2023-10-18T19:30:09.173 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-18T19:30:09.174 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650 --verbose --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} -- /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650/orig.config.yaml 2023-10-18T19:30:09.211 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T19:30:09.288 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650 branch: wip-yuri-testing-2023-10-18-0812-reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7431650' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-10-18-0812-reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 07c1e9af8fb194e5ad8a71499ead549c30345076 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 07c1e9af8fb194e5ad8a71499ead549c30345076 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri-testing-2023-10-18-0812-reef sha1: 07c1e9af8fb194e5ad8a71499ead549c30345076 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 1349 sha1: 07c1e9af8fb194e5ad8a71499ead549c30345076 sleep_before_teardown: 0 subset: 111/120000 suite: rgw suite_branch: wip-yuri-testing-2023-10-18-0812-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 07c1e9af8fb194e5ad8a71499ead549c30345076 targets: smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvyj4AREUB5UtfXpWW0uvjhICdkbhKvR7vk+X63lvYbeR9OxwLhNt/cWMOwhoABu8Uzx6tjPXC/XE/KcEEbuYc= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEoPyxVfTkY/yghFEHuNDraaBPv1OdiYKJ4EOPuHZdPIGA5KTcBjR8Q6W0QaX3LPal9vMY0E9N+bsBvCnjgETOg= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-18_19:20:22 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T19:30:09.289 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa; will attempt to use it 2023-10-18T19:30:09.290 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/tasks 2023-10-18T19:30:09.290 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T19:30:09.292 INFO:teuthology.task.internal:Checking packages... 2023-10-18T19:30:09.315 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '07c1e9af8fb194e5ad8a71499ead549c30345076' 2023-10-18T19:30:09.316 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T19:30:09.316 INFO:teuthology.packaging:ref: None 2023-10-18T19:30:09.316 INFO:teuthology.packaging:tag: None 2023-10-18T19:30:09.317 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-10-18-0812-reef 2023-10-18T19:30:09.317 INFO:teuthology.packaging:sha1: 07c1e9af8fb194e5ad8a71499ead549c30345076 2023-10-18T19:30:09.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-10-18-0812-reef 2023-10-18T19:30:09.538 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-885.g07c1e9af 2023-10-18T19:30:09.538 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T19:30:09.546 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T19:30:09.546 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T19:30:09.555 INFO:teuthology.task.internal:Saving configuration 2023-10-18T19:30:09.570 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T19:30:09.576 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T19:30:09.601 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 19:24:55.861670', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvyj4AREUB5UtfXpWW0uvjhICdkbhKvR7vk+X63lvYbeR9OxwLhNt/cWMOwhoABu8Uzx6tjPXC/XE/KcEEbuYc='} 2023-10-18T19:30:09.624 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 19:24:55.863459', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEoPyxVfTkY/yghFEHuNDraaBPv1OdiYKJ4EOPuHZdPIGA5KTcBjR8Q6W0QaX3LPal9vMY0E9N+bsBvCnjgETOg='} 2023-10-18T19:30:09.625 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T19:30:09.630 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-10-18T19:30:09.631 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-10-18T19:30:09.631 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T19:30:09.705 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f34b307ea60>, signals=[15]) 2023-10-18T19:30:09.706 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T19:30:09.713 INFO:teuthology.task.internal:Opening connections... 2023-10-18T19:30:09.714 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T19:30:09.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:30:09.783 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T19:30:09.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:30:09.850 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T19:30:09.857 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2023-10-18T19:30:09.880 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2023-10-18T19:30:09.880 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2023-10-18T19:30:09.941 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2023-10-18T19:30:09.941 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="8" 2023-10-18T19:30:09.942 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2023-10-18T19:30:09.942 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2023-10-18T19:30:09.942 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="8" 2023-10-18T19:30:09.942 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el8" 2023-10-18T19:30:09.942 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-18T19:30:09.943 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2023-10-18T19:30:09.943 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-18T19:30:09.943 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2023-10-18T19:30:09.943 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T19:30:09.943 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T19:30:09.944 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-18T19:30:09.944 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2023-10-18T19:30:09.972 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2023-10-18T19:30:09.995 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2023-10-18T19:30:09.995 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2023-10-18T19:30:10.056 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2023-10-18T19:30:10.056 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="8" 2023-10-18T19:30:10.057 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2023-10-18T19:30:10.057 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2023-10-18T19:30:10.057 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="8" 2023-10-18T19:30:10.057 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el8" 2023-10-18T19:30:10.057 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-18T19:30:10.058 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2023-10-18T19:30:10.058 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-18T19:30:10.058 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2023-10-18T19:30:10.058 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T19:30:10.058 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T19:30:10.059 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-18T19:30:10.059 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2023-10-18T19:30:10.080 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T19:30:10.089 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T19:30:10.098 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T19:30:10.098 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2023-10-18T19:30:10.101 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2023-10-18T19:30:10.121 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T19:30:10.128 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T19:30:10.128 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2023-10-18T19:30:10.162 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2023-10-18T19:30:10.193 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T19:30:10.256 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T19:30:10.274 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-18T19:30:10.275 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-18T19:30:10.275 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T19:30:10.276 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T19:30:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T19:30:10.296 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2023-10-18T19:30:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T19:30:10.297 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2023-10-18T19:30:10.354 INFO:teuthology.orchestra.run.smithi156.stdout:4.18.0-489.el8.x86_64 2023-10-18T19:30:10.355 INFO:teuthology.task.kernel:Running kernel on smithi156: 4.18.0-489.el8.x86_64 2023-10-18T19:30:10.355 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2023-10-18T19:30:10.358 INFO:teuthology.orchestra.run.smithi082.stdout:4.18.0-489.el8.x86_64 2023-10-18T19:30:10.358 INFO:teuthology.task.kernel:Running kernel on smithi082: 4.18.0-489.el8.x86_64 2023-10-18T19:30:10.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2023-10-18T19:30:16.309 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - AppStream 10 MB/s | 33 MB 00:03 2023-10-18T19:30:17.047 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - AppStream 8.3 MB/s | 33 MB 00:04 2023-10-18T19:30:25.447 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 50 MB 00:02 2023-10-18T19:30:26.300 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 50 MB 00:03 2023-10-18T19:30:32.737 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras 87 kB/s | 18 kB 00:00 2023-10-18T19:30:33.221 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2023-10-18T19:30:33.531 INFO:teuthology.orchestra.run.smithi082.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-10-18T19:30:33.635 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - Extras 80 kB/s | 18 kB 00:00 2023-10-18T19:30:34.092 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 6.9 kB 00:00 2023-10-18T19:30:34.369 INFO:teuthology.orchestra.run.smithi156.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-10-18T19:30:35.381 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 16 MB 00:01 2023-10-18T19:30:39.140 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 549 kB/s | 24 kB 00:00 2023-10-18T19:30:42.656 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-18T19:30:42.656 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-18T19:30:42.938 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:30:42.939 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:30:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2023-10-18T19:30:42.939 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:30:42.939 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:30:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-18T19:30:42.940 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-10-18T19:30:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-18T19:30:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-18T19:30:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:30:42.941 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:30:42.941 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:30:42.941 INFO:teuthology.orchestra.run.smithi082.stdout:Install 3 Packages 2023-10-18T19:30:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:30:42.942 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 89 M 2023-10-18T19:30:42.942 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 96 M 2023-10-18T19:30:42.942 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:30:43.663 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 1.7 MB/s | 16 MB 00:09 2023-10-18T19:30:45.382 INFO:teuthology.orchestra.run.smithi082.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 4.3 MB/s | 10 MB 00:02 2023-10-18T19:30:47.452 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 423 kB/s | 24 kB 00:00 2023-10-18T19:30:48.149 INFO:teuthology.orchestra.run.smithi082.stdout:(2/3): kernel-core-4.18.0-517.el8.x86_64.rpm 8.4 MB/s | 43 MB 00:05 2023-10-18T19:30:48.824 INFO:teuthology.orchestra.run.smithi082.stdout:(3/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 6.2 MB/s | 36 MB 00:05 2023-10-18T19:30:48.824 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:30:48.824 INFO:teuthology.orchestra.run.smithi082.stdout:Total 15 MB/s | 89 MB 00:05 2023-10-18T19:30:49.498 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:30:49.895 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:30:49.895 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:30:51.010 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-18T19:30:51.010 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-18T19:30:51.120 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:30:51.150 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:30:51.269 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:30:51.269 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:30:51.270 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2023-10-18T19:30:51.270 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:30:51.270 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:30:51.270 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-18T19:30:51.271 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-10-18T19:30:51.271 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-18T19:30:51.271 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-18T19:30:51.271 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:30:51.271 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:30:51.272 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:30:51.272 INFO:teuthology.orchestra.run.smithi156.stdout:Install 3 Packages 2023-10-18T19:30:51.272 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:30:51.273 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 89 M 2023-10-18T19:30:51.273 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 96 M 2023-10-18T19:30:51.273 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:30:52.612 INFO:teuthology.orchestra.run.smithi156.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 8.8 MB/s | 10 MB 00:01 2023-10-18T19:30:54.104 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:30:54.605 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T19:30:56.278 INFO:teuthology.orchestra.run.smithi156.stdout:(2/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 7.4 MB/s | 36 MB 00:04 2023-10-18T19:30:57.095 INFO:teuthology.orchestra.run.smithi156.stdout:(3/3): kernel-core-4.18.0-517.el8.x86_64.rpm 7.6 MB/s | 43 MB 00:05 2023-10-18T19:30:57.095 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:30:57.096 INFO:teuthology.orchestra.run.smithi156.stdout:Total 15 MB/s | 89 MB 00:05 2023-10-18T19:30:57.765 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:30:58.181 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:30:58.181 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:30:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T19:30:58.596 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T19:30:59.415 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:30:59.452 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:31:01.975 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T19:31:02.226 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:31:02.260 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:02.749 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T19:31:06.356 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T19:31:06.789 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T19:31:10.174 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T19:31:10.471 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:24.141 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:24.164 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:24.490 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:24.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-18T19:31:24.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-18T19:31:25.201 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:25.202 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:31:25.202 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:31:25.202 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-18T19:31:25.202 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-18T19:31:25.203 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:31:25.203 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:31:25.340 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T19:31:25.557 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T19:31:25.558 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T19:31:25.558 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-18T19:31:25.558 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T19:31:25.559 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T19:31:25.559 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-10-18T19:31:25.559 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T19:31:25.560 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T19:31:25.768 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T19:31:25.768 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T19:31:25.769 DEBUG:teuthology.orchestra.run.smithi082:> sudo rpm -qi grub2-tools 2023-10-18T19:31:25.802 INFO:teuthology.orchestra.run.smithi082.stdout:Name : grub2-tools 2023-10-18T19:31:25.802 INFO:teuthology.orchestra.run.smithi082.stdout:Epoch : 1 2023-10-18T19:31:25.803 INFO:teuthology.orchestra.run.smithi082.stdout:Version : 2.02 2023-10-18T19:31:25.803 INFO:teuthology.orchestra.run.smithi082.stdout:Release : 129.el8 2023-10-18T19:31:25.803 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: x86_64 2023-10-18T19:31:25.803 INFO:teuthology.orchestra.run.smithi082.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-18T19:31:25.804 INFO:teuthology.orchestra.run.smithi082.stdout:Group : System Environment/Base 2023-10-18T19:31:25.804 INFO:teuthology.orchestra.run.smithi082.stdout:Size : 9265586 2023-10-18T19:31:25.804 INFO:teuthology.orchestra.run.smithi082.stdout:License : GPLv3+ 2023-10-18T19:31:25.804 INFO:teuthology.orchestra.run.smithi082.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-18T19:31:25.804 INFO:teuthology.orchestra.run.smithi082.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-18T19:31:25.804 INFO:teuthology.orchestra.run.smithi082.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-18T19:31:25.805 INFO:teuthology.orchestra.run.smithi082.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-18T19:31:25.805 INFO:teuthology.orchestra.run.smithi082.stdout:Relocations : (not relocatable) 2023-10-18T19:31:25.805 INFO:teuthology.orchestra.run.smithi082.stdout:Packager : CentOS BuildSystem 2023-10-18T19:31:25.805 INFO:teuthology.orchestra.run.smithi082.stdout:Vendor : CentOS 2023-10-18T19:31:25.805 INFO:teuthology.orchestra.run.smithi082.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T19:31:25.806 INFO:teuthology.orchestra.run.smithi082.stdout:Summary : Support tools for GRUB. 2023-10-18T19:31:25.806 INFO:teuthology.orchestra.run.smithi082.stdout:Description : 2023-10-18T19:31:25.806 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:31:25.806 INFO:teuthology.orchestra.run.smithi082.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T19:31:25.806 INFO:teuthology.orchestra.run.smithi082.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T19:31:25.807 INFO:teuthology.orchestra.run.smithi082.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T19:31:25.807 INFO:teuthology.orchestra.run.smithi082.stdout:hardware devices. 2023-10-18T19:31:25.807 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:31:25.807 INFO:teuthology.orchestra.run.smithi082.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T19:31:25.808 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T19:31:25.808 INFO:teuthology.task.kernel:Updating grub on smithi082 to boot 4.18.0-517.el8.x86_64 2023-10-18T19:31:25.809 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T19:31:27.472 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2023-10-18T19:31:29.762 INFO:teuthology.orchestra.run.smithi082.stderr:done 2023-10-18T19:31:29.764 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2023-10-18T19:31:29.796 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.dvbZjuPgYT 2023-10-18T19:31:29.797 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dvbZjuPgYT 2023-10-18T19:31:29.871 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.dvbZjuPgYT 2023-10-18T19:31:30.013 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.dvbZjuPgYT is 6KB 2023-10-18T19:31:30.062 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.dvbZjuPgYT 2023-10-18T19:31:30.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T19:31:30.152 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-18T19:31:30.153 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-18T19:31:30.153 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-18T19:31:30.153 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-18T19:31:30.154 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-18T19:31:30.307 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2023-10-18T19:31:32.204 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:32.228 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:32.581 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:32.581 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-18T19:31:32.581 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-18T19:31:33.366 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T19:31:33.367 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:31:33.367 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:31:33.367 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-18T19:31:33.367 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-18T19:31:33.367 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:31:33.368 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:31:33.525 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T19:31:33.738 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T19:31:33.739 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T19:31:33.739 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-18T19:31:33.739 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T19:31:33.739 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T19:31:33.739 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-10-18T19:31:33.740 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T19:31:33.740 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T19:31:33.949 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T19:31:33.949 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T19:31:33.950 DEBUG:teuthology.orchestra.run.smithi156:> sudo rpm -qi grub2-tools 2023-10-18T19:31:33.984 INFO:teuthology.orchestra.run.smithi156.stdout:Name : grub2-tools 2023-10-18T19:31:33.984 INFO:teuthology.orchestra.run.smithi156.stdout:Epoch : 1 2023-10-18T19:31:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:Version : 2.02 2023-10-18T19:31:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:Release : 129.el8 2023-10-18T19:31:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: x86_64 2023-10-18T19:31:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-18T19:31:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:Group : System Environment/Base 2023-10-18T19:31:33.986 INFO:teuthology.orchestra.run.smithi156.stdout:Size : 9265586 2023-10-18T19:31:33.986 INFO:teuthology.orchestra.run.smithi156.stdout:License : GPLv3+ 2023-10-18T19:31:33.986 INFO:teuthology.orchestra.run.smithi156.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-18T19:31:33.986 INFO:teuthology.orchestra.run.smithi156.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-18T19:31:33.986 INFO:teuthology.orchestra.run.smithi156.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-18T19:31:33.987 INFO:teuthology.orchestra.run.smithi156.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-18T19:31:33.987 INFO:teuthology.orchestra.run.smithi156.stdout:Relocations : (not relocatable) 2023-10-18T19:31:33.987 INFO:teuthology.orchestra.run.smithi156.stdout:Packager : CentOS BuildSystem 2023-10-18T19:31:33.987 INFO:teuthology.orchestra.run.smithi156.stdout:Vendor : CentOS 2023-10-18T19:31:33.987 INFO:teuthology.orchestra.run.smithi156.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T19:31:33.987 INFO:teuthology.orchestra.run.smithi156.stdout:Summary : Support tools for GRUB. 2023-10-18T19:31:33.988 INFO:teuthology.orchestra.run.smithi156.stdout:Description : 2023-10-18T19:31:33.988 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:31:33.988 INFO:teuthology.orchestra.run.smithi156.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T19:31:33.988 INFO:teuthology.orchestra.run.smithi156.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T19:31:33.988 INFO:teuthology.orchestra.run.smithi156.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T19:31:33.989 INFO:teuthology.orchestra.run.smithi156.stdout:hardware devices. 2023-10-18T19:31:33.989 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:31:33.989 INFO:teuthology.orchestra.run.smithi156.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T19:31:33.990 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T19:31:33.990 INFO:teuthology.task.kernel:Updating grub on smithi156 to boot 4.18.0-517.el8.x86_64 2023-10-18T19:31:33.991 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T19:31:35.758 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2023-10-18T19:31:38.071 INFO:teuthology.orchestra.run.smithi156.stderr:done 2023-10-18T19:31:38.073 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2023-10-18T19:31:38.136 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.Yu5eMhKcTY 2023-10-18T19:31:38.137 DEBUG:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Yu5eMhKcTY 2023-10-18T19:31:38.211 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.Yu5eMhKcTY 2023-10-18T19:31:38.356 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.Yu5eMhKcTY is 6KB 2023-10-18T19:31:38.404 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.Yu5eMhKcTY 2023-10-18T19:31:38.422 DEBUG:teuthology.orchestra.run.smithi156:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T19:31:38.497 INFO:teuthology.orchestra.run.smithi156.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-18T19:31:38.497 INFO:teuthology.orchestra.run.smithi156.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-18T19:31:38.497 INFO:teuthology.orchestra.run.smithi156.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-18T19:31:38.497 INFO:teuthology.orchestra.run.smithi156.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-18T19:31:38.499 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-18T19:31:38.648 DEBUG:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2023-10-18T19:32:00.330 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-10-18T19:32:00.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-10-18T19:32:00.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:32:08.659 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-10-18T19:32:08.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-10-18T19:32:08.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:32:18.682 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2023-10-18T19:32:27.130 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-10-18T19:32:27.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-10-18T19:32:27.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:32:30.746 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2023-10-18T19:32:36.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-10-18T19:32:36.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:32:39.194 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-10-18T19:32:42.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-10-18T19:32:42.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:32:51.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-10-18T19:32:51.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:33:09.631 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-10-18T19:33:14.907 DEBUG:teuthology.orchestra.run.smithi082:> true 2023-10-18T19:33:15.150 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-10-18T19:33:15.150 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-517.el8.x86_64"... 2023-10-18T19:33:15.151 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2023-10-18T19:33:15.168 INFO:teuthology.orchestra.run.smithi082.stdout:4.18.0-517.el8.x86_64 2023-10-18T19:33:15.169 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-18T19:33:15.169 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T19:33:15.169 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-18T19:33:16.171 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-10-18T19:33:16.171 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T19:33:16.250 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2023-10-18T19:33:16.264 DEBUG:teuthology.parallel:result is None 2023-10-18T19:33:24.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-10-18T19:33:24.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:33:24.965 DEBUG:teuthology.orchestra.run.smithi156:> true 2023-10-18T19:33:25.278 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-10-18T19:33:25.279 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-517.el8.x86_64"... 2023-10-18T19:33:25.279 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2023-10-18T19:33:25.297 INFO:teuthology.orchestra.run.smithi156.stdout:4.18.0-517.el8.x86_64 2023-10-18T19:33:25.297 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-18T19:33:25.297 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T19:33:25.298 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-18T19:33:26.298 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-10-18T19:33:26.299 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T19:33:26.383 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2023-10-18T19:33:26.396 DEBUG:teuthology.parallel:result is None 2023-10-18T19:33:26.397 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T19:33:26.404 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T19:33:26.405 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T19:33:26.407 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T19:33:26.427 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T19:33:26.438 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T19:33:26.444 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T19:33:26.444 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T19:33:26.470 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T19:33:26.504 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T19:33:26.511 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T19:33:26.511 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T19:33:26.542 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T19:33:26.593 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T19:33:26.603 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T19:33:26.607 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T19:33:26.618 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T19:33:26.619 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T19:33:26.626 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T19:33:26.627 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T19:33:26.629 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T19:33:26.658 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T19:33:26.668 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T19:33:26.669 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T19:33:26.701 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T19:33:26.718 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T19:33:26.805 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T19:33:26.856 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T19:33:26.937 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T19:33:26.985 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:33:26.986 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T19:33:27.053 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T19:33:27.100 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T19:33:27.143 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T19:33:27.223 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T19:33:27.265 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:33:27.266 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T19:33:27.331 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2023-10-18T19:33:27.334 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2023-10-18T19:33:27.405 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T19:33:27.427 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T19:33:27.685 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T19:33:27.693 INFO:teuthology.task.internal:Starting timer... 2023-10-18T19:33:27.693 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T19:33:27.703 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T19:33:27.712 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T19:33:27.713 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2023-10-18T19:33:27.804 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2023-10-18T19:33:27.806 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2023-10-18T19:33:27.978 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2023-10-18T19:33:27.979 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T19:33:27.980 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2023-10-18T19:33:28.038 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2023-10-18T19:33:28.038 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2023-10-18T19:33:28.079 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2023-10-18T19:33:28.079 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi082.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2023-10-18T19:33:28.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T19:33:28.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T19:33:28.110 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T19:33:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T19:33:28.390 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T19:33:28.391 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2023-10-18T19:33:28.422 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2023-10-18T19:33:28.843 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T19:33:28.853 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-18T19:33:28.858 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-18T19:33:29.008 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T19:33:29.026 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T19:33:29.027 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi082.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T19:40:52.131 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi082.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2023-10-18T19:40:52.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-10-18T19:40:52.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:40:52.205 DEBUG:teuthology.orchestra.run.smithi082:> true 2023-10-18T19:40:52.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-10-18T19:40:52.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-10-18T19:40:52.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T19:40:52.340 DEBUG:teuthology.orchestra.run.smithi156:> true 2023-10-18T19:40:52.408 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-10-18T19:40:52.409 INFO:teuthology.run_tasks:Running task clock... 2023-10-18T19:40:52.425 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-18T19:40:52.425 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-18T19:40:52.426 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T19:40:52.428 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-18T19:40:52.429 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T19:40:52.460 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-18T19:40:52.474 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-18T19:40:52.479 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-18T19:40:52.494 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-18T19:40:52.543 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2023-10-18T19:40:52.555 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2023-10-18T19:40:52.556 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2023-10-18T19:40:52.567 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2023-10-18T19:40:52.569 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-18T19:40:52.581 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-18T19:40:52.582 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-18T19:40:52.595 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-18T19:40:52.649 INFO:teuthology.orchestra.run.smithi082.stderr:bash: ntpq: command not found 2023-10-18T19:40:52.653 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T19:40:52.653 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2023-10-18T19:40:52.653 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T19:40:52.654 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T19:40:52.654 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T19:40:52.654 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T19:40:52.666 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2023-10-18T19:40:52.669 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T19:40:52.669 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-10-18T19:40:52.670 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T19:40:52.670 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T19:40:52.670 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T19:40:52.670 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T19:40:52.671 INFO:teuthology.run_tasks:Running task install... 2023-10-18T19:40:52.679 DEBUG:teuthology.task.install:project ceph 2023-10-18T19:40:52.680 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076'}} 2023-10-18T19:40:52.680 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076'} 2023-10-18T19:40:52.680 INFO:teuthology.task.install:Using flavor: default 2023-10-18T19:40:52.686 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-18T19:40:52.687 INFO:teuthology.task.install:extra packages: [] 2023-10-18T19:40:52.687 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'tag': None, 'wait_for_package': False} 2023-10-18T19:40:52.688 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=07c1e9af8fb194e5ad8a71499ead549c30345076 2023-10-18T19:40:52.690 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'tag': None, 'wait_for_package': False} 2023-10-18T19:40:52.690 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=07c1e9af8fb194e5ad8a71499ead549c30345076 2023-10-18T19:40:52.838 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-18-0812-reef/07c1e9af8fb194e5ad8a71499ead549c30345076/centos/8/flavors/default/ 2023-10-18T19:40:52.838 INFO:teuthology.task.install.rpm:Package version is 18.2.0-885.g07c1e9af 2023-10-18T19:40:52.852 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-18-0812-reef/07c1e9af8fb194e5ad8a71499ead549c30345076/centos/8/flavors/default/ 2023-10-18T19:40:52.853 INFO:teuthology.task.install.rpm:Package version is 18.2.0-885.g07c1e9af 2023-10-18T19:40:53.031 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-18-0812-reef/07c1e9af8fb194e5ad8a71499ead549c30345076/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-18-0812-reef/07c1e9af8fb194e5ad8a71499ead549c30345076/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-18-0812-reef/07c1e9af8fb194e5ad8a71499ead549c30345076/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-18T19:40:53.031 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:40:53.032 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-18T19:40:53.039 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-18-0812-reef/07c1e9af8fb194e5ad8a71499ead549c30345076/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-18-0812-reef/07c1e9af8fb194e5ad8a71499ead549c30345076/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-18-0812-reef/07c1e9af8fb194e5ad8a71499ead549c30345076/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-18T19:40:53.039 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:40:53.039 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-18T19:40:53.060 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-18T19:40:53.060 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/07c1e9af8fb194e5ad8a71499ead549c30345076/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-18T19:40:53.068 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-18T19:40:53.069 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/07c1e9af8fb194e5ad8a71499ead549c30345076/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-18T19:40:53.129 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-18T19:40:53.139 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-18T19:40:53.206 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-18T19:40:53.218 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-18T19:40:53.274 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2023-10-18T19:40:53.275 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2023-10-18T19:40:53.286 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2023-10-18T19:40:53.287 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2023-10-18T19:40:53.541 INFO:teuthology.orchestra.run.smithi156.stdout:76 files removed 2023-10-18T19:40:53.553 INFO:teuthology.orchestra.run.smithi082.stdout:76 files removed 2023-10-18T19:40:53.569 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw 2023-10-18T19:40:53.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw 2023-10-18T19:40:54.272 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 283 kB/s | 80 kB 00:00 2023-10-18T19:40:54.348 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 221 kB/s | 80 kB 00:00 2023-10-18T19:40:54.623 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 30 kB/s | 9.5 kB 00:00 2023-10-18T19:40:54.665 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 34 kB/s | 9.5 kB 00:00 2023-10-18T19:40:55.023 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 4.0 kB/s | 1.5 kB 00:00 2023-10-18T19:40:55.058 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 4.1 kB/s | 1.5 kB 00:00 2023-10-18T19:40:57.033 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - AppStream 17 MB/s | 33 MB 00:01 2023-10-18T19:40:57.298 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - AppStream 15 MB/s | 33 MB 00:02 2023-10-18T19:41:11.833 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - BaseOS 5.7 MB/s | 50 MB 00:08 2023-10-18T19:41:12.998 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - BaseOS 5.2 MB/s | 50 MB 00:09 2023-10-18T19:41:19.392 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2023-10-18T19:41:19.709 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - Extras common packages 49 kB/s | 6.9 kB 00:00 2023-10-18T19:41:20.364 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras 80 kB/s | 18 kB 00:00 2023-10-18T19:41:20.460 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - PowerTools 11 MB/s | 6.5 MB 00:00 2023-10-18T19:41:20.831 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 6.9 kB 00:00 2023-10-18T19:41:21.649 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - PowerTools 9.7 MB/s | 6.5 MB 00:00 2023-10-18T19:41:21.862 INFO:teuthology.orchestra.run.smithi156.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-10-18T19:41:23.100 INFO:teuthology.orchestra.run.smithi082.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-10-18T19:41:25.996 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 4.0 MB/s | 16 MB 00:03 2023-10-18T19:41:29.904 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 385 kB/s | 24 kB 00:00 2023-10-18T19:41:34.359 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:41:34.362 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:41:34.362 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:41:34.363 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:41:34.363 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:41:34.363 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 13 M 2023-10-18T19:41:34.363 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2023-10-18T19:41:34.363 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 3.7 M 2023-10-18T19:41:34.364 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 3.6 M 2023-10-18T19:41:34.364 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-10-18T19:41:34.364 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 5.4 M 2023-10-18T19:41:34.364 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 22 M 2023-10-18T19:41:34.364 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 24 k 2023-10-18T19:41:34.365 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-18T19:41:34.365 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 771 k 2023-10-18T19:41:34.365 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-18T19:41:34.365 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-18T19:41:34.365 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 475 k 2023-10-18T19:41:34.366 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-18T19:41:34.366 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 6.5 M 2023-10-18T19:41:34.366 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-18T19:41:34.366 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-18T19:41:34.366 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 45 k 2023-10-18T19:41:34.367 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 123 k 2023-10-18T19:41:34.367 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 190 k 2023-10-18T19:41:34.367 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-18T19:41:34.367 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 370 k 2023-10-18T19:41:34.367 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 364 k 2023-10-18T19:41:34.368 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 107 k 2023-10-18T19:41:34.368 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-18T19:41:34.368 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:41:34.368 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:41:34.368 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:41:34.369 INFO:teuthology.orchestra.run.smithi156.stdout:Install 21 Packages 2023-10-18T19:41:34.369 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 2 Packages 2023-10-18T19:41:34.369 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:41:34.370 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 59 M 2023-10-18T19:41:34.370 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:41:37.672 INFO:teuthology.orchestra.run.smithi156.stdout:(1/23): ceph-common-18.2.0-885.g07c1e9af.el8.x8 8.6 MB/s | 22 MB 00:02 2023-10-18T19:41:37.789 INFO:teuthology.orchestra.run.smithi156.stdout:(2/23): ceph-base-18.2.0-885.g07c1e9af.el8.x86_ 2.0 MB/s | 5.4 MB 00:02 2023-10-18T19:41:37.806 INFO:teuthology.orchestra.run.smithi156.stdout:(3/23): ceph-selinux-18.2.0-885.g07c1e9af.el8.x 183 kB/s | 24 kB 00:00 2023-10-18T19:41:37.956 INFO:teuthology.orchestra.run.smithi156.stdout:(4/23): libradosstriper1-18.2.0-885.g07c1e9af.e 3.1 MB/s | 475 kB 00:00 2023-10-18T19:41:38.082 INFO:teuthology.orchestra.run.smithi156.stdout:(5/23): libcephfs2-18.2.0-885.g07c1e9af.el8.x86 2.6 MB/s | 771 kB 00:00 2023-10-18T19:41:38.190 INFO:teuthology.orchestra.run.smithi156.stdout:(6/23): python3-ceph-argparse-18.2.0-885.g07c1e 412 kB/s | 45 kB 00:00 2023-10-18T19:41:38.307 INFO:teuthology.orchestra.run.smithi156.stdout:(7/23): python3-ceph-common-18.2.0-885.g07c1e9a 1.0 MB/s | 123 kB 00:00 2023-10-18T19:41:38.508 INFO:teuthology.orchestra.run.smithi156.stdout:(8/23): ceph-radosgw-18.2.0-885.g07c1e9af.el8.x 3.9 MB/s | 13 MB 00:03 2023-10-18T19:41:38.533 INFO:teuthology.orchestra.run.smithi156.stdout:(9/23): python3-cephfs-18.2.0-885.g07c1e9af.el8 844 kB/s | 190 kB 00:00 2023-10-18T19:41:38.719 INFO:teuthology.orchestra.run.smithi156.stdout:(10/23): librgw2-18.2.0-885.g07c1e9af.el8.x86_6 8.6 MB/s | 6.5 MB 00:00 2023-10-18T19:41:38.750 INFO:teuthology.orchestra.run.smithi156.stdout:(11/23): python3-rados-18.2.0-885.g07c1e9af.el8 1.5 MB/s | 370 kB 00:00 2023-10-18T19:41:38.775 INFO:teuthology.orchestra.run.smithi156.stdout:(12/23): python3-rgw-18.2.0-885.g07c1e9af.el8.x 1.9 MB/s | 107 kB 00:00 2023-10-18T19:41:38.809 INFO:teuthology.orchestra.run.smithi156.stdout:(13/23): python3-rbd-18.2.0-885.g07c1e9af.el8.x 1.3 MB/s | 364 kB 00:00 2023-10-18T19:41:39.001 INFO:teuthology.orchestra.run.smithi156.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 229 kB/s | 44 kB 00:00 2023-10-18T19:41:39.101 INFO:teuthology.orchestra.run.smithi156.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 798 kB/s | 259 kB 00:00 2023-10-18T19:41:39.147 INFO:teuthology.orchestra.run.smithi156.stdout:(16/23): librdkafka-0.11.4-3.el8.x86_64.rpm 893 kB/s | 354 kB 00:00 2023-10-18T19:41:39.177 INFO:teuthology.orchestra.run.smithi156.stdout:(17/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 267 kB/s | 47 kB 00:00 2023-10-18T19:41:39.736 INFO:teuthology.orchestra.run.smithi156.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 100 kB/s | 59 kB 00:00 2023-10-18T19:41:39.761 INFO:teuthology.orchestra.run.smithi156.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 129 kB/s | 75 kB 00:00 2023-10-18T19:41:39.961 INFO:teuthology.orchestra.run.smithi156.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 356 kB/s | 306 kB 00:00 2023-10-18T19:41:40.439 INFO:teuthology.orchestra.run.smithi156.stdout:(21/23): librbd1-18.2.0-885.g07c1e9af.el8.x86_6 7.5 MB/s | 3.6 MB 00:00 2023-10-18T19:41:40.728 INFO:teuthology.orchestra.run.smithi156.stdout:(22/23): librados2-18.2.0-885.g07c1e9af.el8.x86 3.8 MB/s | 3.7 MB 00:00 2023-10-18T19:41:42.112 INFO:teuthology.orchestra.run.smithi156.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 748 kB/s | 1.7 MB 00:02 2023-10-18T19:41:42.116 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:41:42.117 INFO:teuthology.orchestra.run.smithi156.stdout:Total 7.6 MB/s | 59 MB 00:07 2023-10-18T19:41:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:41:42.192 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 860 kB/s | 16 MB 00:18 2023-10-18T19:41:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:41:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:41:42.590 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:41:42.593 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:41:43.837 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:41:44.038 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-18T19:41:44.235 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-10-18T19:41:44.440 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-18T19:41:44.843 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-18T19:41:45.032 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 4/25 2023-10-18T19:41:45.112 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 4/25 2023-10-18T19:41:45.470 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:18.2.0-885.g07c1e9af.el8.x86_64 5/25 2023-10-18T19:41:45.707 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 6/25 2023-10-18T19:41:46.074 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 256 kB/s | 24 kB 00:00 2023-10-18T19:41:46.121 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 6/25 2023-10-18T19:41:46.306 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 7/25 2023-10-18T19:41:46.368 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 7/25 2023-10-18T19:41:46.596 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-18T19:41:46.764 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-18T19:41:46.844 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-18T19:41:47.008 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:18.2.0-885.g07c1e9af.el8.x 10/25 2023-10-18T19:41:47.200 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:18.2.0-885.g07c1e9af.el8.x86_64 11/25 2023-10-18T19:41:47.415 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:18.2.0-885.g07c1e9af.el8.x86_64 12/25 2023-10-18T19:41:47.573 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:18.2.0-885.g07c1e9af.el8.x86_64 13/25 2023-10-18T19:41:47.645 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:18.2.0-885.g07c1e9af.el8.x86_64 13/25 2023-10-18T19:41:47.890 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-10-18T19:41:48.645 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-10-18T19:41:48.789 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:18.2.0-885.g07c1e9af.el8.x86_64 16/25 2023-10-18T19:41:48.866 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:18.2.0-885.g07c1e9af.el8.x86_64 16/25 2023-10-18T19:41:49.072 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:18.2.0-885.g07c1e9af.el8.x86_64 17/25 2023-10-18T19:41:49.248 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-18T19:41:49.431 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:18.2.0-885.g07c1e9af.el8.x86 19/25 2023-10-18T19:41:50.578 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:41:50.581 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:41:50.581 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:41:50.581 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:41:50.581 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:41:50.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 13 M 2023-10-18T19:41:50.582 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2023-10-18T19:41:50.582 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 3.7 M 2023-10-18T19:41:50.582 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 3.6 M 2023-10-18T19:41:50.583 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-10-18T19:41:50.583 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 5.4 M 2023-10-18T19:41:50.583 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 22 M 2023-10-18T19:41:50.584 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 24 k 2023-10-18T19:41:50.584 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-18T19:41:50.584 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 771 k 2023-10-18T19:41:50.584 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-18T19:41:50.585 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-18T19:41:50.585 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 475 k 2023-10-18T19:41:50.585 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-18T19:41:50.585 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 6.5 M 2023-10-18T19:41:50.585 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-18T19:41:50.585 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-18T19:41:50.586 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 45 k 2023-10-18T19:41:50.586 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 123 k 2023-10-18T19:41:50.586 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 190 k 2023-10-18T19:41:50.586 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-18T19:41:50.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 370 k 2023-10-18T19:41:50.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 364 k 2023-10-18T19:41:50.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 107 k 2023-10-18T19:41:50.587 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-18T19:41:50.587 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:41:50.587 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:41:50.588 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:41:50.588 INFO:teuthology.orchestra.run.smithi082.stdout:Install 21 Packages 2023-10-18T19:41:50.588 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 2 Packages 2023-10-18T19:41:50.588 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:41:50.589 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 59 M 2023-10-18T19:41:50.589 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:41:51.856 INFO:teuthology.orchestra.run.smithi082.stdout:(1/23): ceph-base-18.2.0-885.g07c1e9af.el8.x86_ 8.8 MB/s | 5.4 MB 00:00 2023-10-18T19:41:51.915 INFO:teuthology.orchestra.run.smithi082.stdout:(2/23): ceph-selinux-18.2.0-885.g07c1e9af.el8.x 419 kB/s | 24 kB 00:00 2023-10-18T19:41:51.981 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 20/25 2023-10-18T19:41:52.041 INFO:teuthology.orchestra.run.smithi082.stdout:(3/23): libcephfs2-18.2.0-885.g07c1e9af.el8.x86 6.0 MB/s | 771 kB 00:00 2023-10-18T19:41:52.124 INFO:teuthology.orchestra.run.smithi082.stdout:(4/23): libradosstriper1-18.2.0-885.g07c1e9af.e 5.6 MB/s | 475 kB 00:00 2023-10-18T19:41:52.173 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 20/25 2023-10-18T19:41:52.405 INFO:teuthology.orchestra.run.smithi082.stdout:(5/23): ceph-radosgw-18.2.0-885.g07c1e9af.el8.x 11 MB/s | 13 MB 00:01 2023-10-18T19:41:52.534 INFO:teuthology.orchestra.run.smithi082.stdout:(6/23): python3-ceph-argparse-18.2.0-885.g07c1e 349 kB/s | 45 kB 00:00 2023-10-18T19:41:52.642 INFO:teuthology.orchestra.run.smithi082.stdout:(7/23): librgw2-18.2.0-885.g07c1e9af.el8.x86_64 13 MB/s | 6.5 MB 00:00 2023-10-18T19:41:52.668 INFO:teuthology.orchestra.run.smithi082.stdout:(8/23): python3-ceph-common-18.2.0-885.g07c1e9a 926 kB/s | 123 kB 00:00 2023-10-18T19:41:52.693 INFO:teuthology.orchestra.run.smithi082.stdout:(9/23): python3-cephfs-18.2.0-885.g07c1e9af.el8 3.7 MB/s | 190 kB 00:00 2023-10-18T19:41:52.718 INFO:teuthology.orchestra.run.smithi082.stdout:(10/23): python3-rados-18.2.0-885.g07c1e9af.el8 7.2 MB/s | 370 kB 00:00 2023-10-18T19:41:52.743 INFO:teuthology.orchestra.run.smithi082.stdout:(11/23): python3-rbd-18.2.0-885.g07c1e9af.el8.x 7.1 MB/s | 364 kB 00:00 2023-10-18T19:41:52.768 INFO:teuthology.orchestra.run.smithi082.stdout:(12/23): python3-rgw-18.2.0-885.g07c1e9af.el8.x 2.1 MB/s | 107 kB 00:00 2023-10-18T19:41:52.790 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 20/25 2023-10-18T19:41:53.063 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 21/25 2023-10-18T19:41:53.169 INFO:teuthology.orchestra.run.smithi082.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 649 kB/s | 259 kB 00:00 2023-10-18T19:41:53.170 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 21/25 2023-10-18T19:41:53.294 INFO:teuthology.orchestra.run.smithi082.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 643 kB/s | 354 kB 00:00 2023-10-18T19:41:53.372 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:18.2.0-885.g07c1e9af.el8.x86_64 22/25 2023-10-18T19:41:53.419 INFO:teuthology.orchestra.run.smithi082.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 375 kB/s | 47 kB 00:00 2023-10-18T19:41:53.453 INFO:teuthology.orchestra.run.smithi082.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 155 kB/s | 44 kB 00:00 2023-10-18T19:41:53.770 INFO:teuthology.orchestra.run.smithi082.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 186 kB/s | 59 kB 00:00 2023-10-18T19:41:53.996 INFO:teuthology.orchestra.run.smithi082.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 334 kB/s | 75 kB 00:00 2023-10-18T19:41:54.254 INFO:teuthology.orchestra.run.smithi082.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 367 kB/s | 306 kB 00:00 2023-10-18T19:41:54.554 INFO:teuthology.orchestra.run.smithi082.stdout:(20/23): librados2-18.2.0-885.g07c1e9af.el8.x86 12 MB/s | 3.7 MB 00:00 2023-10-18T19:41:54.763 INFO:teuthology.orchestra.run.smithi082.stdout:(21/23): librbd1-18.2.0-885.g07c1e9af.el8.x86_6 17 MB/s | 3.6 MB 00:00 2023-10-18T19:41:55.521 INFO:teuthology.orchestra.run.smithi082.stdout:(22/23): thrift-0.13.0-2.el8.x86_64.rpm 1.1 MB/s | 1.7 MB 00:01 2023-10-18T19:41:58.354 INFO:teuthology.orchestra.run.smithi082.stdout:(23/23): ceph-common-18.2.0-885.g07c1e9af.el8.x 3.0 MB/s | 22 MB 00:07 2023-10-18T19:41:58.357 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:41:58.357 INFO:teuthology.orchestra.run.smithi082.stdout:Total 7.6 MB/s | 59 MB 00:07 2023-10-18T19:41:58.399 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:41:58.444 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:41:58.445 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:41:58.806 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:41:58.810 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:42:00.169 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:42:00.369 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-18T19:42:00.613 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-10-18T19:42:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-18T19:42:01.229 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-18T19:42:01.401 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 4/25 2023-10-18T19:42:01.538 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 4/25 2023-10-18T19:42:01.832 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:18.2.0-885.g07c1e9af.el8.x86_64 5/25 2023-10-18T19:42:02.035 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 6/25 2023-10-18T19:42:02.443 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 6/25 2023-10-18T19:42:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 7/25 2023-10-18T19:42:02.677 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 7/25 2023-10-18T19:42:02.916 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-18T19:42:03.118 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-18T19:42:03.181 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-18T19:42:03.370 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:18.2.0-885.g07c1e9af.el8.x 10/25 2023-10-18T19:42:03.595 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:18.2.0-885.g07c1e9af.el8.x86_64 11/25 2023-10-18T19:42:03.785 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:18.2.0-885.g07c1e9af.el8.x86_64 12/25 2023-10-18T19:42:03.959 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:18.2.0-885.g07c1e9af.el8.x86_64 13/25 2023-10-18T19:42:04.020 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:18.2.0-885.g07c1e9af.el8.x86_64 13/25 2023-10-18T19:42:04.251 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-10-18T19:42:05.053 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-10-18T19:42:05.234 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:18.2.0-885.g07c1e9af.el8.x86_64 16/25 2023-10-18T19:42:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:18.2.0-885.g07c1e9af.el8.x86_64 16/25 2023-10-18T19:42:05.500 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:18.2.0-885.g07c1e9af.el8.x86_64 17/25 2023-10-18T19:42:05.718 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-18T19:42:05.867 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:18.2.0-885.g07c1e9af.el8.x86 19/25 2023-10-18T19:42:08.532 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 20/25 2023-10-18T19:42:08.750 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 20/25 2023-10-18T19:42:09.332 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 20/25 2023-10-18T19:42:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 21/25 2023-10-18T19:42:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 21/25 2023-10-18T19:42:09.925 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:18.2.0-885.g07c1e9af.el8.x86_64 22/25 2023-10-18T19:42:12.229 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:18.2.0-885.g07c1e9af.el8.x86_64 22/25 2023-10-18T19:42:12.229 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2023-10-18T19:42:12.229 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2023-10-18T19:42:12.230 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2023-10-18T19:42:12.230 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /run 2023-10-18T19:42:12.230 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2023-10-18T19:42:12.230 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2023-10-18T19:42:12.230 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2023-10-18T19:42:12.231 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2023-10-18T19:42:12.231 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2023-10-18T19:42:12.231 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:13.601 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:18.2.0-885.g07c1e9af.el8.x86_64 23/25 2023-10-18T19:42:13.622 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-885.g07c1e9af.el8.x86_64 23/25 2023-10-18T19:42:13.622 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-18T19:42:13.880 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-18T19:42:13.880 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-18T19:42:14.633 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-18T19:42:14.633 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 1/25 2023-10-18T19:42:14.633 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 2/25 2023-10-18T19:42:14.634 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:18.2.0-885.g07c1e9af.el8.x86_64 3/25 2023-10-18T19:42:14.634 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:18.2.0-885.g07c1e9af.el8.x86_64 4/25 2023-10-18T19:42:14.634 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 5/25 2023-10-18T19:42:14.634 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:18.2.0-885.g07c1e9af.el8.x86_64 6/25 2023-10-18T19:42:14.634 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:18.2.0-885.g07c1e9af.el8.x86_64 7/25 2023-10-18T19:42:14.635 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:18.2.0-885.g07c1e9af.el8.x 8/25 2023-10-18T19:42:14.635 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:18.2.0-885.g07c1e9af.el8.x86 9/25 2023-10-18T19:42:14.635 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:18.2.0-885.g07c1e9af.el8.x86_64 10/25 2023-10-18T19:42:14.635 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:18.2.0-885.g07c1e9af.el8.x86_64 11/25 2023-10-18T19:42:14.635 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:18.2.0-885.g07c1e9af.el8.x86_64 12/25 2023-10-18T19:42:14.636 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:18.2.0-885.g07c1e9af.el8.x86_64 13/25 2023-10-18T19:42:14.636 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-18T19:42:14.636 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-18T19:42:14.636 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-18T19:42:14.636 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-18T19:42:14.636 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-18T19:42:14.637 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-18T19:42:14.637 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-18T19:42:14.637 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-18T19:42:14.637 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 22/25 2023-10-18T19:42:14.637 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-18T19:42:14.638 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 24/25 2023-10-18T19:42:15.835 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-18T19:42:15.835 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:15.836 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2023-10-18T19:42:15.836 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.836 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.836 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:42:15.836 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.837 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.837 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.837 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.837 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-18T19:42:15.837 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.837 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-18T19:42:15.838 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-18T19:42:15.838 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.838 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-18T19:42:15.838 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.838 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-18T19:42:15.839 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-18T19:42:15.839 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.839 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.839 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.839 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-18T19:42:15.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:15.840 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-18T19:42:15.840 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:15.841 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:42:16.138 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-test 2023-10-18T19:42:16.997 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:47 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:42:18.597 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:42:18.598 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:18.599 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:42:18.599 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:18.599 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:42:18.599 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 47 M 2023-10-18T19:42:18.600 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-10-18T19:42:18.600 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-18T19:42:18.600 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 137 k 2023-10-18T19:42:18.600 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-18T19:42:18.600 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-18T19:42:18.601 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-18T19:42:18.601 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:18.601 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:42:18.601 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:18.601 INFO:teuthology.orchestra.run.smithi156.stdout:Install 6 Packages 2023-10-18T19:42:18.602 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:18.602 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 48 M 2023-10-18T19:42:18.603 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 211 M 2023-10-18T19:42:18.603 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:42:18.828 INFO:teuthology.orchestra.run.smithi156.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2023-10-18T19:42:18.987 INFO:teuthology.orchestra.run.smithi156.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-10-18T19:42:19.012 INFO:teuthology.orchestra.run.smithi156.stdout:(3/6): libcephsqlite-18.2.0-885.g07c1e9af.el8.x 371 kB/s | 137 kB 00:00 2023-10-18T19:42:19.154 INFO:teuthology.orchestra.run.smithi156.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 499 kB/s | 71 kB 00:00 2023-10-18T19:42:19.254 INFO:teuthology.orchestra.run.smithi156.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.2 MB/s | 323 kB 00:00 2023-10-18T19:42:20.996 INFO:teuthology.orchestra.run.smithi156.stdout:(6/6): ceph-test-18.2.0-885.g07c1e9af.el8.x86_6 20 MB/s | 47 MB 00:02 2023-10-18T19:42:20.998 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:42:20.998 INFO:teuthology.orchestra.run.smithi156.stdout:Total 20 MB/s | 48 MB 00:02 2023-10-18T19:42:21.041 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:42:21.063 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:42:21.063 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:42:21.350 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:42:21.353 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:42:21.826 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:42:22.070 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-18T19:42:22.294 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-18T19:42:22.459 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-18T19:42:22.544 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-18T19:42:22.746 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-18T19:42:22.909 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:18.2.0-885.g07c1e9af.el8.x86_64 5/6 2023-10-18T19:42:27.425 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:18.2.0-885.g07c1e9af.el8.x86_64 5/6 2023-10-18T19:42:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:18.2.0-885.g07c1e9af.el8.x86_64 6/6 2023-10-18T19:42:27.958 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-test-2:18.2.0-885.g07c1e9af.el8.x86_64 6/6 2023-10-18T19:42:27.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:18.2.0-885.g07c1e9af.el8.x86_64 1/6 2023-10-18T19:42:27.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:18.2.0-885.g07c1e9af.el8.x86_64 2/6 2023-10-18T19:42:27.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-18T19:42:27.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-18T19:42:27.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-18T19:42:28.526 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:18.2.0-885.g07c1e9af.el8.x86_64 22/25 2023-10-18T19:42:28.527 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2023-10-18T19:42:28.527 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2023-10-18T19:42:28.527 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /dev 2023-10-18T19:42:28.527 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /run 2023-10-18T19:42:28.527 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2023-10-18T19:42:28.528 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2023-10-18T19:42:28.528 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2023-10-18T19:42:28.528 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2023-10-18T19:42:28.528 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /dev 2023-10-18T19:42:28.528 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:42:28.620 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-18T19:42:28.620 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:28.620 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:42:28.620 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:28.621 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-7.el8.x86_64 2023-10-18T19:42:28.621 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:28.621 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-18T19:42:28.621 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-18T19:42:28.621 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-18T19:42:28.622 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:28.622 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:42:28.862 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph 2023-10-18T19:42:29.726 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:00 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:42:29.909 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:18.2.0-885.g07c1e9af.el8.x86_64 23/25 2023-10-18T19:42:29.930 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-885.g07c1e9af.el8.x86_64 23/25 2023-10-18T19:42:29.930 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-18T19:42:30.181 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-18T19:42:30.181 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-18T19:42:30.925 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-18T19:42:30.926 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 1/25 2023-10-18T19:42:30.926 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 2/25 2023-10-18T19:42:30.926 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:18.2.0-885.g07c1e9af.el8.x86_64 3/25 2023-10-18T19:42:30.926 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:18.2.0-885.g07c1e9af.el8.x86_64 4/25 2023-10-18T19:42:30.927 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 5/25 2023-10-18T19:42:30.927 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:18.2.0-885.g07c1e9af.el8.x86_64 6/25 2023-10-18T19:42:30.927 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:18.2.0-885.g07c1e9af.el8.x86_64 7/25 2023-10-18T19:42:30.927 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:18.2.0-885.g07c1e9af.el8.x 8/25 2023-10-18T19:42:30.927 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:18.2.0-885.g07c1e9af.el8.x86 9/25 2023-10-18T19:42:30.928 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:18.2.0-885.g07c1e9af.el8.x86_64 10/25 2023-10-18T19:42:30.928 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:18.2.0-885.g07c1e9af.el8.x86_64 11/25 2023-10-18T19:42:30.928 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:18.2.0-885.g07c1e9af.el8.x86_64 12/25 2023-10-18T19:42:30.928 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:18.2.0-885.g07c1e9af.el8.x86_64 13/25 2023-10-18T19:42:30.928 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-18T19:42:30.928 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-18T19:42:30.929 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-18T19:42:30.929 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-18T19:42:30.929 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-18T19:42:30.929 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-18T19:42:30.929 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-18T19:42:30.930 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-18T19:42:30.930 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 22/25 2023-10-18T19:42:30.930 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-18T19:42:30.930 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 24/25 2023-10-18T19:42:31.329 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:42:31.333 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:31.333 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:42:31.334 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:31.334 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:42:31.334 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 6.4 k 2023-10-18T19:42:31.334 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-10-18T19:42:31.335 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 2.3 M 2023-10-18T19:42:31.335 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 1.5 M 2023-10-18T19:42:31.335 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 242 k 2023-10-18T19:42:31.335 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 4.6 M 2023-10-18T19:42:31.335 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 18 M 2023-10-18T19:42:31.336 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-18T19:42:31.336 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-18T19:42:31.336 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-18T19:42:31.336 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-18T19:42:31.336 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-18T19:42:31.336 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-18T19:42:31.337 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-18T19:42:31.337 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-18T19:42:31.337 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-18T19:42:31.337 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-18T19:42:31.337 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-18T19:42:31.337 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-18T19:42:31.338 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-18T19:42:31.338 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-18T19:42:31.338 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-18T19:42:31.338 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-18T19:42:31.338 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-18T19:42:31.339 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-18T19:42:31.339 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-18T19:42:31.339 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-18T19:42:31.339 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-18T19:42:31.339 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-18T19:42:31.340 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-18T19:42:31.340 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-18T19:42:31.340 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-18T19:42:31.340 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-18T19:42:31.340 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:31.341 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:42:31.341 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:31.341 INFO:teuthology.orchestra.run.smithi156.stdout:Install 32 Packages 2023-10-18T19:42:31.341 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:31.342 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 30 M 2023-10-18T19:42:31.342 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 119 M 2023-10-18T19:42:31.342 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:42:31.740 INFO:teuthology.orchestra.run.smithi156.stdout:(1/32): ceph-mgr-18.2.0-885.g07c1e9af.el8.x86_6 5.3 MB/s | 1.5 MB 00:00 2023-10-18T19:42:31.765 INFO:teuthology.orchestra.run.smithi156.stdout:(2/32): ceph-18.2.0-885.g07c1e9af.el8.x86_64.rp 20 kB/s | 6.4 kB 00:00 2023-10-18T19:42:32.154 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-18T19:42:32.155 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:42:32.155 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2023-10-18T19:42:32.155 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.155 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.156 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:42:32.156 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.156 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.156 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.156 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.157 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-18T19:42:32.157 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.157 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-18T19:42:32.157 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-18T19:42:32.157 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.158 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-18T19:42:32.158 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.158 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-18T19:42:32.158 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-18T19:42:32.158 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-18T19:42:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-18T19:42:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:42:32.160 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:42:32.479 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-test 2023-10-18T19:42:32.840 INFO:teuthology.orchestra.run.smithi156.stdout:(3/32): ceph-osd-18.2.0-885.g07c1e9af.el8.x86_6 16 MB/s | 18 MB 00:01 2023-10-18T19:42:32.899 INFO:teuthology.orchestra.run.smithi156.stdout:(4/32): ceph-mds-18.2.0-885.g07c1e9af.el8.x86_6 1.6 MB/s | 2.3 MB 00:01 2023-10-18T19:42:32.941 INFO:teuthology.orchestra.run.smithi156.stdout:(5/32): ceph-mgr-modules-core-18.2.0-885.g07c1e 2.4 MB/s | 242 kB 00:00 2023-10-18T19:42:33.075 INFO:teuthology.orchestra.run.smithi156.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 290 kB/s | 39 kB 00:00 2023-10-18T19:42:33.117 INFO:teuthology.orchestra.run.smithi156.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 722 kB/s | 157 kB 00:00 2023-10-18T19:42:33.225 INFO:teuthology.orchestra.run.smithi156.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 527 kB/s | 57 kB 00:00 2023-10-18T19:42:33.259 INFO:teuthology.orchestra.run.smithi156.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 562 kB/s | 103 kB 00:00 2023-10-18T19:42:33.325 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:48 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:42:33.384 INFO:teuthology.orchestra.run.smithi156.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.9 MB/s | 249 kB 00:00 2023-10-18T19:42:33.526 INFO:teuthology.orchestra.run.smithi156.stdout:(11/32): ceph-mon-18.2.0-885.g07c1e9af.el8.x86_ 2.6 MB/s | 4.6 MB 00:01 2023-10-18T19:42:33.552 INFO:teuthology.orchestra.run.smithi156.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2023-10-18T19:42:33.619 INFO:teuthology.orchestra.run.smithi156.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2023-10-18T19:42:33.644 INFO:teuthology.orchestra.run.smithi156.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.4 MB/s | 641 kB 00:00 2023-10-18T19:42:33.711 INFO:teuthology.orchestra.run.smithi156.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 274 kB/s | 44 kB 00:00 2023-10-18T19:42:33.797 INFO:teuthology.orchestra.run.smithi156.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2023-10-18T19:42:33.837 INFO:teuthology.orchestra.run.smithi156.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 919 kB/s | 173 kB 00:00 2023-10-18T19:42:33.853 INFO:teuthology.orchestra.run.smithi156.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2023-10-18T19:42:33.879 INFO:teuthology.orchestra.run.smithi156.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.2 MB/s | 384 kB 00:00 2023-10-18T19:42:33.895 INFO:teuthology.orchestra.run.smithi156.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 299 kB/s | 18 kB 00:00 2023-10-18T19:42:33.921 INFO:teuthology.orchestra.run.smithi156.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 725 kB/s | 49 kB 00:00 2023-10-18T19:42:33.946 INFO:teuthology.orchestra.run.smithi156.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-10-18T19:42:33.971 INFO:teuthology.orchestra.run.smithi156.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2023-10-18T19:42:33.997 INFO:teuthology.orchestra.run.smithi156.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 371 kB/s | 19 kB 00:00 2023-10-18T19:42:34.022 INFO:teuthology.orchestra.run.smithi156.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 474 kB/s | 24 kB 00:00 2023-10-18T19:42:34.047 INFO:teuthology.orchestra.run.smithi156.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 584 kB/s | 29 kB 00:00 2023-10-18T19:42:34.081 INFO:teuthology.orchestra.run.smithi156.stdout:(27/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2023-10-18T19:42:34.106 INFO:teuthology.orchestra.run.smithi156.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 328 kB/s | 27 kB 00:00 2023-10-18T19:42:34.148 INFO:teuthology.orchestra.run.smithi156.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2023-10-18T19:42:34.174 INFO:teuthology.orchestra.run.smithi156.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.3 MB/s | 85 kB 00:00 2023-10-18T19:42:34.224 INFO:teuthology.orchestra.run.smithi156.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2023-10-18T19:42:34.249 INFO:teuthology.orchestra.run.smithi156.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2023-10-18T19:42:34.253 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:42:34.254 INFO:teuthology.orchestra.run.smithi156.stdout:Total 10 MB/s | 30 MB 00:02 2023-10-18T19:42:34.333 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:42:34.371 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:42:34.372 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:42:34.697 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:42:34.700 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:42:34.909 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:42:34.910 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:42:34.910 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:42:34.911 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:42:34.911 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:42:34.911 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 47 M 2023-10-18T19:42:34.911 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-10-18T19:42:34.911 INFO:teuthology.orchestra.run.smithi082.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-18T19:42:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 137 k 2023-10-18T19:42:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-18T19:42:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-18T19:42:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-18T19:42:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:42:34.912 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:42:34.913 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:42:34.913 INFO:teuthology.orchestra.run.smithi082.stdout:Install 6 Packages 2023-10-18T19:42:34.913 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:42:34.913 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 48 M 2023-10-18T19:42:34.914 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 211 M 2023-10-18T19:42:34.914 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:42:35.339 INFO:teuthology.orchestra.run.smithi082.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 831 kB/s | 202 kB 00:00 2023-10-18T19:42:35.398 INFO:teuthology.orchestra.run.smithi082.stdout:(2/6): libcephsqlite-18.2.0-885.g07c1e9af.el8.x 453 kB/s | 137 kB 00:00 2023-10-18T19:42:35.548 INFO:teuthology.orchestra.run.smithi082.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 898 kB/s | 187 kB 00:00 2023-10-18T19:42:35.656 INFO:teuthology.orchestra.run.smithi082.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.2 MB/s | 323 kB 00:00 2023-10-18T19:42:35.698 INFO:teuthology.orchestra.run.smithi082.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 470 kB/s | 71 kB 00:00 2023-10-18T19:42:35.849 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:42:36.096 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-18T19:42:36.290 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-18T19:42:36.515 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-18T19:42:36.705 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-18T19:42:36.941 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-18T19:42:37.141 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-18T19:42:37.359 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-18T19:42:37.532 INFO:teuthology.orchestra.run.smithi082.stdout:(6/6): ceph-test-18.2.0-885.g07c1e9af.el8.x86_6 19 MB/s | 47 MB 00:02 2023-10-18T19:42:37.534 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:42:37.534 INFO:teuthology.orchestra.run.smithi082.stdout:Total 18 MB/s | 48 MB 00:02 2023-10-18T19:42:37.576 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:42:37.597 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:42:37.597 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:42:37.646 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-18T19:42:37.873 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-18T19:42:37.891 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:42:37.894 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:42:38.117 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-18T19:42:38.347 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-18T19:42:38.373 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:42:38.596 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-18T19:42:38.646 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-18T19:42:38.887 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-18T19:42:38.980 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-18T19:42:39.084 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-18T19:42:39.174 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-18T19:42:39.217 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-18T19:42:39.388 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-18T19:42:39.447 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-18T19:42:39.534 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:18.2.0-885.g07c1e9af.el8.x86_64 5/6 2023-10-18T19:42:39.697 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-18T19:42:39.940 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-18T19:42:40.285 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-18T19:42:40.503 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-18T19:42:40.734 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-18T19:42:40.894 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-18T19:42:41.164 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-18T19:42:41.385 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-18T19:42:41.685 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-18T19:42:41.999 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-18T19:42:42.312 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-18T19:42:42.667 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-885.g07c1e9af.el8.n 27/32 2023-10-18T19:42:42.888 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 28/32 2023-10-18T19:42:44.129 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:18.2.0-885.g07c1e9af.el8.x86_64 5/6 2023-10-18T19:42:44.365 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:18.2.0-885.g07c1e9af.el8.x86_64 6/6 2023-10-18T19:42:44.477 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 28/32 2023-10-18T19:42:44.666 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-test-2:18.2.0-885.g07c1e9af.el8.x86_64 6/6 2023-10-18T19:42:44.667 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:18.2.0-885.g07c1e9af.el8.x86_64 1/6 2023-10-18T19:42:44.667 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:18.2.0-885.g07c1e9af.el8.x86_64 2/6 2023-10-18T19:42:44.667 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-18T19:42:44.667 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-18T19:42:44.667 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-18T19:42:44.721 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:18.2.0-885.g07c1e9af.el8.x86_64 29/32 2023-10-18T19:42:45.192 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:18.2.0-885.g07c1e9af.el8.x86_64 29/32 2023-10-18T19:42:45.387 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-18T19:42:45.388 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:42:45.388 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:42:45.388 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:45.388 INFO:teuthology.orchestra.run.smithi082.stdout: jq-1.6-7.el8.x86_64 2023-10-18T19:42:45.389 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:45.389 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-18T19:42:45.389 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-18T19:42:45.389 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-18T19:42:45.389 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:42:45.390 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:42:45.390 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:18.2.0-885.g07c1e9af.el8.x86_64 30/32 2023-10-18T19:42:45.605 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:18.2.0-885.g07c1e9af.el8.x86_64 30/32 2023-10-18T19:42:45.626 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph 2023-10-18T19:42:45.797 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:18.2.0-885.g07c1e9af.el8.x86_64 31/32 2023-10-18T19:42:45.815 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:18.2.0-885.g07c1e9af.el8.x86_64 31/32 2023-10-18T19:42:45.938 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:18.2.0-885.g07c1e9af.el8.x86_64 32/32 2023-10-18T19:42:46.336 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-2:18.2.0-885.g07c1e9af.el8.x86_64 32/32 2023-10-18T19:42:46.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:18.2.0-885.g07c1e9af.el8.x86_64 1/32 2023-10-18T19:42:46.336 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:18.2.0-885.g07c1e9af.el8.x86_64 2/32 2023-10-18T19:42:46.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 3/32 2023-10-18T19:42:46.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:18.2.0-885.g07c1e9af.el8.x86_64 4/32 2023-10-18T19:42:46.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:18.2.0-885.g07c1e9af.el8.x86_64 5/32 2023-10-18T19:42:46.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-885.g07c1e9af.el8.n 6/32 2023-10-18T19:42:46.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-18T19:42:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-18T19:42:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-18T19:42:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-18T19:42:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-18T19:42:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-18T19:42:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-18T19:42:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-18T19:42:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-18T19:42:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-18T19:42:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-18T19:42:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-18T19:42:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-18T19:42:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-18T19:42:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-18T19:42:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-18T19:42:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-18T19:42:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-18T19:42:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-18T19:42:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-18T19:42:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-18T19:42:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-18T19:42:46.342 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-18T19:42:46.342 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-18T19:42:46.342 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-18T19:42:46.484 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:01 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:42:47.584 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-18T19:42:47.584 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:47.585 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:42:47.585 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:47.585 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:47.585 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:47.585 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:42:47.585 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:47.586 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:42:47.586 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-18T19:42:47.586 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-18T19:42:47.586 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-18T19:42:47.586 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-18T19:42:47.587 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-18T19:42:47.587 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-18T19:42:47.587 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-18T19:42:47.587 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-18T19:42:47.587 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-18T19:42:47.587 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-18T19:42:47.588 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-18T19:42:47.588 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-18T19:42:47.588 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-18T19:42:47.588 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-2.6-1.el8.noarch 2023-10-18T19:42:47.588 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-18T19:42:47.589 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-18T19:42:47.589 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-18T19:42:47.589 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-18T19:42:47.589 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-18T19:42:47.589 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-18T19:42:47.589 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-18T19:42:47.590 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-18T19:42:47.590 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-18T19:42:47.590 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-18T19:42:47.590 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-18T19:42:47.590 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-18T19:42:47.591 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:47.591 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:42:48.095 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:42:48.098 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:42:48.099 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:42:48.099 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:42:48.099 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:42:48.099 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 6.4 k 2023-10-18T19:42:48.099 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-10-18T19:42:48.100 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 2.3 M 2023-10-18T19:42:48.100 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 1.5 M 2023-10-18T19:42:48.100 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 242 k 2023-10-18T19:42:48.100 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 4.6 M 2023-10-18T19:42:48.100 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 18 M 2023-10-18T19:42:48.101 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-18T19:42:48.101 INFO:teuthology.orchestra.run.smithi082.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-18T19:42:48.101 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-18T19:42:48.101 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-18T19:42:48.101 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-18T19:42:48.102 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-18T19:42:48.102 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-18T19:42:48.102 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-18T19:42:48.102 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-18T19:42:48.102 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-18T19:42:48.103 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-18T19:42:48.103 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-18T19:42:48.103 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-18T19:42:48.103 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-18T19:42:48.103 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-18T19:42:48.104 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-18T19:42:48.104 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-18T19:42:48.104 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-18T19:42:48.104 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-18T19:42:48.104 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-18T19:42:48.105 INFO:teuthology.orchestra.run.smithi082.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-18T19:42:48.105 INFO:teuthology.orchestra.run.smithi082.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-18T19:42:48.105 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-18T19:42:48.105 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-18T19:42:48.105 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-18T19:42:48.105 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-18T19:42:48.106 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:42:48.106 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:42:48.106 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:42:48.106 INFO:teuthology.orchestra.run.smithi082.stdout:Install 32 Packages 2023-10-18T19:42:48.106 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:42:48.107 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 30 M 2023-10-18T19:42:48.107 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 119 M 2023-10-18T19:42:48.108 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:42:48.400 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-base 2023-10-18T19:42:48.741 INFO:teuthology.orchestra.run.smithi082.stdout:(1/32): ceph-18.2.0-885.g07c1e9af.el8.x86_64.rp 34 kB/s | 6.4 kB 00:00 2023-10-18T19:42:48.908 INFO:teuthology.orchestra.run.smithi082.stdout:(2/32): ceph-mds-18.2.0-885.g07c1e9af.el8.x86_6 6.4 MB/s | 2.3 MB 00:00 2023-10-18T19:42:49.134 INFO:teuthology.orchestra.run.smithi082.stdout:(3/32): ceph-mgr-18.2.0-885.g07c1e9af.el8.x86_6 2.7 MB/s | 1.5 MB 00:00 2023-10-18T19:42:49.242 INFO:teuthology.orchestra.run.smithi082.stdout:(4/32): ceph-mgr-modules-core-18.2.0-885.g07c1e 2.2 MB/s | 242 kB 00:00 2023-10-18T19:42:49.266 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:20 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:42:49.442 INFO:teuthology.orchestra.run.smithi082.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 786 kB/s | 157 kB 00:00 2023-10-18T19:42:49.593 INFO:teuthology.orchestra.run.smithi082.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 259 kB/s | 39 kB 00:00 2023-10-18T19:42:49.759 INFO:teuthology.orchestra.run.smithi082.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 621 kB/s | 103 kB 00:00 2023-10-18T19:42:49.926 INFO:teuthology.orchestra.run.smithi082.stdout:(8/32): ceph-mon-18.2.0-885.g07c1e9af.el8.x86_6 3.9 MB/s | 4.6 MB 00:01 2023-10-18T19:42:49.952 INFO:teuthology.orchestra.run.smithi082.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 297 kB/s | 57 kB 00:00 2023-10-18T19:42:50.148 INFO:teuthology.orchestra.run.smithi082.stdout:(10/32): ceph-osd-18.2.0-885.g07c1e9af.el8.x86_ 14 MB/s | 18 MB 00:01 2023-10-18T19:42:50.236 INFO:teuthology.orchestra.run.smithi082.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2023-10-18T19:42:50.261 INFO:teuthology.orchestra.run.smithi082.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 805 kB/s | 249 kB 00:00 2023-10-18T19:42:50.337 INFO:teuthology.orchestra.run.smithi082.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.1 MB/s | 109 kB 00:00 2023-10-18T19:42:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.7 MB/s | 641 kB 00:00 2023-10-18T19:42:50.704 INFO:teuthology.orchestra.run.smithi156.stdout:Package ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:42:50.778 INFO:teuthology.orchestra.run.smithi082.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 84 kB/s | 44 kB 00:00 2023-10-18T19:42:50.898 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:42:50.899 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2023-10-18T19:42:50.899 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:42:50.946 INFO:teuthology.orchestra.run.smithi082.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 304 kB/s | 185 kB 00:00 2023-10-18T19:42:50.971 INFO:teuthology.orchestra.run.smithi082.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 291 kB/s | 173 kB 00:00 2023-10-18T19:42:50.992 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install cephadm 2023-10-18T19:42:51.054 INFO:teuthology.orchestra.run.smithi082.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-10-18T19:42:51.080 INFO:teuthology.orchestra.run.smithi082.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2023-10-18T19:42:51.187 INFO:teuthology.orchestra.run.smithi082.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 368 kB/s | 49 kB 00:00 2023-10-18T19:42:51.214 INFO:teuthology.orchestra.run.smithi082.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 439 kB/s | 59 kB 00:00 2023-10-18T19:42:51.322 INFO:teuthology.orchestra.run.smithi082.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2023-10-18T19:42:51.389 INFO:teuthology.orchestra.run.smithi082.stdout:(23/32): python3-cherrypy-18.4.0-1.el8.noarch.r 630 kB/s | 384 kB 00:00 2023-10-18T19:42:51.458 INFO:teuthology.orchestra.run.smithi082.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 139 kB/s | 19 kB 00:00 2023-10-18T19:42:51.515 INFO:teuthology.orchestra.run.smithi082.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 191 kB/s | 24 kB 00:00 2023-10-18T19:42:51.582 INFO:teuthology.orchestra.run.smithi082.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 236 kB/s | 29 kB 00:00 2023-10-18T19:42:51.632 INFO:teuthology.orchestra.run.smithi082.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2023-10-18T19:42:51.674 INFO:teuthology.orchestra.run.smithi082.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 581 kB/s | 283 kB 00:00 2023-10-18T19:42:51.849 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:22 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:42:51.883 INFO:teuthology.orchestra.run.smithi082.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 410 kB/s | 85 kB 00:00 2023-10-18T19:42:51.983 INFO:teuthology.orchestra.run.smithi082.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 637 kB/s | 255 kB 00:00 2023-10-18T19:42:52.008 INFO:teuthology.orchestra.run.smithi082.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 669 kB/s | 251 kB 00:00 2023-10-18T19:42:52.025 INFO:teuthology.orchestra.run.smithi082.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 162 kB/s | 23 kB 00:00 2023-10-18T19:42:52.030 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:42:52.030 INFO:teuthology.orchestra.run.smithi082.stdout:Total 7.6 MB/s | 30 MB 00:03 2023-10-18T19:42:52.112 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:42:52.149 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:42:52.150 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:42:52.474 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:42:52.477 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:42:53.458 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:42:53.459 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:53.460 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:42:53.460 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:53.460 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:42:53.460 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 167 k 2023-10-18T19:42:53.460 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:53.461 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:42:53.461 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:53.461 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2023-10-18T19:42:53.461 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:53.461 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 167 k 2023-10-18T19:42:53.462 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 674 k 2023-10-18T19:42:53.462 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:42:53.665 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:42:53.903 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-18T19:42:54.025 INFO:teuthology.orchestra.run.smithi156.stdout:cephadm-18.2.0-885.g07c1e9af.el8.noarch.rpm 296 kB/s | 167 kB 00:00 2023-10-18T19:42:54.026 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:42:54.026 INFO:teuthology.orchestra.run.smithi156.stdout:Total 295 kB/s | 167 kB 00:00 2023-10-18T19:42:54.027 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:42:54.034 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:42:54.035 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:42:54.037 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:42:54.040 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:42:54.101 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:42:54.122 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-18T19:42:54.366 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-18T19:42:54.587 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-18T19:42:54.614 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:42:54.757 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:42:54.816 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-18T19:42:54.983 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:42:55.024 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-18T19:42:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-18T19:42:55.487 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-18T19:42:55.683 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:42:55.684 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:55.684 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:42:55.684 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:42:55.684 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:55.684 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:42:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-18T19:42:55.930 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-immutable-object-cache 2023-10-18T19:42:55.992 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-18T19:42:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-18T19:42:56.495 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-18T19:42:56.803 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:27 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:42:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-18T19:42:57.134 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-18T19:42:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-18T19:42:57.596 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-18T19:42:57.823 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-18T19:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-18T19:42:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-18T19:42:58.418 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:42:58.419 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:58.419 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2023-10-18T19:42:58.419 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:58.419 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:42:58.420 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 165 k 2023-10-18T19:42:58.420 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:58.420 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:42:58.420 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:42:58.420 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2023-10-18T19:42:58.421 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:42:58.421 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 165 k 2023-10-18T19:42:58.422 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 503 k 2023-10-18T19:42:58.422 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:42:58.649 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-18T19:42:58.893 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-18T19:42:58.958 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-immutable-object-cache-18.2.0-885.g07c1e9a 308 kB/s | 165 kB 00:00 2023-10-18T19:42:58.959 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:42:58.959 INFO:teuthology.orchestra.run.smithi156.stdout:Total 306 kB/s | 165 kB 00:00 2023-10-18T19:42:58.960 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:42:58.973 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:42:58.973 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:42:59.026 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:42:59.029 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:42:59.198 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-18T19:42:59.450 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-18T19:42:59.750 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-18T19:42:59.785 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:43:00.064 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-18T19:43:00.312 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-885.g07c1e9af.e 1/1 2023-10-18T19:43:00.403 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-18T19:43:00.746 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-885.g07c1e9af.e 1/1 2023-10-18T19:43:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-885.g07c1e9af.el8.n 27/32 2023-10-18T19:43:01.002 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 28/32 2023-10-18T19:43:01.366 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-885.g07c1e9af.e 1/1 2023-10-18T19:43:01.366 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:01.366 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:43:01.366 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:43:01.367 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:01.367 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:01.618 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr 2023-10-18T19:43:02.479 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 28/32 2023-10-18T19:43:02.885 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:18.2.0-885.g07c1e9af.el8.x86_64 29/32 2023-10-18T19:43:03.355 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:18.2.0-885.g07c1e9af.el8.x86_64 29/32 2023-10-18T19:43:03.680 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:18.2.0-885.g07c1e9af.el8.x86_64 30/32 2023-10-18T19:43:03.895 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:18.2.0-885.g07c1e9af.el8.x86_64 30/32 2023-10-18T19:43:03.906 INFO:teuthology.orchestra.run.smithi156.stdout:Package ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:43:04.096 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:18.2.0-885.g07c1e9af.el8.x86_64 31/32 2023-10-18T19:43:04.100 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:43:04.101 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2023-10-18T19:43:04.101 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:04.114 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:18.2.0-885.g07c1e9af.el8.x86_64 31/32 2023-10-18T19:43:04.195 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-dashboard 2023-10-18T19:43:04.237 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:18.2.0-885.g07c1e9af.el8.x86_64 32/32 2023-10-18T19:43:04.687 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-2:18.2.0-885.g07c1e9af.el8.x86_64 32/32 2023-10-18T19:43:04.687 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:18.2.0-885.g07c1e9af.el8.x86_64 1/32 2023-10-18T19:43:04.687 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:18.2.0-885.g07c1e9af.el8.x86_64 2/32 2023-10-18T19:43:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 3/32 2023-10-18T19:43:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:18.2.0-885.g07c1e9af.el8.x86_64 4/32 2023-10-18T19:43:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:18.2.0-885.g07c1e9af.el8.x86_64 5/32 2023-10-18T19:43:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-885.g07c1e9af.el8.n 6/32 2023-10-18T19:43:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-18T19:43:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-18T19:43:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-18T19:43:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-18T19:43:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-18T19:43:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-18T19:43:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-18T19:43:04.690 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-18T19:43:04.690 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-18T19:43:04.690 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-18T19:43:04.690 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-18T19:43:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-18T19:43:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-18T19:43:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-18T19:43:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-18T19:43:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-18T19:43:04.692 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-18T19:43:04.692 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-18T19:43:04.692 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-18T19:43:04.693 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-18T19:43:04.693 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-18T19:43:04.693 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-18T19:43:04.693 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-18T19:43:04.694 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-18T19:43:04.694 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-18T19:43:05.047 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-18T19:43:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:05.814 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:43:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:43:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:43:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:43:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:43:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:43:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-18T19:43:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-18T19:43:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-18T19:43:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-18T19:43:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-18T19:43:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-18T19:43:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-18T19:43:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-18T19:43:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-18T19:43:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-18T19:43:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-18T19:43:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-18T19:43:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-18T19:43:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-2.6-1.el8.noarch 2023-10-18T19:43:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-18T19:43:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-18T19:43:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-18T19:43:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-18T19:43:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-18T19:43:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-18T19:43:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-18T19:43:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-18T19:43:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-18T19:43:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-18T19:43:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-18T19:43:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-18T19:43:05.821 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:05.821 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:43:06.083 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-base 2023-10-18T19:43:06.642 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:43:06.643 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:06.643 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:43:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:43:06.644 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 1.8 M 2023-10-18T19:43:06.644 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-10-18T19:43:06.644 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 23 k 2023-10-18T19:43:06.645 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 15 k 2023-10-18T19:43:06.645 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-18T19:43:06.645 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-18T19:43:06.645 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-18T19:43:06.645 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:06.646 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:43:06.646 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:06.646 INFO:teuthology.orchestra.run.smithi156.stdout:Install 6 Packages 2023-10-18T19:43:06.646 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:06.647 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 2.1 M 2023-10-18T19:43:06.647 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 8.5 M 2023-10-18T19:43:06.647 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:43:06.939 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:21 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:43:07.260 INFO:teuthology.orchestra.run.smithi156.stdout:(1/6): ceph-grafana-dashboards-18.2.0-885.g07c1 147 kB/s | 23 kB 00:00 2023-10-18T19:43:07.327 INFO:teuthology.orchestra.run.smithi156.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 639 kB/s | 43 kB 00:00 2023-10-18T19:43:07.352 INFO:teuthology.orchestra.run.smithi156.stdout:(3/6): ceph-prometheus-alerts-18.2.0-885.g07c1e 58 kB/s | 15 kB 00:00 2023-10-18T19:43:07.561 INFO:teuthology.orchestra.run.smithi156.stdout:(4/6): ceph-mgr-dashboard-18.2.0-885.g07c1e9af. 3.8 MB/s | 1.8 MB 00:00 2023-10-18T19:43:07.586 INFO:teuthology.orchestra.run.smithi156.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 130 kB/s | 34 kB 00:00 2023-10-18T19:43:07.720 INFO:teuthology.orchestra.run.smithi156.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 535 kB/s | 196 kB 00:00 2023-10-18T19:43:07.721 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:07.721 INFO:teuthology.orchestra.run.smithi156.stdout:Total 1.9 MB/s | 2.1 MB 00:01 2023-10-18T19:43:07.732 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:43:07.741 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:43:07.741 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:43:07.818 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:43:07.821 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:43:08.113 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:43:08.299 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-18T19:43:08.323 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-base-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:43:08.516 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:43:08.517 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-10-18T19:43:08.517 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:43:08.527 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-18T19:43:08.606 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install cephadm 2023-10-18T19:43:08.687 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-18T19:43:08.841 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-885.g07c1e9af.el8.no 4/6 2023-10-18T19:43:09.174 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-885.g07c1e9af.el8.n 5/6 2023-10-18T19:43:09.400 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-885.g07c1e9af.el8.noarch 6/6 2023-10-18T19:43:09.454 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:24 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:43:09.684 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-885.g07c1e9af.el8.noarch 6/6 2023-10-18T19:43:09.685 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-885.g07c1e9af.el8.n 1/6 2023-10-18T19:43:09.685 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-885.g07c1e9af.el8.noarch 2/6 2023-10-18T19:43:09.685 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-885.g07c1e9af.el8.no 3/6 2023-10-18T19:43:09.685 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-18T19:43:09.685 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-18T19:43:10.343 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-18T19:43:10.344 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:10.344 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:43:10.344 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:10.344 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:10.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:10.345 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-18T19:43:10.345 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-18T19:43:10.345 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-18T19:43:10.345 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:10.346 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:10.635 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-18T19:43:11.067 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:43:11.069 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:11.069 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:43:11.069 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:11.070 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:43:11.070 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 167 k 2023-10-18T19:43:11.070 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:11.071 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:43:11.071 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:11.071 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2023-10-18T19:43:11.072 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:11.072 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 167 k 2023-10-18T19:43:11.072 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 674 k 2023-10-18T19:43:11.073 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:43:11.374 INFO:teuthology.orchestra.run.smithi082.stdout:cephadm-18.2.0-885.g07c1e9af.el8.noarch.rpm 548 kB/s | 167 kB 00:00 2023-10-18T19:43:11.375 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:11.375 INFO:teuthology.orchestra.run.smithi082.stdout:Total 546 kB/s | 167 kB 00:00 2023-10-18T19:43:11.375 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:43:11.382 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:43:11.382 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:43:11.384 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:43:11.387 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:43:11.433 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:43:11.499 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:12.036 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:43:12.229 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:43:12.451 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:43:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:43:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:13.087 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:43:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:13.087 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:43:13.118 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:43:13.119 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================================ 2023-10-18T19:43:13.120 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:43:13.120 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================================ 2023-10-18T19:43:13.120 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:43:13.121 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 7.4 M 2023-10-18T19:43:13.121 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-10-18T19:43:13.122 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-18T19:43:13.122 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-18T19:43:13.122 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-18T19:43:13.123 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-18T19:43:13.123 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-18T19:43:13.123 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-18T19:43:13.123 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-18T19:43:13.124 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:13.124 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:43:13.124 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================================ 2023-10-18T19:43:13.124 INFO:teuthology.orchestra.run.smithi156.stdout:Install 8 Packages 2023-10-18T19:43:13.124 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:13.125 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 35 M 2023-10-18T19:43:13.125 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 210 M 2023-10-18T19:43:13.125 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:43:13.315 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-immutable-object-cache 2023-10-18T19:43:13.969 INFO:teuthology.orchestra.run.smithi156.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2023-10-18T19:43:14.035 INFO:teuthology.orchestra.run.smithi156.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 9.3 MB/s | 4.7 MB 00:00 2023-10-18T19:43:14.183 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:29 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:43:14.235 INFO:teuthology.orchestra.run.smithi156.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.1 MB/s | 225 kB 00:00 2023-10-18T19:43:14.411 INFO:teuthology.orchestra.run.smithi156.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 8.4 MB/s | 3.7 MB 00:00 2023-10-18T19:43:14.686 INFO:teuthology.orchestra.run.smithi156.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.3 MB/s | 645 kB 00:00 2023-10-18T19:43:14.787 INFO:teuthology.orchestra.run.smithi156.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2023-10-18T19:43:15.087 INFO:teuthology.orchestra.run.smithi156.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 16 MB/s | 14 MB 00:00 2023-10-18T19:43:15.431 INFO:teuthology.orchestra.run.smithi156.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.0-885 3.9 MB/s | 7.4 MB 00:01 2023-10-18T19:43:15.467 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:15.468 INFO:teuthology.orchestra.run.smithi156.stdout:Total 15 MB/s | 35 MB 00:02 2023-10-18T19:43:15.669 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:43:15.687 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:43:15.688 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:43:15.794 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:43:15.795 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:15.795 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2023-10-18T19:43:15.795 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:15.795 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:43:15.796 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 165 k 2023-10-18T19:43:15.796 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:15.796 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:43:15.796 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:15.796 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2023-10-18T19:43:15.797 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:15.797 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 165 k 2023-10-18T19:43:15.797 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 503 k 2023-10-18T19:43:15.797 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:43:16.004 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-immutable-object-cache-18.2.0-885.g07c1e9a 792 kB/s | 165 kB 00:00 2023-10-18T19:43:16.005 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:16.005 INFO:teuthology.orchestra.run.smithi082.stdout:Total 788 kB/s | 165 kB 00:00 2023-10-18T19:43:16.005 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:43:16.016 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:43:16.016 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:43:16.062 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:43:16.065 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:43:16.170 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:43:16.326 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-885.g07c1e9af.e 1/1 2023-10-18T19:43:16.336 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:43:16.339 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:43:16.768 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-885.g07c1e9af.e 1/1 2023-10-18T19:43:16.988 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:43:17.150 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-18T19:43:17.290 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-18T19:43:17.405 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-885.g07c1e9af.e 1/1 2023-10-18T19:43:17.406 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:17.406 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:43:17.406 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:43:17.406 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:17.407 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:43:17.475 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-18T19:43:17.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr 2023-10-18T19:43:18.065 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-18T19:43:18.275 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-18T19:43:18.524 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:43:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-18T19:43:19.042 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-18T19:43:19.764 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-18T19:43:19.946 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-mgr-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:43:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-18T19:43:20.143 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:43:20.144 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-10-18T19:43:20.144 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:43:20.237 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-dashboard 2023-10-18T19:43:20.333 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-18T19:43:21.093 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:43:22.114 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-18T19:43:22.694 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:43:22.695 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:22.695 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:43:22.695 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:22.696 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:43:22.696 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 1.8 M 2023-10-18T19:43:22.696 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-10-18T19:43:22.696 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 23 k 2023-10-18T19:43:22.696 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 15 k 2023-10-18T19:43:22.697 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-18T19:43:22.697 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-18T19:43:22.697 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-18T19:43:22.697 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:22.697 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:43:22.697 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:22.698 INFO:teuthology.orchestra.run.smithi082.stdout:Install 6 Packages 2023-10-18T19:43:22.698 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:22.698 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 2.1 M 2023-10-18T19:43:22.699 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 8.5 M 2023-10-18T19:43:22.699 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:43:23.215 INFO:teuthology.orchestra.run.smithi082.stdout:(1/6): ceph-grafana-dashboards-18.2.0-885.g07c1 53 kB/s | 23 kB 00:00 2023-10-18T19:43:23.240 INFO:teuthology.orchestra.run.smithi082.stdout:(2/6): ceph-prometheus-alerts-18.2.0-885.g07c1e 32 kB/s | 15 kB 00:00 2023-10-18T19:43:23.307 INFO:teuthology.orchestra.run.smithi082.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 463 kB/s | 43 kB 00:00 2023-10-18T19:43:23.319 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-18T19:43:23.382 INFO:teuthology.orchestra.run.smithi082.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2023-10-18T19:43:23.499 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-885.g07c1e9af 8/8 2023-10-18T19:43:23.524 INFO:teuthology.orchestra.run.smithi082.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 905 kB/s | 196 kB 00:00 2023-10-18T19:43:23.814 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-885.g07c1e9af 8/8 2023-10-18T19:43:23.814 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-885.g07c1e9af 1/8 2023-10-18T19:43:23.815 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-18T19:43:23.815 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-18T19:43:23.815 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-18T19:43:23.816 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-18T19:43:23.816 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-18T19:43:23.816 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-18T19:43:23.817 INFO:teuthology.orchestra.run.smithi082.stdout:(6/6): ceph-mgr-dashboard-18.2.0-885.g07c1e9af. 1.7 MB/s | 1.8 MB 00:01 2023-10-18T19:43:23.818 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:23.818 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.8 MB/s | 2.1 MB 00:01 2023-10-18T19:43:23.830 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:43:23.840 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:43:23.840 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:43:23.922 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:43:23.925 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:43:24.196 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:43:24.373 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-18T19:43:24.494 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-18T19:43:24.495 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:24.495 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:43:24.495 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:24.495 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-18T19:43:24.496 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-18T19:43:24.496 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-18T19:43:24.496 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-18T19:43:24.496 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-18T19:43:24.496 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-18T19:43:24.496 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-18T19:43:24.497 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:24.497 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:24.609 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-rook 2023-10-18T19:43:24.617 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-18T19:43:24.811 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-18T19:43:24.931 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-885.g07c1e9af.el8.no 4/6 2023-10-18T19:43:25.329 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-885.g07c1e9af.el8.n 5/6 2023-10-18T19:43:25.465 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:56 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-885.g07c1e9af.el8.noarch 6/6 2023-10-18T19:43:25.962 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-885.g07c1e9af.el8.noarch 6/6 2023-10-18T19:43:25.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-885.g07c1e9af.el8.n 1/6 2023-10-18T19:43:25.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-885.g07c1e9af.el8.noarch 2/6 2023-10-18T19:43:25.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-885.g07c1e9af.el8.no 3/6 2023-10-18T19:43:25.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-18T19:43:25.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-18T19:43:26.666 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-18T19:43:26.667 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:26.667 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:43:26.667 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:26.667 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:26.667 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:26.668 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-18T19:43:26.668 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-18T19:43:26.668 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-18T19:43:26.668 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:26.668 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:43:26.787 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-18T19:43:27.073 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:43:27.075 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:27.075 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:43:27.075 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:27.075 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:43:27.075 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 47 k 2023-10-18T19:43:27.076 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-10-18T19:43:27.076 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-18T19:43:27.076 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-18T19:43:27.076 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-18T19:43:27.076 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-18T19:43:27.077 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-18T19:43:27.077 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-18T19:43:27.077 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-18T19:43:27.077 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-18T19:43:27.077 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-18T19:43:27.078 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-18T19:43:27.078 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-18T19:43:27.078 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-18T19:43:27.078 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:27.078 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:43:27.078 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:27.079 INFO:teuthology.orchestra.run.smithi156.stdout:Install 13 Packages 2023-10-18T19:43:27.079 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:27.079 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 1.9 M 2023-10-18T19:43:27.080 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 25 M 2023-10-18T19:43:27.080 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:43:27.421 INFO:teuthology.orchestra.run.smithi156.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 196 kB/s | 20 kB 00:00 2023-10-18T19:43:27.446 INFO:teuthology.orchestra.run.smithi156.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 209 kB/s | 27 kB 00:00 2023-10-18T19:43:27.620 INFO:teuthology.orchestra.run.smithi156.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 636 kB/s | 126 kB 00:00 2023-10-18T19:43:27.643 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:43:27.655 INFO:teuthology.orchestra.run.smithi156.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 526 kB/s | 110 kB 00:00 2023-10-18T19:43:27.680 INFO:teuthology.orchestra.run.smithi156.stdout:(5/13): ceph-mgr-rook-18.2.0-885.g07c1e9af.el8. 131 kB/s | 47 kB 00:00 2023-10-18T19:43:27.731 INFO:teuthology.orchestra.run.smithi156.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 566 kB/s | 43 kB 00:00 2023-10-18T19:43:27.756 INFO:teuthology.orchestra.run.smithi156.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.1 MB/s | 155 kB 00:00 2023-10-18T19:43:27.781 INFO:teuthology.orchestra.run.smithi156.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 328 kB/s | 33 kB 00:00 2023-10-18T19:43:27.815 INFO:teuthology.orchestra.run.smithi156.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 194 kB/s | 16 kB 00:00 2023-10-18T19:43:27.865 INFO:teuthology.orchestra.run.smithi156.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 747 kB/s | 81 kB 00:00 2023-10-18T19:43:27.891 INFO:teuthology.orchestra.run.smithi156.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 854 kB/s | 64 kB 00:00 2023-10-18T19:43:27.916 INFO:teuthology.orchestra.run.smithi156.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-10-18T19:43:27.949 INFO:teuthology.orchestra.run.smithi156.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 7.0 MB/s | 1.2 MB 00:00 2023-10-18T19:43:27.950 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:27.950 INFO:teuthology.orchestra.run.smithi156.stdout:Total 2.2 MB/s | 1.9 MB 00:00 2023-10-18T19:43:28.009 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:43:28.025 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:43:28.025 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:43:28.111 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:43:28.114 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:43:28.628 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:43:28.842 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-18T19:43:29.065 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-18T19:43:29.237 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:43:29.238 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================================ 2023-10-18T19:43:29.238 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:43:29.239 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================================ 2023-10-18T19:43:29.239 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:43:29.239 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 7.4 M 2023-10-18T19:43:29.239 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-10-18T19:43:29.239 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-18T19:43:29.240 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-18T19:43:29.240 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-18T19:43:29.240 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-18T19:43:29.240 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-18T19:43:29.240 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-18T19:43:29.241 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-18T19:43:29.241 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:29.241 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:43:29.241 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================================ 2023-10-18T19:43:29.241 INFO:teuthology.orchestra.run.smithi082.stdout:Install 8 Packages 2023-10-18T19:43:29.241 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:29.242 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 35 M 2023-10-18T19:43:29.242 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 210 M 2023-10-18T19:43:29.243 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:43:29.275 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-18T19:43:29.513 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-18T19:43:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-18T19:43:29.952 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-18T19:43:30.113 INFO:teuthology.orchestra.run.smithi082.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 9.7 MB/s | 4.7 MB 00:00 2023-10-18T19:43:30.205 INFO:teuthology.orchestra.run.smithi082.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.4 MB/s | 4.9 MB 00:00 2023-10-18T19:43:30.279 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-18T19:43:30.449 INFO:teuthology.orchestra.run.smithi082.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 922 kB/s | 225 kB 00:00 2023-10-18T19:43:30.503 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-18T19:43:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.9 MB/s | 3.7 MB 00:00 2023-10-18T19:43:31.067 INFO:teuthology.orchestra.run.smithi082.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.4 MB/s | 645 kB 00:00 2023-10-18T19:43:31.100 INFO:teuthology.orchestra.run.smithi082.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2023-10-18T19:43:31.566 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-18T19:43:31.584 INFO:teuthology.orchestra.run.smithi082.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 12 MB/s | 14 MB 00:01 2023-10-18T19:43:31.863 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-18T19:43:32.097 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-18T19:43:32.267 INFO:teuthology.orchestra.run.smithi082.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.0-885 2.8 MB/s | 7.4 MB 00:02 2023-10-18T19:43:32.268 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:32.269 INFO:teuthology.orchestra.run.smithi082.stdout:Total 12 MB/s | 35 MB 00:03 2023-10-18T19:43:32.326 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-18T19:43:32.500 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:43:32.514 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:18.2.0-885.g07c1e9af.el8.noarch 13/13 2023-10-18T19:43:32.517 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:43:32.518 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:43:32.802 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-885.g07c1e9af.el8.noarch 13/13 2023-10-18T19:43:32.803 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:18.2.0-885.g07c1e9af.el8.noarch 1/13 2023-10-18T19:43:32.803 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-18T19:43:32.803 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-18T19:43:32.803 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-18T19:43:32.804 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-18T19:43:32.804 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-18T19:43:32.804 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-18T19:43:32.804 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-18T19:43:32.804 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-18T19:43:32.805 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-18T19:43:32.805 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-18T19:43:32.805 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-18T19:43:32.902 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:43:32.905 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:43:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:43:33.531 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-18T19:43:33.531 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:33.532 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:43:33.532 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:33.532 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-18T19:43:33.532 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-18T19:43:33.532 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-18T19:43:33.532 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-18T19:43:33.533 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-18T19:43:33.533 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-18T19:43:33.533 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-18T19:43:33.533 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-18T19:43:33.533 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-18T19:43:33.533 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-18T19:43:33.534 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-18T19:43:33.534 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-18T19:43:33.534 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:33.534 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:33.664 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-18T19:43:33.699 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-cephadm 2023-10-18T19:43:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-18T19:43:33.964 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-18T19:43:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-18T19:43:34.568 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:05 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:34.772 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-18T19:43:35.386 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-18T19:43:35.572 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-18T19:43:36.181 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:43:36.182 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:36.182 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:43:36.182 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:36.182 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:43:36.183 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 132 k 2023-10-18T19:43:36.183 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-10-18T19:43:36.183 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-18T19:43:36.183 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-18T19:43:36.183 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-18T19:43:36.183 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-18T19:43:36.184 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:36.184 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:43:36.184 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:36.184 INFO:teuthology.orchestra.run.smithi156.stdout:Install 5 Packages 2023-10-18T19:43:36.184 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:36.185 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 6.0 M 2023-10-18T19:43:36.185 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 26 M 2023-10-18T19:43:36.185 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:43:36.266 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-18T19:43:36.617 INFO:teuthology.orchestra.run.smithi156.stdout:(1/5): ceph-mgr-cephadm-18.2.0-885.g07c1e9af.el 607 kB/s | 132 kB 00:00 2023-10-18T19:43:36.742 INFO:teuthology.orchestra.run.smithi156.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.5 MB/s | 538 kB 00:00 2023-10-18T19:43:36.884 INFO:teuthology.orchestra.run.smithi156.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.7 MB/s | 476 kB 00:00 2023-10-18T19:43:36.901 INFO:teuthology.orchestra.run.smithi156.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 380 kB/s | 60 kB 00:00 2023-10-18T19:43:37.285 INFO:teuthology.orchestra.run.smithi156.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.4 MB/s | 4.8 MB 00:00 2023-10-18T19:43:37.286 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:37.286 INFO:teuthology.orchestra.run.smithi156.stdout:Total 5.4 MB/s | 6.0 MB 00:01 2023-10-18T19:43:37.337 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:43:37.348 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:43:37.348 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:43:37.448 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-18T19:43:37.452 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:43:37.456 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:43:37.730 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-18T19:43:37.768 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:43:38.112 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-18T19:43:39.103 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-18T19:43:39.468 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-18T19:43:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-18T19:43:39.746 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-18T19:43:39.921 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 5/5 2023-10-18T19:43:40.217 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 5/5 2023-10-18T19:43:40.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/5 2023-10-18T19:43:40.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-18T19:43:40.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-18T19:43:40.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-18T19:43:40.749 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-18T19:43:40.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-18T19:43:40.869 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:40.869 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:43:40.869 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:40.869 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-18T19:43:40.870 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-18T19:43:40.870 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-18T19:43:40.870 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-18T19:43:40.870 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:40.870 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:40.971 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-885.g07c1e9af 8/8 2023-10-18T19:43:41.007 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-fuse 2023-10-18T19:43:41.292 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-885.g07c1e9af 8/8 2023-10-18T19:43:41.293 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-885.g07c1e9af 1/8 2023-10-18T19:43:41.293 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-18T19:43:41.293 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-18T19:43:41.294 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-18T19:43:41.294 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-18T19:43:41.294 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-18T19:43:41.294 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-18T19:43:41.866 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:12 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:41.957 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-18T19:43:41.957 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:41.958 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:43:41.958 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:41.958 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-18T19:43:41.958 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-18T19:43:41.959 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-18T19:43:41.959 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-18T19:43:41.959 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-18T19:43:41.959 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-18T19:43:41.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-18T19:43:41.960 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:41.960 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:43:42.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-rook 2023-10-18T19:43:42.931 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:57 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:43:43.480 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:43:43.482 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:43.482 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:43:43.482 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:43.483 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:43:43.483 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 849 k 2023-10-18T19:43:43.483 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-10-18T19:43:43.483 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-18T19:43:43.483 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:43.484 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:43:43.484 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:43.484 INFO:teuthology.orchestra.run.smithi156.stdout:Install 2 Packages 2023-10-18T19:43:43.484 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:43.484 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 932 k 2023-10-18T19:43:43.484 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 3.0 M 2023-10-18T19:43:43.485 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:43:43.601 INFO:teuthology.orchestra.run.smithi156.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-10-18T19:43:44.531 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:43:44.533 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:44.533 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:43:44.533 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:44.534 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:43:44.534 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 47 k 2023-10-18T19:43:44.534 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-10-18T19:43:44.534 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-18T19:43:44.534 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-18T19:43:44.535 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-18T19:43:44.535 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-18T19:43:44.535 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-18T19:43:44.535 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-18T19:43:44.535 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-18T19:43:44.535 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-18T19:43:44.536 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-18T19:43:44.536 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-18T19:43:44.536 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-18T19:43:44.536 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-18T19:43:44.537 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:44.537 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:43:44.537 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:44.537 INFO:teuthology.orchestra.run.smithi082.stdout:Install 13 Packages 2023-10-18T19:43:44.538 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:44.538 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 1.9 M 2023-10-18T19:43:44.538 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 25 M 2023-10-18T19:43:44.539 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:43:44.810 INFO:teuthology.orchestra.run.smithi156.stdout:(2/2): ceph-fuse-18.2.0-885.g07c1e9af.el8.x86_6 660 kB/s | 849 kB 00:01 2023-10-18T19:43:44.810 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:44.811 INFO:teuthology.orchestra.run.smithi156.stdout:Total 701 kB/s | 932 kB 00:01 2023-10-18T19:43:44.823 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:43:44.845 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:43:44.845 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:43:44.937 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:43:44.941 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:43:45.107 INFO:teuthology.orchestra.run.smithi082.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 198 kB/s | 20 kB 00:00 2023-10-18T19:43:45.111 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:43:45.132 INFO:teuthology.orchestra.run.smithi082.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 210 kB/s | 27 kB 00:00 2023-10-18T19:43:45.308 INFO:teuthology.orchestra.run.smithi082.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 645 kB/s | 126 kB 00:00 2023-10-18T19:43:45.332 INFO:teuthology.orchestra.run.smithi082.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 550 kB/s | 110 kB 00:00 2023-10-18T19:43:45.400 INFO:teuthology.orchestra.run.smithi082.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 638 kB/s | 43 kB 00:00 2023-10-18T19:43:45.402 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-18T19:43:45.425 INFO:teuthology.orchestra.run.smithi082.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.3 MB/s | 155 kB 00:00 2023-10-18T19:43:45.552 INFO:teuthology.orchestra.run.smithi082.stdout:(7/13): ceph-mgr-rook-18.2.0-885.g07c1e9af.el8. 87 kB/s | 47 kB 00:00 2023-10-18T19:43:45.642 INFO:teuthology.orchestra.run.smithi082.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 75 kB/s | 16 kB 00:00 2023-10-18T19:43:45.645 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 2/2 2023-10-18T19:43:45.668 INFO:teuthology.orchestra.run.smithi082.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 123 kB/s | 33 kB 00:00 2023-10-18T19:43:45.751 INFO:teuthology.orchestra.run.smithi082.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 770 kB/s | 64 kB 00:00 2023-10-18T19:43:45.823 INFO:teuthology.orchestra.run.smithi082.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 301 kB/s | 81 kB 00:00 2023-10-18T19:43:45.852 INFO:teuthology.orchestra.run.smithi082.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 612 kB/s | 61 kB 00:00 2023-10-18T19:43:46.028 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 2/2 2023-10-18T19:43:46.028 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 1/2 2023-10-18T19:43:46.277 INFO:teuthology.orchestra.run.smithi082.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.8 MB/s | 1.2 MB 00:00 2023-10-18T19:43:46.280 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:46.280 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2023-10-18T19:43:46.338 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:43:46.354 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:43:46.355 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:43:46.439 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:43:46.442 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:43:46.697 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-18T19:43:46.698 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:46.698 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:43:46.698 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-18T19:43:46.698 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:46.699 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:46.898 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-volume 2023-10-18T19:43:47.000 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:43:47.288 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-18T19:43:47.519 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-18T19:43:47.746 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-18T19:43:47.759 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:18 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-18T19:43:48.249 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-18T19:43:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-18T19:43:48.840 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-18T19:43:49.090 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-18T19:43:49.370 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:43:49.371 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:49.371 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-10-18T19:43:49.372 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:49.372 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:43:49.372 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 261 k 2023-10-18T19:43:49.372 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:49.372 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:43:49.373 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:49.373 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2023-10-18T19:43:49.373 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:49.373 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 261 k 2023-10-18T19:43:49.373 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 1.3 M 2023-10-18T19:43:49.374 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:43:49.755 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-volume-18.2.0-885.g07c1e9af.el8.noarch.rpm 682 kB/s | 261 kB 00:00 2023-10-18T19:43:49.756 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:49.756 INFO:teuthology.orchestra.run.smithi156.stdout:Total 680 kB/s | 261 kB 00:00 2023-10-18T19:43:49.757 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:43:49.766 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:43:49.766 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:43:49.798 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:43:49.802 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:43:49.978 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:43:50.152 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-18T19:43:50.195 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:43:50.442 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-18T19:43:50.620 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:43:50.675 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-18T19:43:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-18T19:43:51.135 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:18.2.0-885.g07c1e9af.el8.noarch 13/13 2023-10-18T19:43:51.250 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:43:51.250 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:51.250 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:43:51.250 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:51.251 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:51.251 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:51.432 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-885.g07c1e9af.el8.noarch 13/13 2023-10-18T19:43:51.432 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:18.2.0-885.g07c1e9af.el8.noarch 1/13 2023-10-18T19:43:51.432 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-18T19:43:51.432 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-18T19:43:51.433 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-18T19:43:51.433 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-18T19:43:51.433 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-18T19:43:51.433 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-18T19:43:51.433 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-18T19:43:51.434 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-18T19:43:51.434 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-18T19:43:51.434 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-18T19:43:51.434 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-18T19:43:51.494 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librados-devel 2023-10-18T19:43:52.194 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-18T19:43:52.195 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:52.195 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:43:52.195 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:52.195 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-18T19:43:52.195 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-18T19:43:52.196 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-18T19:43:52.196 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-18T19:43:52.196 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-18T19:43:52.196 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-18T19:43:52.196 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-18T19:43:52.197 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-18T19:43:52.197 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-18T19:43:52.197 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-18T19:43:52.197 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-18T19:43:52.197 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-18T19:43:52.198 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:52.198 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:43:52.322 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-cephadm 2023-10-18T19:43:52.364 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:23 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:53.185 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:08 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:43:53.947 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:43:53.948 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:53.948 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2023-10-18T19:43:53.948 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:53.948 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:43:53.949 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 139 k 2023-10-18T19:43:53.949 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:53.949 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:43:53.949 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:43:53.949 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2023-10-18T19:43:53.949 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:53.950 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 139 k 2023-10-18T19:43:53.950 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 513 k 2023-10-18T19:43:53.951 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:43:54.397 INFO:teuthology.orchestra.run.smithi156.stdout:librados-devel-18.2.0-885.g07c1e9af.el8.x86_64. 310 kB/s | 139 kB 00:00 2023-10-18T19:43:54.398 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:54.398 INFO:teuthology.orchestra.run.smithi156.stdout:Total 309 kB/s | 139 kB 00:00 2023-10-18T19:43:54.399 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:43:54.404 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:43:54.404 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:43:54.453 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:43:54.455 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:43:54.565 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:43:54.793 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:43:54.801 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:43:54.802 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:54.802 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:43:54.802 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:54.803 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:43:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 132 k 2023-10-18T19:43:54.803 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-10-18T19:43:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-18T19:43:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-18T19:43:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-18T19:43:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-18T19:43:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:54.804 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:43:54.804 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:43:54.805 INFO:teuthology.orchestra.run.smithi082.stdout:Install 5 Packages 2023-10-18T19:43:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:54.805 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 6.0 M 2023-10-18T19:43:54.806 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 26 M 2023-10-18T19:43:54.806 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:43:55.094 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:43:55.176 INFO:teuthology.orchestra.run.smithi082.stdout:(1/5): ceph-mgr-cephadm-18.2.0-885.g07c1e9af.el 673 kB/s | 132 kB 00:00 2023-10-18T19:43:55.268 INFO:teuthology.orchestra.run.smithi082.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.8 MB/s | 538 kB 00:00 2023-10-18T19:43:55.460 INFO:teuthology.orchestra.run.smithi082.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 10 MB/s | 4.8 MB 00:00 2023-10-18T19:43:55.710 INFO:teuthology.orchestra.run.smithi082.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 136 kB/s | 60 kB 00:00 2023-10-18T19:43:55.729 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:43:55.730 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:55.730 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:43:55.730 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:43:55.730 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:43:55.730 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:55.844 INFO:teuthology.orchestra.run.smithi082.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 713 kB/s | 476 kB 00:00 2023-10-18T19:43:55.844 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:43:55.845 INFO:teuthology.orchestra.run.smithi082.stdout:Total 5.7 MB/s | 6.0 MB 00:01 2023-10-18T19:43:55.898 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:43:55.909 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:43:55.909 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:43:55.938 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install libcephfs2 2023-10-18T19:43:56.017 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:43:56.020 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:43:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:43:56.689 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-18T19:43:56.813 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:27 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:57.656 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-18T19:43:58.227 INFO:teuthology.orchestra.run.smithi156.stdout:Package libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:43:58.422 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:43:58.423 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2023-10-18T19:43:58.423 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:43:58.468 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-18T19:43:58.518 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install libcephfs-devel 2023-10-18T19:43:58.759 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-18T19:43:58.958 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 5/5 2023-10-18T19:43:59.242 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 5/5 2023-10-18T19:43:59.242 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 1/5 2023-10-18T19:43:59.242 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-18T19:43:59.243 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-18T19:43:59.243 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-18T19:43:59.380 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:30 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:43:59.881 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-18T19:43:59.881 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:59.882 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:43:59.882 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:43:59.882 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-18T19:43:59.882 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-18T19:43:59.882 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-18T19:43:59.883 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-18T19:43:59.883 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:43:59.883 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:00.021 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-fuse 2023-10-18T19:44:00.870 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:15 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:00.987 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:00.988 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:00.988 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2023-10-18T19:44:00.989 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:00.989 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:44:00.989 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 30 k 2023-10-18T19:44:00.989 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:00.989 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:44:00.989 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:00.990 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2023-10-18T19:44:00.990 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:00.990 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 30 k 2023-10-18T19:44:00.990 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 137 k 2023-10-18T19:44:00.990 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:44:01.151 INFO:teuthology.orchestra.run.smithi156.stdout:libcephfs-devel-18.2.0-885.g07c1e9af.el8.x86_64 189 kB/s | 30 kB 00:00 2023-10-18T19:44:01.152 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:01.152 INFO:teuthology.orchestra.run.smithi156.stdout:Total 187 kB/s | 30 kB 00:00 2023-10-18T19:44:01.152 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:44:01.155 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:44:01.155 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:44:01.177 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:44:01.180 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:44:01.265 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:44:01.433 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:01.701 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:02.314 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:02.315 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:02.315 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:44:02.315 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:44:02.315 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:02.315 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:02.427 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librados2 2023-10-18T19:44:02.471 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:02.471 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:02.472 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:44:02.472 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:02.472 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:44:02.472 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 849 k 2023-10-18T19:44:02.472 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-10-18T19:44:02.473 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-18T19:44:02.473 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:02.473 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:44:02.473 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:02.473 INFO:teuthology.orchestra.run.smithi082.stdout:Install 2 Packages 2023-10-18T19:44:02.474 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:02.474 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 932 k 2023-10-18T19:44:02.474 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 3.0 M 2023-10-18T19:44:02.475 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:44:02.602 INFO:teuthology.orchestra.run.smithi082.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 979 kB/s | 83 kB 00:00 2023-10-18T19:44:02.902 INFO:teuthology.orchestra.run.smithi082.stdout:(2/2): ceph-fuse-18.2.0-885.g07c1e9af.el8.x86_6 2.2 MB/s | 849 kB 00:00 2023-10-18T19:44:02.902 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:02.903 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.1 MB/s | 932 kB 00:00 2023-10-18T19:44:02.913 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:44:02.935 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:44:02.935 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:44:03.021 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:44:03.024 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:44:03.190 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:44:03.296 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:34 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:44:03.480 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-18T19:44:03.707 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 2/2 2023-10-18T19:44:04.100 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 2/2 2023-10-18T19:44:04.101 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 1/2 2023-10-18T19:44:04.709 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:04.743 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-18T19:44:04.743 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:04.743 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:44:04.743 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-18T19:44:04.744 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:04.744 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:04.904 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:04.905 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2023-10-18T19:44:04.906 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:04.977 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-volume 2023-10-18T19:44:05.009 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librbd1 2023-10-18T19:44:05.844 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:20 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:05.867 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:36 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:44:07.271 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:07.435 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:07.436 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:07.436 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-10-18T19:44:07.437 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:07.437 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:44:07.437 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:18.2.0-885.g07c1e9af.el8 ceph-noarch 261 k 2023-10-18T19:44:07.437 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:07.437 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:44:07.438 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:07.438 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2023-10-18T19:44:07.438 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:07.438 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 261 k 2023-10-18T19:44:07.438 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 1.3 M 2023-10-18T19:44:07.439 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:44:07.470 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:07.471 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2023-10-18T19:44:07.471 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:07.560 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rados 2023-10-18T19:44:07.806 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-volume-18.2.0-885.g07c1e9af.el8.noarch.rpm 709 kB/s | 261 kB 00:00 2023-10-18T19:44:07.807 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:07.807 INFO:teuthology.orchestra.run.smithi082.stdout:Total 707 kB/s | 261 kB 00:00 2023-10-18T19:44:07.807 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:44:07.817 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:44:07.817 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:44:07.849 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:44:07.852 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:44:08.034 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:44:08.306 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:44:08.419 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:39 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:44:08.714 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:44:09.853 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-rados-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:10.001 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:18.2.0-885.g07c1e9af.el8.noarch 1/1 2023-10-18T19:44:10.002 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:10.002 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:44:10.002 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:18.2.0-885.g07c1e9af.el8.noarch 2023-10-18T19:44:10.002 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:10.003 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:10.050 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:10.051 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2023-10-18T19:44:10.051 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:10.168 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rgw 2023-10-18T19:44:11.037 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:42 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:44:11.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados-devel 2023-10-18T19:44:12.022 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:27 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:12.451 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-rgw-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:12.647 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:12.648 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2023-10-18T19:44:12.648 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:12.744 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-cephfs 2023-10-18T19:44:13.601 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:44 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:44:13.608 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:13.609 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:13.609 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2023-10-18T19:44:13.609 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:13.609 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:44:13.610 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 139 k 2023-10-18T19:44:13.610 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:13.610 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:44:13.610 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:13.610 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2023-10-18T19:44:13.611 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:13.611 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 139 k 2023-10-18T19:44:13.611 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 513 k 2023-10-18T19:44:13.612 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:44:13.914 INFO:teuthology.orchestra.run.smithi082.stdout:librados-devel-18.2.0-885.g07c1e9af.el8.x86_64. 457 kB/s | 139 kB 00:00 2023-10-18T19:44:13.915 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:13.915 INFO:teuthology.orchestra.run.smithi082.stdout:Total 455 kB/s | 139 kB 00:00 2023-10-18T19:44:13.916 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:44:13.921 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:44:13.921 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:44:13.971 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:44:13.974 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:44:14.079 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:44:14.362 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:14.655 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:14.990 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-cephfs-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:15.186 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:15.187 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2023-10-18T19:44:15.187 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:15.278 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:44:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:44:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:15.279 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:15.296 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rbd 2023-10-18T19:44:15.510 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs2 2023-10-18T19:44:16.160 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:47 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:44:16.375 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:31 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:17.561 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-rbd-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:17.756 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:17.757 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2023-10-18T19:44:17.757 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:17.778 INFO:teuthology.orchestra.run.smithi082.stdout:Package libcephfs2-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:17.854 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-fuse 2023-10-18T19:44:17.978 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:17.979 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-10-18T19:44:17.979 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:18.077 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs-devel 2023-10-18T19:44:18.715 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:49 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:44:18.932 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:33 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:20.305 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:20.306 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:20.306 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2023-10-18T19:44:20.306 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:20.307 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:44:20.307 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 95 k 2023-10-18T19:44:20.307 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:20.307 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:44:20.307 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:20.308 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2023-10-18T19:44:20.308 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:20.308 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 95 k 2023-10-18T19:44:20.308 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 280 k 2023-10-18T19:44:20.308 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:44:20.525 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:20.526 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:20.526 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2023-10-18T19:44:20.526 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:20.527 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:44:20.527 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 30 k 2023-10-18T19:44:20.527 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:20.527 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:44:20.527 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:20.528 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2023-10-18T19:44:20.528 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:20.528 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 30 k 2023-10-18T19:44:20.528 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 137 k 2023-10-18T19:44:20.528 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:44:20.599 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-fuse-18.2.0-885.g07c1e9af.el8.x86_64.rpm 325 kB/s | 95 kB 00:00 2023-10-18T19:44:20.599 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:20.599 INFO:teuthology.orchestra.run.smithi156.stdout:Total 323 kB/s | 95 kB 00:00 2023-10-18T19:44:20.600 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:44:20.610 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:44:20.611 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:44:20.661 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:44:20.664 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:44:20.668 INFO:teuthology.orchestra.run.smithi082.stdout:libcephfs-devel-18.2.0-885.g07c1e9af.el8.x86_64 216 kB/s | 30 kB 00:00 2023-10-18T19:44:20.668 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:20.669 INFO:teuthology.orchestra.run.smithi082.stdout:Total 214 kB/s | 30 kB 00:00 2023-10-18T19:44:20.669 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:44:20.672 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:44:20.672 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:44:20.696 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:44:20.699 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:44:20.765 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:44:20.781 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:44:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:20.997 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:21.272 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:21.288 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:21.928 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:44:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:44:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:21.929 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:22.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:22.071 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:22.071 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:44:22.071 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:44:22.072 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:22.072 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:22.182 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados2 2023-10-18T19:44:22.185 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-mirror 2023-10-18T19:44:23.039 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:38 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:23.058 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:54 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:44:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:24.653 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:24.654 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-10-18T19:44:24.654 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:24.659 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:24.660 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:24.660 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2023-10-18T19:44:24.660 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:24.661 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:44:24.661 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 3.4 M 2023-10-18T19:44:24.661 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:24.661 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:44:24.661 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:24.661 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2023-10-18T19:44:24.662 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:24.662 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 3.4 M 2023-10-18T19:44:24.662 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 16 M 2023-10-18T19:44:24.663 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:44:24.747 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librbd1 2023-10-18T19:44:25.369 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-mirror-18.2.0-885.g07c1e9af.el8.x86_64.rpm 4.9 MB/s | 3.4 MB 00:00 2023-10-18T19:44:25.370 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:25.370 INFO:teuthology.orchestra.run.smithi156.stdout:Total 4.9 MB/s | 3.4 MB 00:00 2023-10-18T19:44:25.371 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:44:25.382 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:44:25.383 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:44:25.446 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:44:25.448 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:44:25.611 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:40 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:25.903 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:44:26.417 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:26.854 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:27.020 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:27.240 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:27.241 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-10-18T19:44:27.242 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:27.328 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rados 2023-10-18T19:44:27.524 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:27.525 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:27.525 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:44:27.525 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:44:27.525 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:27.526 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:27.777 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-nbd 2023-10-18T19:44:28.182 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:43 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:28.648 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:59 ago on Wed 18 Oct 2023 07:41:29 PM UTC. 2023-10-18T19:44:29.585 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rados-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:29.778 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:29.779 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-10-18T19:44:29.779 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:29.868 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rgw 2023-10-18T19:44:30.247 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-10-18T19:44:30.248 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:30.248 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2023-10-18T19:44:30.249 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:30.249 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-10-18T19:44:30.249 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 179 k 2023-10-18T19:44:30.249 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:30.249 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-10-18T19:44:30.250 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-10-18T19:44:30.250 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2023-10-18T19:44:30.250 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:30.251 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 179 k 2023-10-18T19:44:30.251 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 568 k 2023-10-18T19:44:30.251 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-10-18T19:44:30.431 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-nbd-18.2.0-885.g07c1e9af.el8.x86_64.rpm 990 kB/s | 179 kB 00:00 2023-10-18T19:44:30.431 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:30.432 INFO:teuthology.orchestra.run.smithi156.stdout:Total 983 kB/s | 179 kB 00:00 2023-10-18T19:44:30.432 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-10-18T19:44:30.443 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-10-18T19:44:30.443 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-10-18T19:44:30.492 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-10-18T19:44:30.495 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-10-18T19:44:30.613 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-10-18T19:44:30.735 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:45 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:30.838 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:31.129 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-nbd-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:31.776 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:31.777 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:31.777 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-10-18T19:44:31.777 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:44:31.777 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:31.777 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-10-18T19:44:32.009 DEBUG:teuthology.parallel:result is None 2023-10-18T19:44:32.137 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rgw-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:32.334 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:32.335 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-10-18T19:44:32.335 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:32.709 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-cephfs 2023-10-18T19:44:33.563 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:48 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:34.971 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-cephfs-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:35.167 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:35.168 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-10-18T19:44:35.168 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:35.283 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rbd 2023-10-18T19:44:36.146 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:51 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:37.558 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rbd-2:18.2.0-885.g07c1e9af.el8.x86_64 is already installed. 2023-10-18T19:44:37.754 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:37.755 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-10-18T19:44:37.755 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:37.861 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-fuse 2023-10-18T19:44:38.718 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:53 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:40.317 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:40.318 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:40.318 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2023-10-18T19:44:40.318 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:40.318 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:44:40.318 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 95 k 2023-10-18T19:44:40.319 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:40.319 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:44:40.319 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:40.319 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2023-10-18T19:44:40.319 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:40.320 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 95 k 2023-10-18T19:44:40.320 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 280 k 2023-10-18T19:44:40.320 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:44:40.524 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-fuse-18.2.0-885.g07c1e9af.el8.x86_64.rpm 462 kB/s | 95 kB 00:00 2023-10-18T19:44:40.524 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:40.525 INFO:teuthology.orchestra.run.smithi082.stdout:Total 459 kB/s | 95 kB 00:00 2023-10-18T19:44:40.525 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:44:40.535 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:44:40.535 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:44:40.585 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:44:40.588 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:44:40.701 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:44:40.898 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:41.182 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:41.794 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:41.794 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:41.795 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:44:41.795 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:44:41.795 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:41.795 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:42.051 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-mirror 2023-10-18T19:44:42.925 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:57 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:44.531 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:44.531 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:44.531 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2023-10-18T19:44:44.532 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:44.532 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:44:44.532 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 3.4 M 2023-10-18T19:44:44.532 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:44.532 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:44:44.533 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:44.533 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2023-10-18T19:44:44.533 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:44.533 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 3.4 M 2023-10-18T19:44:44.533 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 16 M 2023-10-18T19:44:44.534 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:44:45.118 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-mirror-18.2.0-885.g07c1e9af.el8.x86_64.rpm 5.9 MB/s | 3.4 MB 00:00 2023-10-18T19:44:45.119 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:45.119 INFO:teuthology.orchestra.run.smithi082.stdout:Total 5.9 MB/s | 3.4 MB 00:00 2023-10-18T19:44:45.120 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:44:45.132 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:44:45.132 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:44:45.194 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:44:45.197 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:44:45.666 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:44:45.889 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:46.328 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:46.954 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:46.954 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:46.955 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:44:46.955 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:44:46.955 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:46.955 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:47.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-nbd 2023-10-18T19:44:48.038 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:03:03 ago on Wed 18 Oct 2023 07:41:45 PM UTC. 2023-10-18T19:44:49.644 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-10-18T19:44:49.645 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:49.645 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2023-10-18T19:44:49.645 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:49.645 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-10-18T19:44:49.646 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:18.2.0-885.g07c1e9af.el8 ceph 179 k 2023-10-18T19:44:49.646 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:49.646 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-10-18T19:44:49.646 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-10-18T19:44:49.646 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2023-10-18T19:44:49.647 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:49.647 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 179 k 2023-10-18T19:44:49.647 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 568 k 2023-10-18T19:44:49.647 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-10-18T19:44:49.938 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-nbd-18.2.0-885.g07c1e9af.el8.x86_64.rpm 614 kB/s | 179 kB 00:00 2023-10-18T19:44:49.939 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-10-18T19:44:49.939 INFO:teuthology.orchestra.run.smithi082.stdout:Total 611 kB/s | 179 kB 00:00 2023-10-18T19:44:49.940 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-10-18T19:44:49.950 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-10-18T19:44:49.951 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-10-18T19:44:50.000 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-10-18T19:44:50.003 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-10-18T19:44:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-10-18T19:44:50.304 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:50.630 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-nbd-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:51.329 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:18.2.0-885.g07c1e9af.el8.x86_64 1/1 2023-10-18T19:44:51.329 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:51.329 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-10-18T19:44:51.330 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:18.2.0-885.g07c1e9af.el8.x86_64 2023-10-18T19:44:51.330 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:51.330 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-10-18T19:44:51.592 DEBUG:teuthology.parallel:result is None 2023-10-18T19:44:51.593 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=07c1e9af8fb194e5ad8a71499ead549c30345076 2023-10-18T19:44:51.823 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-18T19:44:51.846 INFO:teuthology.orchestra.run.smithi082.stdout:18.2.0-885.g07c1e9af.el8 2023-10-18T19:44:51.846 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-885.g07c1e9af.el8 2023-10-18T19:44:51.847 INFO:teuthology.task.install:The correct ceph version 18.2.0-885.g07c1e9af is installed. 2023-10-18T19:44:51.847 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=07c1e9af8fb194e5ad8a71499ead549c30345076 2023-10-18T19:44:51.991 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-18T19:44:52.019 INFO:teuthology.orchestra.run.smithi156.stdout:18.2.0-885.g07c1e9af.el8 2023-10-18T19:44:52.019 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-885.g07c1e9af.el8 2023-10-18T19:44:52.020 INFO:teuthology.task.install:The correct ceph version 18.2.0-885.g07c1e9af is installed. 2023-10-18T19:44:52.020 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-18T19:44:52.021 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:52.021 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-18T19:44:52.050 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:44:52.050 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-18T19:44:52.092 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-18T19:44:52.093 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:52.093 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2023-10-18T19:44:52.120 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-18T19:44:52.188 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:44:52.189 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2023-10-18T19:44:52.220 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-18T19:44:52.291 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-18T19:44:52.291 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:52.292 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-18T19:44:52.319 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-18T19:44:52.386 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:44:52.386 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-18T19:44:52.416 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-18T19:44:52.485 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-18T19:44:52.486 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:52.486 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2023-10-18T19:44:52.514 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-18T19:44:52.580 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:44:52.581 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2023-10-18T19:44:52.610 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-18T19:44:52.679 INFO:teuthology.run_tasks:Running task ceph... 2023-10-18T19:44:52.745 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-18T19:44:52.746 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /var/log/ceph 2023-10-18T19:44:52.748 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /var/log/ceph 2023-10-18T19:44:52.776 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-18T19:44:52.776 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-18T19:44:52.815 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-18T19:44:52.844 INFO:tasks.ceph:Creating extra log directories... 2023-10-18T19:44:52.845 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-18T19:44:52.882 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-18T19:44:52.923 INFO:tasks.ceph:Creating ceph cluster c1... 2023-10-18T19:44:52.923 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-18T19:44:52.923 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650', 'branch': 'wip-yuri-testing-2023-10-18-0812-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7431650', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-18-0812-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri-testing-2023-10-18-0812-reef', 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1349, 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-yuri-testing-2023-10-18-0812-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'targets': {'smithi082.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvyj4AREUB5UtfXpWW0uvjhICdkbhKvR7vk+X63lvYbeR9OxwLhNt/cWMOwhoABu8Uzx6tjPXC/XE/KcEEbuYc=', 'smithi156.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEoPyxVfTkY/yghFEHuNDraaBPv1OdiYKJ4EOPuHZdPIGA5KTcBjR8Q6W0QaX3LPal9vMY0E9N+bsBvCnjgETOg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-18_19:20:22', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-18T19:44:52.924 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-18T19:44:52.949 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-18T19:44:52.969 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/run/ceph 2023-10-18T19:44:53.006 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2023-10-18T19:44:53.037 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:53.037 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2023-10-18T19:44:53.091 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-18T19:44:53.091 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2023-10-18T19:44:53.149 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-18T19:44:53.149 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T19:44:53.149 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 390016 Links: 1 2023-10-18T19:44:53.149 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T19:44:53.150 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T19:44:53.150 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2023-10-18 19:44:46.083738819 +0000 2023-10-18T19:44:53.150 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2023-10-18 19:40:14.625682256 +0000 2023-10-18T19:44:53.150 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2023-10-18 19:40:14.625682256 +0000 2023-10-18T19:44:53.150 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2023-10-18T19:44:53.151 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-18T19:44:53.214 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2023-10-18T19:44:53.215 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2023-10-18T19:44:53.215 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000234075 s, 2.2 MB/s 2023-10-18T19:44:53.216 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-18T19:44:53.274 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2023-10-18T19:44:53.332 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-18T19:44:53.333 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T19:44:53.333 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 395549 Links: 1 2023-10-18T19:44:53.333 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T19:44:53.333 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T19:44:53.334 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2023-10-18 19:44:46.083738819 +0000 2023-10-18T19:44:53.334 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2023-10-18 19:40:15.163672149 +0000 2023-10-18T19:44:53.334 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2023-10-18 19:40:15.163672149 +0000 2023-10-18T19:44:53.334 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2023-10-18T19:44:53.335 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-18T19:44:53.398 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2023-10-18T19:44:53.398 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2023-10-18T19:44:53.399 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000211882 s, 2.4 MB/s 2023-10-18T19:44:53.400 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-18T19:44:53.458 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2023-10-18T19:44:53.516 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-18T19:44:53.517 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T19:44:53.517 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 399443 Links: 1 2023-10-18T19:44:53.517 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T19:44:53.517 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T19:44:53.517 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2023-10-18 19:44:46.083738819 +0000 2023-10-18T19:44:53.518 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2023-10-18 19:40:15.711661854 +0000 2023-10-18T19:44:53.518 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2023-10-18 19:40:15.711661854 +0000 2023-10-18T19:44:53.518 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2023-10-18T19:44:53.518 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-18T19:44:53.582 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2023-10-18T19:44:53.582 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2023-10-18T19:44:53.582 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000280156 s, 1.8 MB/s 2023-10-18T19:44:53.583 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-18T19:44:53.642 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2023-10-18T19:44:53.700 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-18T19:44:53.701 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T19:44:53.701 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 401419 Links: 1 2023-10-18T19:44:53.701 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T19:44:53.701 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T19:44:53.701 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2023-10-18 19:44:46.083738819 +0000 2023-10-18T19:44:53.702 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2023-10-18 19:40:16.240651916 +0000 2023-10-18T19:44:53.702 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2023-10-18 19:40:16.240651916 +0000 2023-10-18T19:44:53.702 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2023-10-18T19:44:53.702 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-18T19:44:53.765 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2023-10-18T19:44:53.766 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2023-10-18T19:44:53.766 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000186521 s, 2.7 MB/s 2023-10-18T19:44:53.767 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-18T19:44:53.825 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T19:44:53.825 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-18T19:44:53.826 INFO:tasks.ceph:Generating config... 2023-10-18T19:44:53.827 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-18T19:44:53.827 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-10-18T19:44:53.828 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-18T19:44:53.828 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-18T19:44:53.828 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-18T19:44:53.828 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-18T19:44:53.829 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-18T19:44:53.829 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-18T19:44:53.829 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-10-18T19:44:53.829 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-18T19:44:53.829 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-18T19:44:53.830 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-18T19:44:53.830 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-18T19:44:53.830 INFO:tasks.ceph:[client] setuser = ceph 2023-10-18T19:44:53.830 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-10-18T19:44:53.831 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-18T19:44:53.831 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-18T19:44:53.831 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-18T19:44:53.831 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-18T19:44:53.832 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-18T19:44:53.832 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-18T19:44:53.832 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-18T19:44:53.832 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-18T19:44:53.833 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-18T19:44:53.833 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-18T19:44:53.833 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-18T19:44:53.833 INFO:tasks.ceph:Setting up c1.mon.a... 2023-10-18T19:44:53.834 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-10-18T19:44:53.914 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/c1.keyring 2023-10-18T19:44:53.916 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-10-18T19:44:53.961 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-18T19:44:53.988 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.82')] 2023-10-18T19:44:53.989 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.82'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-18T19:44:53.990 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:53.990 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-18T19:44:54.046 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.82 --print /home/ubuntu/cephtest/c1.monmap 2023-10-18T19:44:54.128 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:44:54.128 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:44:54.129 INFO:teuthology.orchestra.run.smithi082.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-10-18T19:44:54.129 INFO:teuthology.orchestra.run.smithi082.stdout:monmaptool: generated fsid b4c5e45a-fd0d-467e-ac65-1ce5d438401a 2023-10-18T19:44:54.129 INFO:teuthology.orchestra.run.smithi082.stdout:setting min_mon_release = pacific 2023-10-18T19:44:54.130 INFO:teuthology.orchestra.run.smithi082.stdout:epoch 0 2023-10-18T19:44:54.130 INFO:teuthology.orchestra.run.smithi082.stdout:fsid b4c5e45a-fd0d-467e-ac65-1ce5d438401a 2023-10-18T19:44:54.130 INFO:teuthology.orchestra.run.smithi082.stdout:last_changed 2023-10-18T19:44:54.127936+0000 2023-10-18T19:44:54.130 INFO:teuthology.orchestra.run.smithi082.stdout:created 2023-10-18T19:44:54.127936+0000 2023-10-18T19:44:54.131 INFO:teuthology.orchestra.run.smithi082.stdout:min_mon_release 16 (pacific) 2023-10-18T19:44:54.131 INFO:teuthology.orchestra.run.smithi082.stdout:election_strategy: 1 2023-10-18T19:44:54.131 INFO:teuthology.orchestra.run.smithi082.stdout:0: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.a 2023-10-18T19:44:54.131 INFO:teuthology.orchestra.run.smithi082.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-10-18T19:44:54.132 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-18T19:44:54.190 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID b4c5e45a-fd0d-467e-ac65-1ce5d438401a... 2023-10-18T19:44:54.191 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-18T19:44:54.233 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-18T19:44:54.275 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2023-10-18T19:44:54.275 INFO:teuthology.orchestra.run.smithi082.stdout: chdir = "" 2023-10-18T19:44:54.276 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-18T19:44:54.276 INFO:teuthology.orchestra.run.smithi082.stdout: auth supported = cephx 2023-10-18T19:44:54.276 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.276 INFO:teuthology.orchestra.run.smithi082.stdout: filestore xattr use omap = true 2023-10-18T19:44:54.276 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.277 INFO:teuthology.orchestra.run.smithi082.stdout: mon clock drift allowed = 1.000 2023-10-18T19:44:54.277 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.277 INFO:teuthology.orchestra.run.smithi082.stdout: osd crush chooseleaf type = 0 2023-10-18T19:44:54.277 INFO:teuthology.orchestra.run.smithi082.stdout: auth debug = true 2023-10-18T19:44:54.277 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.278 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on old message = true 2023-10-18T19:44:54.278 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on bug = true 2023-10-18T19:44:54.278 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.278 INFO:teuthology.orchestra.run.smithi082.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-18T19:44:54.278 INFO:teuthology.orchestra.run.smithi082.stdout: mon pg warn max object skew = 0 2023-10-18T19:44:54.279 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.279 INFO:teuthology.orchestra.run.smithi082.stdout: # disable pg_autoscaler by default for new pools 2023-10-18T19:44:54.279 INFO:teuthology.orchestra.run.smithi082.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-18T19:44:54.279 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.279 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default size = 2 2023-10-18T19:44:54.280 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.280 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow primary affinity = true 2023-10-18T19:44:54.280 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow pg remap = true 2023-10-18T19:44:54.280 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on legacy crush tunables = false 2023-10-18T19:44:54.281 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on crush straw calc version zero = false 2023-10-18T19:44:54.281 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on no sortbitwise = false 2023-10-18T19:44:54.281 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on osd down out interval zero = false 2023-10-18T19:44:54.281 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on too few osds = false 2023-10-18T19:44:54.281 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-18T19:44:54.281 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-18T19:44:54.282 INFO:teuthology.orchestra.run.smithi082.stdout: mon_allow_pool_size_one = true 2023-10-18T19:44:54.282 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.282 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-18T19:44:54.282 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.282 INFO:teuthology.orchestra.run.smithi082.stdout: osd default data pool replay window = 5 2023-10-18T19:44:54.283 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.283 INFO:teuthology.orchestra.run.smithi082.stdout: mon allow pool delete = true 2023-10-18T19:44:54.283 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.283 INFO:teuthology.orchestra.run.smithi082.stdout: mon cluster log file level = debug 2023-10-18T19:44:54.283 INFO:teuthology.orchestra.run.smithi082.stdout: debug asserts on shutdown = true 2023-10-18T19:44:54.284 INFO:teuthology.orchestra.run.smithi082.stdout: mon health detail to clog = false 2023-10-18T19:44:54.284 INFO:teuthology.orchestra.run.smithi082.stdout: mon host = 172.21.15.82 2023-10-18T19:44:54.284 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = b4c5e45a-fd0d-467e-ac65-1ce5d438401a 2023-10-18T19:44:54.284 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.285 INFO:teuthology.orchestra.run.smithi082.stdout:[osd] 2023-10-18T19:44:54.285 INFO:teuthology.orchestra.run.smithi082.stdout: osd journal size = 100 2023-10-18T19:44:54.285 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.285 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub load threshold = 5.0 2023-10-18T19:44:54.285 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub max interval = 600 2023-10-18T19:44:54.286 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock profile = high_recovery_ops 2023-10-18T19:44:54.286 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.286 INFO:teuthology.orchestra.run.smithi082.stdout: osd recover clone overlap = true 2023-10-18T19:44:54.286 INFO:teuthology.orchestra.run.smithi082.stdout: osd recovery max chunk = 1048576 2023-10-18T19:44:54.286 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.287 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug shutdown = true 2023-10-18T19:44:54.287 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug op order = true 2023-10-18T19:44:54.287 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify stray on activate = true 2023-10-18T19:44:54.287 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.287 INFO:teuthology.orchestra.run.smithi082.stdout: osd open classes on start = true 2023-10-18T19:44:54.288 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug pg log writeout = true 2023-10-18T19:44:54.288 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.288 INFO:teuthology.orchestra.run.smithi082.stdout: osd deep scrub update digest min age = 30 2023-10-18T19:44:54.288 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.288 INFO:teuthology.orchestra.run.smithi082.stdout: osd map max advance = 10 2023-10-18T19:44:54.289 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.289 INFO:teuthology.orchestra.run.smithi082.stdout: journal zero on create = true 2023-10-18T19:44:54.289 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.289 INFO:teuthology.orchestra.run.smithi082.stdout: filestore ondisk finisher threads = 3 2023-10-18T19:44:54.289 INFO:teuthology.orchestra.run.smithi082.stdout: filestore apply finisher threads = 3 2023-10-18T19:44:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: bdev debug aio = true 2023-10-18T19:44:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug misdirected ops = true 2023-10-18T19:44:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-10-18T19:44:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: debug osd = 20 2023-10-18T19:44:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: rocksdb delete range threshold = 0 2023-10-18T19:44:54.291 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.291 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr] 2023-10-18T19:44:54.291 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-10-18T19:44:54.291 INFO:teuthology.orchestra.run.smithi082.stdout: debug mgr = 20 2023-10-18T19:44:54.291 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2023-10-18T19:44:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2023-10-18T19:44:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2023-10-18T19:44:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2023-10-18T19:44:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: mgr/telemetry/nag = false 2023-10-18T19:44:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.293 INFO:teuthology.orchestra.run.smithi082.stdout:[mon] 2023-10-18T19:44:54.293 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-10-18T19:44:54.293 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2023-10-18T19:44:54.293 INFO:teuthology.orchestra.run.smithi082.stdout: debug paxos = 20 2023-10-18T19:44:54.293 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2023-10-18T19:44:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: mon data avail warn = 5 2023-10-18T19:44:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: mon mgr mkfs grace = 240 2023-10-18T19:44:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2023-10-18T19:44:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd reporter subtree level = osd 2023-10-18T19:44:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd prime pg temp = true 2023-10-18T19:44:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2023-10-18T19:44:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-18T19:44:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: auth mon ticket ttl = 660 # 11m 2023-10-18T19:44:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: auth service ticket ttl = 240 # 4m 2023-10-18T19:44:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: # don't complain about insecure global_id in the test suite 2023-10-18T19:44:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-18T19:44:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-18T19:44:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: # 1m isn't quite enough 2023-10-18T19:44:54.297 INFO:teuthology.orchestra.run.smithi082.stdout: mon_down_mkfs_grace = 2m 2023-10-18T19:44:54.297 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.297 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_filestore_osds = false 2023-10-18T19:44:54.297 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:44:54.297 INFO:teuthology.orchestra.run.smithi082.stdout:[client] 2023-10-18T19:44:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: rgw cache enabled = true 2023-10-18T19:44:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable ops log = true 2023-10-18T19:44:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable usage log = true 2023-10-18T19:44:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-18T19:44:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-18T19:44:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: debug rgw = 20 2023-10-18T19:44:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: debug rgw sync = 20 2023-10-18T19:44:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: rgw crypt require ssl = False 2023-10-18T19:44:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: rgw crypt s3 kms backend = testing 2023-10-18T19:44:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-18T19:44:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: rgw curl low speed time = 300 2023-10-18T19:44:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: rgw data log num shards = 4 2023-10-18T19:44:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: rgw md log max shards = 4 2023-10-18T19:44:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: rgw sync data inject err probability = 0.1 2023-10-18T19:44:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: rgw sync log trim interval = 0 2023-10-18T19:44:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: rgw sync meta inject err probability = 0.1 2023-10-18T19:44:54.301 INFO:teuthology.orchestra.run.smithi082.stdout: rgw sync obj etag verify = True 2023-10-18T19:44:54.301 INFO:teuthology.orchestra.run.smithi082.stdout: setgroup = ceph 2023-10-18T19:44:54.301 INFO:teuthology.orchestra.run.smithi082.stdout: setuser = ceph 2023-10-18T19:44:54.301 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.a] 2023-10-18T19:44:54.301 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2023-10-18T19:44:54.302 INFO:teuthology.orchestra.run.smithi082.stdout: rgw data notify interval msec = 0 2023-10-18T19:44:54.302 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2023-10-18T19:44:54.302 INFO:teuthology.orchestra.run.smithi082.stdout: osd_max_omap_entries_per_request = 10 2023-10-18T19:44:54.302 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2023-10-18T19:44:54.302 INFO:teuthology.orchestra.run.smithi082.stdout: osd_max_omap_entries_per_request = 1000 2023-10-18T19:44:54.303 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2023-10-18T19:44:54.303 INFO:teuthology.orchestra.run.smithi082.stdout: osd_max_omap_entries_per_request = 10000 2023-10-18T19:44:54.304 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2023-10-18T19:44:54.305 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2023-10-18T19:44:54.305 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-18T19:44:54.305 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2023-10-18T19:44:54.305 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.306 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2023-10-18T19:44:54.306 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.306 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2023-10-18T19:44:54.306 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.306 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2023-10-18T19:44:54.307 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2023-10-18T19:44:54.307 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.307 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2023-10-18T19:44:54.307 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2023-10-18T19:44:54.307 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.308 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-18T19:44:54.308 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2023-10-18T19:44:54.308 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.308 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2023-10-18T19:44:54.308 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-18T19:44:54.309 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.309 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2023-10-18T19:44:54.309 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.309 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2023-10-18T19:44:54.309 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2023-10-18T19:44:54.309 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2023-10-18T19:44:54.310 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2023-10-18T19:44:54.310 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2023-10-18T19:44:54.310 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2023-10-18T19:44:54.310 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2023-10-18T19:44:54.310 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-18T19:44:54.311 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-18T19:44:54.311 INFO:teuthology.orchestra.run.smithi156.stdout: mon_allow_pool_size_one = true 2023-10-18T19:44:54.311 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.311 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-18T19:44:54.312 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.312 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2023-10-18T19:44:54.312 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.312 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2023-10-18T19:44:54.312 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.313 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2023-10-18T19:44:54.313 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2023-10-18T19:44:54.313 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2023-10-18T19:44:54.313 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = 172.21.15.82 2023-10-18T19:44:54.313 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = b4c5e45a-fd0d-467e-ac65-1ce5d438401a 2023-10-18T19:44:54.314 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.314 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2023-10-18T19:44:54.314 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2023-10-18T19:44:54.314 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.315 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2023-10-18T19:44:54.315 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2023-10-18T19:44:54.315 INFO:teuthology.orchestra.run.smithi156.stdout: osd mclock profile = high_recovery_ops 2023-10-18T19:44:54.315 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.315 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2023-10-18T19:44:54.316 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2023-10-18T19:44:54.316 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.316 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2023-10-18T19:44:54.316 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2023-10-18T19:44:54.316 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2023-10-18T19:44:54.317 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.317 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2023-10-18T19:44:54.317 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2023-10-18T19:44:54.317 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.317 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2023-10-18T19:44:54.317 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.318 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2023-10-18T19:44:54.318 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.318 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2023-10-18T19:44:54.318 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.318 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2023-10-18T19:44:54.319 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2023-10-18T19:44:54.319 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.319 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2023-10-18T19:44:54.319 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2023-10-18T19:44:54.319 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-10-18T19:44:54.320 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2023-10-18T19:44:54.320 INFO:teuthology.orchestra.run.smithi156.stdout: rocksdb delete range threshold = 0 2023-10-18T19:44:54.320 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.320 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2023-10-18T19:44:54.320 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-10-18T19:44:54.320 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2023-10-18T19:44:54.321 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2023-10-18T19:44:54.321 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2023-10-18T19:44:54.321 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2023-10-18T19:44:54.321 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2023-10-18T19:44:54.321 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2023-10-18T19:44:54.322 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.322 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2023-10-18T19:44:54.322 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-10-18T19:44:54.322 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2023-10-18T19:44:54.322 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2023-10-18T19:44:54.323 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2023-10-18T19:44:54.323 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2023-10-18T19:44:54.323 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2023-10-18T19:44:54.323 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2023-10-18T19:44:54.323 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2023-10-18T19:44:54.323 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2023-10-18T19:44:54.324 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2023-10-18T19:44:54.324 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.324 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-18T19:44:54.324 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2023-10-18T19:44:54.324 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2023-10-18T19:44:54.325 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.325 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2023-10-18T19:44:54.325 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-18T19:44:54.325 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-18T19:44:54.325 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.325 INFO:teuthology.orchestra.run.smithi156.stdout: # 1m isn't quite enough 2023-10-18T19:44:54.326 INFO:teuthology.orchestra.run.smithi156.stdout: mon_down_mkfs_grace = 2m 2023-10-18T19:44:54.326 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.326 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_filestore_osds = false 2023-10-18T19:44:54.326 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:44:54.326 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2023-10-18T19:44:54.327 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2023-10-18T19:44:54.327 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2023-10-18T19:44:54.327 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2023-10-18T19:44:54.327 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-18T19:44:54.328 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-18T19:44:54.328 INFO:teuthology.orchestra.run.smithi156.stdout: debug rgw = 20 2023-10-18T19:44:54.328 INFO:teuthology.orchestra.run.smithi156.stdout: debug rgw sync = 20 2023-10-18T19:44:54.328 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt require ssl = False 2023-10-18T19:44:54.328 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt s3 kms backend = testing 2023-10-18T19:44:54.328 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-18T19:44:54.329 INFO:teuthology.orchestra.run.smithi156.stdout: rgw curl low speed time = 300 2023-10-18T19:44:54.329 INFO:teuthology.orchestra.run.smithi156.stdout: rgw data log num shards = 4 2023-10-18T19:44:54.329 INFO:teuthology.orchestra.run.smithi156.stdout: rgw md log max shards = 4 2023-10-18T19:44:54.329 INFO:teuthology.orchestra.run.smithi156.stdout: rgw sync data inject err probability = 0.1 2023-10-18T19:44:54.329 INFO:teuthology.orchestra.run.smithi156.stdout: rgw sync log trim interval = 0 2023-10-18T19:44:54.330 INFO:teuthology.orchestra.run.smithi156.stdout: rgw sync meta inject err probability = 0.1 2023-10-18T19:44:54.330 INFO:teuthology.orchestra.run.smithi156.stdout: rgw sync obj etag verify = True 2023-10-18T19:44:54.330 INFO:teuthology.orchestra.run.smithi156.stdout: setgroup = ceph 2023-10-18T19:44:54.330 INFO:teuthology.orchestra.run.smithi156.stdout: setuser = ceph 2023-10-18T19:44:54.330 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2023-10-18T19:44:54.331 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2023-10-18T19:44:54.331 INFO:teuthology.orchestra.run.smithi156.stdout: rgw data notify interval msec = 0 2023-10-18T19:44:54.331 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2023-10-18T19:44:54.331 INFO:teuthology.orchestra.run.smithi156.stdout: osd_max_omap_entries_per_request = 10 2023-10-18T19:44:54.331 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2023-10-18T19:44:54.331 INFO:teuthology.orchestra.run.smithi156.stdout: osd_max_omap_entries_per_request = 1000 2023-10-18T19:44:54.332 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2023-10-18T19:44:54.332 INFO:teuthology.orchestra.run.smithi156.stdout: osd_max_omap_entries_per_request = 10000 2023-10-18T19:44:54.332 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-10-18T19:44:54.333 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-10-18T19:44:54.402 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-18T19:44:54.403 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:54.403 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-10-18T19:44:54.421 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:54.422 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-10-18T19:44:54.479 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T19:44:54.480 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:54.480 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/c1.keyring 2023-10-18T19:44:54.480 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-18T19:44:54.556 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:44:54.557 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-18T19:44:54.614 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T19:44:54.615 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:44:54.615 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/c1.keyring 2023-10-18T19:44:54.615 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-18T19:44:54.660 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:44:54.660 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-18T19:44:54.719 INFO:tasks.ceph:Setting up mon nodes... 2023-10-18T19:44:54.720 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-18T19:44:54.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-10-18T19:44:54.775 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-10-18T19:44:54.777 INFO:tasks.ceph:Setting up mds nodes... 2023-10-18T19:44:54.778 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-18T19:44:54.778 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-10-18T19:44:54.822 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/c1.client.0.keyring 2023-10-18T19:44:54.834 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-10-18T19:44:54.877 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/c1.client.1.keyring 2023-10-18T19:44:54.889 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-18T19:44:54.890 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-18T19:44:54.890 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-10-18T19:44:54.956 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T19:44:54.957 INFO:tasks.ceph:role: c1.osd.0 2023-10-18T19:44:54.957 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T19:44:54.957 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-18T19:44:55.031 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T19:44:55.031 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T19:44:55.031 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T19:44:55.031 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-18T19:44:55.031 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T19:44:55.032 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2023-10-18T19:44:55.032 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T19:44:55.032 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T19:44:55.032 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T19:44:55.032 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T19:44:55.467 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2023-10-18T19:44:55.469 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2023-10-18T19:44:55.469 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-10-18T19:44:55.636 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-10-18T19:44:55.669 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-10-18T19:44:55.737 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T19:44:55.738 INFO:tasks.ceph:role: c1.osd.1 2023-10-18T19:44:55.738 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T19:44:55.738 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-18T19:44:55.812 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T19:44:55.813 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T19:44:55.813 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T19:44:55.813 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-18T19:44:55.813 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T19:44:55.813 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2023-10-18T19:44:55.814 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T19:44:55.814 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T19:44:55.814 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T19:44:55.814 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T19:44:56.239 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2023-10-18T19:44:56.241 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2023-10-18T19:44:56.241 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-10-18T19:44:56.355 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-10-18T19:44:56.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-10-18T19:44:56.454 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T19:44:56.454 INFO:tasks.ceph:role: c1.osd.2 2023-10-18T19:44:56.454 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T19:44:56.455 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-18T19:44:56.528 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T19:44:56.528 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T19:44:56.528 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T19:44:56.528 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-18T19:44:56.529 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T19:44:56.529 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2023-10-18T19:44:56.529 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T19:44:56.529 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T19:44:56.529 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T19:44:56.530 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T19:44:56.967 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2023-10-18T19:44:56.969 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2023-10-18T19:44:56.969 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-10-18T19:44:57.136 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-10-18T19:44:57.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-18T19:44:57.283 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:57.281+0000 7f3b291ff640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-10-18T19:44:57.285 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:57.283+0000 7f3b291ff640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-10-18T19:44:57.285 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:57.283+0000 7f3b291ff640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-18T19:44:57.286 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:57.283+0000 7f3b291ff640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-18T19:44:57.286 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:57.283+0000 7f3b291ff640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-18T19:44:57.289 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:57.288+0000 7f3b291ff640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-10-18T19:44:59.367 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-10-18T19:44:59.400 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-18T19:44:59.493 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:59.491+0000 7f9e1d5e5640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-10-18T19:44:59.494 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:59.491+0000 7f9e1d5e5640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-10-18T19:44:59.494 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:59.491+0000 7f9e1d5e5640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-18T19:44:59.494 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:59.491+0000 7f9e1d5e5640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-18T19:44:59.494 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:59.491+0000 7f9e1d5e5640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-18T19:44:59.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:44:59.492+0000 7f9e1d5e5640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-10-18T19:45:01.574 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-10-18T19:45:01.606 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-18T19:45:01.699 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:01.697+0000 7f3c62e6a640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-10-18T19:45:01.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:01.697+0000 7f3c62e6a640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-10-18T19:45:01.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:01.697+0000 7f3c62e6a640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-18T19:45:01.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:01.697+0000 7f3c62e6a640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-18T19:45:01.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:01.697+0000 7f3c62e6a640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-18T19:45:01.701 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:01.698+0000 7f3c62e6a640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-10-18T19:45:03.778 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-10-18T19:45:03.811 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-18T19:45:03.812 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:03.812 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-10-18T19:45:03.882 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:03.883 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-10-18T19:45:03.952 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:03.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-10-18T19:45:04.021 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:04.022 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-10-18T19:45:04.090 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:04.091 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-10-18T19:45:04.149 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:04.150 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-10-18T19:45:04.208 INFO:tasks.ceph:Adding keys to all mons... 2023-10-18T19:45:04.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo tee -a /etc/ceph/c1.keyring 2023-10-18T19:45:04.274 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.x] 2023-10-18T19:45:04.274 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQC2NTBlZyslLhAAzINYGXiULttKW60pJcl0IQ== 2023-10-18T19:45:04.275 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2023-10-18T19:45:04.275 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQC5NTBl+rDUEBAAwDr6FhoP3XfS6GWGRcWQDA== 2023-10-18T19:45:04.275 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2023-10-18T19:45:04.275 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQC7NTBlUz9YHRAACKwoyrh3XYXLAAW49zIURg== 2023-10-18T19:45:04.275 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2023-10-18T19:45:04.276 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQC9NTBlyDaeKRAAzvpE+ZoNcpQis2CvqU13Rg== 2023-10-18T19:45:04.276 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2023-10-18T19:45:04.276 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQC2NTBleNP0MBAAcxmIhvs5dGx6n/xO7vwQhQ== 2023-10-18T19:45:04.276 INFO:teuthology.orchestra.run.smithi082.stdout:[client.1] 2023-10-18T19:45:04.276 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQC2NTBleCI8NBAAFIJCXnK1X/xaP572FEsLfA== 2023-10-18T19:45:04.277 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-18T19:45:04.371 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T19:45:04.424 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T19:45:04.478 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T19:45:04.532 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-18T19:45:04.589 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-18T19:45:04.645 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-18T19:45:04.646 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-10-18T19:45:04.673 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-10-18T19:45:04.772 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-10-18T19:45:04.800 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-10-18T19:45:04.858 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-10-18T19:45:04.858 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-10-18T19:45:04.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-10-18T19:45:04.900 INFO:tasks.ceph.c1.mon.a:Started 2023-10-18T19:45:04.900 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-10-18T19:45:04.900 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-10-18T19:45:04.901 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-10-18T19:45:04.902 INFO:tasks.ceph.c1.mgr.x:Started 2023-10-18T19:45:04.903 INFO:tasks.ceph:Setting crush tunables to default 2023-10-18T19:45:04.903 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster c1 osd crush tunables default 2023-10-18T19:45:05.109 INFO:teuthology.orchestra.run.smithi082.stderr:adjusted tunables profile to default 2023-10-18T19:45:05.120 INFO:tasks.ceph:check_enable_crimson: False 2023-10-18T19:45:05.120 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-10-18T19:45:05.120 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:05.121 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-10-18T19:45:05.147 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:05.148 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-10-18T19:45:05.216 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:05.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-10-18T19:45:05.285 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster c1 osd new 5d1e74e8-f2ea-4745-8b16-cf2d1531d1d7 0 2023-10-18T19:45:05.313 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:05.311+0000 7fab2c2fe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-18T19:45:05.497 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:05.495+0000 7fab2c2fe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-18T19:45:05.554 INFO:teuthology.orchestra.run.smithi082.stdout:0 2023-10-18T19:45:05.567 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster c1 osd new 08509149-c1a9-46e9-9724-12e897223e75 1 2023-10-18T19:45:05.779 INFO:teuthology.orchestra.run.smithi082.stdout:1 2023-10-18T19:45:05.792 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster c1 osd new e33dbf72-8228-4550-81cf-f466deb7b6b4 2 2023-10-18T19:45:05.861 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:05.859+0000 7fab2c2fe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-18T19:45:06.023 INFO:teuthology.orchestra.run.smithi082.stdout:2 2023-10-18T19:45:06.035 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-10-18T19:45:06.035 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-10-18T19:45:06.037 INFO:tasks.ceph.c1.osd.0:Started 2023-10-18T19:45:06.038 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-10-18T19:45:06.038 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-10-18T19:45:06.040 INFO:tasks.ceph.c1.osd.1:Started 2023-10-18T19:45:06.041 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-10-18T19:45:06.041 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-10-18T19:45:06.043 INFO:tasks.ceph.c1.osd.2:Started 2023-10-18T19:45:06.043 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-18T19:45:06.272 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:06.272 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":5,"fsid":"b4c5e45a-fd0d-467e-ac65-1ce5d438401a","created":"2023-10-18T19:45:05.013071+0000","modified":"2023-10-18T19:45:06.020600+0000","last_up_change":"0.000000","last_in_change":"2023-10-18T19:45:06.020600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5d1e74e8-f2ea-4745-8b16-cf2d1531d1d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"08509149-c1a9-46e9-9724-12e897223e75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e33dbf72-8228-4550-81cf-f466deb7b6b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:06.289 INFO:tasks.ceph.ceph_manager.c1:[] 2023-10-18T19:45:06.290 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-18T19:45:06.435 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:06.433+0000 7fab2c2fe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-18T19:45:06.592 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-18T19:45:06.634 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:06.632+0000 7fab2c2fe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-18T19:45:06.641 INFO:tasks.ceph.c1.osd.1.smithi082.stderr:2023-10-18T19:45:06.640+0000 7f29ae0a4640 -1 Falling back to public interface 2023-10-18T19:45:06.649 INFO:tasks.ceph.c1.osd.2.smithi082.stderr:2023-10-18T19:45:06.647+0000 7f5fbae78640 -1 Falling back to public interface 2023-10-18T19:45:06.670 INFO:tasks.ceph.c1.osd.0.smithi082.stderr:2023-10-18T19:45:06.668+0000 7fd02a455640 -1 Falling back to public interface 2023-10-18T19:45:06.788 INFO:teuthology.misc.health.smithi082.stdout: 2023-10-18T19:45:06.788 INFO:teuthology.misc.health.smithi082.stdout:{"epoch":5,"fsid":"b4c5e45a-fd0d-467e-ac65-1ce5d438401a","created":"2023-10-18T19:45:05.013071+0000","modified":"2023-10-18T19:45:06.020600+0000","last_up_change":"0.000000","last_in_change":"2023-10-18T19:45:06.020600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5d1e74e8-f2ea-4745-8b16-cf2d1531d1d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"08509149-c1a9-46e9-9724-12e897223e75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e33dbf72-8228-4550-81cf-f466deb7b6b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:06.799 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-18T19:45:07.018 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:07.016+0000 7fab2c2fe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-18T19:45:07.687 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:07.685+0000 7fab2c2fe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-18T19:45:07.760 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:07.758+0000 7fab2c2fe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-18T19:45:07.896 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:07.894+0000 7fab2c2fe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-18T19:45:07.966 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:07.964+0000 7fab2c2fe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-18T19:45:08.147 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:08.145+0000 7fab2c2fe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-18T19:45:08.225 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:08.224+0000 7fab2c2fe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-18T19:45:08.303 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:08.301+0000 7fab2c2fe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-18T19:45:08.402 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:08.400+0000 7fab2c2fe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-18T19:45:08.585 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:08.583+0000 7fab2c2fe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-18T19:45:08.667 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:08.665+0000 7fab2c2fe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-18T19:45:08.962 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:08.960+0000 7fab2c2fe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-18T19:45:09.033 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:09.031+0000 7fab2c2fe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-18T19:45:09.395 INFO:tasks.ceph.c1.osd.1.smithi082.stderr:2023-10-18T19:45:09.393+0000 7f29ae0a4640 -1 osd.1 0 log_to_monitors true 2023-10-18T19:45:09.406 INFO:tasks.ceph.c1.osd.2.smithi082.stderr:2023-10-18T19:45:09.404+0000 7f5fbae78640 -1 osd.2 0 log_to_monitors true 2023-10-18T19:45:09.440 INFO:tasks.ceph.c1.osd.0.smithi082.stderr:2023-10-18T19:45:09.438+0000 7fd02a455640 -1 osd.0 0 log_to_monitors true 2023-10-18T19:45:09.566 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:09.564+0000 7fab2c2fe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-18T19:45:09.881 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:09.879+0000 7fab2c2fe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-18T19:45:10.024 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:10.022+0000 7fab2c2fe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-18T19:45:10.418 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:10.416+0000 7fab2c2fe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-18T19:45:10.589 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:10.587+0000 7fab2c2fe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-18T19:45:10.757 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:10.755+0000 7fab2c2fe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-18T19:45:11.095 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:11.093+0000 7fab2c2fe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-18T19:45:11.673 INFO:tasks.ceph.c1.osd.2.smithi082.stderr:2023-10-18T19:45:11.671+0000 7f5faf219700 -1 osd.2 0 waiting for initial osdmap 2023-10-18T19:45:11.710 INFO:tasks.ceph.c1.osd.2.smithi082.stderr:2023-10-18T19:45:11.708+0000 7f5fa6df3700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T19:45:11.719 INFO:tasks.ceph.c1.osd.1.smithi082.stderr:2023-10-18T19:45:11.717+0000 7f29a2445700 -1 osd.1 0 waiting for initial osdmap 2023-10-18T19:45:11.733 INFO:tasks.ceph.c1.osd.1.smithi082.stderr:2023-10-18T19:45:11.732+0000 7f299a01f700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T19:45:12.079 INFO:tasks.ceph.c1.osd.0.smithi082.stderr:2023-10-18T19:45:12.077+0000 7fd01e7f6700 -1 osd.0 0 waiting for initial osdmap 2023-10-18T19:45:12.093 INFO:tasks.ceph.c1.osd.0.smithi082.stderr:2023-10-18T19:45:12.091+0000 7fd0163d0700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T19:45:12.147 INFO:tasks.ceph.c1.mgr.x.smithi082.stderr:2023-10-18T19:45:12.145+0000 7faaed714700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-18T19:45:13.103 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-18T19:45:13.429 INFO:teuthology.misc.health.smithi082.stdout: 2023-10-18T19:45:13.429 INFO:teuthology.misc.health.smithi082.stdout:{"epoch":8,"fsid":"b4c5e45a-fd0d-467e-ac65-1ce5d438401a","created":"2023-10-18T19:45:05.013071+0000","modified":"2023-10-18T19:45:12.718967+0000","last_up_change":"2023-10-18T19:45:12.718967+0000","last_in_change":"2023-10-18T19:45:06.020600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5d1e74e8-f2ea-4745-8b16-cf2d1531d1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":114214},{"type":"v1","addr":"172.21.15.82:6817","nonce":114214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":114214},{"type":"v1","addr":"172.21.15.82:6819","nonce":114214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":114214},{"type":"v1","addr":"172.21.15.82:6823","nonce":114214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":114214},{"type":"v1","addr":"172.21.15.82:6821","nonce":114214}]},"public_addr":"172.21.15.82:6817/114214","cluster_addr":"172.21.15.82:6819/114214","heartbeat_back_addr":"172.21.15.82:6823/114214","heartbeat_front_addr":"172.21.15.82:6821/114214","state":["exists","up"]},{"osd":1,"uuid":"08509149-c1a9-46e9-9724-12e897223e75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":114212},{"type":"v1","addr":"172.21.15.82:6801","nonce":114212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":114212},{"type":"v1","addr":"172.21.15.82:6803","nonce":114212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":114212},{"type":"v1","addr":"172.21.15.82:6807","nonce":114212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":114212},{"type":"v1","addr":"172.21.15.82:6805","nonce":114212}]},"public_addr":"172.21.15.82:6801/114212","cluster_addr":"172.21.15.82:6803/114212","heartbeat_back_addr":"172.21.15.82:6807/114212","heartbeat_front_addr":"172.21.15.82:6805/114212","state":["exists","up"]},{"osd":2,"uuid":"e33dbf72-8228-4550-81cf-f466deb7b6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":114213},{"type":"v1","addr":"172.21.15.82:6809","nonce":114213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":114213},{"type":"v1","addr":"172.21.15.82:6811","nonce":114213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":114213},{"type":"v1","addr":"172.21.15.82:6815","nonce":114213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":114213},{"type":"v1","addr":"172.21.15.82:6813","nonce":114213}]},"public_addr":"172.21.15.82:6809/114213","cluster_addr":"172.21.15.82:6811/114213","heartbeat_back_addr":"172.21.15.82:6815/114213","heartbeat_front_addr":"172.21.15.82:6813/114213","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:13.441 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-18T19:45:13.441 INFO:tasks.ceph:Creating RBD pool 2023-10-18T19:45:13.442 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster c1 osd pool create rbd 8 2023-10-18T19:45:14.158 INFO:teuthology.orchestra.run.smithi082.stderr:pool 'rbd' created 2023-10-18T19:45:14.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-18T19:45:15.160 INFO:teuthology.orchestra.run.smithi082.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-18T19:45:15.174 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-10-18T19:45:15.174 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-10-18T19:45:15.175 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-10-18T19:45:15.175 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-10-18T19:45:15.176 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-18T19:45:15.593 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:15.606 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":113800},{"type":"v1","addr":"172.21.15.82:6825","nonce":113800}]},"active_addr":"172.21.15.82:6825/113800","active_change":"2023-10-18T19:45:11.106485+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":2174604335}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":3067338936}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":4228569834}]}]} 2023-10-18T19:45:15.609 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-10-18T19:45:15.609 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-10-18T19:45:15.609 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-18T19:45:15.930 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:15.930 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":10,"fsid":"b4c5e45a-fd0d-467e-ac65-1ce5d438401a","created":"2023-10-18T19:45:05.013071+0000","modified":"2023-10-18T19:45:15.149134+0000","last_up_change":"2023-10-18T19:45:12.718967+0000","last_in_change":"2023-10-18T19:45:06.020600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-18T19:45:13.187085+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T19:45:13.774557+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5d1e74e8-f2ea-4745-8b16-cf2d1531d1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":114214},{"type":"v1","addr":"172.21.15.82:6817","nonce":114214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":114214},{"type":"v1","addr":"172.21.15.82:6819","nonce":114214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":114214},{"type":"v1","addr":"172.21.15.82:6823","nonce":114214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":114214},{"type":"v1","addr":"172.21.15.82:6821","nonce":114214}]},"public_addr":"172.21.15.82:6817/114214","cluster_addr":"172.21.15.82:6819/114214","heartbeat_back_addr":"172.21.15.82:6823/114214","heartbeat_front_addr":"172.21.15.82:6821/114214","state":["exists","up"]},{"osd":1,"uuid":"08509149-c1a9-46e9-9724-12e897223e75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":114212},{"type":"v1","addr":"172.21.15.82:6801","nonce":114212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":114212},{"type":"v1","addr":"172.21.15.82:6803","nonce":114212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":114212},{"type":"v1","addr":"172.21.15.82:6807","nonce":114212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":114212},{"type":"v1","addr":"172.21.15.82:6805","nonce":114212}]},"public_addr":"172.21.15.82:6801/114212","cluster_addr":"172.21.15.82:6803/114212","heartbeat_back_addr":"172.21.15.82:6807/114212","heartbeat_front_addr":"172.21.15.82:6805/114212","state":["exists","up"]},{"osd":2,"uuid":"e33dbf72-8228-4550-81cf-f466deb7b6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":114213},{"type":"v1","addr":"172.21.15.82:6809","nonce":114213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":114213},{"type":"v1","addr":"172.21.15.82:6811","nonce":114213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":114213},{"type":"v1","addr":"172.21.15.82:6815","nonce":114213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":114213},{"type":"v1","addr":"172.21.15.82:6813","nonce":114213}]},"public_addr":"172.21.15.82:6809/114213","cluster_addr":"172.21.15.82:6811/114213","heartbeat_back_addr":"172.21.15.82:6815/114213","heartbeat_front_addr":"172.21.15.82:6813/114213","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:12.078249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:10.349599+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:10.381979+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:15.943 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-10-18T19:45:15.944 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-18T19:45:16.264 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:16.265 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":11,"fsid":"b4c5e45a-fd0d-467e-ac65-1ce5d438401a","created":"2023-10-18T19:45:05.013071+0000","modified":"2023-10-18T19:45:16.149750+0000","last_up_change":"2023-10-18T19:45:12.718967+0000","last_in_change":"2023-10-18T19:45:06.020600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-18T19:45:13.187085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T19:45:13.774557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5d1e74e8-f2ea-4745-8b16-cf2d1531d1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":114214},{"type":"v1","addr":"172.21.15.82:6817","nonce":114214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":114214},{"type":"v1","addr":"172.21.15.82:6819","nonce":114214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":114214},{"type":"v1","addr":"172.21.15.82:6823","nonce":114214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":114214},{"type":"v1","addr":"172.21.15.82:6821","nonce":114214}]},"public_addr":"172.21.15.82:6817/114214","cluster_addr":"172.21.15.82:6819/114214","heartbeat_back_addr":"172.21.15.82:6823/114214","heartbeat_front_addr":"172.21.15.82:6821/114214","state":["exists","up"]},{"osd":1,"uuid":"08509149-c1a9-46e9-9724-12e897223e75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":114212},{"type":"v1","addr":"172.21.15.82:6801","nonce":114212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":114212},{"type":"v1","addr":"172.21.15.82:6803","nonce":114212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":114212},{"type":"v1","addr":"172.21.15.82:6807","nonce":114212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":114212},{"type":"v1","addr":"172.21.15.82:6805","nonce":114212}]},"public_addr":"172.21.15.82:6801/114212","cluster_addr":"172.21.15.82:6803/114212","heartbeat_back_addr":"172.21.15.82:6807/114212","heartbeat_front_addr":"172.21.15.82:6805/114212","state":["exists","up"]},{"osd":2,"uuid":"e33dbf72-8228-4550-81cf-f466deb7b6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":114213},{"type":"v1","addr":"172.21.15.82:6809","nonce":114213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":114213},{"type":"v1","addr":"172.21.15.82:6811","nonce":114213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":114213},{"type":"v1","addr":"172.21.15.82:6815","nonce":114213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":114213},{"type":"v1","addr":"172.21.15.82:6813","nonce":114213}]},"public_addr":"172.21.15.82:6809/114213","cluster_addr":"172.21.15.82:6811/114213","heartbeat_back_addr":"172.21.15.82:6815/114213","heartbeat_front_addr":"172.21.15.82:6813/114213","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:12.078249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:10.349599+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:10.381979+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:16.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-10-18T19:45:16.279 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-10-18T19:45:16.279 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-10-18T19:45:16.443 INFO:teuthology.orchestra.run.smithi082.stdout:34359738370 2023-10-18T19:45:16.443 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-10-18T19:45:16.453 INFO:teuthology.orchestra.run.smithi082.stdout:34359738370 2023-10-18T19:45:16.454 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-18T19:45:16.454 INFO:teuthology.orchestra.run.smithi082.stdout:34359738370 2023-10-18T19:45:16.455 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-10-18T19:45:16.779 INFO:teuthology.orchestra.run.smithi082.stdout:34359738369 2023-10-18T19:45:16.792 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2023-10-18T19:45:16.830 INFO:teuthology.orchestra.run.smithi082.stdout:34359738369 2023-10-18T19:45:16.835 INFO:teuthology.orchestra.run.smithi082.stdout:34359738369 2023-10-18T19:45:16.846 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2023-10-18T19:45:16.850 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2023-10-18T19:45:17.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-10-18T19:45:17.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-18T19:45:17.851 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-10-18T19:45:18.159 INFO:teuthology.orchestra.run.smithi082.stdout:34359738370 2023-10-18T19:45:18.173 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.2 2023-10-18T19:45:18.174 DEBUG:teuthology.parallel:result is None 2023-10-18T19:45:18.197 INFO:teuthology.orchestra.run.smithi082.stdout:34359738370 2023-10-18T19:45:18.210 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2023-10-18T19:45:18.211 DEBUG:teuthology.parallel:result is None 2023-10-18T19:45:18.234 INFO:teuthology.orchestra.run.smithi082.stdout:34359738370 2023-10-18T19:45:18.248 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2023-10-18T19:45:18.248 DEBUG:teuthology.parallel:result is None 2023-10-18T19:45:18.249 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-10-18T19:45:18.249 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-18T19:45:18.602 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:18.604 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2023-10-18T19:45:18.616 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-18T19:45:17.144824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82292,"kb_used_data":1436,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490508,"statfs":{"total":322122547200,"available":322038280192,"internally_reserved":0,"allocated":1470464,"data_stored":1255061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.152089+0000","last_change":"2023-10-18T19:45:15.157261+0000","last_active":"2023-10-18T19:45:16.152089+0000","last_peered":"2023-10-18T19:45:16.152089+0000","last_clean":"2023-10-18T19:45:16.152089+0000","last_became_active":"2023-10-18T19:45:15.157022+0000","last_became_peered":"2023-10-18T19:45:15.157022+0000","last_unstale":"2023-10-18T19:45:16.152089+0000","last_undegraded":"2023-10-18T19:45:16.152089+0000","last_fullsized":"2023-10-18T19:45:16.152089+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T20:51:30.705144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.151979+0000","last_change":"2023-10-18T19:45:15.156710+0000","last_active":"2023-10-18T19:45:16.151979+0000","last_peered":"2023-10-18T19:45:16.151979+0000","last_clean":"2023-10-18T19:45:16.151979+0000","last_became_active":"2023-10-18T19:45:15.156119+0000","last_became_peered":"2023-10-18T19:45:15.156119+0000","last_unstale":"2023-10-18T19:45:16.151979+0000","last_undegraded":"2023-10-18T19:45:16.151979+0000","last_fullsized":"2023-10-18T19:45:16.151979+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T05:56:19.134313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.151874+0000","last_change":"2023-10-18T19:45:15.156688+0000","last_active":"2023-10-18T19:45:16.151874+0000","last_peered":"2023-10-18T19:45:16.151874+0000","last_clean":"2023-10-18T19:45:16.151874+0000","last_became_active":"2023-10-18T19:45:15.156133+0000","last_became_peered":"2023-10-18T19:45:15.156133+0000","last_unstale":"2023-10-18T19:45:16.151874+0000","last_undegraded":"2023-10-18T19:45:16.151874+0000","last_fullsized":"2023-10-18T19:45:16.151874+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T22:47:03.704416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.208234+0000","last_change":"2023-10-18T19:45:15.157964+0000","last_active":"2023-10-18T19:45:16.208234+0000","last_peered":"2023-10-18T19:45:16.208234+0000","last_clean":"2023-10-18T19:45:16.208234+0000","last_became_active":"2023-10-18T19:45:15.157781+0000","last_became_peered":"2023-10-18T19:45:15.157781+0000","last_unstale":"2023-10-18T19:45:16.208234+0000","last_undegraded":"2023-10-18T19:45:16.208234+0000","last_fullsized":"2023-10-18T19:45:16.208234+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T22:55:01.004744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.151812+0000","last_change":"2023-10-18T19:45:15.159649+0000","last_active":"2023-10-18T19:45:16.151812+0000","last_peered":"2023-10-18T19:45:16.151812+0000","last_clean":"2023-10-18T19:45:16.151812+0000","last_became_active":"2023-10-18T19:45:15.156708+0000","last_became_peered":"2023-10-18T19:45:15.156708+0000","last_unstale":"2023-10-18T19:45:16.151812+0000","last_undegraded":"2023-10-18T19:45:16.151812+0000","last_fullsized":"2023-10-18T19:45:16.151812+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T00:07:59.629457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.152004+0000","last_change":"2023-10-18T19:45:15.156233+0000","last_active":"2023-10-18T19:45:16.152004+0000","last_peered":"2023-10-18T19:45:16.152004+0000","last_clean":"2023-10-18T19:45:16.152004+0000","last_became_active":"2023-10-18T19:45:15.155948+0000","last_became_peered":"2023-10-18T19:45:15.155948+0000","last_unstale":"2023-10-18T19:45:16.152004+0000","last_undegraded":"2023-10-18T19:45:16.152004+0000","last_fullsized":"2023-10-18T19:45:16.152004+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T02:58:22.244905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.208225+0000","last_change":"2023-10-18T19:45:15.156256+0000","last_active":"2023-10-18T19:45:16.208225+0000","last_peered":"2023-10-18T19:45:16.208225+0000","last_clean":"2023-10-18T19:45:16.208225+0000","last_became_active":"2023-10-18T19:45:15.155937+0000","last_became_peered":"2023-10-18T19:45:15.155937+0000","last_unstale":"2023-10-18T19:45:16.208225+0000","last_undegraded":"2023-10-18T19:45:16.208225+0000","last_fullsized":"2023-10-18T19:45:16.208225+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T02:58:22.244905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.297799+0000","last_change":"2023-10-18T19:45:15.158096+0000","last_active":"2023-10-18T19:45:16.297799+0000","last_peered":"2023-10-18T19:45:16.297799+0000","last_clean":"2023-10-18T19:45:16.297799+0000","last_became_active":"2023-10-18T19:45:15.157782+0000","last_became_peered":"2023-10-18T19:45:15.157782+0000","last_unstale":"2023-10-18T19:45:16.297799+0000","last_undegraded":"2023-10-18T19:45:16.297799+0000","last_fullsized":"2023-10-18T19:45:16.297799+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T02:55:48.965008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.151899+0000","last_change":"2023-10-18T19:45:15.158190+0000","last_active":"2023-10-18T19:45:16.151899+0000","last_peered":"2023-10-18T19:45:16.151899+0000","last_clean":"2023-10-18T19:45:16.151899+0000","last_became_active":"2023-10-18T19:45:15.157964+0000","last_became_peered":"2023-10-18T19:45:15.157964+0000","last_unstale":"2023-10-18T19:45:16.151899+0000","last_undegraded":"2023-10-18T19:45:16.151899+0000","last_fullsized":"2023-10-18T19:45:16.151899+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T07:32:10.913317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27036,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830564,"statfs":{"total":107374182400,"available":107346497536,"internally_reserved":0,"allocated":86016,"data_stored":21469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.218}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-18T19:45:18.617 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-18T19:45:18.933 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:18.934 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2023-10-18T19:45:18.946 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-18T19:45:17.144824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82292,"kb_used_data":1436,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490508,"statfs":{"total":322122547200,"available":322038280192,"internally_reserved":0,"allocated":1470464,"data_stored":1255061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.152089+0000","last_change":"2023-10-18T19:45:15.157261+0000","last_active":"2023-10-18T19:45:16.152089+0000","last_peered":"2023-10-18T19:45:16.152089+0000","last_clean":"2023-10-18T19:45:16.152089+0000","last_became_active":"2023-10-18T19:45:15.157022+0000","last_became_peered":"2023-10-18T19:45:15.157022+0000","last_unstale":"2023-10-18T19:45:16.152089+0000","last_undegraded":"2023-10-18T19:45:16.152089+0000","last_fullsized":"2023-10-18T19:45:16.152089+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T20:51:30.705144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.151979+0000","last_change":"2023-10-18T19:45:15.156710+0000","last_active":"2023-10-18T19:45:16.151979+0000","last_peered":"2023-10-18T19:45:16.151979+0000","last_clean":"2023-10-18T19:45:16.151979+0000","last_became_active":"2023-10-18T19:45:15.156119+0000","last_became_peered":"2023-10-18T19:45:15.156119+0000","last_unstale":"2023-10-18T19:45:16.151979+0000","last_undegraded":"2023-10-18T19:45:16.151979+0000","last_fullsized":"2023-10-18T19:45:16.151979+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T05:56:19.134313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.151874+0000","last_change":"2023-10-18T19:45:15.156688+0000","last_active":"2023-10-18T19:45:16.151874+0000","last_peered":"2023-10-18T19:45:16.151874+0000","last_clean":"2023-10-18T19:45:16.151874+0000","last_became_active":"2023-10-18T19:45:15.156133+0000","last_became_peered":"2023-10-18T19:45:15.156133+0000","last_unstale":"2023-10-18T19:45:16.151874+0000","last_undegraded":"2023-10-18T19:45:16.151874+0000","last_fullsized":"2023-10-18T19:45:16.151874+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T22:47:03.704416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.208234+0000","last_change":"2023-10-18T19:45:15.157964+0000","last_active":"2023-10-18T19:45:16.208234+0000","last_peered":"2023-10-18T19:45:16.208234+0000","last_clean":"2023-10-18T19:45:16.208234+0000","last_became_active":"2023-10-18T19:45:15.157781+0000","last_became_peered":"2023-10-18T19:45:15.157781+0000","last_unstale":"2023-10-18T19:45:16.208234+0000","last_undegraded":"2023-10-18T19:45:16.208234+0000","last_fullsized":"2023-10-18T19:45:16.208234+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T22:55:01.004744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.151812+0000","last_change":"2023-10-18T19:45:15.159649+0000","last_active":"2023-10-18T19:45:16.151812+0000","last_peered":"2023-10-18T19:45:16.151812+0000","last_clean":"2023-10-18T19:45:16.151812+0000","last_became_active":"2023-10-18T19:45:15.156708+0000","last_became_peered":"2023-10-18T19:45:15.156708+0000","last_unstale":"2023-10-18T19:45:16.151812+0000","last_undegraded":"2023-10-18T19:45:16.151812+0000","last_fullsized":"2023-10-18T19:45:16.151812+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T00:07:59.629457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.152004+0000","last_change":"2023-10-18T19:45:15.156233+0000","last_active":"2023-10-18T19:45:16.152004+0000","last_peered":"2023-10-18T19:45:16.152004+0000","last_clean":"2023-10-18T19:45:16.152004+0000","last_became_active":"2023-10-18T19:45:15.155948+0000","last_became_peered":"2023-10-18T19:45:15.155948+0000","last_unstale":"2023-10-18T19:45:16.152004+0000","last_undegraded":"2023-10-18T19:45:16.152004+0000","last_fullsized":"2023-10-18T19:45:16.152004+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T02:58:22.244905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.208225+0000","last_change":"2023-10-18T19:45:15.156256+0000","last_active":"2023-10-18T19:45:16.208225+0000","last_peered":"2023-10-18T19:45:16.208225+0000","last_clean":"2023-10-18T19:45:16.208225+0000","last_became_active":"2023-10-18T19:45:15.155937+0000","last_became_peered":"2023-10-18T19:45:15.155937+0000","last_unstale":"2023-10-18T19:45:16.208225+0000","last_undegraded":"2023-10-18T19:45:16.208225+0000","last_fullsized":"2023-10-18T19:45:16.208225+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T02:58:22.244905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.297799+0000","last_change":"2023-10-18T19:45:15.158096+0000","last_active":"2023-10-18T19:45:16.297799+0000","last_peered":"2023-10-18T19:45:16.297799+0000","last_clean":"2023-10-18T19:45:16.297799+0000","last_became_active":"2023-10-18T19:45:15.157782+0000","last_became_peered":"2023-10-18T19:45:15.157782+0000","last_unstale":"2023-10-18T19:45:16.297799+0000","last_undegraded":"2023-10-18T19:45:16.297799+0000","last_fullsized":"2023-10-18T19:45:16.297799+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T02:55:48.965008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:16.151899+0000","last_change":"2023-10-18T19:45:15.158190+0000","last_active":"2023-10-18T19:45:16.151899+0000","last_peered":"2023-10-18T19:45:16.151899+0000","last_clean":"2023-10-18T19:45:16.151899+0000","last_became_active":"2023-10-18T19:45:15.157964+0000","last_became_peered":"2023-10-18T19:45:15.157964+0000","last_unstale":"2023-10-18T19:45:16.151899+0000","last_undegraded":"2023-10-18T19:45:16.151899+0000","last_fullsized":"2023-10-18T19:45:16.151899+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:14.145641+0000","last_clean_scrub_stamp":"2023-10-18T19:45:14.145641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T07:32:10.913317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27036,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830564,"statfs":{"total":107374182400,"available":107346497536,"internally_reserved":0,"allocated":86016,"data_stored":21469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.218}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-18T19:45:18.947 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-10-18T19:45:18.947 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-10-18T19:45:18.948 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-10-18T19:45:18.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-10-18T19:45:19.307 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:19.307 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-18T19:45:19.321 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-10-18T19:45:19.321 INFO:teuthology.run_tasks:Running task ceph... 2023-10-18T19:45:19.331 INFO:tasks.ceph:Creating ceph cluster c2... 2023-10-18T19:45:19.331 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-18T19:45:19.332 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650', 'branch': 'wip-yuri-testing-2023-10-18-0812-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7431650', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-18-0812-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri-testing-2023-10-18-0812-reef', 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1349, 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-yuri-testing-2023-10-18-0812-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'targets': {'smithi082.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvyj4AREUB5UtfXpWW0uvjhICdkbhKvR7vk+X63lvYbeR9OxwLhNt/cWMOwhoABu8Uzx6tjPXC/XE/KcEEbuYc=', 'smithi156.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEoPyxVfTkY/yghFEHuNDraaBPv1OdiYKJ4EOPuHZdPIGA5KTcBjR8Q6W0QaX3LPal9vMY0E9N+bsBvCnjgETOg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '07c1e9af8fb194e5ad8a71499ead549c30345076', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-18_19:20:22', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-18T19:45:19.332 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-18T19:45:19.335 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-18T19:45:19.357 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/run/ceph 2023-10-18T19:45:19.393 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2023-10-18T19:45:19.430 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:19.430 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2023-10-18T19:45:19.490 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-18T19:45:19.490 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2023-10-18T19:45:19.551 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-18T19:45:19.551 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T19:45:19.551 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 397478 Links: 1 2023-10-18T19:45:19.551 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T19:45:19.552 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T19:45:19.552 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-10-18 19:44:26.598539094 +0000 2023-10-18T19:45:19.552 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-10-18 19:40:14.181878362 +0000 2023-10-18T19:45:19.552 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-10-18 19:40:14.181878362 +0000 2023-10-18T19:45:19.552 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-10-18T19:45:19.553 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-18T19:45:19.618 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-10-18T19:45:19.619 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-10-18T19:45:19.619 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.4169e-05 s, 5.4 MB/s 2023-10-18T19:45:19.620 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-18T19:45:19.681 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2023-10-18T19:45:19.740 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-18T19:45:19.741 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T19:45:19.741 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 396750 Links: 1 2023-10-18T19:45:19.741 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T19:45:19.741 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T19:45:19.742 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-10-18 19:44:26.598539094 +0000 2023-10-18T19:45:19.742 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-10-18 19:40:14.698880508 +0000 2023-10-18T19:45:19.742 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-10-18 19:40:14.698880508 +0000 2023-10-18T19:45:19.742 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-10-18T19:45:19.743 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-18T19:45:19.807 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-10-18T19:45:19.808 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-10-18T19:45:19.808 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 8.4018e-05 s, 6.1 MB/s 2023-10-18T19:45:19.809 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-18T19:45:19.869 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2023-10-18T19:45:19.927 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-18T19:45:19.927 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T19:45:19.927 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 396819 Links: 1 2023-10-18T19:45:19.927 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T19:45:19.928 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T19:45:19.928 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-10-18 19:44:26.598539094 +0000 2023-10-18T19:45:19.928 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-10-18 19:40:15.228882709 +0000 2023-10-18T19:45:19.928 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-10-18 19:40:15.228882709 +0000 2023-10-18T19:45:19.928 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-10-18T19:45:19.929 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-18T19:45:19.992 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-10-18T19:45:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-10-18T19:45:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 8.7101e-05 s, 5.9 MB/s 2023-10-18T19:45:19.994 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-18T19:45:20.053 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2023-10-18T19:45:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-18T19:45:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T19:45:20.112 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 393693 Links: 1 2023-10-18T19:45:20.112 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T19:45:20.112 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T19:45:20.113 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-10-18 19:44:26.598539094 +0000 2023-10-18T19:45:20.113 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-10-18 19:40:15.742884843 +0000 2023-10-18T19:45:20.113 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-10-18 19:40:15.742884843 +0000 2023-10-18T19:45:20.113 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-10-18T19:45:20.113 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-18T19:45:20.178 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-10-18T19:45:20.178 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-10-18T19:45:20.178 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.081e-05 s, 5.6 MB/s 2023-10-18T19:45:20.179 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-18T19:45:20.240 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T19:45:20.240 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-18T19:45:20.240 INFO:tasks.ceph:Generating config... 2023-10-18T19:45:20.242 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-18T19:45:20.242 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-10-18T19:45:20.242 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-18T19:45:20.243 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-18T19:45:20.243 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-18T19:45:20.243 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-18T19:45:20.243 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-18T19:45:20.244 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-18T19:45:20.244 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-10-18T19:45:20.244 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-18T19:45:20.244 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-18T19:45:20.245 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-18T19:45:20.245 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-18T19:45:20.245 INFO:tasks.ceph:[client] setuser = ceph 2023-10-18T19:45:20.245 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-10-18T19:45:20.246 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-18T19:45:20.246 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-18T19:45:20.246 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-18T19:45:20.246 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-18T19:45:20.247 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-18T19:45:20.247 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-18T19:45:20.247 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-18T19:45:20.247 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-18T19:45:20.248 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-18T19:45:20.248 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-18T19:45:20.248 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-18T19:45:20.248 INFO:tasks.ceph:Setting up c2.mon.a... 2023-10-18T19:45:20.249 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-10-18T19:45:20.330 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/c2.keyring 2023-10-18T19:45:20.334 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-10-18T19:45:20.379 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-18T19:45:20.406 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.156')] 2023-10-18T19:45:20.407 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.156'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-18T19:45:20.408 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:20.408 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-18T19:45:20.465 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.156 --print /home/ubuntu/cephtest/c2.monmap 2023-10-18T19:45:20.547 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:45:20.547 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:45:20.548 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-10-18T19:45:20.548 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: generated fsid 4e3dcbf9-06fb-4969-a883-12e13c93f569 2023-10-18T19:45:20.548 INFO:teuthology.orchestra.run.smithi156.stdout:setting min_mon_release = pacific 2023-10-18T19:45:20.549 INFO:teuthology.orchestra.run.smithi156.stdout:epoch 0 2023-10-18T19:45:20.549 INFO:teuthology.orchestra.run.smithi156.stdout:fsid 4e3dcbf9-06fb-4969-a883-12e13c93f569 2023-10-18T19:45:20.549 INFO:teuthology.orchestra.run.smithi156.stdout:last_changed 2023-10-18T19:45:20.546758+0000 2023-10-18T19:45:20.549 INFO:teuthology.orchestra.run.smithi156.stdout:created 2023-10-18T19:45:20.546758+0000 2023-10-18T19:45:20.549 INFO:teuthology.orchestra.run.smithi156.stdout:min_mon_release 16 (pacific) 2023-10-18T19:45:20.550 INFO:teuthology.orchestra.run.smithi156.stdout:election_strategy: 1 2023-10-18T19:45:20.550 INFO:teuthology.orchestra.run.smithi156.stdout:0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2023-10-18T19:45:20.550 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-10-18T19:45:20.551 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-18T19:45:20.610 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 4e3dcbf9-06fb-4969-a883-12e13c93f569... 2023-10-18T19:45:20.611 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-18T19:45:20.613 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-18T19:45:20.664 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2023-10-18T19:45:20.665 INFO:teuthology.orchestra.run.smithi082.stdout: chdir = "" 2023-10-18T19:45:20.665 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-18T19:45:20.665 INFO:teuthology.orchestra.run.smithi082.stdout: auth supported = cephx 2023-10-18T19:45:20.665 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.666 INFO:teuthology.orchestra.run.smithi082.stdout: filestore xattr use omap = true 2023-10-18T19:45:20.666 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.666 INFO:teuthology.orchestra.run.smithi082.stdout: mon clock drift allowed = 1.000 2023-10-18T19:45:20.666 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.666 INFO:teuthology.orchestra.run.smithi082.stdout: osd crush chooseleaf type = 0 2023-10-18T19:45:20.667 INFO:teuthology.orchestra.run.smithi082.stdout: auth debug = true 2023-10-18T19:45:20.667 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.667 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on old message = true 2023-10-18T19:45:20.667 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on bug = true 2023-10-18T19:45:20.667 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.667 INFO:teuthology.orchestra.run.smithi082.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-18T19:45:20.668 INFO:teuthology.orchestra.run.smithi082.stdout: mon pg warn max object skew = 0 2023-10-18T19:45:20.668 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.668 INFO:teuthology.orchestra.run.smithi082.stdout: # disable pg_autoscaler by default for new pools 2023-10-18T19:45:20.668 INFO:teuthology.orchestra.run.smithi082.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-18T19:45:20.668 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.669 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default size = 2 2023-10-18T19:45:20.669 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.669 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow primary affinity = true 2023-10-18T19:45:20.669 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow pg remap = true 2023-10-18T19:45:20.669 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on legacy crush tunables = false 2023-10-18T19:45:20.669 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on crush straw calc version zero = false 2023-10-18T19:45:20.670 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on no sortbitwise = false 2023-10-18T19:45:20.670 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on osd down out interval zero = false 2023-10-18T19:45:20.670 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on too few osds = false 2023-10-18T19:45:20.670 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-18T19:45:20.670 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-18T19:45:20.671 INFO:teuthology.orchestra.run.smithi082.stdout: mon_allow_pool_size_one = true 2023-10-18T19:45:20.671 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.671 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-18T19:45:20.671 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.671 INFO:teuthology.orchestra.run.smithi082.stdout: osd default data pool replay window = 5 2023-10-18T19:45:20.671 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.672 INFO:teuthology.orchestra.run.smithi082.stdout: mon allow pool delete = true 2023-10-18T19:45:20.672 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.672 INFO:teuthology.orchestra.run.smithi082.stdout: mon cluster log file level = debug 2023-10-18T19:45:20.672 INFO:teuthology.orchestra.run.smithi082.stdout: debug asserts on shutdown = true 2023-10-18T19:45:20.672 INFO:teuthology.orchestra.run.smithi082.stdout: mon health detail to clog = false 2023-10-18T19:45:20.673 INFO:teuthology.orchestra.run.smithi082.stdout: mon host = 172.21.15.156 2023-10-18T19:45:20.673 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = 4e3dcbf9-06fb-4969-a883-12e13c93f569 2023-10-18T19:45:20.673 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.673 INFO:teuthology.orchestra.run.smithi082.stdout:[osd] 2023-10-18T19:45:20.673 INFO:teuthology.orchestra.run.smithi082.stdout: osd journal size = 100 2023-10-18T19:45:20.674 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.674 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub load threshold = 5.0 2023-10-18T19:45:20.674 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub max interval = 600 2023-10-18T19:45:20.674 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock profile = high_recovery_ops 2023-10-18T19:45:20.674 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.675 INFO:teuthology.orchestra.run.smithi082.stdout: osd recover clone overlap = true 2023-10-18T19:45:20.675 INFO:teuthology.orchestra.run.smithi082.stdout: osd recovery max chunk = 1048576 2023-10-18T19:45:20.675 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.675 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug shutdown = true 2023-10-18T19:45:20.675 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug op order = true 2023-10-18T19:45:20.676 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify stray on activate = true 2023-10-18T19:45:20.676 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.676 INFO:teuthology.orchestra.run.smithi082.stdout: osd open classes on start = true 2023-10-18T19:45:20.676 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug pg log writeout = true 2023-10-18T19:45:20.676 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.677 INFO:teuthology.orchestra.run.smithi082.stdout: osd deep scrub update digest min age = 30 2023-10-18T19:45:20.677 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.677 INFO:teuthology.orchestra.run.smithi082.stdout: osd map max advance = 10 2023-10-18T19:45:20.677 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.677 INFO:teuthology.orchestra.run.smithi082.stdout: journal zero on create = true 2023-10-18T19:45:20.678 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.678 INFO:teuthology.orchestra.run.smithi082.stdout: filestore ondisk finisher threads = 3 2023-10-18T19:45:20.678 INFO:teuthology.orchestra.run.smithi082.stdout: filestore apply finisher threads = 3 2023-10-18T19:45:20.678 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.678 INFO:teuthology.orchestra.run.smithi082.stdout: bdev debug aio = true 2023-10-18T19:45:20.678 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug misdirected ops = true 2023-10-18T19:45:20.679 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-10-18T19:45:20.679 INFO:teuthology.orchestra.run.smithi082.stdout: debug osd = 20 2023-10-18T19:45:20.679 INFO:teuthology.orchestra.run.smithi082.stdout: rocksdb delete range threshold = 0 2023-10-18T19:45:20.679 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.679 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr] 2023-10-18T19:45:20.680 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-10-18T19:45:20.680 INFO:teuthology.orchestra.run.smithi082.stdout: debug mgr = 20 2023-10-18T19:45:20.680 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2023-10-18T19:45:20.680 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2023-10-18T19:45:20.680 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2023-10-18T19:45:20.680 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2023-10-18T19:45:20.681 INFO:teuthology.orchestra.run.smithi082.stdout: mgr/telemetry/nag = false 2023-10-18T19:45:20.681 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.681 INFO:teuthology.orchestra.run.smithi082.stdout:[mon] 2023-10-18T19:45:20.681 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-10-18T19:45:20.681 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2023-10-18T19:45:20.681 INFO:teuthology.orchestra.run.smithi082.stdout: debug paxos = 20 2023-10-18T19:45:20.682 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2023-10-18T19:45:20.682 INFO:teuthology.orchestra.run.smithi082.stdout: mon data avail warn = 5 2023-10-18T19:45:20.682 INFO:teuthology.orchestra.run.smithi082.stdout: mon mgr mkfs grace = 240 2023-10-18T19:45:20.682 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2023-10-18T19:45:20.682 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd reporter subtree level = osd 2023-10-18T19:45:20.683 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd prime pg temp = true 2023-10-18T19:45:20.683 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2023-10-18T19:45:20.683 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.683 INFO:teuthology.orchestra.run.smithi082.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-18T19:45:20.683 INFO:teuthology.orchestra.run.smithi082.stdout: auth mon ticket ttl = 660 # 11m 2023-10-18T19:45:20.683 INFO:teuthology.orchestra.run.smithi082.stdout: auth service ticket ttl = 240 # 4m 2023-10-18T19:45:20.684 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.684 INFO:teuthology.orchestra.run.smithi082.stdout: # don't complain about insecure global_id in the test suite 2023-10-18T19:45:20.684 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-18T19:45:20.684 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-18T19:45:20.684 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.685 INFO:teuthology.orchestra.run.smithi082.stdout: # 1m isn't quite enough 2023-10-18T19:45:20.685 INFO:teuthology.orchestra.run.smithi082.stdout: mon_down_mkfs_grace = 2m 2023-10-18T19:45:20.685 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.685 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_filestore_osds = false 2023-10-18T19:45:20.685 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:45:20.686 INFO:teuthology.orchestra.run.smithi082.stdout:[client] 2023-10-18T19:45:20.686 INFO:teuthology.orchestra.run.smithi082.stdout: rgw cache enabled = true 2023-10-18T19:45:20.686 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable ops log = true 2023-10-18T19:45:20.686 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable usage log = true 2023-10-18T19:45:20.686 INFO:teuthology.orchestra.run.smithi082.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-18T19:45:20.687 INFO:teuthology.orchestra.run.smithi082.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-18T19:45:20.687 INFO:teuthology.orchestra.run.smithi082.stdout: debug rgw = 20 2023-10-18T19:45:20.687 INFO:teuthology.orchestra.run.smithi082.stdout: debug rgw sync = 20 2023-10-18T19:45:20.687 INFO:teuthology.orchestra.run.smithi082.stdout: rgw crypt require ssl = False 2023-10-18T19:45:20.687 INFO:teuthology.orchestra.run.smithi082.stdout: rgw crypt s3 kms backend = testing 2023-10-18T19:45:20.688 INFO:teuthology.orchestra.run.smithi082.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-18T19:45:20.688 INFO:teuthology.orchestra.run.smithi082.stdout: rgw curl low speed time = 300 2023-10-18T19:45:20.688 INFO:teuthology.orchestra.run.smithi082.stdout: rgw data log num shards = 4 2023-10-18T19:45:20.688 INFO:teuthology.orchestra.run.smithi082.stdout: rgw md log max shards = 4 2023-10-18T19:45:20.688 INFO:teuthology.orchestra.run.smithi082.stdout: rgw sync data inject err probability = 0.1 2023-10-18T19:45:20.689 INFO:teuthology.orchestra.run.smithi082.stdout: rgw sync log trim interval = 0 2023-10-18T19:45:20.689 INFO:teuthology.orchestra.run.smithi082.stdout: rgw sync meta inject err probability = 0.1 2023-10-18T19:45:20.689 INFO:teuthology.orchestra.run.smithi082.stdout: rgw sync obj etag verify = True 2023-10-18T19:45:20.689 INFO:teuthology.orchestra.run.smithi082.stdout: setgroup = ceph 2023-10-18T19:45:20.689 INFO:teuthology.orchestra.run.smithi082.stdout: setuser = ceph 2023-10-18T19:45:20.689 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.a] 2023-10-18T19:45:20.690 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2023-10-18T19:45:20.690 INFO:teuthology.orchestra.run.smithi082.stdout: rgw data notify interval msec = 0 2023-10-18T19:45:20.690 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2023-10-18T19:45:20.690 INFO:teuthology.orchestra.run.smithi082.stdout: osd_max_omap_entries_per_request = 10 2023-10-18T19:45:20.690 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2023-10-18T19:45:20.691 INFO:teuthology.orchestra.run.smithi082.stdout: osd_max_omap_entries_per_request = 1000 2023-10-18T19:45:20.691 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2023-10-18T19:45:20.691 INFO:teuthology.orchestra.run.smithi082.stdout: osd_max_omap_entries_per_request = 10000 2023-10-18T19:45:20.695 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2023-10-18T19:45:20.695 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2023-10-18T19:45:20.695 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-18T19:45:20.695 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2023-10-18T19:45:20.695 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.696 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2023-10-18T19:45:20.696 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.696 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2023-10-18T19:45:20.696 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.696 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2023-10-18T19:45:20.697 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2023-10-18T19:45:20.697 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.697 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2023-10-18T19:45:20.697 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2023-10-18T19:45:20.697 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.698 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-18T19:45:20.698 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2023-10-18T19:45:20.698 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.698 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2023-10-18T19:45:20.698 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-18T19:45:20.698 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.699 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2023-10-18T19:45:20.699 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.699 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2023-10-18T19:45:20.699 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2023-10-18T19:45:20.699 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2023-10-18T19:45:20.699 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2023-10-18T19:45:20.700 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2023-10-18T19:45:20.700 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2023-10-18T19:45:20.700 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2023-10-18T19:45:20.700 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-18T19:45:20.700 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-18T19:45:20.701 INFO:teuthology.orchestra.run.smithi156.stdout: mon_allow_pool_size_one = true 2023-10-18T19:45:20.701 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.701 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-18T19:45:20.701 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.701 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2023-10-18T19:45:20.701 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.702 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2023-10-18T19:45:20.702 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.702 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2023-10-18T19:45:20.702 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2023-10-18T19:45:20.703 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2023-10-18T19:45:20.703 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = 172.21.15.156 2023-10-18T19:45:20.703 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = 4e3dcbf9-06fb-4969-a883-12e13c93f569 2023-10-18T19:45:20.703 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.703 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2023-10-18T19:45:20.703 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2023-10-18T19:45:20.704 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.704 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2023-10-18T19:45:20.704 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2023-10-18T19:45:20.704 INFO:teuthology.orchestra.run.smithi156.stdout: osd mclock profile = high_recovery_ops 2023-10-18T19:45:20.704 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.705 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2023-10-18T19:45:20.705 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2023-10-18T19:45:20.705 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.705 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2023-10-18T19:45:20.705 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2023-10-18T19:45:20.705 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2023-10-18T19:45:20.706 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.706 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2023-10-18T19:45:20.706 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2023-10-18T19:45:20.706 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.706 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2023-10-18T19:45:20.707 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.707 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2023-10-18T19:45:20.707 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.707 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2023-10-18T19:45:20.707 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.708 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2023-10-18T19:45:20.708 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2023-10-18T19:45:20.708 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.708 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2023-10-18T19:45:20.708 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2023-10-18T19:45:20.709 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-10-18T19:45:20.709 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2023-10-18T19:45:20.709 INFO:teuthology.orchestra.run.smithi156.stdout: rocksdb delete range threshold = 0 2023-10-18T19:45:20.709 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.709 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2023-10-18T19:45:20.709 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-10-18T19:45:20.710 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2023-10-18T19:45:20.710 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2023-10-18T19:45:20.710 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2023-10-18T19:45:20.710 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2023-10-18T19:45:20.710 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2023-10-18T19:45:20.710 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2023-10-18T19:45:20.711 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.711 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2023-10-18T19:45:20.711 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-10-18T19:45:20.711 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2023-10-18T19:45:20.711 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2023-10-18T19:45:20.711 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2023-10-18T19:45:20.712 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2023-10-18T19:45:20.712 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2023-10-18T19:45:20.712 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2023-10-18T19:45:20.712 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2023-10-18T19:45:20.712 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2023-10-18T19:45:20.713 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2023-10-18T19:45:20.713 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.713 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-18T19:45:20.713 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2023-10-18T19:45:20.713 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2023-10-18T19:45:20.714 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.714 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2023-10-18T19:45:20.714 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-18T19:45:20.714 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-18T19:45:20.715 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.715 INFO:teuthology.orchestra.run.smithi156.stdout: # 1m isn't quite enough 2023-10-18T19:45:20.715 INFO:teuthology.orchestra.run.smithi156.stdout: mon_down_mkfs_grace = 2m 2023-10-18T19:45:20.715 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.715 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_filestore_osds = false 2023-10-18T19:45:20.715 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:20.716 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2023-10-18T19:45:20.716 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2023-10-18T19:45:20.716 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2023-10-18T19:45:20.716 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2023-10-18T19:45:20.716 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-18T19:45:20.717 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-18T19:45:20.717 INFO:teuthology.orchestra.run.smithi156.stdout: debug rgw = 20 2023-10-18T19:45:20.717 INFO:teuthology.orchestra.run.smithi156.stdout: debug rgw sync = 20 2023-10-18T19:45:20.717 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt require ssl = False 2023-10-18T19:45:20.717 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt s3 kms backend = testing 2023-10-18T19:45:20.717 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-18T19:45:20.718 INFO:teuthology.orchestra.run.smithi156.stdout: rgw curl low speed time = 300 2023-10-18T19:45:20.718 INFO:teuthology.orchestra.run.smithi156.stdout: rgw data log num shards = 4 2023-10-18T19:45:20.718 INFO:teuthology.orchestra.run.smithi156.stdout: rgw md log max shards = 4 2023-10-18T19:45:20.718 INFO:teuthology.orchestra.run.smithi156.stdout: rgw sync data inject err probability = 0.1 2023-10-18T19:45:20.718 INFO:teuthology.orchestra.run.smithi156.stdout: rgw sync log trim interval = 0 2023-10-18T19:45:20.719 INFO:teuthology.orchestra.run.smithi156.stdout: rgw sync meta inject err probability = 0.1 2023-10-18T19:45:20.719 INFO:teuthology.orchestra.run.smithi156.stdout: rgw sync obj etag verify = True 2023-10-18T19:45:20.719 INFO:teuthology.orchestra.run.smithi156.stdout: setgroup = ceph 2023-10-18T19:45:20.719 INFO:teuthology.orchestra.run.smithi156.stdout: setuser = ceph 2023-10-18T19:45:20.719 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2023-10-18T19:45:20.719 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2023-10-18T19:45:20.720 INFO:teuthology.orchestra.run.smithi156.stdout: rgw data notify interval msec = 0 2023-10-18T19:45:20.720 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2023-10-18T19:45:20.720 INFO:teuthology.orchestra.run.smithi156.stdout: osd_max_omap_entries_per_request = 10 2023-10-18T19:45:20.720 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2023-10-18T19:45:20.720 INFO:teuthology.orchestra.run.smithi156.stdout: osd_max_omap_entries_per_request = 1000 2023-10-18T19:45:20.721 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2023-10-18T19:45:20.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd_max_omap_entries_per_request = 10000 2023-10-18T19:45:20.722 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-10-18T19:45:20.722 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-10-18T19:45:20.819 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-18T19:45:20.820 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:20.820 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-10-18T19:45:20.839 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:20.840 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-10-18T19:45:20.897 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T19:45:20.898 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:20.898 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/c2.keyring 2023-10-18T19:45:20.898 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-18T19:45:20.942 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-10-18T19:45:20.942 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-18T19:45:21.003 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T19:45:21.004 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:21.004 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/c2.keyring 2023-10-18T19:45:21.005 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-18T19:45:21.042 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:21.042 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-18T19:45:21.100 INFO:tasks.ceph:Setting up mon nodes... 2023-10-18T19:45:21.100 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-18T19:45:21.101 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-10-18T19:45:21.193 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-10-18T19:45:21.195 INFO:tasks.ceph:Setting up mds nodes... 2023-10-18T19:45:21.196 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-18T19:45:21.196 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-10-18T19:45:21.240 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/c2.client.0.keyring 2023-10-18T19:45:21.253 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-10-18T19:45:21.297 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/c2.client.1.keyring 2023-10-18T19:45:21.309 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-18T19:45:21.310 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-18T19:45:21.310 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-10-18T19:45:21.379 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T19:45:21.379 INFO:tasks.ceph:role: c2.osd.0 2023-10-18T19:45:21.379 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T19:45:21.380 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-18T19:45:21.446 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T19:45:21.446 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T19:45:21.446 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T19:45:21.446 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-18T19:45:21.447 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T19:45:21.447 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2023-10-18T19:45:21.447 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T19:45:21.447 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T19:45:21.447 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T19:45:21.448 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T19:45:21.468 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2023-10-18T19:45:21.469 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2023-10-18T19:45:21.470 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-10-18T19:45:21.551 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-10-18T19:45:21.624 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-10-18T19:45:21.691 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T19:45:21.692 INFO:tasks.ceph:role: c2.osd.1 2023-10-18T19:45:21.692 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T19:45:21.693 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-18T19:45:21.758 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T19:45:21.759 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T19:45:21.759 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T19:45:21.759 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-18T19:45:21.759 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T19:45:21.759 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2023-10-18T19:45:21.760 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T19:45:21.761 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T19:45:21.761 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T19:45:21.761 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T19:45:21.781 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2023-10-18T19:45:21.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2023-10-18T19:45:21.783 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-10-18T19:45:21.862 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-10-18T19:45:21.932 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-10-18T19:45:22.000 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T19:45:22.000 INFO:tasks.ceph:role: c2.osd.2 2023-10-18T19:45:22.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T19:45:22.001 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-18T19:45:22.067 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T19:45:22.068 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T19:45:22.068 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T19:45:22.068 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-18T19:45:22.068 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T19:45:22.068 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2023-10-18T19:45:22.069 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T19:45:22.069 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-18T19:45:22.069 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T19:45:22.069 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T19:45:22.090 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2023-10-18T19:45:22.092 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2023-10-18T19:45:22.092 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-10-18T19:45:22.172 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-10-18T19:45:22.242 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-18T19:45:22.337 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:22.333+0000 7f82ce3c6640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-10-18T19:45:22.337 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:22.334+0000 7f82ce3c6640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-10-18T19:45:22.337 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:22.334+0000 7f82ce3c6640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-18T19:45:22.337 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:22.334+0000 7f82ce3c6640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-18T19:45:22.338 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:22.334+0000 7f82ce3c6640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-18T19:45:22.338 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:22.334+0000 7f82ce3c6640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-10-18T19:45:24.416 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-10-18T19:45:24.449 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-18T19:45:24.543 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:24.540+0000 7f8a1edb2640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-10-18T19:45:24.543 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:24.540+0000 7f8a1edb2640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-10-18T19:45:24.544 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:24.540+0000 7f8a1edb2640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-18T19:45:24.544 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:24.540+0000 7f8a1edb2640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-18T19:45:24.544 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:24.540+0000 7f8a1edb2640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-18T19:45:24.544 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:24.540+0000 7f8a1edb2640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-10-18T19:45:26.624 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-10-18T19:45:26.657 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-18T19:45:26.750 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:26.746+0000 7f240fae7640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-10-18T19:45:26.750 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:26.746+0000 7f240fae7640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-10-18T19:45:26.750 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:26.746+0000 7f240fae7640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-18T19:45:26.751 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:26.747+0000 7f240fae7640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-18T19:45:26.751 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:26.747+0000 7f240fae7640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-18T19:45:26.751 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:45:26.747+0000 7f240fae7640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-10-18T19:45:28.820 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-10-18T19:45:28.853 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-18T19:45:28.854 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:28.854 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-10-18T19:45:28.924 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:28.925 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-10-18T19:45:28.994 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:28.994 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-10-18T19:45:29.063 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:29.063 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-10-18T19:45:29.131 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:29.131 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-10-18T19:45:29.189 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:29.190 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-10-18T19:45:29.248 INFO:tasks.ceph:Adding keys to all mons... 2023-10-18T19:45:29.248 DEBUG:teuthology.orchestra.run.smithi156:> sudo tee -a /etc/ceph/c2.keyring 2023-10-18T19:45:29.314 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.x] 2023-10-18T19:45:29.315 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDRNTBl/6dzCxAAfjDoOdhdoGwVkSBL29Akig== 2023-10-18T19:45:29.315 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2023-10-18T19:45:29.315 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDSNTBlYpf+ExAATBb/BUwqRTmz79MJZTBUMg== 2023-10-18T19:45:29.316 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2023-10-18T19:45:29.316 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDUNTBltmpJIBAAlSF5b1D6hCLzfD3e4jiW6w== 2023-10-18T19:45:29.316 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2023-10-18T19:45:29.316 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDWNTBllSibLBAACZVlwrS1Lx1SxtLb+Lj8yw== 2023-10-18T19:45:29.316 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2023-10-18T19:45:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDRNTBlHL5GDhAAnwanc1YJAid3+21XzugK0Q== 2023-10-18T19:45:29.317 INFO:teuthology.orchestra.run.smithi156.stdout:[client.1] 2023-10-18T19:45:29.317 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDRNTBlKkCoERAAE0t1wSbSvW2GZLn52PZTFA== 2023-10-18T19:45:29.318 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-18T19:45:29.412 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T19:45:29.466 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T19:45:29.520 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T19:45:29.575 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-18T19:45:29.632 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-18T19:45:29.689 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-18T19:45:29.689 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-10-18T19:45:29.717 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-10-18T19:45:29.816 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-10-18T19:45:29.844 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-10-18T19:45:29.901 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-10-18T19:45:29.902 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-10-18T19:45:29.902 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-10-18T19:45:29.944 INFO:tasks.ceph.c2.mon.a:Started 2023-10-18T19:45:29.944 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-10-18T19:45:29.945 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-10-18T19:45:29.945 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-10-18T19:45:29.947 INFO:tasks.ceph.c2.mgr.x:Started 2023-10-18T19:45:29.947 INFO:tasks.ceph:Setting crush tunables to default 2023-10-18T19:45:29.948 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c2 osd crush tunables default 2023-10-18T19:45:30.153 INFO:teuthology.orchestra.run.smithi156.stderr:adjusted tunables profile to default 2023-10-18T19:45:30.164 INFO:tasks.ceph:check_enable_crimson: False 2023-10-18T19:45:30.165 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-10-18T19:45:30.165 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:30.165 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-10-18T19:45:30.192 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:30.193 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-10-18T19:45:30.260 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-10-18T19:45:30.261 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-10-18T19:45:30.328 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c2 osd new d6031b86-8bd0-4c3e-a5be-2e373dca4c49 0 2023-10-18T19:45:30.355 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:30.352+0000 7f839a9ed200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-18T19:45:30.541 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:30.539+0000 7f839a9ed200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-18T19:45:30.601 INFO:teuthology.orchestra.run.smithi156.stdout:0 2023-10-18T19:45:30.618 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c2 osd new 608ad7bb-a3e6-4575-83ca-1f23d7d7f99b 1 2023-10-18T19:45:30.831 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-10-18T19:45:30.844 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c2 osd new fc8c0211-200f-4ce8-9313-d338dfb4060f 2 2023-10-18T19:45:30.911 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:30.908+0000 7f839a9ed200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-18T19:45:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:2 2023-10-18T19:45:31.084 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-10-18T19:45:31.085 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-10-18T19:45:31.087 INFO:tasks.ceph.c2.osd.0:Started 2023-10-18T19:45:31.088 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-10-18T19:45:31.088 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-10-18T19:45:31.091 INFO:tasks.ceph.c2.osd.1:Started 2023-10-18T19:45:31.092 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-10-18T19:45:31.092 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-10-18T19:45:31.094 INFO:tasks.ceph.c2.osd.2:Started 2023-10-18T19:45:31.095 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-18T19:45:31.306 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:31.307 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":5,"fsid":"4e3dcbf9-06fb-4969-a883-12e13c93f569","created":"2023-10-18T19:45:30.048018+0000","modified":"2023-10-18T19:45:31.068707+0000","last_up_change":"0.000000","last_in_change":"2023-10-18T19:45:31.068707+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d6031b86-8bd0-4c3e-a5be-2e373dca4c49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"608ad7bb-a3e6-4575-83ca-1f23d7d7f99b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc8c0211-200f-4ce8-9313-d338dfb4060f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:31.319 INFO:tasks.ceph.ceph_manager.c2:[] 2023-10-18T19:45:31.320 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-18T19:45:31.466 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:31.463+0000 7f839a9ed200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-18T19:45:31.622 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-18T19:45:31.671 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:31.668+0000 7f839a9ed200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-18T19:45:31.693 INFO:tasks.ceph.c2.osd.1.smithi156.stderr:2023-10-18T19:45:31.690+0000 7fc1c2210640 -1 Falling back to public interface 2023-10-18T19:45:31.699 INFO:tasks.ceph.c2.osd.2.smithi156.stderr:2023-10-18T19:45:31.697+0000 7f9ac4794640 -1 Falling back to public interface 2023-10-18T19:45:31.719 INFO:tasks.ceph.c2.osd.0.smithi156.stderr:2023-10-18T19:45:31.717+0000 7f4d8829d640 -1 Falling back to public interface 2023-10-18T19:45:31.817 INFO:teuthology.misc.health.smithi156.stdout: 2023-10-18T19:45:31.817 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":5,"fsid":"4e3dcbf9-06fb-4969-a883-12e13c93f569","created":"2023-10-18T19:45:30.048018+0000","modified":"2023-10-18T19:45:31.068707+0000","last_up_change":"0.000000","last_in_change":"2023-10-18T19:45:31.068707+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d6031b86-8bd0-4c3e-a5be-2e373dca4c49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"608ad7bb-a3e6-4575-83ca-1f23d7d7f99b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc8c0211-200f-4ce8-9313-d338dfb4060f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:31.828 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-18T19:45:32.057 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:32.054+0000 7f839a9ed200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-18T19:45:32.727 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:32.724+0000 7f839a9ed200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-18T19:45:32.800 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:32.797+0000 7f839a9ed200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-18T19:45:32.935 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:32.933+0000 7f839a9ed200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-18T19:45:33.005 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:33.003+0000 7f839a9ed200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-18T19:45:33.191 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:33.188+0000 7f839a9ed200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-18T19:45:33.270 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:33.268+0000 7f839a9ed200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-18T19:45:33.349 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:33.346+0000 7f839a9ed200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-18T19:45:33.441 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:33.438+0000 7f839a9ed200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-18T19:45:33.625 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:33.622+0000 7f839a9ed200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-18T19:45:33.707 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:33.704+0000 7f839a9ed200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-18T19:45:33.999 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:33.996+0000 7f839a9ed200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-18T19:45:34.070 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:34.067+0000 7f839a9ed200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-18T19:45:34.419 INFO:tasks.ceph.c2.osd.1.smithi156.stderr:2023-10-18T19:45:34.416+0000 7fc1c2210640 -1 osd.1 0 log_to_monitors true 2023-10-18T19:45:34.465 INFO:tasks.ceph.c2.osd.2.smithi156.stderr:2023-10-18T19:45:34.461+0000 7f9ac4794640 -1 osd.2 0 log_to_monitors true 2023-10-18T19:45:34.468 INFO:tasks.ceph.c2.osd.0.smithi156.stderr:2023-10-18T19:45:34.465+0000 7f4d8829d640 -1 osd.0 0 log_to_monitors true 2023-10-18T19:45:34.608 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:34.606+0000 7f839a9ed200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-18T19:45:34.931 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:34.928+0000 7f839a9ed200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-18T19:45:35.077 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:35.073+0000 7f839a9ed200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-18T19:45:35.473 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:35.470+0000 7f839a9ed200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-18T19:45:35.670 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:35.667+0000 7f839a9ed200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-18T19:45:35.846 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:35.842+0000 7f839a9ed200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-18T19:45:36.126 INFO:tasks.ceph.c2.osd.0.smithi156.stderr:2023-10-18T19:45:36.123+0000 7f4d7c63e700 -1 osd.0 0 waiting for initial osdmap 2023-10-18T19:45:36.134 INFO:tasks.ceph.c2.osd.0.smithi156.stderr:2023-10-18T19:45:36.132+0000 7f4d74218700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T19:45:36.155 INFO:tasks.ceph.c2.osd.1.smithi156.stderr:2023-10-18T19:45:36.153+0000 7fc1b65b1700 -1 osd.1 0 waiting for initial osdmap 2023-10-18T19:45:36.173 INFO:tasks.ceph.c2.osd.1.smithi156.stderr:2023-10-18T19:45:36.170+0000 7fc1ae18b700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T19:45:36.197 INFO:tasks.ceph.c2.osd.2.smithi156.stderr:2023-10-18T19:45:36.194+0000 7f9ab8b35700 -1 osd.2 0 waiting for initial osdmap 2023-10-18T19:45:36.209 INFO:tasks.ceph.c2.osd.2.smithi156.stderr:2023-10-18T19:45:36.206+0000 7f9ab070f700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-18T19:45:36.219 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:36.216+0000 7f839a9ed200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-18T19:45:37.229 INFO:tasks.ceph.c2.mgr.x.smithi156.stderr:2023-10-18T19:45:37.226+0000 7f835be03700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-18T19:45:38.131 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-18T19:45:38.456 INFO:teuthology.misc.health.smithi156.stdout: 2023-10-18T19:45:38.457 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":9,"fsid":"4e3dcbf9-06fb-4969-a883-12e13c93f569","created":"2023-10-18T19:45:30.048018+0000","modified":"2023-10-18T19:45:38.229310+0000","last_up_change":"2023-10-18T19:45:37.126203+0000","last_in_change":"2023-10-18T19:45:31.068707+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d6031b86-8bd0-4c3e-a5be-2e373dca4c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":114361},{"type":"v1","addr":"172.21.15.156:6817","nonce":114361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":114361},{"type":"v1","addr":"172.21.15.156:6819","nonce":114361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":114361},{"type":"v1","addr":"172.21.15.156:6823","nonce":114361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":114361},{"type":"v1","addr":"172.21.15.156:6821","nonce":114361}]},"public_addr":"172.21.15.156:6817/114361","cluster_addr":"172.21.15.156:6819/114361","heartbeat_back_addr":"172.21.15.156:6823/114361","heartbeat_front_addr":"172.21.15.156:6821/114361","state":["exists","up"]},{"osd":1,"uuid":"608ad7bb-a3e6-4575-83ca-1f23d7d7f99b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":114351},{"type":"v1","addr":"172.21.15.156:6801","nonce":114351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":114351},{"type":"v1","addr":"172.21.15.156:6803","nonce":114351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":114351},{"type":"v1","addr":"172.21.15.156:6807","nonce":114351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":114351},{"type":"v1","addr":"172.21.15.156:6805","nonce":114351}]},"public_addr":"172.21.15.156:6801/114351","cluster_addr":"172.21.15.156:6803/114351","heartbeat_back_addr":"172.21.15.156:6807/114351","heartbeat_front_addr":"172.21.15.156:6805/114351","state":["exists","up"]},{"osd":2,"uuid":"fc8c0211-200f-4ce8-9313-d338dfb4060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":114352},{"type":"v1","addr":"172.21.15.156:6809","nonce":114352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":114352},{"type":"v1","addr":"172.21.15.156:6811","nonce":114352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":114352},{"type":"v1","addr":"172.21.15.156:6815","nonce":114352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":114352},{"type":"v1","addr":"172.21.15.156:6813","nonce":114352}]},"public_addr":"172.21.15.156:6809/114352","cluster_addr":"172.21.15.156:6811/114352","heartbeat_back_addr":"172.21.15.156:6815/114352","heartbeat_front_addr":"172.21.15.156:6813/114352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:35.454444+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:35.447401+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:36.195906+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:38.469 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-18T19:45:38.470 INFO:tasks.ceph:Creating RBD pool 2023-10-18T19:45:38.470 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c2 osd pool create rbd 8 2023-10-18T19:45:39.242 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'rbd' created 2023-10-18T19:45:39.254 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-18T19:45:40.246 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-18T19:45:40.261 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-10-18T19:45:40.263 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-10-18T19:45:40.263 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-10-18T19:45:40.263 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-10-18T19:45:40.264 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-18T19:45:40.681 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:40.694 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":113939},{"type":"v1","addr":"172.21.15.156:6825","nonce":113939}]},"active_addr":"172.21.15.156:6825/113939","active_change":"2023-10-18T19:45:36.219499+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":1091489988}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":3889667726}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":4215601511}]}]} 2023-10-18T19:45:40.697 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-10-18T19:45:40.698 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-10-18T19:45:40.698 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-18T19:45:41.019 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:41.019 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":11,"fsid":"4e3dcbf9-06fb-4969-a883-12e13c93f569","created":"2023-10-18T19:45:30.048018+0000","modified":"2023-10-18T19:45:40.233876+0000","last_up_change":"2023-10-18T19:45:37.126203+0000","last_in_change":"2023-10-18T19:45:31.068707+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-18T19:45:38.252552+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T19:45:38.802078+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d6031b86-8bd0-4c3e-a5be-2e373dca4c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":114361},{"type":"v1","addr":"172.21.15.156:6817","nonce":114361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":114361},{"type":"v1","addr":"172.21.15.156:6819","nonce":114361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":114361},{"type":"v1","addr":"172.21.15.156:6823","nonce":114361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":114361},{"type":"v1","addr":"172.21.15.156:6821","nonce":114361}]},"public_addr":"172.21.15.156:6817/114361","cluster_addr":"172.21.15.156:6819/114361","heartbeat_back_addr":"172.21.15.156:6823/114361","heartbeat_front_addr":"172.21.15.156:6821/114361","state":["exists","up"]},{"osd":1,"uuid":"608ad7bb-a3e6-4575-83ca-1f23d7d7f99b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":114351},{"type":"v1","addr":"172.21.15.156:6801","nonce":114351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":114351},{"type":"v1","addr":"172.21.15.156:6803","nonce":114351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":114351},{"type":"v1","addr":"172.21.15.156:6807","nonce":114351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":114351},{"type":"v1","addr":"172.21.15.156:6805","nonce":114351}]},"public_addr":"172.21.15.156:6801/114351","cluster_addr":"172.21.15.156:6803/114351","heartbeat_back_addr":"172.21.15.156:6807/114351","heartbeat_front_addr":"172.21.15.156:6805/114351","state":["exists","up"]},{"osd":2,"uuid":"fc8c0211-200f-4ce8-9313-d338dfb4060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":114352},{"type":"v1","addr":"172.21.15.156:6809","nonce":114352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":114352},{"type":"v1","addr":"172.21.15.156:6811","nonce":114352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":114352},{"type":"v1","addr":"172.21.15.156:6815","nonce":114352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":114352},{"type":"v1","addr":"172.21.15.156:6813","nonce":114352}]},"public_addr":"172.21.15.156:6809/114352","cluster_addr":"172.21.15.156:6811/114352","heartbeat_back_addr":"172.21.15.156:6815/114352","heartbeat_front_addr":"172.21.15.156:6813/114352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:35.454444+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:35.447401+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:36.195906+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:41.033 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-10-18T19:45:41.033 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-18T19:45:41.354 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:41.354 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":12,"fsid":"4e3dcbf9-06fb-4969-a883-12e13c93f569","created":"2023-10-18T19:45:30.048018+0000","modified":"2023-10-18T19:45:41.234757+0000","last_up_change":"2023-10-18T19:45:37.126203+0000","last_in_change":"2023-10-18T19:45:31.068707+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-18T19:45:38.252552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T19:45:38.802078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d6031b86-8bd0-4c3e-a5be-2e373dca4c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":114361},{"type":"v1","addr":"172.21.15.156:6817","nonce":114361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":114361},{"type":"v1","addr":"172.21.15.156:6819","nonce":114361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":114361},{"type":"v1","addr":"172.21.15.156:6823","nonce":114361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":114361},{"type":"v1","addr":"172.21.15.156:6821","nonce":114361}]},"public_addr":"172.21.15.156:6817/114361","cluster_addr":"172.21.15.156:6819/114361","heartbeat_back_addr":"172.21.15.156:6823/114361","heartbeat_front_addr":"172.21.15.156:6821/114361","state":["exists","up"]},{"osd":1,"uuid":"608ad7bb-a3e6-4575-83ca-1f23d7d7f99b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":114351},{"type":"v1","addr":"172.21.15.156:6801","nonce":114351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":114351},{"type":"v1","addr":"172.21.15.156:6803","nonce":114351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":114351},{"type":"v1","addr":"172.21.15.156:6807","nonce":114351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":114351},{"type":"v1","addr":"172.21.15.156:6805","nonce":114351}]},"public_addr":"172.21.15.156:6801/114351","cluster_addr":"172.21.15.156:6803/114351","heartbeat_back_addr":"172.21.15.156:6807/114351","heartbeat_front_addr":"172.21.15.156:6805/114351","state":["exists","up"]},{"osd":2,"uuid":"fc8c0211-200f-4ce8-9313-d338dfb4060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":114352},{"type":"v1","addr":"172.21.15.156:6809","nonce":114352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":114352},{"type":"v1","addr":"172.21.15.156:6811","nonce":114352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":114352},{"type":"v1","addr":"172.21.15.156:6815","nonce":114352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":114352},{"type":"v1","addr":"172.21.15.156:6813","nonce":114352}]},"public_addr":"172.21.15.156:6809/114352","cluster_addr":"172.21.15.156:6811/114352","heartbeat_back_addr":"172.21.15.156:6815/114352","heartbeat_front_addr":"172.21.15.156:6813/114352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:35.454444+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:35.447401+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-18T19:45:36.195906+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T19:45:41.368 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-10-18T19:45:41.368 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-10-18T19:45:41.369 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-10-18T19:45:41.538 INFO:teuthology.orchestra.run.smithi156.stdout:34359738370 2023-10-18T19:45:41.539 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-10-18T19:45:41.545 INFO:teuthology.orchestra.run.smithi156.stdout:34359738370 2023-10-18T19:45:41.546 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-18T19:45:41.552 INFO:teuthology.orchestra.run.smithi156.stdout:34359738370 2023-10-18T19:45:41.552 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-10-18T19:45:41.905 INFO:teuthology.orchestra.run.smithi156.stdout:34359738369 2023-10-18T19:45:41.920 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2023-10-18T19:45:41.931 INFO:teuthology.orchestra.run.smithi156.stdout:34359738369 2023-10-18T19:45:41.945 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2023-10-18T19:45:41.948 INFO:teuthology.orchestra.run.smithi156.stdout:34359738369 2023-10-18T19:45:41.962 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2023-10-18T19:45:42.920 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-10-18T19:45:42.946 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-10-18T19:45:42.963 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-18T19:45:43.297 INFO:teuthology.orchestra.run.smithi156.stdout:34359738370 2023-10-18T19:45:43.313 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.2 2023-10-18T19:45:43.314 DEBUG:teuthology.parallel:result is None 2023-10-18T19:45:43.315 INFO:teuthology.orchestra.run.smithi156.stdout:34359738370 2023-10-18T19:45:43.334 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.1 2023-10-18T19:45:43.334 DEBUG:teuthology.parallel:result is None 2023-10-18T19:45:43.336 INFO:teuthology.orchestra.run.smithi156.stdout:34359738370 2023-10-18T19:45:43.351 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2023-10-18T19:45:43.351 DEBUG:teuthology.parallel:result is None 2023-10-18T19:45:43.351 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-10-18T19:45:43.352 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-18T19:45:43.706 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:43.708 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2023-10-18T19:45:43.720 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-18T19:45:42.224421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82752,"kb_used_data":1452,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490048,"statfs":{"total":322122547200,"available":322037809152,"internally_reserved":0,"allocated":1486848,"data_stored":1258566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422151+0000","last_change":"2023-10-18T19:45:40.241602+0000","last_active":"2023-10-18T19:45:41.422151+0000","last_peered":"2023-10-18T19:45:41.422151+0000","last_clean":"2023-10-18T19:45:41.422151+0000","last_became_active":"2023-10-18T19:45:40.241327+0000","last_became_peered":"2023-10-18T19:45:40.241327+0000","last_unstale":"2023-10-18T19:45:41.422151+0000","last_undegraded":"2023-10-18T19:45:41.422151+0000","last_fullsized":"2023-10-18T19:45:41.422151+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T23:18:35.498103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422126+0000","last_change":"2023-10-18T19:45:40.241970+0000","last_active":"2023-10-18T19:45:41.422126+0000","last_peered":"2023-10-18T19:45:41.422126+0000","last_clean":"2023-10-18T19:45:41.422126+0000","last_became_active":"2023-10-18T19:45:40.241698+0000","last_became_peered":"2023-10-18T19:45:40.241698+0000","last_unstale":"2023-10-18T19:45:41.422126+0000","last_undegraded":"2023-10-18T19:45:41.422126+0000","last_fullsized":"2023-10-18T19:45:41.422126+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T23:47:08.667045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422198+0000","last_change":"2023-10-18T19:45:40.241489+0000","last_active":"2023-10-18T19:45:41.422198+0000","last_peered":"2023-10-18T19:45:41.422198+0000","last_clean":"2023-10-18T19:45:41.422198+0000","last_became_active":"2023-10-18T19:45:40.241251+0000","last_became_peered":"2023-10-18T19:45:40.241251+0000","last_unstale":"2023-10-18T19:45:41.422198+0000","last_undegraded":"2023-10-18T19:45:41.422198+0000","last_fullsized":"2023-10-18T19:45:41.422198+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T23:03:41.745457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.238718+0000","last_change":"2023-10-18T19:45:40.240482+0000","last_active":"2023-10-18T19:45:41.238718+0000","last_peered":"2023-10-18T19:45:41.238718+0000","last_clean":"2023-10-18T19:45:41.238718+0000","last_became_active":"2023-10-18T19:45:40.240212+0000","last_became_peered":"2023-10-18T19:45:40.240212+0000","last_unstale":"2023-10-18T19:45:41.238718+0000","last_undegraded":"2023-10-18T19:45:41.238718+0000","last_fullsized":"2023-10-18T19:45:41.238718+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T22:43:33.681259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422116+0000","last_change":"2023-10-18T19:45:40.241994+0000","last_active":"2023-10-18T19:45:41.422116+0000","last_peered":"2023-10-18T19:45:41.422116+0000","last_clean":"2023-10-18T19:45:41.422116+0000","last_became_active":"2023-10-18T19:45:40.241762+0000","last_became_peered":"2023-10-18T19:45:40.241762+0000","last_unstale":"2023-10-18T19:45:41.422116+0000","last_undegraded":"2023-10-18T19:45:41.422116+0000","last_fullsized":"2023-10-18T19:45:41.422116+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:27:18.534648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422009+0000","last_change":"2023-10-18T19:45:40.241614+0000","last_active":"2023-10-18T19:45:41.422009+0000","last_peered":"2023-10-18T19:45:41.422009+0000","last_clean":"2023-10-18T19:45:41.422009+0000","last_became_active":"2023-10-18T19:45:40.241323+0000","last_became_peered":"2023-10-18T19:45:40.241323+0000","last_unstale":"2023-10-18T19:45:41.422009+0000","last_undegraded":"2023-10-18T19:45:41.422009+0000","last_fullsized":"2023-10-18T19:45:41.422009+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T05:05:52.315972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.238718+0000","last_change":"2023-10-18T19:45:40.240772+0000","last_active":"2023-10-18T19:45:41.238718+0000","last_peered":"2023-10-18T19:45:41.238718+0000","last_clean":"2023-10-18T19:45:41.238718+0000","last_became_active":"2023-10-18T19:45:40.240466+0000","last_became_peered":"2023-10-18T19:45:40.240466+0000","last_unstale":"2023-10-18T19:45:41.238718+0000","last_undegraded":"2023-10-18T19:45:41.238718+0000","last_fullsized":"2023-10-18T19:45:41.238718+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T21:47:14.845812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:40.244490+0000","last_change":"2023-10-18T19:45:40.244163+0000","last_active":"2023-10-18T19:45:40.244490+0000","last_peered":"2023-10-18T19:45:40.244490+0000","last_clean":"2023-10-18T19:45:40.244490+0000","last_became_active":"2023-10-18T19:45:40.243908+0000","last_became_peered":"2023-10-18T19:45:40.243908+0000","last_unstale":"2023-10-18T19:45:40.244490+0000","last_undegraded":"2023-10-18T19:45:40.244490+0000","last_fullsized":"2023-10-18T19:45:40.244490+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T22:43:33.681259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422040+0000","last_change":"2023-10-18T19:45:40.242148+0000","last_active":"2023-10-18T19:45:41.422040+0000","last_peered":"2023-10-18T19:45:41.422040+0000","last_clean":"2023-10-18T19:45:41.422040+0000","last_became_active":"2023-10-18T19:45:40.241157+0000","last_became_peered":"2023-10-18T19:45:40.241157+0000","last_unstale":"2023-10-18T19:45:41.422040+0000","last_undegraded":"2023-10-18T19:45:41.422040+0000","last_fullsized":"2023-10-18T19:45:41.422040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T06:27:19.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27944,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829656,"statfs":{"total":107374182400,"available":107345567744,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27752,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829848,"statfs":{"total":107374182400,"available":107345764352,"internally_reserved":0,"allocated":688128,"data_stored":614388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-18T19:45:43.722 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-18T19:45:44.040 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:44.042 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2023-10-18T19:45:44.053 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-18T19:45:42.224421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82752,"kb_used_data":1452,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490048,"statfs":{"total":322122547200,"available":322037809152,"internally_reserved":0,"allocated":1486848,"data_stored":1258566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422151+0000","last_change":"2023-10-18T19:45:40.241602+0000","last_active":"2023-10-18T19:45:41.422151+0000","last_peered":"2023-10-18T19:45:41.422151+0000","last_clean":"2023-10-18T19:45:41.422151+0000","last_became_active":"2023-10-18T19:45:40.241327+0000","last_became_peered":"2023-10-18T19:45:40.241327+0000","last_unstale":"2023-10-18T19:45:41.422151+0000","last_undegraded":"2023-10-18T19:45:41.422151+0000","last_fullsized":"2023-10-18T19:45:41.422151+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T23:18:35.498103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422126+0000","last_change":"2023-10-18T19:45:40.241970+0000","last_active":"2023-10-18T19:45:41.422126+0000","last_peered":"2023-10-18T19:45:41.422126+0000","last_clean":"2023-10-18T19:45:41.422126+0000","last_became_active":"2023-10-18T19:45:40.241698+0000","last_became_peered":"2023-10-18T19:45:40.241698+0000","last_unstale":"2023-10-18T19:45:41.422126+0000","last_undegraded":"2023-10-18T19:45:41.422126+0000","last_fullsized":"2023-10-18T19:45:41.422126+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T23:47:08.667045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422198+0000","last_change":"2023-10-18T19:45:40.241489+0000","last_active":"2023-10-18T19:45:41.422198+0000","last_peered":"2023-10-18T19:45:41.422198+0000","last_clean":"2023-10-18T19:45:41.422198+0000","last_became_active":"2023-10-18T19:45:40.241251+0000","last_became_peered":"2023-10-18T19:45:40.241251+0000","last_unstale":"2023-10-18T19:45:41.422198+0000","last_undegraded":"2023-10-18T19:45:41.422198+0000","last_fullsized":"2023-10-18T19:45:41.422198+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T23:03:41.745457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.238718+0000","last_change":"2023-10-18T19:45:40.240482+0000","last_active":"2023-10-18T19:45:41.238718+0000","last_peered":"2023-10-18T19:45:41.238718+0000","last_clean":"2023-10-18T19:45:41.238718+0000","last_became_active":"2023-10-18T19:45:40.240212+0000","last_became_peered":"2023-10-18T19:45:40.240212+0000","last_unstale":"2023-10-18T19:45:41.238718+0000","last_undegraded":"2023-10-18T19:45:41.238718+0000","last_fullsized":"2023-10-18T19:45:41.238718+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T22:43:33.681259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422116+0000","last_change":"2023-10-18T19:45:40.241994+0000","last_active":"2023-10-18T19:45:41.422116+0000","last_peered":"2023-10-18T19:45:41.422116+0000","last_clean":"2023-10-18T19:45:41.422116+0000","last_became_active":"2023-10-18T19:45:40.241762+0000","last_became_peered":"2023-10-18T19:45:40.241762+0000","last_unstale":"2023-10-18T19:45:41.422116+0000","last_undegraded":"2023-10-18T19:45:41.422116+0000","last_fullsized":"2023-10-18T19:45:41.422116+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:27:18.534648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422009+0000","last_change":"2023-10-18T19:45:40.241614+0000","last_active":"2023-10-18T19:45:41.422009+0000","last_peered":"2023-10-18T19:45:41.422009+0000","last_clean":"2023-10-18T19:45:41.422009+0000","last_became_active":"2023-10-18T19:45:40.241323+0000","last_became_peered":"2023-10-18T19:45:40.241323+0000","last_unstale":"2023-10-18T19:45:41.422009+0000","last_undegraded":"2023-10-18T19:45:41.422009+0000","last_fullsized":"2023-10-18T19:45:41.422009+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T05:05:52.315972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.238718+0000","last_change":"2023-10-18T19:45:40.240772+0000","last_active":"2023-10-18T19:45:41.238718+0000","last_peered":"2023-10-18T19:45:41.238718+0000","last_clean":"2023-10-18T19:45:41.238718+0000","last_became_active":"2023-10-18T19:45:40.240466+0000","last_became_peered":"2023-10-18T19:45:40.240466+0000","last_unstale":"2023-10-18T19:45:41.238718+0000","last_undegraded":"2023-10-18T19:45:41.238718+0000","last_fullsized":"2023-10-18T19:45:41.238718+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T21:47:14.845812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-18T19:45:40.244490+0000","last_change":"2023-10-18T19:45:40.244163+0000","last_active":"2023-10-18T19:45:40.244490+0000","last_peered":"2023-10-18T19:45:40.244490+0000","last_clean":"2023-10-18T19:45:40.244490+0000","last_became_active":"2023-10-18T19:45:40.243908+0000","last_became_peered":"2023-10-18T19:45:40.243908+0000","last_unstale":"2023-10-18T19:45:40.244490+0000","last_undegraded":"2023-10-18T19:45:40.244490+0000","last_fullsized":"2023-10-18T19:45:40.244490+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-19T22:43:33.681259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T19:45:41.422040+0000","last_change":"2023-10-18T19:45:40.242148+0000","last_active":"2023-10-18T19:45:41.422040+0000","last_peered":"2023-10-18T19:45:41.422040+0000","last_clean":"2023-10-18T19:45:41.422040+0000","last_became_active":"2023-10-18T19:45:40.241157+0000","last_became_peered":"2023-10-18T19:45:40.241157+0000","last_unstale":"2023-10-18T19:45:41.422040+0000","last_undegraded":"2023-10-18T19:45:41.422040+0000","last_fullsized":"2023-10-18T19:45:41.422040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T19:45:39.230301+0000","last_clean_scrub_stamp":"2023-10-18T19:45:39.230301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T06:27:19.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27944,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829656,"statfs":{"total":107374182400,"available":107345567744,"internally_reserved":0,"allocated":692224,"data_stored":615349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27752,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829848,"statfs":{"total":107374182400,"available":107345764352,"internally_reserved":0,"allocated":688128,"data_stored":614388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-18T19:45:44.054 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-10-18T19:45:44.055 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-10-18T19:45:44.055 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-10-18T19:45:44.055 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-10-18T19:45:44.420 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-10-18T19:45:44.420 INFO:teuthology.orchestra.run.smithi156.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-18T19:45:44.433 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-10-18T19:45:44.434 INFO:teuthology.run_tasks:Running task rgw... 2023-10-18T19:45:44.450 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-10-18T19:45:44.451 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-10-18T19:45:44.451 INFO:tasks.rgw:Creating data pools 2023-10-18T19:45:44.452 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-10-18T19:45:44.452 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-18T19:45:44.861 INFO:teuthology.orchestra.run.smithi082.stderr:pool 'default.rgw.buckets.data' created 2023-10-18T19:45:44.877 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-18T19:45:45.896 INFO:teuthology.orchestra.run.smithi082.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-18T19:45:45.906 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-18T19:45:46.886 INFO:teuthology.orchestra.run.smithi082.stderr:pool 'default.rgw.buckets.index' created 2023-10-18T19:45:46.900 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-18T19:45:47.897 INFO:teuthology.orchestra.run.smithi082.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-18T19:45:47.917 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-10-18T19:45:47.917 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-18T19:45:48.244 INFO:teuthology.orchestra.run.smithi082.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-18T19:45:48.255 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-18T19:45:48.883 INFO:teuthology.orchestra.run.smithi082.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-18T19:45:48.895 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-18T19:45:49.219 INFO:teuthology.orchestra.run.smithi082.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-18T19:45:49.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-18T19:45:49.890 INFO:teuthology.orchestra.run.smithi082.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-18T19:45:49.901 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-10-18T19:45:49.902 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-18T19:45:51.055 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.data' created 2023-10-18T19:45:51.067 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-18T19:45:52.060 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-18T19:45:52.082 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-18T19:45:53.062 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.index' created 2023-10-18T19:45:53.072 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-18T19:45:54.059 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-18T19:45:54.083 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-10-18T19:45:54.085 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-18T19:45:54.412 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-18T19:45:54.423 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-18T19:45:55.056 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-18T19:45:55.068 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-18T19:45:55.391 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-18T19:45:55.402 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-18T19:45:56.058 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-18T19:45:56.069 DEBUG:tasks.rgw:Pools created 2023-10-18T19:45:56.070 INFO:tasks.rgw:Configuring compression type = random 2023-10-18T19:45:56.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-10-18T19:45:56.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-10-18T19:45:56.071 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-10-18T19:45:56.189 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:45:56.189 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:45:56.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:56.197+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.014 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.012+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.012+0000 7f692ad1c780 20 realm 2023-10-18T19:45:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.012+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.013+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.016 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.013+0000 7f692ad1c780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-18T19:45:58.016 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.013+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.016 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.013+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.016 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.013+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.020+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.022 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.020+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.020+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.020+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.021+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.023 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.021+0000 7f692ad1c780 10 Creating default zonegroup 2023-10-18T19:45:58.024 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.021+0000 7f692ad1c780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-18T19:45:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.032+0000 7f692ad1c780 10 WARNING: driver->list_zones() returned r=-2 2023-10-18T19:45:58.034 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.032+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.033+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.035+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.035+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.037 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.035+0000 7f692ad1c780 10 could not read realm id: (2) No such file or directory 2023-10-18T19:45:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.035+0000 7f692ad1c780 10 WARNING: failed to set zone as default, r=-22 2023-10-18T19:45:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.035+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.035+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.040 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.038+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.038+0000 7f692ad1c780 20 rados_obj.operate() r=0 bl.length=874 2023-10-18T19:45:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.038+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.039+0000 7f692ad1c780 20 rados_obj.operate() r=0 bl.length=420 2023-10-18T19:45:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.039+0000 7f692ad1c780 10 Using default name default 2023-10-18T19:45:58.041 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.039+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.042 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.039+0000 7f692ad1c780 20 rados_obj.operate() r=0 bl.length=46 2023-10-18T19:45:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.039+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.039+0000 7f692ad1c780 20 rados_obj.operate() r=0 bl.length=874 2023-10-18T19:45:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.040+0000 7f692ad1c780 20 zone default found 2023-10-18T19:45:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.040+0000 7f692ad1c780 4 Realm: () 2023-10-18T19:45:58.043 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.040+0000 7f692ad1c780 4 ZoneGroup: default (172feeac-636c-43de-9f26-4ee4d1c0a18c) 2023-10-18T19:45:58.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.040+0000 7f692ad1c780 4 Zone: default (136b8c7c-ea7b-4fcd-93e3-75760d92cbba) 2023-10-18T19:45:58.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.040+0000 7f692ad1c780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-18T19:45:58.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.040+0000 7f692ad1c780 20 zonegroup default 2023-10-18T19:45:58.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.040+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:45:58.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.040+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:45:58.045 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:45:58.040+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:46:00.032 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:00.030+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:00.032 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:00.030+0000 7f692ad1c780 20 rados->read ofs=0 len=0 2023-10-18T19:46:00.033 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:00.031+0000 7f692ad1c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:00.033 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:00.031+0000 7f692ad1c780 20 started sync module instance, tier type = 2023-10-18T19:46:00.034 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:00.031+0000 7f692ad1c780 20 started zone id=136b8c7c-ea7b-4fcd-93e3-75760d92cbba (name=default) with tier type = 2023-10-18T19:46:02.069 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.067+0000 7f692ad1c780 20 add_watcher() i=0 2023-10-18T19:46:02.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.067+0000 7f692ad1c780 20 add_watcher() i=1 2023-10-18T19:46:02.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.067+0000 7f692ad1c780 20 add_watcher() i=2 2023-10-18T19:46:02.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.067+0000 7f692ad1c780 20 add_watcher() i=3 2023-10-18T19:46:02.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.067+0000 7f692ad1c780 20 add_watcher() i=4 2023-10-18T19:46:02.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.067+0000 7f692ad1c780 20 add_watcher() i=5 2023-10-18T19:46:02.071 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.067+0000 7f692ad1c780 20 add_watcher() i=6 2023-10-18T19:46:02.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.068+0000 7f692ad1c780 20 add_watcher() i=7 2023-10-18T19:46:02.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.068+0000 7f692ad1c780 2 all 8 watchers are set, enabling cache 2023-10-18T19:46:02.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.068+0000 7f692ad1c780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-10-18T19:46:02.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.068+0000 7f692ad1c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-10-18T19:46:02.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.068+0000 7f692ad1c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:02.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-10-18T19:46:02.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-10-18T19:46:02.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:02.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-10-18T19:46:02.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-10-18T19:46:02.074 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:02.074 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-10-18T19:46:02.074 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-10-18T19:46:02.074 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:02.074 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-10-18T19:46:02.075 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.069+0000 7f692ad1c780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-10-18T19:46:02.075 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.070+0000 7f692ad1c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:02.076 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.073+0000 7f690fc98700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-18T19:46:02.076 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.073+0000 7f692ad1c780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-18T19:46:02.076 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:02.073+0000 7f692ad1c780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-18T19:46:05.043 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.037+0000 7f692ad1c780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-18T19:46:05.043 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.037+0000 7f692ad1c780 5 note: GC not initialized 2023-10-18T19:46:05.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.038+0000 7f690b48f700 20 reqs_thread_entry: start 2023-10-18T19:46:05.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.039+0000 7f692ad1c780 20 init_complete bucket index max shards: 11 2023-10-18T19:46:05.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.039+0000 7f692ad1c780 10 rgw notify: Started notification manager with: 1 workers 2023-10-18T19:46:05.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.039+0000 7f690948b700 20 rgw notify: INFO: next queues processing will happen at: Wed Oct 18 19:46:35 2023 2023-10-18T19:46:05.045 INFO:teuthology.orchestra.run.smithi082.stderr: 2023-10-18T19:46:05.045 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.040+0000 7f6908c69700 20 reqs_thread_entry: start 2023-10-18T19:46:05.064 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.061+0000 7f690fc98700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-18T19:46:05.065 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.063+0000 7f692ad1c780 20 remove_watcher() i=0 2023-10-18T19:46:05.065 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.063+0000 7f692ad1c780 2 removed watcher, disabling cache 2023-10-18T19:46:05.067 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.066+0000 7f692ad1c780 20 remove_watcher() i=1 2023-10-18T19:46:05.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.068+0000 7f692ad1c780 20 remove_watcher() i=2 2023-10-18T19:46:05.076 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.074+0000 7f692ad1c780 20 remove_watcher() i=3 2023-10-18T19:46:05.077 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.075+0000 7f692ad1c780 20 remove_watcher() i=4 2023-10-18T19:46:05.079 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.077+0000 7f692ad1c780 20 remove_watcher() i=5 2023-10-18T19:46:05.080 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.078+0000 7f692ad1c780 20 remove_watcher() i=6 2023-10-18T19:46:05.081 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.079+0000 7f692ad1c780 20 remove_watcher() i=7 2023-10-18T19:46:05.084 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T19:46:05.086 DEBUG:tasks.util.rgw: json result: [] 2023-10-18T19:46:05.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-18T19:46:05.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-18T19:46:05.087 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-18T19:46:05.141 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:05.141 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:05.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.148+0000 7fa4c54d1780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-18T19:46:05.150 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.148+0000 7fa4c54d1780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-18T19:46:05.153 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.151+0000 7fa4a6c46700 20 reqs_thread_entry: start 2023-10-18T19:46:05.159 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:05.159 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "136b8c7c-ea7b-4fcd-93e3-75760d92cbba", 2023-10-18T19:46:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default", 2023-10-18T19:46:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "default.rgw.meta:root", 2023-10-18T19:46:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "default.rgw.control", 2023-10-18T19:46:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-18T19:46:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-18T19:46:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "default.rgw.log", 2023-10-18T19:46:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-18T19:46:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-18T19:46:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-18T19:46:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-18T19:46:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-18T19:46:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-18T19:46:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-18T19:46:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-18T19:46:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "default.rgw.otp", 2023-10-18T19:46:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T19:46:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "", 2023-10-18T19:46:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "" 2023-10-18T19:46:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T19:46:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:05.164 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T19:46:05.164 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:46:05.164 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-18T19:46:05.164 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T19:46:05.164 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T19:46:05.165 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-18T19:46:05.165 INFO:teuthology.orchestra.run.smithi082.stdout: "compression_type": "random" 2023-10-18T19:46:05.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:05.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:05.165 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-18T19:46:05.166 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T19:46:05.166 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T19:46:05.166 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:05.166 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:05.166 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:05.167 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "", 2023-10-18T19:46:05.167 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-18T19:46:05.167 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:05.168 DEBUG:tasks.util.rgw: json result: {'id': '136b8c7c-ea7b-4fcd-93e3-75760d92cbba', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-18T19:46:05.168 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-10-18T19:46:05.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-10-18T19:46:05.169 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-10-18T19:46:05.214 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:05.214 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:05.222 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.220+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.223 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.221+0000 7f48394c4780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:05.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.221+0000 7f48394c4780 20 realm 2023-10-18T19:46:05.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.221+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.222+0000 7f48394c4780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:05.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.222+0000 7f48394c4780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-18T19:46:05.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.222+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.222+0000 7f48394c4780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:05.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.222+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.222+0000 7f48394c4780 20 rados_obj.operate() r=0 bl.length=46 2023-10-18T19:46:05.226 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.222+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.226 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.222+0000 7f48394c4780 20 rados_obj.operate() r=0 bl.length=890 2023-10-18T19:46:05.226 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.222+0000 7f48394c4780 20 searching for the correct realm 2023-10-18T19:46:05.231 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 RGWRados::pool_iterate: got zonegroup_info.172feeac-636c-43de-9f26-4ee4d1c0a18c 2023-10-18T19:46:05.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 RGWRados::pool_iterate: got zone_info.136b8c7c-ea7b-4fcd-93e3-75760d92cbba 2023-10-18T19:46:05.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 RGWRados::pool_iterate: got zone_names.default 2023-10-18T19:46:05.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-18T19:46:05.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:05.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 rados_obj.operate() r=0 bl.length=46 2023-10-18T19:46:05.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 rados_obj.operate() r=0 bl.length=420 2023-10-18T19:46:05.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 zone default found 2023-10-18T19:46:05.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 4 Realm: () 2023-10-18T19:46:05.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 4 ZoneGroup: default (172feeac-636c-43de-9f26-4ee4d1c0a18c) 2023-10-18T19:46:05.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 4 Zone: default (136b8c7c-ea7b-4fcd-93e3-75760d92cbba) 2023-10-18T19:46:05.236 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-18T19:46:05.236 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 zonegroup default 2023-10-18T19:46:05.236 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.236 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:05.237 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.230+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.237 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.231+0000 7f48394c4780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:05.237 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.231+0000 7f48394c4780 20 rados->read ofs=0 len=0 2023-10-18T19:46:05.237 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.231+0000 7f48394c4780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:05.237 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.231+0000 7f48394c4780 20 started sync module instance, tier type = 2023-10-18T19:46:05.238 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.231+0000 7f48394c4780 20 started zone id=136b8c7c-ea7b-4fcd-93e3-75760d92cbba (name=default) with tier type = 2023-10-18T19:46:05.269 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.267+0000 7f48394c4780 20 add_watcher() i=0 2023-10-18T19:46:05.269 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.267+0000 7f48394c4780 20 add_watcher() i=1 2023-10-18T19:46:05.270 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.267+0000 7f48394c4780 20 add_watcher() i=2 2023-10-18T19:46:05.270 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.267+0000 7f48394c4780 20 add_watcher() i=3 2023-10-18T19:46:05.270 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.267+0000 7f48394c4780 20 add_watcher() i=4 2023-10-18T19:46:05.270 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.267+0000 7f48394c4780 20 add_watcher() i=5 2023-10-18T19:46:05.271 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.267+0000 7f48394c4780 20 add_watcher() i=6 2023-10-18T19:46:05.271 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.268+0000 7f48394c4780 20 add_watcher() i=7 2023-10-18T19:46:05.271 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.268+0000 7f48394c4780 2 all 8 watchers are set, enabling cache 2023-10-18T19:46:05.272 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.270+0000 7f481e440700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-18T19:46:05.272 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.270+0000 7f48394c4780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-18T19:46:05.272 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.270+0000 7f48394c4780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-18T19:46:05.275 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.274+0000 7f48394c4780 5 note: GC not initialized 2023-10-18T19:46:05.277 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.275+0000 7f4819c37700 20 reqs_thread_entry: start 2023-10-18T19:46:05.277 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.275+0000 7f48394c4780 20 init_complete bucket index max shards: 11 2023-10-18T19:46:05.278 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.275+0000 7f48394c4780 10 rgw notify: Started notification manager with: 1 workers 2023-10-18T19:46:05.278 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.275+0000 7f4817c33700 20 rgw notify: INFO: next queues processing will happen at: Wed Oct 18 19:46:35 2023 2023-10-18T19:46:05.278 INFO:teuthology.orchestra.run.smithi082.stderr: 2023-10-18T19:46:05.278 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.275+0000 7f4817411700 20 reqs_thread_entry: start 2023-10-18T19:46:05.284 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.282+0000 7f481e440700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-18T19:46:05.287 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.285+0000 7f48394c4780 20 remove_watcher() i=0 2023-10-18T19:46:05.287 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.285+0000 7f48394c4780 2 removed watcher, disabling cache 2023-10-18T19:46:05.290 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.288+0000 7f48394c4780 20 remove_watcher() i=1 2023-10-18T19:46:05.292 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.290+0000 7f48394c4780 20 remove_watcher() i=2 2023-10-18T19:46:05.295 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.293+0000 7f48394c4780 20 remove_watcher() i=3 2023-10-18T19:46:05.298 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.296+0000 7f48394c4780 20 remove_watcher() i=4 2023-10-18T19:46:05.299 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.297+0000 7f48394c4780 20 remove_watcher() i=5 2023-10-18T19:46:05.301 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.299+0000 7f48394c4780 20 remove_watcher() i=6 2023-10-18T19:46:05.301 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.299+0000 7f48394c4780 20 remove_watcher() i=7 2023-10-18T19:46:05.303 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T19:46:05.305 DEBUG:tasks.util.rgw: json result: [] 2023-10-18T19:46:05.305 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-18T19:46:05.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-18T19:46:05.306 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-18T19:46:05.356 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:05.356 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:05.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.362+0000 7fe0fded7780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-18T19:46:05.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.362+0000 7fe0fded7780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-18T19:46:05.367 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:05.365+0000 7fe0df64c700 20 reqs_thread_entry: start 2023-10-18T19:46:05.372 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "136b8c7c-ea7b-4fcd-93e3-75760d92cbba", 2023-10-18T19:46:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default", 2023-10-18T19:46:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "default.rgw.meta:root", 2023-10-18T19:46:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "default.rgw.control", 2023-10-18T19:46:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-18T19:46:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-18T19:46:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "default.rgw.log", 2023-10-18T19:46:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-18T19:46:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-18T19:46:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-18T19:46:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-18T19:46:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-18T19:46:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-18T19:46:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-18T19:46:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-18T19:46:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "default.rgw.otp", 2023-10-18T19:46:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T19:46:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "", 2023-10-18T19:46:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "" 2023-10-18T19:46:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T19:46:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T19:46:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:46:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-18T19:46:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T19:46:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T19:46:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-18T19:46:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "compression_type": "random" 2023-10-18T19:46:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-18T19:46:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T19:46:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T19:46:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "", 2023-10-18T19:46:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-18T19:46:05.384 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:05.385 DEBUG:tasks.util.rgw: json result: {'id': '136b8c7c-ea7b-4fcd-93e3-75760d92cbba', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-18T19:46:05.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-10-18T19:46:05.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-10-18T19:46:05.386 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-10-18T19:46:05.512 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:05.513 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:05.522 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:05.520+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.058+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.062 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.058+0000 7fd3a1394780 20 realm 2023-10-18T19:46:07.062 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.058+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.060+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.060+0000 7fd3a1394780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-18T19:46:07.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.060+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.061+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.061+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.065 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.061+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.065 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.061+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.065 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.061+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.065 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.061+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.066 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.062+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.066 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.062+0000 7fd3a1394780 10 Creating default zonegroup 2023-10-18T19:46:07.066 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.062+0000 7fd3a1394780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-18T19:46:07.074 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.071+0000 7fd3a1394780 10 WARNING: driver->list_zones() returned r=-2 2023-10-18T19:46:07.074 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.071+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.074 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.071+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.074+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.074+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.074+0000 7fd3a1394780 10 could not read realm id: (2) No such file or directory 2023-10-18T19:46:07.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.074+0000 7fd3a1394780 10 WARNING: failed to set zone as default, r=-22 2023-10-18T19:46:07.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.074+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.078 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.074+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.079 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.076+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.080 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 rados_obj.operate() r=0 bl.length=874 2023-10-18T19:46:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 rados_obj.operate() r=0 bl.length=420 2023-10-18T19:46:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 10 Using default name default 2023-10-18T19:46:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 rados_obj.operate() r=0 bl.length=46 2023-10-18T19:46:07.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 rados_obj.operate() r=0 bl.length=874 2023-10-18T19:46:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 zone default found 2023-10-18T19:46:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 4 Realm: () 2023-10-18T19:46:07.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 4 ZoneGroup: default (48302423-9890-40bd-8780-0613c3527979) 2023-10-18T19:46:07.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 4 Zone: default (02d207aa-78c1-4ca9-9ff1-0debece50378) 2023-10-18T19:46:07.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-18T19:46:07.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 zonegroup default 2023-10-18T19:46:07.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.077+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:07.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.078+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:07.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:07.078+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:09.069 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:09.066+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:09.069 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:09.066+0000 7fd3a1394780 20 rados->read ofs=0 len=0 2023-10-18T19:46:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:09.067+0000 7fd3a1394780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:09.067+0000 7fd3a1394780 20 started sync module instance, tier type = 2023-10-18T19:46:09.070 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:09.067+0000 7fd3a1394780 20 started zone id=02d207aa-78c1-4ca9-9ff1-0debece50378 (name=default) with tier type = 2023-10-18T19:46:11.087 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.084+0000 7fd3a1394780 20 add_watcher() i=0 2023-10-18T19:46:11.088 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.084+0000 7fd3a1394780 20 add_watcher() i=1 2023-10-18T19:46:11.088 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.084+0000 7fd3a1394780 20 add_watcher() i=2 2023-10-18T19:46:11.088 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.084+0000 7fd3a1394780 20 add_watcher() i=3 2023-10-18T19:46:11.088 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.084+0000 7fd3a1394780 20 add_watcher() i=4 2023-10-18T19:46:11.088 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.084+0000 7fd3a1394780 20 add_watcher() i=5 2023-10-18T19:46:11.089 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.084+0000 7fd3a1394780 20 add_watcher() i=6 2023-10-18T19:46:11.090 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.085+0000 7fd3a1394780 20 add_watcher() i=7 2023-10-18T19:46:11.090 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.085+0000 7fd3a1394780 2 all 8 watchers are set, enabling cache 2023-10-18T19:46:11.090 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-10-18T19:46:11.090 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-10-18T19:46:11.091 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:11.091 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-10-18T19:46:11.091 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-10-18T19:46:11.091 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:11.091 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-10-18T19:46:11.092 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-10-18T19:46:11.092 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:11.092 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-10-18T19:46:11.092 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-10-18T19:46:11.092 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.086+0000 7fd3a1394780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:11.093 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.087+0000 7fd3a1394780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-10-18T19:46:11.093 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.087+0000 7fd3a1394780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-10-18T19:46:11.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.088+0000 7fd3a1394780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-18T19:46:11.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.090+0000 7fd386310700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-18T19:46:11.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.090+0000 7fd3a1394780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-18T19:46:11.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:11.090+0000 7fd3a1394780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-18T19:46:14.067 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.064+0000 7fd3a1394780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-18T19:46:14.067 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.064+0000 7fd3a1394780 5 note: GC not initialized 2023-10-18T19:46:14.068 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.065+0000 7fd381b07700 20 reqs_thread_entry: start 2023-10-18T19:46:14.069 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.066+0000 7fd3a1394780 20 init_complete bucket index max shards: 11 2023-10-18T19:46:14.069 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.066+0000 7fd3a1394780 10 rgw notify: Started notification manager with: 1 workers 2023-10-18T19:46:14.069 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.066+0000 7fd37fb03700 20 rgw notify: INFO: next queues processing will happen at: Wed Oct 18 19:46:44 2023 2023-10-18T19:46:14.069 INFO:teuthology.orchestra.run.smithi156.stderr: 2023-10-18T19:46:14.069 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.066+0000 7fd37f2e1700 20 reqs_thread_entry: start 2023-10-18T19:46:14.088 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.085+0000 7fd386310700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-18T19:46:14.090 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.088+0000 7fd3a1394780 20 remove_watcher() i=0 2023-10-18T19:46:14.090 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.088+0000 7fd3a1394780 2 removed watcher, disabling cache 2023-10-18T19:46:14.091 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.089+0000 7fd3a1394780 20 remove_watcher() i=1 2023-10-18T19:46:14.092 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.090+0000 7fd3a1394780 20 remove_watcher() i=2 2023-10-18T19:46:14.093 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.091+0000 7fd3a1394780 20 remove_watcher() i=3 2023-10-18T19:46:14.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.092+0000 7fd3a1394780 20 remove_watcher() i=4 2023-10-18T19:46:14.095 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.093+0000 7fd3a1394780 20 remove_watcher() i=5 2023-10-18T19:46:14.096 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.094+0000 7fd3a1394780 20 remove_watcher() i=6 2023-10-18T19:46:14.097 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.095+0000 7fd3a1394780 20 remove_watcher() i=7 2023-10-18T19:46:14.100 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2023-10-18T19:46:14.102 DEBUG:tasks.util.rgw: json result: [] 2023-10-18T19:46:14.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-18T19:46:14.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-18T19:46:14.103 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-18T19:46:14.157 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:14.158 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:14.167 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.164+0000 7f39a202a780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-18T19:46:14.167 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.164+0000 7f39a202a780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-18T19:46:14.171 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.169+0000 7f398379f700 20 reqs_thread_entry: start 2023-10-18T19:46:14.178 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:46:14.178 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "02d207aa-78c1-4ca9-9ff1-0debece50378", 2023-10-18T19:46:14.178 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "default", 2023-10-18T19:46:14.178 INFO:teuthology.orchestra.run.smithi156.stdout: "domain_root": "default.rgw.meta:root", 2023-10-18T19:46:14.178 INFO:teuthology.orchestra.run.smithi156.stdout: "control_pool": "default.rgw.control", 2023-10-18T19:46:14.179 INFO:teuthology.orchestra.run.smithi156.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-18T19:46:14.179 INFO:teuthology.orchestra.run.smithi156.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-18T19:46:14.179 INFO:teuthology.orchestra.run.smithi156.stdout: "log_pool": "default.rgw.log", 2023-10-18T19:46:14.179 INFO:teuthology.orchestra.run.smithi156.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-18T19:46:14.179 INFO:teuthology.orchestra.run.smithi156.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-18T19:46:14.180 INFO:teuthology.orchestra.run.smithi156.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-18T19:46:14.180 INFO:teuthology.orchestra.run.smithi156.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-18T19:46:14.180 INFO:teuthology.orchestra.run.smithi156.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-18T19:46:14.180 INFO:teuthology.orchestra.run.smithi156.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-18T19:46:14.180 INFO:teuthology.orchestra.run.smithi156.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-18T19:46:14.181 INFO:teuthology.orchestra.run.smithi156.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-18T19:46:14.181 INFO:teuthology.orchestra.run.smithi156.stdout: "otp_pool": "default.rgw.otp", 2023-10-18T19:46:14.181 INFO:teuthology.orchestra.run.smithi156.stdout: "system_key": { 2023-10-18T19:46:14.181 INFO:teuthology.orchestra.run.smithi156.stdout: "access_key": "", 2023-10-18T19:46:14.181 INFO:teuthology.orchestra.run.smithi156.stdout: "secret_key": "" 2023-10-18T19:46:14.181 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:14.182 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_pools": [ 2023-10-18T19:46:14.182 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:14.182 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "default-placement", 2023-10-18T19:46:14.182 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:46:14.182 INFO:teuthology.orchestra.run.smithi156.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-18T19:46:14.183 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": { 2023-10-18T19:46:14.183 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD": { 2023-10-18T19:46:14.183 INFO:teuthology.orchestra.run.smithi156.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-18T19:46:14.183 INFO:teuthology.orchestra.run.smithi156.stdout: "compression_type": "random" 2023-10-18T19:46:14.183 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:14.183 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:14.184 INFO:teuthology.orchestra.run.smithi156.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-18T19:46:14.184 INFO:teuthology.orchestra.run.smithi156.stdout: "index_type": 0, 2023-10-18T19:46:14.184 INFO:teuthology.orchestra.run.smithi156.stdout: "inline_data": true 2023-10-18T19:46:14.184 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:14.184 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:14.185 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:14.185 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "", 2023-10-18T19:46:14.185 INFO:teuthology.orchestra.run.smithi156.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-18T19:46:14.185 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:46:14.186 DEBUG:tasks.util.rgw: json result: {'id': '02d207aa-78c1-4ca9-9ff1-0debece50378', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-18T19:46:14.186 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-10-18T19:46:14.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-10-18T19:46:14.187 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-10-18T19:46:14.232 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:14.232 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:14.241 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.238+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.242 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.239+0000 7fd8e3a69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:14.243 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.239+0000 7fd8e3a69780 20 realm 2023-10-18T19:46:14.243 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.239+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.243 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.239+0000 7fd8e3a69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:14.243 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.239+0000 7fd8e3a69780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-18T19:46:14.243 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.239+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.244 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.240+0000 7fd8e3a69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:14.244 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.240+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.244 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.240+0000 7fd8e3a69780 20 rados_obj.operate() r=0 bl.length=46 2023-10-18T19:46:14.244 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.240+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.245 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.241+0000 7fd8e3a69780 20 rados_obj.operate() r=0 bl.length=890 2023-10-18T19:46:14.245 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.241+0000 7fd8e3a69780 20 searching for the correct realm 2023-10-18T19:46:14.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.247+0000 7fd8e3a69780 20 RGWRados::pool_iterate: got zone_info.02d207aa-78c1-4ca9-9ff1-0debece50378 2023-10-18T19:46:14.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.247+0000 7fd8e3a69780 20 RGWRados::pool_iterate: got zonegroup_info.48302423-9890-40bd-8780-0613c3527979 2023-10-18T19:46:14.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.247+0000 7fd8e3a69780 20 RGWRados::pool_iterate: got zone_names.default 2023-10-18T19:46:14.253 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.247+0000 7fd8e3a69780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-18T19:46:14.253 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.247+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.253 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.247+0000 7fd8e3a69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:14.254 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.247+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.254 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 rados_obj.operate() r=0 bl.length=46 2023-10-18T19:46:14.254 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.254 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 rados_obj.operate() r=0 bl.length=420 2023-10-18T19:46:14.254 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 zone default found 2023-10-18T19:46:14.255 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 4 Realm: () 2023-10-18T19:46:14.255 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 4 ZoneGroup: default (48302423-9890-40bd-8780-0613c3527979) 2023-10-18T19:46:14.255 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 4 Zone: default (02d207aa-78c1-4ca9-9ff1-0debece50378) 2023-10-18T19:46:14.255 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-18T19:46:14.255 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 zonegroup default 2023-10-18T19:46:14.256 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.256 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:14.256 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.256 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:14.256 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.248+0000 7fd8e3a69780 20 rados->read ofs=0 len=0 2023-10-18T19:46:14.257 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.249+0000 7fd8e3a69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-18T19:46:14.257 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.249+0000 7fd8e3a69780 20 started sync module instance, tier type = 2023-10-18T19:46:14.257 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.249+0000 7fd8e3a69780 20 started zone id=02d207aa-78c1-4ca9-9ff1-0debece50378 (name=default) with tier type = 2023-10-18T19:46:14.259 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.256+0000 7fd8e3a69780 20 add_watcher() i=0 2023-10-18T19:46:14.259 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.256+0000 7fd8e3a69780 20 add_watcher() i=1 2023-10-18T19:46:14.259 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.256+0000 7fd8e3a69780 20 add_watcher() i=2 2023-10-18T19:46:14.259 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.256+0000 7fd8e3a69780 20 add_watcher() i=3 2023-10-18T19:46:14.259 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.256+0000 7fd8e3a69780 20 add_watcher() i=4 2023-10-18T19:46:14.260 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.256+0000 7fd8e3a69780 20 add_watcher() i=5 2023-10-18T19:46:14.260 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.256+0000 7fd8e3a69780 20 add_watcher() i=6 2023-10-18T19:46:14.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.257+0000 7fd8e3a69780 20 add_watcher() i=7 2023-10-18T19:46:14.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.257+0000 7fd8e3a69780 2 all 8 watchers are set, enabling cache 2023-10-18T19:46:14.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.258+0000 7fd8c89e5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-18T19:46:14.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.258+0000 7fd8e3a69780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-18T19:46:14.261 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.258+0000 7fd8e3a69780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-18T19:46:14.264 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.262+0000 7fd8e3a69780 5 note: GC not initialized 2023-10-18T19:46:14.266 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.263+0000 7fd8c41dc700 20 reqs_thread_entry: start 2023-10-18T19:46:14.266 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.263+0000 7fd8e3a69780 20 init_complete bucket index max shards: 11 2023-10-18T19:46:14.266 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.263+0000 7fd8e3a69780 10 rgw notify: Started notification manager with: 1 workers 2023-10-18T19:46:14.266 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.263+0000 7fd8c21d8700 20 rgw notify: INFO: next queues processing will happen at: Wed Oct 18 19:46:44 2023 2023-10-18T19:46:14.266 INFO:teuthology.orchestra.run.smithi156.stderr: 2023-10-18T19:46:14.267 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.263+0000 7fd8c19b6700 20 reqs_thread_entry: start 2023-10-18T19:46:14.273 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.270+0000 7fd8c89e5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-18T19:46:14.275 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.272+0000 7fd8e3a69780 20 remove_watcher() i=0 2023-10-18T19:46:14.275 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.272+0000 7fd8e3a69780 2 removed watcher, disabling cache 2023-10-18T19:46:14.275 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.273+0000 7fd8e3a69780 20 remove_watcher() i=1 2023-10-18T19:46:14.276 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.274+0000 7fd8e3a69780 20 remove_watcher() i=2 2023-10-18T19:46:14.277 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.274+0000 7fd8e3a69780 20 remove_watcher() i=3 2023-10-18T19:46:14.278 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.275+0000 7fd8e3a69780 20 remove_watcher() i=4 2023-10-18T19:46:14.278 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.276+0000 7fd8e3a69780 20 remove_watcher() i=5 2023-10-18T19:46:14.279 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.277+0000 7fd8e3a69780 20 remove_watcher() i=6 2023-10-18T19:46:14.280 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.278+0000 7fd8e3a69780 20 remove_watcher() i=7 2023-10-18T19:46:14.282 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2023-10-18T19:46:14.284 DEBUG:tasks.util.rgw: json result: [] 2023-10-18T19:46:14.285 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-18T19:46:14.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-18T19:46:14.285 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-18T19:46:14.335 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:14.336 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:14.343 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.341+0000 7f8a5b44d780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-18T19:46:14.344 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.341+0000 7f8a5b44d780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-18T19:46:14.347 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:14.344+0000 7f8a3cbc2700 20 reqs_thread_entry: start 2023-10-18T19:46:14.353 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:46:14.353 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "02d207aa-78c1-4ca9-9ff1-0debece50378", 2023-10-18T19:46:14.353 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "default", 2023-10-18T19:46:14.353 INFO:teuthology.orchestra.run.smithi156.stdout: "domain_root": "default.rgw.meta:root", 2023-10-18T19:46:14.354 INFO:teuthology.orchestra.run.smithi156.stdout: "control_pool": "default.rgw.control", 2023-10-18T19:46:14.354 INFO:teuthology.orchestra.run.smithi156.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-18T19:46:14.354 INFO:teuthology.orchestra.run.smithi156.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-18T19:46:14.354 INFO:teuthology.orchestra.run.smithi156.stdout: "log_pool": "default.rgw.log", 2023-10-18T19:46:14.354 INFO:teuthology.orchestra.run.smithi156.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-18T19:46:14.355 INFO:teuthology.orchestra.run.smithi156.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-18T19:46:14.355 INFO:teuthology.orchestra.run.smithi156.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-18T19:46:14.355 INFO:teuthology.orchestra.run.smithi156.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-18T19:46:14.355 INFO:teuthology.orchestra.run.smithi156.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-18T19:46:14.355 INFO:teuthology.orchestra.run.smithi156.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-18T19:46:14.355 INFO:teuthology.orchestra.run.smithi156.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-18T19:46:14.356 INFO:teuthology.orchestra.run.smithi156.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-18T19:46:14.356 INFO:teuthology.orchestra.run.smithi156.stdout: "otp_pool": "default.rgw.otp", 2023-10-18T19:46:14.356 INFO:teuthology.orchestra.run.smithi156.stdout: "system_key": { 2023-10-18T19:46:14.356 INFO:teuthology.orchestra.run.smithi156.stdout: "access_key": "", 2023-10-18T19:46:14.356 INFO:teuthology.orchestra.run.smithi156.stdout: "secret_key": "" 2023-10-18T19:46:14.357 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:14.357 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_pools": [ 2023-10-18T19:46:14.357 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:14.359 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "default-placement", 2023-10-18T19:46:14.359 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:46:14.359 INFO:teuthology.orchestra.run.smithi156.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-18T19:46:14.360 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": { 2023-10-18T19:46:14.360 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD": { 2023-10-18T19:46:14.360 INFO:teuthology.orchestra.run.smithi156.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-18T19:46:14.360 INFO:teuthology.orchestra.run.smithi156.stdout: "compression_type": "random" 2023-10-18T19:46:14.360 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:14.361 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:14.361 INFO:teuthology.orchestra.run.smithi156.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-18T19:46:14.361 INFO:teuthology.orchestra.run.smithi156.stdout: "index_type": 0, 2023-10-18T19:46:14.361 INFO:teuthology.orchestra.run.smithi156.stdout: "inline_data": true 2023-10-18T19:46:14.362 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:14.362 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:14.362 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:14.362 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "", 2023-10-18T19:46:14.362 INFO:teuthology.orchestra.run.smithi156.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-18T19:46:14.363 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:46:14.364 DEBUG:tasks.util.rgw: json result: {'id': '02d207aa-78c1-4ca9-9ff1-0debece50378', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-18T19:46:14.364 INFO:tasks.rgw:Starting rgw... 2023-10-18T19:46:14.364 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-10-18T19:46:14.365 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-18T19:46:14.365 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-18T19:46:14.366 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-18T19:46:14.368 INFO:tasks.rgw.c1.client.0:Started 2023-10-18T19:46:14.369 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-10-18T19:46:14.369 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-18T19:46:14.369 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-18T19:46:14.370 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T19:46:14.372 INFO:tasks.rgw.c1.client.1:Started 2023-10-18T19:46:14.372 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-10-18T19:46:14.372 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-18T19:46:14.373 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-18T19:46:14.373 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-18T19:46:14.375 INFO:tasks.rgw.c2.client.0:Started 2023-10-18T19:46:14.375 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-10-18T19:46:14.376 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-18T19:46:14.376 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-18T19:46:14.376 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-18T19:46:14.378 INFO:tasks.rgw.c2.client.1:Started 2023-10-18T19:46:14.379 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi082.front.sepia.ceph.com:8000/ 2023-10-18T19:46:14.379 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8000/ 2023-10-18T19:46:14.452 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:46:14.453 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:46:14.455 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T19:46:14.455 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8000: Connection refused 2023-10-18T19:46:15.456 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8000/ 2023-10-18T19:46:15.479 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:46:15.479 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:46:15.479 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T19:46:15.481 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T19:46:15.482 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi082.front.sepia.ceph.com:8001/ 2023-10-18T19:46:15.482 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001/ 2023-10-18T19:46:15.543 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:46:15.543 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:46:15.544 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T19:46:15.545 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T19:46:15.545 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi156.front.sepia.ceph.com:8000/ 2023-10-18T19:46:15.546 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8000/ 2023-10-18T19:46:15.584 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:46:15.584 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:46:15.585 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T19:46:15.586 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2023-10-18T19:46:15.586 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi156.front.sepia.ceph.com:8001/ 2023-10-18T19:46:15.587 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001/ 2023-10-18T19:46:15.648 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:46:15.648 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:46:15.649 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T19:46:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2023-10-18T19:46:15.650 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-10-18T19:46:17.476 INFO:tasks.rgw_multisite:creating realm.. 2023-10-18T19:46:17.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:17.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:17.477 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:17.531 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:17.531 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:17.559 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:17.559 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:17.559 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "test-realm", 2023-10-18T19:46:17.559 INFO:teuthology.orchestra.run.smithi082.stdout: "current_period": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T19:46:17.560 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T19:46:17.560 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:17.561 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-10-18T19:46:17.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:17.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:17.562 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:17.612 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:17.612 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:17.625 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:17.623+0000 7f4141bd4780 -1 old format 2023-10-18T19:46:17.632 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:17.632 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T19:46:17.633 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T19:46:17.633 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T19:46:17.633 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T19:46:17.633 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T19:46:17.633 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T19:46:17.634 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:17.634 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T19:46:17.634 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T19:46:17.634 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "", 2023-10-18T19:46:17.634 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [], 2023-10-18T19:46:17.635 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T19:46:17.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:17.635 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T19:46:17.635 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T19:46:17.635 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [] 2023-10-18T19:46:17.636 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:17.636 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:17.636 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T19:46:17.636 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:17.636 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T19:46:17.636 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T19:46:17.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:17.637 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T19:46:17.637 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T19:46:17.637 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:17.637 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:17.638 INFO:tasks.rgw_multisite:creating master zone.. 2023-10-18T19:46:17.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--endpoints', 'http://smithi082.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:17.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--endpoints', 'http://smithi082.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:17.640 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key HD6X1XK4CX1C8MO8 --secret iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX --endpoints http://smithi082.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:17.734 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:17.734 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:17.748 INFO:teuthology.orchestra.run.smithi082.stderr:NOTICE: set zone's realm_id=7565323d-5a72-4e51-af8d-a123f7dfc36a 2023-10-18T19:46:17.766 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T19:46:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-18T19:46:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "a1.rgw.control", 2023-10-18T19:46:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-18T19:46:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-18T19:46:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "a1.rgw.log", 2023-10-18T19:46:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-18T19:46:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-18T19:46:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-18T19:46:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-18T19:46:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-18T19:46:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-18T19:46:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-18T19:46:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-18T19:46:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "a1.rgw.otp", 2023-10-18T19:46:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T19:46:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T19:46:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T19:46:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T19:46:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T19:46:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:46:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-18T19:46:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T19:46:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T19:46:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "a1.rgw.buckets.data" 2023-10-18T19:46:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-18T19:46:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T19:46:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T19:46:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-18T19:46:17.774 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:17.775 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-10-18T19:46:18.264 INFO:teuthology.orchestra.run.smithi082.stderr:pool 'a1.rgw.buckets.data' created 2023-10-18T19:46:18.281 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-10-18T19:46:19.294 INFO:teuthology.orchestra.run.smithi082.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-10-18T19:46:19.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:19.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:19.309 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:19.363 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:19.364 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:19.382 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:19.383 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:19.383 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T19:46:19.383 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-18T19:46:19.383 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "a1.rgw.control", 2023-10-18T19:46:19.383 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-18T19:46:19.384 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-18T19:46:19.384 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "a1.rgw.log", 2023-10-18T19:46:19.384 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-18T19:46:19.384 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-18T19:46:19.384 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-18T19:46:19.385 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-18T19:46:19.385 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-18T19:46:19.385 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-18T19:46:19.385 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-18T19:46:19.385 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-18T19:46:19.385 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "a1.rgw.otp", 2023-10-18T19:46:19.386 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T19:46:19.386 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T19:46:19.386 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T19:46:19.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:19.386 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T19:46:19.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:19.387 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T19:46:19.387 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:46:19.387 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-18T19:46:19.387 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T19:46:19.388 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T19:46:19.388 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "a1.rgw.buckets.data", 2023-10-18T19:46:19.388 INFO:teuthology.orchestra.run.smithi082.stdout: "compression_type": "random" 2023-10-18T19:46:19.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:19.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:19.389 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-18T19:46:19.389 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T19:46:19.389 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T19:46:19.390 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:19.390 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:19.390 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:19.390 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:19.390 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-18T19:46:19.391 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:19.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:19.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:19.393 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:19.441 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:19.441 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:19.452 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:19.449+0000 7f318df41780 0 period (aaf2721a-72fd-4d7e-add3-5a97d3603481 does not have zone 403a921b-fcfc-4ab2-b434-bf29d35a51b6 configured 2023-10-18T19:46:26.361 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:26.362 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:46:26.362 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1, 2023-10-18T19:46:26.362 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T19:46:26.362 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T19:46:26.362 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T19:46:26.363 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:46:26.363 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T19:46:26.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:26.363 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T19:46:26.363 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T19:46:26.364 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T19:46:26.364 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T19:46:26.364 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T19:46:26.364 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T19:46:26.364 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:26.365 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T19:46:26.365 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T19:46:26.365 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:26.365 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T19:46:26.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:26.366 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:26.366 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T19:46:26.366 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T19:46:26.366 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T19:46:26.366 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:26.367 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T19:46:26.367 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": false, 2023-10-18T19:46:26.367 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:46:26.367 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T19:46:26.367 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T19:46:26.367 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T19:46:26.368 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T19:46:26.368 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T19:46:26.368 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T19:46:26.368 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T19:46:26.368 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T19:46:26.369 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:26.369 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:26.369 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:26.369 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T19:46:26.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:26.370 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T19:46:26.370 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T19:46:26.370 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T19:46:26.370 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T19:46:26.370 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:26.371 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:26.371 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:26.371 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T19:46:26.371 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:26.371 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T19:46:26.371 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T19:46:26.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:26.372 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T19:46:26.372 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T19:46:26.372 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:26.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:26.373 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:26.373 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T19:46:26.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:26.373 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:26.373 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T19:46:26.374 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:26.374 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:26.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:26.374 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T19:46:26.374 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:26.375 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T19:46:26.375 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T19:46:26.375 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T19:46:26.375 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T19:46:26.375 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T19:46:26.375 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T19:46:26.376 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T19:46:26.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:26.376 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T19:46:26.376 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T19:46:26.376 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T19:46:26.377 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T19:46:26.377 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T19:46:26.377 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T19:46:26.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:26.377 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T19:46:26.377 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T19:46:26.378 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T19:46:26.378 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T19:46:26.378 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T19:46:26.378 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T19:46:26.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:26.379 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T19:46:26.379 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T19:46:26.379 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T19:46:26.379 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T19:46:26.379 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T19:46:26.379 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T19:46:26.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:26.380 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T19:46:26.380 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T19:46:26.380 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T19:46:26.380 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T19:46:26.381 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T19:46:26.381 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T19:46:26.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:26.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:26.381 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:26.382 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T19:46:26.382 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:46:26.382 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:26.383 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-18T19:46:26.384 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T19:46:26.385 INFO:tasks.rgw.c1.client.0.smithi082.stdout:2023-10-18T19:46:26.381+0000 7f313a8de700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 120431) UID: 0 2023-10-18T19:46:26.385 INFO:tasks.rgw.c1.client.0.smithi082.stdout:2023-10-18T19:46:26.381+0000 7f313eb477c0 -1 shutting down 2023-10-18T19:46:29.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-18T19:46:32.485 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-18T19:46:32.486 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-18T19:46:32.489 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-18T19:46:32.493 INFO:tasks.rgw.c1.client.0:Started 2023-10-18T19:46:32.493 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8000 2023-10-18T19:46:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T19:46:32.521 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:46:32.522 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:46:32.522 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8000: Connection refused 2023-10-18T19:46:33.523 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8000 2023-10-18T19:46:33.545 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:46:33.546 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:46:33.547 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T19:46:33.548 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T19:46:33.548 INFO:tasks.rgw_multisite:creating admin user.. 2023-10-18T19:46:33.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:33.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:33.549 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key HD6X1XK4CX1C8MO8 --secret iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:33.643 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:33.644 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:33.674 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:33.674 INFO:teuthology.orchestra.run.smithi082.stdout: "user_id": "realm-admin", 2023-10-18T19:46:33.674 INFO:teuthology.orchestra.run.smithi082.stdout: "display_name": "Realm Admin", 2023-10-18T19:46:33.674 INFO:teuthology.orchestra.run.smithi082.stdout: "email": "", 2023-10-18T19:46:33.675 INFO:teuthology.orchestra.run.smithi082.stdout: "suspended": 0, 2023-10-18T19:46:33.675 INFO:teuthology.orchestra.run.smithi082.stdout: "max_buckets": 1000, 2023-10-18T19:46:33.675 INFO:teuthology.orchestra.run.smithi082.stdout: "subusers": [], 2023-10-18T19:46:33.675 INFO:teuthology.orchestra.run.smithi082.stdout: "keys": [ 2023-10-18T19:46:33.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:33.676 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "realm-admin", 2023-10-18T19:46:33.676 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T19:46:33.676 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T19:46:33.676 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:33.676 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:33.677 INFO:teuthology.orchestra.run.smithi082.stdout: "swift_keys": [], 2023-10-18T19:46:33.677 INFO:teuthology.orchestra.run.smithi082.stdout: "caps": [], 2023-10-18T19:46:33.677 INFO:teuthology.orchestra.run.smithi082.stdout: "op_mask": "read, write, delete", 2023-10-18T19:46:33.677 INFO:teuthology.orchestra.run.smithi082.stdout: "system": true, 2023-10-18T19:46:33.677 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "", 2023-10-18T19:46:33.678 INFO:teuthology.orchestra.run.smithi082.stdout: "default_storage_class": "", 2023-10-18T19:46:33.678 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_tags": [], 2023-10-18T19:46:33.678 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T19:46:33.678 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T19:46:33.678 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T19:46:33.679 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T19:46:33.679 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T19:46:33.679 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T19:46:33.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:33.679 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T19:46:33.679 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T19:46:33.680 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T19:46:33.680 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T19:46:33.680 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T19:46:33.680 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T19:46:33.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:33.681 INFO:teuthology.orchestra.run.smithi082.stdout: "temp_url_keys": [], 2023-10-18T19:46:33.681 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "rgw", 2023-10-18T19:46:33.681 INFO:teuthology.orchestra.run.smithi082.stdout: "mfa_ids": [] 2023-10-18T19:46:33.681 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:33.681 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:46:33.685 INFO:tasks.rgw_multisite:creating zone.. 2023-10-18T19:46:33.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--endpoints', 'http://smithi082.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:33.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--endpoints', 'http://smithi082.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:33.686 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key HD6X1XK4CX1C8MO8 --secret iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX --endpoints http://smithi082.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:33.738 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:33.739 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:33.752 INFO:teuthology.orchestra.run.smithi082.stderr:NOTICE: set zone's realm_id=7565323d-5a72-4e51-af8d-a123f7dfc36a 2023-10-18T19:46:33.766 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T19:46:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T19:46:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-18T19:46:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "a2.rgw.control", 2023-10-18T19:46:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-18T19:46:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-18T19:46:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "a2.rgw.log", 2023-10-18T19:46:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-18T19:46:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-18T19:46:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-18T19:46:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-18T19:46:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-18T19:46:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-18T19:46:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-18T19:46:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-18T19:46:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "a2.rgw.otp", 2023-10-18T19:46:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T19:46:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T19:46:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T19:46:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T19:46:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T19:46:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:46:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-18T19:46:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T19:46:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T19:46:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "a2.rgw.buckets.data" 2023-10-18T19:46:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-18T19:46:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T19:46:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T19:46:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-18T19:46:33.773 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:33.774 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-10-18T19:46:34.260 INFO:teuthology.orchestra.run.smithi082.stderr:pool 'a2.rgw.buckets.data' created 2023-10-18T19:46:34.276 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-10-18T19:46:35.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-18T19:46:35.298 INFO:teuthology.orchestra.run.smithi082.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-10-18T19:46:35.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:35.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:35.317 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:35.370 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:35.370 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:35.388 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:35.388 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T19:46:35.388 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T19:46:35.389 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-18T19:46:35.389 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "a2.rgw.control", 2023-10-18T19:46:35.389 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-18T19:46:35.389 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-18T19:46:35.389 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "a2.rgw.log", 2023-10-18T19:46:35.390 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-18T19:46:35.390 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-18T19:46:35.390 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-18T19:46:35.390 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-18T19:46:35.390 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-18T19:46:35.391 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-18T19:46:35.391 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-18T19:46:35.391 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-18T19:46:35.391 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "a2.rgw.otp", 2023-10-18T19:46:35.391 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T19:46:35.392 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T19:46:35.392 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T19:46:35.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:35.392 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T19:46:35.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:35.393 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T19:46:35.393 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:46:35.393 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-18T19:46:35.393 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T19:46:35.393 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T19:46:35.394 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-18T19:46:35.394 INFO:teuthology.orchestra.run.smithi082.stdout: "compression_type": "random" 2023-10-18T19:46:35.394 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:35.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:35.394 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-18T19:46:35.395 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T19:46:35.395 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T19:46:35.395 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:35.395 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:35.395 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:35.396 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:35.396 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-18T19:46:35.396 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:35.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:35.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:35.398 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:35.445 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:35.446 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:35.457 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:46:35.455+0000 7f3a5d408780 0 period (67bf1060-aa43-46d9-8036-efd54ef55d7d does not have zone 419020ed-d3b2-4ce0-b36c-97abb284c459 configured 2023-10-18T19:46:42.326 INFO:teuthology.orchestra.run.smithi082.stderr:Sending period to new master zone 403a921b-fcfc-4ab2-b434-bf29d35a51b6 2023-10-18T19:46:42.349 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:46:42.350 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:46:42.350 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2, 2023-10-18T19:46:42.350 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T19:46:42.350 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T19:46:42.351 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T19:46:42.351 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:46:42.351 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T19:46:42.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:42.351 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T19:46:42.352 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T19:46:42.352 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T19:46:42.352 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T19:46:42.352 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T19:46:42.352 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T19:46:42.353 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:42.353 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T19:46:42.353 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T19:46:42.353 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:42.353 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T19:46:42.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:42.354 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:42.354 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T19:46:42.354 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T19:46:42.354 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T19:46:42.354 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:42.355 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T19:46:42.355 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T19:46:42.355 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:46:42.355 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T19:46:42.355 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T19:46:42.356 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T19:46:42.356 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T19:46:42.356 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T19:46:42.356 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T19:46:42.356 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T19:46:42.357 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T19:46:42.357 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:42.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:42.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:42.357 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T19:46:42.357 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T19:46:42.358 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T19:46:42.358 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T19:46:42.358 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:42.358 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T19:46:42.358 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T19:46:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:46:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T19:46:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T19:46:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T19:46:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T19:46:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T19:46:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T19:46:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T19:46:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T19:46:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T19:46:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T19:46:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T19:46:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T19:46:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T19:46:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T19:46:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T19:46:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T19:46:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T19:46:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T19:46:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:46:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T19:46:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T19:46:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:46:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T19:46:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T19:46:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:46:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T19:46:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T19:46:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T19:46:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T19:46:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T19:46:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T19:46:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T19:46:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T19:46:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T19:46:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T19:46:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T19:46:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T19:46:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T19:46:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T19:46:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T19:46:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T19:46:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T19:46:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T19:46:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T19:46:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T19:46:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T19:46:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T19:46:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T19:46:42.372 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T19:46:42.372 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T19:46:42.373 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T19:46:42.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:42.373 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T19:46:42.373 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T19:46:42.374 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T19:46:42.374 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T19:46:42.374 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T19:46:42.374 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T19:46:42.374 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:46:42.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:46:42.375 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:42.375 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T19:46:42.375 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:46:42.375 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:46:42.376 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-18T19:46:42.376 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T19:46:42.377 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T19:46:42.375+0000 7eff46016700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 120437) UID: 0 2023-10-18T19:46:42.377 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T19:46:42.375+0000 7eff4a27f7c0 -1 shutting down 2023-10-18T19:46:46.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T19:46:48.477 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-18T19:46:48.478 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-18T19:46:48.478 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T19:46:48.481 INFO:tasks.rgw.c1.client.1:Started 2023-10-18T19:46:48.481 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T19:46:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T19:46:48.507 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:46:48.507 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:46:48.508 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T19:46:49.509 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T19:46:49.532 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:46:49.532 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:46:49.532 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T19:46:49.533 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T19:46:49.534 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-18T19:46:49.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi082.front.sepia.ceph.com:8000', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:49.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi082.front.sepia.ceph.com:8000', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:49.535 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi082.front.sepia.ceph.com:8000 --access-key HD6X1XK4CX1C8MO8 --secret iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:49.594 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:49.595 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:49.629 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:46:49.630 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:49.630 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "test-realm", 2023-10-18T19:46:49.630 INFO:teuthology.orchestra.run.smithi156.stdout: "current_period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:46:49.630 INFO:teuthology.orchestra.run.smithi156.stdout: "epoch": 2 2023-10-18T19:46:49.631 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:46:49.632 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-10-18T19:46:49.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:49.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:49.633 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:49.681 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:49.682 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:49.696 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:49.694+0000 7f3e7783b780 -1 old format 2023-10-18T19:46:49.702 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:46:49.702 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "b", 2023-10-18T19:46:49.702 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b", 2023-10-18T19:46:49.702 INFO:teuthology.orchestra.run.smithi156.stdout: "api_name": "", 2023-10-18T19:46:49.703 INFO:teuthology.orchestra.run.smithi156.stdout: "is_master": false, 2023-10-18T19:46:49.703 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:46:49.703 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T19:46:49.703 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:49.703 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames": [], 2023-10-18T19:46:49.704 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames_s3website": [], 2023-10-18T19:46:49.704 INFO:teuthology.orchestra.run.smithi156.stdout: "master_zone": "", 2023-10-18T19:46:49.704 INFO:teuthology.orchestra.run.smithi156.stdout: "zones": [], 2023-10-18T19:46:49.704 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_targets": [ 2023-10-18T19:46:49.704 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:49.705 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "default-placement", 2023-10-18T19:46:49.705 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2023-10-18T19:46:49.705 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": [] 2023-10-18T19:46:49.705 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:49.705 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:49.706 INFO:teuthology.orchestra.run.smithi156.stdout: "default_placement": "default-placement", 2023-10-18T19:46:49.706 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:49.706 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_policy": { 2023-10-18T19:46:49.706 INFO:teuthology.orchestra.run.smithi156.stdout: "groups": [] 2023-10-18T19:46:49.706 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:49.707 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled_features": [ 2023-10-18T19:46:49.707 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:46:49.707 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:46:49.707 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:46:49.708 INFO:tasks.rgw_multisite:creating zone.. 2023-10-18T19:46:49.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--endpoints', 'http://smithi156.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:49.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--endpoints', 'http://smithi156.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:49.709 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key HD6X1XK4CX1C8MO8 --secret iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX --endpoints http://smithi156.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:49.804 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:49.804 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:49.817 INFO:teuthology.orchestra.run.smithi156.stderr:NOTICE: set zone's realm_id=7565323d-5a72-4e51-af8d-a123f7dfc36a 2023-10-18T19:46:49.832 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:46:49.832 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T19:46:49.832 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b1", 2023-10-18T19:46:49.832 INFO:teuthology.orchestra.run.smithi156.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-18T19:46:49.833 INFO:teuthology.orchestra.run.smithi156.stdout: "control_pool": "b1.rgw.control", 2023-10-18T19:46:49.833 INFO:teuthology.orchestra.run.smithi156.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-18T19:46:49.833 INFO:teuthology.orchestra.run.smithi156.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-18T19:46:49.833 INFO:teuthology.orchestra.run.smithi156.stdout: "log_pool": "b1.rgw.log", 2023-10-18T19:46:49.833 INFO:teuthology.orchestra.run.smithi156.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-18T19:46:49.834 INFO:teuthology.orchestra.run.smithi156.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-18T19:46:49.834 INFO:teuthology.orchestra.run.smithi156.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-18T19:46:49.834 INFO:teuthology.orchestra.run.smithi156.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-18T19:46:49.834 INFO:teuthology.orchestra.run.smithi156.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-18T19:46:49.834 INFO:teuthology.orchestra.run.smithi156.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-18T19:46:49.835 INFO:teuthology.orchestra.run.smithi156.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-18T19:46:49.835 INFO:teuthology.orchestra.run.smithi156.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-18T19:46:49.835 INFO:teuthology.orchestra.run.smithi156.stdout: "otp_pool": "b1.rgw.otp", 2023-10-18T19:46:49.835 INFO:teuthology.orchestra.run.smithi156.stdout: "system_key": { 2023-10-18T19:46:49.835 INFO:teuthology.orchestra.run.smithi156.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T19:46:49.836 INFO:teuthology.orchestra.run.smithi156.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T19:46:49.836 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:49.836 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_pools": [ 2023-10-18T19:46:49.836 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:49.836 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "default-placement", 2023-10-18T19:46:49.837 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:46:49.837 INFO:teuthology.orchestra.run.smithi156.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-18T19:46:49.837 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": { 2023-10-18T19:46:49.837 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD": { 2023-10-18T19:46:49.837 INFO:teuthology.orchestra.run.smithi156.stdout: "data_pool": "b1.rgw.buckets.data" 2023-10-18T19:46:49.837 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:49.838 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:49.838 INFO:teuthology.orchestra.run.smithi156.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-18T19:46:49.838 INFO:teuthology.orchestra.run.smithi156.stdout: "index_type": 0, 2023-10-18T19:46:49.838 INFO:teuthology.orchestra.run.smithi156.stdout: "inline_data": true 2023-10-18T19:46:49.838 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:49.839 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:49.839 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:49.839 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:49.839 INFO:teuthology.orchestra.run.smithi156.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-18T19:46:49.839 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:46:49.840 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-10-18T19:46:51.082 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'b1.rgw.buckets.data' created 2023-10-18T19:46:51.102 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-10-18T19:46:52.095 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-10-18T19:46:52.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:52.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:52.115 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:52.169 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:52.170 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:52.188 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:46:52.188 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T19:46:52.188 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b1", 2023-10-18T19:46:52.188 INFO:teuthology.orchestra.run.smithi156.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-18T19:46:52.189 INFO:teuthology.orchestra.run.smithi156.stdout: "control_pool": "b1.rgw.control", 2023-10-18T19:46:52.189 INFO:teuthology.orchestra.run.smithi156.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-18T19:46:52.189 INFO:teuthology.orchestra.run.smithi156.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-18T19:46:52.189 INFO:teuthology.orchestra.run.smithi156.stdout: "log_pool": "b1.rgw.log", 2023-10-18T19:46:52.189 INFO:teuthology.orchestra.run.smithi156.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-18T19:46:52.189 INFO:teuthology.orchestra.run.smithi156.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-18T19:46:52.190 INFO:teuthology.orchestra.run.smithi156.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-18T19:46:52.190 INFO:teuthology.orchestra.run.smithi156.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-18T19:46:52.190 INFO:teuthology.orchestra.run.smithi156.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-18T19:46:52.190 INFO:teuthology.orchestra.run.smithi156.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-18T19:46:52.190 INFO:teuthology.orchestra.run.smithi156.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-18T19:46:52.191 INFO:teuthology.orchestra.run.smithi156.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-18T19:46:52.191 INFO:teuthology.orchestra.run.smithi156.stdout: "otp_pool": "b1.rgw.otp", 2023-10-18T19:46:52.191 INFO:teuthology.orchestra.run.smithi156.stdout: "system_key": { 2023-10-18T19:46:52.191 INFO:teuthology.orchestra.run.smithi156.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T19:46:52.192 INFO:teuthology.orchestra.run.smithi156.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T19:46:52.192 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:52.193 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_pools": [ 2023-10-18T19:46:52.193 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:52.193 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "default-placement", 2023-10-18T19:46:52.194 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:46:52.194 INFO:teuthology.orchestra.run.smithi156.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-18T19:46:52.194 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": { 2023-10-18T19:46:52.194 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD": { 2023-10-18T19:46:52.194 INFO:teuthology.orchestra.run.smithi156.stdout: "data_pool": "b1.rgw.buckets.data", 2023-10-18T19:46:52.195 INFO:teuthology.orchestra.run.smithi156.stdout: "compression_type": "random" 2023-10-18T19:46:52.195 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:52.195 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:52.195 INFO:teuthology.orchestra.run.smithi156.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-18T19:46:52.195 INFO:teuthology.orchestra.run.smithi156.stdout: "index_type": 0, 2023-10-18T19:46:52.196 INFO:teuthology.orchestra.run.smithi156.stdout: "inline_data": true 2023-10-18T19:46:52.196 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:52.196 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:52.196 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:52.196 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:52.196 INFO:teuthology.orchestra.run.smithi156.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-18T19:46:52.197 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:46:52.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:52.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:46:52.199 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-18T19:46:52.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-18T19:46:52.247 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:46:52.248 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:46:52.258 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:46:52.257+0000 7f02b0c93780 0 period (67bf1060-aa43-46d9-8036-efd54ef55d7d does not have zone 576f1485-1a9f-4b57-9613-5ba13a3f0a36 configured 2023-10-18T19:46:59.133 INFO:teuthology.orchestra.run.smithi156.stderr:Sending period to new master zone 403a921b-fcfc-4ab2-b434-bf29d35a51b6 2023-10-18T19:46:59.161 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:46:59.162 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:46:59.162 INFO:teuthology.orchestra.run.smithi156.stdout: "epoch": 3, 2023-10-18T19:46:59.162 INFO:teuthology.orchestra.run.smithi156.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T19:46:59.162 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_status": [], 2023-10-18T19:46:59.162 INFO:teuthology.orchestra.run.smithi156.stdout: "period_map": { 2023-10-18T19:46:59.163 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:46:59.163 INFO:teuthology.orchestra.run.smithi156.stdout: "zonegroups": [ 2023-10-18T19:46:59.163 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.163 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "a", 2023-10-18T19:46:59.163 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "a", 2023-10-18T19:46:59.164 INFO:teuthology.orchestra.run.smithi156.stdout: "api_name": "", 2023-10-18T19:46:59.164 INFO:teuthology.orchestra.run.smithi156.stdout: "is_master": true, 2023-10-18T19:46:59.164 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:46:59.164 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T19:46:59.164 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.165 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames": [], 2023-10-18T19:46:59.165 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames_s3website": [], 2023-10-18T19:46:59.165 INFO:teuthology.orchestra.run.smithi156.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:59.165 INFO:teuthology.orchestra.run.smithi156.stdout: "zones": [ 2023-10-18T19:46:59.165 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.166 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:59.166 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "a1", 2023-10-18T19:46:59.166 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:46:59.166 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T19:46:59.167 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.167 INFO:teuthology.orchestra.run.smithi156.stdout: "log_meta": false, 2023-10-18T19:46:59.167 INFO:teuthology.orchestra.run.smithi156.stdout: "log_data": true, 2023-10-18T19:46:59.167 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:46:59.167 INFO:teuthology.orchestra.run.smithi156.stdout: "read_only": false, 2023-10-18T19:46:59.168 INFO:teuthology.orchestra.run.smithi156.stdout: "tier_type": "", 2023-10-18T19:46:59.168 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from_all": true, 2023-10-18T19:46:59.168 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from": [], 2023-10-18T19:46:59.168 INFO:teuthology.orchestra.run.smithi156.stdout: "redirect_zone": "", 2023-10-18T19:46:59.168 INFO:teuthology.orchestra.run.smithi156.stdout: "supported_features": [ 2023-10-18T19:46:59.168 INFO:teuthology.orchestra.run.smithi156.stdout: "compress-encrypted", 2023-10-18T19:46:59.169 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:46:59.169 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:46:59.169 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.169 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.169 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T19:46:59.170 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "a2", 2023-10-18T19:46:59.170 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:46:59.170 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T19:46:59.170 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.171 INFO:teuthology.orchestra.run.smithi156.stdout: "log_meta": false, 2023-10-18T19:46:59.171 INFO:teuthology.orchestra.run.smithi156.stdout: "log_data": true, 2023-10-18T19:46:59.171 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:46:59.171 INFO:teuthology.orchestra.run.smithi156.stdout: "read_only": false, 2023-10-18T19:46:59.171 INFO:teuthology.orchestra.run.smithi156.stdout: "tier_type": "", 2023-10-18T19:46:59.172 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from_all": true, 2023-10-18T19:46:59.172 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from": [], 2023-10-18T19:46:59.172 INFO:teuthology.orchestra.run.smithi156.stdout: "redirect_zone": "", 2023-10-18T19:46:59.172 INFO:teuthology.orchestra.run.smithi156.stdout: "supported_features": [ 2023-10-18T19:46:59.172 INFO:teuthology.orchestra.run.smithi156.stdout: "compress-encrypted", 2023-10-18T19:46:59.173 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:46:59.173 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:46:59.173 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:59.173 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.173 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_targets": [ 2023-10-18T19:46:59.174 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.174 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "default-placement", 2023-10-18T19:46:59.174 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2023-10-18T19:46:59.174 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": [ 2023-10-18T19:46:59.174 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD" 2023-10-18T19:46:59.174 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:46:59.175 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:59.175 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.175 INFO:teuthology.orchestra.run.smithi156.stdout: "default_placement": "default-placement", 2023-10-18T19:46:59.175 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:59.175 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_policy": { 2023-10-18T19:46:59.176 INFO:teuthology.orchestra.run.smithi156.stdout: "groups": [] 2023-10-18T19:46:59.176 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.176 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled_features": [ 2023-10-18T19:46:59.176 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:46:59.176 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:46:59.177 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.177 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.177 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "b", 2023-10-18T19:46:59.177 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b", 2023-10-18T19:46:59.177 INFO:teuthology.orchestra.run.smithi156.stdout: "api_name": "", 2023-10-18T19:46:59.177 INFO:teuthology.orchestra.run.smithi156.stdout: "is_master": false, 2023-10-18T19:46:59.178 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:46:59.178 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T19:46:59.178 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.178 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames": [], 2023-10-18T19:46:59.179 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames_s3website": [], 2023-10-18T19:46:59.179 INFO:teuthology.orchestra.run.smithi156.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T19:46:59.179 INFO:teuthology.orchestra.run.smithi156.stdout: "zones": [ 2023-10-18T19:46:59.179 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.179 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T19:46:59.180 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b1", 2023-10-18T19:46:59.180 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:46:59.181 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T19:46:59.181 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.181 INFO:teuthology.orchestra.run.smithi156.stdout: "log_meta": false, 2023-10-18T19:46:59.181 INFO:teuthology.orchestra.run.smithi156.stdout: "log_data": false, 2023-10-18T19:46:59.182 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:46:59.182 INFO:teuthology.orchestra.run.smithi156.stdout: "read_only": false, 2023-10-18T19:46:59.182 INFO:teuthology.orchestra.run.smithi156.stdout: "tier_type": "", 2023-10-18T19:46:59.182 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from_all": true, 2023-10-18T19:46:59.182 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from": [], 2023-10-18T19:46:59.183 INFO:teuthology.orchestra.run.smithi156.stdout: "redirect_zone": "", 2023-10-18T19:46:59.183 INFO:teuthology.orchestra.run.smithi156.stdout: "supported_features": [ 2023-10-18T19:46:59.183 INFO:teuthology.orchestra.run.smithi156.stdout: "compress-encrypted", 2023-10-18T19:46:59.184 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:46:59.184 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:46:59.185 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:59.185 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.185 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_targets": [ 2023-10-18T19:46:59.185 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.186 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "default-placement", 2023-10-18T19:46:59.186 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2023-10-18T19:46:59.186 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": [ 2023-10-18T19:46:59.186 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD" 2023-10-18T19:46:59.186 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:46:59.187 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:59.187 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.187 INFO:teuthology.orchestra.run.smithi156.stdout: "default_placement": "default-placement", 2023-10-18T19:46:59.187 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:59.187 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_policy": { 2023-10-18T19:46:59.187 INFO:teuthology.orchestra.run.smithi156.stdout: "groups": [] 2023-10-18T19:46:59.188 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.188 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled_features": [ 2023-10-18T19:46:59.188 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:46:59.188 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:46:59.188 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:59.189 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:46:59.189 INFO:teuthology.orchestra.run.smithi156.stdout: "short_zone_ids": [ 2023-10-18T19:46:59.189 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.189 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:59.189 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 3791896933 2023-10-18T19:46:59.190 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.190 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.190 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T19:46:59.190 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 2277887411 2023-10-18T19:46:59.190 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.191 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:46:59.191 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T19:46:59.191 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 953824181 2023-10-18T19:46:59.191 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:59.192 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:46:59.192 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.192 INFO:teuthology.orchestra.run.smithi156.stdout: "master_zonegroup": "a", 2023-10-18T19:46:59.192 INFO:teuthology.orchestra.run.smithi156.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:46:59.192 INFO:teuthology.orchestra.run.smithi156.stdout: "period_config": { 2023-10-18T19:46:59.193 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_quota": { 2023-10-18T19:46:59.193 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2023-10-18T19:46:59.193 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2023-10-18T19:46:59.193 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2023-10-18T19:46:59.193 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2023-10-18T19:46:59.194 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2023-10-18T19:46:59.194 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.194 INFO:teuthology.orchestra.run.smithi156.stdout: "user_quota": { 2023-10-18T19:46:59.194 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2023-10-18T19:46:59.194 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2023-10-18T19:46:59.195 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2023-10-18T19:46:59.195 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2023-10-18T19:46:59.195 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2023-10-18T19:46:59.195 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.195 INFO:teuthology.orchestra.run.smithi156.stdout: "user_ratelimit": { 2023-10-18T19:46:59.195 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_ops": 0, 2023-10-18T19:46:59.196 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_ops": 0, 2023-10-18T19:46:59.196 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_bytes": 0, 2023-10-18T19:46:59.196 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_bytes": 0, 2023-10-18T19:46:59.196 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false 2023-10-18T19:46:59.197 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.197 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_ratelimit": { 2023-10-18T19:46:59.197 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_ops": 0, 2023-10-18T19:46:59.197 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_ops": 0, 2023-10-18T19:46:59.197 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_bytes": 0, 2023-10-18T19:46:59.198 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_bytes": 0, 2023-10-18T19:46:59.198 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false 2023-10-18T19:46:59.198 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.198 INFO:teuthology.orchestra.run.smithi156.stdout: "anonymous_ratelimit": { 2023-10-18T19:46:59.198 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_ops": 0, 2023-10-18T19:46:59.199 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_ops": 0, 2023-10-18T19:46:59.199 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_bytes": 0, 2023-10-18T19:46:59.199 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_bytes": 0, 2023-10-18T19:46:59.199 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false 2023-10-18T19:46:59.199 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:46:59.200 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:46:59.200 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:46:59.200 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_name": "test-realm", 2023-10-18T19:46:59.200 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:46:59.200 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:46:59.201 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-18T19:46:59.202 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T19:46:59.203 INFO:tasks.rgw.c2.client.0.smithi156.stdout:2023-10-18T19:46:59.200+0000 7f89be121700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 119553) UID: 0 2023-10-18T19:46:59.203 INFO:tasks.rgw.c2.client.0.smithi156.stdout:2023-10-18T19:46:59.200+0000 7f89c238a7c0 -1 shutting down 2023-10-18T19:47:05.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-18T19:47:05.303 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-18T19:47:05.304 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-18T19:47:05.304 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-18T19:47:05.307 INFO:tasks.rgw.c2.client.0:Started 2023-10-18T19:47:05.307 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8000 2023-10-18T19:47:05.332 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:47:05.333 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:47:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T19:47:05.334 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi156.front.sepia.ceph.com port 8000: Connection refused 2023-10-18T19:47:06.335 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8000 2023-10-18T19:47:06.359 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:47:06.360 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:47:06.360 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T19:47:06.361 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2023-10-18T19:47:06.362 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-18T19:47:06.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi082.front.sepia.ceph.com:8000', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--url', 'http://smithi082.front.sepia.ceph.com:8000', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:06.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi082.front.sepia.ceph.com:8000', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--url', 'http://smithi082.front.sepia.ceph.com:8000', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:06.363 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi082.front.sepia.ceph.com:8000 --access-key HD6X1XK4CX1C8MO8 --secret iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX --url http://smithi082.front.sepia.ceph.com:8000 --access-key HD6X1XK4CX1C8MO8 --secret iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-18T19:47:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:06.492 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:47:06.493 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:47:06.493 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "test-realm", 2023-10-18T19:47:06.493 INFO:teuthology.orchestra.run.smithi156.stdout: "current_period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:06.493 INFO:teuthology.orchestra.run.smithi156.stdout: "epoch": 2 2023-10-18T19:47:06.493 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:47:06.494 INFO:tasks.rgw_multisite:creating zone.. 2023-10-18T19:47:06.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--endpoints', 'http://smithi156.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:06.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--endpoints', 'http://smithi156.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:06.496 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key HD6X1XK4CX1C8MO8 --secret iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX --endpoints http://smithi156.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-18T19:47:06.549 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:06.549 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:06.563 INFO:teuthology.orchestra.run.smithi156.stderr:NOTICE: set zone's realm_id=7565323d-5a72-4e51-af8d-a123f7dfc36a 2023-10-18T19:47:06.577 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:47:06.577 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T19:47:06.577 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b2", 2023-10-18T19:47:06.577 INFO:teuthology.orchestra.run.smithi156.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-18T19:47:06.578 INFO:teuthology.orchestra.run.smithi156.stdout: "control_pool": "b2.rgw.control", 2023-10-18T19:47:06.578 INFO:teuthology.orchestra.run.smithi156.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-18T19:47:06.578 INFO:teuthology.orchestra.run.smithi156.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-18T19:47:06.578 INFO:teuthology.orchestra.run.smithi156.stdout: "log_pool": "b2.rgw.log", 2023-10-18T19:47:06.578 INFO:teuthology.orchestra.run.smithi156.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-18T19:47:06.579 INFO:teuthology.orchestra.run.smithi156.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-18T19:47:06.579 INFO:teuthology.orchestra.run.smithi156.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-18T19:47:06.579 INFO:teuthology.orchestra.run.smithi156.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-18T19:47:06.579 INFO:teuthology.orchestra.run.smithi156.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-18T19:47:06.579 INFO:teuthology.orchestra.run.smithi156.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-18T19:47:06.580 INFO:teuthology.orchestra.run.smithi156.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-18T19:47:06.580 INFO:teuthology.orchestra.run.smithi156.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-18T19:47:06.580 INFO:teuthology.orchestra.run.smithi156.stdout: "otp_pool": "b2.rgw.otp", 2023-10-18T19:47:06.580 INFO:teuthology.orchestra.run.smithi156.stdout: "system_key": { 2023-10-18T19:47:06.580 INFO:teuthology.orchestra.run.smithi156.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T19:47:06.580 INFO:teuthology.orchestra.run.smithi156.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T19:47:06.581 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:06.581 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_pools": [ 2023-10-18T19:47:06.581 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:06.581 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "default-placement", 2023-10-18T19:47:06.581 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:47:06.582 INFO:teuthology.orchestra.run.smithi156.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-18T19:47:06.582 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": { 2023-10-18T19:47:06.582 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD": { 2023-10-18T19:47:06.582 INFO:teuthology.orchestra.run.smithi156.stdout: "data_pool": "b2.rgw.buckets.data" 2023-10-18T19:47:06.583 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:06.583 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:06.583 INFO:teuthology.orchestra.run.smithi156.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-18T19:47:06.583 INFO:teuthology.orchestra.run.smithi156.stdout: "index_type": 0, 2023-10-18T19:47:06.583 INFO:teuthology.orchestra.run.smithi156.stdout: "inline_data": true 2023-10-18T19:47:06.584 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:06.584 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:06.584 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:06.584 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:47:06.584 INFO:teuthology.orchestra.run.smithi156.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-18T19:47:06.585 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:47:06.586 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-10-18T19:47:07.121 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'b2.rgw.buckets.data' created 2023-10-18T19:47:07.135 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-10-18T19:47:08.132 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-10-18T19:47:08.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:08.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:08.154 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-18T19:47:08.207 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:08.208 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:08.228 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:47:08.228 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T19:47:08.228 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b2", 2023-10-18T19:47:08.229 INFO:teuthology.orchestra.run.smithi156.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-18T19:47:08.229 INFO:teuthology.orchestra.run.smithi156.stdout: "control_pool": "b2.rgw.control", 2023-10-18T19:47:08.229 INFO:teuthology.orchestra.run.smithi156.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-18T19:47:08.229 INFO:teuthology.orchestra.run.smithi156.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-18T19:47:08.229 INFO:teuthology.orchestra.run.smithi156.stdout: "log_pool": "b2.rgw.log", 2023-10-18T19:47:08.230 INFO:teuthology.orchestra.run.smithi156.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-18T19:47:08.230 INFO:teuthology.orchestra.run.smithi156.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-18T19:47:08.230 INFO:teuthology.orchestra.run.smithi156.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-18T19:47:08.230 INFO:teuthology.orchestra.run.smithi156.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-18T19:47:08.230 INFO:teuthology.orchestra.run.smithi156.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-18T19:47:08.231 INFO:teuthology.orchestra.run.smithi156.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-18T19:47:08.231 INFO:teuthology.orchestra.run.smithi156.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-18T19:47:08.231 INFO:teuthology.orchestra.run.smithi156.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-18T19:47:08.231 INFO:teuthology.orchestra.run.smithi156.stdout: "otp_pool": "b2.rgw.otp", 2023-10-18T19:47:08.231 INFO:teuthology.orchestra.run.smithi156.stdout: "system_key": { 2023-10-18T19:47:08.232 INFO:teuthology.orchestra.run.smithi156.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T19:47:08.232 INFO:teuthology.orchestra.run.smithi156.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T19:47:08.232 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:08.232 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_pools": [ 2023-10-18T19:47:08.232 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:08.233 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "default-placement", 2023-10-18T19:47:08.233 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:47:08.233 INFO:teuthology.orchestra.run.smithi156.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-18T19:47:08.233 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": { 2023-10-18T19:47:08.234 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD": { 2023-10-18T19:47:08.234 INFO:teuthology.orchestra.run.smithi156.stdout: "data_pool": "b2.rgw.buckets.data", 2023-10-18T19:47:08.234 INFO:teuthology.orchestra.run.smithi156.stdout: "compression_type": "random" 2023-10-18T19:47:08.234 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:08.234 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:08.235 INFO:teuthology.orchestra.run.smithi156.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-18T19:47:08.235 INFO:teuthology.orchestra.run.smithi156.stdout: "index_type": 0, 2023-10-18T19:47:08.235 INFO:teuthology.orchestra.run.smithi156.stdout: "inline_data": true 2023-10-18T19:47:08.235 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:08.235 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:08.236 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:08.236 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:47:08.236 INFO:teuthology.orchestra.run.smithi156.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-18T19:47:08.236 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:47:08.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:08.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:08.238 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-18T19:47:08.286 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:08.286 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:08.297 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:47:08.296+0000 7f8a45fcb780 0 period (67bf1060-aa43-46d9-8036-efd54ef55d7d does not have zone a08fbfd2-309c-4698-8327-be8d0aa187fe configured 2023-10-18T19:47:10.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-18T19:47:15.186 INFO:teuthology.orchestra.run.smithi156.stderr:Sending period to new master zone 403a921b-fcfc-4ab2-b434-bf29d35a51b6 2023-10-18T19:47:15.215 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:47:15.215 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:15.216 INFO:teuthology.orchestra.run.smithi156.stdout: "epoch": 4, 2023-10-18T19:47:15.216 INFO:teuthology.orchestra.run.smithi156.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T19:47:15.216 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_status": [], 2023-10-18T19:47:15.216 INFO:teuthology.orchestra.run.smithi156.stdout: "period_map": { 2023-10-18T19:47:15.216 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:15.217 INFO:teuthology.orchestra.run.smithi156.stdout: "zonegroups": [ 2023-10-18T19:47:15.217 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.217 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "a", 2023-10-18T19:47:15.217 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "a", 2023-10-18T19:47:15.217 INFO:teuthology.orchestra.run.smithi156.stdout: "api_name": "", 2023-10-18T19:47:15.218 INFO:teuthology.orchestra.run.smithi156.stdout: "is_master": true, 2023-10-18T19:47:15.218 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:47:15.218 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T19:47:15.218 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.218 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames": [], 2023-10-18T19:47:15.219 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames_s3website": [], 2023-10-18T19:47:15.219 INFO:teuthology.orchestra.run.smithi156.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:47:15.219 INFO:teuthology.orchestra.run.smithi156.stdout: "zones": [ 2023-10-18T19:47:15.219 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.219 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:47:15.220 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "a1", 2023-10-18T19:47:15.220 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:47:15.220 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T19:47:15.220 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.220 INFO:teuthology.orchestra.run.smithi156.stdout: "log_meta": false, 2023-10-18T19:47:15.220 INFO:teuthology.orchestra.run.smithi156.stdout: "log_data": true, 2023-10-18T19:47:15.221 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:47:15.221 INFO:teuthology.orchestra.run.smithi156.stdout: "read_only": false, 2023-10-18T19:47:15.221 INFO:teuthology.orchestra.run.smithi156.stdout: "tier_type": "", 2023-10-18T19:47:15.221 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from_all": true, 2023-10-18T19:47:15.221 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from": [], 2023-10-18T19:47:15.222 INFO:teuthology.orchestra.run.smithi156.stdout: "redirect_zone": "", 2023-10-18T19:47:15.222 INFO:teuthology.orchestra.run.smithi156.stdout: "supported_features": [ 2023-10-18T19:47:15.222 INFO:teuthology.orchestra.run.smithi156.stdout: "compress-encrypted", 2023-10-18T19:47:15.222 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:47:15.222 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:47:15.223 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.223 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.223 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T19:47:15.223 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "a2", 2023-10-18T19:47:15.223 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:47:15.223 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T19:47:15.224 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.224 INFO:teuthology.orchestra.run.smithi156.stdout: "log_meta": false, 2023-10-18T19:47:15.224 INFO:teuthology.orchestra.run.smithi156.stdout: "log_data": true, 2023-10-18T19:47:15.224 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:47:15.224 INFO:teuthology.orchestra.run.smithi156.stdout: "read_only": false, 2023-10-18T19:47:15.225 INFO:teuthology.orchestra.run.smithi156.stdout: "tier_type": "", 2023-10-18T19:47:15.225 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from_all": true, 2023-10-18T19:47:15.225 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from": [], 2023-10-18T19:47:15.225 INFO:teuthology.orchestra.run.smithi156.stdout: "redirect_zone": "", 2023-10-18T19:47:15.225 INFO:teuthology.orchestra.run.smithi156.stdout: "supported_features": [ 2023-10-18T19:47:15.226 INFO:teuthology.orchestra.run.smithi156.stdout: "compress-encrypted", 2023-10-18T19:47:15.226 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:47:15.226 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:47:15.226 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:15.226 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.226 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_targets": [ 2023-10-18T19:47:15.227 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.227 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "default-placement", 2023-10-18T19:47:15.227 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2023-10-18T19:47:15.227 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": [ 2023-10-18T19:47:15.227 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD" 2023-10-18T19:47:15.228 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:47:15.228 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:15.228 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.228 INFO:teuthology.orchestra.run.smithi156.stdout: "default_placement": "default-placement", 2023-10-18T19:47:15.228 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:47:15.229 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_policy": { 2023-10-18T19:47:15.229 INFO:teuthology.orchestra.run.smithi156.stdout: "groups": [] 2023-10-18T19:47:15.229 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.229 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled_features": [ 2023-10-18T19:47:15.229 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:47:15.229 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:47:15.230 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.230 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.230 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "b", 2023-10-18T19:47:15.230 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b", 2023-10-18T19:47:15.230 INFO:teuthology.orchestra.run.smithi156.stdout: "api_name": "", 2023-10-18T19:47:15.231 INFO:teuthology.orchestra.run.smithi156.stdout: "is_master": false, 2023-10-18T19:47:15.231 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:47:15.231 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T19:47:15.231 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.231 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames": [], 2023-10-18T19:47:15.232 INFO:teuthology.orchestra.run.smithi156.stdout: "hostnames_s3website": [], 2023-10-18T19:47:15.232 INFO:teuthology.orchestra.run.smithi156.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T19:47:15.232 INFO:teuthology.orchestra.run.smithi156.stdout: "zones": [ 2023-10-18T19:47:15.232 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.232 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T19:47:15.232 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b1", 2023-10-18T19:47:15.233 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:47:15.233 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T19:47:15.233 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.233 INFO:teuthology.orchestra.run.smithi156.stdout: "log_meta": false, 2023-10-18T19:47:15.233 INFO:teuthology.orchestra.run.smithi156.stdout: "log_data": true, 2023-10-18T19:47:15.234 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:47:15.234 INFO:teuthology.orchestra.run.smithi156.stdout: "read_only": false, 2023-10-18T19:47:15.234 INFO:teuthology.orchestra.run.smithi156.stdout: "tier_type": "", 2023-10-18T19:47:15.234 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from_all": true, 2023-10-18T19:47:15.234 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from": [], 2023-10-18T19:47:15.234 INFO:teuthology.orchestra.run.smithi156.stdout: "redirect_zone": "", 2023-10-18T19:47:15.235 INFO:teuthology.orchestra.run.smithi156.stdout: "supported_features": [ 2023-10-18T19:47:15.235 INFO:teuthology.orchestra.run.smithi156.stdout: "compress-encrypted", 2023-10-18T19:47:15.236 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:47:15.236 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:47:15.236 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.237 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.237 INFO:teuthology.orchestra.run.smithi156.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T19:47:15.237 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "b2", 2023-10-18T19:47:15.237 INFO:teuthology.orchestra.run.smithi156.stdout: "endpoints": [ 2023-10-18T19:47:15.237 INFO:teuthology.orchestra.run.smithi156.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T19:47:15.238 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.238 INFO:teuthology.orchestra.run.smithi156.stdout: "log_meta": false, 2023-10-18T19:47:15.238 INFO:teuthology.orchestra.run.smithi156.stdout: "log_data": true, 2023-10-18T19:47:15.238 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_index_max_shards": 11, 2023-10-18T19:47:15.238 INFO:teuthology.orchestra.run.smithi156.stdout: "read_only": false, 2023-10-18T19:47:15.239 INFO:teuthology.orchestra.run.smithi156.stdout: "tier_type": "", 2023-10-18T19:47:15.239 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from_all": true, 2023-10-18T19:47:15.239 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_from": [], 2023-10-18T19:47:15.239 INFO:teuthology.orchestra.run.smithi156.stdout: "redirect_zone": "", 2023-10-18T19:47:15.239 INFO:teuthology.orchestra.run.smithi156.stdout: "supported_features": [ 2023-10-18T19:47:15.240 INFO:teuthology.orchestra.run.smithi156.stdout: "compress-encrypted", 2023-10-18T19:47:15.240 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:47:15.240 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:47:15.240 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:15.240 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.240 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_targets": [ 2023-10-18T19:47:15.241 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.241 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "default-placement", 2023-10-18T19:47:15.241 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2023-10-18T19:47:15.241 INFO:teuthology.orchestra.run.smithi156.stdout: "storage_classes": [ 2023-10-18T19:47:15.241 INFO:teuthology.orchestra.run.smithi156.stdout: "STANDARD" 2023-10-18T19:47:15.242 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:47:15.242 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:15.242 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.242 INFO:teuthology.orchestra.run.smithi156.stdout: "default_placement": "default-placement", 2023-10-18T19:47:15.242 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:47:15.243 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_policy": { 2023-10-18T19:47:15.243 INFO:teuthology.orchestra.run.smithi156.stdout: "groups": [] 2023-10-18T19:47:15.243 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.243 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled_features": [ 2023-10-18T19:47:15.243 INFO:teuthology.orchestra.run.smithi156.stdout: "resharding" 2023-10-18T19:47:15.244 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:47:15.244 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:15.244 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2023-10-18T19:47:15.244 INFO:teuthology.orchestra.run.smithi156.stdout: "short_zone_ids": [ 2023-10-18T19:47:15.244 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.244 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:47:15.245 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 3791896933 2023-10-18T19:47:15.245 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.245 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.245 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T19:47:15.245 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 2277887411 2023-10-18T19:47:15.246 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.246 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.246 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T19:47:15.246 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 953824181 2023-10-18T19:47:15.246 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.247 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:47:15.247 INFO:teuthology.orchestra.run.smithi156.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T19:47:15.247 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 2717429769 2023-10-18T19:47:15.247 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:15.247 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:47:15.247 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.248 INFO:teuthology.orchestra.run.smithi156.stdout: "master_zonegroup": "a", 2023-10-18T19:47:15.248 INFO:teuthology.orchestra.run.smithi156.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T19:47:15.248 INFO:teuthology.orchestra.run.smithi156.stdout: "period_config": { 2023-10-18T19:47:15.248 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_quota": { 2023-10-18T19:47:15.248 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2023-10-18T19:47:15.249 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2023-10-18T19:47:15.249 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2023-10-18T19:47:15.249 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2023-10-18T19:47:15.249 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2023-10-18T19:47:15.249 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.249 INFO:teuthology.orchestra.run.smithi156.stdout: "user_quota": { 2023-10-18T19:47:15.250 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2023-10-18T19:47:15.250 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2023-10-18T19:47:15.250 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2023-10-18T19:47:15.250 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2023-10-18T19:47:15.250 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2023-10-18T19:47:15.251 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.251 INFO:teuthology.orchestra.run.smithi156.stdout: "user_ratelimit": { 2023-10-18T19:47:15.251 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_ops": 0, 2023-10-18T19:47:15.251 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_ops": 0, 2023-10-18T19:47:15.251 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_bytes": 0, 2023-10-18T19:47:15.252 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_bytes": 0, 2023-10-18T19:47:15.252 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false 2023-10-18T19:47:15.252 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.252 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_ratelimit": { 2023-10-18T19:47:15.252 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_ops": 0, 2023-10-18T19:47:15.252 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_ops": 0, 2023-10-18T19:47:15.253 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_bytes": 0, 2023-10-18T19:47:15.253 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_bytes": 0, 2023-10-18T19:47:15.253 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false 2023-10-18T19:47:15.253 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.253 INFO:teuthology.orchestra.run.smithi156.stdout: "anonymous_ratelimit": { 2023-10-18T19:47:15.254 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_ops": 0, 2023-10-18T19:47:15.254 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_ops": 0, 2023-10-18T19:47:15.254 INFO:teuthology.orchestra.run.smithi156.stdout: "max_read_bytes": 0, 2023-10-18T19:47:15.254 INFO:teuthology.orchestra.run.smithi156.stdout: "max_write_bytes": 0, 2023-10-18T19:47:15.254 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false 2023-10-18T19:47:15.255 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:47:15.255 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:47:15.255 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T19:47:15.255 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_name": "test-realm", 2023-10-18T19:47:15.255 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:47:15.256 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:47:15.256 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-18T19:47:15.257 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T19:47:15.257 INFO:tasks.rgw.c2.client.1.smithi156.stdout:2023-10-18T19:47:15.255+0000 7f636725e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 119556) UID: 0 2023-10-18T19:47:15.258 INFO:tasks.rgw.c2.client.1.smithi156.stdout:2023-10-18T19:47:15.255+0000 7f636b4c77c0 -1 shutting down 2023-10-18T19:47:16.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-18T19:47:21.358 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-18T19:47:21.359 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-18T19:47:21.359 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-18T19:47:21.362 INFO:tasks.rgw.c2.client.1:Started 2023-10-18T19:47:21.363 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2023-10-18T19:47:21.387 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:47:21.388 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:47:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T19:47:21.389 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi156.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T19:47:22.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-18T19:47:22.390 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2023-10-18T19:47:22.413 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T19:47:22.413 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2023-10-18T19:47:22.414 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T19:47:22.415 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2023-10-18T19:47:22.416 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-10-18T19:47:22.416 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-10-18T19:47:22.442 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi 2023-10-18T19:47:22.498 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-10-18T19:47:22.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:22.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T19:47:22.500 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T19:47:22.563 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:22.563 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:22.600 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:47:22.600 INFO:teuthology.orchestra.run.smithi082.stdout: "user_id": "rgw-multisite-test-user", 2023-10-18T19:47:22.600 INFO:teuthology.orchestra.run.smithi082.stdout: "display_name": "Multisite Test User", 2023-10-18T19:47:22.601 INFO:teuthology.orchestra.run.smithi082.stdout: "email": "", 2023-10-18T19:47:22.601 INFO:teuthology.orchestra.run.smithi082.stdout: "suspended": 0, 2023-10-18T19:47:22.601 INFO:teuthology.orchestra.run.smithi082.stdout: "max_buckets": 1000, 2023-10-18T19:47:22.602 INFO:teuthology.orchestra.run.smithi082.stdout: "subusers": [], 2023-10-18T19:47:22.602 INFO:teuthology.orchestra.run.smithi082.stdout: "keys": [ 2023-10-18T19:47:22.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.602 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "rgw-multisite-test-user", 2023-10-18T19:47:22.603 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "6TC2VRGERC8I036KRP19", 2023-10-18T19:47:22.603 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "9At4Dr0CUQqgS3jTVEr6oIuh7ZOdQxRHLDvh76gs" 2023-10-18T19:47:22.603 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:22.603 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:47:22.604 INFO:teuthology.orchestra.run.smithi082.stdout: "swift_keys": [], 2023-10-18T19:47:22.604 INFO:teuthology.orchestra.run.smithi082.stdout: "caps": [ 2023-10-18T19:47:22.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.604 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "roles", 2023-10-18T19:47:22.605 INFO:teuthology.orchestra.run.smithi082.stdout: "perm": "*" 2023-10-18T19:47:22.605 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:22.605 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T19:47:22.605 INFO:teuthology.orchestra.run.smithi082.stdout: "op_mask": "read, write, delete", 2023-10-18T19:47:22.605 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "", 2023-10-18T19:47:22.606 INFO:teuthology.orchestra.run.smithi082.stdout: "default_storage_class": "", 2023-10-18T19:47:22.606 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_tags": [], 2023-10-18T19:47:22.606 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T19:47:22.606 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T19:47:22.606 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T19:47:22.607 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T19:47:22.607 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T19:47:22.607 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T19:47:22.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.608 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T19:47:22.608 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T19:47:22.608 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T19:47:22.608 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T19:47:22.609 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T19:47:22.609 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T19:47:22.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.609 INFO:teuthology.orchestra.run.smithi082.stdout: "temp_url_keys": [], 2023-10-18T19:47:22.609 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "rgw", 2023-10-18T19:47:22.610 INFO:teuthology.orchestra.run.smithi082.stdout: "mfa_ids": [] 2023-10-18T19:47:22.610 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:47:22.610 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T19:47:22.612 INFO:rgw_multi.tests:meta checkpoint 2023-10-18T19:47:22.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:22.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:22.614 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:47:22.662 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:22.663 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:22.693 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:47:22.704 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:47:22.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.704 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:47:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:47:22.596445Z" 2023-10-18T19:47:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:47:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:47:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:47:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:22.707 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:47:22.709 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:47:22.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:22.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:22.710 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:47:22.759 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:22.759 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:22.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:22.784+0000 7efd7b63b780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:22.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:22.785+0000 7efd7b63b780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:22.799 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:47:22.799 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:47:22.799 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:47:22.800 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:47:22.800 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:47:22.800 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:22.800 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:22.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.801 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:47:22.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.801 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:47:22.801 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:22.801 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:22.802 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:22.802 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:22.802 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:47:22.803 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:22.803 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:22.803 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:22.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:22.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.804 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:47:22.804 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:22.804 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:22.805 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:22.805 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:22.805 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:22.805 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:22.805 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:22.806 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:22.806 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:22.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.807 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:47:22.807 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:22.807 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:22.807 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:22.807 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:22.808 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:22.808 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:22.808 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:22.808 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:22.808 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:22.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:22.809 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:47:22.809 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:22.809 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:22.809 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:22.810 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:22.810 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:22.810 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:22.810 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:22.810 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:22.811 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:22.811 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:22.811 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:47:22.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.811 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:47:22.811 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:47:22.812 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:47:22.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:22.812 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:47:22Z" 2023-10-18T19:47:22.812 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:47:22.813 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:47:27.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:27.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:27.816 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:47:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:27.903 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:27.900+0000 7fe4d28ce780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:27.903 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:27.901+0000 7fe4d28ce780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:27.916 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:47:27.916 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:47:27.916 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:47:27.916 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:47:27.917 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:47:27.917 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:27.917 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:27.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:27.917 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:47:27.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:27.918 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:47:27.918 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:27.918 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:27.918 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:27.919 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:27.919 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:47:27.919 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:27.919 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:27.919 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:27.920 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:27.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:27.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:27.920 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:47:27.920 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:27.920 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:27.921 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:27.921 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:27.921 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:27.921 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:27.921 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:27.922 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:27.922 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:27.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:27.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:27.922 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:47:27.923 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:27.923 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:27.923 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:27.924 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:27.924 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:27.924 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:27.924 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:27.924 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:27.925 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:27.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:27.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:27.925 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:47:27.925 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:27.926 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:27.926 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:27.926 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:27.926 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:27.926 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:27.927 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:27.927 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:27.927 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:27.927 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:27.927 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:47:27.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:27.928 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:47:27.928 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:47:27.928 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:47:27.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:27.928 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:47:27Z" 2023-10-18T19:47:27.929 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:47:27.930 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:47:32.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:32.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:32.932 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:47:32.988 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:32.989 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:33.018 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:33.016+0000 7feb599c1780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:33.019 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:33.017+0000 7feb599c1780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:33.032 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:47:33.033 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:47:33.033 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:47:33.033 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:47:33.033 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:47:33.033 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:33.034 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:33.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:33.034 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:47:33.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:33.034 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:47:33.035 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:33.035 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:33.035 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:33.035 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:33.035 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:47:33.036 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:33.036 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:33.036 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:33.036 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:33.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:33.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:33.037 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:47:33.037 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:33.037 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:33.037 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:33.037 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:33.038 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:33.038 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:33.038 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:33.038 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:33.038 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:33.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:33.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:33.039 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:47:33.039 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:33.039 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:33.040 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:33.040 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:33.040 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:33.040 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:33.040 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:33.040 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:33.041 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:33.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:33.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:33.041 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:47:33.041 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:33.042 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:33.042 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:33.042 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:33.042 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:33.042 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:33.043 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:33.043 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:33.043 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:33.043 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:33.043 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:47:33.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:33.044 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:47:33.044 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:47:33.044 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:47:33.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:33.044 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:47:33Z" 2023-10-18T19:47:33.045 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:47:33.046 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:47:38.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:38.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:38.047 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:47:38.104 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:38.104 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:38.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:38.132+0000 7fe9c31d2780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:38.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:38.133+0000 7fe9c31d2780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:38.146 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:47:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:47:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:47:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:47:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:47:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:47:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:47:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:47:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:47:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:47:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:47:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:47:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:47:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:47:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:47:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:47:38Z" 2023-10-18T19:47:38.158 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:47:38.159 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:47:43.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:43.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:43.161 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:47:43.218 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:43.219 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:43.250+0000 7f9326f21780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:43.251+0000 7f9326f21780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:43.264 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:47:43.264 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:47:43.265 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:47:43.265 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:47:43.265 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:47:43.265 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:43.265 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:43.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:43.266 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:47:43.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:43.266 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:47:43.266 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:43.267 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:43.267 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:43.267 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:43.267 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:47:43.267 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:43.268 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:43.268 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:43.268 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:43.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:43.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:43.268 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:47:43.269 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:43.269 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:43.269 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:43.269 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:43.269 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:43.270 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:43.270 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:43.270 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:43.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:43.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:43.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:43.271 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:47:43.271 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:43.271 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:43.271 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:43.271 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:43.272 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:43.272 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:43.272 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:43.272 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:43.272 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:43.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:43.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:43.273 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:47:43.273 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:43.273 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:43.273 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:47:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:47:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:47:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:47:43.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:43.276 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:47:43Z" 2023-10-18T19:47:43.276 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:47:43.277 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:47:48.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:48.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:48.279 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:47:48.336 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:48.336 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:48.366 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:48.364+0000 7f8a0b44d780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:48.366 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:48.365+0000 7f8a0b44d780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:48.379 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:47:48.379 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:47:48.379 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:47:48.380 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:47:48.380 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:47:48.380 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:48.380 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:48.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:48.381 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:47:48.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:48.381 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:47:48.381 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:48.381 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:48.382 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:48.382 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:48.382 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:47:48.382 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:48.382 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:48.383 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:48.383 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:48.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:48.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:48.383 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:47:48.384 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:48.384 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:48.384 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:48.384 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:48.385 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:48.385 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:48.385 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:48.385 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:48.386 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:48.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:48.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:48.386 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:47:48.386 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:48.387 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:48.387 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:48.387 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:48.387 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:48.387 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:48.387 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:48.388 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:48.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:48.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:48.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:48.388 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:47:48.389 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:48.389 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:48.389 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:48.389 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:48.389 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:48.390 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:48.390 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:48.390 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:48.390 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:48.390 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:48.390 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:47:48.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:48.391 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:47:48.391 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:47:48.391 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:47:48.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:48.392 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:47:48Z" 2023-10-18T19:47:48.392 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:47:48.393 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:47:53.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:53.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:53.395 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:47:53.451 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:53.452 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:53.481 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:53.479+0000 7f7b134db780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:53.482 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:53.480+0000 7f7b134db780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:53.494 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:47:53.494 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:47:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:47:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:47:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:47:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:47:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:47:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:53.497 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:53.497 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:53.497 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:53.497 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:47:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:47:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:53.500 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:53.500 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:53.500 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:53.500 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:53.500 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:53.501 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:53.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:53.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:53.501 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:47:53.501 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:53.502 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:53.502 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:53.502 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:53.502 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:53.502 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:53.502 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:53.503 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:53.503 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:53.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:53.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:53.503 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:47:53.504 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:53.504 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:53.504 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:53.504 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:53.504 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:53.505 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:53.505 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:53.505 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:53.505 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:53.505 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:53.505 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:47:53.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:53.506 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:47:53.506 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:47:53.506 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:47:53.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:53.507 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:47:53Z" 2023-10-18T19:47:53.507 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:47:53.508 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:47:58.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:58.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:47:58.510 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:47:58.567 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:47:58.568 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:47:58.598 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:58.596+0000 7fdd71cd9780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:58.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:47:58.598+0000 7fdd71cd9780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:47:58.611 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:47:58.611 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:47:58.612 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:47:58.612 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:47:58.612 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:47:58.612 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:47:58.612 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:58.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:58.613 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:47:58.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:58.613 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:47:58.613 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:58.614 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:58.614 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:58.614 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:58.614 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:47:58.614 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:58.615 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:58.615 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:58.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:58.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:58.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:58.616 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:47:58.616 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:58.616 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:58.617 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:58.617 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:58.617 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:58.617 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:58.617 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:58.617 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:58.618 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:58.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:58.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:58.618 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:47:58.618 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:58.619 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:58.619 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:58.619 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:58.619 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:58.619 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:58.620 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:58.620 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:58.620 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:58.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:58.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:47:58.620 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:47:58.621 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:47:58.621 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:47:58.621 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:47:58.621 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:47:58.621 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:47:58.622 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:47:58.622 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:47:58.622 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:47:58.622 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:58.622 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:47:58.623 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:47:58.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:58.623 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:47:58.623 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:47:58.623 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:47:58.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:47:58.624 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:47:58Z" 2023-10-18T19:47:58.624 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:47:58.625 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:03.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:03.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:03.627 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:03.684 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:03.685 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:03.717 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:03.715+0000 7fd0b74a0780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:03.717 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:03.716+0000 7fd0b74a0780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:03.729 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:03.730 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:03.730 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:03.730 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:03.730 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:03.731 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:03.731 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:03.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:03.731 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:03.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:03.732 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:03.732 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:03.732 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:03.732 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:03.732 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:03.733 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:03.733 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:03.733 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:03.733 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:03.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:03.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:03.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:03.734 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:03.734 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:03.734 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:03.735 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:03.735 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:03.735 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:03.735 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:03.735 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:03.735 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:03.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:03.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:03.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:03.736 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:03.736 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:03.737 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:03.737 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:03.737 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:03.737 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:03.737 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:03.738 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:03.738 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:03.738 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:03.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:03.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:03.739 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:03.739 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:03.739 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:03.739 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:03.739 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:03.740 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:03.740 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:03.740 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:03.740 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:03.740 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:03.740 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:03.741 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:03.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:03.741 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:03.741 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:03.741 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:03.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:03.742 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:03Z" 2023-10-18T19:48:03.742 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:03.743 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:08.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:08.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:08.745 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:08.801 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:08.802 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:08.832 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:08.829+0000 7f22f2e18780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:08.833 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:08.830+0000 7f22f2e18780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:08.845 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:08.845 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:08.845 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:08.845 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:08.846 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:08.846 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:08.846 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:08.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:08.846 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:08.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:08.847 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:08.847 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:08.847 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:08.847 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:08.848 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:08.848 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:08.848 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:08.848 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:08.848 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:08.849 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:08.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:08.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:08.849 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:08.849 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:08.849 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:08.850 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:08.850 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:08.850 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:08.850 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:08.850 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:08.851 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:08.851 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:08.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:08.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:08.851 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:08.852 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:08.852 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:08.852 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:08.852 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:08.852 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:08.853 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:08.853 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:08.853 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:08.853 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:08.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:08.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:08.854 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:08.855 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:08.855 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:08.855 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:08.856 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:08.856 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:08.856 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:08.856 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:08.857 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:08.857 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:08.859 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:08.859 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:08.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:08.859 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:08.859 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:08.860 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:08.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:08.860 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:08Z" 2023-10-18T19:48:08.860 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:08.861 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:13.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:13.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:13.863 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:13.920 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:13.921 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:13.952 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:13.949+0000 7f56794a0780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:13.953 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:13.950+0000 7f56794a0780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:13.965 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:13.966 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:13.966 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:13.966 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:13.966 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:13.967 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:13.967 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:13.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:13.967 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:13.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:13.968 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:13.968 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:13.968 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:13.968 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:13.968 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:13.968 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:13.969 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:13.969 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:13.969 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:13.969 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:13.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:13.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:13.970 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:13.970 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:13.970 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:13.970 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:13.971 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:13.971 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:13.971 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:13.972 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:13.972 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:13.972 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:13.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:13.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:13.973 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:13.973 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:13.973 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:13.973 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:13.974 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:13.974 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:13.974 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:13.974 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:13.975 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:13.975 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:13.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:13.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:13.975 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:13.975 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:13.976 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:13.976 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:13.976 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:13.976 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:13.976 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:13.977 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:13.977 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:13.977 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:13.977 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:13.978 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:13.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:13.978 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:13.978 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:13.978 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:13.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:13.979 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:13Z" 2023-10-18T19:48:13.979 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:13.980 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:18.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:18.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:18.982 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:19.039 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:19.039 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:19.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:19.067+0000 7f70accc3780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:19.071 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:19.068+0000 7f70accc3780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:19.083 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:19Z" 2023-10-18T19:48:19.097 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:19.098 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:24.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:24.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:24.100 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:24.157 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:24.158 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:24.187 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:24.185+0000 7ff546fdf780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:24.188 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:24.186+0000 7ff546fdf780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:24.201 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:24.202 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:24.202 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:24.202 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:24.202 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:24.203 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:24.203 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:24.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:24.203 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:24.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:24.204 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:24.204 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:24.204 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:24.204 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:24.204 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:24.205 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:24.205 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:24.205 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:24.205 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:24.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:24.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:24.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:24.206 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:24.206 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:24.206 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:24.206 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:24.207 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:24.207 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:24.207 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:24.207 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:24.207 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:24.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:24.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:24.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:24.208 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:24.208 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:24.209 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:24.209 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:24.209 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:24.209 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:24.209 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:24.209 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:24.210 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:24.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:24.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:24.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:24.210 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:24.211 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:24.211 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:24.211 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:24.211 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:24.211 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:24.212 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:24.212 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:24.212 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:24.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:24.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:24.213 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:24.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:24.213 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:24.213 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:24.213 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:24.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:24.214 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:24Z" 2023-10-18T19:48:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:24.215 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:29.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:29.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:29.217 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:29.274 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:29.275 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:29.307 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:29.305+0000 7f7285670780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:29.308 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:29.306+0000 7f7285670780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:29.320 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:29.320 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:29.321 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:29.321 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:29.321 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:29.321 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:29.321 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:29.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:29.322 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:29.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:29.322 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:29.322 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:29.322 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:29.323 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:29.323 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:29.323 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:29.323 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:29.323 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:29.324 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:29.324 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:29.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:29.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:29.324 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:29.325 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:29.325 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:29.325 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:29.325 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:29.325 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:29.325 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:29.326 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:29.326 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:29.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:29.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:29.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:29.327 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:29.327 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:29.327 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:29.327 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:29.327 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:29.328 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:29.328 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:29.328 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:29.328 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:29.328 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:29.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:29.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:29.329 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:29.329 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:29.329 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:29.329 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:29.330 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:29.330 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:29.330 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:29.330 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:29.330 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:29.330 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:29.331 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:29.331 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:29.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:29.331 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:29.331 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:29.331 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:29.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:29.332 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:29Z" 2023-10-18T19:48:29.332 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:29.333 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:34.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:34.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:34.336 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:34.392 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:34.393 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:34.423 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:34.421+0000 7efe54eff780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:34.424 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:34.422+0000 7efe54eff780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:34.436 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:34.436 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:34.437 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:34.437 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:34.437 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:34.437 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:34.437 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:34.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:34.438 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:34.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:34.438 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:34.438 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:34.442 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:34.442 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:34.442 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:34.442 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:34.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:34.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:34.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:34.445 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:34.445 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:34.445 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:34.446 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:34.446 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:34.446 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:34.446 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:34.446 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:34.447 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:34.447 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:34.447 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:34.447 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:34.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:34.447 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:34.448 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:34.448 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:34.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:34.448 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:34Z" 2023-10-18T19:48:34.448 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:34.450 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:39.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:39.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:39.453 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:39.510 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:39.511 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:39.547 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:39.545+0000 7fdb53509780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:39.547 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:39.546+0000 7fdb53509780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:39.559 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:39.560 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:39.560 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:39.560 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:39.561 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:39.561 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:39.561 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:39.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:39.561 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:39.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:39.562 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:39.562 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:39.563 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:39.563 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:39.563 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:39.563 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:39.563 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:39.564 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:39.564 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:39.564 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:39.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:39.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:39.565 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:39.565 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:39.565 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:39.565 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:39.565 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:39.566 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:39.566 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:39.566 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:39.566 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:39.567 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:39.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:39.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:39.567 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:39.567 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:39.568 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:39.568 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:39.568 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:39.568 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:39.568 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:39.568 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:39.569 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:39.569 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:39.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:39.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:39.569 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:39.571 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:39.571 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:39.571 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:39.571 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:39.572 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:39.572 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:39.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:39.572 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:39.573 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:39.573 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:39.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:39.574 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:39Z" 2023-10-18T19:48:39.574 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:39.576 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:44.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:44.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:44.578 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:44.635 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:44.636 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:44.665 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:44.663+0000 7febac8b4780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:44.666 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:44.664+0000 7febac8b4780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:44.677 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:44.678 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:44.678 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:44.678 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:44.678 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:44.679 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:44.679 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:44.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:44.680 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:44.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:44.680 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:44.681 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:44.681 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:44.681 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:44.682 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:44.682 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:44.682 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:44.682 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:44.683 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:44.683 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:44.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:44.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:44.684 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:44.684 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:44.685 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:44.685 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:44.685 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:44.686 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:44.686 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:44.686 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:44.687 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:44.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:44.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:44.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:44.688 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:44.688 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:44.688 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:44.689 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:44.689 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:44.689 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:44.690 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:44.690 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:44.690 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:44.691 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:44.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:44.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:44.691 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:44.692 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:44.692 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:44.692 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:44.693 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:44.693 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:44.693 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:44.693 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:44.694 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:44.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:44.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:44.694 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:44.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:44.695 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:44.695 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:44.695 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:44.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:44.695 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:44Z" 2023-10-18T19:48:44.696 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:44.697 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:49.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:49.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:49.699 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:49.756 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:49.757 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:49.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:49.785+0000 7f1dd81ee780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:49.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:49.786+0000 7f1dd81ee780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:49.801 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:49.801 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:49.801 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:49.802 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:49.802 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:49.802 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:49.802 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:49.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:49.803 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:49.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:49.803 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:49.803 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:49.803 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:49.804 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:49.804 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:49.804 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:49.804 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:49.804 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:49.804 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:49.805 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:49.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:49.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:49.805 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:49.806 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:49.806 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:49.806 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:49.806 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:49.806 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:49.807 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:49.807 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:49.807 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:49.807 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:49.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:49.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:49.808 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:49.808 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:49.808 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:49.808 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:49.809 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:49.809 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:49.809 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:49.809 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:49.809 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:49.810 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:49.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:49.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:49.810 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:49.810 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:49.811 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:49.811 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:49.811 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:49.811 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:49.811 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:49.812 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:49.812 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:49.812 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:49.812 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:49.812 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:49.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:49.813 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:49.813 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:49.813 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:49.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:49.813 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:49Z" 2023-10-18T19:48:49.814 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:49.815 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:54.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:54.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:54.817 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:54.873 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:54.873 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:48:54.904 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:54.902+0000 7f9d88149780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:54.905 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:48:54.903+0000 7f9d88149780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:48:54.917 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:48:54.917 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:48:54.917 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:48:54.918 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:48:54.918 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:48:54.918 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:48:54.918 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:54.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:54.919 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:48:54.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:54.919 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:48:54.919 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:54.919 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:54.920 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:54.920 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:54.920 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:48:54.920 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:54.920 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:54.921 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:54.921 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:54.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:54.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:54.921 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:48:54.922 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:54.922 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:54.922 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:54.922 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:54.922 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:54.923 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:54.923 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:54.923 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:54.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:54.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:54.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:54.924 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:48:54.924 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:54.924 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:54.924 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:54.925 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:54.925 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:54.925 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:54.925 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:54.925 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:54.925 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:54.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:54.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:48:54.926 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:48:54.926 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:48:54.927 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:48:54.927 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:48:54.927 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:48:54.927 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:48:54.927 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:48:54.928 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:48:54.928 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:48:54.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:54.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:48:54.928 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:48:54.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:54.929 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:48:54.929 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:48:54.929 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:48:54.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:48:54.929 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:48:54Z" 2023-10-18T19:48:54.930 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:48:54.931 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:48:59.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:59.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:48:59.933 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:48:59.991 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:48:59.991 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:00.023 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:00.021+0000 7f0fe83ff780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:00.024 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:00.022+0000 7f0fe83ff780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:00.036 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:00.037 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:00.037 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:00.037 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:00.037 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:00.038 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:00.038 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:00.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:00.038 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:00.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:00.039 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:00.039 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:00.039 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:00.039 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:00.039 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:00.040 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:00.040 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:00.040 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:00.040 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:00.040 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:00.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:00.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:00.041 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:00.041 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:00.041 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:00.041 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:00.042 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:00.042 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:00.042 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:00.042 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:00.042 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:00.042 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:00.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:00.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:00.043 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:00.043 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:00.043 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:00.043 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:00.044 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:00.044 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:00.044 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:00.044 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:00.044 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:00.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:00.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:00.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:00.045 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:00.045 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:00.045 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:00.046 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:00.046 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:00.046 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:00.046 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:00.046 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:00.047 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:00.047 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:00.047 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:00.047 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:00.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:00.047 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:00.048 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:00.048 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:00.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:00.048 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:00Z" 2023-10-18T19:49:00.048 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:00.049 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:49:05.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:05.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:05.052 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:05.108 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:05.108 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:05.141 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:05.137+0000 7f782fdba780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:05.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:05.138+0000 7f782fdba780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:05.152 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:05.152 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:05.152 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:05.153 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:05.153 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:05.153 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:05.153 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:05.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:05.154 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:05.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:05.154 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:05.154 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:05.154 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:05.155 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:05.155 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:05.155 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:05.155 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:05.155 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:05.155 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:05.156 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:05.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:05.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:05.156 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:05.156 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:05.157 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:05.157 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:05.157 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:05.157 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:05.157 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:05.158 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:05.158 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:05.158 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:05.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:05.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:05.158 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:05.159 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:05.159 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:05.159 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:05.159 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:05.159 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:05.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:05.161 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:05.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:05.163 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:05.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:05.164 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:05Z" 2023-10-18T19:49:05.164 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:05.165 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:49:10.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:10.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:10.167 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:10.223 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:10.223 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:10.255 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:10.253+0000 7f804e285780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:10.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:10.254+0000 7f804e285780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:10.270 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:10.275 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:10.275 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:10.275 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:10.275 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:10.275 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:10.276 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:10.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:10.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:10.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:10.276 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:10.277 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:10.277 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:10.277 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:10.277 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:10.277 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:10.277 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:10Z" 2023-10-18T19:49:10.282 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:10.282 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:49:15.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:15.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:15.285 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:15.344 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:15.345 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:15.374 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:15.371+0000 7f5311b0f780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:15.375 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:15.372+0000 7f5311b0f780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:15.387 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:15.387 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:15.387 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:15.387 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:15.388 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:15.388 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:15.388 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:15.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:15.388 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:15.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:15.389 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:15.389 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:15.389 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:15.389 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:15.390 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:15.390 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:15.390 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:15.390 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:15.390 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:15.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:15.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:15.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:15.391 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:15.391 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:15.391 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:15.392 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:15.392 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:15.392 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:15.392 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:15.392 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:15.393 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:15.393 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:15.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:15.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:15.393 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:15.394 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:15.394 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:15.394 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:15.394 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:15.394 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:15.394 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:15.395 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:15.395 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:15.395 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:15.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:15.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:15.396 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:15.396 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:15.396 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:15.396 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:15.396 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:15.396 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:15.397 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:15.397 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:15.397 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:15.397 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:15.397 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:15.398 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:15.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:15.398 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:15.398 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:15.398 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:15.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:15.399 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:15Z" 2023-10-18T19:49:15.399 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:15.400 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658442.596445_106.1 target= 2023-10-18T19:49:20.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.402 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:20.459 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:20.459 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:20.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:20.493+0000 7f3b3f43c780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:20.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:20.494+0000 7f3b3f43c780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:20.508 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:20.508 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:20.508 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:20.508 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:20.509 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:20.509 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:20.509 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.509 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:20.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.510 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:20.510 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:20.510 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:20.510 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:20.511 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:20.511 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:20.511 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:20.511 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:20.511 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.512 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.512 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:20.512 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:20.513 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:20.513 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:20.513 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:20.513 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:20.513 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:20.514 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.514 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.514 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.515 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:20.515 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:20.515 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:20.515 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:20.515 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:20.515 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:20.516 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:20.516 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.516 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.516 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.517 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:20.517 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:20.517 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:20.517 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:20.517 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:20.518 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:20.518 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:20.518 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.518 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.518 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.519 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:20.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.519 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:20.519 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:20.521 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:20.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.521 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:20Z" 2023-10-18T19:49:20.521 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:20.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:49:20.523 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-18T19:49:20.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.524 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:20.581 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:20.581 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:20.612 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:49:20.611+0000 7f68a88ff780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:20.613 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:49:20.612+0000 7f68a88ff780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:20.625 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:49:20.625 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_status": { 2023-10-18T19:49:20.625 INFO:teuthology.orchestra.run.smithi156.stdout: "info": { 2023-10-18T19:49:20.626 INFO:teuthology.orchestra.run.smithi156.stdout: "status": "sync", 2023-10-18T19:49:20.626 INFO:teuthology.orchestra.run.smithi156.stdout: "num_shards": 4, 2023-10-18T19:49:20.626 INFO:teuthology.orchestra.run.smithi156.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:20.626 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.626 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.627 INFO:teuthology.orchestra.run.smithi156.stdout: "markers": [ 2023-10-18T19:49:20.627 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:20.627 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 0, 2023-10-18T19:49:20.627 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:20.627 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:20.628 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:20.628 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:20.628 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 1, 2023-10-18T19:49:20.628 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:20.628 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:20.629 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.629 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.629 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.629 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:20.629 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 1, 2023-10-18T19:49:20.630 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:20.630 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:20.630 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:20.630 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:20.631 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:20.631 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:20.631 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.631 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.631 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.632 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.632 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:20.632 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 2, 2023-10-18T19:49:20.632 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:20.632 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:20.632 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:20.633 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:20.633 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:20.633 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:20.633 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.633 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.634 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.634 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.634 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:20.634 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 3, 2023-10-18T19:49:20.634 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:20.635 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:20.635 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:20.635 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:20.635 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:20.635 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:20.635 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.636 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.636 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.636 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.636 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:49:20.636 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.637 INFO:teuthology.orchestra.run.smithi156.stdout: "full_sync": { 2023-10-18T19:49:20.637 INFO:teuthology.orchestra.run.smithi156.stdout: "total": 1, 2023-10-18T19:49:20.637 INFO:teuthology.orchestra.run.smithi156.stdout: "complete": 1 2023-10-18T19:49:20.637 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.637 INFO:teuthology.orchestra.run.smithi156.stdout: "current_time": "2023-10-18T19:49:20Z" 2023-10-18T19:49:20.637 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:49:20.639 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-18T19:49:20.639 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-18T19:49:20.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.640 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:20.691 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:20.691 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:20.717 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:49:20.716+0000 7f17f4b5f780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:20.718 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:49:20.717+0000 7f17f4b5f780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:20.729 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:49:20.730 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_status": { 2023-10-18T19:49:20.730 INFO:teuthology.orchestra.run.smithi156.stdout: "info": { 2023-10-18T19:49:20.730 INFO:teuthology.orchestra.run.smithi156.stdout: "status": "sync", 2023-10-18T19:49:20.730 INFO:teuthology.orchestra.run.smithi156.stdout: "num_shards": 4, 2023-10-18T19:49:20.730 INFO:teuthology.orchestra.run.smithi156.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:20.731 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.731 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.731 INFO:teuthology.orchestra.run.smithi156.stdout: "markers": [ 2023-10-18T19:49:20.731 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:20.731 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 0, 2023-10-18T19:49:20.732 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:20.732 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:20.732 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:20.732 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:20.732 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 1, 2023-10-18T19:49:20.732 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:20.733 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:20.733 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.733 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.733 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.733 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:20.734 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 1, 2023-10-18T19:49:20.734 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:20.734 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:20.734 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:20.734 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:20.735 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:20.735 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:20.735 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.735 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.735 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.735 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.736 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:20.736 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 2, 2023-10-18T19:49:20.736 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:20.736 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:20.736 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:20.737 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:20.737 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:20.737 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:20.737 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.737 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.738 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.738 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.738 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:20.738 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 3, 2023-10-18T19:49:20.738 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:20.738 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:20.739 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:20.739 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:20.739 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:20.739 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:20.739 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.740 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:20.740 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.740 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:20.740 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:49:20.740 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.740 INFO:teuthology.orchestra.run.smithi156.stdout: "full_sync": { 2023-10-18T19:49:20.741 INFO:teuthology.orchestra.run.smithi156.stdout: "total": 1, 2023-10-18T19:49:20.741 INFO:teuthology.orchestra.run.smithi156.stdout: "complete": 1 2023-10-18T19:49:20.741 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:20.741 INFO:teuthology.orchestra.run.smithi156.stdout: "current_time": "2023-10-18T19:49:20Z" 2023-10-18T19:49:20.741 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:49:20.742 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-18T19:49:20.743 INFO:rgw_multi.tests:meta checkpoint 2023-10-18T19:49:20.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.744 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:20.796 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:20.797 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:20.827 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:49:20.838 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:49:20.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.839 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:20.839 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:47:22.596445Z" 2023-10-18T19:49:20.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.839 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:20.840 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:20.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.840 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:20.840 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:20.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.841 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:20.841 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:20.841 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.842 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:49:20.843 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:49:20.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.844 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:20.893 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:20.893 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:20.920 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:20.918+0000 7fc6c0b55780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:20.921 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:20.919+0000 7fc6c0b55780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:20.934 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:20.935 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:20.935 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:20.935 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:20.935 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:20.935 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:20.936 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.936 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:20.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.936 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:20.936 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:20.937 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:20.937 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:20.937 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:20.937 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:20.937 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:20.938 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:20.938 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.938 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.939 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:20.939 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:20.939 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:20.939 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:20.939 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:20.939 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:20.940 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:20.940 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.940 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.940 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.941 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:20.941 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:20.941 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:20.941 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:20.942 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:20.942 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:20.942 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:20.942 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.942 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.942 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:20.943 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:20.943 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:20.943 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:20.944 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:20.944 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:20.944 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:20.944 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:20.944 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:20.944 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:20.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:20.946 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:20.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.947 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:20.947 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:20.947 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:20.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:20.947 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:20Z" 2023-10-18T19:49:20.948 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:20.949 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:49:20.949 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-18T19:49:20.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:20.950 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:21.001 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:21.001 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:21.028 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:49:21.026+0000 7fc7f1951780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:21.028 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:49:21.027+0000 7fc7f1951780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:21.039 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:49:21.039 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_status": { 2023-10-18T19:49:21.040 INFO:teuthology.orchestra.run.smithi156.stdout: "info": { 2023-10-18T19:49:21.040 INFO:teuthology.orchestra.run.smithi156.stdout: "status": "sync", 2023-10-18T19:49:21.040 INFO:teuthology.orchestra.run.smithi156.stdout: "num_shards": 4, 2023-10-18T19:49:21.041 INFO:teuthology.orchestra.run.smithi156.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:21.041 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.041 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.042 INFO:teuthology.orchestra.run.smithi156.stdout: "markers": [ 2023-10-18T19:49:21.042 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:21.042 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 0, 2023-10-18T19:49:21.043 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:21.043 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:21.043 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:21.044 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:21.044 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 1, 2023-10-18T19:49:21.044 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:21.044 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:21.045 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.045 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.045 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.046 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:21.046 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 1, 2023-10-18T19:49:21.046 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:21.047 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:21.047 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:21.047 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:21.048 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:21.048 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:21.048 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:21.049 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.049 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.049 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.050 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:21.050 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 2, 2023-10-18T19:49:21.051 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:21.051 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:21.051 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:21.051 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:21.052 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:21.052 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:21.052 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:21.053 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.053 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.053 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.053 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:21.054 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 3, 2023-10-18T19:49:21.054 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:21.054 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:21.055 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:21.055 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:21.055 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:21.055 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:21.055 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:21.056 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.056 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.056 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.056 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:49:21.056 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.057 INFO:teuthology.orchestra.run.smithi156.stdout: "full_sync": { 2023-10-18T19:49:21.057 INFO:teuthology.orchestra.run.smithi156.stdout: "total": 1, 2023-10-18T19:49:21.057 INFO:teuthology.orchestra.run.smithi156.stdout: "complete": 1 2023-10-18T19:49:21.057 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.057 INFO:teuthology.orchestra.run.smithi156.stdout: "current_time": "2023-10-18T19:49:21Z" 2023-10-18T19:49:21.058 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:49:21.059 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-18T19:49:21.060 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-18T19:49:21.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:21.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:21.061 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:21.110 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:21.111 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:21.137 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:49:21.135+0000 7f9d9310d780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:21.137 INFO:teuthology.orchestra.run.smithi156.stderr:2023-10-18T19:49:21.136+0000 7f9d9310d780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:21.148 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-10-18T19:49:21.149 INFO:teuthology.orchestra.run.smithi156.stdout: "sync_status": { 2023-10-18T19:49:21.149 INFO:teuthology.orchestra.run.smithi156.stdout: "info": { 2023-10-18T19:49:21.149 INFO:teuthology.orchestra.run.smithi156.stdout: "status": "sync", 2023-10-18T19:49:21.149 INFO:teuthology.orchestra.run.smithi156.stdout: "num_shards": 4, 2023-10-18T19:49:21.149 INFO:teuthology.orchestra.run.smithi156.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:21.150 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.150 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.150 INFO:teuthology.orchestra.run.smithi156.stdout: "markers": [ 2023-10-18T19:49:21.150 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:21.150 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 0, 2023-10-18T19:49:21.151 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:21.151 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:21.151 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:21.151 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:21.151 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 1, 2023-10-18T19:49:21.152 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:21.152 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:21.152 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.152 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.152 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.153 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:21.153 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 1, 2023-10-18T19:49:21.153 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:21.153 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:21.153 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:21.153 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:21.154 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:21.154 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:21.154 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:21.154 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.154 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.155 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.155 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:21.155 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 2, 2023-10-18T19:49:21.155 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:21.155 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:21.156 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:21.156 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:21.156 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:21.156 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:21.156 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:21.156 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.157 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.157 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.157 INFO:teuthology.orchestra.run.smithi156.stdout: { 2023-10-18T19:49:21.157 INFO:teuthology.orchestra.run.smithi156.stdout: "key": 3, 2023-10-18T19:49:21.158 INFO:teuthology.orchestra.run.smithi156.stdout: "val": { 2023-10-18T19:49:21.158 INFO:teuthology.orchestra.run.smithi156.stdout: "state": 1, 2023-10-18T19:49:21.158 INFO:teuthology.orchestra.run.smithi156.stdout: "marker": "", 2023-10-18T19:49:21.159 INFO:teuthology.orchestra.run.smithi156.stdout: "next_step_marker": "", 2023-10-18T19:49:21.159 INFO:teuthology.orchestra.run.smithi156.stdout: "total_entries": 0, 2023-10-18T19:49:21.159 INFO:teuthology.orchestra.run.smithi156.stdout: "pos": 0, 2023-10-18T19:49:21.160 INFO:teuthology.orchestra.run.smithi156.stdout: "timestamp": "0.000000", 2023-10-18T19:49:21.160 INFO:teuthology.orchestra.run.smithi156.stdout: "realm_epoch": 2 2023-10-18T19:49:21.160 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.161 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-10-18T19:49:21.161 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-10-18T19:49:21.161 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.162 INFO:teuthology.orchestra.run.smithi156.stdout: "full_sync": { 2023-10-18T19:49:21.162 INFO:teuthology.orchestra.run.smithi156.stdout: "total": 1, 2023-10-18T19:49:21.162 INFO:teuthology.orchestra.run.smithi156.stdout: "complete": 1 2023-10-18T19:49:21.163 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-10-18T19:49:21.163 INFO:teuthology.orchestra.run.smithi156.stdout: "current_time": "2023-10-18T19:49:21Z" 2023-10-18T19:49:21.163 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-10-18T19:49:21.165 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-18T19:49:21.166 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-10-18T19:49:21.557 DEBUG:boto:Using access key provided by client. 2023-10-18T19:49:21.722 DEBUG:boto:Using secret key provided by client. 2023-10-18T19:49:21.723 DEBUG:boto:Using access key provided by client. 2023-10-18T19:49:21.742 DEBUG:boto:Using secret key provided by client. 2023-10-18T19:49:21.743 DEBUG:boto:Using access key provided by client. 2023-10-18T19:49:21.744 DEBUG:boto:Using secret key provided by client. 2023-10-18T19:49:21.745 DEBUG:boto:Using access key provided by client. 2023-10-18T19:49:21.747 DEBUG:boto:Using secret key provided by client. 2023-10-18T19:49:21.747 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-1 2023-10-18T19:49:21.748 DEBUG:boto:path=/bofrnf-1/ 2023-10-18T19:49:21.748 DEBUG:boto:auth_path=/bofrnf-1/ 2023-10-18T19:49:21.748 DEBUG:boto:Method: PUT 2023-10-18T19:49:21.749 DEBUG:boto:Path: /bofrnf-1/ 2023-10-18T19:49:21.749 DEBUG:boto:Data: 2023-10-18T19:49:21.750 DEBUG:boto:Headers: {} 2023-10-18T19:49:21.750 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:21.750 DEBUG:boto:Port: 8000 2023-10-18T19:49:21.751 DEBUG:boto:Params: {} 2023-10-18T19:49:21.751 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T19:49:21.752 DEBUG:boto:Token: None 2023-10-18T19:49:21.752 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:21 GMT /bofrnf-1/ 2023-10-18T19:49:21.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JKzNqTFj+wqrJuVoPYX5AURD+4s= 2023-10-18T19:49:21.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JKzNqTFj+wqrJuVoPYX5AURD+4s=', 'Content-Length': '0'} 2023-10-18T19:49:24.128 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e5a83b4df8fdcc40-00653036c1-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:24.129 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-2 2023-10-18T19:49:24.129 DEBUG:boto:path=/bofrnf-2/ 2023-10-18T19:49:24.130 DEBUG:boto:auth_path=/bofrnf-2/ 2023-10-18T19:49:24.130 DEBUG:boto:Method: PUT 2023-10-18T19:49:24.131 DEBUG:boto:Path: /bofrnf-2/ 2023-10-18T19:49:24.131 DEBUG:boto:Data: 2023-10-18T19:49:24.131 DEBUG:boto:Headers: {} 2023-10-18T19:49:24.132 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:24.132 DEBUG:boto:Port: 8001 2023-10-18T19:49:24.133 DEBUG:boto:Params: {} 2023-10-18T19:49:24.133 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T19:49:24.134 DEBUG:boto:Token: None 2023-10-18T19:49:24.134 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:24 GMT /bofrnf-2/ 2023-10-18T19:49:24.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:agFyaaZhwZtEGho3ZLz6Gss5k7o= 2023-10-18T19:49:24.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:agFyaaZhwZtEGho3ZLz6Gss5k7o=', 'Content-Length': '0'} 2023-10-18T19:49:27.141 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ae5762871cb0500a-00653036c4-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:27.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:27.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:27.143 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:27.195 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:27.195 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:27.229 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:49:27.240 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:49:27.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:27.241 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:27.241 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:47:22.596445Z" 2023-10-18T19:49:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658564.153199_76.1", 2023-10-18T19:49:27.243 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:24.153199Z" 2023-10-18T19:49:27.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:27.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:27.244 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:27.244 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:27.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:27.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:27.246 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:27.246 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:27.246 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:27.247 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:49:27.248 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697658442.596445_106.1', 1: '1_1697658564.153199_76.1', 2: '', 3: ''} 2023-10-18T19:49:27.249 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:49:27.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:27.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:27.251 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:27.301 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:27.301 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:27.333 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:27.331+0000 7feb927bb780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:27.334 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:27.332+0000 7feb927bb780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:27.345 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:27.345 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:27.346 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:27.346 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:27.346 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:27.346 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:27.347 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:27.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:27.347 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:27.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:27.347 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:27.348 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:27.348 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:27.348 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:27.348 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:27.348 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:27.349 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:27.349 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:27.349 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:27.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:27.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:27.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:27.350 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:27.350 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:27.350 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:27.351 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658564.109200_69.1", 2023-10-18T19:49:27.351 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:27.351 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:27.351 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:27.351 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:24.109200Z", 2023-10-18T19:49:27.352 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:27.352 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:27.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:27.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:27.352 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:27.353 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:27.353 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:27.353 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:27.353 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:27.353 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:27.354 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:27.354 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:27.354 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:27.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:27.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:27.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:27.355 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:27.355 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:27.355 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:27.355 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:27.356 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:27.356 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:27.356 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:27.357 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:27.357 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:27.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:27.358 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:27.358 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:27.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:27.358 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:27.359 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:27.359 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:27.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:27.359 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:27Z" 2023-10-18T19:49:27.359 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:27.361 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658442.596445_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:47:22.596445Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658564.109200_69.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:24.109200Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:49:27Z" } 2023-10-18T19:49:27.361 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658442.596445_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:47:22.596445Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658564.109200_69.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:24.109200Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:49:27.362 DEBUG:rgw_multi.tests:log_status={0: '1_1697658442.596445_106.1', 1: '1_1697658564.153199_76.1', 2: '', 3: ''} 2023-10-18T19:49:27.362 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658442.596445_106.1', 1: '1_1697658564.109200_69.1', 2: '', 3: ''} 2023-10-18T19:49:27.362 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697658564.153199_76.1 target=1_1697658564.109200_69.1 2023-10-18T19:49:32.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:32.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:32.365 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:32.422 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:32.423 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:32.452 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:32.450+0000 7f0f7202b780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:32.453 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:32.451+0000 7f0f7202b780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:32.465 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:32.465 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:32.465 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:32.466 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:32.466 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:32.466 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:32.466 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.467 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:32.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.467 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:32.468 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:32.468 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:32.468 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:32.468 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:32.468 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:32.469 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:32.469 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:32.469 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.470 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:32.470 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:32.470 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658564.153199_76.1", 2023-10-18T19:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:24.153199Z", 2023-10-18T19:49:32.472 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.472 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.472 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:32.474 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:32.474 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:32.474 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.474 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.478 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:32.478 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:32.478 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:32.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.479 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:32Z" 2023-10-18T19:49:32.479 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:32.480 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658442.596445_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:47:22.596445Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658564.153199_76.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:24.153199Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:49:32Z" } 2023-10-18T19:49:32.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658442.596445_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:47:22.596445Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658564.153199_76.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:24.153199Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:49:32.481 DEBUG:rgw_multi.tests:log_status={0: '1_1697658442.596445_106.1', 1: '1_1697658564.153199_76.1', 2: '', 3: ''} 2023-10-18T19:49:32.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658442.596445_106.1', 1: '1_1697658564.153199_76.1', 2: '', 3: ''} 2023-10-18T19:49:32.482 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:49:32.482 DEBUG:boto:path=/bofrnf-1/ 2023-10-18T19:49:32.482 DEBUG:boto:auth_path=/bofrnf-1/ 2023-10-18T19:49:32.483 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.483 DEBUG:boto:Path: /bofrnf-1/ 2023-10-18T19:49:32.484 DEBUG:boto:Data: 2023-10-18T19:49:32.484 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.484 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:32.485 DEBUG:boto:Port: 8000 2023-10-18T19:49:32.485 DEBUG:boto:Params: {} 2023-10-18T19:49:32.486 DEBUG:boto:Token: None 2023-10-18T19:49:32.486 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-1/ 2023-10-18T19:49:32.487 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q397kXE9h9g0LwPOkyrm0qnhsmA= 2023-10-18T19:49:32.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q397kXE9h9g0LwPOkyrm0qnhsmA=', 'Content-Length': '0'} 2023-10-18T19:49:32.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dc32af5f966ad49-00653036cc-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.490 DEBUG:boto:path=/bofrnf-2/ 2023-10-18T19:49:32.491 DEBUG:boto:auth_path=/bofrnf-2/ 2023-10-18T19:49:32.491 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.492 DEBUG:boto:Path: /bofrnf-2/ 2023-10-18T19:49:32.492 DEBUG:boto:Data: 2023-10-18T19:49:32.492 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.493 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:32.493 DEBUG:boto:Port: 8000 2023-10-18T19:49:32.494 DEBUG:boto:Params: {} 2023-10-18T19:49:32.494 DEBUG:boto:Token: None 2023-10-18T19:49:32.495 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-2/ 2023-10-18T19:49:32.495 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:re32ADyBbd767yzEp0z09FMp2q0= 2023-10-18T19:49:32.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:re32ADyBbd767yzEp0z09FMp2q0=', 'Content-Length': '0'} 2023-10-18T19:49:32.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000003e6ca003d283-00653036cc-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.498 DEBUG:boto:path=/bofrnf-1/ 2023-10-18T19:49:32.499 DEBUG:boto:auth_path=/bofrnf-1/ 2023-10-18T19:49:32.499 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.499 DEBUG:boto:Path: /bofrnf-1/ 2023-10-18T19:49:32.500 DEBUG:boto:Data: 2023-10-18T19:49:32.500 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.501 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:32.501 DEBUG:boto:Port: 8001 2023-10-18T19:49:32.502 DEBUG:boto:Params: {} 2023-10-18T19:49:32.502 DEBUG:boto:Token: None 2023-10-18T19:49:32.503 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-1/ 2023-10-18T19:49:32.503 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q397kXE9h9g0LwPOkyrm0qnhsmA= 2023-10-18T19:49:32.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q397kXE9h9g0LwPOkyrm0qnhsmA=', 'Content-Length': '0'} 2023-10-18T19:49:32.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000342908a7934aefb7-00653036cc-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.506 DEBUG:boto:path=/bofrnf-2/ 2023-10-18T19:49:32.506 DEBUG:boto:auth_path=/bofrnf-2/ 2023-10-18T19:49:32.507 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.507 DEBUG:boto:Path: /bofrnf-2/ 2023-10-18T19:49:32.508 DEBUG:boto:Data: 2023-10-18T19:49:32.508 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.509 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:32.509 DEBUG:boto:Port: 8001 2023-10-18T19:49:32.509 DEBUG:boto:Params: {} 2023-10-18T19:49:32.510 DEBUG:boto:Token: None 2023-10-18T19:49:32.510 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-2/ 2023-10-18T19:49:32.511 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:re32ADyBbd767yzEp0z09FMp2q0= 2023-10-18T19:49:32.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:re32ADyBbd767yzEp0z09FMp2q0=', 'Content-Length': '0'} 2023-10-18T19:49:32.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059411c391e44f745-00653036cc-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-10-18T19:49:32.522 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-3 2023-10-18T19:49:32.523 DEBUG:boto:path=/bofrnf-3/ 2023-10-18T19:49:32.523 DEBUG:boto:auth_path=/bofrnf-3/ 2023-10-18T19:49:32.524 DEBUG:boto:Method: PUT 2023-10-18T19:49:32.524 DEBUG:boto:Path: /bofrnf-3/ 2023-10-18T19:49:32.525 DEBUG:boto:Data: 2023-10-18T19:49:32.525 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:32.526 DEBUG:boto:Port: 8000 2023-10-18T19:49:32.526 DEBUG:boto:Params: {} 2023-10-18T19:49:32.527 DEBUG:boto:Token: None 2023-10-18T19:49:32.527 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-3/ 2023-10-18T19:49:32.527 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MJ+EKOul5ECToyPgIss1BNlxfVQ= 2023-10-18T19:49:32.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MJ+EKOul5ECToyPgIss1BNlxfVQ=', 'Content-Length': '0'} 2023-10-18T19:49:32.541 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3ddf9deb50a6f36-00653036cc-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.541 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-4 2023-10-18T19:49:32.542 DEBUG:boto:path=/bofrnf-4/ 2023-10-18T19:49:32.542 DEBUG:boto:auth_path=/bofrnf-4/ 2023-10-18T19:49:32.543 DEBUG:boto:Method: PUT 2023-10-18T19:49:32.543 DEBUG:boto:Path: /bofrnf-4/ 2023-10-18T19:49:32.543 DEBUG:boto:Data: 2023-10-18T19:49:32.544 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.544 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:32.545 DEBUG:boto:Port: 8001 2023-10-18T19:49:32.545 DEBUG:boto:Params: {} 2023-10-18T19:49:32.545 DEBUG:boto:Token: None 2023-10-18T19:49:32.546 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-4/ 2023-10-18T19:49:32.546 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:83Cu9mYNpXqbkN5qy7Q9RyciHMA= 2023-10-18T19:49:32.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:83Cu9mYNpXqbkN5qy7Q9RyciHMA=', 'Content-Length': '0'} 2023-10-18T19:49:32.572 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5e0dec425017867-00653036cc-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:32.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:32.574 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:32.625 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:32.626 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:32.661 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:49:32.674 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:49:32.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.675 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:32.675 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:47:22.596445Z" 2023-10-18T19:49:32.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.676 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658572.559870_84.1", 2023-10-18T19:49:32.677 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:32.559870Z" 2023-10-18T19:49:32.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.678 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:32.678 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:32.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.679 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:32.680 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:32.680 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.680 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:49:32.682 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697658442.596445_106.1', 1: '1_1697658572.559870_84.1', 2: '', 3: ''} 2023-10-18T19:49:32.683 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:49:32.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:32.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:32.685 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:32.734 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:32.735 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:32.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:32.759+0000 7f9c44d4d780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:32.762 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:32.760+0000 7f9c44d4d780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:32.773 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:32.773 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:32.774 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:32.774 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:32.775 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:32.775 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:32.775 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.776 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:32.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.777 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:32.777 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:32.777 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:32.778 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:32.778 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:32.778 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:32.779 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:32.779 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:32.780 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.780 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.781 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:32.782 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:32.782 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:32.783 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658572.559870_84.1", 2023-10-18T19:49:32.783 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:32.783 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:32.784 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:32.784 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:32.559870Z", 2023-10-18T19:49:32.785 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.785 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.786 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:32.786 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:32.787 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:32.787 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:32.788 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:32.788 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:32.788 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:32.789 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:32.789 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:32.791 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:32.791 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:32.792 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:32.792 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:32.792 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:32.793 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:32.793 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:32.793 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:32.794 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:32.794 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.795 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:32.795 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:32.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.796 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:32.796 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:32.797 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:32.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:32.797 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:32Z" 2023-10-18T19:49:32.798 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:32.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658442.596445_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:47:22.596445Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658572.559870_84.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:32.559870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:49:32Z" } 2023-10-18T19:49:32.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658442.596445_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:47:22.596445Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658572.559870_84.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:32.559870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:49:32.801 DEBUG:rgw_multi.tests:log_status={0: '1_1697658442.596445_106.1', 1: '1_1697658572.559870_84.1', 2: '', 3: ''} 2023-10-18T19:49:32.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658442.596445_106.1', 1: '1_1697658572.559870_84.1', 2: '', 3: ''} 2023-10-18T19:49:32.802 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:49:32.803 DEBUG:boto:path=/bofrnf-3/ 2023-10-18T19:49:32.804 DEBUG:boto:auth_path=/bofrnf-3/ 2023-10-18T19:49:32.804 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.805 DEBUG:boto:Path: /bofrnf-3/ 2023-10-18T19:49:32.806 DEBUG:boto:Data: 2023-10-18T19:49:32.806 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.807 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:32.808 DEBUG:boto:Port: 8000 2023-10-18T19:49:32.809 DEBUG:boto:Params: {} 2023-10-18T19:49:32.810 DEBUG:boto:Token: None 2023-10-18T19:49:32.810 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-3/ 2023-10-18T19:49:32.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Eqo56kd1om2E1Q1ngvC4yJgxQE4= 2023-10-18T19:49:32.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Eqo56kd1om2E1Q1ngvC4yJgxQE4=', 'Content-Length': '0'} 2023-10-18T19:49:32.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062693407480a4c35-00653036cc-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.816 DEBUG:boto:path=/bofrnf-4/ 2023-10-18T19:49:32.816 DEBUG:boto:auth_path=/bofrnf-4/ 2023-10-18T19:49:32.817 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.818 DEBUG:boto:Path: /bofrnf-4/ 2023-10-18T19:49:32.818 DEBUG:boto:Data: 2023-10-18T19:49:32.819 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.820 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:32.821 DEBUG:boto:Port: 8000 2023-10-18T19:49:32.821 DEBUG:boto:Params: {} 2023-10-18T19:49:32.822 DEBUG:boto:Token: None 2023-10-18T19:49:32.823 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-4/ 2023-10-18T19:49:32.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kSE/TPtBI+Rz8zJVMvrKdc1m+us= 2023-10-18T19:49:32.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kSE/TPtBI+Rz8zJVMvrKdc1m+us=', 'Content-Length': '0'} 2023-10-18T19:49:32.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8fb7e290a442d5f-00653036cc-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.828 DEBUG:boto:path=/bofrnf-3/ 2023-10-18T19:49:32.829 DEBUG:boto:auth_path=/bofrnf-3/ 2023-10-18T19:49:32.829 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.830 DEBUG:boto:Path: /bofrnf-3/ 2023-10-18T19:49:32.831 DEBUG:boto:Data: 2023-10-18T19:49:32.832 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.832 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:32.833 DEBUG:boto:Port: 8001 2023-10-18T19:49:32.834 DEBUG:boto:Params: {} 2023-10-18T19:49:32.835 DEBUG:boto:Token: None 2023-10-18T19:49:32.835 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-3/ 2023-10-18T19:49:32.836 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Eqo56kd1om2E1Q1ngvC4yJgxQE4= 2023-10-18T19:49:32.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Eqo56kd1om2E1Q1ngvC4yJgxQE4=', 'Content-Length': '0'} 2023-10-18T19:49:32.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e3bd27c1a025737-00653036cc-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.841 DEBUG:boto:path=/bofrnf-4/ 2023-10-18T19:49:32.842 DEBUG:boto:auth_path=/bofrnf-4/ 2023-10-18T19:49:32.842 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.843 DEBUG:boto:Path: /bofrnf-4/ 2023-10-18T19:49:32.844 DEBUG:boto:Data: 2023-10-18T19:49:32.845 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.845 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:32.846 DEBUG:boto:Port: 8001 2023-10-18T19:49:32.847 DEBUG:boto:Params: {} 2023-10-18T19:49:32.847 DEBUG:boto:Token: None 2023-10-18T19:49:32.848 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-4/ 2023-10-18T19:49:32.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kSE/TPtBI+Rz8zJVMvrKdc1m+us= 2023-10-18T19:49:32.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kSE/TPtBI+Rz8zJVMvrKdc1m+us=', 'Content-Length': '0'} 2023-10-18T19:49:32.851 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7f97071d2a8a772-00653036cc-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.851 DEBUG:boto:path=/bofrnf-3/ 2023-10-18T19:49:32.852 DEBUG:boto:auth_path=/bofrnf-3/ 2023-10-18T19:49:32.852 DEBUG:boto:Method: PUT 2023-10-18T19:49:32.853 DEBUG:boto:Path: /bofrnf-3/ 2023-10-18T19:49:32.853 DEBUG:boto:Data: 2023-10-18T19:49:32.854 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.854 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:32.854 DEBUG:boto:Port: 8000 2023-10-18T19:49:32.855 DEBUG:boto:Params: {} 2023-10-18T19:49:32.855 DEBUG:boto:Token: None 2023-10-18T19:49:32.856 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-3/ 2023-10-18T19:49:32.856 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MJ+EKOul5ECToyPgIss1BNlxfVQ= 2023-10-18T19:49:32.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MJ+EKOul5ECToyPgIss1BNlxfVQ=', 'Content-Length': '0'} 2023-10-18T19:49:32.859 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d2277f44f598c2e-00653036cc-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.860 DEBUG:boto:path=/bofrnf-4/ 2023-10-18T19:49:32.860 DEBUG:boto:auth_path=/bofrnf-4/ 2023-10-18T19:49:32.860 DEBUG:boto:Method: PUT 2023-10-18T19:49:32.861 DEBUG:boto:Path: /bofrnf-4/ 2023-10-18T19:49:32.861 DEBUG:boto:Data: 2023-10-18T19:49:32.862 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.862 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:32.862 DEBUG:boto:Port: 8000 2023-10-18T19:49:32.863 DEBUG:boto:Params: {} 2023-10-18T19:49:32.863 DEBUG:boto:Token: None 2023-10-18T19:49:32.864 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-4/ 2023-10-18T19:49:32.864 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:83Cu9mYNpXqbkN5qy7Q9RyciHMA= 2023-10-18T19:49:32.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:83Cu9mYNpXqbkN5qy7Q9RyciHMA=', 'Content-Length': '0'} 2023-10-18T19:49:32.867 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ce62bd462f68ec6b-00653036cc-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.868 DEBUG:boto:path=/bofrnf-3/ 2023-10-18T19:49:32.868 DEBUG:boto:auth_path=/bofrnf-3/ 2023-10-18T19:49:32.868 DEBUG:boto:Method: PUT 2023-10-18T19:49:32.869 DEBUG:boto:Path: /bofrnf-3/ 2023-10-18T19:49:32.869 DEBUG:boto:Data: 2023-10-18T19:49:32.869 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.870 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:32.870 DEBUG:boto:Port: 8001 2023-10-18T19:49:32.871 DEBUG:boto:Params: {} 2023-10-18T19:49:32.871 DEBUG:boto:Token: None 2023-10-18T19:49:32.871 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-3/ 2023-10-18T19:49:32.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MJ+EKOul5ECToyPgIss1BNlxfVQ= 2023-10-18T19:49:32.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MJ+EKOul5ECToyPgIss1BNlxfVQ=', 'Content-Length': '0'} 2023-10-18T19:49:32.877 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000175eef73d5f9ed27-00653036cc-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.877 DEBUG:boto:path=/bofrnf-4/ 2023-10-18T19:49:32.878 DEBUG:boto:auth_path=/bofrnf-4/ 2023-10-18T19:49:32.878 DEBUG:boto:Method: PUT 2023-10-18T19:49:32.879 DEBUG:boto:Path: /bofrnf-4/ 2023-10-18T19:49:32.879 DEBUG:boto:Data: 2023-10-18T19:49:32.879 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.880 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:32.880 DEBUG:boto:Port: 8001 2023-10-18T19:49:32.881 DEBUG:boto:Params: {} 2023-10-18T19:49:32.881 DEBUG:boto:Token: None 2023-10-18T19:49:32.881 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-4/ 2023-10-18T19:49:32.882 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:83Cu9mYNpXqbkN5qy7Q9RyciHMA= 2023-10-18T19:49:32.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:83Cu9mYNpXqbkN5qy7Q9RyciHMA=', 'Content-Length': '0'} 2023-10-18T19:49:32.928 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fdf60fab532920b5-00653036cc-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.928 DEBUG:boto:path=/bofrnf-3/ 2023-10-18T19:49:32.929 DEBUG:boto:auth_path=/bofrnf-3/ 2023-10-18T19:49:32.929 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.930 DEBUG:boto:Path: /bofrnf-3/ 2023-10-18T19:49:32.930 DEBUG:boto:Data: 2023-10-18T19:49:32.931 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:32.932 DEBUG:boto:Port: 8000 2023-10-18T19:49:32.932 DEBUG:boto:Params: {} 2023-10-18T19:49:32.932 DEBUG:boto:Token: None 2023-10-18T19:49:32.933 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-3/ 2023-10-18T19:49:32.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Eqo56kd1om2E1Q1ngvC4yJgxQE4= 2023-10-18T19:49:32.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Eqo56kd1om2E1Q1ngvC4yJgxQE4=', 'Content-Length': '0'} 2023-10-18T19:49:32.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b8ef5f35f2f7346-00653036cc-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.937 DEBUG:boto:path=/bofrnf-4/ 2023-10-18T19:49:32.937 DEBUG:boto:auth_path=/bofrnf-4/ 2023-10-18T19:49:32.937 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.938 DEBUG:boto:Path: /bofrnf-4/ 2023-10-18T19:49:32.938 DEBUG:boto:Data: 2023-10-18T19:49:32.939 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.939 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:32.939 DEBUG:boto:Port: 8000 2023-10-18T19:49:32.940 DEBUG:boto:Params: {} 2023-10-18T19:49:32.940 DEBUG:boto:Token: None 2023-10-18T19:49:32.941 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-4/ 2023-10-18T19:49:32.941 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kSE/TPtBI+Rz8zJVMvrKdc1m+us= 2023-10-18T19:49:32.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kSE/TPtBI+Rz8zJVMvrKdc1m+us=', 'Content-Length': '0'} 2023-10-18T19:49:32.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc13b919756f6292-00653036cc-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.945 DEBUG:boto:path=/bofrnf-3/ 2023-10-18T19:49:32.945 DEBUG:boto:auth_path=/bofrnf-3/ 2023-10-18T19:49:32.946 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.946 DEBUG:boto:Path: /bofrnf-3/ 2023-10-18T19:49:32.947 DEBUG:boto:Data: 2023-10-18T19:49:32.947 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.947 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:32.948 DEBUG:boto:Port: 8001 2023-10-18T19:49:32.948 DEBUG:boto:Params: {} 2023-10-18T19:49:32.949 DEBUG:boto:Token: None 2023-10-18T19:49:32.950 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-3/ 2023-10-18T19:49:32.950 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Eqo56kd1om2E1Q1ngvC4yJgxQE4= 2023-10-18T19:49:32.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Eqo56kd1om2E1Q1ngvC4yJgxQE4=', 'Content-Length': '0'} 2023-10-18T19:49:32.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b07ad80ae012d664-00653036cc-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.954 DEBUG:boto:path=/bofrnf-4/ 2023-10-18T19:49:32.954 DEBUG:boto:auth_path=/bofrnf-4/ 2023-10-18T19:49:32.955 DEBUG:boto:Method: HEAD 2023-10-18T19:49:32.955 DEBUG:boto:Path: /bofrnf-4/ 2023-10-18T19:49:32.955 DEBUG:boto:Data: 2023-10-18T19:49:32.956 DEBUG:boto:Headers: {} 2023-10-18T19:49:32.956 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:32.957 DEBUG:boto:Port: 8001 2023-10-18T19:49:32.957 DEBUG:boto:Params: {} 2023-10-18T19:49:32.957 DEBUG:boto:Token: None 2023-10-18T19:49:32.958 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:32 GMT /bofrnf-4/ 2023-10-18T19:49:32.958 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kSE/TPtBI+Rz8zJVMvrKdc1m+us= 2023-10-18T19:49:32.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kSE/TPtBI+Rz8zJVMvrKdc1m+us=', 'Content-Length': '0'} 2023-10-18T19:49:32.961 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb361d906ec8de33-00653036cc-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:32.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:32.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:32.962 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:33.011 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:33.011 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:33.038 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:49:33.049 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:49:33.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.050 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:33.050 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:47:22.596445Z" 2023-10-18T19:49:33.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.051 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658572.559870_84.1", 2023-10-18T19:49:33.051 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:32.559870Z" 2023-10-18T19:49:33.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.052 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:33.052 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:33.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.052 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:33.053 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:33.053 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.053 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:49:33.054 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697658442.596445_106.1', 1: '1_1697658572.559870_84.1', 2: '', 3: ''} 2023-10-18T19:49:33.055 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:49:33.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:33.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:33.056 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:33.105 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:33.106 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:33.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:33.133+0000 7fc428257780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:33.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:33.134+0000 7fc428257780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:33.146 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:33.147 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:33.147 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:33.147 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:33.147 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:33.147 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:33.148 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.148 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:33.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.149 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:33.149 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:33.149 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:33.149 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:33.149 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:33.150 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:33.150 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:33.150 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:33.150 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.150 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.151 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:33.151 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:33.152 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:33.152 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658572.559870_84.1", 2023-10-18T19:49:33.152 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:33.152 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:33.152 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:33.153 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:32.559870Z", 2023-10-18T19:49:33.153 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.153 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.154 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:33.154 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:33.154 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:33.154 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:33.154 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:33.155 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:33.155 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:33.155 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:33.155 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.156 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.156 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:33.156 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:33.157 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:33.157 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:33.157 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:33.157 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:33.157 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:33.158 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:33.158 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.158 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.158 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.158 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.160 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:33Z" 2023-10-18T19:49:33.160 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:33.161 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658442.596445_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:47:22.596445Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658572.559870_84.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:32.559870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:49:33Z" } 2023-10-18T19:49:33.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658442.596445_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:47:22.596445Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658572.559870_84.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:32.559870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:49:33.162 DEBUG:rgw_multi.tests:log_status={0: '1_1697658442.596445_106.1', 1: '1_1697658572.559870_84.1', 2: '', 3: ''} 2023-10-18T19:49:33.162 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658442.596445_106.1', 1: '1_1697658572.559870_84.1', 2: '', 3: ''} 2023-10-18T19:49:33.163 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:49:33.163 DEBUG:boto:path=/bofrnf-3/ 2023-10-18T19:49:33.164 DEBUG:boto:auth_path=/bofrnf-3/ 2023-10-18T19:49:33.164 DEBUG:boto:Method: HEAD 2023-10-18T19:49:33.164 DEBUG:boto:Path: /bofrnf-3/ 2023-10-18T19:49:33.165 DEBUG:boto:Data: 2023-10-18T19:49:33.165 DEBUG:boto:Headers: {} 2023-10-18T19:49:33.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:33.166 DEBUG:boto:Port: 8000 2023-10-18T19:49:33.166 DEBUG:boto:Params: {} 2023-10-18T19:49:33.167 DEBUG:boto:Token: None 2023-10-18T19:49:33.167 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:33 GMT /bofrnf-3/ 2023-10-18T19:49:33.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ofZg1f7eikx+Zuz09Pt1jUHYRyI= 2023-10-18T19:49:33.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ofZg1f7eikx+Zuz09Pt1jUHYRyI=', 'Content-Length': '0'} 2023-10-18T19:49:33.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099ecdba832e121a3-00653036cd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:33.171 DEBUG:boto:path=/bofrnf-4/ 2023-10-18T19:49:33.171 DEBUG:boto:auth_path=/bofrnf-4/ 2023-10-18T19:49:33.172 DEBUG:boto:Method: HEAD 2023-10-18T19:49:33.172 DEBUG:boto:Path: /bofrnf-4/ 2023-10-18T19:49:33.173 DEBUG:boto:Data: 2023-10-18T19:49:33.173 DEBUG:boto:Headers: {} 2023-10-18T19:49:33.173 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:33.174 DEBUG:boto:Port: 8000 2023-10-18T19:49:33.174 DEBUG:boto:Params: {} 2023-10-18T19:49:33.175 DEBUG:boto:Token: None 2023-10-18T19:49:33.175 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:33 GMT /bofrnf-4/ 2023-10-18T19:49:33.176 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+B+1IzlV8GAU5K/HUF5hb/VBPqE= 2023-10-18T19:49:33.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+B+1IzlV8GAU5K/HUF5hb/VBPqE=', 'Content-Length': '0'} 2023-10-18T19:49:33.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8efda647219fd8f-00653036cd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:33.179 DEBUG:boto:path=/bofrnf-3/ 2023-10-18T19:49:33.179 DEBUG:boto:auth_path=/bofrnf-3/ 2023-10-18T19:49:33.179 DEBUG:boto:Method: HEAD 2023-10-18T19:49:33.180 DEBUG:boto:Path: /bofrnf-3/ 2023-10-18T19:49:33.180 DEBUG:boto:Data: 2023-10-18T19:49:33.181 DEBUG:boto:Headers: {} 2023-10-18T19:49:33.181 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:33.181 DEBUG:boto:Port: 8001 2023-10-18T19:49:33.182 DEBUG:boto:Params: {} 2023-10-18T19:49:33.182 DEBUG:boto:Token: None 2023-10-18T19:49:33.183 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:33 GMT /bofrnf-3/ 2023-10-18T19:49:33.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ofZg1f7eikx+Zuz09Pt1jUHYRyI= 2023-10-18T19:49:33.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ofZg1f7eikx+Zuz09Pt1jUHYRyI=', 'Content-Length': '0'} 2023-10-18T19:49:33.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a3f5f43e413af01-00653036cd-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:33.186 DEBUG:boto:path=/bofrnf-4/ 2023-10-18T19:49:33.187 DEBUG:boto:auth_path=/bofrnf-4/ 2023-10-18T19:49:33.187 DEBUG:boto:Method: HEAD 2023-10-18T19:49:33.187 DEBUG:boto:Path: /bofrnf-4/ 2023-10-18T19:49:33.188 DEBUG:boto:Data: 2023-10-18T19:49:33.188 DEBUG:boto:Headers: {} 2023-10-18T19:49:33.189 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:33.189 DEBUG:boto:Port: 8001 2023-10-18T19:49:33.189 DEBUG:boto:Params: {} 2023-10-18T19:49:33.190 DEBUG:boto:Token: None 2023-10-18T19:49:33.190 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:33 GMT /bofrnf-4/ 2023-10-18T19:49:33.191 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+B+1IzlV8GAU5K/HUF5hb/VBPqE= 2023-10-18T19:49:33.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+B+1IzlV8GAU5K/HUF5hb/VBPqE=', 'Content-Length': '0'} 2023-10-18T19:49:33.193 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca117d565c34de43-00653036cd-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:33.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-10-18T19:49:33.195 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-5 2023-10-18T19:49:33.195 DEBUG:boto:path=/bofrnf-5/ 2023-10-18T19:49:33.195 DEBUG:boto:auth_path=/bofrnf-5/ 2023-10-18T19:49:33.196 DEBUG:boto:Method: PUT 2023-10-18T19:49:33.196 DEBUG:boto:Path: /bofrnf-5/ 2023-10-18T19:49:33.197 DEBUG:boto:Data: 2023-10-18T19:49:33.197 DEBUG:boto:Headers: {} 2023-10-18T19:49:33.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:33.198 DEBUG:boto:Port: 8000 2023-10-18T19:49:33.198 DEBUG:boto:Params: {} 2023-10-18T19:49:33.199 DEBUG:boto:Token: None 2023-10-18T19:49:33.199 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:33 GMT /bofrnf-5/ 2023-10-18T19:49:33.199 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XQgDQpbjOWlkdHotOU9pQrViLsg= 2023-10-18T19:49:33.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XQgDQpbjOWlkdHotOU9pQrViLsg=', 'Content-Length': '0'} 2023-10-18T19:49:33.217 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000059ca288c673100cf-00653036cd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:33.218 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-6 2023-10-18T19:49:33.218 DEBUG:boto:path=/bofrnf-6/ 2023-10-18T19:49:33.218 DEBUG:boto:auth_path=/bofrnf-6/ 2023-10-18T19:49:33.219 DEBUG:boto:Method: PUT 2023-10-18T19:49:33.219 DEBUG:boto:Path: /bofrnf-6/ 2023-10-18T19:49:33.220 DEBUG:boto:Data: 2023-10-18T19:49:33.220 DEBUG:boto:Headers: {} 2023-10-18T19:49:33.220 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:33.221 DEBUG:boto:Port: 8001 2023-10-18T19:49:33.221 DEBUG:boto:Params: {} 2023-10-18T19:49:33.222 DEBUG:boto:Token: None 2023-10-18T19:49:33.222 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:33 GMT /bofrnf-6/ 2023-10-18T19:49:33.223 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iU1caKrmQJ7oplGBoaifChN3LBs= 2023-10-18T19:49:33.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iU1caKrmQJ7oplGBoaifChN3LBs=', 'Content-Length': '0'} 2023-10-18T19:49:33.249 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000aa9e9de87cac267-00653036cd-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:33.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:33.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:33.251 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:33.299 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:33.300 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:33.329 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:49:33.341 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:49:33.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.341 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:33.342 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:47:22.596445Z" 2023-10-18T19:49:33.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.342 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658573.236271_92.1", 2023-10-18T19:49:33.343 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:33.236271Z" 2023-10-18T19:49:33.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.343 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:33.343 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:33.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.344 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:33.344 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:33.344 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.345 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:49:33.346 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697658442.596445_106.1', 1: '1_1697658573.236271_92.1', 2: '', 3: ''} 2023-10-18T19:49:33.346 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:49:33.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:33.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:33.347 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:33.396 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:33.396 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:33.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:33.426+0000 7f1ab3688780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:33.429 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:33.427+0000 7f1ab3688780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:33.440 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658573.208915_85.1", 2023-10-18T19:49:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:33.208915Z", 2023-10-18T19:49:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:33Z" 2023-10-18T19:49:33.454 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:33.455 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658442.596445_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:47:22.596445Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658573.208915_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:33.208915Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:49:33Z" } 2023-10-18T19:49:33.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658442.596445_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:47:22.596445Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658573.208915_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:33.208915Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:49:33.456 DEBUG:rgw_multi.tests:log_status={0: '1_1697658442.596445_106.1', 1: '1_1697658573.236271_92.1', 2: '', 3: ''} 2023-10-18T19:49:33.456 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658442.596445_106.1', 1: '1_1697658573.208915_85.1', 2: '', 3: ''} 2023-10-18T19:49:33.457 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697658573.236271_92.1 target=1_1697658573.208915_85.1 2023-10-18T19:49:38.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:38.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:38.459 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:38.516 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:38.516 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:38.547 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:38.545+0000 7efd038cc780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:38.548 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:38.547+0000 7efd038cc780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:38.560 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658573.236271_92.1", 2023-10-18T19:49:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:33.236271Z", 2023-10-18T19:49:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:38Z" 2023-10-18T19:49:38.574 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:38.575 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658442.596445_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:47:22.596445Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658573.236271_92.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:33.236271Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:49:38Z" } 2023-10-18T19:49:38.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658442.596445_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:47:22.596445Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658573.236271_92.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:33.236271Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:49:38.576 DEBUG:rgw_multi.tests:log_status={0: '1_1697658442.596445_106.1', 1: '1_1697658573.236271_92.1', 2: '', 3: ''} 2023-10-18T19:49:38.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658442.596445_106.1', 1: '1_1697658573.236271_92.1', 2: '', 3: ''} 2023-10-18T19:49:38.577 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:49:38.577 DEBUG:boto:path=/bofrnf-5/ 2023-10-18T19:49:38.578 DEBUG:boto:auth_path=/bofrnf-5/ 2023-10-18T19:49:38.578 DEBUG:boto:Method: HEAD 2023-10-18T19:49:38.579 DEBUG:boto:Path: /bofrnf-5/ 2023-10-18T19:49:38.579 DEBUG:boto:Data: 2023-10-18T19:49:38.580 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.580 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:38.580 DEBUG:boto:Port: 8000 2023-10-18T19:49:38.581 DEBUG:boto:Params: {} 2023-10-18T19:49:38.581 DEBUG:boto:Token: None 2023-10-18T19:49:38.582 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-5/ 2023-10-18T19:49:38.582 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YBy44ThJMML9w/iFq9T0oB8xvtk= 2023-10-18T19:49:38.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YBy44ThJMML9w/iFq9T0oB8xvtk=', 'Content-Length': '0'} 2023-10-18T19:49:38.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062dd701be4622819-00653036d2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.586 DEBUG:boto:path=/bofrnf-6/ 2023-10-18T19:49:38.586 DEBUG:boto:auth_path=/bofrnf-6/ 2023-10-18T19:49:38.587 DEBUG:boto:Method: HEAD 2023-10-18T19:49:38.587 DEBUG:boto:Path: /bofrnf-6/ 2023-10-18T19:49:38.587 DEBUG:boto:Data: 2023-10-18T19:49:38.588 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.588 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:38.589 DEBUG:boto:Port: 8000 2023-10-18T19:49:38.589 DEBUG:boto:Params: {} 2023-10-18T19:49:38.589 DEBUG:boto:Token: None 2023-10-18T19:49:38.590 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-6/ 2023-10-18T19:49:38.590 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:18dJDU+OI7AphEtvr88tXwJThaI= 2023-10-18T19:49:38.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:18dJDU+OI7AphEtvr88tXwJThaI=', 'Content-Length': '0'} 2023-10-18T19:49:38.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bea3fdfe066377ce-00653036d2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.593 DEBUG:boto:path=/bofrnf-5/ 2023-10-18T19:49:38.594 DEBUG:boto:auth_path=/bofrnf-5/ 2023-10-18T19:49:38.594 DEBUG:boto:Method: HEAD 2023-10-18T19:49:38.595 DEBUG:boto:Path: /bofrnf-5/ 2023-10-18T19:49:38.595 DEBUG:boto:Data: 2023-10-18T19:49:38.595 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.596 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:38.596 DEBUG:boto:Port: 8001 2023-10-18T19:49:38.597 DEBUG:boto:Params: {} 2023-10-18T19:49:38.597 DEBUG:boto:Token: None 2023-10-18T19:49:38.598 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-5/ 2023-10-18T19:49:38.598 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YBy44ThJMML9w/iFq9T0oB8xvtk= 2023-10-18T19:49:38.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YBy44ThJMML9w/iFq9T0oB8xvtk=', 'Content-Length': '0'} 2023-10-18T19:49:38.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000102aebada3023ae2-00653036d2-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.601 DEBUG:boto:path=/bofrnf-6/ 2023-10-18T19:49:38.602 DEBUG:boto:auth_path=/bofrnf-6/ 2023-10-18T19:49:38.602 DEBUG:boto:Method: HEAD 2023-10-18T19:49:38.602 DEBUG:boto:Path: /bofrnf-6/ 2023-10-18T19:49:38.603 DEBUG:boto:Data: 2023-10-18T19:49:38.603 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.604 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:38.604 DEBUG:boto:Port: 8001 2023-10-18T19:49:38.604 DEBUG:boto:Params: {} 2023-10-18T19:49:38.605 DEBUG:boto:Token: None 2023-10-18T19:49:38.605 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-6/ 2023-10-18T19:49:38.606 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:18dJDU+OI7AphEtvr88tXwJThaI= 2023-10-18T19:49:38.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:18dJDU+OI7AphEtvr88tXwJThaI=', 'Content-Length': '0'} 2023-10-18T19:49:38.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9ed7a00a41747ba-00653036d2-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.609 DEBUG:boto:path=/bofrnf-5/ 2023-10-18T19:49:38.609 DEBUG:boto:auth_path=/bofrnf-5/ 2023-10-18T19:49:38.610 DEBUG:boto:Method: DELETE 2023-10-18T19:49:38.610 DEBUG:boto:Path: /bofrnf-5/ 2023-10-18T19:49:38.610 DEBUG:boto:Data: 2023-10-18T19:49:38.611 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.611 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:38.611 DEBUG:boto:Port: 8000 2023-10-18T19:49:38.612 DEBUG:boto:Params: {} 2023-10-18T19:49:38.612 DEBUG:boto:Token: None 2023-10-18T19:49:38.613 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-5/ 2023-10-18T19:49:38.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LqYoLNXMNU+ZR7D8YKR9Nr6oG9k= 2023-10-18T19:49:38.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LqYoLNXMNU+ZR7D8YKR9Nr6oG9k=', 'Content-Length': '0'} 2023-10-18T19:49:38.639 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b0e43e09edbbcf86-00653036d2-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.640 DEBUG:boto:path=/bofrnf-6/ 2023-10-18T19:49:38.640 DEBUG:boto:auth_path=/bofrnf-6/ 2023-10-18T19:49:38.641 DEBUG:boto:Method: DELETE 2023-10-18T19:49:38.641 DEBUG:boto:Path: /bofrnf-6/ 2023-10-18T19:49:38.642 DEBUG:boto:Data: 2023-10-18T19:49:38.642 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.642 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:38.643 DEBUG:boto:Port: 8001 2023-10-18T19:49:38.643 DEBUG:boto:Params: {} 2023-10-18T19:49:38.644 DEBUG:boto:Token: None 2023-10-18T19:49:38.644 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-6/ 2023-10-18T19:49:38.645 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WJFo4OFYzOgPNQQBjqSL8WirDIo= 2023-10-18T19:49:38.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WJFo4OFYzOgPNQQBjqSL8WirDIo=', 'Content-Length': '0'} 2023-10-18T19:49:38.698 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000786260e5a4077836-00653036d2-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:38.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:38.699 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:38.749 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:38.750 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:38.776 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:49:38.787 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:49:38.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.787 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:38.787 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:47:22.596445Z" 2023-10-18T19:49:38.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.788 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658578.672290_100.1", 2023-10-18T19:49:38.788 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:38.672290Z" 2023-10-18T19:49:38.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.789 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:38.789 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:38.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.790 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:38.790 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:38.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.790 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:49:38.791 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697658442.596445_106.1', 1: '1_1697658578.672290_100.1', 2: '', 3: ''} 2023-10-18T19:49:38.792 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:49:38.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:38.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:38.793 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:38.846 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:38.847 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:38.877 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:38.875+0000 7f2f783ee780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:38.877 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:38.876+0000 7f2f783ee780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:38.889 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:38.890 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:38.890 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:38.890 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:38.890 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:38.891 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:38.891 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.891 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:38.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.892 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:38.892 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:38.892 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:38.893 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:38.893 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:38.893 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:38.893 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:38.893 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:38.894 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.894 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.895 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:38.895 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:38.895 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:38.895 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658578.672290_100.1", 2023-10-18T19:49:38.895 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:38.896 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:38.896 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:38.896 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:38.672290Z", 2023-10-18T19:49:38.896 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.896 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.897 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:38.897 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:38.898 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:38.898 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:38.898 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:38.898 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:38.898 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:38.899 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:38.899 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.899 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:38.900 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:38.900 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:38.900 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:38.900 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:38.901 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:38.901 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:38.901 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:38.901 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:38.901 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:38.902 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.902 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:38.902 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:38.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.903 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:38.903 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:38.903 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:38.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:38.903 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:38Z" 2023-10-18T19:49:38.904 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:38.904 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658442.596445_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:47:22.596445Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658578.672290_100.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:38.672290Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:49:38Z" } 2023-10-18T19:49:38.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658442.596445_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:47:22.596445Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658578.672290_100.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:38.672290Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:49:38.906 DEBUG:rgw_multi.tests:log_status={0: '1_1697658442.596445_106.1', 1: '1_1697658578.672290_100.1', 2: '', 3: ''} 2023-10-18T19:49:38.906 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658442.596445_106.1', 1: '1_1697658578.672290_100.1', 2: '', 3: ''} 2023-10-18T19:49:38.906 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:49:38.907 DEBUG:boto:path=/bofrnf-5/ 2023-10-18T19:49:38.907 DEBUG:boto:auth_path=/bofrnf-5/ 2023-10-18T19:49:38.908 DEBUG:boto:Method: HEAD 2023-10-18T19:49:38.908 DEBUG:boto:Path: /bofrnf-5/ 2023-10-18T19:49:38.908 DEBUG:boto:Data: 2023-10-18T19:49:38.909 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.909 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:38.910 DEBUG:boto:Port: 8000 2023-10-18T19:49:38.910 DEBUG:boto:Params: {} 2023-10-18T19:49:38.910 DEBUG:boto:Token: None 2023-10-18T19:49:38.911 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-5/ 2023-10-18T19:49:38.911 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YBy44ThJMML9w/iFq9T0oB8xvtk= 2023-10-18T19:49:38.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YBy44ThJMML9w/iFq9T0oB8xvtk=', 'Content-Length': '0'} 2023-10-18T19:49:38.913 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000001bfebbf3f1d08ee5-00653036d2-1112-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.914 DEBUG:boto:path=/bofrnf-6/ 2023-10-18T19:49:38.914 DEBUG:boto:auth_path=/bofrnf-6/ 2023-10-18T19:49:38.915 DEBUG:boto:Method: HEAD 2023-10-18T19:49:38.915 DEBUG:boto:Path: /bofrnf-6/ 2023-10-18T19:49:38.915 DEBUG:boto:Data: 2023-10-18T19:49:38.916 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.916 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:38.917 DEBUG:boto:Port: 8000 2023-10-18T19:49:38.917 DEBUG:boto:Params: {} 2023-10-18T19:49:38.917 DEBUG:boto:Token: None 2023-10-18T19:49:38.918 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-6/ 2023-10-18T19:49:38.918 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:18dJDU+OI7AphEtvr88tXwJThaI= 2023-10-18T19:49:38.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:18dJDU+OI7AphEtvr88tXwJThaI=', 'Content-Length': '0'} 2023-10-18T19:49:38.920 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000c9e49a61d717e442-00653036d2-1112-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.920 DEBUG:boto:path=/bofrnf-5/ 2023-10-18T19:49:38.921 DEBUG:boto:auth_path=/bofrnf-5/ 2023-10-18T19:49:38.921 DEBUG:boto:Method: HEAD 2023-10-18T19:49:38.921 DEBUG:boto:Path: /bofrnf-5/ 2023-10-18T19:49:38.922 DEBUG:boto:Data: 2023-10-18T19:49:38.922 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:38.924 DEBUG:boto:Port: 8001 2023-10-18T19:49:38.924 DEBUG:boto:Params: {} 2023-10-18T19:49:38.924 DEBUG:boto:Token: None 2023-10-18T19:49:38.925 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-5/ 2023-10-18T19:49:38.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YBy44ThJMML9w/iFq9T0oB8xvtk= 2023-10-18T19:49:38.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YBy44ThJMML9w/iFq9T0oB8xvtk=', 'Content-Length': '0'} 2023-10-18T19:49:38.928 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000ff77bf089f62abc8-00653036d2-1116-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.928 DEBUG:boto:path=/bofrnf-6/ 2023-10-18T19:49:38.929 DEBUG:boto:auth_path=/bofrnf-6/ 2023-10-18T19:49:38.929 DEBUG:boto:Method: HEAD 2023-10-18T19:49:38.930 DEBUG:boto:Path: /bofrnf-6/ 2023-10-18T19:49:38.930 DEBUG:boto:Data: 2023-10-18T19:49:38.931 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:38.931 DEBUG:boto:Port: 8001 2023-10-18T19:49:38.932 DEBUG:boto:Params: {} 2023-10-18T19:49:38.932 DEBUG:boto:Token: None 2023-10-18T19:49:38.933 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-6/ 2023-10-18T19:49:38.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:18dJDU+OI7AphEtvr88tXwJThaI= 2023-10-18T19:49:38.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:18dJDU+OI7AphEtvr88tXwJThaI=', 'Content-Length': '0'} 2023-10-18T19:49:38.935 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000085375308517d6c7f-00653036d2-1116-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-10-18T19:49:38.936 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-7 2023-10-18T19:49:38.936 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:49:38.937 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:49:38.937 DEBUG:boto:Method: PUT 2023-10-18T19:49:38.937 DEBUG:boto:Path: /bofrnf-7/ 2023-10-18T19:49:38.938 DEBUG:boto:Data: 2023-10-18T19:49:38.938 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.939 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:38.939 DEBUG:boto:Port: 8000 2023-10-18T19:49:38.939 DEBUG:boto:Params: {} 2023-10-18T19:49:38.940 DEBUG:boto:Token: None 2023-10-18T19:49:38.940 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-7/ 2023-10-18T19:49:38.941 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zZU+okD1CCOsUWtwqznsIWJmoIo= 2023-10-18T19:49:38.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zZU+okD1CCOsUWtwqznsIWJmoIo=', 'Content-Length': '0'} 2023-10-18T19:49:38.959 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4e8899a29a55173-00653036d2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.959 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-8 2023-10-18T19:49:38.960 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:49:38.960 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:49:38.961 DEBUG:boto:Method: PUT 2023-10-18T19:49:38.961 DEBUG:boto:Path: /bofrnf-8/ 2023-10-18T19:49:38.961 DEBUG:boto:Data: 2023-10-18T19:49:38.962 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.962 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:38.963 DEBUG:boto:Port: 8001 2023-10-18T19:49:38.963 DEBUG:boto:Params: {} 2023-10-18T19:49:38.964 DEBUG:boto:Token: None 2023-10-18T19:49:38.964 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-8/ 2023-10-18T19:49:38.964 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TOkzLNliiT254EvQWRXpD1L4aLw= 2023-10-18T19:49:38.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TOkzLNliiT254EvQWRXpD1L4aLw=', 'Content-Length': '0'} 2023-10-18T19:49:38.988 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c203ddef0fb2cb95-00653036d2-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.989 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:49:38.989 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:49:38.989 DEBUG:boto:Method: HEAD 2023-10-18T19:49:38.990 DEBUG:boto:Path: /bofrnf-7/ 2023-10-18T19:49:38.990 DEBUG:boto:Data: 2023-10-18T19:49:38.991 DEBUG:boto:Headers: {} 2023-10-18T19:49:38.991 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:38.991 DEBUG:boto:Port: 8000 2023-10-18T19:49:38.992 DEBUG:boto:Params: {} 2023-10-18T19:49:38.992 DEBUG:boto:Token: None 2023-10-18T19:49:38.993 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:38 GMT /bofrnf-7/ 2023-10-18T19:49:38.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pEOyuFaLU1ulfnwiwpt47nWH4PU= 2023-10-18T19:49:38.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pEOyuFaLU1ulfnwiwpt47nWH4PU=', 'Content-Length': '0'} 2023-10-18T19:49:38.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028ac5acff44b5d24-00653036d2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:38.996 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:49:39.023 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:49:39.024 DEBUG:boto:Method: PUT 2023-10-18T19:49:39.024 DEBUG:boto:Path: /bofrnf-7/myobj 2023-10-18T19:49:39.024 DEBUG:boto:Data: 2023-10-18T19:49:39.025 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:49:39.025 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:39.025 DEBUG:boto:Port: 8000 2023-10-18T19:49:39.026 DEBUG:boto:Params: {} 2023-10-18T19:49:39.026 DEBUG:boto:Token: None 2023-10-18T19:49:39.027 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-7/myobj 2023-10-18T19:49:39.027 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ywsWGG62AQkwbCbGKqPVkJ+LDw0= 2023-10-18T19:49:39.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ywsWGG62AQkwbCbGKqPVkJ+LDw0='} 2023-10-18T19:49:39.037 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000403a2037cf4aff04-00653036d3-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.038 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:49:39.038 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:49:39.038 DEBUG:boto:Method: HEAD 2023-10-18T19:49:39.039 DEBUG:boto:Path: /bofrnf-7/ 2023-10-18T19:49:39.039 DEBUG:boto:Data: 2023-10-18T19:49:39.040 DEBUG:boto:Headers: {} 2023-10-18T19:49:39.040 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:39.040 DEBUG:boto:Port: 8000 2023-10-18T19:49:39.041 DEBUG:boto:Params: {} 2023-10-18T19:49:39.041 DEBUG:boto:Token: None 2023-10-18T19:49:39.042 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-7/ 2023-10-18T19:49:39.042 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lzsKKVnKTPUd3FMx9xfmPVp0vAs= 2023-10-18T19:49:39.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lzsKKVnKTPUd3FMx9xfmPVp0vAs=', 'Content-Length': '0'} 2023-10-18T19:49:39.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000843b1366f403a3f3-00653036d3-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.047 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:49:39.047 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:49:39.047 DEBUG:boto:Method: PUT 2023-10-18T19:49:39.048 DEBUG:boto:Path: /bofrnf-7/_myobj 2023-10-18T19:49:39.048 DEBUG:boto:Data: 2023-10-18T19:49:39.049 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:49:39.049 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:39.049 DEBUG:boto:Port: 8000 2023-10-18T19:49:39.050 DEBUG:boto:Params: {} 2023-10-18T19:49:39.050 DEBUG:boto:Token: None 2023-10-18T19:49:39.051 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-7/_myobj 2023-10-18T19:49:39.051 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7X1XkiOpRIc97t8QQQZntSZ9Zc= 2023-10-18T19:49:39.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7X1XkiOpRIc97t8QQQZntSZ9Zc='} 2023-10-18T19:49:39.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e5be3dfec2d3831-00653036d3-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.095 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:49:39.095 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:49:39.096 DEBUG:boto:Method: HEAD 2023-10-18T19:49:39.096 DEBUG:boto:Path: /bofrnf-7/ 2023-10-18T19:49:39.096 DEBUG:boto:Data: 2023-10-18T19:49:39.097 DEBUG:boto:Headers: {} 2023-10-18T19:49:39.097 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:39.097 DEBUG:boto:Port: 8000 2023-10-18T19:49:39.098 DEBUG:boto:Params: {} 2023-10-18T19:49:39.098 DEBUG:boto:Token: None 2023-10-18T19:49:39.099 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-7/ 2023-10-18T19:49:39.099 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lzsKKVnKTPUd3FMx9xfmPVp0vAs= 2023-10-18T19:49:39.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lzsKKVnKTPUd3FMx9xfmPVp0vAs=', 'Content-Length': '0'} 2023-10-18T19:49:39.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005250c53120426e31-00653036d3-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.103 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:49:39.103 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:49:39.103 DEBUG:boto:Method: PUT 2023-10-18T19:49:39.104 DEBUG:boto:Path: /bofrnf-7/%3A 2023-10-18T19:49:39.104 DEBUG:boto:Data: 2023-10-18T19:49:39.104 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:49:39.105 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:39.105 DEBUG:boto:Port: 8000 2023-10-18T19:49:39.106 DEBUG:boto:Params: {} 2023-10-18T19:49:39.106 DEBUG:boto:Token: None 2023-10-18T19:49:39.106 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-7/%3A 2023-10-18T19:49:39.107 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MfswqPGANLfEd4u+GSO60pbb4Cc= 2023-10-18T19:49:39.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MfswqPGANLfEd4u+GSO60pbb4Cc='} 2023-10-18T19:49:39.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c262db5bc6d692d3-00653036d3-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.120 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:49:39.120 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:49:39.121 DEBUG:boto:Method: HEAD 2023-10-18T19:49:39.121 DEBUG:boto:Path: /bofrnf-7/ 2023-10-18T19:49:39.121 DEBUG:boto:Data: 2023-10-18T19:49:39.122 DEBUG:boto:Headers: {} 2023-10-18T19:49:39.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:39.123 DEBUG:boto:Port: 8000 2023-10-18T19:49:39.123 DEBUG:boto:Params: {} 2023-10-18T19:49:39.123 DEBUG:boto:Token: None 2023-10-18T19:49:39.124 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-7/ 2023-10-18T19:49:39.124 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lzsKKVnKTPUd3FMx9xfmPVp0vAs= 2023-10-18T19:49:39.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lzsKKVnKTPUd3FMx9xfmPVp0vAs=', 'Content-Length': '0'} 2023-10-18T19:49:39.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c7df0130e437a24-00653036d3-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.127 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:49:39.128 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:49:39.128 DEBUG:boto:Method: PUT 2023-10-18T19:49:39.129 DEBUG:boto:Path: /bofrnf-7/%26 2023-10-18T19:49:39.129 DEBUG:boto:Data: 2023-10-18T19:49:39.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:49:39.130 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:49:39.130 DEBUG:boto:Port: 8000 2023-10-18T19:49:39.131 DEBUG:boto:Params: {} 2023-10-18T19:49:39.131 DEBUG:boto:Token: None 2023-10-18T19:49:39.132 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-7/%26 2023-10-18T19:49:39.132 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+pHxlniXSSwaiEYeaZkOtNSFmmQ= 2023-10-18T19:49:39.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+pHxlniXSSwaiEYeaZkOtNSFmmQ='} 2023-10-18T19:49:39.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abb64916ea1aa2f8-00653036d3-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.175 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:49:39.176 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:49:39.176 DEBUG:boto:Method: HEAD 2023-10-18T19:49:39.176 DEBUG:boto:Path: /bofrnf-8/ 2023-10-18T19:49:39.177 DEBUG:boto:Data: 2023-10-18T19:49:39.177 DEBUG:boto:Headers: {} 2023-10-18T19:49:39.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:39.178 DEBUG:boto:Port: 8001 2023-10-18T19:49:39.178 DEBUG:boto:Params: {} 2023-10-18T19:49:39.179 DEBUG:boto:Token: None 2023-10-18T19:49:39.179 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-8/ 2023-10-18T19:49:39.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lIf6tn0sAQbU60+CuqP8EeDzfVw= 2023-10-18T19:49:39.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lIf6tn0sAQbU60+CuqP8EeDzfVw=', 'Content-Length': '0'} 2023-10-18T19:49:39.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062d8201b2ae8c66f-00653036d3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.183 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:49:39.183 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:49:39.184 DEBUG:boto:Method: PUT 2023-10-18T19:49:39.184 DEBUG:boto:Path: /bofrnf-8/myobj 2023-10-18T19:49:39.185 DEBUG:boto:Data: 2023-10-18T19:49:39.185 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:49:39.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:39.186 DEBUG:boto:Port: 8001 2023-10-18T19:49:39.186 DEBUG:boto:Params: {} 2023-10-18T19:49:39.187 DEBUG:boto:Token: None 2023-10-18T19:49:39.187 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-8/myobj 2023-10-18T19:49:39.188 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:frO3lx/L+zRg7G0EV1e1+gvxyj0= 2023-10-18T19:49:39.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:frO3lx/L+zRg7G0EV1e1+gvxyj0='} 2023-10-18T19:49:39.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011616a11df7032ae-00653036d3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.200 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:49:39.200 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:49:39.201 DEBUG:boto:Method: HEAD 2023-10-18T19:49:39.201 DEBUG:boto:Path: /bofrnf-8/ 2023-10-18T19:49:39.202 DEBUG:boto:Data: 2023-10-18T19:49:39.202 DEBUG:boto:Headers: {} 2023-10-18T19:49:39.202 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:39.203 DEBUG:boto:Port: 8001 2023-10-18T19:49:39.203 DEBUG:boto:Params: {} 2023-10-18T19:49:39.204 DEBUG:boto:Token: None 2023-10-18T19:49:39.204 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-8/ 2023-10-18T19:49:39.205 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lIf6tn0sAQbU60+CuqP8EeDzfVw= 2023-10-18T19:49:39.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lIf6tn0sAQbU60+CuqP8EeDzfVw=', 'Content-Length': '0'} 2023-10-18T19:49:39.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b647813347dc26b-00653036d3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.208 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:49:39.208 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:49:39.208 DEBUG:boto:Method: PUT 2023-10-18T19:49:39.209 DEBUG:boto:Path: /bofrnf-8/_myobj 2023-10-18T19:49:39.209 DEBUG:boto:Data: 2023-10-18T19:49:39.210 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:49:39.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:39.211 DEBUG:boto:Port: 8001 2023-10-18T19:49:39.211 DEBUG:boto:Params: {} 2023-10-18T19:49:39.212 DEBUG:boto:Token: None 2023-10-18T19:49:39.212 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-8/_myobj 2023-10-18T19:49:39.212 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MIHbkNbBD2sIsHhXPPgmHZJpkQw= 2023-10-18T19:49:39.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MIHbkNbBD2sIsHhXPPgmHZJpkQw='} 2023-10-18T19:49:39.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c6f818167826b67-00653036d3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.259 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:49:39.260 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:49:39.260 DEBUG:boto:Method: HEAD 2023-10-18T19:49:39.261 DEBUG:boto:Path: /bofrnf-8/ 2023-10-18T19:49:39.261 DEBUG:boto:Data: 2023-10-18T19:49:39.261 DEBUG:boto:Headers: {} 2023-10-18T19:49:39.262 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:39.262 DEBUG:boto:Port: 8001 2023-10-18T19:49:39.263 DEBUG:boto:Params: {} 2023-10-18T19:49:39.263 DEBUG:boto:Token: None 2023-10-18T19:49:39.264 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-8/ 2023-10-18T19:49:39.264 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lIf6tn0sAQbU60+CuqP8EeDzfVw= 2023-10-18T19:49:39.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lIf6tn0sAQbU60+CuqP8EeDzfVw=', 'Content-Length': '0'} 2023-10-18T19:49:39.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba0091590e8d5ab4-00653036d3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.268 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:49:39.268 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:49:39.269 DEBUG:boto:Method: PUT 2023-10-18T19:49:39.269 DEBUG:boto:Path: /bofrnf-8/%3A 2023-10-18T19:49:39.269 DEBUG:boto:Data: 2023-10-18T19:49:39.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:49:39.270 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:39.271 DEBUG:boto:Port: 8001 2023-10-18T19:49:39.271 DEBUG:boto:Params: {} 2023-10-18T19:49:39.272 DEBUG:boto:Token: None 2023-10-18T19:49:39.272 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-8/%3A 2023-10-18T19:49:39.273 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KIJ+0A4Gk37+iFIdTUz+X0kdEww= 2023-10-18T19:49:39.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KIJ+0A4Gk37+iFIdTUz+X0kdEww='} 2023-10-18T19:49:39.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032861d3e3bbc26b0-00653036d3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.285 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:49:39.285 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:49:39.285 DEBUG:boto:Method: HEAD 2023-10-18T19:49:39.286 DEBUG:boto:Path: /bofrnf-8/ 2023-10-18T19:49:39.286 DEBUG:boto:Data: 2023-10-18T19:49:39.287 DEBUG:boto:Headers: {} 2023-10-18T19:49:39.287 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:39.288 DEBUG:boto:Port: 8001 2023-10-18T19:49:39.288 DEBUG:boto:Params: {} 2023-10-18T19:49:39.288 DEBUG:boto:Token: None 2023-10-18T19:49:39.289 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-8/ 2023-10-18T19:49:39.289 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lIf6tn0sAQbU60+CuqP8EeDzfVw= 2023-10-18T19:49:39.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lIf6tn0sAQbU60+CuqP8EeDzfVw=', 'Content-Length': '0'} 2023-10-18T19:49:39.293 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006887776209415bca-00653036d3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.293 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:49:39.294 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:49:39.294 DEBUG:boto:Method: PUT 2023-10-18T19:49:39.295 DEBUG:boto:Path: /bofrnf-8/%26 2023-10-18T19:49:39.295 DEBUG:boto:Data: 2023-10-18T19:49:39.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:49:39.296 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:49:39.296 DEBUG:boto:Port: 8001 2023-10-18T19:49:39.297 DEBUG:boto:Params: {} 2023-10-18T19:49:39.297 DEBUG:boto:Token: None 2023-10-18T19:49:39.298 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:49:39 GMT /bofrnf-8/%26 2023-10-18T19:49:39.298 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2jL9mQApOCh6Mb8SlVF3ddNRjL0= 2023-10-18T19:49:39.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:49:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2jL9mQApOCh6Mb8SlVF3ddNRjL0='} 2023-10-18T19:49:39.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd8854f44ce51fe6-00653036d3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:49:39.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:39.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:39.344 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:39.394 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:39.394 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:39.421 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:49:39.432 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:49:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:47:22.596445Z" 2023-10-18T19:49:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658578.977815_108.1", 2023-10-18T19:49:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:38.977815Z" 2023-10-18T19:49:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:49:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:39.436 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:49:39.437 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697658442.596445_106.1', 1: '1_1697658578.977815_108.1', 2: '', 3: ''} 2023-10-18T19:49:39.437 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:49:39.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:39.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:49:39.439 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:49:39.487 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:39.488 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:39.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:39.512+0000 7fde6edeb780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:39.514 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:39.513+0000 7fde6edeb780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:49:39.525 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:49:39.526 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:49:39.526 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:49:39.526 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:49:39.526 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:49:39.527 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:49:39.527 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:39.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:39.527 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:49:39.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:39.528 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:49:39.528 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:39.528 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:39.528 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658442.596445_106.1", 2023-10-18T19:49:39.529 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:39.529 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:49:39.529 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:39.529 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:47:22.596445Z", 2023-10-18T19:49:39.529 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:39.530 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:39.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:39.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:39.530 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:49:39.530 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:39.531 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:39.531 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658578.977815_108.1", 2023-10-18T19:49:39.531 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:39.531 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:39.532 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:39.532 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:38.977815Z", 2023-10-18T19:49:39.532 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:39.532 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:39.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:39.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:39.533 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:49:39.533 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:39.533 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:39.533 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:39.534 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:39.534 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:39.534 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:39.534 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:39.535 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:39.535 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:39.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:39.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:49:39.535 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:49:39.536 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:49:39.536 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:49:39.536 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:49:39.536 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:49:39.536 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:49:39.537 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:49:39.537 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:49:39.537 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:49:39.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:39.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:49:39.538 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:49:39.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:39.538 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:49:39.538 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:49:39.538 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:49:39.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:49:39.539 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:49:39Z" 2023-10-18T19:49:39.539 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:49:39.540 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658442.596445_106.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:47:22.596445Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658578.977815_108.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:38.977815Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:49:39Z" } 2023-10-18T19:49:39.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658442.596445_106.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:47:22.596445Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658578.977815_108.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:38.977815Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:49:39.541 DEBUG:rgw_multi.tests:log_status={0: '1_1697658442.596445_106.1', 1: '1_1697658578.977815_108.1', 2: '', 3: ''} 2023-10-18T19:49:39.542 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658442.596445_106.1', 1: '1_1697658578.977815_108.1', 2: '', 3: ''} 2023-10-18T19:49:39.542 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:49:39.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:49:39.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:49:39.543 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:49:39.599 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:49:39.600 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:49:39.633 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:39.631+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:49:40.421 INFO:tasks.rgw.c2.client.0.smithi156.stdout:2023-10-18T19:49:40.418+0000 7f29dd326700 -1 rgw async rados processor: could not delete bucket=bofrnf-5 2023-10-18T19:49:40.428 INFO:tasks.rgw.c2.client.0.smithi156.stdout:2023-10-18T19:49:40.426+0000 7f29e9b3f700 -1 rgw async rados processor: could not delete bucket=bofrnf-6 2023-10-18T19:49:44.634 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:44.632+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:49:49.634 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:49.632+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:49:54.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:54.632+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:49:59.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:49:59.633+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:04.637 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:04.634+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:09.637 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:09.635+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:14.638 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:14.636+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:19.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:19.637+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:24.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:24.637+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:29.640 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:29.638+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:34.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:34.639+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:39.642 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:39.640+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:44.642 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:44.640+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:49.640+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:54.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:54.641+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:50:59.644 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:50:59.641+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:04.645 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:04.643+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:09.645 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:09.643+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:14.646 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:14.644+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:19.646 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:19.644+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:24.647 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:24.645+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:29.647 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:29.646+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:34.648 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:34.646+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:39.649 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:39.647+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:44.649 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:44.646+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:49.647+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:54.651 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:54.648+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:51:59.651 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:51:59.649+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:04.652 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:04.650+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:09.652 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:09.650+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:14.653 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:14.651+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:19.653 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:19.652+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:24.654 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:24.652+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:29.655 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:29.653+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:34.655 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:34.652+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:39.656 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:39.653+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:44.656 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:44.654+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:49.657 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:49.655+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:54.658 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:54.656+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:52:59.658 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:52:59.656+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:04.659 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:04.657+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:09.660 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:09.658+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:14.660 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:14.658+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:19.661 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:19.659+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:24.662 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:24.660+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:29.662 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:29.660+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:34.663 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:34.660+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:39.663 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:39.661+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:44.664 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:44.662+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:49.665 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:49.663+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:54.666 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:54.664+0000 7ffb4d136780 1 waiting to reach incremental sync.. 2023-10-18T19:53:59.667 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:59.666+0000 7ffb4d136780 1 bucket sync caught up with source: 2023-10-18T19:53:59.668 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.3.6, , 00000000001.6.6, , , , 00000000001.5.6, 00000000001.5.6, , ] 2023-10-18T19:53:59.668 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.3.6, , 00000000001.6.6, , , , 00000000001.5.6, 00000000001.5.6, , ] 2023-10-18T19:53:59.668 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:53:59.666+0000 7ffb4d136780 0 bucket checkpoint complete 2023-10-18T19:53:59.683 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-7 zones={a2, a1} 2023-10-18T19:53:59.684 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.684 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.685 DEBUG:boto:Method: HEAD 2023-10-18T19:53:59.685 DEBUG:boto:Path: /bofrnf-7/ 2023-10-18T19:53:59.685 DEBUG:boto:Data: 2023-10-18T19:53:59.686 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.687 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.687 DEBUG:boto:Params: {} 2023-10-18T19:53:59.688 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T19:53:59.688 DEBUG:boto:Token: None 2023-10-18T19:53:59.689 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/ 2023-10-18T19:53:59.689 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hliToppIUO/o4byXm+HXxem20po= 2023-10-18T19:53:59.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hliToppIUO/o4byXm+HXxem20po=', 'Content-Length': '0'} 2023-10-18T19:53:59.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e914be057d3f7f4e-00653037d7-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.695 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.696 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.696 DEBUG:boto:Method: HEAD 2023-10-18T19:53:59.696 DEBUG:boto:Path: /bofrnf-7/ 2023-10-18T19:53:59.697 DEBUG:boto:Data: 2023-10-18T19:53:59.697 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.698 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.698 DEBUG:boto:Params: {} 2023-10-18T19:53:59.699 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T19:53:59.699 DEBUG:boto:Token: None 2023-10-18T19:53:59.700 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/ 2023-10-18T19:53:59.700 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hliToppIUO/o4byXm+HXxem20po= 2023-10-18T19:53:59.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hliToppIUO/o4byXm+HXxem20po=', 'Content-Length': '0'} 2023-10-18T19:53:59.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcd5246fe7454b61-00653037d7-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.705 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:53:59.705 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.753 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.754 DEBUG:boto:path=/bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.754 DEBUG:boto:auth_path=/bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.755 DEBUG:boto:Method: GET 2023-10-18T19:53:59.755 DEBUG:boto:Path: /bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.755 DEBUG:boto:Data: 2023-10-18T19:53:59.756 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.756 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.757 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.757 DEBUG:boto:Params: {} 2023-10-18T19:53:59.758 DEBUG:boto:Token: None 2023-10-18T19:53:59.758 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/?versions 2023-10-18T19:53:59.758 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jZDun91zhuMcyjNZxZVMaCFnONc= 2023-10-18T19:53:59.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jZDun91zhuMcyjNZxZVMaCFnONc=', 'Content-Length': '0'} 2023-10-18T19:53:59.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd88de10133615ff-00653037d7-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.762 DEBUG:boto:b'bofrnf-7999false&nulltrue2023-10-18T19:49:39.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-18T19:49:39.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-18T19:49:39.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-18T19:49:39.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:53:59.764 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-18T19:53:59.765 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-18T19:53:59.765 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-18T19:53:59.765 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-18T19:53:59.766 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:53:59.766 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.767 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.767 DEBUG:boto:path=/bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.768 DEBUG:boto:auth_path=/bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.768 DEBUG:boto:Method: GET 2023-10-18T19:53:59.768 DEBUG:boto:Path: /bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.769 DEBUG:boto:Data: 2023-10-18T19:53:59.769 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.770 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.770 DEBUG:boto:Params: {} 2023-10-18T19:53:59.771 DEBUG:boto:Token: None 2023-10-18T19:53:59.771 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/?versions 2023-10-18T19:53:59.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jZDun91zhuMcyjNZxZVMaCFnONc= 2023-10-18T19:53:59.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jZDun91zhuMcyjNZxZVMaCFnONc=', 'Content-Length': '0'} 2023-10-18T19:53:59.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dcdfaaca4d2743c-00653037d7-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.775 DEBUG:boto:b'bofrnf-7999false&nulltrue2023-10-18T19:49:39.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-18T19:49:39.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-18T19:49:39.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-18T19:49:39.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:53:59.776 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-18T19:53:59.776 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-18T19:53:59.776 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-18T19:53:59.777 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-18T19:53:59.777 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.778 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.778 DEBUG:boto:path=/bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.779 DEBUG:boto:auth_path=/bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.779 DEBUG:boto:Method: GET 2023-10-18T19:53:59.779 DEBUG:boto:Path: /bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.780 DEBUG:boto:Data: 2023-10-18T19:53:59.780 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.781 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.781 DEBUG:boto:Params: {} 2023-10-18T19:53:59.782 DEBUG:boto:Token: None 2023-10-18T19:53:59.782 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/?versions 2023-10-18T19:53:59.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jZDun91zhuMcyjNZxZVMaCFnONc= 2023-10-18T19:53:59.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jZDun91zhuMcyjNZxZVMaCFnONc=', 'Content-Length': '0'} 2023-10-18T19:53:59.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa883e9c836eead9-00653037d7-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.786 DEBUG:boto:b'bofrnf-7999false&nulltrue2023-10-18T19:49:39.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-18T19:49:39.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-18T19:49:39.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-18T19:49:39.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:53:59.787 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.787 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.788 DEBUG:boto:path=/bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.788 DEBUG:boto:auth_path=/bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.789 DEBUG:boto:Method: GET 2023-10-18T19:53:59.789 DEBUG:boto:Path: /bofrnf-7/?versions&max-keys=999 2023-10-18T19:53:59.789 DEBUG:boto:Data: 2023-10-18T19:53:59.790 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.791 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.791 DEBUG:boto:Params: {} 2023-10-18T19:53:59.792 DEBUG:boto:Token: None 2023-10-18T19:53:59.792 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/?versions 2023-10-18T19:53:59.793 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jZDun91zhuMcyjNZxZVMaCFnONc= 2023-10-18T19:53:59.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jZDun91zhuMcyjNZxZVMaCFnONc=', 'Content-Length': '0'} 2023-10-18T19:53:59.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004644d59c0935e1c4-00653037d7-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.796 DEBUG:boto:b'bofrnf-7999false&nulltrue2023-10-18T19:49:39.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-18T19:49:39.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-18T19:49:39.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-18T19:49:39.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:53:59.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-18T19:53:59.797 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.798 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.798 DEBUG:boto:path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.798 DEBUG:boto:auth_path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.799 DEBUG:boto:Method: GET 2023-10-18T19:53:59.799 DEBUG:boto:Path: /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.800 DEBUG:boto:Data: 2023-10-18T19:53:59.800 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.800 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.801 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.801 DEBUG:boto:Params: {} 2023-10-18T19:53:59.802 DEBUG:boto:Token: None 2023-10-18T19:53:59.802 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.803 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jcqhQG3JsTplQA4oAHAX629Fc3w= 2023-10-18T19:53:59.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jcqhQG3JsTplQA4oAHAX629Fc3w=', 'Content-Length': '0'} 2023-10-18T19:53:59.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090773084c04a752c-00653037d7-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.806 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.807 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.807 DEBUG:boto:path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.807 DEBUG:boto:auth_path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.808 DEBUG:boto:Method: GET 2023-10-18T19:53:59.808 DEBUG:boto:Path: /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.809 DEBUG:boto:Data: 2023-10-18T19:53:59.809 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.810 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.810 DEBUG:boto:Params: {} 2023-10-18T19:53:59.811 DEBUG:boto:Token: None 2023-10-18T19:53:59.811 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jcqhQG3JsTplQA4oAHAX629Fc3w= 2023-10-18T19:53:59.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jcqhQG3JsTplQA4oAHAX629Fc3w=', 'Content-Length': '0'} 2023-10-18T19:53:59.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cd2df4f1d3c55d3-00653037d7-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.815 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.815 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.816 DEBUG:boto:path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.816 DEBUG:boto:auth_path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.817 DEBUG:boto:Method: HEAD 2023-10-18T19:53:59.817 DEBUG:boto:Path: /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.817 DEBUG:boto:Data: 2023-10-18T19:53:59.818 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.818 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.819 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.819 DEBUG:boto:Params: {} 2023-10-18T19:53:59.820 DEBUG:boto:Token: None 2023-10-18T19:53:59.820 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.820 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1URo6vybvoqYsmGX2WuUBlPEvy8= 2023-10-18T19:53:59.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1URo6vybvoqYsmGX2WuUBlPEvy8=', 'Content-Length': '0'} 2023-10-18T19:53:59.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d842f412c648d5c-00653037d7-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.823 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.824 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.824 DEBUG:boto:path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.825 DEBUG:boto:auth_path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.825 DEBUG:boto:Method: HEAD 2023-10-18T19:53:59.825 DEBUG:boto:Path: /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.826 DEBUG:boto:Data: 2023-10-18T19:53:59.826 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.827 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.827 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.827 DEBUG:boto:Params: {} 2023-10-18T19:53:59.828 DEBUG:boto:Token: None 2023-10-18T19:53:59.828 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.829 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1URo6vybvoqYsmGX2WuUBlPEvy8= 2023-10-18T19:53:59.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1URo6vybvoqYsmGX2WuUBlPEvy8=', 'Content-Length': '0'} 2023-10-18T19:53:59.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001c9236371402718-00653037d7-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-18T19:53:59.832 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.832 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.833 DEBUG:boto:path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.833 DEBUG:boto:auth_path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.834 DEBUG:boto:Method: GET 2023-10-18T19:53:59.834 DEBUG:boto:Path: /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.834 DEBUG:boto:Data: 2023-10-18T19:53:59.835 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.835 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.836 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.836 DEBUG:boto:Params: {} 2023-10-18T19:53:59.837 DEBUG:boto:Token: None 2023-10-18T19:53:59.837 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.838 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jcqhQG3JsTplQA4oAHAX629Fc3w= 2023-10-18T19:53:59.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jcqhQG3JsTplQA4oAHAX629Fc3w=', 'Content-Length': '0'} 2023-10-18T19:53:59.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f317947ad5cfa80a-00653037d7-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.841 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.841 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.841 DEBUG:boto:path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.842 DEBUG:boto:auth_path=/bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.842 DEBUG:boto:Method: GET 2023-10-18T19:53:59.843 DEBUG:boto:Path: /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.843 DEBUG:boto:Data: 2023-10-18T19:53:59.844 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.844 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.844 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.845 DEBUG:boto:Params: {} 2023-10-18T19:53:59.845 DEBUG:boto:Token: None 2023-10-18T19:53:59.846 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26?versionId=null 2023-10-18T19:53:59.846 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jcqhQG3JsTplQA4oAHAX629Fc3w= 2023-10-18T19:53:59.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jcqhQG3JsTplQA4oAHAX629Fc3w=', 'Content-Length': '0'} 2023-10-18T19:53:59.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da9bb847e0538d89-00653037d7-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.894 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.895 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.895 DEBUG:boto:path=/bofrnf-7/?prefix=%26 2023-10-18T19:53:59.896 DEBUG:boto:auth_path=/bofrnf-7/?prefix=%26 2023-10-18T19:53:59.896 DEBUG:boto:Method: GET 2023-10-18T19:53:59.896 DEBUG:boto:Path: /bofrnf-7/?prefix=%26 2023-10-18T19:53:59.897 DEBUG:boto:Data: 2023-10-18T19:53:59.897 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.898 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.898 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.899 DEBUG:boto:Params: {} 2023-10-18T19:53:59.899 DEBUG:boto:Token: None 2023-10-18T19:53:59.899 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/ 2023-10-18T19:53:59.900 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1OgVujNu72cGQAuPIAba/EsgK1E= 2023-10-18T19:53:59.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1OgVujNu72cGQAuPIAba/EsgK1E=', 'Content-Length': '0'} 2023-10-18T19:53:59.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1089a9113c069c-00653037d7-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.903 DEBUG:boto:b'bofrnf-7&1000false&2023-10-18T19:49:39.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:53:59.904 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.904 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.905 DEBUG:boto:path=/bofrnf-7/?prefix=%26 2023-10-18T19:53:59.905 DEBUG:boto:auth_path=/bofrnf-7/?prefix=%26 2023-10-18T19:53:59.906 DEBUG:boto:Method: GET 2023-10-18T19:53:59.906 DEBUG:boto:Path: /bofrnf-7/?prefix=%26 2023-10-18T19:53:59.907 DEBUG:boto:Data: 2023-10-18T19:53:59.907 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.907 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.908 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.908 DEBUG:boto:Params: {} 2023-10-18T19:53:59.909 DEBUG:boto:Token: None 2023-10-18T19:53:59.909 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/ 2023-10-18T19:53:59.910 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1OgVujNu72cGQAuPIAba/EsgK1E= 2023-10-18T19:53:59.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1OgVujNu72cGQAuPIAba/EsgK1E=', 'Content-Length': '0'} 2023-10-18T19:53:59.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054a4e9db5b58495e-00653037d7-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.914 DEBUG:boto:b'bofrnf-7&1000false&2023-10-18T19:49:39.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:53:59.914 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.915 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.915 DEBUG:boto:Method: HEAD 2023-10-18T19:53:59.915 DEBUG:boto:Path: /bofrnf-7/%26 2023-10-18T19:53:59.916 DEBUG:boto:Data: 2023-10-18T19:53:59.916 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.916 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.917 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.917 DEBUG:boto:Params: {} 2023-10-18T19:53:59.917 DEBUG:boto:Token: None 2023-10-18T19:53:59.918 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26 2023-10-18T19:53:59.918 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qJkqaAxYa8Ftir6Hm4wcc0DcFAE= 2023-10-18T19:53:59.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qJkqaAxYa8Ftir6Hm4wcc0DcFAE=', 'Content-Length': '0'} 2023-10-18T19:53:59.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e9f75e0736a5574-00653037d7-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.921 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.922 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.922 DEBUG:boto:Method: HEAD 2023-10-18T19:53:59.922 DEBUG:boto:Path: /bofrnf-7/%26 2023-10-18T19:53:59.923 DEBUG:boto:Data: 2023-10-18T19:53:59.923 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.924 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.924 DEBUG:boto:Params: {} 2023-10-18T19:53:59.925 DEBUG:boto:Token: None 2023-10-18T19:53:59.925 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26 2023-10-18T19:53:59.925 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qJkqaAxYa8Ftir6Hm4wcc0DcFAE= 2023-10-18T19:53:59.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qJkqaAxYa8Ftir6Hm4wcc0DcFAE=', 'Content-Length': '0'} 2023-10-18T19:53:59.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014ef0437dc40374f-00653037d7-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-18T19:53:59.929 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.929 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.929 DEBUG:boto:Method: GET 2023-10-18T19:53:59.930 DEBUG:boto:Path: /bofrnf-7/%26 2023-10-18T19:53:59.930 DEBUG:boto:Data: 2023-10-18T19:53:59.931 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.932 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.932 DEBUG:boto:Params: {} 2023-10-18T19:53:59.932 DEBUG:boto:Token: None 2023-10-18T19:53:59.933 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26 2023-10-18T19:53:59.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KMfgVCcH9FyNzo6gJ0fH2gNyjR8= 2023-10-18T19:53:59.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KMfgVCcH9FyNzo6gJ0fH2gNyjR8=', 'Content-Length': '0'} 2023-10-18T19:53:59.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5abf6e8c3120603-00653037d7-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.936 DEBUG:boto:path=/bofrnf-7/%26 2023-10-18T19:53:59.937 DEBUG:boto:auth_path=/bofrnf-7/%26 2023-10-18T19:53:59.937 DEBUG:boto:Method: GET 2023-10-18T19:53:59.938 DEBUG:boto:Path: /bofrnf-7/%26 2023-10-18T19:53:59.938 DEBUG:boto:Data: 2023-10-18T19:53:59.939 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.939 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.939 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.940 DEBUG:boto:Params: {} 2023-10-18T19:53:59.940 DEBUG:boto:Token: None 2023-10-18T19:53:59.941 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%26 2023-10-18T19:53:59.941 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KMfgVCcH9FyNzo6gJ0fH2gNyjR8= 2023-10-18T19:53:59.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KMfgVCcH9FyNzo6gJ0fH2gNyjR8=', 'Content-Length': '0'} 2023-10-18T19:53:59.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ad5922a6b884c8c-00653037d7-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.944 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.945 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.945 DEBUG:boto:path=/bofrnf-7/?prefix=%26 2023-10-18T19:53:59.945 DEBUG:boto:auth_path=/bofrnf-7/?prefix=%26 2023-10-18T19:53:59.946 DEBUG:boto:Method: GET 2023-10-18T19:53:59.946 DEBUG:boto:Path: /bofrnf-7/?prefix=%26 2023-10-18T19:53:59.947 DEBUG:boto:Data: 2023-10-18T19:53:59.947 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.948 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.948 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.948 DEBUG:boto:Params: {} 2023-10-18T19:53:59.949 DEBUG:boto:Token: None 2023-10-18T19:53:59.949 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/ 2023-10-18T19:53:59.950 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1OgVujNu72cGQAuPIAba/EsgK1E= 2023-10-18T19:53:59.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1OgVujNu72cGQAuPIAba/EsgK1E=', 'Content-Length': '0'} 2023-10-18T19:53:59.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ab3e95b686bf60c-00653037d7-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.953 DEBUG:boto:b'bofrnf-7&1000false&2023-10-18T19:49:39.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:53:59.954 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:53:59.954 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:53:59.955 DEBUG:boto:path=/bofrnf-7/?prefix=%26 2023-10-18T19:53:59.955 DEBUG:boto:auth_path=/bofrnf-7/?prefix=%26 2023-10-18T19:53:59.955 DEBUG:boto:Method: GET 2023-10-18T19:53:59.956 DEBUG:boto:Path: /bofrnf-7/?prefix=%26 2023-10-18T19:53:59.956 DEBUG:boto:Data: 2023-10-18T19:53:59.957 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.958 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.958 DEBUG:boto:Params: {} 2023-10-18T19:53:59.958 DEBUG:boto:Token: None 2023-10-18T19:53:59.959 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/ 2023-10-18T19:53:59.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1OgVujNu72cGQAuPIAba/EsgK1E= 2023-10-18T19:53:59.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1OgVujNu72cGQAuPIAba/EsgK1E=', 'Content-Length': '0'} 2023-10-18T19:53:59.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002deb3211209bde4f-00653037d7-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.963 DEBUG:boto:b'bofrnf-7&1000false&2023-10-18T19:49:39.133Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:53:59.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-18T19:53:59.964 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:53:59.964 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:53:59.965 DEBUG:boto:path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.965 DEBUG:boto:auth_path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.966 DEBUG:boto:Method: GET 2023-10-18T19:53:59.966 DEBUG:boto:Path: /bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.966 DEBUG:boto:Data: 2023-10-18T19:53:59.967 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.967 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.968 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.968 DEBUG:boto:Params: {} 2023-10-18T19:53:59.968 DEBUG:boto:Token: None 2023-10-18T19:53:59.969 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.969 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LoRGxrWVhyM67xvKsrl3uw1lagc= 2023-10-18T19:53:59.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LoRGxrWVhyM67xvKsrl3uw1lagc=', 'Content-Length': '0'} 2023-10-18T19:53:59.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f5c3c95c3abadc5-00653037d7-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.973 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:53:59.973 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:53:59.973 DEBUG:boto:path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.974 DEBUG:boto:auth_path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.974 DEBUG:boto:Method: GET 2023-10-18T19:53:59.975 DEBUG:boto:Path: /bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.975 DEBUG:boto:Data: 2023-10-18T19:53:59.975 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.976 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.976 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.977 DEBUG:boto:Params: {} 2023-10-18T19:53:59.977 DEBUG:boto:Token: None 2023-10-18T19:53:59.978 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.978 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LoRGxrWVhyM67xvKsrl3uw1lagc= 2023-10-18T19:53:59.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LoRGxrWVhyM67xvKsrl3uw1lagc=', 'Content-Length': '0'} 2023-10-18T19:53:59.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002064910553481ef7-00653037d7-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.981 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:53:59.982 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:53:59.982 DEBUG:boto:path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.982 DEBUG:boto:auth_path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.983 DEBUG:boto:Method: HEAD 2023-10-18T19:53:59.983 DEBUG:boto:Path: /bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.984 DEBUG:boto:Data: 2023-10-18T19:53:59.984 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.984 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:53:59.985 DEBUG:boto:Port: 8001 2023-10-18T19:53:59.985 DEBUG:boto:Params: {} 2023-10-18T19:53:59.986 DEBUG:boto:Token: None 2023-10-18T19:53:59.986 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.986 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ev1jgjDXrVLRTj+XDKSFlDRP6KU= 2023-10-18T19:53:59.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ev1jgjDXrVLRTj+XDKSFlDRP6KU=', 'Content-Length': '0'} 2023-10-18T19:53:59.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c806e2f6d8251c6a-00653037d7-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.989 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:53:59.990 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:53:59.990 DEBUG:boto:path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.991 DEBUG:boto:auth_path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.991 DEBUG:boto:Method: HEAD 2023-10-18T19:53:59.992 DEBUG:boto:Path: /bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.992 DEBUG:boto:Data: 2023-10-18T19:53:59.992 DEBUG:boto:Headers: {} 2023-10-18T19:53:59.993 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:53:59.993 DEBUG:boto:Port: 8000 2023-10-18T19:53:59.993 DEBUG:boto:Params: {} 2023-10-18T19:53:59.994 DEBUG:boto:Token: None 2023-10-18T19:53:59.994 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:53:59 GMT /bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.995 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ev1jgjDXrVLRTj+XDKSFlDRP6KU= 2023-10-18T19:53:59.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:53:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ev1jgjDXrVLRTj+XDKSFlDRP6KU=', 'Content-Length': '0'} 2023-10-18T19:53:59.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5cc2f48e93abd5b-00653037d7-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:53:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:53:59.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-18T19:53:59.998 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:53:59.998 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:53:59.999 DEBUG:boto:path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:53:59.999 DEBUG:boto:auth_path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:54:00.000 DEBUG:boto:Method: GET 2023-10-18T19:54:00.000 DEBUG:boto:Path: /bofrnf-7/%3A?versionId=null 2023-10-18T19:54:00.000 DEBUG:boto:Data: 2023-10-18T19:54:00.001 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.002 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.002 DEBUG:boto:Params: {} 2023-10-18T19:54:00.002 DEBUG:boto:Token: None 2023-10-18T19:54:00.003 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/%3A?versionId=null 2023-10-18T19:54:00.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X0rT/0mqqOT79Rh76mqEz3LuhDg= 2023-10-18T19:54:00.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X0rT/0mqqOT79Rh76mqEz3LuhDg=', 'Content-Length': '0'} 2023-10-18T19:54:00.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004af003b7fb498ea6-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.006 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:54:00.007 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:54:00.007 DEBUG:boto:path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:54:00.008 DEBUG:boto:auth_path=/bofrnf-7/%3A?versionId=null 2023-10-18T19:54:00.008 DEBUG:boto:Method: GET 2023-10-18T19:54:00.008 DEBUG:boto:Path: /bofrnf-7/%3A?versionId=null 2023-10-18T19:54:00.009 DEBUG:boto:Data: 2023-10-18T19:54:00.009 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.010 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.010 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.011 DEBUG:boto:Params: {} 2023-10-18T19:54:00.011 DEBUG:boto:Token: None 2023-10-18T19:54:00.011 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/%3A?versionId=null 2023-10-18T19:54:00.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X0rT/0mqqOT79Rh76mqEz3LuhDg= 2023-10-18T19:54:00.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X0rT/0mqqOT79Rh76mqEz3LuhDg=', 'Content-Length': '0'} 2023-10-18T19:54:00.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1023a7a4e956042-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.015 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.016 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.016 DEBUG:boto:path=/bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.016 DEBUG:boto:auth_path=/bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.017 DEBUG:boto:Method: GET 2023-10-18T19:54:00.017 DEBUG:boto:Path: /bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.018 DEBUG:boto:Data: 2023-10-18T19:54:00.018 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.019 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.019 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.020 DEBUG:boto:Params: {} 2023-10-18T19:54:00.020 DEBUG:boto:Token: None 2023-10-18T19:54:00.021 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.022 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052991d4eba434128-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.025 DEBUG:boto:b'bofrnf-7:1000false:2023-10-18T19:49:39.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.026 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.026 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.027 DEBUG:boto:path=/bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.027 DEBUG:boto:auth_path=/bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.028 DEBUG:boto:Method: GET 2023-10-18T19:54:00.028 DEBUG:boto:Path: /bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.029 DEBUG:boto:Data: 2023-10-18T19:54:00.029 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.029 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.030 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.030 DEBUG:boto:Params: {} 2023-10-18T19:54:00.031 DEBUG:boto:Token: None 2023-10-18T19:54:00.031 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f295a923b4d3d9b9-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.078 DEBUG:boto:b'bofrnf-7:1000false:2023-10-18T19:49:39.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.079 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:54:00.079 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:54:00.080 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.080 DEBUG:boto:Path: /bofrnf-7/%3A 2023-10-18T19:54:00.080 DEBUG:boto:Data: 2023-10-18T19:54:00.081 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.081 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.081 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.082 DEBUG:boto:Params: {} 2023-10-18T19:54:00.082 DEBUG:boto:Token: None 2023-10-18T19:54:00.083 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/%3A 2023-10-18T19:54:00.083 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:47/t8l47zDwdsd8pVpIgfwwkqTI= 2023-10-18T19:54:00.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:47/t8l47zDwdsd8pVpIgfwwkqTI=', 'Content-Length': '0'} 2023-10-18T19:54:00.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099af26df0d87c2a4-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.086 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:54:00.087 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:54:00.087 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.087 DEBUG:boto:Path: /bofrnf-7/%3A 2023-10-18T19:54:00.088 DEBUG:boto:Data: 2023-10-18T19:54:00.088 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.088 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.089 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.089 DEBUG:boto:Params: {} 2023-10-18T19:54:00.090 DEBUG:boto:Token: None 2023-10-18T19:54:00.090 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/%3A 2023-10-18T19:54:00.090 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:47/t8l47zDwdsd8pVpIgfwwkqTI= 2023-10-18T19:54:00.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:47/t8l47zDwdsd8pVpIgfwwkqTI=', 'Content-Length': '0'} 2023-10-18T19:54:00.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a4b62e4ed678ae1-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-18T19:54:00.094 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:54:00.094 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:54:00.095 DEBUG:boto:Method: GET 2023-10-18T19:54:00.095 DEBUG:boto:Path: /bofrnf-7/%3A 2023-10-18T19:54:00.095 DEBUG:boto:Data: 2023-10-18T19:54:00.096 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.096 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.096 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.097 DEBUG:boto:Params: {} 2023-10-18T19:54:00.097 DEBUG:boto:Token: None 2023-10-18T19:54:00.098 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/%3A 2023-10-18T19:54:00.098 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8W4N+pLgNQ6V20DeFB9Sre8vmn0= 2023-10-18T19:54:00.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8W4N+pLgNQ6V20DeFB9Sre8vmn0=', 'Content-Length': '0'} 2023-10-18T19:54:00.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a474a43d39e0953-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.102 DEBUG:boto:path=/bofrnf-7/%3A 2023-10-18T19:54:00.102 DEBUG:boto:auth_path=/bofrnf-7/%3A 2023-10-18T19:54:00.102 DEBUG:boto:Method: GET 2023-10-18T19:54:00.103 DEBUG:boto:Path: /bofrnf-7/%3A 2023-10-18T19:54:00.103 DEBUG:boto:Data: 2023-10-18T19:54:00.104 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.104 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.104 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.105 DEBUG:boto:Params: {} 2023-10-18T19:54:00.105 DEBUG:boto:Token: None 2023-10-18T19:54:00.106 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/%3A 2023-10-18T19:54:00.106 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8W4N+pLgNQ6V20DeFB9Sre8vmn0= 2023-10-18T19:54:00.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8W4N+pLgNQ6V20DeFB9Sre8vmn0=', 'Content-Length': '0'} 2023-10-18T19:54:00.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000170a0eae4e6be339-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.109 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.109 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.110 DEBUG:boto:path=/bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.110 DEBUG:boto:auth_path=/bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.111 DEBUG:boto:Method: GET 2023-10-18T19:54:00.111 DEBUG:boto:Path: /bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.111 DEBUG:boto:Data: 2023-10-18T19:54:00.112 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.112 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.112 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.113 DEBUG:boto:Params: {} 2023-10-18T19:54:00.113 DEBUG:boto:Token: None 2023-10-18T19:54:00.114 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.114 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a4c324497c240c2-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.118 DEBUG:boto:b'bofrnf-7:1000false:2023-10-18T19:49:39.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.118 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.119 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.119 DEBUG:boto:path=/bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.120 DEBUG:boto:auth_path=/bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.120 DEBUG:boto:Method: GET 2023-10-18T19:54:00.120 DEBUG:boto:Path: /bofrnf-7/?prefix=%3A 2023-10-18T19:54:00.121 DEBUG:boto:Data: 2023-10-18T19:54:00.121 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.122 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.122 DEBUG:boto:Params: {} 2023-10-18T19:54:00.123 DEBUG:boto:Token: None 2023-10-18T19:54:00.123 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.124 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b836f1835d7ae4a8-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.127 DEBUG:boto:b'bofrnf-7:1000false:2023-10-18T19:49:39.108Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-18T19:54:00.128 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.129 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.129 DEBUG:boto:path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.129 DEBUG:boto:auth_path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.130 DEBUG:boto:Method: GET 2023-10-18T19:54:00.130 DEBUG:boto:Path: /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.130 DEBUG:boto:Data: 2023-10-18T19:54:00.131 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.131 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.132 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.132 DEBUG:boto:Params: {} 2023-10-18T19:54:00.132 DEBUG:boto:Token: None 2023-10-18T19:54:00.133 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dmrFhE8Go33x6bVmXT94alQGIVs= 2023-10-18T19:54:00.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dmrFhE8Go33x6bVmXT94alQGIVs=', 'Content-Length': '0'} 2023-10-18T19:54:00.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000678f060b9f71544a-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.136 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.137 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.137 DEBUG:boto:path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.138 DEBUG:boto:auth_path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.138 DEBUG:boto:Method: GET 2023-10-18T19:54:00.138 DEBUG:boto:Path: /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.139 DEBUG:boto:Data: 2023-10-18T19:54:00.139 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.140 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.140 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.140 DEBUG:boto:Params: {} 2023-10-18T19:54:00.141 DEBUG:boto:Token: None 2023-10-18T19:54:00.141 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.142 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dmrFhE8Go33x6bVmXT94alQGIVs= 2023-10-18T19:54:00.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dmrFhE8Go33x6bVmXT94alQGIVs=', 'Content-Length': '0'} 2023-10-18T19:54:00.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087195c3f878b7ef9-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.145 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.145 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.145 DEBUG:boto:path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.146 DEBUG:boto:auth_path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.146 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.146 DEBUG:boto:Path: /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.147 DEBUG:boto:Data: 2023-10-18T19:54:00.147 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.148 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.148 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.148 DEBUG:boto:Params: {} 2023-10-18T19:54:00.149 DEBUG:boto:Token: None 2023-10-18T19:54:00.149 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.150 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I83ZXmDq7Re4oDsrxtfdf4Ozjf8= 2023-10-18T19:54:00.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I83ZXmDq7Re4oDsrxtfdf4Ozjf8=', 'Content-Length': '0'} 2023-10-18T19:54:00.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000169ba89db047f083-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.153 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.153 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.153 DEBUG:boto:path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.154 DEBUG:boto:auth_path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.154 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.154 DEBUG:boto:Path: /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.155 DEBUG:boto:Data: 2023-10-18T19:54:00.155 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.156 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.156 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.156 DEBUG:boto:Params: {} 2023-10-18T19:54:00.157 DEBUG:boto:Token: None 2023-10-18T19:54:00.157 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.158 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I83ZXmDq7Re4oDsrxtfdf4Ozjf8= 2023-10-18T19:54:00.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I83ZXmDq7Re4oDsrxtfdf4Ozjf8=', 'Content-Length': '0'} 2023-10-18T19:54:00.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d11fb805ae63695-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-18T19:54:00.161 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.161 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.161 DEBUG:boto:path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.162 DEBUG:boto:auth_path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.162 DEBUG:boto:Method: GET 2023-10-18T19:54:00.163 DEBUG:boto:Path: /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.163 DEBUG:boto:Data: 2023-10-18T19:54:00.163 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.164 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.164 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.165 DEBUG:boto:Params: {} 2023-10-18T19:54:00.165 DEBUG:boto:Token: None 2023-10-18T19:54:00.166 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.166 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dmrFhE8Go33x6bVmXT94alQGIVs= 2023-10-18T19:54:00.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dmrFhE8Go33x6bVmXT94alQGIVs=', 'Content-Length': '0'} 2023-10-18T19:54:00.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8893d6112172b83-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.169 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.169 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.170 DEBUG:boto:path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.170 DEBUG:boto:auth_path=/bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.171 DEBUG:boto:Method: GET 2023-10-18T19:54:00.171 DEBUG:boto:Path: /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.172 DEBUG:boto:Data: 2023-10-18T19:54:00.172 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.173 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.173 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.173 DEBUG:boto:Params: {} 2023-10-18T19:54:00.174 DEBUG:boto:Token: None 2023-10-18T19:54:00.174 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj?versionId=null 2023-10-18T19:54:00.175 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dmrFhE8Go33x6bVmXT94alQGIVs= 2023-10-18T19:54:00.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dmrFhE8Go33x6bVmXT94alQGIVs=', 'Content-Length': '0'} 2023-10-18T19:54:00.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e30be5bfb80522e1-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.178 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.178 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.179 DEBUG:boto:path=/bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.179 DEBUG:boto:auth_path=/bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.180 DEBUG:boto:Method: GET 2023-10-18T19:54:00.180 DEBUG:boto:Path: /bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.180 DEBUG:boto:Data: 2023-10-18T19:54:00.181 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.181 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.181 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.182 DEBUG:boto:Params: {} 2023-10-18T19:54:00.182 DEBUG:boto:Token: None 2023-10-18T19:54:00.183 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004523427e0738ffef-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.187 DEBUG:boto:b'bofrnf-7_myobj1000false_myobj2023-10-18T19:49:39.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.187 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.188 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.188 DEBUG:boto:path=/bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.189 DEBUG:boto:auth_path=/bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.189 DEBUG:boto:Method: GET 2023-10-18T19:54:00.189 DEBUG:boto:Path: /bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.190 DEBUG:boto:Data: 2023-10-18T19:54:00.190 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.191 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.191 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.192 DEBUG:boto:Params: {} 2023-10-18T19:54:00.192 DEBUG:boto:Token: None 2023-10-18T19:54:00.193 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.193 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9c2baab980f6b4-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.197 DEBUG:boto:b'bofrnf-7_myobj1000false_myobj2023-10-18T19:49:39.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.197 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.198 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.198 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.198 DEBUG:boto:Path: /bofrnf-7/_myobj 2023-10-18T19:54:00.199 DEBUG:boto:Data: 2023-10-18T19:54:00.199 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.200 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.200 DEBUG:boto:Params: {} 2023-10-18T19:54:00.201 DEBUG:boto:Token: None 2023-10-18T19:54:00.201 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj 2023-10-18T19:54:00.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r7eV+FRbav7oDqpDLt9OcrqZRbA= 2023-10-18T19:54:00.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r7eV+FRbav7oDqpDLt9OcrqZRbA=', 'Content-Length': '0'} 2023-10-18T19:54:00.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ce253714a1fa2ff-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.204 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.204 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.205 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.205 DEBUG:boto:Path: /bofrnf-7/_myobj 2023-10-18T19:54:00.206 DEBUG:boto:Data: 2023-10-18T19:54:00.206 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.206 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.207 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.207 DEBUG:boto:Params: {} 2023-10-18T19:54:00.208 DEBUG:boto:Token: None 2023-10-18T19:54:00.208 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj 2023-10-18T19:54:00.209 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r7eV+FRbav7oDqpDLt9OcrqZRbA= 2023-10-18T19:54:00.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r7eV+FRbav7oDqpDLt9OcrqZRbA=', 'Content-Length': '0'} 2023-10-18T19:54:00.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022268d8f1995bea9-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-18T19:54:00.212 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.212 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.213 DEBUG:boto:Method: GET 2023-10-18T19:54:00.213 DEBUG:boto:Path: /bofrnf-7/_myobj 2023-10-18T19:54:00.213 DEBUG:boto:Data: 2023-10-18T19:54:00.214 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.214 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.215 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.215 DEBUG:boto:Params: {} 2023-10-18T19:54:00.216 DEBUG:boto:Token: None 2023-10-18T19:54:00.216 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj 2023-10-18T19:54:00.216 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eOrkC/pqIOFrfjR7yXQdmQe2XsM= 2023-10-18T19:54:00.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eOrkC/pqIOFrfjR7yXQdmQe2XsM=', 'Content-Length': '0'} 2023-10-18T19:54:00.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000465e030e72be5ad-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.262 DEBUG:boto:path=/bofrnf-7/_myobj 2023-10-18T19:54:00.263 DEBUG:boto:auth_path=/bofrnf-7/_myobj 2023-10-18T19:54:00.263 DEBUG:boto:Method: GET 2023-10-18T19:54:00.264 DEBUG:boto:Path: /bofrnf-7/_myobj 2023-10-18T19:54:00.264 DEBUG:boto:Data: 2023-10-18T19:54:00.264 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.265 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.265 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.266 DEBUG:boto:Params: {} 2023-10-18T19:54:00.266 DEBUG:boto:Token: None 2023-10-18T19:54:00.266 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/_myobj 2023-10-18T19:54:00.267 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eOrkC/pqIOFrfjR7yXQdmQe2XsM= 2023-10-18T19:54:00.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eOrkC/pqIOFrfjR7yXQdmQe2XsM=', 'Content-Length': '0'} 2023-10-18T19:54:00.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2b2c20f60fd6f97-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.270 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.271 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.271 DEBUG:boto:path=/bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.271 DEBUG:boto:auth_path=/bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.272 DEBUG:boto:Method: GET 2023-10-18T19:54:00.272 DEBUG:boto:Path: /bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.273 DEBUG:boto:Data: 2023-10-18T19:54:00.273 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.273 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.274 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.274 DEBUG:boto:Params: {} 2023-10-18T19:54:00.275 DEBUG:boto:Token: None 2023-10-18T19:54:00.275 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002954081af270d67e-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.279 DEBUG:boto:b'bofrnf-7_myobj1000false_myobj2023-10-18T19:49:39.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.280 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.280 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.280 DEBUG:boto:path=/bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.281 DEBUG:boto:auth_path=/bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.281 DEBUG:boto:Method: GET 2023-10-18T19:54:00.282 DEBUG:boto:Path: /bofrnf-7/?prefix=_myobj 2023-10-18T19:54:00.282 DEBUG:boto:Data: 2023-10-18T19:54:00.283 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.283 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.283 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.284 DEBUG:boto:Params: {} 2023-10-18T19:54:00.284 DEBUG:boto:Token: None 2023-10-18T19:54:00.285 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.285 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab3c1e8ac304705d-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.288 DEBUG:boto:b'bofrnf-7_myobj1000false_myobj2023-10-18T19:49:39.053Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:00.289 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.290 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.290 DEBUG:boto:path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.291 DEBUG:boto:auth_path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.291 DEBUG:boto:Method: GET 2023-10-18T19:54:00.291 DEBUG:boto:Path: /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.292 DEBUG:boto:Data: 2023-10-18T19:54:00.292 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.293 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.293 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.293 DEBUG:boto:Params: {} 2023-10-18T19:54:00.294 DEBUG:boto:Token: None 2023-10-18T19:54:00.294 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.295 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UViaUmpetidDG66PorILek5mbqk= 2023-10-18T19:54:00.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UViaUmpetidDG66PorILek5mbqk=', 'Content-Length': '0'} 2023-10-18T19:54:00.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ecc2998204dd34b-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.298 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.299 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.299 DEBUG:boto:path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.299 DEBUG:boto:auth_path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.300 DEBUG:boto:Method: GET 2023-10-18T19:54:00.300 DEBUG:boto:Path: /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.301 DEBUG:boto:Data: 2023-10-18T19:54:00.301 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.301 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.302 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.302 DEBUG:boto:Params: {} 2023-10-18T19:54:00.303 DEBUG:boto:Token: None 2023-10-18T19:54:00.303 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UViaUmpetidDG66PorILek5mbqk= 2023-10-18T19:54:00.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UViaUmpetidDG66PorILek5mbqk=', 'Content-Length': '0'} 2023-10-18T19:54:00.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047ff13a456c9e9d9-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.307 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.307 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.308 DEBUG:boto:path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.308 DEBUG:boto:auth_path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.308 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.309 DEBUG:boto:Path: /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.309 DEBUG:boto:Data: 2023-10-18T19:54:00.309 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.310 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.311 DEBUG:boto:Params: {} 2023-10-18T19:54:00.311 DEBUG:boto:Token: None 2023-10-18T19:54:00.311 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xoPQ7FbYPzLYVNN0CbWfpfnlen4= 2023-10-18T19:54:00.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xoPQ7FbYPzLYVNN0CbWfpfnlen4=', 'Content-Length': '0'} 2023-10-18T19:54:00.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d67e9c2aca9c1c6-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.315 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.315 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.315 DEBUG:boto:path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.316 DEBUG:boto:auth_path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.316 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.317 DEBUG:boto:Path: /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.317 DEBUG:boto:Data: 2023-10-18T19:54:00.317 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.318 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.319 DEBUG:boto:Params: {} 2023-10-18T19:54:00.319 DEBUG:boto:Token: None 2023-10-18T19:54:00.319 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xoPQ7FbYPzLYVNN0CbWfpfnlen4= 2023-10-18T19:54:00.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xoPQ7FbYPzLYVNN0CbWfpfnlen4=', 'Content-Length': '0'} 2023-10-18T19:54:00.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017a3ec65b75ae162-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:00.323 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.324 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.324 DEBUG:boto:path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.324 DEBUG:boto:auth_path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.325 DEBUG:boto:Method: GET 2023-10-18T19:54:00.325 DEBUG:boto:Path: /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.326 DEBUG:boto:Data: 2023-10-18T19:54:00.326 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.326 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.327 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.327 DEBUG:boto:Params: {} 2023-10-18T19:54:00.328 DEBUG:boto:Token: None 2023-10-18T19:54:00.328 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UViaUmpetidDG66PorILek5mbqk= 2023-10-18T19:54:00.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UViaUmpetidDG66PorILek5mbqk=', 'Content-Length': '0'} 2023-10-18T19:54:00.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000480e12988796ba16-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.331 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.332 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.332 DEBUG:boto:path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.332 DEBUG:boto:auth_path=/bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.333 DEBUG:boto:Method: GET 2023-10-18T19:54:00.333 DEBUG:boto:Path: /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.333 DEBUG:boto:Data: 2023-10-18T19:54:00.334 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.334 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.335 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.335 DEBUG:boto:Params: {} 2023-10-18T19:54:00.336 DEBUG:boto:Token: None 2023-10-18T19:54:00.336 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj?versionId=null 2023-10-18T19:54:00.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UViaUmpetidDG66PorILek5mbqk= 2023-10-18T19:54:00.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UViaUmpetidDG66PorILek5mbqk=', 'Content-Length': '0'} 2023-10-18T19:54:00.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000444bf6031cd9f284-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.340 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.340 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.341 DEBUG:boto:path=/bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.341 DEBUG:boto:auth_path=/bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.341 DEBUG:boto:Method: GET 2023-10-18T19:54:00.342 DEBUG:boto:Path: /bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.342 DEBUG:boto:Data: 2023-10-18T19:54:00.343 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.343 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.344 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.344 DEBUG:boto:Params: {} 2023-10-18T19:54:00.344 DEBUG:boto:Token: None 2023-10-18T19:54:00.345 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.345 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf6cf2431579638e-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.390 DEBUG:boto:b'bofrnf-7myobj1000falsemyobj2023-10-18T19:49:39.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.391 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.391 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.392 DEBUG:boto:path=/bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.392 DEBUG:boto:auth_path=/bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.392 DEBUG:boto:Method: GET 2023-10-18T19:54:00.393 DEBUG:boto:Path: /bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.393 DEBUG:boto:Data: 2023-10-18T19:54:00.393 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.394 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.394 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.395 DEBUG:boto:Params: {} 2023-10-18T19:54:00.395 DEBUG:boto:Token: None 2023-10-18T19:54:00.396 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.396 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6f767ba1d9b8657-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.399 DEBUG:boto:b'bofrnf-7myobj1000falsemyobj2023-10-18T19:49:39.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.400 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.400 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.401 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.401 DEBUG:boto:Path: /bofrnf-7/myobj 2023-10-18T19:54:00.402 DEBUG:boto:Data: 2023-10-18T19:54:00.402 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.402 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.403 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.403 DEBUG:boto:Params: {} 2023-10-18T19:54:00.403 DEBUG:boto:Token: None 2023-10-18T19:54:00.404 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj 2023-10-18T19:54:00.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CfapNlSMdNZmYD/6xrMi4eD3z0A= 2023-10-18T19:54:00.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CfapNlSMdNZmYD/6xrMi4eD3z0A=', 'Content-Length': '0'} 2023-10-18T19:54:00.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a2c9bdf64aad6ca-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.408 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.408 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.408 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.409 DEBUG:boto:Path: /bofrnf-7/myobj 2023-10-18T19:54:00.409 DEBUG:boto:Data: 2023-10-18T19:54:00.410 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.410 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.410 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.411 DEBUG:boto:Params: {} 2023-10-18T19:54:00.411 DEBUG:boto:Token: None 2023-10-18T19:54:00.411 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj 2023-10-18T19:54:00.412 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CfapNlSMdNZmYD/6xrMi4eD3z0A= 2023-10-18T19:54:00.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CfapNlSMdNZmYD/6xrMi4eD3z0A=', 'Content-Length': '0'} 2023-10-18T19:54:00.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000755cd3886d15bf95-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:00.415 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.415 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.416 DEBUG:boto:Method: GET 2023-10-18T19:54:00.416 DEBUG:boto:Path: /bofrnf-7/myobj 2023-10-18T19:54:00.416 DEBUG:boto:Data: 2023-10-18T19:54:00.417 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.418 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.418 DEBUG:boto:Params: {} 2023-10-18T19:54:00.418 DEBUG:boto:Token: None 2023-10-18T19:54:00.419 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj 2023-10-18T19:54:00.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EFMxLneOp5cg6vIKP4/M22UrSZY= 2023-10-18T19:54:00.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EFMxLneOp5cg6vIKP4/M22UrSZY=', 'Content-Length': '0'} 2023-10-18T19:54:00.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000982778b2ae2ee45b-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.423 DEBUG:boto:path=/bofrnf-7/myobj 2023-10-18T19:54:00.423 DEBUG:boto:auth_path=/bofrnf-7/myobj 2023-10-18T19:54:00.424 DEBUG:boto:Method: GET 2023-10-18T19:54:00.424 DEBUG:boto:Path: /bofrnf-7/myobj 2023-10-18T19:54:00.424 DEBUG:boto:Data: 2023-10-18T19:54:00.425 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.425 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.426 DEBUG:boto:Params: {} 2023-10-18T19:54:00.426 DEBUG:boto:Token: None 2023-10-18T19:54:00.427 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/myobj 2023-10-18T19:54:00.427 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EFMxLneOp5cg6vIKP4/M22UrSZY= 2023-10-18T19:54:00.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EFMxLneOp5cg6vIKP4/M22UrSZY=', 'Content-Length': '0'} 2023-10-18T19:54:00.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b2df3b91f0226b7-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.474 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.475 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.475 DEBUG:boto:path=/bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.475 DEBUG:boto:auth_path=/bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.476 DEBUG:boto:Method: GET 2023-10-18T19:54:00.476 DEBUG:boto:Path: /bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.477 DEBUG:boto:Data: 2023-10-18T19:54:00.477 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.477 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.478 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.478 DEBUG:boto:Params: {} 2023-10-18T19:54:00.479 DEBUG:boto:Token: None 2023-10-18T19:54:00.479 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.480 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073e3708de258f229-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.483 DEBUG:boto:b'bofrnf-7myobj1000falsemyobj2023-10-18T19:49:39.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.484 DEBUG:boto:path=/bofrnf-7/ 2023-10-18T19:54:00.484 DEBUG:boto:auth_path=/bofrnf-7/ 2023-10-18T19:54:00.484 DEBUG:boto:path=/bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.485 DEBUG:boto:auth_path=/bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.485 DEBUG:boto:Method: GET 2023-10-18T19:54:00.486 DEBUG:boto:Path: /bofrnf-7/?prefix=myobj 2023-10-18T19:54:00.486 DEBUG:boto:Data: 2023-10-18T19:54:00.486 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.487 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.487 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.487 DEBUG:boto:Params: {} 2023-10-18T19:54:00.488 DEBUG:boto:Token: None 2023-10-18T19:54:00.488 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-7/ 2023-10-18T19:54:00.489 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY= 2023-10-18T19:54:00.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whcX4Uu8I7Q6DrgXUlY2iblJ4MY=', 'Content-Length': '0'} 2023-10-18T19:54:00.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa3534958d4d867f-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.492 DEBUG:boto:b'bofrnf-7myobj1000falsemyobj2023-10-18T19:49:39.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-7 zones={a2, a1} 2023-10-18T19:54:00.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:00.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:00.494 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:54:00.558 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:54:00.559 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:54:00.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:00.594+0000 7f1361d6b780 1 bucket sync caught up with source: 2023-10-18T19:54:00.596 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.18.6, , 00000000001.8.6, , , , 00000000001.10.6, 00000000001.10.6, , ] 2023-10-18T19:54:00.596 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.18.6, , 00000000001.8.6, , , , 00000000001.10.6, 00000000001.10.6, , ] 2023-10-18T19:54:00.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:00.594+0000 7f1361d6b780 0 bucket checkpoint complete 2023-10-18T19:54:00.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-8 zones={a1, a2} 2023-10-18T19:54:00.609 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.609 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.609 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.610 DEBUG:boto:Path: /bofrnf-8/ 2023-10-18T19:54:00.610 DEBUG:boto:Data: 2023-10-18T19:54:00.610 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.611 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.611 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.612 DEBUG:boto:Params: {} 2023-10-18T19:54:00.612 DEBUG:boto:Token: None 2023-10-18T19:54:00.612 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/ 2023-10-18T19:54:00.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m2qub+S8gpoEbxO39nPhM+/jh9c= 2023-10-18T19:54:00.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m2qub+S8gpoEbxO39nPhM+/jh9c=', 'Content-Length': '0'} 2023-10-18T19:54:00.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fc2aeedadcb718f-00653037d8-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.616 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.617 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.617 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.617 DEBUG:boto:Path: /bofrnf-8/ 2023-10-18T19:54:00.618 DEBUG:boto:Data: 2023-10-18T19:54:00.618 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.618 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.619 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.619 DEBUG:boto:Params: {} 2023-10-18T19:54:00.620 DEBUG:boto:Token: None 2023-10-18T19:54:00.620 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/ 2023-10-18T19:54:00.620 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m2qub+S8gpoEbxO39nPhM+/jh9c= 2023-10-18T19:54:00.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m2qub+S8gpoEbxO39nPhM+/jh9c=', 'Content-Length': '0'} 2023-10-18T19:54:00.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013add2af230ec96b-00653037d8-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.623 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:54:00.624 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.624 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.625 DEBUG:boto:path=/bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.625 DEBUG:boto:auth_path=/bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.625 DEBUG:boto:Method: GET 2023-10-18T19:54:00.626 DEBUG:boto:Path: /bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.626 DEBUG:boto:Data: 2023-10-18T19:54:00.626 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.627 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.627 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.628 DEBUG:boto:Params: {} 2023-10-18T19:54:00.628 DEBUG:boto:Token: None 2023-10-18T19:54:00.629 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/?versions 2023-10-18T19:54:00.629 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9gz1kOqNbXj1IA+kGLy17bMjmvM= 2023-10-18T19:54:00.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9gz1kOqNbXj1IA+kGLy17bMjmvM=', 'Content-Length': '0'} 2023-10-18T19:54:00.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a58bdc9dec5556ab-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.632 DEBUG:boto:b'bofrnf-8999false&nulltrue2023-10-18T19:49:39.299Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-18T19:49:39.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-18T19:49:39.214Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-18T19:49:39.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.633 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-18T19:54:00.633 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-18T19:54:00.633 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-18T19:54:00.634 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-18T19:54:00.634 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:54:00.635 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.635 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.635 DEBUG:boto:path=/bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.636 DEBUG:boto:auth_path=/bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.636 DEBUG:boto:Method: GET 2023-10-18T19:54:00.636 DEBUG:boto:Path: /bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.637 DEBUG:boto:Data: 2023-10-18T19:54:00.637 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.638 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.638 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.638 DEBUG:boto:Params: {} 2023-10-18T19:54:00.639 DEBUG:boto:Token: None 2023-10-18T19:54:00.639 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/?versions 2023-10-18T19:54:00.640 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9gz1kOqNbXj1IA+kGLy17bMjmvM= 2023-10-18T19:54:00.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9gz1kOqNbXj1IA+kGLy17bMjmvM=', 'Content-Length': '0'} 2023-10-18T19:54:00.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de9d7b9aefea53a-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.643 DEBUG:boto:b'bofrnf-8999false&nulltrue2023-10-18T19:49:39.299Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-18T19:49:39.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-18T19:49:39.214Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-18T19:49:39.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.643 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-18T19:54:00.644 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-18T19:54:00.644 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-18T19:54:00.644 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-18T19:54:00.645 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.645 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.645 DEBUG:boto:path=/bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.646 DEBUG:boto:auth_path=/bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.646 DEBUG:boto:Method: GET 2023-10-18T19:54:00.647 DEBUG:boto:Path: /bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.647 DEBUG:boto:Data: 2023-10-18T19:54:00.647 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.648 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.648 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.649 DEBUG:boto:Params: {} 2023-10-18T19:54:00.649 DEBUG:boto:Token: None 2023-10-18T19:54:00.649 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/?versions 2023-10-18T19:54:00.650 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9gz1kOqNbXj1IA+kGLy17bMjmvM= 2023-10-18T19:54:00.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9gz1kOqNbXj1IA+kGLy17bMjmvM=', 'Content-Length': '0'} 2023-10-18T19:54:00.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000369d795edea16dc7-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.654 DEBUG:boto:b'bofrnf-8999false&nulltrue2023-10-18T19:49:39.299Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-18T19:49:39.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-18T19:49:39.214Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-18T19:49:39.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.654 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.655 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.655 DEBUG:boto:path=/bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.656 DEBUG:boto:auth_path=/bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.656 DEBUG:boto:Method: GET 2023-10-18T19:54:00.656 DEBUG:boto:Path: /bofrnf-8/?versions&max-keys=999 2023-10-18T19:54:00.657 DEBUG:boto:Data: 2023-10-18T19:54:00.657 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.658 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.658 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.658 DEBUG:boto:Params: {} 2023-10-18T19:54:00.659 DEBUG:boto:Token: None 2023-10-18T19:54:00.659 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/?versions 2023-10-18T19:54:00.660 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9gz1kOqNbXj1IA+kGLy17bMjmvM= 2023-10-18T19:54:00.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9gz1kOqNbXj1IA+kGLy17bMjmvM=', 'Content-Length': '0'} 2023-10-18T19:54:00.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bc95489ee766239-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.663 DEBUG:boto:b'bofrnf-8999false&nulltrue2023-10-18T19:49:39.299Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-18T19:49:39.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-18T19:49:39.214Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-18T19:49:39.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-18T19:54:00.664 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.664 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.665 DEBUG:boto:path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.665 DEBUG:boto:auth_path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.665 DEBUG:boto:Method: GET 2023-10-18T19:54:00.666 DEBUG:boto:Path: /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.666 DEBUG:boto:Data: 2023-10-18T19:54:00.667 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.667 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.667 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.668 DEBUG:boto:Params: {} 2023-10-18T19:54:00.668 DEBUG:boto:Token: None 2023-10-18T19:54:00.669 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.669 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ns5G8DoBqNsdNskRcU/bEz3l5K8= 2023-10-18T19:54:00.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ns5G8DoBqNsdNskRcU/bEz3l5K8=', 'Content-Length': '0'} 2023-10-18T19:54:00.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000909b95296a51444-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.672 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.672 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.673 DEBUG:boto:path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.673 DEBUG:boto:auth_path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.673 DEBUG:boto:Method: GET 2023-10-18T19:54:00.674 DEBUG:boto:Path: /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.674 DEBUG:boto:Data: 2023-10-18T19:54:00.675 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.675 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.676 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.676 DEBUG:boto:Params: {} 2023-10-18T19:54:00.676 DEBUG:boto:Token: None 2023-10-18T19:54:00.677 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.677 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ns5G8DoBqNsdNskRcU/bEz3l5K8= 2023-10-18T19:54:00.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ns5G8DoBqNsdNskRcU/bEz3l5K8=', 'Content-Length': '0'} 2023-10-18T19:54:00.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e34cb23df37ddb1-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.722 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.723 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.723 DEBUG:boto:path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.723 DEBUG:boto:auth_path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.724 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.724 DEBUG:boto:Path: /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.725 DEBUG:boto:Data: 2023-10-18T19:54:00.725 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.726 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.726 DEBUG:boto:Params: {} 2023-10-18T19:54:00.726 DEBUG:boto:Token: None 2023-10-18T19:54:00.727 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.727 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yIc3lipqXMF/ANuOIqi5qqAJmsc= 2023-10-18T19:54:00.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yIc3lipqXMF/ANuOIqi5qqAJmsc=', 'Content-Length': '0'} 2023-10-18T19:54:00.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e3ca224e480806b-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.730 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.731 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.731 DEBUG:boto:path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.731 DEBUG:boto:auth_path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.732 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.732 DEBUG:boto:Path: /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.732 DEBUG:boto:Data: 2023-10-18T19:54:00.733 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.733 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.734 DEBUG:boto:Params: {} 2023-10-18T19:54:00.734 DEBUG:boto:Token: None 2023-10-18T19:54:00.735 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yIc3lipqXMF/ANuOIqi5qqAJmsc= 2023-10-18T19:54:00.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yIc3lipqXMF/ANuOIqi5qqAJmsc=', 'Content-Length': '0'} 2023-10-18T19:54:00.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038463e12b6126783-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-18T19:54:00.738 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.739 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.739 DEBUG:boto:path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.739 DEBUG:boto:auth_path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.740 DEBUG:boto:Method: GET 2023-10-18T19:54:00.740 DEBUG:boto:Path: /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.741 DEBUG:boto:Data: 2023-10-18T19:54:00.741 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.741 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.742 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.742 DEBUG:boto:Params: {} 2023-10-18T19:54:00.743 DEBUG:boto:Token: None 2023-10-18T19:54:00.743 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.743 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ns5G8DoBqNsdNskRcU/bEz3l5K8= 2023-10-18T19:54:00.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ns5G8DoBqNsdNskRcU/bEz3l5K8=', 'Content-Length': '0'} 2023-10-18T19:54:00.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008aeb0c156f22d8c0-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.746 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.747 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.747 DEBUG:boto:path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.747 DEBUG:boto:auth_path=/bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.748 DEBUG:boto:Method: GET 2023-10-18T19:54:00.748 DEBUG:boto:Path: /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.749 DEBUG:boto:Data: 2023-10-18T19:54:00.749 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.749 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.750 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.750 DEBUG:boto:Params: {} 2023-10-18T19:54:00.750 DEBUG:boto:Token: None 2023-10-18T19:54:00.751 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26?versionId=null 2023-10-18T19:54:00.751 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ns5G8DoBqNsdNskRcU/bEz3l5K8= 2023-10-18T19:54:00.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ns5G8DoBqNsdNskRcU/bEz3l5K8=', 'Content-Length': '0'} 2023-10-18T19:54:00.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ad4ec653686b033-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.754 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.754 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.755 DEBUG:boto:path=/bofrnf-8/?prefix=%26 2023-10-18T19:54:00.755 DEBUG:boto:auth_path=/bofrnf-8/?prefix=%26 2023-10-18T19:54:00.755 DEBUG:boto:Method: GET 2023-10-18T19:54:00.756 DEBUG:boto:Path: /bofrnf-8/?prefix=%26 2023-10-18T19:54:00.756 DEBUG:boto:Data: 2023-10-18T19:54:00.757 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.757 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.758 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.758 DEBUG:boto:Params: {} 2023-10-18T19:54:00.759 DEBUG:boto:Token: None 2023-10-18T19:54:00.759 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/ 2023-10-18T19:54:00.759 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo= 2023-10-18T19:54:00.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo=', 'Content-Length': '0'} 2023-10-18T19:54:00.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc6316c9c95114e1-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.763 DEBUG:boto:b'bofrnf-8&1000false&2023-10-18T19:49:39.299Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.763 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.764 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.764 DEBUG:boto:path=/bofrnf-8/?prefix=%26 2023-10-18T19:54:00.765 DEBUG:boto:auth_path=/bofrnf-8/?prefix=%26 2023-10-18T19:54:00.765 DEBUG:boto:Method: GET 2023-10-18T19:54:00.765 DEBUG:boto:Path: /bofrnf-8/?prefix=%26 2023-10-18T19:54:00.766 DEBUG:boto:Data: 2023-10-18T19:54:00.766 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.767 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.767 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.768 DEBUG:boto:Params: {} 2023-10-18T19:54:00.768 DEBUG:boto:Token: None 2023-10-18T19:54:00.768 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/ 2023-10-18T19:54:00.769 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo= 2023-10-18T19:54:00.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo=', 'Content-Length': '0'} 2023-10-18T19:54:00.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a43856b44ca9e1aa-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.772 DEBUG:boto:b'bofrnf-8&1000false&2023-10-18T19:49:39.299Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.773 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.773 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.773 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.774 DEBUG:boto:Path: /bofrnf-8/%26 2023-10-18T19:54:00.774 DEBUG:boto:Data: 2023-10-18T19:54:00.775 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.775 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.775 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.776 DEBUG:boto:Params: {} 2023-10-18T19:54:00.776 DEBUG:boto:Token: None 2023-10-18T19:54:00.777 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26 2023-10-18T19:54:00.777 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UDMAE81Fj7TxBdDRrDqQJk5ZTxw= 2023-10-18T19:54:00.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UDMAE81Fj7TxBdDRrDqQJk5ZTxw=', 'Content-Length': '0'} 2023-10-18T19:54:00.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dda39230c51ac8ff-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.780 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.780 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.781 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.781 DEBUG:boto:Path: /bofrnf-8/%26 2023-10-18T19:54:00.781 DEBUG:boto:Data: 2023-10-18T19:54:00.782 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.782 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.783 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.783 DEBUG:boto:Params: {} 2023-10-18T19:54:00.783 DEBUG:boto:Token: None 2023-10-18T19:54:00.784 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26 2023-10-18T19:54:00.784 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UDMAE81Fj7TxBdDRrDqQJk5ZTxw= 2023-10-18T19:54:00.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UDMAE81Fj7TxBdDRrDqQJk5ZTxw=', 'Content-Length': '0'} 2023-10-18T19:54:00.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000269e97c951846602-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-18T19:54:00.787 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.788 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.788 DEBUG:boto:Method: GET 2023-10-18T19:54:00.788 DEBUG:boto:Path: /bofrnf-8/%26 2023-10-18T19:54:00.789 DEBUG:boto:Data: 2023-10-18T19:54:00.789 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.790 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.791 DEBUG:boto:Params: {} 2023-10-18T19:54:00.791 DEBUG:boto:Token: None 2023-10-18T19:54:00.791 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26 2023-10-18T19:54:00.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W/4v/Ipi7gnmeONaxXBke/cd11A= 2023-10-18T19:54:00.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W/4v/Ipi7gnmeONaxXBke/cd11A=', 'Content-Length': '0'} 2023-10-18T19:54:00.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000199f2625d886d4ea-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.834 DEBUG:boto:path=/bofrnf-8/%26 2023-10-18T19:54:00.835 DEBUG:boto:auth_path=/bofrnf-8/%26 2023-10-18T19:54:00.835 DEBUG:boto:Method: GET 2023-10-18T19:54:00.836 DEBUG:boto:Path: /bofrnf-8/%26 2023-10-18T19:54:00.836 DEBUG:boto:Data: 2023-10-18T19:54:00.836 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.837 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.838 DEBUG:boto:Params: {} 2023-10-18T19:54:00.838 DEBUG:boto:Token: None 2023-10-18T19:54:00.838 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%26 2023-10-18T19:54:00.839 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W/4v/Ipi7gnmeONaxXBke/cd11A= 2023-10-18T19:54:00.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W/4v/Ipi7gnmeONaxXBke/cd11A=', 'Content-Length': '0'} 2023-10-18T19:54:00.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c155779d1eed412e-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.842 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.842 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.843 DEBUG:boto:path=/bofrnf-8/?prefix=%26 2023-10-18T19:54:00.843 DEBUG:boto:auth_path=/bofrnf-8/?prefix=%26 2023-10-18T19:54:00.844 DEBUG:boto:Method: GET 2023-10-18T19:54:00.844 DEBUG:boto:Path: /bofrnf-8/?prefix=%26 2023-10-18T19:54:00.844 DEBUG:boto:Data: 2023-10-18T19:54:00.845 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.845 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.846 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.846 DEBUG:boto:Params: {} 2023-10-18T19:54:00.847 DEBUG:boto:Token: None 2023-10-18T19:54:00.847 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/ 2023-10-18T19:54:00.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo= 2023-10-18T19:54:00.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo=', 'Content-Length': '0'} 2023-10-18T19:54:00.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002df0bb5ff778fc2b-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.851 DEBUG:boto:b'bofrnf-8&1000false&2023-10-18T19:49:39.299Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.851 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.852 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.852 DEBUG:boto:path=/bofrnf-8/?prefix=%26 2023-10-18T19:54:00.852 DEBUG:boto:auth_path=/bofrnf-8/?prefix=%26 2023-10-18T19:54:00.853 DEBUG:boto:Method: GET 2023-10-18T19:54:00.853 DEBUG:boto:Path: /bofrnf-8/?prefix=%26 2023-10-18T19:54:00.854 DEBUG:boto:Data: 2023-10-18T19:54:00.854 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.855 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.855 DEBUG:boto:Params: {} 2023-10-18T19:54:00.856 DEBUG:boto:Token: None 2023-10-18T19:54:00.856 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/ 2023-10-18T19:54:00.857 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo= 2023-10-18T19:54:00.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo=', 'Content-Length': '0'} 2023-10-18T19:54:00.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c68b292c9df6c7-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.860 DEBUG:boto:b'bofrnf-8&1000false&2023-10-18T19:49:39.299Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-18T19:54:00.861 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.861 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.862 DEBUG:boto:path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.862 DEBUG:boto:auth_path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.863 DEBUG:boto:Method: GET 2023-10-18T19:54:00.863 DEBUG:boto:Path: /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.863 DEBUG:boto:Data: 2023-10-18T19:54:00.864 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.864 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.864 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.865 DEBUG:boto:Params: {} 2023-10-18T19:54:00.865 DEBUG:boto:Token: None 2023-10-18T19:54:00.866 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.866 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SMx7Nitnc1EV+kfxMZTZUVaU1e8= 2023-10-18T19:54:00.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SMx7Nitnc1EV+kfxMZTZUVaU1e8=', 'Content-Length': '0'} 2023-10-18T19:54:00.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c64ae031a26243a-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.869 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.869 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.870 DEBUG:boto:path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.870 DEBUG:boto:auth_path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.871 DEBUG:boto:Method: GET 2023-10-18T19:54:00.871 DEBUG:boto:Path: /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.872 DEBUG:boto:Data: 2023-10-18T19:54:00.872 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.872 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.873 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.873 DEBUG:boto:Params: {} 2023-10-18T19:54:00.874 DEBUG:boto:Token: None 2023-10-18T19:54:00.874 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.875 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SMx7Nitnc1EV+kfxMZTZUVaU1e8= 2023-10-18T19:54:00.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SMx7Nitnc1EV+kfxMZTZUVaU1e8=', 'Content-Length': '0'} 2023-10-18T19:54:00.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042fa64fffe117d4c-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.877 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.878 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.878 DEBUG:boto:path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.879 DEBUG:boto:auth_path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.879 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.879 DEBUG:boto:Path: /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.880 DEBUG:boto:Data: 2023-10-18T19:54:00.880 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.880 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.881 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.881 DEBUG:boto:Params: {} 2023-10-18T19:54:00.881 DEBUG:boto:Token: None 2023-10-18T19:54:00.882 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.882 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MrYCp+WWbxsf3yEtbJt0ZChWro8= 2023-10-18T19:54:00.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MrYCp+WWbxsf3yEtbJt0ZChWro8=', 'Content-Length': '0'} 2023-10-18T19:54:00.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c74e990255079d80-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.885 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.885 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.886 DEBUG:boto:path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.886 DEBUG:boto:auth_path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.887 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.887 DEBUG:boto:Path: /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.887 DEBUG:boto:Data: 2023-10-18T19:54:00.888 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.888 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.889 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.889 DEBUG:boto:Params: {} 2023-10-18T19:54:00.889 DEBUG:boto:Token: None 2023-10-18T19:54:00.890 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.890 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MrYCp+WWbxsf3yEtbJt0ZChWro8= 2023-10-18T19:54:00.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MrYCp+WWbxsf3yEtbJt0ZChWro8=', 'Content-Length': '0'} 2023-10-18T19:54:00.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eec344027042b744-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-18T19:54:00.894 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.894 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.895 DEBUG:boto:path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.895 DEBUG:boto:auth_path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.895 DEBUG:boto:Method: GET 2023-10-18T19:54:00.896 DEBUG:boto:Path: /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.896 DEBUG:boto:Data: 2023-10-18T19:54:00.897 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.897 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.897 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.898 DEBUG:boto:Params: {} 2023-10-18T19:54:00.898 DEBUG:boto:Token: None 2023-10-18T19:54:00.899 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.899 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SMx7Nitnc1EV+kfxMZTZUVaU1e8= 2023-10-18T19:54:00.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SMx7Nitnc1EV+kfxMZTZUVaU1e8=', 'Content-Length': '0'} 2023-10-18T19:54:00.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e411b9f8ff53cef-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.902 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.902 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.903 DEBUG:boto:path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.903 DEBUG:boto:auth_path=/bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.904 DEBUG:boto:Method: GET 2023-10-18T19:54:00.904 DEBUG:boto:Path: /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.904 DEBUG:boto:Data: 2023-10-18T19:54:00.905 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.905 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.906 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.906 DEBUG:boto:Params: {} 2023-10-18T19:54:00.906 DEBUG:boto:Token: None 2023-10-18T19:54:00.907 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A?versionId=null 2023-10-18T19:54:00.908 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SMx7Nitnc1EV+kfxMZTZUVaU1e8= 2023-10-18T19:54:00.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SMx7Nitnc1EV+kfxMZTZUVaU1e8=', 'Content-Length': '0'} 2023-10-18T19:54:00.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cd7ae3b748297ff-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.911 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.911 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.911 DEBUG:boto:path=/bofrnf-8/?prefix=%3A 2023-10-18T19:54:00.912 DEBUG:boto:auth_path=/bofrnf-8/?prefix=%3A 2023-10-18T19:54:00.912 DEBUG:boto:Method: GET 2023-10-18T19:54:00.913 DEBUG:boto:Path: /bofrnf-8/?prefix=%3A 2023-10-18T19:54:00.913 DEBUG:boto:Data: 2023-10-18T19:54:00.913 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.914 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.914 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.914 DEBUG:boto:Params: {} 2023-10-18T19:54:00.915 DEBUG:boto:Token: None 2023-10-18T19:54:00.915 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/ 2023-10-18T19:54:00.916 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo= 2023-10-18T19:54:00.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo=', 'Content-Length': '0'} 2023-10-18T19:54:00.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075c1b13827aea337-00653037d8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.919 DEBUG:boto:b'bofrnf-8:1000false:2023-10-18T19:49:39.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.919 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.920 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.920 DEBUG:boto:path=/bofrnf-8/?prefix=%3A 2023-10-18T19:54:00.921 DEBUG:boto:auth_path=/bofrnf-8/?prefix=%3A 2023-10-18T19:54:00.921 DEBUG:boto:Method: GET 2023-10-18T19:54:00.922 DEBUG:boto:Path: /bofrnf-8/?prefix=%3A 2023-10-18T19:54:00.922 DEBUG:boto:Data: 2023-10-18T19:54:00.922 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.923 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.924 DEBUG:boto:Params: {} 2023-10-18T19:54:00.924 DEBUG:boto:Token: None 2023-10-18T19:54:00.924 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/ 2023-10-18T19:54:00.925 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo= 2023-10-18T19:54:00.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SO7Kptf0zhDGXPuTw0Qs41KFFgo=', 'Content-Length': '0'} 2023-10-18T19:54:00.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b83eadcf551236a8-00653037d8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.928 DEBUG:boto:b'bofrnf-8:1000false:2023-10-18T19:49:39.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:00.928 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.929 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.929 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.930 DEBUG:boto:Path: /bofrnf-8/%3A 2023-10-18T19:54:00.930 DEBUG:boto:Data: 2023-10-18T19:54:00.930 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.931 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.932 DEBUG:boto:Params: {} 2023-10-18T19:54:00.932 DEBUG:boto:Token: None 2023-10-18T19:54:00.932 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A 2023-10-18T19:54:00.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Qs2F407lO/3VmR5Pnj7fNX5RQ8g= 2023-10-18T19:54:00.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Qs2F407lO/3VmR5Pnj7fNX5RQ8g=', 'Content-Length': '0'} 2023-10-18T19:54:00.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d63b9b4aedb5b0ea-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.936 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.936 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.936 DEBUG:boto:Method: HEAD 2023-10-18T19:54:00.937 DEBUG:boto:Path: /bofrnf-8/%3A 2023-10-18T19:54:00.937 DEBUG:boto:Data: 2023-10-18T19:54:00.938 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.938 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.938 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.939 DEBUG:boto:Params: {} 2023-10-18T19:54:00.939 DEBUG:boto:Token: None 2023-10-18T19:54:00.940 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A 2023-10-18T19:54:00.940 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Qs2F407lO/3VmR5Pnj7fNX5RQ8g= 2023-10-18T19:54:00.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Qs2F407lO/3VmR5Pnj7fNX5RQ8g=', 'Content-Length': '0'} 2023-10-18T19:54:00.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000930ba5fcf750d481-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-18T19:54:00.943 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.943 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.944 DEBUG:boto:Method: GET 2023-10-18T19:54:00.944 DEBUG:boto:Path: /bofrnf-8/%3A 2023-10-18T19:54:00.945 DEBUG:boto:Data: 2023-10-18T19:54:00.945 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:00.946 DEBUG:boto:Port: 8000 2023-10-18T19:54:00.946 DEBUG:boto:Params: {} 2023-10-18T19:54:00.947 DEBUG:boto:Token: None 2023-10-18T19:54:00.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A 2023-10-18T19:54:00.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fCxlZef+5uw3N3RqOnzt0nTr9V8= 2023-10-18T19:54:00.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fCxlZef+5uw3N3RqOnzt0nTr9V8=', 'Content-Length': '0'} 2023-10-18T19:54:00.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057cf6f3007dd148a-00653037d8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.990 DEBUG:boto:path=/bofrnf-8/%3A 2023-10-18T19:54:00.991 DEBUG:boto:auth_path=/bofrnf-8/%3A 2023-10-18T19:54:00.991 DEBUG:boto:Method: GET 2023-10-18T19:54:00.992 DEBUG:boto:Path: /bofrnf-8/%3A 2023-10-18T19:54:00.992 DEBUG:boto:Data: 2023-10-18T19:54:00.993 DEBUG:boto:Headers: {} 2023-10-18T19:54:00.993 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:00.993 DEBUG:boto:Port: 8001 2023-10-18T19:54:00.994 DEBUG:boto:Params: {} 2023-10-18T19:54:00.994 DEBUG:boto:Token: None 2023-10-18T19:54:00.995 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:00 GMT /bofrnf-8/%3A 2023-10-18T19:54:00.995 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fCxlZef+5uw3N3RqOnzt0nTr9V8= 2023-10-18T19:54:00.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fCxlZef+5uw3N3RqOnzt0nTr9V8=', 'Content-Length': '0'} 2023-10-18T19:54:00.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018cd1daaa1796f85-00653037d8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:00.998 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:00.999 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:00.999 DEBUG:boto:path=/bofrnf-8/?prefix=%3A 2023-10-18T19:54:01.000 DEBUG:boto:auth_path=/bofrnf-8/?prefix=%3A 2023-10-18T19:54:01.000 DEBUG:boto:Method: GET 2023-10-18T19:54:01.001 DEBUG:boto:Path: /bofrnf-8/?prefix=%3A 2023-10-18T19:54:01.001 DEBUG:boto:Data: 2023-10-18T19:54:01.002 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.002 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.002 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.003 DEBUG:boto:Params: {} 2023-10-18T19:54:01.003 DEBUG:boto:Token: None 2023-10-18T19:54:01.004 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081858e49c86d0e24-00653037d9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.008 DEBUG:boto:b'bofrnf-8:1000false:2023-10-18T19:49:39.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.008 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:01.009 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:01.009 DEBUG:boto:path=/bofrnf-8/?prefix=%3A 2023-10-18T19:54:01.009 DEBUG:boto:auth_path=/bofrnf-8/?prefix=%3A 2023-10-18T19:54:01.010 DEBUG:boto:Method: GET 2023-10-18T19:54:01.010 DEBUG:boto:Path: /bofrnf-8/?prefix=%3A 2023-10-18T19:54:01.011 DEBUG:boto:Data: 2023-10-18T19:54:01.011 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.012 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.012 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.012 DEBUG:boto:Params: {} 2023-10-18T19:54:01.013 DEBUG:boto:Token: None 2023-10-18T19:54:01.013 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.014 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8b1941c022292da-00653037d9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.017 DEBUG:boto:b'bofrnf-8:1000false:2023-10-18T19:49:39.274Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-18T19:54:01.018 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.019 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.019 DEBUG:boto:path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.020 DEBUG:boto:auth_path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.020 DEBUG:boto:Method: GET 2023-10-18T19:54:01.021 DEBUG:boto:Path: /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.021 DEBUG:boto:Data: 2023-10-18T19:54:01.022 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.022 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.023 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.023 DEBUG:boto:Params: {} 2023-10-18T19:54:01.023 DEBUG:boto:Token: None 2023-10-18T19:54:01.024 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.024 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OeGKkb+UT/xZilG/buTt9imOERQ= 2023-10-18T19:54:01.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OeGKkb+UT/xZilG/buTt9imOERQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a6a9de0519cf825-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.027 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.028 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.028 DEBUG:boto:path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.029 DEBUG:boto:auth_path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.029 DEBUG:boto:Method: GET 2023-10-18T19:54:01.030 DEBUG:boto:Path: /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.030 DEBUG:boto:Data: 2023-10-18T19:54:01.030 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.031 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.031 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.031 DEBUG:boto:Params: {} 2023-10-18T19:54:01.032 DEBUG:boto:Token: None 2023-10-18T19:54:01.032 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.033 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OeGKkb+UT/xZilG/buTt9imOERQ= 2023-10-18T19:54:01.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OeGKkb+UT/xZilG/buTt9imOERQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6857c4cf1340d41-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.036 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.036 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.037 DEBUG:boto:path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.037 DEBUG:boto:auth_path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.038 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.038 DEBUG:boto:Path: /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.038 DEBUG:boto:Data: 2023-10-18T19:54:01.039 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.039 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.040 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.040 DEBUG:boto:Params: {} 2023-10-18T19:54:01.040 DEBUG:boto:Token: None 2023-10-18T19:54:01.041 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.041 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:x9rpHSoOZs7GZpI0VXHB2vGmvfU= 2023-10-18T19:54:01.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x9rpHSoOZs7GZpI0VXHB2vGmvfU=', 'Content-Length': '0'} 2023-10-18T19:54:01.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7b670b2366b58da-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.044 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.044 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.045 DEBUG:boto:path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.045 DEBUG:boto:auth_path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.045 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.046 DEBUG:boto:Path: /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.046 DEBUG:boto:Data: 2023-10-18T19:54:01.046 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.047 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.047 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.047 DEBUG:boto:Params: {} 2023-10-18T19:54:01.048 DEBUG:boto:Token: None 2023-10-18T19:54:01.048 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:x9rpHSoOZs7GZpI0VXHB2vGmvfU= 2023-10-18T19:54:01.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x9rpHSoOZs7GZpI0VXHB2vGmvfU=', 'Content-Length': '0'} 2023-10-18T19:54:01.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000549fb1792dfbd705-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-18T19:54:01.052 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.053 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.053 DEBUG:boto:path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.053 DEBUG:boto:auth_path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.054 DEBUG:boto:Method: GET 2023-10-18T19:54:01.054 DEBUG:boto:Path: /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.055 DEBUG:boto:Data: 2023-10-18T19:54:01.055 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.056 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.056 DEBUG:boto:Params: {} 2023-10-18T19:54:01.057 DEBUG:boto:Token: None 2023-10-18T19:54:01.057 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.058 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OeGKkb+UT/xZilG/buTt9imOERQ= 2023-10-18T19:54:01.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OeGKkb+UT/xZilG/buTt9imOERQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a7037cad0fdc0eb-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.102 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.103 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.103 DEBUG:boto:path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.104 DEBUG:boto:auth_path=/bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.104 DEBUG:boto:Method: GET 2023-10-18T19:54:01.104 DEBUG:boto:Path: /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.105 DEBUG:boto:Data: 2023-10-18T19:54:01.105 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.106 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.106 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.106 DEBUG:boto:Params: {} 2023-10-18T19:54:01.107 DEBUG:boto:Token: None 2023-10-18T19:54:01.107 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj?versionId=null 2023-10-18T19:54:01.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OeGKkb+UT/xZilG/buTt9imOERQ= 2023-10-18T19:54:01.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OeGKkb+UT/xZilG/buTt9imOERQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096759a072bd2c28b-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.111 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:01.111 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:01.112 DEBUG:boto:path=/bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.112 DEBUG:boto:auth_path=/bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.112 DEBUG:boto:Method: GET 2023-10-18T19:54:01.113 DEBUG:boto:Path: /bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.113 DEBUG:boto:Data: 2023-10-18T19:54:01.114 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.114 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.114 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.115 DEBUG:boto:Params: {} 2023-10-18T19:54:01.115 DEBUG:boto:Token: None 2023-10-18T19:54:01.116 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.116 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000428c876c25ecb413-00653037d9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.119 DEBUG:boto:b'bofrnf-8_myobj1000false_myobj2023-10-18T19:49:39.214Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.120 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:01.120 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:01.121 DEBUG:boto:path=/bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.121 DEBUG:boto:auth_path=/bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.121 DEBUG:boto:Method: GET 2023-10-18T19:54:01.122 DEBUG:boto:Path: /bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.122 DEBUG:boto:Data: 2023-10-18T19:54:01.122 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.123 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.123 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.124 DEBUG:boto:Params: {} 2023-10-18T19:54:01.124 DEBUG:boto:Token: None 2023-10-18T19:54:01.124 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.125 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030dae406a1f5f2c9-00653037d9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.128 DEBUG:boto:b'bofrnf-8_myobj1000false_myobj2023-10-18T19:49:39.214Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.128 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.129 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.129 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.130 DEBUG:boto:Path: /bofrnf-8/_myobj 2023-10-18T19:54:01.130 DEBUG:boto:Data: 2023-10-18T19:54:01.130 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.131 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.131 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.131 DEBUG:boto:Params: {} 2023-10-18T19:54:01.132 DEBUG:boto:Token: None 2023-10-18T19:54:01.132 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj 2023-10-18T19:54:01.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nCS3+6AcWwhSMe2Is+V84kKSpWg= 2023-10-18T19:54:01.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nCS3+6AcWwhSMe2Is+V84kKSpWg=', 'Content-Length': '0'} 2023-10-18T19:54:01.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003317bd03ea9666dc-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.136 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.136 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.136 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.137 DEBUG:boto:Path: /bofrnf-8/_myobj 2023-10-18T19:54:01.137 DEBUG:boto:Data: 2023-10-18T19:54:01.137 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.138 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.138 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.139 DEBUG:boto:Params: {} 2023-10-18T19:54:01.139 DEBUG:boto:Token: None 2023-10-18T19:54:01.140 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj 2023-10-18T19:54:01.140 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nCS3+6AcWwhSMe2Is+V84kKSpWg= 2023-10-18T19:54:01.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nCS3+6AcWwhSMe2Is+V84kKSpWg=', 'Content-Length': '0'} 2023-10-18T19:54:01.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006174a44ddbccd31b-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-18T19:54:01.143 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.144 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.144 DEBUG:boto:Method: GET 2023-10-18T19:54:01.144 DEBUG:boto:Path: /bofrnf-8/_myobj 2023-10-18T19:54:01.145 DEBUG:boto:Data: 2023-10-18T19:54:01.145 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.145 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.146 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.146 DEBUG:boto:Params: {} 2023-10-18T19:54:01.147 DEBUG:boto:Token: None 2023-10-18T19:54:01.147 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj 2023-10-18T19:54:01.148 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:evm10OZY5fAZ4079n5qw1U28Zsw= 2023-10-18T19:54:01.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:evm10OZY5fAZ4079n5qw1U28Zsw=', 'Content-Length': '0'} 2023-10-18T19:54:01.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047ea5325b24989b3-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.190 DEBUG:boto:path=/bofrnf-8/_myobj 2023-10-18T19:54:01.191 DEBUG:boto:auth_path=/bofrnf-8/_myobj 2023-10-18T19:54:01.191 DEBUG:boto:Method: GET 2023-10-18T19:54:01.191 DEBUG:boto:Path: /bofrnf-8/_myobj 2023-10-18T19:54:01.192 DEBUG:boto:Data: 2023-10-18T19:54:01.192 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.193 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.193 DEBUG:boto:Params: {} 2023-10-18T19:54:01.194 DEBUG:boto:Token: None 2023-10-18T19:54:01.194 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/_myobj 2023-10-18T19:54:01.195 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:evm10OZY5fAZ4079n5qw1U28Zsw= 2023-10-18T19:54:01.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:evm10OZY5fAZ4079n5qw1U28Zsw=', 'Content-Length': '0'} 2023-10-18T19:54:01.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c21ba91243866ccb-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.198 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:01.198 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:01.199 DEBUG:boto:path=/bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.199 DEBUG:boto:auth_path=/bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.199 DEBUG:boto:Method: GET 2023-10-18T19:54:01.200 DEBUG:boto:Path: /bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.200 DEBUG:boto:Data: 2023-10-18T19:54:01.201 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.201 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.201 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.202 DEBUG:boto:Params: {} 2023-10-18T19:54:01.202 DEBUG:boto:Token: None 2023-10-18T19:54:01.203 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.203 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f91d3123f29dce1-00653037d9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.207 DEBUG:boto:b'bofrnf-8_myobj1000false_myobj2023-10-18T19:49:39.214Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.207 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:01.207 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:01.208 DEBUG:boto:path=/bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.208 DEBUG:boto:auth_path=/bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.209 DEBUG:boto:Method: GET 2023-10-18T19:54:01.209 DEBUG:boto:Path: /bofrnf-8/?prefix=_myobj 2023-10-18T19:54:01.209 DEBUG:boto:Data: 2023-10-18T19:54:01.210 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.211 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.211 DEBUG:boto:Params: {} 2023-10-18T19:54:01.211 DEBUG:boto:Token: None 2023-10-18T19:54:01.212 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ce24b2bc5e1ae68-00653037d9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.216 DEBUG:boto:b'bofrnf-8_myobj1000false_myobj2023-10-18T19:49:39.214Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:01.217 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.217 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.217 DEBUG:boto:path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.218 DEBUG:boto:auth_path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.218 DEBUG:boto:Method: GET 2023-10-18T19:54:01.219 DEBUG:boto:Path: /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.219 DEBUG:boto:Data: 2023-10-18T19:54:01.220 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.220 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.220 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.221 DEBUG:boto:Params: {} 2023-10-18T19:54:01.221 DEBUG:boto:Token: None 2023-10-18T19:54:01.222 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.222 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lapONr9waWi4Wm+6A9rBeWx8TYs= 2023-10-18T19:54:01.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lapONr9waWi4Wm+6A9rBeWx8TYs=', 'Content-Length': '0'} 2023-10-18T19:54:01.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe3de28e28d63f9d-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.225 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.226 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.226 DEBUG:boto:path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.226 DEBUG:boto:auth_path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.227 DEBUG:boto:Method: GET 2023-10-18T19:54:01.227 DEBUG:boto:Path: /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.228 DEBUG:boto:Data: 2023-10-18T19:54:01.228 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.228 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.229 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.229 DEBUG:boto:Params: {} 2023-10-18T19:54:01.230 DEBUG:boto:Token: None 2023-10-18T19:54:01.230 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.230 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lapONr9waWi4Wm+6A9rBeWx8TYs= 2023-10-18T19:54:01.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lapONr9waWi4Wm+6A9rBeWx8TYs=', 'Content-Length': '0'} 2023-10-18T19:54:01.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d24a790ce595b1c-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.274 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.275 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.275 DEBUG:boto:path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.275 DEBUG:boto:auth_path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.276 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.276 DEBUG:boto:Path: /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.276 DEBUG:boto:Data: 2023-10-18T19:54:01.277 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.277 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.278 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.278 DEBUG:boto:Params: {} 2023-10-18T19:54:01.278 DEBUG:boto:Token: None 2023-10-18T19:54:01.279 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.279 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KST3uQW3rYi+svMbvOfjHb6+D0U= 2023-10-18T19:54:01.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KST3uQW3rYi+svMbvOfjHb6+D0U=', 'Content-Length': '0'} 2023-10-18T19:54:01.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011347aa0d3ec28d6-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.282 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.283 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.283 DEBUG:boto:path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.283 DEBUG:boto:auth_path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.284 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.284 DEBUG:boto:Path: /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.284 DEBUG:boto:Data: 2023-10-18T19:54:01.285 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.285 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.286 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.286 DEBUG:boto:Params: {} 2023-10-18T19:54:01.286 DEBUG:boto:Token: None 2023-10-18T19:54:01.287 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.287 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KST3uQW3rYi+svMbvOfjHb6+D0U= 2023-10-18T19:54:01.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KST3uQW3rYi+svMbvOfjHb6+D0U=', 'Content-Length': '0'} 2023-10-18T19:54:01.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b2a5b57e6fdc322-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:01.290 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.291 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.291 DEBUG:boto:path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.292 DEBUG:boto:auth_path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.292 DEBUG:boto:Method: GET 2023-10-18T19:54:01.293 DEBUG:boto:Path: /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.293 DEBUG:boto:Data: 2023-10-18T19:54:01.293 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.294 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.294 DEBUG:boto:Params: {} 2023-10-18T19:54:01.295 DEBUG:boto:Token: None 2023-10-18T19:54:01.295 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.296 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lapONr9waWi4Wm+6A9rBeWx8TYs= 2023-10-18T19:54:01.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lapONr9waWi4Wm+6A9rBeWx8TYs=', 'Content-Length': '0'} 2023-10-18T19:54:01.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed499adcc1895ec4-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.299 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.299 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.300 DEBUG:boto:path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.300 DEBUG:boto:auth_path=/bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.300 DEBUG:boto:Method: GET 2023-10-18T19:54:01.301 DEBUG:boto:Path: /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.301 DEBUG:boto:Data: 2023-10-18T19:54:01.302 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.303 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.303 DEBUG:boto:Params: {} 2023-10-18T19:54:01.303 DEBUG:boto:Token: None 2023-10-18T19:54:01.304 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj?versionId=null 2023-10-18T19:54:01.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lapONr9waWi4Wm+6A9rBeWx8TYs= 2023-10-18T19:54:01.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lapONr9waWi4Wm+6A9rBeWx8TYs=', 'Content-Length': '0'} 2023-10-18T19:54:01.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2c752902ce158c9-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.307 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:01.307 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:01.308 DEBUG:boto:path=/bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.308 DEBUG:boto:auth_path=/bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.309 DEBUG:boto:Method: GET 2023-10-18T19:54:01.309 DEBUG:boto:Path: /bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.310 DEBUG:boto:Data: 2023-10-18T19:54:01.310 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.311 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.311 DEBUG:boto:Params: {} 2023-10-18T19:54:01.311 DEBUG:boto:Token: None 2023-10-18T19:54:01.312 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b21f4827fefa1775-00653037d9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.315 DEBUG:boto:b'bofrnf-8myobj1000falsemyobj2023-10-18T19:49:39.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.316 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:01.316 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:01.317 DEBUG:boto:path=/bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.317 DEBUG:boto:auth_path=/bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.318 DEBUG:boto:Method: GET 2023-10-18T19:54:01.318 DEBUG:boto:Path: /bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.318 DEBUG:boto:Data: 2023-10-18T19:54:01.319 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.320 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.320 DEBUG:boto:Params: {} 2023-10-18T19:54:01.320 DEBUG:boto:Token: None 2023-10-18T19:54:01.321 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000162950f28c5dd5fa-00653037d9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.325 DEBUG:boto:b'bofrnf-8myobj1000falsemyobj2023-10-18T19:49:39.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.325 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.326 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.326 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.326 DEBUG:boto:Path: /bofrnf-8/myobj 2023-10-18T19:54:01.327 DEBUG:boto:Data: 2023-10-18T19:54:01.327 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.328 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.328 DEBUG:boto:Params: {} 2023-10-18T19:54:01.329 DEBUG:boto:Token: None 2023-10-18T19:54:01.329 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj 2023-10-18T19:54:01.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wJiGT8Rax5o2d8g1gBuGYGc8Xfo= 2023-10-18T19:54:01.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wJiGT8Rax5o2d8g1gBuGYGc8Xfo=', 'Content-Length': '0'} 2023-10-18T19:54:01.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cc63dfb63064e6c-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.332 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.333 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.333 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.334 DEBUG:boto:Path: /bofrnf-8/myobj 2023-10-18T19:54:01.334 DEBUG:boto:Data: 2023-10-18T19:54:01.334 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.335 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.335 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.335 DEBUG:boto:Params: {} 2023-10-18T19:54:01.336 DEBUG:boto:Token: None 2023-10-18T19:54:01.336 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj 2023-10-18T19:54:01.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wJiGT8Rax5o2d8g1gBuGYGc8Xfo= 2023-10-18T19:54:01.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wJiGT8Rax5o2d8g1gBuGYGc8Xfo=', 'Content-Length': '0'} 2023-10-18T19:54:01.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054c7871c1bd1bfa6-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:01.340 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.340 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.340 DEBUG:boto:Method: GET 2023-10-18T19:54:01.341 DEBUG:boto:Path: /bofrnf-8/myobj 2023-10-18T19:54:01.341 DEBUG:boto:Data: 2023-10-18T19:54:01.341 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.342 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.342 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.343 DEBUG:boto:Params: {} 2023-10-18T19:54:01.343 DEBUG:boto:Token: None 2023-10-18T19:54:01.344 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj 2023-10-18T19:54:01.344 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:crfJl/g8YjqrJJFLzd6+vii6f7o= 2023-10-18T19:54:01.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:crfJl/g8YjqrJJFLzd6+vii6f7o=', 'Content-Length': '0'} 2023-10-18T19:54:01.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b69be8d0914a01e-00653037d9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.390 DEBUG:boto:path=/bofrnf-8/myobj 2023-10-18T19:54:01.391 DEBUG:boto:auth_path=/bofrnf-8/myobj 2023-10-18T19:54:01.391 DEBUG:boto:Method: GET 2023-10-18T19:54:01.391 DEBUG:boto:Path: /bofrnf-8/myobj 2023-10-18T19:54:01.392 DEBUG:boto:Data: 2023-10-18T19:54:01.392 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.393 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.393 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.393 DEBUG:boto:Params: {} 2023-10-18T19:54:01.394 DEBUG:boto:Token: None 2023-10-18T19:54:01.394 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/myobj 2023-10-18T19:54:01.395 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:crfJl/g8YjqrJJFLzd6+vii6f7o= 2023-10-18T19:54:01.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:crfJl/g8YjqrJJFLzd6+vii6f7o=', 'Content-Length': '0'} 2023-10-18T19:54:01.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:49:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de9fb8bf95b91c58-00653037d9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.398 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:01.398 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:01.399 DEBUG:boto:path=/bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.399 DEBUG:boto:auth_path=/bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.399 DEBUG:boto:Method: GET 2023-10-18T19:54:01.400 DEBUG:boto:Path: /bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.400 DEBUG:boto:Data: 2023-10-18T19:54:01.401 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.402 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.402 DEBUG:boto:Params: {} 2023-10-18T19:54:01.402 DEBUG:boto:Token: None 2023-10-18T19:54:01.403 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.403 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053e7d24ee1a2947f-00653037d9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.407 DEBUG:boto:b'bofrnf-8myobj1000falsemyobj2023-10-18T19:49:39.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.407 DEBUG:boto:path=/bofrnf-8/ 2023-10-18T19:54:01.408 DEBUG:boto:auth_path=/bofrnf-8/ 2023-10-18T19:54:01.408 DEBUG:boto:path=/bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.408 DEBUG:boto:auth_path=/bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.409 DEBUG:boto:Method: GET 2023-10-18T19:54:01.409 DEBUG:boto:Path: /bofrnf-8/?prefix=myobj 2023-10-18T19:54:01.410 DEBUG:boto:Data: 2023-10-18T19:54:01.410 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.411 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.411 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.411 DEBUG:boto:Params: {} 2023-10-18T19:54:01.412 DEBUG:boto:Token: None 2023-10-18T19:54:01.412 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-8/ 2023-10-18T19:54:01.413 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ= 2023-10-18T19:54:01.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3cCq73FSCeVbX+J1FKodfAuCJWQ=', 'Content-Length': '0'} 2023-10-18T19:54:01.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003793b20824cc8c43-00653037d9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.416 DEBUG:boto:b'bofrnf-8myobj1000falsemyobj2023-10-18T19:49:39.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:01.417 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-8 zones={a1, a2} 2023-10-18T19:54:01.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-10-18T19:54:01.419 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-9 2023-10-18T19:54:01.419 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:01.419 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:01.420 DEBUG:boto:Method: PUT 2023-10-18T19:54:01.420 DEBUG:boto:Path: /bofrnf-9/ 2023-10-18T19:54:01.420 DEBUG:boto:Data: 2023-10-18T19:54:01.421 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.421 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.421 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.422 DEBUG:boto:Params: {} 2023-10-18T19:54:01.422 DEBUG:boto:Token: None 2023-10-18T19:54:01.423 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-9/ 2023-10-18T19:54:01.423 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XmUF4oUm2SQkH3JesYFCZEL3rlU= 2023-10-18T19:54:01.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XmUF4oUm2SQkH3JesYFCZEL3rlU=', 'Content-Length': '0'} 2023-10-18T19:54:01.437 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e6674e6b7726dcb-00653037d9-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.438 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-10 2023-10-18T19:54:01.438 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:01.439 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:01.439 DEBUG:boto:Method: PUT 2023-10-18T19:54:01.439 DEBUG:boto:Path: /bofrnf-10/ 2023-10-18T19:54:01.440 DEBUG:boto:Data: 2023-10-18T19:54:01.440 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.441 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.441 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.441 DEBUG:boto:Params: {} 2023-10-18T19:54:01.442 DEBUG:boto:Token: None 2023-10-18T19:54:01.442 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-10/ 2023-10-18T19:54:01.443 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ljlbBRhGIYfjtFjiXyxxIHobSNc= 2023-10-18T19:54:01.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ljlbBRhGIYfjtFjiXyxxIHobSNc=', 'Content-Length': '0'} 2023-10-18T19:54:01.465 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a262b5dd739b5ca2-00653037d9-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.466 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:01.466 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:01.467 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.467 DEBUG:boto:Path: /bofrnf-9/ 2023-10-18T19:54:01.468 DEBUG:boto:Data: 2023-10-18T19:54:01.468 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.468 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.469 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.469 DEBUG:boto:Params: {} 2023-10-18T19:54:01.470 DEBUG:boto:Token: None 2023-10-18T19:54:01.470 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-9/ 2023-10-18T19:54:01.471 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y1luTRu3DSo1WpdYnVd2cqO7kpI= 2023-10-18T19:54:01.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y1luTRu3DSo1WpdYnVd2cqO7kpI=', 'Content-Length': '0'} 2023-10-18T19:54:01.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a129d9f180a246cf-00653037d9-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.474 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:01.474 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:01.475 DEBUG:boto:Method: PUT 2023-10-18T19:54:01.475 DEBUG:boto:Path: /bofrnf-9/myobj 2023-10-18T19:54:01.475 DEBUG:boto:Data: 2023-10-18T19:54:01.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:54:01.476 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:01.477 DEBUG:boto:Port: 8000 2023-10-18T19:54:01.477 DEBUG:boto:Params: {} 2023-10-18T19:54:01.477 DEBUG:boto:Token: None 2023-10-18T19:54:01.478 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-9/myobj 2023-10-18T19:54:01.478 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H2z4Y4docJFGgQ00sI96s6sMMKE= 2023-10-18T19:54:01.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H2z4Y4docJFGgQ00sI96s6sMMKE='} 2023-10-18T19:54:01.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d9ae27110a75264-00653037d9-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.485 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:01.486 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:01.486 DEBUG:boto:Method: HEAD 2023-10-18T19:54:01.486 DEBUG:boto:Path: /bofrnf-10/ 2023-10-18T19:54:01.487 DEBUG:boto:Data: 2023-10-18T19:54:01.487 DEBUG:boto:Headers: {} 2023-10-18T19:54:01.488 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.488 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.488 DEBUG:boto:Params: {} 2023-10-18T19:54:01.489 DEBUG:boto:Token: None 2023-10-18T19:54:01.489 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-10/ 2023-10-18T19:54:01.490 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HRKT95UEf2UJh1AjGOm598X1Gwo= 2023-10-18T19:54:01.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HRKT95UEf2UJh1AjGOm598X1Gwo=', 'Content-Length': '0'} 2023-10-18T19:54:01.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068f5f205b1f279fe-00653037d9-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.493 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:01.493 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:01.494 DEBUG:boto:Method: PUT 2023-10-18T19:54:01.494 DEBUG:boto:Path: /bofrnf-10/myobj 2023-10-18T19:54:01.495 DEBUG:boto:Data: 2023-10-18T19:54:01.495 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T19:54:01.495 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:01.496 DEBUG:boto:Port: 8001 2023-10-18T19:54:01.496 DEBUG:boto:Params: {} 2023-10-18T19:54:01.496 DEBUG:boto:Token: None 2023-10-18T19:54:01.497 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 19:54:01 GMT /bofrnf-10/myobj 2023-10-18T19:54:01.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3SHJLg/MT76KFYGm1zFNSbypeFc= 2023-10-18T19:54:01.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:54:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3SHJLg/MT76KFYGm1zFNSbypeFc='} 2023-10-18T19:54:01.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fcb51d9bba89947-00653037d9-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:01.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:54:01.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:54:01.548 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:54:01.599 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:54:01.600 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:54:01.633 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:54:01.644 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:54:01.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:01.644 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.455783_144.1", 2023-10-18T19:54:01.645 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.455783Z" 2023-10-18T19:54:01.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:01.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:01.645 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:54:01.645 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T19:54:01.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:01.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:01.646 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:54:01.646 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:54:01.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:01.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:01.647 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:54:01.647 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:54:01.647 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:01.647 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:54:01.648 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697658841.455783_144.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:54:01.649 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:54:01.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:54:01.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:54:01.650 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:54:01.699 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:54:01.700 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:54:01.727 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:01.725+0000 7f0fffca7780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:54:01.728 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:01.726+0000 7f0fffca7780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:54:01.739 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:54:01.739 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:54:01.739 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:54:01.740 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:54:01.740 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:54:01.740 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:54:01.740 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:01.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:01.741 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:54:01.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:01.741 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:54:01.741 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:54:01.741 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:54:01.742 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.455783_144.1", 2023-10-18T19:54:01.742 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:54:01.742 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:54:01.742 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:54:01.742 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.455783Z", 2023-10-18T19:54:01.743 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:01.743 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:01.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:01.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:01.743 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:54:01.744 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:54:01.744 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:54:01.744 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:54:01.744 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:54:01.744 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:54:01.744 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:54:01.745 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T19:54:01.745 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:01.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:01.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:01.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:01.746 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:54:01.746 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:54:01.746 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:54:01.746 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:54:01.746 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:54:01.747 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:54:01.747 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:54:01.747 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:54:01.747 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:01.747 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:01.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:01.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:01.748 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:54:01.748 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:54:01.748 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:54:01.749 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:54:01.749 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:54:01.749 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:54:01.749 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:54:01.749 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:54:01.749 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:01.750 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:01.750 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:01.750 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:54:01.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:01.750 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:54:01.751 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:54:01.751 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:54:01.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:01.751 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:54:01Z" 2023-10-18T19:54:01.751 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:54:01.752 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658841.455783_144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:54:01.455783Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:54:01Z" } 2023-10-18T19:54:01.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658841.455783_144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.455783Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:54:01.753 DEBUG:rgw_multi.tests:log_status={0: '1_1697658841.455783_144.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:54:01.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658841.455783_144.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:54:01.754 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:54:01.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:01.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:01.755 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:54:01.811 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:54:01.811 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:54:01.843 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:01.841+0000 7f374ba82780 1 waiting to reach incremental sync.. 2023-10-18T19:54:06.843 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:06.842+0000 7f374ba82780 1 waiting to reach incremental sync.. 2023-10-18T19:54:11.844 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:11.842+0000 7f374ba82780 1 waiting to reach incremental sync.. 2023-10-18T19:54:16.845 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:16.844+0000 7f374ba82780 1 bucket sync caught up with source: 2023-10-18T19:54:16.846 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-10-18T19:54:16.846 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.7.6, , , , , , , , , ] 2023-10-18T19:54:16.847 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:16.844+0000 7f374ba82780 0 bucket checkpoint complete 2023-10-18T19:54:16.861 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-9 zones={a2, a1} 2023-10-18T19:54:16.861 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:16.862 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:16.862 DEBUG:boto:Method: HEAD 2023-10-18T19:54:16.863 DEBUG:boto:Path: /bofrnf-9/ 2023-10-18T19:54:16.863 DEBUG:boto:Data: 2023-10-18T19:54:16.864 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.864 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:16.864 DEBUG:boto:Port: 8001 2023-10-18T19:54:16.865 DEBUG:boto:Params: {} 2023-10-18T19:54:16.865 DEBUG:boto:Token: None 2023-10-18T19:54:16.866 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/ 2023-10-18T19:54:16.866 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:empdEePQhzm/sYsheBzYT7rOGFI= 2023-10-18T19:54:16.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:empdEePQhzm/sYsheBzYT7rOGFI=', 'Content-Length': '0'} 2023-10-18T19:54:16.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f204f0677c763ecd-00653037e8-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.870 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:16.870 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:16.871 DEBUG:boto:Method: HEAD 2023-10-18T19:54:16.871 DEBUG:boto:Path: /bofrnf-9/ 2023-10-18T19:54:16.871 DEBUG:boto:Data: 2023-10-18T19:54:16.872 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.872 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:16.873 DEBUG:boto:Port: 8000 2023-10-18T19:54:16.873 DEBUG:boto:Params: {} 2023-10-18T19:54:16.873 DEBUG:boto:Token: None 2023-10-18T19:54:16.874 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/ 2023-10-18T19:54:16.874 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:empdEePQhzm/sYsheBzYT7rOGFI= 2023-10-18T19:54:16.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:empdEePQhzm/sYsheBzYT7rOGFI=', 'Content-Length': '0'} 2023-10-18T19:54:16.877 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b129b965901679c-00653037e8-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.878 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:54:16.878 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:16.879 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:16.879 DEBUG:boto:path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.880 DEBUG:boto:auth_path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.880 DEBUG:boto:Method: GET 2023-10-18T19:54:16.880 DEBUG:boto:Path: /bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.881 DEBUG:boto:Data: 2023-10-18T19:54:16.881 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:16.882 DEBUG:boto:Port: 8001 2023-10-18T19:54:16.883 DEBUG:boto:Params: {} 2023-10-18T19:54:16.883 DEBUG:boto:Token: None 2023-10-18T19:54:16.884 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/?versions 2023-10-18T19:54:16.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:69vyzir9nW/YOdHyPnKb1WG680E= 2023-10-18T19:54:16.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:69vyzir9nW/YOdHyPnKb1WG680E=', 'Content-Length': '0'} 2023-10-18T19:54:16.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8a2c02d07654e2c-00653037e8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.887 DEBUG:boto:b'bofrnf-9999falsemyobjnulltrue2023-10-18T19:54:01.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:16.888 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-18T19:54:16.888 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:54:16.889 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:16.889 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:16.890 DEBUG:boto:path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.890 DEBUG:boto:auth_path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.891 DEBUG:boto:Method: GET 2023-10-18T19:54:16.891 DEBUG:boto:Path: /bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.891 DEBUG:boto:Data: 2023-10-18T19:54:16.892 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.892 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:16.893 DEBUG:boto:Port: 8000 2023-10-18T19:54:16.893 DEBUG:boto:Params: {} 2023-10-18T19:54:16.893 DEBUG:boto:Token: None 2023-10-18T19:54:16.894 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/?versions 2023-10-18T19:54:16.894 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:69vyzir9nW/YOdHyPnKb1WG680E= 2023-10-18T19:54:16.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:69vyzir9nW/YOdHyPnKb1WG680E=', 'Content-Length': '0'} 2023-10-18T19:54:16.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ea4f8a486ee18d8-00653037e8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.898 DEBUG:boto:b'bofrnf-9999falsemyobjnulltrue2023-10-18T19:54:01.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:16.899 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-18T19:54:16.899 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:16.900 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:16.900 DEBUG:boto:path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.901 DEBUG:boto:auth_path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.901 DEBUG:boto:Method: GET 2023-10-18T19:54:16.901 DEBUG:boto:Path: /bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.902 DEBUG:boto:Data: 2023-10-18T19:54:16.902 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.903 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:16.903 DEBUG:boto:Port: 8001 2023-10-18T19:54:16.904 DEBUG:boto:Params: {} 2023-10-18T19:54:16.904 DEBUG:boto:Token: None 2023-10-18T19:54:16.904 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/?versions 2023-10-18T19:54:16.905 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:69vyzir9nW/YOdHyPnKb1WG680E= 2023-10-18T19:54:16.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:69vyzir9nW/YOdHyPnKb1WG680E=', 'Content-Length': '0'} 2023-10-18T19:54:16.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153bc2c437f131bb-00653037e8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.908 DEBUG:boto:b'bofrnf-9999falsemyobjnulltrue2023-10-18T19:54:01.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:16.909 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:16.909 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:16.910 DEBUG:boto:path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.910 DEBUG:boto:auth_path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.911 DEBUG:boto:Method: GET 2023-10-18T19:54:16.911 DEBUG:boto:Path: /bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:16.911 DEBUG:boto:Data: 2023-10-18T19:54:16.912 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.912 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:16.913 DEBUG:boto:Port: 8000 2023-10-18T19:54:16.913 DEBUG:boto:Params: {} 2023-10-18T19:54:16.913 DEBUG:boto:Token: None 2023-10-18T19:54:16.914 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/?versions 2023-10-18T19:54:16.914 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:69vyzir9nW/YOdHyPnKb1WG680E= 2023-10-18T19:54:16.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:69vyzir9nW/YOdHyPnKb1WG680E=', 'Content-Length': '0'} 2023-10-18T19:54:16.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ed6134335bddfc3-00653037e8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.918 DEBUG:boto:b'bofrnf-9999falsemyobjnulltrue2023-10-18T19:54:01.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:16.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:16.919 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:16.919 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:16.920 DEBUG:boto:path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.920 DEBUG:boto:auth_path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.921 DEBUG:boto:Method: GET 2023-10-18T19:54:16.921 DEBUG:boto:Path: /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.921 DEBUG:boto:Data: 2023-10-18T19:54:16.922 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.922 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:16.923 DEBUG:boto:Port: 8001 2023-10-18T19:54:16.923 DEBUG:boto:Params: {} 2023-10-18T19:54:16.924 DEBUG:boto:Token: None 2023-10-18T19:54:16.924 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.925 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uNX057jemO2rRlCc75JmJvjQ4XA= 2023-10-18T19:54:16.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uNX057jemO2rRlCc75JmJvjQ4XA=', 'Content-Length': '0'} 2023-10-18T19:54:16.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002494d421bb931f00-00653037e8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.928 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:16.928 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:16.929 DEBUG:boto:path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.929 DEBUG:boto:auth_path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.930 DEBUG:boto:Method: GET 2023-10-18T19:54:16.930 DEBUG:boto:Path: /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.930 DEBUG:boto:Data: 2023-10-18T19:54:16.931 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:16.932 DEBUG:boto:Port: 8000 2023-10-18T19:54:16.932 DEBUG:boto:Params: {} 2023-10-18T19:54:16.933 DEBUG:boto:Token: None 2023-10-18T19:54:16.933 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uNX057jemO2rRlCc75JmJvjQ4XA= 2023-10-18T19:54:16.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uNX057jemO2rRlCc75JmJvjQ4XA=', 'Content-Length': '0'} 2023-10-18T19:54:16.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000185b7a505a08e767-00653037e8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.937 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:16.937 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:16.938 DEBUG:boto:path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.938 DEBUG:boto:auth_path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.939 DEBUG:boto:Method: HEAD 2023-10-18T19:54:16.939 DEBUG:boto:Path: /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.939 DEBUG:boto:Data: 2023-10-18T19:54:16.940 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.940 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:16.941 DEBUG:boto:Port: 8001 2023-10-18T19:54:16.941 DEBUG:boto:Params: {} 2023-10-18T19:54:16.941 DEBUG:boto:Token: None 2023-10-18T19:54:16.942 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.942 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNRMxJs05WukxeA2dIoJukXwDYg= 2023-10-18T19:54:16.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNRMxJs05WukxeA2dIoJukXwDYg=', 'Content-Length': '0'} 2023-10-18T19:54:16.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007dace1b4cb7aa679-00653037e8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.945 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:16.946 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:16.946 DEBUG:boto:path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.947 DEBUG:boto:auth_path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.947 DEBUG:boto:Method: HEAD 2023-10-18T19:54:16.948 DEBUG:boto:Path: /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.948 DEBUG:boto:Data: 2023-10-18T19:54:16.949 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:16.949 DEBUG:boto:Port: 8000 2023-10-18T19:54:16.950 DEBUG:boto:Params: {} 2023-10-18T19:54:16.950 DEBUG:boto:Token: None 2023-10-18T19:54:16.951 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNRMxJs05WukxeA2dIoJukXwDYg= 2023-10-18T19:54:16.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNRMxJs05WukxeA2dIoJukXwDYg=', 'Content-Length': '0'} 2023-10-18T19:54:16.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8f8eea5ba1666d9-00653037e8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:16.955 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:16.955 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:16.956 DEBUG:boto:path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.956 DEBUG:boto:auth_path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.956 DEBUG:boto:Method: GET 2023-10-18T19:54:16.957 DEBUG:boto:Path: /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.957 DEBUG:boto:Data: 2023-10-18T19:54:16.958 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:16.959 DEBUG:boto:Port: 8001 2023-10-18T19:54:16.959 DEBUG:boto:Params: {} 2023-10-18T19:54:16.960 DEBUG:boto:Token: None 2023-10-18T19:54:16.960 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.961 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uNX057jemO2rRlCc75JmJvjQ4XA= 2023-10-18T19:54:16.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uNX057jemO2rRlCc75JmJvjQ4XA=', 'Content-Length': '0'} 2023-10-18T19:54:16.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f9389f6e0411180-00653037e8-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.964 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:16.964 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:16.965 DEBUG:boto:path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.965 DEBUG:boto:auth_path=/bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.965 DEBUG:boto:Method: GET 2023-10-18T19:54:16.966 DEBUG:boto:Path: /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.966 DEBUG:boto:Data: 2023-10-18T19:54:16.967 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.967 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:16.968 DEBUG:boto:Port: 8000 2023-10-18T19:54:16.968 DEBUG:boto:Params: {} 2023-10-18T19:54:16.969 DEBUG:boto:Token: None 2023-10-18T19:54:16.969 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/myobj?versionId=null 2023-10-18T19:54:16.970 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uNX057jemO2rRlCc75JmJvjQ4XA= 2023-10-18T19:54:16.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uNX057jemO2rRlCc75JmJvjQ4XA=', 'Content-Length': '0'} 2023-10-18T19:54:16.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1bb48b8f65d1818-00653037e8-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:16.973 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:16.974 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:16.974 DEBUG:boto:path=/bofrnf-9/?prefix=myobj 2023-10-18T19:54:16.975 DEBUG:boto:auth_path=/bofrnf-9/?prefix=myobj 2023-10-18T19:54:16.975 DEBUG:boto:Method: GET 2023-10-18T19:54:16.975 DEBUG:boto:Path: /bofrnf-9/?prefix=myobj 2023-10-18T19:54:16.976 DEBUG:boto:Data: 2023-10-18T19:54:16.976 DEBUG:boto:Headers: {} 2023-10-18T19:54:16.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:16.977 DEBUG:boto:Port: 8001 2023-10-18T19:54:16.978 DEBUG:boto:Params: {} 2023-10-18T19:54:16.978 DEBUG:boto:Token: None 2023-10-18T19:54:16.979 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:16 GMT /bofrnf-9/ 2023-10-18T19:54:16.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hYlwFN6POGhSTSYEZmSkDKuGEdw= 2023-10-18T19:54:16.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hYlwFN6POGhSTSYEZmSkDKuGEdw=', 'Content-Length': '0'} 2023-10-18T19:54:16.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e91b1b8e40f36249-00653037e8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.026 DEBUG:boto:b'bofrnf-9myobj1000falsemyobj2023-10-18T19:54:01.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.027 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:17.028 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:17.028 DEBUG:boto:path=/bofrnf-9/?prefix=myobj 2023-10-18T19:54:17.029 DEBUG:boto:auth_path=/bofrnf-9/?prefix=myobj 2023-10-18T19:54:17.029 DEBUG:boto:Method: GET 2023-10-18T19:54:17.029 DEBUG:boto:Path: /bofrnf-9/?prefix=myobj 2023-10-18T19:54:17.030 DEBUG:boto:Data: 2023-10-18T19:54:17.030 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.031 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.031 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.032 DEBUG:boto:Params: {} 2023-10-18T19:54:17.032 DEBUG:boto:Token: None 2023-10-18T19:54:17.033 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/ 2023-10-18T19:54:17.033 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tOk9X5IfuN7v/IyLe2iFgNmXiYU= 2023-10-18T19:54:17.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tOk9X5IfuN7v/IyLe2iFgNmXiYU=', 'Content-Length': '0'} 2023-10-18T19:54:17.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5eac6f199aed447-00653037e9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.037 DEBUG:boto:b'bofrnf-9myobj1000falsemyobj2023-10-18T19:54:01.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.038 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:17.038 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:17.038 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.039 DEBUG:boto:Path: /bofrnf-9/myobj 2023-10-18T19:54:17.039 DEBUG:boto:Data: 2023-10-18T19:54:17.042 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.042 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.043 DEBUG:boto:Params: {} 2023-10-18T19:54:17.043 DEBUG:boto:Token: None 2023-10-18T19:54:17.044 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/myobj 2023-10-18T19:54:17.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XMSLk2QUvkoz6ZX1ilxk3CQVCVU= 2023-10-18T19:54:17.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XMSLk2QUvkoz6ZX1ilxk3CQVCVU=', 'Content-Length': '0'} 2023-10-18T19:54:17.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d018b64e4c724422-00653037e9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.047 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:17.048 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:17.048 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.048 DEBUG:boto:Path: /bofrnf-9/myobj 2023-10-18T19:54:17.049 DEBUG:boto:Data: 2023-10-18T19:54:17.049 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.049 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.050 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.050 DEBUG:boto:Params: {} 2023-10-18T19:54:17.051 DEBUG:boto:Token: None 2023-10-18T19:54:17.051 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/myobj 2023-10-18T19:54:17.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XMSLk2QUvkoz6ZX1ilxk3CQVCVU= 2023-10-18T19:54:17.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XMSLk2QUvkoz6ZX1ilxk3CQVCVU=', 'Content-Length': '0'} 2023-10-18T19:54:17.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e0fdef1cfadaf08-00653037e9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:17.055 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:17.055 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:17.056 DEBUG:boto:Method: GET 2023-10-18T19:54:17.056 DEBUG:boto:Path: /bofrnf-9/myobj 2023-10-18T19:54:17.057 DEBUG:boto:Data: 2023-10-18T19:54:17.057 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.057 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.058 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.058 DEBUG:boto:Params: {} 2023-10-18T19:54:17.059 DEBUG:boto:Token: None 2023-10-18T19:54:17.059 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/myobj 2023-10-18T19:54:17.060 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QWVBYyv70c1kH0pCg099AsjA28E= 2023-10-18T19:54:17.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QWVBYyv70c1kH0pCg099AsjA28E=', 'Content-Length': '0'} 2023-10-18T19:54:17.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b7e424f8e704484-00653037e9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.063 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:17.063 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:17.064 DEBUG:boto:Method: GET 2023-10-18T19:54:17.064 DEBUG:boto:Path: /bofrnf-9/myobj 2023-10-18T19:54:17.064 DEBUG:boto:Data: 2023-10-18T19:54:17.065 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.065 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.066 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.066 DEBUG:boto:Params: {} 2023-10-18T19:54:17.067 DEBUG:boto:Token: None 2023-10-18T19:54:17.067 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/myobj 2023-10-18T19:54:17.068 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QWVBYyv70c1kH0pCg099AsjA28E= 2023-10-18T19:54:17.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QWVBYyv70c1kH0pCg099AsjA28E=', 'Content-Length': '0'} 2023-10-18T19:54:17.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9d3f2ff2ca0cec0-00653037e9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.071 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:17.072 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:17.072 DEBUG:boto:path=/bofrnf-9/?prefix=myobj 2023-10-18T19:54:17.072 DEBUG:boto:auth_path=/bofrnf-9/?prefix=myobj 2023-10-18T19:54:17.073 DEBUG:boto:Method: GET 2023-10-18T19:54:17.073 DEBUG:boto:Path: /bofrnf-9/?prefix=myobj 2023-10-18T19:54:17.074 DEBUG:boto:Data: 2023-10-18T19:54:17.074 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.075 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.075 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.075 DEBUG:boto:Params: {} 2023-10-18T19:54:17.076 DEBUG:boto:Token: None 2023-10-18T19:54:17.076 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/ 2023-10-18T19:54:17.077 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tOk9X5IfuN7v/IyLe2iFgNmXiYU= 2023-10-18T19:54:17.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tOk9X5IfuN7v/IyLe2iFgNmXiYU=', 'Content-Length': '0'} 2023-10-18T19:54:17.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003deef4cb2e5973b-00653037e9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.122 DEBUG:boto:b'bofrnf-9myobj1000falsemyobj2023-10-18T19:54:01.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.124 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:17.124 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:17.125 DEBUG:boto:path=/bofrnf-9/?prefix=myobj 2023-10-18T19:54:17.126 DEBUG:boto:auth_path=/bofrnf-9/?prefix=myobj 2023-10-18T19:54:17.126 DEBUG:boto:Method: GET 2023-10-18T19:54:17.127 DEBUG:boto:Path: /bofrnf-9/?prefix=myobj 2023-10-18T19:54:17.128 DEBUG:boto:Data: 2023-10-18T19:54:17.128 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.129 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.130 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.131 DEBUG:boto:Params: {} 2023-10-18T19:54:17.132 DEBUG:boto:Token: None 2023-10-18T19:54:17.132 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/ 2023-10-18T19:54:17.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tOk9X5IfuN7v/IyLe2iFgNmXiYU= 2023-10-18T19:54:17.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tOk9X5IfuN7v/IyLe2iFgNmXiYU=', 'Content-Length': '0'} 2023-10-18T19:54:17.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1cd18fe19169b1c-00653037e9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.138 DEBUG:boto:b'bofrnf-9myobj1000falsemyobj2023-10-18T19:54:01.480Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.140 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-9 zones={a2, a1} 2023-10-18T19:54:17.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:17.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:17.142 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:54:17.207 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:54:17.208 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:54:17.246 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:17.241+0000 7fc95675f780 1 bucket sync caught up with source: 2023-10-18T19:54:17.246 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-10-18T19:54:17.247 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.14.6, , , , , , , , , ] 2023-10-18T19:54:17.247 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:17.241+0000 7fc95675f780 0 bucket checkpoint complete 2023-10-18T19:54:17.257 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-10 zones={a1, a2} 2023-10-18T19:54:17.258 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.258 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.259 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.259 DEBUG:boto:Path: /bofrnf-10/ 2023-10-18T19:54:17.259 DEBUG:boto:Data: 2023-10-18T19:54:17.260 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.260 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.261 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.261 DEBUG:boto:Params: {} 2023-10-18T19:54:17.262 DEBUG:boto:Token: None 2023-10-18T19:54:17.262 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/ 2023-10-18T19:54:17.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EkgFU4GviBp4F16ciKF1I883/uU= 2023-10-18T19:54:17.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EkgFU4GviBp4F16ciKF1I883/uU=', 'Content-Length': '0'} 2023-10-18T19:54:17.266 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000712140d237c9eefc-00653037e9-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.266 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.267 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.267 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.267 DEBUG:boto:Path: /bofrnf-10/ 2023-10-18T19:54:17.268 DEBUG:boto:Data: 2023-10-18T19:54:17.268 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.268 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.269 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.269 DEBUG:boto:Params: {} 2023-10-18T19:54:17.270 DEBUG:boto:Token: None 2023-10-18T19:54:17.270 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/ 2023-10-18T19:54:17.271 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EkgFU4GviBp4F16ciKF1I883/uU= 2023-10-18T19:54:17.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EkgFU4GviBp4F16ciKF1I883/uU=', 'Content-Length': '0'} 2023-10-18T19:54:17.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d84bc8fe4dc5c36b-00653037e9-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.274 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:54:17.274 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.275 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.275 DEBUG:boto:path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.275 DEBUG:boto:auth_path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.276 DEBUG:boto:Method: GET 2023-10-18T19:54:17.276 DEBUG:boto:Path: /bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.277 DEBUG:boto:Data: 2023-10-18T19:54:17.277 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.277 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.278 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.278 DEBUG:boto:Params: {} 2023-10-18T19:54:17.279 DEBUG:boto:Token: None 2023-10-18T19:54:17.279 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/?versions 2023-10-18T19:54:17.280 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NFpk2dKC4oy93DmU7bs732fQvHU= 2023-10-18T19:54:17.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NFpk2dKC4oy93DmU7bs732fQvHU=', 'Content-Length': '0'} 2023-10-18T19:54:17.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033faf55ef21b4313-00653037e9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.283 DEBUG:boto:b'bofrnf-10999falsemyobjnulltrue2023-10-18T19:54:01.499Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.284 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-18T19:54:17.284 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:54:17.284 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.285 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.285 DEBUG:boto:path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.286 DEBUG:boto:auth_path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.286 DEBUG:boto:Method: GET 2023-10-18T19:54:17.287 DEBUG:boto:Path: /bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.287 DEBUG:boto:Data: 2023-10-18T19:54:17.287 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.288 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.289 DEBUG:boto:Params: {} 2023-10-18T19:54:17.289 DEBUG:boto:Token: None 2023-10-18T19:54:17.289 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/?versions 2023-10-18T19:54:17.290 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NFpk2dKC4oy93DmU7bs732fQvHU= 2023-10-18T19:54:17.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NFpk2dKC4oy93DmU7bs732fQvHU=', 'Content-Length': '0'} 2023-10-18T19:54:17.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc2d5b003ae47ce-00653037e9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.293 DEBUG:boto:b'bofrnf-10999falsemyobjnulltrue2023-10-18T19:54:01.499Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.294 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-18T19:54:17.294 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.295 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.295 DEBUG:boto:path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.295 DEBUG:boto:auth_path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.296 DEBUG:boto:Method: GET 2023-10-18T19:54:17.296 DEBUG:boto:Path: /bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.297 DEBUG:boto:Data: 2023-10-18T19:54:17.297 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.297 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.298 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.298 DEBUG:boto:Params: {} 2023-10-18T19:54:17.299 DEBUG:boto:Token: None 2023-10-18T19:54:17.299 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/?versions 2023-10-18T19:54:17.300 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NFpk2dKC4oy93DmU7bs732fQvHU= 2023-10-18T19:54:17.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NFpk2dKC4oy93DmU7bs732fQvHU=', 'Content-Length': '0'} 2023-10-18T19:54:17.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4d2738b363576f-00653037e9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.346 DEBUG:boto:b'bofrnf-10999falsemyobjnulltrue2023-10-18T19:54:01.499Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.347 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.347 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.348 DEBUG:boto:path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.348 DEBUG:boto:auth_path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.349 DEBUG:boto:Method: GET 2023-10-18T19:54:17.349 DEBUG:boto:Path: /bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:17.349 DEBUG:boto:Data: 2023-10-18T19:54:17.350 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.350 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.350 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.351 DEBUG:boto:Params: {} 2023-10-18T19:54:17.351 DEBUG:boto:Token: None 2023-10-18T19:54:17.352 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/?versions 2023-10-18T19:54:17.352 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NFpk2dKC4oy93DmU7bs732fQvHU= 2023-10-18T19:54:17.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NFpk2dKC4oy93DmU7bs732fQvHU=', 'Content-Length': '0'} 2023-10-18T19:54:17.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb10714dfcd85e1-00653037e9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.356 DEBUG:boto:b'bofrnf-10999falsemyobjnulltrue2023-10-18T19:54:01.499Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:17.357 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.357 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.358 DEBUG:boto:path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.358 DEBUG:boto:auth_path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.359 DEBUG:boto:Method: GET 2023-10-18T19:54:17.359 DEBUG:boto:Path: /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.359 DEBUG:boto:Data: 2023-10-18T19:54:17.360 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.360 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.361 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.361 DEBUG:boto:Params: {} 2023-10-18T19:54:17.361 DEBUG:boto:Token: None 2023-10-18T19:54:17.362 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.362 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xu/IZsNfaapmF26SFzp6dw68P1s= 2023-10-18T19:54:17.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xu/IZsNfaapmF26SFzp6dw68P1s=', 'Content-Length': '0'} 2023-10-18T19:54:17.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7049f6264e8d69a-00653037e9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.366 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.366 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.367 DEBUG:boto:path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.367 DEBUG:boto:auth_path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.368 DEBUG:boto:Method: GET 2023-10-18T19:54:17.368 DEBUG:boto:Path: /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.368 DEBUG:boto:Data: 2023-10-18T19:54:17.369 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.369 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.369 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.370 DEBUG:boto:Params: {} 2023-10-18T19:54:17.370 DEBUG:boto:Token: None 2023-10-18T19:54:17.371 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.371 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xu/IZsNfaapmF26SFzp6dw68P1s= 2023-10-18T19:54:17.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xu/IZsNfaapmF26SFzp6dw68P1s=', 'Content-Length': '0'} 2023-10-18T19:54:17.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6fdbd983d61a711-00653037e9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.374 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.375 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.375 DEBUG:boto:path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.375 DEBUG:boto:auth_path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.376 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.376 DEBUG:boto:Path: /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.376 DEBUG:boto:Data: 2023-10-18T19:54:17.377 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.378 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.378 DEBUG:boto:Params: {} 2023-10-18T19:54:17.378 DEBUG:boto:Token: None 2023-10-18T19:54:17.379 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.379 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:00zfRro7oLssqNb2yBb7U60llTE= 2023-10-18T19:54:17.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:00zfRro7oLssqNb2yBb7U60llTE=', 'Content-Length': '0'} 2023-10-18T19:54:17.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000334d85a37fcc03ae-00653037e9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.382 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.382 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.383 DEBUG:boto:path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.383 DEBUG:boto:auth_path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.383 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.384 DEBUG:boto:Path: /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.384 DEBUG:boto:Data: 2023-10-18T19:54:17.384 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.385 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.385 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.386 DEBUG:boto:Params: {} 2023-10-18T19:54:17.386 DEBUG:boto:Token: None 2023-10-18T19:54:17.386 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.387 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:00zfRro7oLssqNb2yBb7U60llTE= 2023-10-18T19:54:17.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:00zfRro7oLssqNb2yBb7U60llTE=', 'Content-Length': '0'} 2023-10-18T19:54:17.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b5a2841eaa9b65f-00653037e9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:17.390 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.390 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.391 DEBUG:boto:path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.391 DEBUG:boto:auth_path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.391 DEBUG:boto:Method: GET 2023-10-18T19:54:17.392 DEBUG:boto:Path: /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.392 DEBUG:boto:Data: 2023-10-18T19:54:17.393 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.393 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.393 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.394 DEBUG:boto:Params: {} 2023-10-18T19:54:17.394 DEBUG:boto:Token: None 2023-10-18T19:54:17.395 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.395 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xu/IZsNfaapmF26SFzp6dw68P1s= 2023-10-18T19:54:17.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xu/IZsNfaapmF26SFzp6dw68P1s=', 'Content-Length': '0'} 2023-10-18T19:54:17.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000065a01c3e4ebd52c-00653037e9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.398 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.398 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.399 DEBUG:boto:path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.399 DEBUG:boto:auth_path=/bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.399 DEBUG:boto:Method: GET 2023-10-18T19:54:17.400 DEBUG:boto:Path: /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.400 DEBUG:boto:Data: 2023-10-18T19:54:17.401 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.401 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.402 DEBUG:boto:Params: {} 2023-10-18T19:54:17.402 DEBUG:boto:Token: None 2023-10-18T19:54:17.403 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj?versionId=null 2023-10-18T19:54:17.403 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xu/IZsNfaapmF26SFzp6dw68P1s= 2023-10-18T19:54:17.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xu/IZsNfaapmF26SFzp6dw68P1s=', 'Content-Length': '0'} 2023-10-18T19:54:17.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032a3e0f57f7d5eb4-00653037e9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.406 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.407 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.407 DEBUG:boto:path=/bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.407 DEBUG:boto:auth_path=/bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.408 DEBUG:boto:Method: GET 2023-10-18T19:54:17.408 DEBUG:boto:Path: /bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.409 DEBUG:boto:Data: 2023-10-18T19:54:17.409 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.410 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.410 DEBUG:boto:Params: {} 2023-10-18T19:54:17.411 DEBUG:boto:Token: None 2023-10-18T19:54:17.411 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/ 2023-10-18T19:54:17.412 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dk39Q6GJ92rFBsr/8adIHdpaRNE= 2023-10-18T19:54:17.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dk39Q6GJ92rFBsr/8adIHdpaRNE=', 'Content-Length': '0'} 2023-10-18T19:54:17.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b84cd5e0991768-00653037e9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.415 DEBUG:boto:b'bofrnf-10myobj1000falsemyobj2023-10-18T19:54:01.499Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.415 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.416 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.416 DEBUG:boto:path=/bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.417 DEBUG:boto:auth_path=/bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.417 DEBUG:boto:Method: GET 2023-10-18T19:54:17.417 DEBUG:boto:Path: /bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.418 DEBUG:boto:Data: 2023-10-18T19:54:17.418 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.419 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.419 DEBUG:boto:Params: {} 2023-10-18T19:54:17.420 DEBUG:boto:Token: None 2023-10-18T19:54:17.420 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/ 2023-10-18T19:54:17.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dk39Q6GJ92rFBsr/8adIHdpaRNE= 2023-10-18T19:54:17.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dk39Q6GJ92rFBsr/8adIHdpaRNE=', 'Content-Length': '0'} 2023-10-18T19:54:17.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060cdba3fc1c80064-00653037e9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.424 DEBUG:boto:b'bofrnf-10myobj1000falsemyobj2023-10-18T19:54:01.499Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.424 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.424 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.425 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.425 DEBUG:boto:Path: /bofrnf-10/myobj 2023-10-18T19:54:17.426 DEBUG:boto:Data: 2023-10-18T19:54:17.426 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.427 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.427 DEBUG:boto:Params: {} 2023-10-18T19:54:17.428 DEBUG:boto:Token: None 2023-10-18T19:54:17.428 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj 2023-10-18T19:54:17.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tPSYXFvrQoLLDqO2hcFl4l+dQas= 2023-10-18T19:54:17.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tPSYXFvrQoLLDqO2hcFl4l+dQas=', 'Content-Length': '0'} 2023-10-18T19:54:17.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001402e64344ab7030-00653037e9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.431 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.431 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.432 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.432 DEBUG:boto:Path: /bofrnf-10/myobj 2023-10-18T19:54:17.433 DEBUG:boto:Data: 2023-10-18T19:54:17.433 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.434 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.434 DEBUG:boto:Params: {} 2023-10-18T19:54:17.435 DEBUG:boto:Token: None 2023-10-18T19:54:17.435 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj 2023-10-18T19:54:17.436 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tPSYXFvrQoLLDqO2hcFl4l+dQas= 2023-10-18T19:54:17.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tPSYXFvrQoLLDqO2hcFl4l+dQas=', 'Content-Length': '0'} 2023-10-18T19:54:17.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000311182e025092a60-00653037e9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-18T19:54:17.439 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.440 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.440 DEBUG:boto:Method: GET 2023-10-18T19:54:17.440 DEBUG:boto:Path: /bofrnf-10/myobj 2023-10-18T19:54:17.441 DEBUG:boto:Data: 2023-10-18T19:54:17.441 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.442 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.443 DEBUG:boto:Params: {} 2023-10-18T19:54:17.443 DEBUG:boto:Token: None 2023-10-18T19:54:17.443 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj 2023-10-18T19:54:17.444 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bG3xJSYNXsw4A3Vb+UJV1SbebAE= 2023-10-18T19:54:17.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bG3xJSYNXsw4A3Vb+UJV1SbebAE=', 'Content-Length': '0'} 2023-10-18T19:54:17.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4a92f8fcc423cb8-00653037e9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.490 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:17.491 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:17.491 DEBUG:boto:Method: GET 2023-10-18T19:54:17.492 DEBUG:boto:Path: /bofrnf-10/myobj 2023-10-18T19:54:17.492 DEBUG:boto:Data: 2023-10-18T19:54:17.492 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.493 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.493 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.494 DEBUG:boto:Params: {} 2023-10-18T19:54:17.494 DEBUG:boto:Token: None 2023-10-18T19:54:17.495 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/myobj 2023-10-18T19:54:17.495 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bG3xJSYNXsw4A3Vb+UJV1SbebAE= 2023-10-18T19:54:17.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bG3xJSYNXsw4A3Vb+UJV1SbebAE=', 'Content-Length': '0'} 2023-10-18T19:54:17.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d3cb72f054a6039-00653037e9-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.498 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.499 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.499 DEBUG:boto:path=/bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.500 DEBUG:boto:auth_path=/bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.500 DEBUG:boto:Method: GET 2023-10-18T19:54:17.501 DEBUG:boto:Path: /bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.501 DEBUG:boto:Data: 2023-10-18T19:54:17.501 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.502 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.502 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.503 DEBUG:boto:Params: {} 2023-10-18T19:54:17.503 DEBUG:boto:Token: None 2023-10-18T19:54:17.504 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/ 2023-10-18T19:54:17.504 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dk39Q6GJ92rFBsr/8adIHdpaRNE= 2023-10-18T19:54:17.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dk39Q6GJ92rFBsr/8adIHdpaRNE=', 'Content-Length': '0'} 2023-10-18T19:54:17.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c21de477e89088a4-00653037e9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.508 DEBUG:boto:b'bofrnf-10myobj1000falsemyobj2023-10-18T19:54:01.499Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.508 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:17.509 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:17.509 DEBUG:boto:path=/bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.510 DEBUG:boto:auth_path=/bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.510 DEBUG:boto:Method: GET 2023-10-18T19:54:17.511 DEBUG:boto:Path: /bofrnf-10/?prefix=myobj 2023-10-18T19:54:17.511 DEBUG:boto:Data: 2023-10-18T19:54:17.512 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.512 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:17.513 DEBUG:boto:Port: 8001 2023-10-18T19:54:17.513 DEBUG:boto:Params: {} 2023-10-18T19:54:17.513 DEBUG:boto:Token: None 2023-10-18T19:54:17.514 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-10/ 2023-10-18T19:54:17.514 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dk39Q6GJ92rFBsr/8adIHdpaRNE= 2023-10-18T19:54:17.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dk39Q6GJ92rFBsr/8adIHdpaRNE=', 'Content-Length': '0'} 2023-10-18T19:54:17.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005266dd997522f5fd-00653037e9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.518 DEBUG:boto:b'bofrnf-10myobj1000falsemyobj2023-10-18T19:54:01.499Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:54:17.518 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-10 zones={a1, a2} 2023-10-18T19:54:17.519 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:17.519 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:17.519 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.520 DEBUG:boto:Path: /bofrnf-9/ 2023-10-18T19:54:17.520 DEBUG:boto:Data: 2023-10-18T19:54:17.520 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.521 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.521 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.521 DEBUG:boto:Params: {} 2023-10-18T19:54:17.522 DEBUG:boto:Token: None 2023-10-18T19:54:17.522 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/ 2023-10-18T19:54:17.523 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kHSthmdIJDD+Jh1wWFNEtU173tg= 2023-10-18T19:54:17.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kHSthmdIJDD+Jh1wWFNEtU173tg=', 'Content-Length': '0'} 2023-10-18T19:54:17.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073b8e7e3121def8f-00653037e9-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.526 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:17.526 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:17.527 DEBUG:boto:Method: HEAD 2023-10-18T19:54:17.527 DEBUG:boto:Path: /bofrnf-9/myobj 2023-10-18T19:54:17.527 DEBUG:boto:Data: 2023-10-18T19:54:17.528 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.528 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.529 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.529 DEBUG:boto:Params: {} 2023-10-18T19:54:17.530 DEBUG:boto:Token: None 2023-10-18T19:54:17.530 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/myobj 2023-10-18T19:54:17.531 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XMSLk2QUvkoz6ZX1ilxk3CQVCVU= 2023-10-18T19:54:17.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XMSLk2QUvkoz6ZX1ilxk3CQVCVU=', 'Content-Length': '0'} 2023-10-18T19:54:17.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005315214a54261ada-00653037e9-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.533 DEBUG:boto:path=/bofrnf-9/myobj 2023-10-18T19:54:17.534 DEBUG:boto:auth_path=/bofrnf-9/myobj 2023-10-18T19:54:17.534 DEBUG:boto:Method: DELETE 2023-10-18T19:54:17.535 DEBUG:boto:Path: /bofrnf-9/myobj 2023-10-18T19:54:17.535 DEBUG:boto:Data: 2023-10-18T19:54:17.535 DEBUG:boto:Headers: {} 2023-10-18T19:54:17.536 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:17.536 DEBUG:boto:Port: 8000 2023-10-18T19:54:17.536 DEBUG:boto:Params: {} 2023-10-18T19:54:17.537 DEBUG:boto:Token: None 2023-10-18T19:54:17.537 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:54:17 GMT /bofrnf-9/myobj 2023-10-18T19:54:17.538 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lRZ+WRvmozDUIbCnhIBGokZzC9M= 2023-10-18T19:54:17.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lRZ+WRvmozDUIbCnhIBGokZzC9M=', 'Content-Length': '0'} 2023-10-18T19:54:17.542 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002699f7a9723dfe63-00653037e9-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:54:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:17.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:17.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:17.544 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:54:17.600 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:54:17.601 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:54:17.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:17.633+0000 7f0694e9f780 1 waiting for incremental sync to catch up: 2023-10-18T19:54:17.636 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-10-18T19:54:17.636 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.9.6, , , , , , , , , ] 2023-10-18T19:54:22.637 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:22.635+0000 7f0694e9f780 1 waiting for incremental sync to catch up: 2023-10-18T19:54:22.638 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-10-18T19:54:22.638 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.9.6, , , , , , , , , ] 2023-10-18T19:54:27.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:27.636+0000 7f0694e9f780 1 waiting for incremental sync to catch up: 2023-10-18T19:54:27.639 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-10-18T19:54:27.639 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.9.6, , , , , , , , , ] 2023-10-18T19:54:32.640 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:32.638+0000 7f0694e9f780 1 waiting for incremental sync to catch up: 2023-10-18T19:54:32.640 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-10-18T19:54:32.640 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.9.6, , , , , , , , , ] 2023-10-18T19:54:37.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:37.639+0000 7f0694e9f780 1 bucket sync caught up with source: 2023-10-18T19:54:37.642 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.9.6, , , , , , , , , ] 2023-10-18T19:54:37.642 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.9.6, , , , , , , , , ] 2023-10-18T19:54:37.642 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:37.639+0000 7f0694e9f780 0 bucket checkpoint complete 2023-10-18T19:54:37.657 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-9 zones={a2, a1} 2023-10-18T19:54:37.657 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:37.658 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:37.658 DEBUG:boto:Method: HEAD 2023-10-18T19:54:37.659 DEBUG:boto:Path: /bofrnf-9/ 2023-10-18T19:54:37.659 DEBUG:boto:Data: 2023-10-18T19:54:37.659 DEBUG:boto:Headers: {} 2023-10-18T19:54:37.660 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:37.660 DEBUG:boto:Port: 8001 2023-10-18T19:54:37.661 DEBUG:boto:Params: {} 2023-10-18T19:54:37.661 DEBUG:boto:Token: None 2023-10-18T19:54:37.662 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:37 GMT /bofrnf-9/ 2023-10-18T19:54:37.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jZGx82YwC2XQLMiH/0zG8eVj7Ko= 2023-10-18T19:54:37.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jZGx82YwC2XQLMiH/0zG8eVj7Ko=', 'Content-Length': '0'} 2023-10-18T19:54:37.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000711fc27ff3b77547-00653037fd-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:37.666 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:37.666 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:37.667 DEBUG:boto:Method: HEAD 2023-10-18T19:54:37.667 DEBUG:boto:Path: /bofrnf-9/ 2023-10-18T19:54:37.668 DEBUG:boto:Data: 2023-10-18T19:54:37.668 DEBUG:boto:Headers: {} 2023-10-18T19:54:37.668 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:37.669 DEBUG:boto:Port: 8000 2023-10-18T19:54:37.669 DEBUG:boto:Params: {} 2023-10-18T19:54:37.670 DEBUG:boto:Token: None 2023-10-18T19:54:37.670 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:37 GMT /bofrnf-9/ 2023-10-18T19:54:37.670 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jZGx82YwC2XQLMiH/0zG8eVj7Ko= 2023-10-18T19:54:37.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jZGx82YwC2XQLMiH/0zG8eVj7Ko=', 'Content-Length': '0'} 2023-10-18T19:54:37.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec99265042e3c7f2-00653037fd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:37.674 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:54:37.674 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:37.675 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:37.675 DEBUG:boto:path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.676 DEBUG:boto:auth_path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.676 DEBUG:boto:Method: GET 2023-10-18T19:54:37.676 DEBUG:boto:Path: /bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.677 DEBUG:boto:Data: 2023-10-18T19:54:37.677 DEBUG:boto:Headers: {} 2023-10-18T19:54:37.679 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:37.679 DEBUG:boto:Port: 8001 2023-10-18T19:54:37.680 DEBUG:boto:Params: {} 2023-10-18T19:54:37.680 DEBUG:boto:Token: None 2023-10-18T19:54:37.680 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:37 GMT /bofrnf-9/?versions 2023-10-18T19:54:37.681 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F9kSKqvMMEJH6nkJ2C1mSnJWjK0= 2023-10-18T19:54:37.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F9kSKqvMMEJH6nkJ2C1mSnJWjK0=', 'Content-Length': '0'} 2023-10-18T19:54:37.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059dcabad65a4cb6c-00653037fd-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:37.684 DEBUG:boto:b'bofrnf-9999false' 2023-10-18T19:54:37.685 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:54:37.686 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:37.686 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:37.687 DEBUG:boto:path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.687 DEBUG:boto:auth_path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.688 DEBUG:boto:Method: GET 2023-10-18T19:54:37.688 DEBUG:boto:Path: /bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.688 DEBUG:boto:Data: 2023-10-18T19:54:37.689 DEBUG:boto:Headers: {} 2023-10-18T19:54:37.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:37.690 DEBUG:boto:Port: 8000 2023-10-18T19:54:37.693 DEBUG:boto:Params: {} 2023-10-18T19:54:37.694 DEBUG:boto:Token: None 2023-10-18T19:54:37.695 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:37 GMT /bofrnf-9/?versions 2023-10-18T19:54:37.695 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F9kSKqvMMEJH6nkJ2C1mSnJWjK0= 2023-10-18T19:54:37.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F9kSKqvMMEJH6nkJ2C1mSnJWjK0=', 'Content-Length': '0'} 2023-10-18T19:54:37.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6dc5a7fde0515e1-00653037fd-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:37.699 DEBUG:boto:b'bofrnf-9999false' 2023-10-18T19:54:37.700 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:37.701 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:37.701 DEBUG:boto:path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.701 DEBUG:boto:auth_path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.702 DEBUG:boto:Method: GET 2023-10-18T19:54:37.702 DEBUG:boto:Path: /bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.703 DEBUG:boto:Data: 2023-10-18T19:54:37.703 DEBUG:boto:Headers: {} 2023-10-18T19:54:37.704 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:37.704 DEBUG:boto:Port: 8001 2023-10-18T19:54:37.704 DEBUG:boto:Params: {} 2023-10-18T19:54:37.705 DEBUG:boto:Token: None 2023-10-18T19:54:37.705 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:37 GMT /bofrnf-9/?versions 2023-10-18T19:54:37.706 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F9kSKqvMMEJH6nkJ2C1mSnJWjK0= 2023-10-18T19:54:37.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F9kSKqvMMEJH6nkJ2C1mSnJWjK0=', 'Content-Length': '0'} 2023-10-18T19:54:37.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f61a11107a6f296-00653037fd-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:37.709 DEBUG:boto:b'bofrnf-9999false' 2023-10-18T19:54:37.710 DEBUG:boto:path=/bofrnf-9/ 2023-10-18T19:54:37.710 DEBUG:boto:auth_path=/bofrnf-9/ 2023-10-18T19:54:37.711 DEBUG:boto:path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.711 DEBUG:boto:auth_path=/bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.712 DEBUG:boto:Method: GET 2023-10-18T19:54:37.712 DEBUG:boto:Path: /bofrnf-9/?versions&max-keys=999 2023-10-18T19:54:37.712 DEBUG:boto:Data: 2023-10-18T19:54:37.713 DEBUG:boto:Headers: {} 2023-10-18T19:54:37.713 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:37.714 DEBUG:boto:Port: 8000 2023-10-18T19:54:37.714 DEBUG:boto:Params: {} 2023-10-18T19:54:37.714 DEBUG:boto:Token: None 2023-10-18T19:54:37.715 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:37 GMT /bofrnf-9/?versions 2023-10-18T19:54:37.715 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F9kSKqvMMEJH6nkJ2C1mSnJWjK0= 2023-10-18T19:54:37.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F9kSKqvMMEJH6nkJ2C1mSnJWjK0=', 'Content-Length': '0'} 2023-10-18T19:54:37.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a863c8653c66561b-00653037fd-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:37.762 DEBUG:boto:b'bofrnf-9999false' 2023-10-18T19:54:37.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-9 zones={a2, a1} 2023-10-18T19:54:37.763 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:37.764 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:37.764 DEBUG:boto:Method: HEAD 2023-10-18T19:54:37.765 DEBUG:boto:Path: /bofrnf-10/ 2023-10-18T19:54:37.765 DEBUG:boto:Data: 2023-10-18T19:54:37.765 DEBUG:boto:Headers: {} 2023-10-18T19:54:37.766 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:37.766 DEBUG:boto:Port: 8001 2023-10-18T19:54:37.767 DEBUG:boto:Params: {} 2023-10-18T19:54:37.767 DEBUG:boto:Token: None 2023-10-18T19:54:37.767 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:37 GMT /bofrnf-10/ 2023-10-18T19:54:37.768 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gC5tbz99fxbFwrh/hlaAUHq9KUU= 2023-10-18T19:54:37.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gC5tbz99fxbFwrh/hlaAUHq9KUU=', 'Content-Length': '0'} 2023-10-18T19:54:37.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c795fe9772ae6e9-00653037fd-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:37.771 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:37.772 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:37.772 DEBUG:boto:Method: HEAD 2023-10-18T19:54:37.772 DEBUG:boto:Path: /bofrnf-10/myobj 2023-10-18T19:54:37.773 DEBUG:boto:Data: 2023-10-18T19:54:37.773 DEBUG:boto:Headers: {} 2023-10-18T19:54:37.773 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:37.774 DEBUG:boto:Port: 8001 2023-10-18T19:54:37.774 DEBUG:boto:Params: {} 2023-10-18T19:54:37.775 DEBUG:boto:Token: None 2023-10-18T19:54:37.775 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:37 GMT /bofrnf-10/myobj 2023-10-18T19:54:37.776 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QhAcRdkewao1DgMuJ8POGeSKG7Y= 2023-10-18T19:54:37.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QhAcRdkewao1DgMuJ8POGeSKG7Y=', 'Content-Length': '0'} 2023-10-18T19:54:37.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:54:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fc0763b8a24feb1-00653037fd-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:37.779 DEBUG:boto:path=/bofrnf-10/myobj 2023-10-18T19:54:37.779 DEBUG:boto:auth_path=/bofrnf-10/myobj 2023-10-18T19:54:37.779 DEBUG:boto:Method: DELETE 2023-10-18T19:54:37.780 DEBUG:boto:Path: /bofrnf-10/myobj 2023-10-18T19:54:37.780 DEBUG:boto:Data: 2023-10-18T19:54:37.780 DEBUG:boto:Headers: {} 2023-10-18T19:54:37.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:37.781 DEBUG:boto:Port: 8001 2023-10-18T19:54:37.782 DEBUG:boto:Params: {} 2023-10-18T19:54:37.782 DEBUG:boto:Token: None 2023-10-18T19:54:37.782 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:54:37 GMT /bofrnf-10/myobj 2023-10-18T19:54:37.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U2or5Oi06HxJwYD5NPvHfWUSmaI= 2023-10-18T19:54:37.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U2or5Oi06HxJwYD5NPvHfWUSmaI=', 'Content-Length': '0'} 2023-10-18T19:54:37.790 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000931114e340a2d6a8-00653037fd-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:37.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:37.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:54:37.791 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:54:37.854 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:54:37.855 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:54:37.891 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:37.889+0000 7f8c21700780 1 waiting for incremental sync to catch up: 2023-10-18T19:54:37.891 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-10-18T19:54:37.891 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-18T19:54:42.892 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:42.890+0000 7f8c21700780 1 waiting for incremental sync to catch up: 2023-10-18T19:54:42.892 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-10-18T19:54:42.893 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-18T19:54:47.893 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:47.892+0000 7f8c21700780 1 waiting for incremental sync to catch up: 2023-10-18T19:54:47.894 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-10-18T19:54:47.894 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-18T19:54:52.894 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:52.893+0000 7f8c21700780 1 waiting for incremental sync to catch up: 2023-10-18T19:54:52.895 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-10-18T19:54:52.895 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-18T19:54:57.896 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:57.895+0000 7f8c21700780 1 bucket sync caught up with source: 2023-10-18T19:54:57.897 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.18.6, , , , , , , , , ] 2023-10-18T19:54:57.897 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-18T19:54:57.897 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:57.895+0000 7f8c21700780 0 bucket checkpoint complete 2023-10-18T19:54:57.915 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-10 zones={a1, a2} 2023-10-18T19:54:57.916 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:57.916 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:57.917 DEBUG:boto:Method: HEAD 2023-10-18T19:54:57.917 DEBUG:boto:Path: /bofrnf-10/ 2023-10-18T19:54:57.918 DEBUG:boto:Data: 2023-10-18T19:54:57.918 DEBUG:boto:Headers: {} 2023-10-18T19:54:57.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:57.919 DEBUG:boto:Port: 8000 2023-10-18T19:54:57.919 DEBUG:boto:Params: {} 2023-10-18T19:54:57.920 DEBUG:boto:Token: None 2023-10-18T19:54:57.920 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:57 GMT /bofrnf-10/ 2023-10-18T19:54:57.921 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WdpR/Q53oAfDpIBDUB+Kc/FahGA= 2023-10-18T19:54:57.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WdpR/Q53oAfDpIBDUB+Kc/FahGA=', 'Content-Length': '0'} 2023-10-18T19:54:57.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acd3eef5e88db393-0065303811-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:57.925 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:57.925 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:57.926 DEBUG:boto:Method: HEAD 2023-10-18T19:54:57.926 DEBUG:boto:Path: /bofrnf-10/ 2023-10-18T19:54:57.926 DEBUG:boto:Data: 2023-10-18T19:54:57.927 DEBUG:boto:Headers: {} 2023-10-18T19:54:57.927 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:57.928 DEBUG:boto:Port: 8001 2023-10-18T19:54:57.928 DEBUG:boto:Params: {} 2023-10-18T19:54:57.929 DEBUG:boto:Token: None 2023-10-18T19:54:57.929 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:54:57 GMT /bofrnf-10/ 2023-10-18T19:54:57.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WdpR/Q53oAfDpIBDUB+Kc/FahGA= 2023-10-18T19:54:57.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WdpR/Q53oAfDpIBDUB+Kc/FahGA=', 'Content-Length': '0'} 2023-10-18T19:54:57.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000879dce37f4269ad6-0065303811-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:57.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:54:57.933 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:57.934 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:57.934 DEBUG:boto:path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.935 DEBUG:boto:auth_path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.935 DEBUG:boto:Method: GET 2023-10-18T19:54:57.935 DEBUG:boto:Path: /bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.936 DEBUG:boto:Data: 2023-10-18T19:54:57.936 DEBUG:boto:Headers: {} 2023-10-18T19:54:57.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:57.937 DEBUG:boto:Port: 8000 2023-10-18T19:54:57.937 DEBUG:boto:Params: {} 2023-10-18T19:54:57.938 DEBUG:boto:Token: None 2023-10-18T19:54:57.939 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:57 GMT /bofrnf-10/?versions 2023-10-18T19:54:57.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:piAqH/a8cfLD1mIdkWlgZci8Od8= 2023-10-18T19:54:57.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:piAqH/a8cfLD1mIdkWlgZci8Od8=', 'Content-Length': '0'} 2023-10-18T19:54:57.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e85975cc71277314-0065303811-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:57.943 DEBUG:boto:b'bofrnf-10999false' 2023-10-18T19:54:57.943 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:54:57.944 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:57.944 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:57.945 DEBUG:boto:path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.945 DEBUG:boto:auth_path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.946 DEBUG:boto:Method: GET 2023-10-18T19:54:57.946 DEBUG:boto:Path: /bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.946 DEBUG:boto:Data: 2023-10-18T19:54:57.947 DEBUG:boto:Headers: {} 2023-10-18T19:54:57.947 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:57.948 DEBUG:boto:Port: 8001 2023-10-18T19:54:57.948 DEBUG:boto:Params: {} 2023-10-18T19:54:57.949 DEBUG:boto:Token: None 2023-10-18T19:54:57.949 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:57 GMT /bofrnf-10/?versions 2023-10-18T19:54:57.950 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:piAqH/a8cfLD1mIdkWlgZci8Od8= 2023-10-18T19:54:57.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:piAqH/a8cfLD1mIdkWlgZci8Od8=', 'Content-Length': '0'} 2023-10-18T19:54:57.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e15b92ba809192b4-0065303811-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:57.953 DEBUG:boto:b'bofrnf-10999false' 2023-10-18T19:54:57.954 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:57.954 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:57.955 DEBUG:boto:path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.955 DEBUG:boto:auth_path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.955 DEBUG:boto:Method: GET 2023-10-18T19:54:57.956 DEBUG:boto:Path: /bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.956 DEBUG:boto:Data: 2023-10-18T19:54:57.957 DEBUG:boto:Headers: {} 2023-10-18T19:54:57.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:57.958 DEBUG:boto:Port: 8000 2023-10-18T19:54:57.958 DEBUG:boto:Params: {} 2023-10-18T19:54:57.959 DEBUG:boto:Token: None 2023-10-18T19:54:57.959 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:57 GMT /bofrnf-10/?versions 2023-10-18T19:54:57.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:piAqH/a8cfLD1mIdkWlgZci8Od8= 2023-10-18T19:54:57.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:piAqH/a8cfLD1mIdkWlgZci8Od8=', 'Content-Length': '0'} 2023-10-18T19:54:57.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef7ff037b31422c-0065303811-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:57.963 DEBUG:boto:b'bofrnf-10999false' 2023-10-18T19:54:57.964 DEBUG:boto:path=/bofrnf-10/ 2023-10-18T19:54:57.964 DEBUG:boto:auth_path=/bofrnf-10/ 2023-10-18T19:54:57.965 DEBUG:boto:path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.965 DEBUG:boto:auth_path=/bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.966 DEBUG:boto:Method: GET 2023-10-18T19:54:57.966 DEBUG:boto:Path: /bofrnf-10/?versions&max-keys=999 2023-10-18T19:54:57.966 DEBUG:boto:Data: 2023-10-18T19:54:57.967 DEBUG:boto:Headers: {} 2023-10-18T19:54:57.967 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:57.968 DEBUG:boto:Port: 8001 2023-10-18T19:54:57.968 DEBUG:boto:Params: {} 2023-10-18T19:54:57.969 DEBUG:boto:Token: None 2023-10-18T19:54:57.969 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:54:57 GMT /bofrnf-10/?versions 2023-10-18T19:54:57.970 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:piAqH/a8cfLD1mIdkWlgZci8Od8= 2023-10-18T19:54:57.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:piAqH/a8cfLD1mIdkWlgZci8Od8=', 'Content-Length': '0'} 2023-10-18T19:54:57.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f57a66aa980a6cb-0065303811-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:57.974 DEBUG:boto:b'bofrnf-10999false' 2023-10-18T19:54:57.974 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-10 zones={a1, a2} 2023-10-18T19:54:57.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-10-18T19:54:57.976 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-11 2023-10-18T19:54:57.976 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:54:57.977 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:54:57.977 DEBUG:boto:Method: PUT 2023-10-18T19:54:57.978 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:54:57.978 DEBUG:boto:Data: 2023-10-18T19:54:57.979 DEBUG:boto:Headers: {} 2023-10-18T19:54:57.979 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:57.979 DEBUG:boto:Port: 8000 2023-10-18T19:54:57.980 DEBUG:boto:Params: {} 2023-10-18T19:54:57.980 DEBUG:boto:Token: None 2023-10-18T19:54:57.981 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:54:57 GMT /bofrnf-11/ 2023-10-18T19:54:57.981 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TO7sFDD4wLn9XfiHtGa2YIWxMYI= 2023-10-18T19:54:57.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TO7sFDD4wLn9XfiHtGa2YIWxMYI=', 'Content-Length': '0'} 2023-10-18T19:54:58.001 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000070f766d0c1226788-0065303811-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:58.002 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-12 2023-10-18T19:54:58.002 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:54:58.002 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:54:58.003 DEBUG:boto:Method: PUT 2023-10-18T19:54:58.003 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:54:58.004 DEBUG:boto:Data: 2023-10-18T19:54:58.004 DEBUG:boto:Headers: {} 2023-10-18T19:54:58.004 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:58.005 DEBUG:boto:Port: 8001 2023-10-18T19:54:58.005 DEBUG:boto:Params: {} 2023-10-18T19:54:58.006 DEBUG:boto:Token: None 2023-10-18T19:54:58.006 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:54:58 GMT /bofrnf-12/ 2023-10-18T19:54:58.006 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pUQvD/Xqi44PjzSncrJd7U6ddbM= 2023-10-18T19:54:58.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pUQvD/Xqi44PjzSncrJd7U6ddbM=', 'Content-Length': '0'} 2023-10-18T19:54:58.041 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e0caf11054089756-0065303812-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:58.041 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:54:58.042 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:54:58.042 DEBUG:boto:path=/bofrnf-11/?versioning 2023-10-18T19:54:58.043 DEBUG:boto:auth_path=/bofrnf-11/?versioning 2023-10-18T19:54:58.043 DEBUG:boto:Method: PUT 2023-10-18T19:54:58.043 DEBUG:boto:Path: /bofrnf-11/?versioning 2023-10-18T19:54:58.044 DEBUG:boto:Data: Enabled Disabled 2023-10-18T19:54:58.044 DEBUG:boto:Headers: {} 2023-10-18T19:54:58.044 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:54:58.045 DEBUG:boto:Port: 8000 2023-10-18T19:54:58.045 DEBUG:boto:Params: {} 2023-10-18T19:54:58.046 DEBUG:boto:Token: None 2023-10-18T19:54:58.046 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:54:58 GMT /bofrnf-11/?versioning 2023-10-18T19:54:58.047 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0YDvGxTL7SIP5f5GQw31bWaqUtA= 2023-10-18T19:54:58.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0YDvGxTL7SIP5f5GQw31bWaqUtA=', 'Content-Length': '228'} 2023-10-18T19:54:58.052 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000821ab10ac451dd87-0065303812-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:58.053 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:54:58.053 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:54:58.054 DEBUG:boto:path=/bofrnf-12/?versioning 2023-10-18T19:54:58.054 DEBUG:boto:auth_path=/bofrnf-12/?versioning 2023-10-18T19:54:58.054 DEBUG:boto:Method: PUT 2023-10-18T19:54:58.055 DEBUG:boto:Path: /bofrnf-12/?versioning 2023-10-18T19:54:58.055 DEBUG:boto:Data: Enabled Disabled 2023-10-18T19:54:58.055 DEBUG:boto:Headers: {} 2023-10-18T19:54:58.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:54:58.056 DEBUG:boto:Port: 8001 2023-10-18T19:54:58.057 DEBUG:boto:Params: {} 2023-10-18T19:54:58.057 DEBUG:boto:Token: None 2023-10-18T19:54:58.057 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:54:58 GMT /bofrnf-12/?versioning 2023-10-18T19:54:58.058 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j0Zc3kpC0F+ekNjiTcSYTU1mPzY= 2023-10-18T19:54:58.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:54:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j0Zc3kpC0F+ekNjiTcSYTU1mPzY=', 'Content-Length': '228'} 2023-10-18T19:54:58.067 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000001b48a8d61167727-0065303812-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:54:58.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:54:58.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:54:58.069 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:54:58.120 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:54:58.120 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:54:58.151 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:54:58.164 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:54:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658898.062804_165.1", 2023-10-18T19:54:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:58.062804Z" 2023-10-18T19:54:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:54:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T19:54:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:54:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:54:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:54:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:54:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:58.168 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:54:58.169 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697658898.062804_165.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:54:58.169 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:54:58.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:54:58.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:54:58.173 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:54:58.225 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:54:58.225 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:54:58.254 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:58.252+0000 7f03b2e43780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:54:58.255 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:54:58.253+0000 7f03b2e43780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:54:58.266 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:54:58.267 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:54:58.267 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:54:58.267 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:54:58.267 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:54:58.268 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:54:58.268 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:58.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:58.268 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:54:58.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:58.269 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:54:58.269 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:54:58.269 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:54:58.269 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.455783_144.1", 2023-10-18T19:54:58.269 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:54:58.270 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:54:58.270 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:54:58.270 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.455783Z", 2023-10-18T19:54:58.270 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:58.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:58.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:58.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:58.271 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:54:58.271 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:54:58.272 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:54:58.272 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:54:58.272 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:54:58.272 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:54:58.272 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:54:58.273 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T19:54:58.273 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:58.273 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:58.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:58.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:58.274 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:54:58.274 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:54:58.274 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:54:58.274 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:54:58.275 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:54:58.275 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:54:58.275 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:54:58.276 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:54:58.276 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:58.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:58.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:58.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:54:58.277 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:54:58.277 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:54:58.277 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:54:58.277 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:54:58.277 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:54:58.278 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:54:58.278 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:54:58.278 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:54:58.278 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:54:58.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:58.279 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:54:58.279 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:54:58.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:58.279 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:54:58.279 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:54:58.280 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:54:58.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:54:58.280 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:54:58Z" 2023-10-18T19:54:58.280 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:54:58.281 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658841.455783_144.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:54:01.455783Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:54:58Z" } 2023-10-18T19:54:58.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658841.455783_144.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.455783Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:54:58.283 DEBUG:rgw_multi.tests:log_status={0: '1_1697658898.062804_165.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:54:58.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658841.455783_144.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:54:58.283 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697658898.062804_165.1 target=1_1697658841.455783_144.1 2023-10-18T19:55:03.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:55:03.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:55:03.286 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:55:03.344 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:55:03.344 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:55:03.374 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:03.373+0000 7f9dd3e08780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:55:03.375 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:03.374+0000 7f9dd3e08780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:55:03.387 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:55:03.388 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:55:03.388 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:55:03.388 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:55:03.388 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:55:03.389 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:55:03.389 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:55:03.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:55:03.389 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:55:03.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:55:03.390 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:55:03.390 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:55:03.390 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:55:03.390 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658898.062804_165.1", 2023-10-18T19:55:03.391 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:55:03.391 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:55:03.391 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:55:03.391 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:58.062804Z", 2023-10-18T19:55:03.392 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:55:03.392 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:55:03.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:55:03.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:55:03.393 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:55:03.393 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:55:03.393 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:55:03.393 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:55:03.393 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:55:03.394 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:55:03.394 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:55:03.394 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T19:55:03.394 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:55:03.395 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:55:03.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:55:03.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:55:03.395 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:55:03.396 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:55:03.396 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:55:03.396 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:55:03.396 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:55:03.396 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:55:03.397 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:55:03.397 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:55:03.397 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:55:03.397 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:55:03.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:55:03.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:55:03.398 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:55:03.398 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:55:03.398 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:55:03.399 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:55:03.399 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:55:03.399 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:55:03.399 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:55:03.400 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:55:03.400 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:55:03.400 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:55:03.400 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:55:03.401 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:55:03.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:55:03.401 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:55:03.401 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:55:03.401 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:55:03.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:55:03.402 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:55:03Z" 2023-10-18T19:55:03.402 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:55:03.403 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697658898.062804_165.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:54:58.062804Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:55:03Z" } 2023-10-18T19:55:03.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697658898.062804_165.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:54:58.062804Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:55:03.404 DEBUG:rgw_multi.tests:log_status={0: '1_1697658898.062804_165.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:55:03.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1697658898.062804_165.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:55:03.405 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:55:03.406 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:55:03.406 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:55:03.406 DEBUG:boto:Method: HEAD 2023-10-18T19:55:03.407 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:55:03.407 DEBUG:boto:Data: 2023-10-18T19:55:03.408 DEBUG:boto:Headers: {} 2023-10-18T19:55:03.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:03.409 DEBUG:boto:Port: 8000 2023-10-18T19:55:03.409 DEBUG:boto:Params: {} 2023-10-18T19:55:03.410 DEBUG:boto:Token: None 2023-10-18T19:55:03.410 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:55:03 GMT /bofrnf-11/ 2023-10-18T19:55:03.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OrutPAuPwxJ44E22CuhA56dOZxc= 2023-10-18T19:55:03.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OrutPAuPwxJ44E22CuhA56dOZxc=', 'Content-Length': '0'} 2023-10-18T19:55:03.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092176bbfb6ea853f-0065303817-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:03.414 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:55:03.415 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:55:03.415 DEBUG:boto:Method: PUT 2023-10-18T19:55:03.416 DEBUG:boto:Path: /bofrnf-11/dummy 2023-10-18T19:55:03.416 DEBUG:boto:Data: 2023-10-18T19:55:03.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T19:55:03.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:03.417 DEBUG:boto:Port: 8000 2023-10-18T19:55:03.418 DEBUG:boto:Params: {} 2023-10-18T19:55:03.418 DEBUG:boto:Token: None 2023-10-18T19:55:03.419 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 19:55:03 GMT /bofrnf-11/dummy 2023-10-18T19:55:03.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W7x1F5OWunuNMHZB8B7mg/Iixug= 2023-10-18T19:55:03.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W7x1F5OWunuNMHZB8B7mg/Iixug='} 2023-10-18T19:55:03.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-amz-request-id', 'tx0000048835a022d085b8d-0065303817-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:03.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:55:03.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:55:03.432 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:55:03.488 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:55:03.488 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:55:03.521 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:03.519+0000 7fc79a1ba780 1 waiting to reach incremental sync.. 2023-10-18T19:55:08.521 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:08.520+0000 7fc79a1ba780 1 waiting to reach incremental sync.. 2023-10-18T19:55:13.522 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:13.519+0000 7fc79a1ba780 1 waiting to reach incremental sync.. 2023-10-18T19:55:18.524 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:18.521+0000 7fc79a1ba780 1 bucket sync caught up with source: 2023-10-18T19:55:18.524 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.7.11, , , , , ] 2023-10-18T19:55:18.524 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.7.11, , , , , ] 2023-10-18T19:55:18.524 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:18.521+0000 7fc79a1ba780 0 bucket checkpoint complete 2023-10-18T19:55:18.539 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:55:18.539 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:55:18.540 DEBUG:boto:Method: HEAD 2023-10-18T19:55:18.540 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:55:18.540 DEBUG:boto:Data: 2023-10-18T19:55:18.541 DEBUG:boto:Headers: {} 2023-10-18T19:55:18.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:18.541 DEBUG:boto:Port: 8001 2023-10-18T19:55:18.542 DEBUG:boto:Params: {} 2023-10-18T19:55:18.542 DEBUG:boto:Token: None 2023-10-18T19:55:18.543 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:55:18 GMT /bofrnf-12/ 2023-10-18T19:55:18.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Fayp5btUdOyDNV0fr99gxsC4C+M= 2023-10-18T19:55:18.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Fayp5btUdOyDNV0fr99gxsC4C+M=', 'Content-Length': '0'} 2023-10-18T19:55:18.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018fbbb34899055f3-0065303826-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:18.547 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:55:18.547 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:55:18.547 DEBUG:boto:Method: PUT 2023-10-18T19:55:18.548 DEBUG:boto:Path: /bofrnf-12/dummy 2023-10-18T19:55:18.548 DEBUG:boto:Data: 2023-10-18T19:55:18.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T19:55:18.549 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:18.549 DEBUG:boto:Port: 8001 2023-10-18T19:55:18.550 DEBUG:boto:Params: {} 2023-10-18T19:55:18.550 DEBUG:boto:Token: None 2023-10-18T19:55:18.551 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 19:55:18 GMT /bofrnf-12/dummy 2023-10-18T19:55:18.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:py4h/DXPjJbSiH49fbSOGEtTgtc= 2023-10-18T19:55:18.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:py4h/DXPjJbSiH49fbSOGEtTgtc='} 2023-10-18T19:55:18.563 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-amz-request-id', 'tx000007220d7a24bd839c8-0065303826-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:18.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:55:18.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:55:18.564 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:55:18.625 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:55:18.625 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:55:18.661 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:18.659+0000 7fc693483780 1 waiting to reach incremental sync.. 2023-10-18T19:55:23.662 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:23.660+0000 7fc693483780 1 waiting to reach incremental sync.. 2023-10-18T19:55:28.662 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:28.661+0000 7fc693483780 1 waiting to reach incremental sync.. 2023-10-18T19:55:33.663 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:33.661+0000 7fc693483780 1 waiting to reach incremental sync.. 2023-10-18T19:55:38.665 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:38.663+0000 7fc693483780 1 bucket sync caught up with source: 2023-10-18T19:55:38.665 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.25.11, , , , , ] 2023-10-18T19:55:38.666 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.25.11, , , , , ] 2023-10-18T19:55:38.666 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:38.663+0000 7fc693483780 0 bucket checkpoint complete 2023-10-18T19:55:38.680 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:55:38.681 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:55:38.681 DEBUG:boto:Method: HEAD 2023-10-18T19:55:38.681 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:55:38.682 DEBUG:boto:Data: 2023-10-18T19:55:38.682 DEBUG:boto:Headers: {} 2023-10-18T19:55:38.683 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.683 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.683 DEBUG:boto:Params: {} 2023-10-18T19:55:38.684 DEBUG:boto:Token: None 2023-10-18T19:55:38.684 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/ 2023-10-18T19:55:38.685 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o0+qbl88K7lRAFQ1Zk/Fhczhi6M= 2023-10-18T19:55:38.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o0+qbl88K7lRAFQ1Zk/Fhczhi6M=', 'Content-Length': '0'} 2023-10-18T19:55:38.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca36b434f20a963a-006530383a-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.688 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.688 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.689 DEBUG:boto:Method: PUT 2023-10-18T19:55:38.689 DEBUG:boto:Path: /bofrnf-11/obj-a1 2023-10-18T19:55:38.689 DEBUG:boto:Data: 2023-10-18T19:55:38.690 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:38.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.691 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.691 DEBUG:boto:Params: {} 2023-10-18T19:55:38.691 DEBUG:boto:Token: None 2023-10-18T19:55:38.692 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a1 2023-10-18T19:55:38.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bFrFxMIbDPP41bvb78Y2532A8gw= 2023-10-18T19:55:38.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bFrFxMIbDPP41bvb78Y2532A8gw='} 2023-10-18T19:55:38.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-amz-request-id', 'tx00000d4520458d7f23700-006530383a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.705 DEBUG:rgw_multi.tests:version1 id=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:55:38.705 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.705 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.706 DEBUG:boto:Method: PUT 2023-10-18T19:55:38.706 DEBUG:boto:Path: /bofrnf-11/obj-a1 2023-10-18T19:55:38.707 DEBUG:boto:Data: 2023-10-18T19:55:38.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:38.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.708 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.708 DEBUG:boto:Params: {} 2023-10-18T19:55:38.708 DEBUG:boto:Token: None 2023-10-18T19:55:38.709 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a1 2023-10-18T19:55:38.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZF/rXToUc2Gd4NPVqGEkLXXFwIA= 2023-10-18T19:55:38.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZF/rXToUc2Gd4NPVqGEkLXXFwIA='} 2023-10-18T19:55:38.719 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Lhers2vxodKopssnzdaYnXKgDDPQZOv'), ('x-amz-request-id', 'tx00000960798934147fa92-006530383a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.720 DEBUG:rgw_multi.tests:version2 id=Lhers2vxodKopssnzdaYnXKgDDPQZOv 2023-10-18T19:55:38.720 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.721 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.721 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=Lhers2vxodKopssnzdaYnXKgDDPQZOv 2023-10-18T19:55:38.721 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=Lhers2vxodKopssnzdaYnXKgDDPQZOv 2023-10-18T19:55:38.722 DEBUG:boto:Method: DELETE 2023-10-18T19:55:38.722 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=Lhers2vxodKopssnzdaYnXKgDDPQZOv 2023-10-18T19:55:38.722 DEBUG:boto:Data: 2023-10-18T19:55:38.723 DEBUG:boto:Headers: {} 2023-10-18T19:55:38.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.724 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.724 DEBUG:boto:Params: {} 2023-10-18T19:55:38.724 DEBUG:boto:Token: None 2023-10-18T19:55:38.725 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a1?versionId=Lhers2vxodKopssnzdaYnXKgDDPQZOv 2023-10-18T19:55:38.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bFehdz9GW/xBU6J2HzfeALYumN8= 2023-10-18T19:55:38.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bFehdz9GW/xBU6J2HzfeALYumN8=', 'Content-Length': '0'} 2023-10-18T19:55:38.735 DEBUG:boto:Response headers: [('x-amz-version-id', 'Lhers2vxodKopssnzdaYnXKgDDPQZOv'), ('x-amz-request-id', 'tx0000009aad6833727566e-006530383a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.736 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.736 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.737 DEBUG:boto:Method: PUT 2023-10-18T19:55:38.737 DEBUG:boto:Path: /bofrnf-11/obj-a1 2023-10-18T19:55:38.738 DEBUG:boto:Data: 2023-10-18T19:55:38.738 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:38.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.739 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.739 DEBUG:boto:Params: {} 2023-10-18T19:55:38.740 DEBUG:boto:Token: None 2023-10-18T19:55:38.740 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a1 2023-10-18T19:55:38.741 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aawBsc+pn0AtTSoqOWtfNqb3dRI= 2023-10-18T19:55:38.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aawBsc+pn0AtTSoqOWtfNqb3dRI='} 2023-10-18T19:55:38.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '86mVj2MtR4ZjM5PZcC0I9Dd0oJ0nqj8'), ('x-amz-request-id', 'tx00000a5c552a7e4585224-006530383a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.791 DEBUG:rgw_multi.tests:version3 id=86mVj2MtR4ZjM5PZcC0I9Dd0oJ0nqj8 2023-10-18T19:55:38.792 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.792 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:55:38.792 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=86mVj2MtR4ZjM5PZcC0I9Dd0oJ0nqj8 2023-10-18T19:55:38.793 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=86mVj2MtR4ZjM5PZcC0I9Dd0oJ0nqj8 2023-10-18T19:55:38.793 DEBUG:boto:Method: DELETE 2023-10-18T19:55:38.794 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=86mVj2MtR4ZjM5PZcC0I9Dd0oJ0nqj8 2023-10-18T19:55:38.794 DEBUG:boto:Data: 2023-10-18T19:55:38.794 DEBUG:boto:Headers: {} 2023-10-18T19:55:38.795 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.795 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.796 DEBUG:boto:Params: {} 2023-10-18T19:55:38.796 DEBUG:boto:Token: None 2023-10-18T19:55:38.796 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a1?versionId=86mVj2MtR4ZjM5PZcC0I9Dd0oJ0nqj8 2023-10-18T19:55:38.797 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7j7hYKILk9Bg1sYpwD+K+PskVY= 2023-10-18T19:55:38.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7j7hYKILk9Bg1sYpwD+K+PskVY=', 'Content-Length': '0'} 2023-10-18T19:55:38.807 DEBUG:boto:Response headers: [('x-amz-version-id', '86mVj2MtR4ZjM5PZcC0I9Dd0oJ0nqj8'), ('x-amz-request-id', 'tx000004cbecdbe527149e2-006530383a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.808 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:55:38.808 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:55:38.809 DEBUG:boto:Method: HEAD 2023-10-18T19:55:38.809 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:55:38.809 DEBUG:boto:Data: 2023-10-18T19:55:38.810 DEBUG:boto:Headers: {} 2023-10-18T19:55:38.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:38.811 DEBUG:boto:Port: 8001 2023-10-18T19:55:38.811 DEBUG:boto:Params: {} 2023-10-18T19:55:38.811 DEBUG:boto:Token: None 2023-10-18T19:55:38.812 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/ 2023-10-18T19:55:38.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o0+qbl88K7lRAFQ1Zk/Fhczhi6M= 2023-10-18T19:55:38.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o0+qbl88K7lRAFQ1Zk/Fhczhi6M=', 'Content-Length': '0'} 2023-10-18T19:55:38.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063a28c8f7aebbe6c-006530383a-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.816 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.816 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.817 DEBUG:boto:Method: PUT 2023-10-18T19:55:38.817 DEBUG:boto:Path: /bofrnf-11/obj-a2 2023-10-18T19:55:38.818 DEBUG:boto:Data: 2023-10-18T19:55:38.818 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:38.818 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:38.819 DEBUG:boto:Port: 8001 2023-10-18T19:55:38.819 DEBUG:boto:Params: {} 2023-10-18T19:55:38.819 DEBUG:boto:Token: None 2023-10-18T19:55:38.820 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a2 2023-10-18T19:55:38.820 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FZG3hq0ffcRS+Y/9uTgXEkAecUg= 2023-10-18T19:55:38.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FZG3hq0ffcRS+Y/9uTgXEkAecUg='} 2023-10-18T19:55:38.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-amz-request-id', 'tx000005b94ea2face3a669-006530383a-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.832 DEBUG:rgw_multi.tests:version1 id=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:55:38.832 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.833 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.833 DEBUG:boto:Method: PUT 2023-10-18T19:55:38.834 DEBUG:boto:Path: /bofrnf-11/obj-a2 2023-10-18T19:55:38.834 DEBUG:boto:Data: 2023-10-18T19:55:38.834 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:38.835 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:38.835 DEBUG:boto:Port: 8001 2023-10-18T19:55:38.835 DEBUG:boto:Params: {} 2023-10-18T19:55:38.836 DEBUG:boto:Token: None 2023-10-18T19:55:38.836 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a2 2023-10-18T19:55:38.837 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IiW+ftS1ifoBQxFOhldomxalOJ4= 2023-10-18T19:55:38.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IiW+ftS1ifoBQxFOhldomxalOJ4='} 2023-10-18T19:55:38.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'AXHqWpe.NGnsVoQIAcmN2yf1mPl6SMw'), ('x-amz-request-id', 'tx0000048a103c8e37c4432-006530383a-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.883 DEBUG:rgw_multi.tests:version2 id=AXHqWpe.NGnsVoQIAcmN2yf1mPl6SMw 2023-10-18T19:55:38.883 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.884 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.884 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=AXHqWpe.NGnsVoQIAcmN2yf1mPl6SMw 2023-10-18T19:55:38.884 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=AXHqWpe.NGnsVoQIAcmN2yf1mPl6SMw 2023-10-18T19:55:38.885 DEBUG:boto:Method: DELETE 2023-10-18T19:55:38.885 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=AXHqWpe.NGnsVoQIAcmN2yf1mPl6SMw 2023-10-18T19:55:38.885 DEBUG:boto:Data: 2023-10-18T19:55:38.886 DEBUG:boto:Headers: {} 2023-10-18T19:55:38.886 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:38.886 DEBUG:boto:Port: 8001 2023-10-18T19:55:38.887 DEBUG:boto:Params: {} 2023-10-18T19:55:38.887 DEBUG:boto:Token: None 2023-10-18T19:55:38.888 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a2?versionId=AXHqWpe.NGnsVoQIAcmN2yf1mPl6SMw 2023-10-18T19:55:38.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p5N2zdRpqGisNIY9FKPPrtrciTU= 2023-10-18T19:55:38.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p5N2zdRpqGisNIY9FKPPrtrciTU=', 'Content-Length': '0'} 2023-10-18T19:55:38.899 DEBUG:boto:Response headers: [('x-amz-version-id', 'AXHqWpe.NGnsVoQIAcmN2yf1mPl6SMw'), ('x-amz-request-id', 'tx0000002cb39354d04ecd5-006530383a-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.900 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.900 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.901 DEBUG:boto:Method: PUT 2023-10-18T19:55:38.901 DEBUG:boto:Path: /bofrnf-11/obj-a2 2023-10-18T19:55:38.901 DEBUG:boto:Data: 2023-10-18T19:55:38.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:38.902 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:38.902 DEBUG:boto:Port: 8001 2023-10-18T19:55:38.903 DEBUG:boto:Params: {} 2023-10-18T19:55:38.903 DEBUG:boto:Token: None 2023-10-18T19:55:38.904 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a2 2023-10-18T19:55:38.904 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vPbrFXpsVRXYQu1W9yR2pm6SRUQ= 2023-10-18T19:55:38.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vPbrFXpsVRXYQu1W9yR2pm6SRUQ='} 2023-10-18T19:55:38.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'x.1A4xITVIaPDa94jfRqTqCgZ3JX1DG'), ('x-amz-request-id', 'tx0000001e066825f526b26-006530383a-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.913 DEBUG:rgw_multi.tests:version3 id=x.1A4xITVIaPDa94jfRqTqCgZ3JX1DG 2023-10-18T19:55:38.914 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.914 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:55:38.914 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=x.1A4xITVIaPDa94jfRqTqCgZ3JX1DG 2023-10-18T19:55:38.915 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=x.1A4xITVIaPDa94jfRqTqCgZ3JX1DG 2023-10-18T19:55:38.915 DEBUG:boto:Method: DELETE 2023-10-18T19:55:38.915 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=x.1A4xITVIaPDa94jfRqTqCgZ3JX1DG 2023-10-18T19:55:38.916 DEBUG:boto:Data: 2023-10-18T19:55:38.916 DEBUG:boto:Headers: {} 2023-10-18T19:55:38.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:38.917 DEBUG:boto:Port: 8001 2023-10-18T19:55:38.917 DEBUG:boto:Params: {} 2023-10-18T19:55:38.918 DEBUG:boto:Token: None 2023-10-18T19:55:38.918 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-11/obj-a2?versionId=x.1A4xITVIaPDa94jfRqTqCgZ3JX1DG 2023-10-18T19:55:38.918 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oeG/Hma/do2QKOGFXJt+eTV760o= 2023-10-18T19:55:38.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oeG/Hma/do2QKOGFXJt+eTV760o=', 'Content-Length': '0'} 2023-10-18T19:55:38.929 DEBUG:boto:Response headers: [('x-amz-version-id', 'x.1A4xITVIaPDa94jfRqTqCgZ3JX1DG'), ('x-amz-request-id', 'tx000000915a21632326753-006530383a-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.929 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:55:38.930 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:55:38.930 DEBUG:boto:Method: HEAD 2023-10-18T19:55:38.930 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:55:38.931 DEBUG:boto:Data: 2023-10-18T19:55:38.931 DEBUG:boto:Headers: {} 2023-10-18T19:55:38.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.932 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.932 DEBUG:boto:Params: {} 2023-10-18T19:55:38.933 DEBUG:boto:Token: None 2023-10-18T19:55:38.933 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-12/ 2023-10-18T19:55:38.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ilzyvSFdfosYRRaQocKw2m3Kl7w= 2023-10-18T19:55:38.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ilzyvSFdfosYRRaQocKw2m3Kl7w=', 'Content-Length': '0'} 2023-10-18T19:55:38.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037ac662dd01fcfb3-006530383a-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.937 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:55:38.937 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:55:38.938 DEBUG:boto:Method: PUT 2023-10-18T19:55:38.938 DEBUG:boto:Path: /bofrnf-12/obj-a1 2023-10-18T19:55:38.938 DEBUG:boto:Data: 2023-10-18T19:55:38.939 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:38.939 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.939 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.940 DEBUG:boto:Params: {} 2023-10-18T19:55:38.940 DEBUG:boto:Token: None 2023-10-18T19:55:38.941 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-12/obj-a1 2023-10-18T19:55:38.941 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GP1/7wl/JYkXnrnMJK+J15xQdIo= 2023-10-18T19:55:38.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GP1/7wl/JYkXnrnMJK+J15xQdIo='} 2023-10-18T19:55:38.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-amz-request-id', 'tx000008ddb8f105baf2952-006530383a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.951 DEBUG:rgw_multi.tests:version1 id=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:55:38.952 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:55:38.952 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:55:38.952 DEBUG:boto:Method: PUT 2023-10-18T19:55:38.953 DEBUG:boto:Path: /bofrnf-12/obj-a1 2023-10-18T19:55:38.953 DEBUG:boto:Data: 2023-10-18T19:55:38.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:38.954 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.954 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.955 DEBUG:boto:Params: {} 2023-10-18T19:55:38.955 DEBUG:boto:Token: None 2023-10-18T19:55:38.955 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-12/obj-a1 2023-10-18T19:55:38.956 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g0bwoH5WCOWKjGKUs8j0wi3de1Y= 2023-10-18T19:55:38.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g0bwoH5WCOWKjGKUs8j0wi3de1Y='} 2023-10-18T19:55:38.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JELeO0MN2zzaAI8T3sTw391zZBGa-f4'), ('x-amz-request-id', 'tx00000024dd564feb2a235-006530383a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.965 DEBUG:rgw_multi.tests:version2 id=JELeO0MN2zzaAI8T3sTw391zZBGa-f4 2023-10-18T19:55:38.965 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:55:38.965 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:55:38.966 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=JELeO0MN2zzaAI8T3sTw391zZBGa-f4 2023-10-18T19:55:38.966 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=JELeO0MN2zzaAI8T3sTw391zZBGa-f4 2023-10-18T19:55:38.966 DEBUG:boto:Method: DELETE 2023-10-18T19:55:38.967 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=JELeO0MN2zzaAI8T3sTw391zZBGa-f4 2023-10-18T19:55:38.967 DEBUG:boto:Data: 2023-10-18T19:55:38.967 DEBUG:boto:Headers: {} 2023-10-18T19:55:38.968 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.968 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.968 DEBUG:boto:Params: {} 2023-10-18T19:55:38.969 DEBUG:boto:Token: None 2023-10-18T19:55:38.969 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-12/obj-a1?versionId=JELeO0MN2zzaAI8T3sTw391zZBGa-f4 2023-10-18T19:55:38.970 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X46IHPAv9jUfqzWozvFmJ+dOdcw= 2023-10-18T19:55:38.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X46IHPAv9jUfqzWozvFmJ+dOdcw=', 'Content-Length': '0'} 2023-10-18T19:55:38.978 DEBUG:boto:Response headers: [('x-amz-version-id', 'JELeO0MN2zzaAI8T3sTw391zZBGa-f4'), ('x-amz-request-id', 'tx00000eac8e874ad0a9223-006530383a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:38.979 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:55:38.979 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:55:38.980 DEBUG:boto:Method: PUT 2023-10-18T19:55:38.980 DEBUG:boto:Path: /bofrnf-12/obj-a1 2023-10-18T19:55:38.980 DEBUG:boto:Data: 2023-10-18T19:55:38.981 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:38.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:38.981 DEBUG:boto:Port: 8000 2023-10-18T19:55:38.982 DEBUG:boto:Params: {} 2023-10-18T19:55:38.982 DEBUG:boto:Token: None 2023-10-18T19:55:38.983 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 18 Oct 2023 19:55:38 GMT /bofrnf-12/obj-a1 2023-10-18T19:55:38.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M/7YGbyKynTDANBtGY41mmHbGZw= 2023-10-18T19:55:38.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M/7YGbyKynTDANBtGY41mmHbGZw='} 2023-10-18T19:55:39.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4iC9oo3oTCDn1OaKpxUxAHOdaGp3T3k'), ('x-amz-request-id', 'tx000002b6cebc3ca21e0d9-006530383a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:39.027 DEBUG:rgw_multi.tests:version3 id=4iC9oo3oTCDn1OaKpxUxAHOdaGp3T3k 2023-10-18T19:55:39.027 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:55:39.027 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:55:39.028 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=4iC9oo3oTCDn1OaKpxUxAHOdaGp3T3k 2023-10-18T19:55:39.028 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=4iC9oo3oTCDn1OaKpxUxAHOdaGp3T3k 2023-10-18T19:55:39.029 DEBUG:boto:Method: DELETE 2023-10-18T19:55:39.029 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=4iC9oo3oTCDn1OaKpxUxAHOdaGp3T3k 2023-10-18T19:55:39.029 DEBUG:boto:Data: 2023-10-18T19:55:39.030 DEBUG:boto:Headers: {} 2023-10-18T19:55:39.030 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:55:39.030 DEBUG:boto:Port: 8000 2023-10-18T19:55:39.031 DEBUG:boto:Params: {} 2023-10-18T19:55:39.031 DEBUG:boto:Token: None 2023-10-18T19:55:39.031 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:55:39 GMT /bofrnf-12/obj-a1?versionId=4iC9oo3oTCDn1OaKpxUxAHOdaGp3T3k 2023-10-18T19:55:39.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xZpwsMATpyqCwY+oRDBmrEG8E4w= 2023-10-18T19:55:39.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xZpwsMATpyqCwY+oRDBmrEG8E4w=', 'Content-Length': '0'} 2023-10-18T19:55:39.041 DEBUG:boto:Response headers: [('x-amz-version-id', '4iC9oo3oTCDn1OaKpxUxAHOdaGp3T3k'), ('x-amz-request-id', 'tx00000165be5347cda7742-006530383b-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:39.041 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:55:39.041 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:55:39.042 DEBUG:boto:Method: HEAD 2023-10-18T19:55:39.042 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:55:39.043 DEBUG:boto:Data: 2023-10-18T19:55:39.043 DEBUG:boto:Headers: {} 2023-10-18T19:55:39.043 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:39.044 DEBUG:boto:Port: 8001 2023-10-18T19:55:39.044 DEBUG:boto:Params: {} 2023-10-18T19:55:39.044 DEBUG:boto:Token: None 2023-10-18T19:55:39.045 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:55:39 GMT /bofrnf-12/ 2023-10-18T19:55:39.045 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5n398+rc6H9UJb6GHZWQ/x48+4c= 2023-10-18T19:55:39.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5n398+rc6H9UJb6GHZWQ/x48+4c=', 'Content-Length': '0'} 2023-10-18T19:55:39.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a69b80e9bbed7814-006530383b-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:39.048 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.049 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.049 DEBUG:boto:Method: PUT 2023-10-18T19:55:39.049 DEBUG:boto:Path: /bofrnf-12/obj-a2 2023-10-18T19:55:39.050 DEBUG:boto:Data: 2023-10-18T19:55:39.050 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:39.051 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:39.051 DEBUG:boto:Port: 8001 2023-10-18T19:55:39.051 DEBUG:boto:Params: {} 2023-10-18T19:55:39.052 DEBUG:boto:Token: None 2023-10-18T19:55:39.052 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:55:39 GMT /bofrnf-12/obj-a2 2023-10-18T19:55:39.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rn2MBLxeHDOL6Vp31U34O7amUos= 2023-10-18T19:55:39.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rn2MBLxeHDOL6Vp31U34O7amUos='} 2023-10-18T19:55:39.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-amz-request-id', 'tx00000509600d10dc888ab-006530383b-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:39.066 DEBUG:rgw_multi.tests:version1 id=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:55:39.066 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.066 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.067 DEBUG:boto:Method: PUT 2023-10-18T19:55:39.067 DEBUG:boto:Path: /bofrnf-12/obj-a2 2023-10-18T19:55:39.068 DEBUG:boto:Data: 2023-10-18T19:55:39.068 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:39.068 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:39.069 DEBUG:boto:Port: 8001 2023-10-18T19:55:39.069 DEBUG:boto:Params: {} 2023-10-18T19:55:39.069 DEBUG:boto:Token: None 2023-10-18T19:55:39.070 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 18 Oct 2023 19:55:39 GMT /bofrnf-12/obj-a2 2023-10-18T19:55:39.070 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sYsJ1KsQkDTUmkCKaZSEBsrco1Y= 2023-10-18T19:55:39.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sYsJ1KsQkDTUmkCKaZSEBsrco1Y='} 2023-10-18T19:55:39.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fZZ1PWxc-myuqvg246L4MoGo3UlMgpg'), ('x-amz-request-id', 'tx0000092301d4360e46181-006530383b-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:39.081 DEBUG:rgw_multi.tests:version2 id=fZZ1PWxc-myuqvg246L4MoGo3UlMgpg 2023-10-18T19:55:39.082 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.082 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.083 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=fZZ1PWxc-myuqvg246L4MoGo3UlMgpg 2023-10-18T19:55:39.083 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=fZZ1PWxc-myuqvg246L4MoGo3UlMgpg 2023-10-18T19:55:39.083 DEBUG:boto:Method: DELETE 2023-10-18T19:55:39.084 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=fZZ1PWxc-myuqvg246L4MoGo3UlMgpg 2023-10-18T19:55:39.084 DEBUG:boto:Data: 2023-10-18T19:55:39.084 DEBUG:boto:Headers: {} 2023-10-18T19:55:39.085 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:39.085 DEBUG:boto:Port: 8001 2023-10-18T19:55:39.085 DEBUG:boto:Params: {} 2023-10-18T19:55:39.086 DEBUG:boto:Token: None 2023-10-18T19:55:39.086 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:55:39 GMT /bofrnf-12/obj-a2?versionId=fZZ1PWxc-myuqvg246L4MoGo3UlMgpg 2023-10-18T19:55:39.087 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6JlYucSTPTdr4D94W2bDLw0Wc9A= 2023-10-18T19:55:39.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6JlYucSTPTdr4D94W2bDLw0Wc9A=', 'Content-Length': '0'} 2023-10-18T19:55:39.096 DEBUG:boto:Response headers: [('x-amz-version-id', 'fZZ1PWxc-myuqvg246L4MoGo3UlMgpg'), ('x-amz-request-id', 'tx000004201d3c403e5ccf5-006530383b-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:39.097 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.097 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.098 DEBUG:boto:Method: PUT 2023-10-18T19:55:39.098 DEBUG:boto:Path: /bofrnf-12/obj-a2 2023-10-18T19:55:39.098 DEBUG:boto:Data: 2023-10-18T19:55:39.099 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:55:39.099 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:39.099 DEBUG:boto:Port: 8001 2023-10-18T19:55:39.100 DEBUG:boto:Params: {} 2023-10-18T19:55:39.100 DEBUG:boto:Token: None 2023-10-18T19:55:39.101 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 18 Oct 2023 19:55:39 GMT /bofrnf-12/obj-a2 2023-10-18T19:55:39.101 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z6ZyX0EFTbGDbB1RlZ78lY7BWV4= 2023-10-18T19:55:39.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:55:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z6ZyX0EFTbGDbB1RlZ78lY7BWV4='} 2023-10-18T19:55:39.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9QlSUuH1iuINvUixgLWbMU3n1lWzMff'), ('x-amz-request-id', 'tx000002bb10e7cbb989599-006530383b-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:39.147 DEBUG:rgw_multi.tests:version3 id=9QlSUuH1iuINvUixgLWbMU3n1lWzMff 2023-10-18T19:55:39.147 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.148 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:55:39.148 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=9QlSUuH1iuINvUixgLWbMU3n1lWzMff 2023-10-18T19:55:39.148 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=9QlSUuH1iuINvUixgLWbMU3n1lWzMff 2023-10-18T19:55:39.149 DEBUG:boto:Method: DELETE 2023-10-18T19:55:39.149 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=9QlSUuH1iuINvUixgLWbMU3n1lWzMff 2023-10-18T19:55:39.149 DEBUG:boto:Data: 2023-10-18T19:55:39.150 DEBUG:boto:Headers: {} 2023-10-18T19:55:39.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:55:39.151 DEBUG:boto:Port: 8001 2023-10-18T19:55:39.151 DEBUG:boto:Params: {} 2023-10-18T19:55:39.151 DEBUG:boto:Token: None 2023-10-18T19:55:39.152 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 19:55:39 GMT /bofrnf-12/obj-a2?versionId=9QlSUuH1iuINvUixgLWbMU3n1lWzMff 2023-10-18T19:55:39.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tXcpl7jycXI2jVfwBa2AArATp/Q= 2023-10-18T19:55:39.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:55:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tXcpl7jycXI2jVfwBa2AArATp/Q=', 'Content-Length': '0'} 2023-10-18T19:55:39.161 DEBUG:boto:Response headers: [('x-amz-version-id', '9QlSUuH1iuINvUixgLWbMU3n1lWzMff'), ('x-amz-request-id', 'tx00000d9270b10bff65525-006530383b-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:55:39.162 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-11 2023-10-18T19:55:39.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:55:39.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:55:39.163 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:55:39.220 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:55:39.220 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:55:39.253 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:39.251+0000 7f8f76b6e780 1 waiting for incremental sync to catch up: 2023-10-18T19:55:39.253 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , , , ] 2023-10-18T19:55:39.253 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , , 00000000010.21.6, ] 2023-10-18T19:55:44.254 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:44.253+0000 7f8f76b6e780 1 waiting for incremental sync to catch up: 2023-10-18T19:55:44.255 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , , , ] 2023-10-18T19:55:44.255 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , , 00000000010.21.6, ] 2023-10-18T19:55:49.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:49.254+0000 7f8f76b6e780 1 waiting for incremental sync to catch up: 2023-10-18T19:55:49.256 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , , , ] 2023-10-18T19:55:49.256 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , , 00000000010.21.6, ] 2023-10-18T19:55:54.257 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:54.255+0000 7f8f76b6e780 1 waiting for incremental sync to catch up: 2023-10-18T19:55:54.257 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , , , ] 2023-10-18T19:55:54.257 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , , 00000000010.21.6, ] 2023-10-18T19:55:59.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:59.256+0000 7f8f76b6e780 1 bucket sync caught up with source: 2023-10-18T19:55:59.259 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , , 00000000010.21.6, ] 2023-10-18T19:55:59.259 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , , 00000000010.21.6, ] 2023-10-18T19:55:59.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:59.256+0000 7f8f76b6e780 0 bucket checkpoint complete 2023-10-18T19:55:59.274 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-11 2023-10-18T19:55:59.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:55:59.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:55:59.275 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:55:59.339 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:55:59.340 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:55:59.376 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:55:59.374+0000 7f14e3efd780 1 waiting for incremental sync to catch up: 2023-10-18T19:55:59.376 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.26.6, , ] 2023-10-18T19:55:59.376 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.26.6, 00000000002.12.11, ] 2023-10-18T19:56:04.377 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:04.375+0000 7f14e3efd780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:04.378 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.26.6, , ] 2023-10-18T19:56:04.378 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.26.6, 00000000002.12.11, ] 2023-10-18T19:56:09.379 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:09.376+0000 7f14e3efd780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:09.379 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.26.6, , ] 2023-10-18T19:56:09.379 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.26.6, 00000000002.12.11, ] 2023-10-18T19:56:14.380 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:14.377+0000 7f14e3efd780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:14.381 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.26.6, , ] 2023-10-18T19:56:14.381 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.26.6, 00000000002.12.11, ] 2023-10-18T19:56:19.383 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:19.381+0000 7f14e3efd780 1 bucket sync caught up with source: 2023-10-18T19:56:19.383 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000010.26.6, 00000000002.12.11, ] 2023-10-18T19:56:19.384 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000010.26.6, 00000000002.12.11, ] 2023-10-18T19:56:19.384 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:19.381+0000 7f14e3efd780 0 bucket checkpoint complete 2023-10-18T19:56:19.398 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-11 zones={a2, a1} 2023-10-18T19:56:19.399 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.399 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.400 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.400 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:56:19.401 DEBUG:boto:Data: 2023-10-18T19:56:19.401 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.402 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.402 DEBUG:boto:Params: {} 2023-10-18T19:56:19.403 DEBUG:boto:Token: None 2023-10-18T19:56:19.403 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FcZDXQqntJAmg3ITiBtZqWQ2lWg= 2023-10-18T19:56:19.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FcZDXQqntJAmg3ITiBtZqWQ2lWg=', 'Content-Length': '0'} 2023-10-18T19:56:19.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dba4ad854268eae0-0065303863-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.408 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.408 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.409 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.409 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:56:19.409 DEBUG:boto:Data: 2023-10-18T19:56:19.410 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.410 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.411 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.411 DEBUG:boto:Params: {} 2023-10-18T19:56:19.412 DEBUG:boto:Token: None 2023-10-18T19:56:19.412 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.412 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FcZDXQqntJAmg3ITiBtZqWQ2lWg= 2023-10-18T19:56:19.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FcZDXQqntJAmg3ITiBtZqWQ2lWg=', 'Content-Length': '0'} 2023-10-18T19:56:19.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b0f9131a9cea6fd-0065303863-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.416 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:56:19.416 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.417 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.417 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.418 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.418 DEBUG:boto:Method: GET 2023-10-18T19:56:19.419 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.419 DEBUG:boto:Data: 2023-10-18T19:56:19.419 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.420 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.420 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.421 DEBUG:boto:Params: {} 2023-10-18T19:56:19.421 DEBUG:boto:Token: None 2023-10-18T19:56:19.421 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/?versions 2023-10-18T19:56:19.422 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n97+U7rH6NZOz1d5QXnAV+PY8Xk= 2023-10-18T19:56:19.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n97+U7rH6NZOz1d5QXnAV+PY8Xk=', 'Content-Length': '0'} 2023-10-18T19:56:19.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba555a5789ebfef-0065303863-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.426 DEBUG:boto:b'bofrnf-11999falsedummyejuhtdELxhW-1atXJMLKJcENH4sJdCgtrue2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkXtrue2023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2SPXUXHYcw3.HZZ0z2coSO05aV.6WDqVtrue2023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.427 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:56:19.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-18T19:56:19.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-18T19:56:19.428 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:56:19.428 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.429 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.429 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.429 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.430 DEBUG:boto:Method: GET 2023-10-18T19:56:19.430 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.431 DEBUG:boto:Data: 2023-10-18T19:56:19.431 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.432 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.432 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.432 DEBUG:boto:Params: {} 2023-10-18T19:56:19.433 DEBUG:boto:Token: None 2023-10-18T19:56:19.433 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/?versions 2023-10-18T19:56:19.434 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n97+U7rH6NZOz1d5QXnAV+PY8Xk= 2023-10-18T19:56:19.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n97+U7rH6NZOz1d5QXnAV+PY8Xk=', 'Content-Length': '0'} 2023-10-18T19:56:19.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034d8fbfbcfab8459-0065303863-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.437 DEBUG:boto:b'bofrnf-11999falsedummyejuhtdELxhW-1atXJMLKJcENH4sJdCgtrue2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkXtrue2023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2SPXUXHYcw3.HZZ0z2coSO05aV.6WDqVtrue2023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.438 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:56:19.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-18T19:56:19.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-18T19:56:19.439 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.440 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.440 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.441 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.441 DEBUG:boto:Method: GET 2023-10-18T19:56:19.441 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.442 DEBUG:boto:Data: 2023-10-18T19:56:19.442 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.443 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.443 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.443 DEBUG:boto:Params: {} 2023-10-18T19:56:19.444 DEBUG:boto:Token: None 2023-10-18T19:56:19.444 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/?versions 2023-10-18T19:56:19.445 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n97+U7rH6NZOz1d5QXnAV+PY8Xk= 2023-10-18T19:56:19.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n97+U7rH6NZOz1d5QXnAV+PY8Xk=', 'Content-Length': '0'} 2023-10-18T19:56:19.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3ec1f3c76405671-0065303863-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.449 DEBUG:boto:b'bofrnf-11999falsedummyejuhtdELxhW-1atXJMLKJcENH4sJdCgtrue2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkXtrue2023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2SPXUXHYcw3.HZZ0z2coSO05aV.6WDqVtrue2023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.449 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.450 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.450 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.451 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.451 DEBUG:boto:Method: GET 2023-10-18T19:56:19.451 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999 2023-10-18T19:56:19.452 DEBUG:boto:Data: 2023-10-18T19:56:19.452 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.453 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.454 DEBUG:boto:Params: {} 2023-10-18T19:56:19.454 DEBUG:boto:Token: None 2023-10-18T19:56:19.454 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/?versions 2023-10-18T19:56:19.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n97+U7rH6NZOz1d5QXnAV+PY8Xk= 2023-10-18T19:56:19.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n97+U7rH6NZOz1d5QXnAV+PY8Xk=', 'Content-Length': '0'} 2023-10-18T19:56:19.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd9c15222df19327-0065303863-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.458 DEBUG:boto:b'bofrnf-11999falsedummyejuhtdELxhW-1atXJMLKJcENH4sJdCgtrue2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkXtrue2023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2SPXUXHYcw3.HZZ0z2coSO05aV.6WDqVtrue2023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:56:19.459 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.460 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.460 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.461 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.461 DEBUG:boto:Method: GET 2023-10-18T19:56:19.462 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.462 DEBUG:boto:Data: 2023-10-18T19:56:19.463 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.463 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.463 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.464 DEBUG:boto:Params: {} 2023-10-18T19:56:19.464 DEBUG:boto:Token: None 2023-10-18T19:56:19.465 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.465 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY= 2023-10-18T19:56:19.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY=', 'Content-Length': '0'} 2023-10-18T19:56:19.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001683cf2e751f07af-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.469 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.469 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.470 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.470 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.471 DEBUG:boto:Method: GET 2023-10-18T19:56:19.471 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.471 DEBUG:boto:Data: 2023-10-18T19:56:19.472 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.472 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.473 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.473 DEBUG:boto:Params: {} 2023-10-18T19:56:19.474 DEBUG:boto:Token: None 2023-10-18T19:56:19.474 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.474 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY= 2023-10-18T19:56:19.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY=', 'Content-Length': '0'} 2023-10-18T19:56:19.477 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008adbbca995119f78-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.477 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.478 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.478 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.479 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.479 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.480 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.480 DEBUG:boto:Data: 2023-10-18T19:56:19.480 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.481 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.482 DEBUG:boto:Params: {} 2023-10-18T19:56:19.482 DEBUG:boto:Token: None 2023-10-18T19:56:19.483 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.483 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1ih1g1OoXWdbMfNoFqfLe9kt8Bc= 2023-10-18T19:56:19.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1ih1g1OoXWdbMfNoFqfLe9kt8Bc=', 'Content-Length': '0'} 2023-10-18T19:56:19.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd515e987cf0c4ac-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.486 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.486 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.487 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.487 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.488 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.488 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.488 DEBUG:boto:Data: 2023-10-18T19:56:19.489 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.489 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.490 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.490 DEBUG:boto:Params: {} 2023-10-18T19:56:19.490 DEBUG:boto:Token: None 2023-10-18T19:56:19.491 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.491 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1ih1g1OoXWdbMfNoFqfLe9kt8Bc= 2023-10-18T19:56:19.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1ih1g1OoXWdbMfNoFqfLe9kt8Bc=', 'Content-Length': '0'} 2023-10-18T19:56:19.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005329add2de0093e4-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:56:19.495 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.495 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.496 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.496 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.497 DEBUG:boto:Method: GET 2023-10-18T19:56:19.497 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.497 DEBUG:boto:Data: 2023-10-18T19:56:19.498 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.499 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.499 DEBUG:boto:Params: {} 2023-10-18T19:56:19.500 DEBUG:boto:Token: None 2023-10-18T19:56:19.500 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.501 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY= 2023-10-18T19:56:19.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY=', 'Content-Length': '0'} 2023-10-18T19:56:19.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003544003931c9093-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.504 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.504 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.504 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.505 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.505 DEBUG:boto:Method: GET 2023-10-18T19:56:19.506 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.506 DEBUG:boto:Data: 2023-10-18T19:56:19.507 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.507 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.508 DEBUG:boto:Params: {} 2023-10-18T19:56:19.508 DEBUG:boto:Token: None 2023-10-18T19:56:19.509 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY= 2023-10-18T19:56:19.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY=', 'Content-Length': '0'} 2023-10-18T19:56:19.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009daf8a0568612360-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.512 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.513 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.513 DEBUG:boto:path=/bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.514 DEBUG:boto:auth_path=/bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.514 DEBUG:boto:Method: GET 2023-10-18T19:56:19.514 DEBUG:boto:Path: /bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.515 DEBUG:boto:Data: 2023-10-18T19:56:19.515 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.516 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.516 DEBUG:boto:Params: {} 2023-10-18T19:56:19.517 DEBUG:boto:Token: None 2023-10-18T19:56:19.517 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.518 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b59482077c1c87-0065303863-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.562 DEBUG:boto:b'bofrnf-11dummy1000falsedummy2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.563 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.564 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.564 DEBUG:boto:path=/bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.564 DEBUG:boto:auth_path=/bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.565 DEBUG:boto:Method: GET 2023-10-18T19:56:19.565 DEBUG:boto:Path: /bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.566 DEBUG:boto:Data: 2023-10-18T19:56:19.566 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.567 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.567 DEBUG:boto:Params: {} 2023-10-18T19:56:19.568 DEBUG:boto:Token: None 2023-10-18T19:56:19.568 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e505ed383c1df73d-0065303863-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.572 DEBUG:boto:b'bofrnf-11dummy1000falsedummy2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.573 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.573 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.573 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.574 DEBUG:boto:Path: /bofrnf-11/dummy 2023-10-18T19:56:19.574 DEBUG:boto:Data: 2023-10-18T19:56:19.575 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.575 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.575 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.576 DEBUG:boto:Params: {} 2023-10-18T19:56:19.576 DEBUG:boto:Token: None 2023-10-18T19:56:19.577 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy 2023-10-18T19:56:19.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JS5yMkqJ2XjI1C2pS1k50+DwDV0= 2023-10-18T19:56:19.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JS5yMkqJ2XjI1C2pS1k50+DwDV0=', 'Content-Length': '0'} 2023-10-18T19:56:19.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a9eca3df3648ce0-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.581 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.581 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.581 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.582 DEBUG:boto:Path: /bofrnf-11/dummy 2023-10-18T19:56:19.582 DEBUG:boto:Data: 2023-10-18T19:56:19.583 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.583 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.584 DEBUG:boto:Params: {} 2023-10-18T19:56:19.584 DEBUG:boto:Token: None 2023-10-18T19:56:19.585 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy 2023-10-18T19:56:19.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JS5yMkqJ2XjI1C2pS1k50+DwDV0= 2023-10-18T19:56:19.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JS5yMkqJ2XjI1C2pS1k50+DwDV0=', 'Content-Length': '0'} 2023-10-18T19:56:19.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000652c5f00f372a94f-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:56:19.589 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.590 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.590 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.590 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.591 DEBUG:boto:Method: GET 2023-10-18T19:56:19.591 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.592 DEBUG:boto:Data: 2023-10-18T19:56:19.592 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.593 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.593 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.593 DEBUG:boto:Params: {} 2023-10-18T19:56:19.594 DEBUG:boto:Token: None 2023-10-18T19:56:19.594 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY= 2023-10-18T19:56:19.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY=', 'Content-Length': '0'} 2023-10-18T19:56:19.597 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c4b91b039678852-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.598 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:56:19.599 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:56:19.599 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.599 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.600 DEBUG:boto:Method: GET 2023-10-18T19:56:19.600 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.601 DEBUG:boto:Data: 2023-10-18T19:56:19.601 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.602 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.602 DEBUG:boto:Params: {} 2023-10-18T19:56:19.603 DEBUG:boto:Token: None 2023-10-18T19:56:19.603 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:56:19.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY= 2023-10-18T19:56:19.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6WEQR4+eI/c3DLXnt+De2uXobqY=', 'Content-Length': '0'} 2023-10-18T19:56:19.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d221c97007dbf267-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.607 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.607 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.608 DEBUG:boto:path=/bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.608 DEBUG:boto:auth_path=/bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.609 DEBUG:boto:Method: GET 2023-10-18T19:56:19.609 DEBUG:boto:Path: /bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.609 DEBUG:boto:Data: 2023-10-18T19:56:19.610 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.610 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.611 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.611 DEBUG:boto:Params: {} 2023-10-18T19:56:19.612 DEBUG:boto:Token: None 2023-10-18T19:56:19.612 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002418c5f19c7474a1-0065303863-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.616 DEBUG:boto:b'bofrnf-11dummy1000falsedummy2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.617 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.617 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.618 DEBUG:boto:path=/bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.618 DEBUG:boto:auth_path=/bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.618 DEBUG:boto:Method: GET 2023-10-18T19:56:19.619 DEBUG:boto:Path: /bofrnf-11/?prefix=dummy 2023-10-18T19:56:19.619 DEBUG:boto:Data: 2023-10-18T19:56:19.620 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.620 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.620 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.621 DEBUG:boto:Params: {} 2023-10-18T19:56:19.621 DEBUG:boto:Token: None 2023-10-18T19:56:19.622 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.622 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5d02b261443b2aa-0065303863-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.666 DEBUG:boto:b'bofrnf-11dummy1000falsedummy2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:56:19.668 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.668 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.669 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.669 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.669 DEBUG:boto:Method: GET 2023-10-18T19:56:19.670 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.670 DEBUG:boto:Data: 2023-10-18T19:56:19.671 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.671 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.672 DEBUG:boto:Params: {} 2023-10-18T19:56:19.672 DEBUG:boto:Token: None 2023-10-18T19:56:19.673 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY= 2023-10-18T19:56:19.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY=', 'Content-Length': '0'} 2023-10-18T19:56:19.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5a9b6d85ba26b9c-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.677 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.678 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.678 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.678 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.679 DEBUG:boto:Method: GET 2023-10-18T19:56:19.679 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.680 DEBUG:boto:Data: 2023-10-18T19:56:19.680 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.681 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.681 DEBUG:boto:Params: {} 2023-10-18T19:56:19.682 DEBUG:boto:Token: None 2023-10-18T19:56:19.682 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY= 2023-10-18T19:56:19.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY=', 'Content-Length': '0'} 2023-10-18T19:56:19.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc569b4a4282d169-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.686 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.687 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.687 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.687 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.688 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.688 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.689 DEBUG:boto:Data: 2023-10-18T19:56:19.689 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.690 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.690 DEBUG:boto:Params: {} 2023-10-18T19:56:19.691 DEBUG:boto:Token: None 2023-10-18T19:56:19.691 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofp6e1xQ2srxZP7ei6bN5EY56V4= 2023-10-18T19:56:19.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofp6e1xQ2srxZP7ei6bN5EY56V4=', 'Content-Length': '0'} 2023-10-18T19:56:19.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e99245fe5e1d38c9-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.695 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.695 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.695 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.696 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.696 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.697 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.697 DEBUG:boto:Data: 2023-10-18T19:56:19.697 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.698 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.699 DEBUG:boto:Params: {} 2023-10-18T19:56:19.699 DEBUG:boto:Token: None 2023-10-18T19:56:19.700 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.700 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofp6e1xQ2srxZP7ei6bN5EY56V4= 2023-10-18T19:56:19.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofp6e1xQ2srxZP7ei6bN5EY56V4=', 'Content-Length': '0'} 2023-10-18T19:56:19.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000229bb6b52f82161d-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:56:19.703 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.704 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.704 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.705 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.705 DEBUG:boto:Method: GET 2023-10-18T19:56:19.706 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.706 DEBUG:boto:Data: 2023-10-18T19:56:19.707 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.707 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.708 DEBUG:boto:Params: {} 2023-10-18T19:56:19.708 DEBUG:boto:Token: None 2023-10-18T19:56:19.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY= 2023-10-18T19:56:19.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY=', 'Content-Length': '0'} 2023-10-18T19:56:19.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf223c0664a01255-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.712 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.713 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.713 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.714 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.714 DEBUG:boto:Method: GET 2023-10-18T19:56:19.714 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.715 DEBUG:boto:Data: 2023-10-18T19:56:19.715 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.716 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.716 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.716 DEBUG:boto:Params: {} 2023-10-18T19:56:19.717 DEBUG:boto:Token: None 2023-10-18T19:56:19.717 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.718 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY= 2023-10-18T19:56:19.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY=', 'Content-Length': '0'} 2023-10-18T19:56:19.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beafa84381e3fecb-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.721 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.722 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.722 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.722 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.723 DEBUG:boto:Method: GET 2023-10-18T19:56:19.723 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.724 DEBUG:boto:Data: 2023-10-18T19:56:19.724 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.725 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.725 DEBUG:boto:Params: {} 2023-10-18T19:56:19.726 DEBUG:boto:Token: None 2023-10-18T19:56:19.726 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.727 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6a1951d5a06b015-0065303863-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.730 DEBUG:boto:b'bofrnf-11obj-a11000falseobj-a12023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.731 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.731 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.732 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.732 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.733 DEBUG:boto:Method: GET 2023-10-18T19:56:19.733 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.734 DEBUG:boto:Data: 2023-10-18T19:56:19.734 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.734 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.735 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.735 DEBUG:boto:Params: {} 2023-10-18T19:56:19.736 DEBUG:boto:Token: None 2023-10-18T19:56:19.736 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9098fda4163e42f-0065303863-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.740 DEBUG:boto:b'bofrnf-11obj-a11000falseobj-a12023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.741 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.741 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.742 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.742 DEBUG:boto:Path: /bofrnf-11/obj-a1 2023-10-18T19:56:19.743 DEBUG:boto:Data: 2023-10-18T19:56:19.743 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.743 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.744 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.744 DEBUG:boto:Params: {} 2023-10-18T19:56:19.745 DEBUG:boto:Token: None 2023-10-18T19:56:19.745 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1 2023-10-18T19:56:19.746 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OK7s/J4kR+fVyTewpCDjxApmnhM= 2023-10-18T19:56:19.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OK7s/J4kR+fVyTewpCDjxApmnhM=', 'Content-Length': '0'} 2023-10-18T19:56:19.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098af01e63091e849-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.749 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.749 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.750 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.750 DEBUG:boto:Path: /bofrnf-11/obj-a1 2023-10-18T19:56:19.751 DEBUG:boto:Data: 2023-10-18T19:56:19.751 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.752 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.752 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.752 DEBUG:boto:Params: {} 2023-10-18T19:56:19.753 DEBUG:boto:Token: None 2023-10-18T19:56:19.753 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1 2023-10-18T19:56:19.754 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OK7s/J4kR+fVyTewpCDjxApmnhM= 2023-10-18T19:56:19.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OK7s/J4kR+fVyTewpCDjxApmnhM=', 'Content-Length': '0'} 2023-10-18T19:56:19.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000311221647654ca2e-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:56:19.758 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.758 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.759 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.759 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.759 DEBUG:boto:Method: GET 2023-10-18T19:56:19.760 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.760 DEBUG:boto:Data: 2023-10-18T19:56:19.761 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.762 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.762 DEBUG:boto:Params: {} 2023-10-18T19:56:19.763 DEBUG:boto:Token: None 2023-10-18T19:56:19.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY= 2023-10-18T19:56:19.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY=', 'Content-Length': '0'} 2023-10-18T19:56:19.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057df5f9db480de67-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.767 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.767 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:19.768 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.768 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.769 DEBUG:boto:Method: GET 2023-10-18T19:56:19.769 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.769 DEBUG:boto:Data: 2023-10-18T19:56:19.770 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.771 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.771 DEBUG:boto:Params: {} 2023-10-18T19:56:19.772 DEBUG:boto:Token: None 2023-10-18T19:56:19.772 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:56:19.773 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY= 2023-10-18T19:56:19.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBjVoZZc0CKqxE7SgNh84KUa4xY=', 'Content-Length': '0'} 2023-10-18T19:56:19.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f697f59a8a9edd3c-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.818 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.819 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.819 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.820 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.820 DEBUG:boto:Method: GET 2023-10-18T19:56:19.821 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.821 DEBUG:boto:Data: 2023-10-18T19:56:19.822 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.822 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.822 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.823 DEBUG:boto:Params: {} 2023-10-18T19:56:19.823 DEBUG:boto:Token: None 2023-10-18T19:56:19.824 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000089227a4c488ed7d-0065303863-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.828 DEBUG:boto:b'bofrnf-11obj-a11000falseobj-a12023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.829 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.829 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.830 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.830 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.830 DEBUG:boto:Method: GET 2023-10-18T19:56:19.831 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a1 2023-10-18T19:56:19.831 DEBUG:boto:Data: 2023-10-18T19:56:19.832 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.832 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.833 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.833 DEBUG:boto:Params: {} 2023-10-18T19:56:19.833 DEBUG:boto:Token: None 2023-10-18T19:56:19.834 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002de06c4f9a4cafd5-0065303863-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.838 DEBUG:boto:b'bofrnf-11obj-a11000falseobj-a12023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:56:19.839 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.839 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.840 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.840 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.840 DEBUG:boto:Method: GET 2023-10-18T19:56:19.841 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.841 DEBUG:boto:Data: 2023-10-18T19:56:19.842 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.842 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.843 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.843 DEBUG:boto:Params: {} 2023-10-18T19:56:19.844 DEBUG:boto:Token: None 2023-10-18T19:56:19.844 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.845 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc= 2023-10-18T19:56:19.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc=', 'Content-Length': '0'} 2023-10-18T19:56:19.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae3745e2a4bfdadc-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.848 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.848 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.849 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.849 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.849 DEBUG:boto:Method: GET 2023-10-18T19:56:19.850 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.850 DEBUG:boto:Data: 2023-10-18T19:56:19.851 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.851 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.851 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.852 DEBUG:boto:Params: {} 2023-10-18T19:56:19.852 DEBUG:boto:Token: None 2023-10-18T19:56:19.853 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.853 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc= 2023-10-18T19:56:19.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc=', 'Content-Length': '0'} 2023-10-18T19:56:19.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009aa575c1df18ba3f-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.857 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.857 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.857 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.858 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.858 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.859 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.859 DEBUG:boto:Data: 2023-10-18T19:56:19.859 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.860 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.860 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.861 DEBUG:boto:Params: {} 2023-10-18T19:56:19.861 DEBUG:boto:Token: None 2023-10-18T19:56:19.861 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d2qaxQ6jew3nKxsCPmtbBO/YKfY= 2023-10-18T19:56:19.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d2qaxQ6jew3nKxsCPmtbBO/YKfY=', 'Content-Length': '0'} 2023-10-18T19:56:19.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5a3619edc320096-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.865 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.865 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.866 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.866 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.866 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.867 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.867 DEBUG:boto:Data: 2023-10-18T19:56:19.868 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.868 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.869 DEBUG:boto:Params: {} 2023-10-18T19:56:19.869 DEBUG:boto:Token: None 2023-10-18T19:56:19.870 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.870 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d2qaxQ6jew3nKxsCPmtbBO/YKfY= 2023-10-18T19:56:19.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d2qaxQ6jew3nKxsCPmtbBO/YKfY=', 'Content-Length': '0'} 2023-10-18T19:56:19.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076117c8993ca1d98-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:56:19.873 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.874 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.874 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.875 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.875 DEBUG:boto:Method: GET 2023-10-18T19:56:19.876 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.876 DEBUG:boto:Data: 2023-10-18T19:56:19.876 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.877 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.878 DEBUG:boto:Params: {} 2023-10-18T19:56:19.878 DEBUG:boto:Token: None 2023-10-18T19:56:19.879 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.879 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc= 2023-10-18T19:56:19.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc=', 'Content-Length': '0'} 2023-10-18T19:56:19.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000edd6e2b9b626bf8-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.922 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.923 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.924 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.924 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.924 DEBUG:boto:Method: GET 2023-10-18T19:56:19.925 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.925 DEBUG:boto:Data: 2023-10-18T19:56:19.926 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.926 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.927 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.927 DEBUG:boto:Params: {} 2023-10-18T19:56:19.928 DEBUG:boto:Token: None 2023-10-18T19:56:19.928 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc= 2023-10-18T19:56:19.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc=', 'Content-Length': '0'} 2023-10-18T19:56:19.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000546f3fb4d5c36686-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.932 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.932 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.933 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:19.933 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:19.934 DEBUG:boto:Method: GET 2023-10-18T19:56:19.934 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:19.934 DEBUG:boto:Data: 2023-10-18T19:56:19.935 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.936 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.936 DEBUG:boto:Params: {} 2023-10-18T19:56:19.937 DEBUG:boto:Token: None 2023-10-18T19:56:19.937 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.938 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d001a99cfa6dab0-0065303863-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.941 DEBUG:boto:b'bofrnf-11obj-a21000falseobj-a22023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.942 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:19.942 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:19.943 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:19.943 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:19.944 DEBUG:boto:Method: GET 2023-10-18T19:56:19.944 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:19.944 DEBUG:boto:Data: 2023-10-18T19:56:19.945 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.946 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.946 DEBUG:boto:Params: {} 2023-10-18T19:56:19.946 DEBUG:boto:Token: None 2023-10-18T19:56:19.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/ 2023-10-18T19:56:19.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI= 2023-10-18T19:56:19.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+0hXREnHYAzyd0h9EbgUNBqkoI=', 'Content-Length': '0'} 2023-10-18T19:56:19.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000199ce61a311a6f15-0065303863-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.951 DEBUG:boto:b'bofrnf-11obj-a21000falseobj-a22023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:19.952 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.952 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.953 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.953 DEBUG:boto:Path: /bofrnf-11/obj-a2 2023-10-18T19:56:19.953 DEBUG:boto:Data: 2023-10-18T19:56:19.954 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.954 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.955 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.955 DEBUG:boto:Params: {} 2023-10-18T19:56:19.955 DEBUG:boto:Token: None 2023-10-18T19:56:19.956 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2 2023-10-18T19:56:19.956 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OcY1tLsuS484eq9vAfye36ed4rw= 2023-10-18T19:56:19.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OcY1tLsuS484eq9vAfye36ed4rw=', 'Content-Length': '0'} 2023-10-18T19:56:19.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b057a4f952c41eb1-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.960 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.960 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.961 DEBUG:boto:Method: HEAD 2023-10-18T19:56:19.961 DEBUG:boto:Path: /bofrnf-11/obj-a2 2023-10-18T19:56:19.962 DEBUG:boto:Data: 2023-10-18T19:56:19.962 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.963 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.963 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.963 DEBUG:boto:Params: {} 2023-10-18T19:56:19.964 DEBUG:boto:Token: None 2023-10-18T19:56:19.964 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2 2023-10-18T19:56:19.965 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OcY1tLsuS484eq9vAfye36ed4rw= 2023-10-18T19:56:19.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OcY1tLsuS484eq9vAfye36ed4rw=', 'Content-Length': '0'} 2023-10-18T19:56:19.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d367c226b45b7d9c-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:56:19.968 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.969 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.969 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.970 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.970 DEBUG:boto:Method: GET 2023-10-18T19:56:19.971 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.971 DEBUG:boto:Data: 2023-10-18T19:56:19.971 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.972 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:19.972 DEBUG:boto:Port: 8001 2023-10-18T19:56:19.973 DEBUG:boto:Params: {} 2023-10-18T19:56:19.973 DEBUG:boto:Token: None 2023-10-18T19:56:19.974 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.974 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc= 2023-10-18T19:56:19.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc=', 'Content-Length': '0'} 2023-10-18T19:56:19.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acef43d6ece35fb9-0065303863-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:19.977 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.978 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:19.978 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.979 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.979 DEBUG:boto:Method: GET 2023-10-18T19:56:19.980 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.980 DEBUG:boto:Data: 2023-10-18T19:56:19.980 DEBUG:boto:Headers: {} 2023-10-18T19:56:19.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:19.981 DEBUG:boto:Port: 8000 2023-10-18T19:56:19.982 DEBUG:boto:Params: {} 2023-10-18T19:56:19.982 DEBUG:boto:Token: None 2023-10-18T19:56:19.983 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:19 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:56:19.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc= 2023-10-18T19:56:19.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fzrdmvhccUyAtra7HaYJ/NgbLXc=', 'Content-Length': '0'} 2023-10-18T19:56:19.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be5a3f5b3484f364-0065303863-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:20.026 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:20.027 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:20.027 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:20.028 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:20.028 DEBUG:boto:Method: GET 2023-10-18T19:56:20.029 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:20.029 DEBUG:boto:Data: 2023-10-18T19:56:20.030 DEBUG:boto:Headers: {} 2023-10-18T19:56:20.030 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:20.031 DEBUG:boto:Port: 8001 2023-10-18T19:56:20.031 DEBUG:boto:Params: {} 2023-10-18T19:56:20.031 DEBUG:boto:Token: None 2023-10-18T19:56:20.032 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:20 GMT /bofrnf-11/ 2023-10-18T19:56:20.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ykpn5ta3GqooF6JTSDK8dasxtHE= 2023-10-18T19:56:20.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ykpn5ta3GqooF6JTSDK8dasxtHE=', 'Content-Length': '0'} 2023-10-18T19:56:20.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002146dc56613c5849-0065303864-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:20.036 DEBUG:boto:b'bofrnf-11obj-a21000falseobj-a22023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:20.037 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:20.037 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:20.038 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:20.038 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:20.038 DEBUG:boto:Method: GET 2023-10-18T19:56:20.039 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a2 2023-10-18T19:56:20.039 DEBUG:boto:Data: 2023-10-18T19:56:20.040 DEBUG:boto:Headers: {} 2023-10-18T19:56:20.040 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:20.041 DEBUG:boto:Port: 8000 2023-10-18T19:56:20.041 DEBUG:boto:Params: {} 2023-10-18T19:56:20.041 DEBUG:boto:Token: None 2023-10-18T19:56:20.042 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:20 GMT /bofrnf-11/ 2023-10-18T19:56:20.042 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ykpn5ta3GqooF6JTSDK8dasxtHE= 2023-10-18T19:56:20.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ykpn5ta3GqooF6JTSDK8dasxtHE=', 'Content-Length': '0'} 2023-10-18T19:56:20.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acda6616396182d9-0065303864-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:20.046 DEBUG:boto:b'bofrnf-11obj-a21000falseobj-a22023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:20.047 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-11 zones={a2, a1} 2023-10-18T19:56:20.047 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-12 2023-10-18T19:56:20.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:56:20.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:56:20.049 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:56:20.114 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:56:20.114 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:56:20.151 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:20.150+0000 7fac2f5ab780 1 bucket sync caught up with source: 2023-10-18T19:56:20.151 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.8.11, , , 00000000002.6.11, 00000000010.23.6, ] 2023-10-18T19:56:20.152 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.8.11, , , 00000000002.6.11, 00000000010.23.6, ] 2023-10-18T19:56:20.152 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:20.150+0000 7fac2f5ab780 0 bucket checkpoint complete 2023-10-18T19:56:20.164 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-12 2023-10-18T19:56:20.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:56:20.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:56:20.165 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:56:20.222 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:56:20.222 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:56:20.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:20.254+0000 7fa4a6318780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:20.256 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.27.10, , , 00000000010.34.6, , ] 2023-10-18T19:56:20.257 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000010.34.6, 00000000002.15.11, ] 2023-10-18T19:56:25.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:25.256+0000 7fa4a6318780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:25.258 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.27.10, , , 00000000010.34.6, , ] 2023-10-18T19:56:25.258 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000010.34.6, 00000000002.15.11, ] 2023-10-18T19:56:30.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:30.257+0000 7fa4a6318780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:30.259 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.27.10, , , 00000000010.34.6, , ] 2023-10-18T19:56:30.260 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000010.34.6, 00000000002.15.11, ] 2023-10-18T19:56:35.260 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:35.258+0000 7fa4a6318780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:35.260 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.27.10, , , 00000000010.34.6, , ] 2023-10-18T19:56:35.261 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000010.34.6, 00000000002.15.11, ] 2023-10-18T19:56:40.261 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:40.259+0000 7fa4a6318780 1 bucket sync caught up with source: 2023-10-18T19:56:40.262 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000010.34.6, 00000000002.15.11, ] 2023-10-18T19:56:40.262 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000010.34.6, 00000000002.15.11, ] 2023-10-18T19:56:40.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:40.259+0000 7fa4a6318780 0 bucket checkpoint complete 2023-10-18T19:56:40.276 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-12 zones={a2, a1} 2023-10-18T19:56:40.277 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.277 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.278 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.278 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:56:40.279 DEBUG:boto:Data: 2023-10-18T19:56:40.279 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.280 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.280 DEBUG:boto:Params: {} 2023-10-18T19:56:40.281 DEBUG:boto:Token: None 2023-10-18T19:56:40.281 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.281 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G2ouc01yDZ+CO3l8Bpgf1akWSpA= 2023-10-18T19:56:40.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G2ouc01yDZ+CO3l8Bpgf1akWSpA=', 'Content-Length': '0'} 2023-10-18T19:56:40.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a194fe867352d7a9-0065303878-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.285 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.286 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.286 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.286 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:56:40.287 DEBUG:boto:Data: 2023-10-18T19:56:40.287 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.288 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.288 DEBUG:boto:Params: {} 2023-10-18T19:56:40.289 DEBUG:boto:Token: None 2023-10-18T19:56:40.289 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.290 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G2ouc01yDZ+CO3l8Bpgf1akWSpA= 2023-10-18T19:56:40.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G2ouc01yDZ+CO3l8Bpgf1akWSpA=', 'Content-Length': '0'} 2023-10-18T19:56:40.293 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ae3f23c4ad90629-0065303878-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.293 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:56:40.294 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.294 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.294 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.295 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.295 DEBUG:boto:Method: GET 2023-10-18T19:56:40.296 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.296 DEBUG:boto:Data: 2023-10-18T19:56:40.297 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.297 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.298 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.298 DEBUG:boto:Params: {} 2023-10-18T19:56:40.298 DEBUG:boto:Token: None 2023-10-18T19:56:40.299 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/?versions 2023-10-18T19:56:40.299 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lFYIwWMk22tXkoONf7Xw2QJEllI= 2023-10-18T19:56:40.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lFYIwWMk22tXkoONf7Xw2QJEllI=', 'Content-Length': '0'} 2023-10-18T19:56:40.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d355ad3dee6e372f-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.303 DEBUG:boto:b'bofrnf-12999falsedummyytzJEO2gslxPWNYVg6CTpsims4XGqOgtrue2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1toth3ilqud.qHjmYoshz5E--Nxv2rQYtrue2023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2yipaUQxzrtbVwuxaybttOFpenBTsbAbtrue2023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.304 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:56:40.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-18T19:56:40.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-18T19:56:40.305 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:56:40.305 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.306 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.306 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.307 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.307 DEBUG:boto:Method: GET 2023-10-18T19:56:40.308 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.308 DEBUG:boto:Data: 2023-10-18T19:56:40.308 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.309 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.309 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.310 DEBUG:boto:Params: {} 2023-10-18T19:56:40.310 DEBUG:boto:Token: None 2023-10-18T19:56:40.311 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/?versions 2023-10-18T19:56:40.311 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lFYIwWMk22tXkoONf7Xw2QJEllI= 2023-10-18T19:56:40.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lFYIwWMk22tXkoONf7Xw2QJEllI=', 'Content-Length': '0'} 2023-10-18T19:56:40.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b02ef386f43bf3f-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.315 DEBUG:boto:b'bofrnf-12999falsedummyytzJEO2gslxPWNYVg6CTpsims4XGqOgtrue2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1toth3ilqud.qHjmYoshz5E--Nxv2rQYtrue2023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2yipaUQxzrtbVwuxaybttOFpenBTsbAbtrue2023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.315 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:56:40.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-18T19:56:40.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-18T19:56:40.316 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.317 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.317 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.318 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.318 DEBUG:boto:Method: GET 2023-10-18T19:56:40.319 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.319 DEBUG:boto:Data: 2023-10-18T19:56:40.319 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.320 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.320 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.321 DEBUG:boto:Params: {} 2023-10-18T19:56:40.321 DEBUG:boto:Token: None 2023-10-18T19:56:40.322 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/?versions 2023-10-18T19:56:40.322 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lFYIwWMk22tXkoONf7Xw2QJEllI= 2023-10-18T19:56:40.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lFYIwWMk22tXkoONf7Xw2QJEllI=', 'Content-Length': '0'} 2023-10-18T19:56:40.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000418efc5a1e7849db-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.326 DEBUG:boto:b'bofrnf-12999falsedummyytzJEO2gslxPWNYVg6CTpsims4XGqOgtrue2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1toth3ilqud.qHjmYoshz5E--Nxv2rQYtrue2023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2yipaUQxzrtbVwuxaybttOFpenBTsbAbtrue2023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.326 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.327 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.327 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.328 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.328 DEBUG:boto:Method: GET 2023-10-18T19:56:40.329 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999 2023-10-18T19:56:40.329 DEBUG:boto:Data: 2023-10-18T19:56:40.329 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.330 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.330 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.331 DEBUG:boto:Params: {} 2023-10-18T19:56:40.331 DEBUG:boto:Token: None 2023-10-18T19:56:40.332 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/?versions 2023-10-18T19:56:40.332 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lFYIwWMk22tXkoONf7Xw2QJEllI= 2023-10-18T19:56:40.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lFYIwWMk22tXkoONf7Xw2QJEllI=', 'Content-Length': '0'} 2023-10-18T19:56:40.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d90f7c5a2b1fb0cc-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.336 DEBUG:boto:b'bofrnf-12999falsedummyytzJEO2gslxPWNYVg6CTpsims4XGqOgtrue2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1toth3ilqud.qHjmYoshz5E--Nxv2rQYtrue2023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2yipaUQxzrtbVwuxaybttOFpenBTsbAbtrue2023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:56:40.337 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.337 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.338 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.338 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.339 DEBUG:boto:Method: GET 2023-10-18T19:56:40.339 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.340 DEBUG:boto:Data: 2023-10-18T19:56:40.340 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.340 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.341 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.341 DEBUG:boto:Params: {} 2023-10-18T19:56:40.342 DEBUG:boto:Token: None 2023-10-18T19:56:40.342 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.343 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ= 2023-10-18T19:56:40.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009240f6d50518ba21-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.346 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.347 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.347 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.348 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.348 DEBUG:boto:Method: GET 2023-10-18T19:56:40.349 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.349 DEBUG:boto:Data: 2023-10-18T19:56:40.350 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.350 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.351 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.351 DEBUG:boto:Params: {} 2023-10-18T19:56:40.352 DEBUG:boto:Token: None 2023-10-18T19:56:40.352 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.353 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ= 2023-10-18T19:56:40.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.355 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000428164536fbec3cd-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.356 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.356 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.357 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.357 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.358 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.358 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.358 DEBUG:boto:Data: 2023-10-18T19:56:40.359 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.359 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.360 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.360 DEBUG:boto:Params: {} 2023-10-18T19:56:40.361 DEBUG:boto:Token: None 2023-10-18T19:56:40.361 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.362 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UsdZiIX/3dRiVMJ0W9pX/JSjncg= 2023-10-18T19:56:40.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UsdZiIX/3dRiVMJ0W9pX/JSjncg=', 'Content-Length': '0'} 2023-10-18T19:56:40.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002de7e93a71630ccb-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.365 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.365 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.366 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.366 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.367 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.367 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.368 DEBUG:boto:Data: 2023-10-18T19:56:40.368 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.368 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.369 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.369 DEBUG:boto:Params: {} 2023-10-18T19:56:40.370 DEBUG:boto:Token: None 2023-10-18T19:56:40.370 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.371 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UsdZiIX/3dRiVMJ0W9pX/JSjncg= 2023-10-18T19:56:40.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UsdZiIX/3dRiVMJ0W9pX/JSjncg=', 'Content-Length': '0'} 2023-10-18T19:56:40.373 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000badf2904b6823751-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:56:40.374 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.374 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.375 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.375 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.375 DEBUG:boto:Method: GET 2023-10-18T19:56:40.376 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.376 DEBUG:boto:Data: 2023-10-18T19:56:40.377 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.378 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.378 DEBUG:boto:Params: {} 2023-10-18T19:56:40.379 DEBUG:boto:Token: None 2023-10-18T19:56:40.379 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.380 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ= 2023-10-18T19:56:40.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080d1e4ad24b2f0af-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.383 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.383 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.383 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.384 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.384 DEBUG:boto:Method: GET 2023-10-18T19:56:40.385 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.385 DEBUG:boto:Data: 2023-10-18T19:56:40.386 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.387 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.387 DEBUG:boto:Params: {} 2023-10-18T19:56:40.388 DEBUG:boto:Token: None 2023-10-18T19:56:40.388 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.389 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ= 2023-10-18T19:56:40.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000349105a2e7cec355-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.392 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.392 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.393 DEBUG:boto:path=/bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.393 DEBUG:boto:auth_path=/bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.394 DEBUG:boto:Method: GET 2023-10-18T19:56:40.394 DEBUG:boto:Path: /bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.395 DEBUG:boto:Data: 2023-10-18T19:56:40.395 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.395 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.396 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.396 DEBUG:boto:Params: {} 2023-10-18T19:56:40.397 DEBUG:boto:Token: None 2023-10-18T19:56:40.397 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.398 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa4dc37f05873b96-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.446 DEBUG:boto:b'bofrnf-12dummy1000falsedummy2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.447 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.447 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.448 DEBUG:boto:path=/bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.448 DEBUG:boto:auth_path=/bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.449 DEBUG:boto:Method: GET 2023-10-18T19:56:40.449 DEBUG:boto:Path: /bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.449 DEBUG:boto:Data: 2023-10-18T19:56:40.450 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.450 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.451 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.451 DEBUG:boto:Params: {} 2023-10-18T19:56:40.452 DEBUG:boto:Token: None 2023-10-18T19:56:40.452 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.453 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011cf138b9cdc5cbd-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.456 DEBUG:boto:b'bofrnf-12dummy1000falsedummy2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.457 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.457 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.458 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.458 DEBUG:boto:Path: /bofrnf-12/dummy 2023-10-18T19:56:40.458 DEBUG:boto:Data: 2023-10-18T19:56:40.459 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.459 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.460 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.460 DEBUG:boto:Params: {} 2023-10-18T19:56:40.460 DEBUG:boto:Token: None 2023-10-18T19:56:40.461 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy 2023-10-18T19:56:40.461 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5E/29Z8kdX/gq+GXqHoz1WpfyKw= 2023-10-18T19:56:40.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5E/29Z8kdX/gq+GXqHoz1WpfyKw=', 'Content-Length': '0'} 2023-10-18T19:56:40.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0720a14bc221930-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.465 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.465 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.466 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.466 DEBUG:boto:Path: /bofrnf-12/dummy 2023-10-18T19:56:40.467 DEBUG:boto:Data: 2023-10-18T19:56:40.467 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.467 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.468 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.468 DEBUG:boto:Params: {} 2023-10-18T19:56:40.469 DEBUG:boto:Token: None 2023-10-18T19:56:40.469 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy 2023-10-18T19:56:40.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5E/29Z8kdX/gq+GXqHoz1WpfyKw= 2023-10-18T19:56:40.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5E/29Z8kdX/gq+GXqHoz1WpfyKw=', 'Content-Length': '0'} 2023-10-18T19:56:40.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000151a9527b402ecdc-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:56:40.473 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.474 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.474 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.475 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.475 DEBUG:boto:Method: GET 2023-10-18T19:56:40.475 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.476 DEBUG:boto:Data: 2023-10-18T19:56:40.476 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.477 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.477 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.478 DEBUG:boto:Params: {} 2023-10-18T19:56:40.478 DEBUG:boto:Token: None 2023-10-18T19:56:40.479 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.479 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ= 2023-10-18T19:56:40.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d9c9ce873911733-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.482 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:56:40.483 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:56:40.483 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.484 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.484 DEBUG:boto:Method: GET 2023-10-18T19:56:40.484 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.485 DEBUG:boto:Data: 2023-10-18T19:56:40.485 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.486 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.486 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.486 DEBUG:boto:Params: {} 2023-10-18T19:56:40.487 DEBUG:boto:Token: None 2023-10-18T19:56:40.487 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:56:40.488 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ= 2023-10-18T19:56:40.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2yHjU2XG07H3wpyk8OQ6t1SMTjQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c9bacef4e7eda50-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.491 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.491 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.492 DEBUG:boto:path=/bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.492 DEBUG:boto:auth_path=/bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.492 DEBUG:boto:Method: GET 2023-10-18T19:56:40.493 DEBUG:boto:Path: /bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.493 DEBUG:boto:Data: 2023-10-18T19:56:40.493 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.494 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.495 DEBUG:boto:Params: {} 2023-10-18T19:56:40.495 DEBUG:boto:Token: None 2023-10-18T19:56:40.495 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.496 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007af00a2c5e9a000-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.500 DEBUG:boto:b'bofrnf-12dummy1000falsedummy2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.500 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.500 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.501 DEBUG:boto:path=/bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.501 DEBUG:boto:auth_path=/bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.502 DEBUG:boto:Method: GET 2023-10-18T19:56:40.502 DEBUG:boto:Path: /bofrnf-12/?prefix=dummy 2023-10-18T19:56:40.502 DEBUG:boto:Data: 2023-10-18T19:56:40.503 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.504 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.504 DEBUG:boto:Params: {} 2023-10-18T19:56:40.504 DEBUG:boto:Token: None 2023-10-18T19:56:40.505 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e3816d86fe7c4ba-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.509 DEBUG:boto:b'bofrnf-12dummy1000falsedummy2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:56:40.510 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.511 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.511 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.511 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.512 DEBUG:boto:Method: GET 2023-10-18T19:56:40.512 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.512 DEBUG:boto:Data: 2023-10-18T19:56:40.513 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.513 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.514 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.514 DEBUG:boto:Params: {} 2023-10-18T19:56:40.515 DEBUG:boto:Token: None 2023-10-18T19:56:40.515 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg= 2023-10-18T19:56:40.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg=', 'Content-Length': '0'} 2023-10-18T19:56:40.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000326698a691c60a9f-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.519 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.520 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.520 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.520 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.521 DEBUG:boto:Method: GET 2023-10-18T19:56:40.521 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.522 DEBUG:boto:Data: 2023-10-18T19:56:40.522 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.523 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.523 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.523 DEBUG:boto:Params: {} 2023-10-18T19:56:40.524 DEBUG:boto:Token: None 2023-10-18T19:56:40.524 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.525 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg= 2023-10-18T19:56:40.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg=', 'Content-Length': '0'} 2023-10-18T19:56:40.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052ef98a1145ddd34-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.528 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.528 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.529 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.529 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.529 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.530 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.530 DEBUG:boto:Data: 2023-10-18T19:56:40.531 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.531 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.531 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.532 DEBUG:boto:Params: {} 2023-10-18T19:56:40.532 DEBUG:boto:Token: None 2023-10-18T19:56:40.533 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CbnSYVN6xXaB48Emdc9qGqBpJJo= 2023-10-18T19:56:40.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CbnSYVN6xXaB48Emdc9qGqBpJJo=', 'Content-Length': '0'} 2023-10-18T19:56:40.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004707897e638a87df-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.536 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.536 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.536 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.537 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.537 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.538 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.538 DEBUG:boto:Data: 2023-10-18T19:56:40.538 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.539 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.539 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.540 DEBUG:boto:Params: {} 2023-10-18T19:56:40.540 DEBUG:boto:Token: None 2023-10-18T19:56:40.540 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.541 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CbnSYVN6xXaB48Emdc9qGqBpJJo= 2023-10-18T19:56:40.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CbnSYVN6xXaB48Emdc9qGqBpJJo=', 'Content-Length': '0'} 2023-10-18T19:56:40.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003408fc8ca3d59ec8-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:56:40.544 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.544 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.545 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.545 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.545 DEBUG:boto:Method: GET 2023-10-18T19:56:40.546 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.546 DEBUG:boto:Data: 2023-10-18T19:56:40.547 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.547 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.548 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.548 DEBUG:boto:Params: {} 2023-10-18T19:56:40.548 DEBUG:boto:Token: None 2023-10-18T19:56:40.549 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.549 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg= 2023-10-18T19:56:40.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg=', 'Content-Length': '0'} 2023-10-18T19:56:40.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5d494ccb79a5fe6-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.598 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.599 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.599 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.600 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.600 DEBUG:boto:Method: GET 2023-10-18T19:56:40.600 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.601 DEBUG:boto:Data: 2023-10-18T19:56:40.601 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.602 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.603 DEBUG:boto:Params: {} 2023-10-18T19:56:40.603 DEBUG:boto:Token: None 2023-10-18T19:56:40.605 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.606 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg= 2023-10-18T19:56:40.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg=', 'Content-Length': '0'} 2023-10-18T19:56:40.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc1be4f2e63bf981-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.609 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.610 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.610 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.610 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.611 DEBUG:boto:Method: GET 2023-10-18T19:56:40.611 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.612 DEBUG:boto:Data: 2023-10-18T19:56:40.612 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.613 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.613 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.613 DEBUG:boto:Params: {} 2023-10-18T19:56:40.614 DEBUG:boto:Token: None 2023-10-18T19:56:40.614 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003873ccae1336f17f-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.619 DEBUG:boto:b'bofrnf-12obj-a11000falseobj-a12023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.619 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.620 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.620 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.620 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.621 DEBUG:boto:Method: GET 2023-10-18T19:56:40.621 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.622 DEBUG:boto:Data: 2023-10-18T19:56:40.622 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.623 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.623 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.624 DEBUG:boto:Params: {} 2023-10-18T19:56:40.624 DEBUG:boto:Token: None 2023-10-18T19:56:40.625 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.625 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000273ae1b37884c3ec-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.629 DEBUG:boto:b'bofrnf-12obj-a11000falseobj-a12023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.629 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.629 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.630 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.630 DEBUG:boto:Path: /bofrnf-12/obj-a1 2023-10-18T19:56:40.631 DEBUG:boto:Data: 2023-10-18T19:56:40.631 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.632 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.632 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.632 DEBUG:boto:Params: {} 2023-10-18T19:56:40.633 DEBUG:boto:Token: None 2023-10-18T19:56:40.633 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1 2023-10-18T19:56:40.634 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:al6eVyn2E95NtsKP269GnIV/W9M= 2023-10-18T19:56:40.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:al6eVyn2E95NtsKP269GnIV/W9M=', 'Content-Length': '0'} 2023-10-18T19:56:40.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe68b4431e867d1b-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.637 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.638 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.638 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.639 DEBUG:boto:Path: /bofrnf-12/obj-a1 2023-10-18T19:56:40.639 DEBUG:boto:Data: 2023-10-18T19:56:40.639 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.640 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.640 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.640 DEBUG:boto:Params: {} 2023-10-18T19:56:40.641 DEBUG:boto:Token: None 2023-10-18T19:56:40.641 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1 2023-10-18T19:56:40.642 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:al6eVyn2E95NtsKP269GnIV/W9M= 2023-10-18T19:56:40.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:al6eVyn2E95NtsKP269GnIV/W9M=', 'Content-Length': '0'} 2023-10-18T19:56:40.644 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006003ff124caf9bee-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:56:40.645 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.646 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.646 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.647 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.647 DEBUG:boto:Method: GET 2023-10-18T19:56:40.647 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.648 DEBUG:boto:Data: 2023-10-18T19:56:40.648 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.649 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.649 DEBUG:boto:Params: {} 2023-10-18T19:56:40.650 DEBUG:boto:Token: None 2023-10-18T19:56:40.650 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.651 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg= 2023-10-18T19:56:40.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg=', 'Content-Length': '0'} 2023-10-18T19:56:40.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0619000c8e69357-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.654 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.654 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:40.655 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.655 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.655 DEBUG:boto:Method: GET 2023-10-18T19:56:40.656 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.656 DEBUG:boto:Data: 2023-10-18T19:56:40.657 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.657 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.658 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.658 DEBUG:boto:Params: {} 2023-10-18T19:56:40.658 DEBUG:boto:Token: None 2023-10-18T19:56:40.659 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:56:40.659 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg= 2023-10-18T19:56:40.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2JpX3Enopt0noUAP7LWULlaPSUg=', 'Content-Length': '0'} 2023-10-18T19:56:40.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ec4a01102c05fb7-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.663 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.663 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.663 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.664 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.664 DEBUG:boto:Method: GET 2023-10-18T19:56:40.665 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.665 DEBUG:boto:Data: 2023-10-18T19:56:40.665 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.666 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.666 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.667 DEBUG:boto:Params: {} 2023-10-18T19:56:40.667 DEBUG:boto:Token: None 2023-10-18T19:56:40.668 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006000ca1827a65da7-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.672 DEBUG:boto:b'bofrnf-12obj-a11000falseobj-a12023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.672 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.673 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.673 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.674 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.674 DEBUG:boto:Method: GET 2023-10-18T19:56:40.674 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a1 2023-10-18T19:56:40.675 DEBUG:boto:Data: 2023-10-18T19:56:40.675 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.676 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.676 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.676 DEBUG:boto:Params: {} 2023-10-18T19:56:40.677 DEBUG:boto:Token: None 2023-10-18T19:56:40.677 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.678 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007218503b6c18679-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.681 DEBUG:boto:b'bofrnf-12obj-a11000falseobj-a12023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:56:40.682 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.683 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.683 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.684 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.684 DEBUG:boto:Method: GET 2023-10-18T19:56:40.684 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.685 DEBUG:boto:Data: 2023-10-18T19:56:40.685 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.685 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.686 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.686 DEBUG:boto:Params: {} 2023-10-18T19:56:40.687 DEBUG:boto:Token: None 2023-10-18T19:56:40.687 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY= 2023-10-18T19:56:40.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY=', 'Content-Length': '0'} 2023-10-18T19:56:40.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000696bd3a4a67a0f6d-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.734 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.735 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.735 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.736 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.736 DEBUG:boto:Method: GET 2023-10-18T19:56:40.736 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.737 DEBUG:boto:Data: 2023-10-18T19:56:40.737 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.738 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.739 DEBUG:boto:Params: {} 2023-10-18T19:56:40.739 DEBUG:boto:Token: None 2023-10-18T19:56:40.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY= 2023-10-18T19:56:40.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY=', 'Content-Length': '0'} 2023-10-18T19:56:40.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034a7565bd990c3e6-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.743 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.744 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.744 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.745 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.745 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.745 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.746 DEBUG:boto:Data: 2023-10-18T19:56:40.746 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.747 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.747 DEBUG:boto:Params: {} 2023-10-18T19:56:40.748 DEBUG:boto:Token: None 2023-10-18T19:56:40.748 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SsWg3wDhGkBo8+mfAhf75ZWIh5g= 2023-10-18T19:56:40.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SsWg3wDhGkBo8+mfAhf75ZWIh5g=', 'Content-Length': '0'} 2023-10-18T19:56:40.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ed68fb59590c26d-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.751 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.752 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.752 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.753 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.753 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.753 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.754 DEBUG:boto:Data: 2023-10-18T19:56:40.754 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.755 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.755 DEBUG:boto:Params: {} 2023-10-18T19:56:40.755 DEBUG:boto:Token: None 2023-10-18T19:56:40.756 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SsWg3wDhGkBo8+mfAhf75ZWIh5g= 2023-10-18T19:56:40.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SsWg3wDhGkBo8+mfAhf75ZWIh5g=', 'Content-Length': '0'} 2023-10-18T19:56:40.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2da7e6a61dc8377-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:56:40.760 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.760 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.760 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.761 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.761 DEBUG:boto:Method: GET 2023-10-18T19:56:40.761 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.762 DEBUG:boto:Data: 2023-10-18T19:56:40.762 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.763 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.764 DEBUG:boto:Params: {} 2023-10-18T19:56:40.764 DEBUG:boto:Token: None 2023-10-18T19:56:40.765 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.765 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY= 2023-10-18T19:56:40.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY=', 'Content-Length': '0'} 2023-10-18T19:56:40.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000629421344bd0e263-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.768 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.769 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.769 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.770 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.770 DEBUG:boto:Method: GET 2023-10-18T19:56:40.770 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.771 DEBUG:boto:Data: 2023-10-18T19:56:40.771 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.772 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.772 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.773 DEBUG:boto:Params: {} 2023-10-18T19:56:40.773 DEBUG:boto:Token: None 2023-10-18T19:56:40.773 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY= 2023-10-18T19:56:40.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY=', 'Content-Length': '0'} 2023-10-18T19:56:40.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a6a9ed1b217bb73-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.818 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.819 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.819 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.820 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.820 DEBUG:boto:Method: GET 2023-10-18T19:56:40.820 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.821 DEBUG:boto:Data: 2023-10-18T19:56:40.821 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.822 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.822 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.823 DEBUG:boto:Params: {} 2023-10-18T19:56:40.823 DEBUG:boto:Token: None 2023-10-18T19:56:40.824 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e015093b89d1526-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.828 DEBUG:boto:b'bofrnf-12obj-a21000falseobj-a22023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.828 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.829 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.829 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.830 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.830 DEBUG:boto:Method: GET 2023-10-18T19:56:40.830 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.831 DEBUG:boto:Data: 2023-10-18T19:56:40.831 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.832 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.832 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.832 DEBUG:boto:Params: {} 2023-10-18T19:56:40.833 DEBUG:boto:Token: None 2023-10-18T19:56:40.833 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eefa73b5bd9f1a3a-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.837 DEBUG:boto:b'bofrnf-12obj-a21000falseobj-a22023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.838 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.839 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.839 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.839 DEBUG:boto:Path: /bofrnf-12/obj-a2 2023-10-18T19:56:40.840 DEBUG:boto:Data: 2023-10-18T19:56:40.840 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.841 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.841 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.841 DEBUG:boto:Params: {} 2023-10-18T19:56:40.842 DEBUG:boto:Token: None 2023-10-18T19:56:40.842 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2 2023-10-18T19:56:40.843 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kJVk8bK+NGhNWd5KKyFes2a6pEk= 2023-10-18T19:56:40.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kJVk8bK+NGhNWd5KKyFes2a6pEk=', 'Content-Length': '0'} 2023-10-18T19:56:40.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b5b1b64a75b0851-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.847 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.848 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.848 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.848 DEBUG:boto:Path: /bofrnf-12/obj-a2 2023-10-18T19:56:40.849 DEBUG:boto:Data: 2023-10-18T19:56:40.849 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.850 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.850 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.850 DEBUG:boto:Params: {} 2023-10-18T19:56:40.851 DEBUG:boto:Token: None 2023-10-18T19:56:40.851 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2 2023-10-18T19:56:40.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kJVk8bK+NGhNWd5KKyFes2a6pEk= 2023-10-18T19:56:40.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kJVk8bK+NGhNWd5KKyFes2a6pEk=', 'Content-Length': '0'} 2023-10-18T19:56:40.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfa87717bba6d8c8-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:56:40.856 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.856 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.857 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.857 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.858 DEBUG:boto:Method: GET 2023-10-18T19:56:40.858 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.858 DEBUG:boto:Data: 2023-10-18T19:56:40.859 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.860 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.860 DEBUG:boto:Params: {} 2023-10-18T19:56:40.861 DEBUG:boto:Token: None 2023-10-18T19:56:40.861 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.861 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY= 2023-10-18T19:56:40.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY=', 'Content-Length': '0'} 2023-10-18T19:56:40.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016d7ae67f897fb24-0065303878-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.906 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.907 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:40.907 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.908 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.908 DEBUG:boto:Method: GET 2023-10-18T19:56:40.908 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.909 DEBUG:boto:Data: 2023-10-18T19:56:40.909 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.910 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.910 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.910 DEBUG:boto:Params: {} 2023-10-18T19:56:40.911 DEBUG:boto:Token: None 2023-10-18T19:56:40.911 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:56:40.912 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY= 2023-10-18T19:56:40.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9TB+su1C0mGbtONGQJDxuxgqCMY=', 'Content-Length': '0'} 2023-10-18T19:56:40.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dad0ec7fa96d4df0-0065303878-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.915 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.915 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.916 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.916 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.917 DEBUG:boto:Method: GET 2023-10-18T19:56:40.917 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.918 DEBUG:boto:Data: 2023-10-18T19:56:40.918 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.919 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.919 DEBUG:boto:Params: {} 2023-10-18T19:56:40.920 DEBUG:boto:Token: None 2023-10-18T19:56:40.920 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.921 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8f854124ad133e4-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.924 DEBUG:boto:b'bofrnf-12obj-a21000falseobj-a22023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.925 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.925 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.926 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.926 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.926 DEBUG:boto:Method: GET 2023-10-18T19:56:40.927 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a2 2023-10-18T19:56:40.927 DEBUG:boto:Data: 2023-10-18T19:56:40.928 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.928 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.929 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.929 DEBUG:boto:Params: {} 2023-10-18T19:56:40.929 DEBUG:boto:Token: None 2023-10-18T19:56:40.930 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-12/ 2023-10-18T19:56:40.930 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ= 2023-10-18T19:56:40.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P1fyD6HR5TB9Ht1BdkMigQAt+eQ=', 'Content-Length': '0'} 2023-10-18T19:56:40.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065e78e87f2e07c15-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.934 DEBUG:boto:b'bofrnf-12obj-a21000falseobj-a22023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.935 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-12 zones={a2, a1} 2023-10-18T19:56:40.935 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:40.935 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:40.936 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.936 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:56:40.937 DEBUG:boto:Data: 2023-10-18T19:56:40.937 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.938 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.938 DEBUG:boto:Params: {} 2023-10-18T19:56:40.939 DEBUG:boto:Token: None 2023-10-18T19:56:40.939 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-11/ 2023-10-18T19:56:40.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QhkYAMuGIyUW7JvJaDA8Ln4n0Ao= 2023-10-18T19:56:40.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QhkYAMuGIyUW7JvJaDA8Ln4n0Ao=', 'Content-Length': '0'} 2023-10-18T19:56:40.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f08edc0ea3ecddc-0065303878-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.943 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:40.943 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:40.944 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999&prefix=obj-a1 2023-10-18T19:56:40.944 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999&prefix=obj-a1 2023-10-18T19:56:40.944 DEBUG:boto:Method: GET 2023-10-18T19:56:40.945 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999&prefix=obj-a1 2023-10-18T19:56:40.945 DEBUG:boto:Data: 2023-10-18T19:56:40.946 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.946 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.946 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.947 DEBUG:boto:Params: {} 2023-10-18T19:56:40.947 DEBUG:boto:Token: None 2023-10-18T19:56:40.948 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-11/?versions 2023-10-18T19:56:40.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O4yC1nJGNHFihzb+yMFVbY0stmA= 2023-10-18T19:56:40.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O4yC1nJGNHFihzb+yMFVbY0stmA=', 'Content-Length': '0'} 2023-10-18T19:56:40.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001843588af9c713b7-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.951 DEBUG:boto:b'bofrnf-11obj-a1999falseobj-a1b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkXtrue2023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.952 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:56:40.952 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:56:40.953 DEBUG:boto:path=/bofrnf-11/obj-a1?acl 2023-10-18T19:56:40.953 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?acl 2023-10-18T19:56:40.953 DEBUG:boto:Method: PUT 2023-10-18T19:56:40.954 DEBUG:boto:Path: /bofrnf-11/obj-a1?acl 2023-10-18T19:56:40.954 DEBUG:boto:Data: 2023-10-18T19:56:40.955 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-18T19:56:40.955 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.955 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.956 DEBUG:boto:Params: {} 2023-10-18T19:56:40.956 DEBUG:boto:Token: None 2023-10-18T19:56:40.956 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:56:40 GMT x-amz-acl:public-read /bofrnf-11/obj-a1?acl 2023-10-18T19:56:40.957 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tE4P0p96yjkWNYJdOq4rIt0CV+A= 2023-10-18T19:56:40.957 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tE4P0p96yjkWNYJdOq4rIt0CV+A=', 'Content-Length': '0'} 2023-10-18T19:56:40.964 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ac611200804833a6-0065303878-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.965 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:40.965 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:40.966 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.966 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:56:40.966 DEBUG:boto:Data: 2023-10-18T19:56:40.967 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.967 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.968 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.968 DEBUG:boto:Params: {} 2023-10-18T19:56:40.968 DEBUG:boto:Token: None 2023-10-18T19:56:40.969 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-11/ 2023-10-18T19:56:40.969 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QhkYAMuGIyUW7JvJaDA8Ln4n0Ao= 2023-10-18T19:56:40.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QhkYAMuGIyUW7JvJaDA8Ln4n0Ao=', 'Content-Length': '0'} 2023-10-18T19:56:40.972 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085da57986bf2b06c-0065303878-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.973 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:56:40.973 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:56:40.974 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999&prefix=obj-a2 2023-10-18T19:56:40.974 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999&prefix=obj-a2 2023-10-18T19:56:40.975 DEBUG:boto:Method: GET 2023-10-18T19:56:40.975 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999&prefix=obj-a2 2023-10-18T19:56:40.976 DEBUG:boto:Data: 2023-10-18T19:56:40.976 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.976 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.977 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.977 DEBUG:boto:Params: {} 2023-10-18T19:56:40.978 DEBUG:boto:Token: None 2023-10-18T19:56:40.978 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:40 GMT /bofrnf-11/?versions 2023-10-18T19:56:40.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O4yC1nJGNHFihzb+yMFVbY0stmA= 2023-10-18T19:56:40.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O4yC1nJGNHFihzb+yMFVbY0stmA=', 'Content-Length': '0'} 2023-10-18T19:56:40.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f97383f9b7f0e646-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.982 DEBUG:boto:b'bofrnf-11obj-a2999falseobj-a2SPXUXHYcw3.HZZ0z2coSO05aV.6WDqVtrue2023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:40.983 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:56:40.983 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:56:40.983 DEBUG:boto:path=/bofrnf-11/obj-a2?acl 2023-10-18T19:56:40.984 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?acl 2023-10-18T19:56:40.984 DEBUG:boto:Method: PUT 2023-10-18T19:56:40.984 DEBUG:boto:Path: /bofrnf-11/obj-a2?acl 2023-10-18T19:56:40.985 DEBUG:boto:Data: 2023-10-18T19:56:40.985 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-18T19:56:40.985 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:40.986 DEBUG:boto:Port: 8001 2023-10-18T19:56:40.986 DEBUG:boto:Params: {} 2023-10-18T19:56:40.987 DEBUG:boto:Token: None 2023-10-18T19:56:40.987 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:56:40 GMT x-amz-acl:public-read /bofrnf-11/obj-a2?acl 2023-10-18T19:56:40.988 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vaxT/d8bBqNs+5sGdpI3uR78AQc= 2023-10-18T19:56:40.988 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vaxT/d8bBqNs+5sGdpI3uR78AQc=', 'Content-Length': '0'} 2023-10-18T19:56:40.995 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006fba4bdea0ae84a9-0065303878-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:40.995 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:40.996 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:40.996 DEBUG:boto:Method: HEAD 2023-10-18T19:56:40.997 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:56:40.997 DEBUG:boto:Data: 2023-10-18T19:56:40.997 DEBUG:boto:Headers: {} 2023-10-18T19:56:40.998 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:40.998 DEBUG:boto:Port: 8000 2023-10-18T19:56:40.999 DEBUG:boto:Params: {} 2023-10-18T19:56:40.999 DEBUG:boto:Token: None 2023-10-18T19:56:41.000 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:41 GMT /bofrnf-12/ 2023-10-18T19:56:41.000 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gRSFlVdyojrhkvEuiyg91YCHTeM= 2023-10-18T19:56:41.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gRSFlVdyojrhkvEuiyg91YCHTeM=', 'Content-Length': '0'} 2023-10-18T19:56:41.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca72de59ccff7284-0065303879-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:41.004 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:41.004 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:41.004 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999&prefix=obj-a1 2023-10-18T19:56:41.005 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999&prefix=obj-a1 2023-10-18T19:56:41.005 DEBUG:boto:Method: GET 2023-10-18T19:56:41.006 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999&prefix=obj-a1 2023-10-18T19:56:41.006 DEBUG:boto:Data: 2023-10-18T19:56:41.006 DEBUG:boto:Headers: {} 2023-10-18T19:56:41.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:41.007 DEBUG:boto:Port: 8000 2023-10-18T19:56:41.008 DEBUG:boto:Params: {} 2023-10-18T19:56:41.008 DEBUG:boto:Token: None 2023-10-18T19:56:41.009 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:41 GMT /bofrnf-12/?versions 2023-10-18T19:56:41.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2T3foHncqzg0YZEPx/9+l7RGGvQ= 2023-10-18T19:56:41.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2T3foHncqzg0YZEPx/9+l7RGGvQ=', 'Content-Length': '0'} 2023-10-18T19:56:41.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad1586360bed46b-0065303879-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:41.058 DEBUG:boto:b'bofrnf-12obj-a1999falseobj-a1toth3ilqud.qHjmYoshz5E--Nxv2rQYtrue2023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:41.059 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:56:41.059 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:56:41.060 DEBUG:boto:path=/bofrnf-12/obj-a1?acl 2023-10-18T19:56:41.060 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?acl 2023-10-18T19:56:41.061 DEBUG:boto:Method: PUT 2023-10-18T19:56:41.061 DEBUG:boto:Path: /bofrnf-12/obj-a1?acl 2023-10-18T19:56:41.061 DEBUG:boto:Data: 2023-10-18T19:56:41.062 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-18T19:56:41.062 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:56:41.063 DEBUG:boto:Port: 8000 2023-10-18T19:56:41.063 DEBUG:boto:Params: {} 2023-10-18T19:56:41.063 DEBUG:boto:Token: None 2023-10-18T19:56:41.064 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:56:41 GMT x-amz-acl:public-read /bofrnf-12/obj-a1?acl 2023-10-18T19:56:41.064 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9yYQ5A2adeUO2y7yrBez3WLt4To= 2023-10-18T19:56:41.065 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9yYQ5A2adeUO2y7yrBez3WLt4To=', 'Content-Length': '0'} 2023-10-18T19:56:41.072 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000df7db9d87074f7d0-0065303879-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:41.072 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:41.072 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:41.073 DEBUG:boto:Method: HEAD 2023-10-18T19:56:41.073 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:56:41.074 DEBUG:boto:Data: 2023-10-18T19:56:41.074 DEBUG:boto:Headers: {} 2023-10-18T19:56:41.074 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:41.075 DEBUG:boto:Port: 8001 2023-10-18T19:56:41.075 DEBUG:boto:Params: {} 2023-10-18T19:56:41.075 DEBUG:boto:Token: None 2023-10-18T19:56:41.076 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:56:41 GMT /bofrnf-12/ 2023-10-18T19:56:41.076 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gRSFlVdyojrhkvEuiyg91YCHTeM= 2023-10-18T19:56:41.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gRSFlVdyojrhkvEuiyg91YCHTeM=', 'Content-Length': '0'} 2023-10-18T19:56:41.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c68e5f3a3ff15cb-0065303879-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:41.079 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:56:41.080 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:56:41.080 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999&prefix=obj-a2 2023-10-18T19:56:41.081 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999&prefix=obj-a2 2023-10-18T19:56:41.081 DEBUG:boto:Method: GET 2023-10-18T19:56:41.081 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999&prefix=obj-a2 2023-10-18T19:56:41.082 DEBUG:boto:Data: 2023-10-18T19:56:41.082 DEBUG:boto:Headers: {} 2023-10-18T19:56:41.083 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:41.083 DEBUG:boto:Port: 8001 2023-10-18T19:56:41.084 DEBUG:boto:Params: {} 2023-10-18T19:56:41.084 DEBUG:boto:Token: None 2023-10-18T19:56:41.084 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:56:41 GMT /bofrnf-12/?versions 2023-10-18T19:56:41.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2T3foHncqzg0YZEPx/9+l7RGGvQ= 2023-10-18T19:56:41.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2T3foHncqzg0YZEPx/9+l7RGGvQ=', 'Content-Length': '0'} 2023-10-18T19:56:41.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0c166600f4d7a3e-0065303879-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:41.089 DEBUG:boto:b'bofrnf-12obj-a2999falseobj-a2yipaUQxzrtbVwuxaybttOFpenBTsbAbtrue2023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:56:41.089 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:56:41.090 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:56:41.090 DEBUG:boto:path=/bofrnf-12/obj-a2?acl 2023-10-18T19:56:41.091 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?acl 2023-10-18T19:56:41.091 DEBUG:boto:Method: PUT 2023-10-18T19:56:41.091 DEBUG:boto:Path: /bofrnf-12/obj-a2?acl 2023-10-18T19:56:41.092 DEBUG:boto:Data: 2023-10-18T19:56:41.092 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-18T19:56:41.092 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:56:41.093 DEBUG:boto:Port: 8001 2023-10-18T19:56:41.093 DEBUG:boto:Params: {} 2023-10-18T19:56:41.094 DEBUG:boto:Token: None 2023-10-18T19:56:41.094 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:56:41 GMT x-amz-acl:public-read /bofrnf-12/obj-a2?acl 2023-10-18T19:56:41.094 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bunr2vVVwCgkKFV5Y6l4au4feXo= 2023-10-18T19:56:41.095 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:56:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bunr2vVVwCgkKFV5Y6l4au4feXo=', 'Content-Length': '0'} 2023-10-18T19:56:41.101 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000044fb5b27104e9ba0-0065303879-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:56:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:56:41.101 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-11 2023-10-18T19:56:41.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:56:41.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:56:41.103 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:56:41.166 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:56:41.166 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:56:41.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:41.199+0000 7fefa76c7780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:41.201 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000010.21.6, ] 2023-10-18T19:56:41.201 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000011.24.6, ] 2023-10-18T19:56:46.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:46.200+0000 7fefa76c7780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:46.203 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000010.21.6, ] 2023-10-18T19:56:46.203 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000011.24.6, ] 2023-10-18T19:56:51.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:51.202+0000 7fefa76c7780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:51.204 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000010.21.6, ] 2023-10-18T19:56:51.204 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000011.24.6, ] 2023-10-18T19:56:56.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:56:56.202+0000 7fefa76c7780 1 waiting for incremental sync to catch up: 2023-10-18T19:56:56.205 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000010.21.6, ] 2023-10-18T19:56:56.205 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000011.24.6, ] 2023-10-18T19:57:01.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:01.203+0000 7fefa76c7780 1 bucket sync caught up with source: 2023-10-18T19:57:01.206 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000011.24.6, ] 2023-10-18T19:57:01.207 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000002.4.11, 00000000011.24.6, ] 2023-10-18T19:57:01.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:01.203+0000 7fefa76c7780 0 bucket checkpoint complete 2023-10-18T19:57:01.221 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-11 2023-10-18T19:57:01.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:01.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:01.223 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:57:01.285 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:57:01.285 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:57:01.322 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:01.320+0000 7f37d8486780 1 bucket sync caught up with source: 2023-10-18T19:57:01.323 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.20.11, , , 00000000011.29.6, 00000000002.12.11, ] 2023-10-18T19:57:01.323 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.20.11, , , 00000000011.29.6, 00000000002.12.11, ] 2023-10-18T19:57:01.323 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:01.320+0000 7f37d8486780 0 bucket checkpoint complete 2023-10-18T19:57:01.336 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-11 zones={a2, a1} 2023-10-18T19:57:01.336 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.337 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.337 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.338 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:57:01.338 DEBUG:boto:Data: 2023-10-18T19:57:01.339 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.339 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.339 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.340 DEBUG:boto:Params: {} 2023-10-18T19:57:01.340 DEBUG:boto:Token: None 2023-10-18T19:57:01.341 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.341 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m49MehCoKx5hYKm98Rayd+BsCLQ= 2023-10-18T19:57:01.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m49MehCoKx5hYKm98Rayd+BsCLQ=', 'Content-Length': '0'} 2023-10-18T19:57:01.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2923a87523ab3f7-006530388d-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.345 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.345 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.346 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.346 DEBUG:boto:Path: /bofrnf-11/ 2023-10-18T19:57:01.346 DEBUG:boto:Data: 2023-10-18T19:57:01.347 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.347 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.348 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.349 DEBUG:boto:Params: {} 2023-10-18T19:57:01.350 DEBUG:boto:Token: None 2023-10-18T19:57:01.350 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m49MehCoKx5hYKm98Rayd+BsCLQ= 2023-10-18T19:57:01.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m49MehCoKx5hYKm98Rayd+BsCLQ=', 'Content-Length': '0'} 2023-10-18T19:57:01.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011c47e5daf76b366-006530388d-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:57:01.355 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.355 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.356 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.356 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.356 DEBUG:boto:Method: GET 2023-10-18T19:57:01.357 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.357 DEBUG:boto:Data: 2023-10-18T19:57:01.358 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.358 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.359 DEBUG:boto:Params: {} 2023-10-18T19:57:01.359 DEBUG:boto:Token: None 2023-10-18T19:57:01.360 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/?versions 2023-10-18T19:57:01.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qhkE02jY7qS2AC/GZOZ3F3uG87s= 2023-10-18T19:57:01.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qhkE02jY7qS2AC/GZOZ3F3uG87s=', 'Content-Length': '0'} 2023-10-18T19:57:01.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051f6dcad0449abad-006530388d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.364 DEBUG:boto:b'bofrnf-11999falsedummyejuhtdELxhW-1atXJMLKJcENH4sJdCgtrue2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkXtrue2023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2SPXUXHYcw3.HZZ0z2coSO05aV.6WDqVtrue2023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.364 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:57:01.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-18T19:57:01.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-18T19:57:01.366 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:57:01.366 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.367 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.367 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.367 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.368 DEBUG:boto:Method: GET 2023-10-18T19:57:01.368 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.369 DEBUG:boto:Data: 2023-10-18T19:57:01.369 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.369 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.370 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.370 DEBUG:boto:Params: {} 2023-10-18T19:57:01.371 DEBUG:boto:Token: None 2023-10-18T19:57:01.371 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/?versions 2023-10-18T19:57:01.372 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qhkE02jY7qS2AC/GZOZ3F3uG87s= 2023-10-18T19:57:01.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qhkE02jY7qS2AC/GZOZ3F3uG87s=', 'Content-Length': '0'} 2023-10-18T19:57:01.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4f1bd8f4218b38-006530388d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.375 DEBUG:boto:b'bofrnf-11999falsedummyejuhtdELxhW-1atXJMLKJcENH4sJdCgtrue2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkXtrue2023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2SPXUXHYcw3.HZZ0z2coSO05aV.6WDqVtrue2023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.375 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:57:01.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-18T19:57:01.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-18T19:57:01.377 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.377 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.377 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.378 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.378 DEBUG:boto:Method: GET 2023-10-18T19:57:01.379 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.379 DEBUG:boto:Data: 2023-10-18T19:57:01.379 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.380 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.381 DEBUG:boto:Params: {} 2023-10-18T19:57:01.381 DEBUG:boto:Token: None 2023-10-18T19:57:01.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/?versions 2023-10-18T19:57:01.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qhkE02jY7qS2AC/GZOZ3F3uG87s= 2023-10-18T19:57:01.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qhkE02jY7qS2AC/GZOZ3F3uG87s=', 'Content-Length': '0'} 2023-10-18T19:57:01.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028a4edec256d3734-006530388d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.385 DEBUG:boto:b'bofrnf-11999falsedummyejuhtdELxhW-1atXJMLKJcENH4sJdCgtrue2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkXtrue2023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2SPXUXHYcw3.HZZ0z2coSO05aV.6WDqVtrue2023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.386 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.387 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.387 DEBUG:boto:path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.387 DEBUG:boto:auth_path=/bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.388 DEBUG:boto:Method: GET 2023-10-18T19:57:01.388 DEBUG:boto:Path: /bofrnf-11/?versions&max-keys=999 2023-10-18T19:57:01.389 DEBUG:boto:Data: 2023-10-18T19:57:01.389 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.390 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.390 DEBUG:boto:Params: {} 2023-10-18T19:57:01.391 DEBUG:boto:Token: None 2023-10-18T19:57:01.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/?versions 2023-10-18T19:57:01.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qhkE02jY7qS2AC/GZOZ3F3uG87s= 2023-10-18T19:57:01.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qhkE02jY7qS2AC/GZOZ3F3uG87s=', 'Content-Length': '0'} 2023-10-18T19:57:01.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adac5340626cd17a-006530388d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.395 DEBUG:boto:b'bofrnf-11999falsedummyejuhtdELxhW-1atXJMLKJcENH4sJdCgtrue2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkXtrue2023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2SPXUXHYcw3.HZZ0z2coSO05aV.6WDqVtrue2023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:57:01.396 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.396 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.397 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.397 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.398 DEBUG:boto:Method: GET 2023-10-18T19:57:01.398 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.399 DEBUG:boto:Data: 2023-10-18T19:57:01.399 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.400 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.400 DEBUG:boto:Params: {} 2023-10-18T19:57:01.401 DEBUG:boto:Token: None 2023-10-18T19:57:01.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg= 2023-10-18T19:57:01.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg=', 'Content-Length': '0'} 2023-10-18T19:57:01.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c99f1bddb343c5d3-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.405 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.405 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.406 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.406 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.407 DEBUG:boto:Method: GET 2023-10-18T19:57:01.407 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.408 DEBUG:boto:Data: 2023-10-18T19:57:01.408 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.409 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.409 DEBUG:boto:Params: {} 2023-10-18T19:57:01.410 DEBUG:boto:Token: None 2023-10-18T19:57:01.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg= 2023-10-18T19:57:01.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg=', 'Content-Length': '0'} 2023-10-18T19:57:01.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013278d7346d1532f-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.414 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.414 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.415 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.415 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.415 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.416 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.416 DEBUG:boto:Data: 2023-10-18T19:57:01.416 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.417 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.418 DEBUG:boto:Params: {} 2023-10-18T19:57:01.418 DEBUG:boto:Token: None 2023-10-18T19:57:01.419 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nBtwqJPkmqGfY+SohOhG6qUOuyY= 2023-10-18T19:57:01.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nBtwqJPkmqGfY+SohOhG6qUOuyY=', 'Content-Length': '0'} 2023-10-18T19:57:01.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b2e985f537199b2-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.422 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.423 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.423 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.423 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.424 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.424 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.425 DEBUG:boto:Data: 2023-10-18T19:57:01.425 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.426 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.426 DEBUG:boto:Params: {} 2023-10-18T19:57:01.427 DEBUG:boto:Token: None 2023-10-18T19:57:01.427 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nBtwqJPkmqGfY+SohOhG6qUOuyY= 2023-10-18T19:57:01.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nBtwqJPkmqGfY+SohOhG6qUOuyY=', 'Content-Length': '0'} 2023-10-18T19:57:01.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce9f30b1aff24c3c-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:57:01.431 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.431 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.432 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.432 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.433 DEBUG:boto:Method: GET 2023-10-18T19:57:01.433 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.433 DEBUG:boto:Data: 2023-10-18T19:57:01.434 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.434 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.435 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.435 DEBUG:boto:Params: {} 2023-10-18T19:57:01.436 DEBUG:boto:Token: None 2023-10-18T19:57:01.436 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.437 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg= 2023-10-18T19:57:01.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg=', 'Content-Length': '0'} 2023-10-18T19:57:01.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f0b8d6342b0d8b2-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.440 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.440 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.440 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.441 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.441 DEBUG:boto:Method: GET 2023-10-18T19:57:01.442 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.442 DEBUG:boto:Data: 2023-10-18T19:57:01.443 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.443 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.443 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.444 DEBUG:boto:Params: {} 2023-10-18T19:57:01.444 DEBUG:boto:Token: None 2023-10-18T19:57:01.445 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.445 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg= 2023-10-18T19:57:01.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg=', 'Content-Length': '0'} 2023-10-18T19:57:01.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008188255e9c7c7939-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.448 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.449 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.449 DEBUG:boto:path=/bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.449 DEBUG:boto:auth_path=/bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.450 DEBUG:boto:Method: GET 2023-10-18T19:57:01.450 DEBUG:boto:Path: /bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.451 DEBUG:boto:Data: 2023-10-18T19:57:01.451 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.452 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.452 DEBUG:boto:Params: {} 2023-10-18T19:57:01.453 DEBUG:boto:Token: None 2023-10-18T19:57:01.453 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.454 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000726fc5fc46a12e1d-006530388d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.457 DEBUG:boto:b'bofrnf-11dummy1000falsedummy2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.458 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.459 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.459 DEBUG:boto:path=/bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.459 DEBUG:boto:auth_path=/bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.460 DEBUG:boto:Method: GET 2023-10-18T19:57:01.460 DEBUG:boto:Path: /bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.460 DEBUG:boto:Data: 2023-10-18T19:57:01.461 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.461 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.462 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.462 DEBUG:boto:Params: {} 2023-10-18T19:57:01.463 DEBUG:boto:Token: None 2023-10-18T19:57:01.463 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab798a20c306540-006530388d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.510 DEBUG:boto:b'bofrnf-11dummy1000falsedummy2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.511 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.512 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.512 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.512 DEBUG:boto:Path: /bofrnf-11/dummy 2023-10-18T19:57:01.513 DEBUG:boto:Data: 2023-10-18T19:57:01.513 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.513 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.514 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.514 DEBUG:boto:Params: {} 2023-10-18T19:57:01.515 DEBUG:boto:Token: None 2023-10-18T19:57:01.515 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy 2023-10-18T19:57:01.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BhsoafixoHoC2WnhimVEM9V9Gnk= 2023-10-18T19:57:01.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BhsoafixoHoC2WnhimVEM9V9Gnk=', 'Content-Length': '0'} 2023-10-18T19:57:01.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a46e62c804779baa-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.519 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.519 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.520 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.520 DEBUG:boto:Path: /bofrnf-11/dummy 2023-10-18T19:57:01.521 DEBUG:boto:Data: 2023-10-18T19:57:01.521 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.521 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.522 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.522 DEBUG:boto:Params: {} 2023-10-18T19:57:01.523 DEBUG:boto:Token: None 2023-10-18T19:57:01.523 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy 2023-10-18T19:57:01.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BhsoafixoHoC2WnhimVEM9V9Gnk= 2023-10-18T19:57:01.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BhsoafixoHoC2WnhimVEM9V9Gnk=', 'Content-Length': '0'} 2023-10-18T19:57:01.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec49d19f139eb977-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:57:01.527 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.528 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.528 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.528 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.529 DEBUG:boto:Method: GET 2023-10-18T19:57:01.529 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.530 DEBUG:boto:Data: 2023-10-18T19:57:01.530 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.531 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.531 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.531 DEBUG:boto:Params: {} 2023-10-18T19:57:01.532 DEBUG:boto:Token: None 2023-10-18T19:57:01.532 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg= 2023-10-18T19:57:01.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg=', 'Content-Length': '0'} 2023-10-18T19:57:01.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c5036d2fb3e9327-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.536 DEBUG:boto:path=/bofrnf-11/dummy 2023-10-18T19:57:01.536 DEBUG:boto:auth_path=/bofrnf-11/dummy 2023-10-18T19:57:01.536 DEBUG:boto:path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.537 DEBUG:boto:auth_path=/bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.537 DEBUG:boto:Method: GET 2023-10-18T19:57:01.538 DEBUG:boto:Path: /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.538 DEBUG:boto:Data: 2023-10-18T19:57:01.539 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.539 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.539 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.540 DEBUG:boto:Params: {} 2023-10-18T19:57:01.540 DEBUG:boto:Token: None 2023-10-18T19:57:01.541 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/dummy?versionId=ejuhtdELxhW-1atXJMLKJcENH4sJdCg 2023-10-18T19:57:01.541 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg= 2023-10-18T19:57:01.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VM0XkAQJfEb9+Y0w6Wu1qXZ2LEg=', 'Content-Length': '0'} 2023-10-18T19:57:01.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:03 GMT'), ('x-amz-version-id', 'ejuhtdELxhW-1atXJMLKJcENH4sJdCg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084c8ee2fb3f97bf9-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.544 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.545 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.545 DEBUG:boto:path=/bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.545 DEBUG:boto:auth_path=/bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.546 DEBUG:boto:Method: GET 2023-10-18T19:57:01.546 DEBUG:boto:Path: /bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.547 DEBUG:boto:Data: 2023-10-18T19:57:01.547 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.547 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.548 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.548 DEBUG:boto:Params: {} 2023-10-18T19:57:01.549 DEBUG:boto:Token: None 2023-10-18T19:57:01.549 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.550 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000108e7ca832fa5331-006530388d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.598 DEBUG:boto:b'bofrnf-11dummy1000falsedummy2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.599 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.600 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.600 DEBUG:boto:path=/bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.601 DEBUG:boto:auth_path=/bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.601 DEBUG:boto:Method: GET 2023-10-18T19:57:01.602 DEBUG:boto:Path: /bofrnf-11/?prefix=dummy 2023-10-18T19:57:01.602 DEBUG:boto:Data: 2023-10-18T19:57:01.603 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.603 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.603 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.604 DEBUG:boto:Params: {} 2023-10-18T19:57:01.604 DEBUG:boto:Token: None 2023-10-18T19:57:01.605 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.606 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000345675f65f5bf5bc-006530388d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.609 DEBUG:boto:b'bofrnf-11dummy1000falsedummy2023-10-18T19:55:03.421Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:57:01.610 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.611 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.611 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.612 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.612 DEBUG:boto:Method: GET 2023-10-18T19:57:01.612 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.613 DEBUG:boto:Data: 2023-10-18T19:57:01.613 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.614 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.614 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.614 DEBUG:boto:Params: {} 2023-10-18T19:57:01.615 DEBUG:boto:Token: None 2023-10-18T19:57:01.615 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.616 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas= 2023-10-18T19:57:01.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas=', 'Content-Length': '0'} 2023-10-18T19:57:01.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000536aea59d2d5282d-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.619 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.620 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.620 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.621 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.621 DEBUG:boto:Method: GET 2023-10-18T19:57:01.622 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.622 DEBUG:boto:Data: 2023-10-18T19:57:01.623 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.623 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.623 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.624 DEBUG:boto:Params: {} 2023-10-18T19:57:01.624 DEBUG:boto:Token: None 2023-10-18T19:57:01.625 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.625 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas= 2023-10-18T19:57:01.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas=', 'Content-Length': '0'} 2023-10-18T19:57:01.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9d7f35239f35fc2-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.628 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.628 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.629 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.629 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.629 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.630 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.630 DEBUG:boto:Data: 2023-10-18T19:57:01.631 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.631 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.632 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.632 DEBUG:boto:Params: {} 2023-10-18T19:57:01.633 DEBUG:boto:Token: None 2023-10-18T19:57:01.633 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.634 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GBbSaLpp9/MKOxuH8MVlISj+Opo= 2023-10-18T19:57:01.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GBbSaLpp9/MKOxuH8MVlISj+Opo=', 'Content-Length': '0'} 2023-10-18T19:57:01.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fbe787f84b86527-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.636 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.637 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.637 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.638 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.638 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.639 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.639 DEBUG:boto:Data: 2023-10-18T19:57:01.639 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.640 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.640 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.641 DEBUG:boto:Params: {} 2023-10-18T19:57:01.641 DEBUG:boto:Token: None 2023-10-18T19:57:01.642 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.642 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GBbSaLpp9/MKOxuH8MVlISj+Opo= 2023-10-18T19:57:01.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GBbSaLpp9/MKOxuH8MVlISj+Opo=', 'Content-Length': '0'} 2023-10-18T19:57:01.644 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a81bf1be64cb1631-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:57:01.645 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.646 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.647 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.647 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.647 DEBUG:boto:Method: GET 2023-10-18T19:57:01.648 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.648 DEBUG:boto:Data: 2023-10-18T19:57:01.649 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.650 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.650 DEBUG:boto:Params: {} 2023-10-18T19:57:01.651 DEBUG:boto:Token: None 2023-10-18T19:57:01.651 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas= 2023-10-18T19:57:01.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas=', 'Content-Length': '0'} 2023-10-18T19:57:01.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097c10ca1a746f408-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.654 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.655 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.655 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.656 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.656 DEBUG:boto:Method: GET 2023-10-18T19:57:01.657 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.657 DEBUG:boto:Data: 2023-10-18T19:57:01.658 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.658 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.659 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.659 DEBUG:boto:Params: {} 2023-10-18T19:57:01.660 DEBUG:boto:Token: None 2023-10-18T19:57:01.660 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.661 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas= 2023-10-18T19:57:01.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas=', 'Content-Length': '0'} 2023-10-18T19:57:01.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dda946a3b232f9f-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.664 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.664 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.665 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.665 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.665 DEBUG:boto:Method: GET 2023-10-18T19:57:01.666 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.666 DEBUG:boto:Data: 2023-10-18T19:57:01.667 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.667 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.668 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.668 DEBUG:boto:Params: {} 2023-10-18T19:57:01.668 DEBUG:boto:Token: None 2023-10-18T19:57:01.669 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.669 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000339e1ffe5615334f-006530388d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.673 DEBUG:boto:b'bofrnf-11obj-a11000falseobj-a12023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.674 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.674 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.675 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.675 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.675 DEBUG:boto:Method: GET 2023-10-18T19:57:01.676 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.676 DEBUG:boto:Data: 2023-10-18T19:57:01.677 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.677 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.677 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.678 DEBUG:boto:Params: {} 2023-10-18T19:57:01.678 DEBUG:boto:Token: None 2023-10-18T19:57:01.679 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023950d858f01e0e2-006530388d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.682 DEBUG:boto:b'bofrnf-11obj-a11000falseobj-a12023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.683 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.683 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.684 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.684 DEBUG:boto:Path: /bofrnf-11/obj-a1 2023-10-18T19:57:01.685 DEBUG:boto:Data: 2023-10-18T19:57:01.685 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.686 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.686 DEBUG:boto:Params: {} 2023-10-18T19:57:01.687 DEBUG:boto:Token: None 2023-10-18T19:57:01.687 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1 2023-10-18T19:57:01.689 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7kmQnFA/IA8iqJcuGvgCf5B4TX4= 2023-10-18T19:57:01.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7kmQnFA/IA8iqJcuGvgCf5B4TX4=', 'Content-Length': '0'} 2023-10-18T19:57:01.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c03248054966b613-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.693 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.693 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.694 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.694 DEBUG:boto:Path: /bofrnf-11/obj-a1 2023-10-18T19:57:01.694 DEBUG:boto:Data: 2023-10-18T19:57:01.695 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.695 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.695 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.696 DEBUG:boto:Params: {} 2023-10-18T19:57:01.696 DEBUG:boto:Token: None 2023-10-18T19:57:01.697 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1 2023-10-18T19:57:01.697 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7kmQnFA/IA8iqJcuGvgCf5B4TX4= 2023-10-18T19:57:01.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7kmQnFA/IA8iqJcuGvgCf5B4TX4=', 'Content-Length': '0'} 2023-10-18T19:57:01.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000851bfbee4c03f0df-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:57:01.701 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.701 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.702 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.702 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.703 DEBUG:boto:Method: GET 2023-10-18T19:57:01.703 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.704 DEBUG:boto:Data: 2023-10-18T19:57:01.704 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.705 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.705 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.705 DEBUG:boto:Params: {} 2023-10-18T19:57:01.706 DEBUG:boto:Token: None 2023-10-18T19:57:01.707 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.707 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas= 2023-10-18T19:57:01.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas=', 'Content-Length': '0'} 2023-10-18T19:57:01.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c82122eece7ff89-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.710 DEBUG:boto:path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.710 DEBUG:boto:auth_path=/bofrnf-11/obj-a1 2023-10-18T19:57:01.711 DEBUG:boto:path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.711 DEBUG:boto:auth_path=/bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.712 DEBUG:boto:Method: GET 2023-10-18T19:57:01.712 DEBUG:boto:Path: /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.713 DEBUG:boto:Data: 2023-10-18T19:57:01.713 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.714 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.714 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.715 DEBUG:boto:Params: {} 2023-10-18T19:57:01.715 DEBUG:boto:Token: None 2023-10-18T19:57:01.715 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a1?versionId=b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX 2023-10-18T19:57:01.716 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas= 2023-10-18T19:57:01.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFu6b6hbjy2F9vXJmGkaifvaYas=', 'Content-Length': '0'} 2023-10-18T19:57:01.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'b6R8vJz8GDXUGyZ.93pHd9QXFfaUbkX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f8508a60fe2ca25-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.719 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.720 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.720 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.720 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.721 DEBUG:boto:Method: GET 2023-10-18T19:57:01.721 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.722 DEBUG:boto:Data: 2023-10-18T19:57:01.722 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.723 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.723 DEBUG:boto:Params: {} 2023-10-18T19:57:01.724 DEBUG:boto:Token: None 2023-10-18T19:57:01.724 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b4d92d3168f094b-006530388d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.728 DEBUG:boto:b'bofrnf-11obj-a11000falseobj-a12023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.729 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.729 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.729 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.730 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.730 DEBUG:boto:Method: GET 2023-10-18T19:57:01.731 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a1 2023-10-18T19:57:01.731 DEBUG:boto:Data: 2023-10-18T19:57:01.732 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.732 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.733 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.733 DEBUG:boto:Params: {} 2023-10-18T19:57:01.734 DEBUG:boto:Token: None 2023-10-18T19:57:01.734 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029fb9cb9e3ef2a20-006530388d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.778 DEBUG:boto:b'bofrnf-11obj-a11000falseobj-a12023-10-18T19:55:38.694Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:57:01.780 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.780 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.781 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.781 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.781 DEBUG:boto:Method: GET 2023-10-18T19:57:01.782 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.782 DEBUG:boto:Data: 2023-10-18T19:57:01.783 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.783 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.783 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.784 DEBUG:boto:Params: {} 2023-10-18T19:57:01.784 DEBUG:boto:Token: None 2023-10-18T19:57:01.785 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.785 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro= 2023-10-18T19:57:01.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro=', 'Content-Length': '0'} 2023-10-18T19:57:01.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d66ca07011831cce-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.788 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.789 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.789 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.790 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.790 DEBUG:boto:Method: GET 2023-10-18T19:57:01.791 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.791 DEBUG:boto:Data: 2023-10-18T19:57:01.792 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.792 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.793 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.793 DEBUG:boto:Params: {} 2023-10-18T19:57:01.794 DEBUG:boto:Token: None 2023-10-18T19:57:01.794 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.795 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro= 2023-10-18T19:57:01.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro=', 'Content-Length': '0'} 2023-10-18T19:57:01.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c40da825a925d0a-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.798 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.798 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.798 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.799 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.799 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.800 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.800 DEBUG:boto:Data: 2023-10-18T19:57:01.800 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.801 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.801 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.801 DEBUG:boto:Params: {} 2023-10-18T19:57:01.802 DEBUG:boto:Token: None 2023-10-18T19:57:01.802 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.803 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bmmPc2Zr2qBL4C9kMq2y38/4VDU= 2023-10-18T19:57:01.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bmmPc2Zr2qBL4C9kMq2y38/4VDU=', 'Content-Length': '0'} 2023-10-18T19:57:01.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000274704ff32169a4b-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.806 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.806 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.806 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.807 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.807 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.808 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.808 DEBUG:boto:Data: 2023-10-18T19:57:01.809 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.809 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.810 DEBUG:boto:Params: {} 2023-10-18T19:57:01.810 DEBUG:boto:Token: None 2023-10-18T19:57:01.811 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bmmPc2Zr2qBL4C9kMq2y38/4VDU= 2023-10-18T19:57:01.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bmmPc2Zr2qBL4C9kMq2y38/4VDU=', 'Content-Length': '0'} 2023-10-18T19:57:01.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a758a3b43b6726df-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:57:01.814 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.815 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.815 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.816 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.816 DEBUG:boto:Method: GET 2023-10-18T19:57:01.816 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.817 DEBUG:boto:Data: 2023-10-18T19:57:01.817 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.818 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.818 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.818 DEBUG:boto:Params: {} 2023-10-18T19:57:01.819 DEBUG:boto:Token: None 2023-10-18T19:57:01.819 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.820 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro= 2023-10-18T19:57:01.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro=', 'Content-Length': '0'} 2023-10-18T19:57:01.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000560f8334d0a16b31-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.823 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.823 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.824 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.824 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.824 DEBUG:boto:Method: GET 2023-10-18T19:57:01.825 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.825 DEBUG:boto:Data: 2023-10-18T19:57:01.826 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.826 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.826 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.827 DEBUG:boto:Params: {} 2023-10-18T19:57:01.827 DEBUG:boto:Token: None 2023-10-18T19:57:01.828 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.829 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro= 2023-10-18T19:57:01.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro=', 'Content-Length': '0'} 2023-10-18T19:57:01.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000702773111b4ab132-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.832 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.832 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.833 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.833 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.833 DEBUG:boto:Method: GET 2023-10-18T19:57:01.834 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.834 DEBUG:boto:Data: 2023-10-18T19:57:01.835 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.835 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.835 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.836 DEBUG:boto:Params: {} 2023-10-18T19:57:01.836 DEBUG:boto:Token: None 2023-10-18T19:57:01.837 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.837 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b727d807551311ba-006530388d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.882 DEBUG:boto:b'bofrnf-11obj-a21000falseobj-a22023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.883 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.884 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.884 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.884 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.885 DEBUG:boto:Method: GET 2023-10-18T19:57:01.885 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.886 DEBUG:boto:Data: 2023-10-18T19:57:01.886 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.887 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.887 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.887 DEBUG:boto:Params: {} 2023-10-18T19:57:01.888 DEBUG:boto:Token: None 2023-10-18T19:57:01.888 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.889 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e2cd23fe88d502a-006530388d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.892 DEBUG:boto:b'bofrnf-11obj-a21000falseobj-a22023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.893 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.893 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.894 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.894 DEBUG:boto:Path: /bofrnf-11/obj-a2 2023-10-18T19:57:01.895 DEBUG:boto:Data: 2023-10-18T19:57:01.895 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.895 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.896 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.896 DEBUG:boto:Params: {} 2023-10-18T19:57:01.897 DEBUG:boto:Token: None 2023-10-18T19:57:01.897 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2 2023-10-18T19:57:01.898 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PfcGGAC0AeWJqmcsTCvMlB+3WS4= 2023-10-18T19:57:01.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PfcGGAC0AeWJqmcsTCvMlB+3WS4=', 'Content-Length': '0'} 2023-10-18T19:57:01.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c366082c31b039a2-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.901 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.901 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.902 DEBUG:boto:Method: HEAD 2023-10-18T19:57:01.902 DEBUG:boto:Path: /bofrnf-11/obj-a2 2023-10-18T19:57:01.903 DEBUG:boto:Data: 2023-10-18T19:57:01.903 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.903 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.904 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.904 DEBUG:boto:Params: {} 2023-10-18T19:57:01.905 DEBUG:boto:Token: None 2023-10-18T19:57:01.905 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2 2023-10-18T19:57:01.905 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PfcGGAC0AeWJqmcsTCvMlB+3WS4= 2023-10-18T19:57:01.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PfcGGAC0AeWJqmcsTCvMlB+3WS4=', 'Content-Length': '0'} 2023-10-18T19:57:01.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009022a744659ec6b0-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:57:01.909 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.910 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.910 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.910 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.911 DEBUG:boto:Method: GET 2023-10-18T19:57:01.911 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.912 DEBUG:boto:Data: 2023-10-18T19:57:01.912 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.913 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.913 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.913 DEBUG:boto:Params: {} 2023-10-18T19:57:01.914 DEBUG:boto:Token: None 2023-10-18T19:57:01.914 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.915 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro= 2023-10-18T19:57:01.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro=', 'Content-Length': '0'} 2023-10-18T19:57:01.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000180793aad2c18aba-006530388d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.918 DEBUG:boto:path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.919 DEBUG:boto:auth_path=/bofrnf-11/obj-a2 2023-10-18T19:57:01.919 DEBUG:boto:path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.919 DEBUG:boto:auth_path=/bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.920 DEBUG:boto:Method: GET 2023-10-18T19:57:01.920 DEBUG:boto:Path: /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.921 DEBUG:boto:Data: 2023-10-18T19:57:01.921 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.921 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.922 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.922 DEBUG:boto:Params: {} 2023-10-18T19:57:01.923 DEBUG:boto:Token: None 2023-10-18T19:57:01.924 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/obj-a2?versionId=SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV 2023-10-18T19:57:01.924 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro= 2023-10-18T19:57:01.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pJu0HyLXytyOOBSwR0uvr80xXro=', 'Content-Length': '0'} 2023-10-18T19:57:01.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'SPXUXHYcw3.HZZ0z2coSO05aV.6WDqV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d68a401c42c88a4a-006530388d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.927 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.927 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.928 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.928 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.929 DEBUG:boto:Method: GET 2023-10-18T19:57:01.929 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.930 DEBUG:boto:Data: 2023-10-18T19:57:01.930 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:01.931 DEBUG:boto:Port: 8001 2023-10-18T19:57:01.931 DEBUG:boto:Params: {} 2023-10-18T19:57:01.932 DEBUG:boto:Token: None 2023-10-18T19:57:01.932 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003452c0f578aae56-006530388d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.937 DEBUG:boto:b'bofrnf-11obj-a21000falseobj-a22023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.937 DEBUG:boto:path=/bofrnf-11/ 2023-10-18T19:57:01.938 DEBUG:boto:auth_path=/bofrnf-11/ 2023-10-18T19:57:01.938 DEBUG:boto:path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.939 DEBUG:boto:auth_path=/bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.939 DEBUG:boto:Method: GET 2023-10-18T19:57:01.940 DEBUG:boto:Path: /bofrnf-11/?prefix=obj-a2 2023-10-18T19:57:01.940 DEBUG:boto:Data: 2023-10-18T19:57:01.941 DEBUG:boto:Headers: {} 2023-10-18T19:57:01.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:01.942 DEBUG:boto:Port: 8000 2023-10-18T19:57:01.942 DEBUG:boto:Params: {} 2023-10-18T19:57:01.943 DEBUG:boto:Token: None 2023-10-18T19:57:01.943 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:01 GMT /bofrnf-11/ 2023-10-18T19:57:01.944 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU= 2023-10-18T19:57:01.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAiDNL+ANfY1nFljxb11uvhIMNU=', 'Content-Length': '0'} 2023-10-18T19:57:01.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059797977064663c5-006530388d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:01.947 DEBUG:boto:b'bofrnf-11obj-a21000falseobj-a22023-10-18T19:55:38.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:01.947 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-11 zones={a2, a1} 2023-10-18T19:57:01.948 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-12 2023-10-18T19:57:01.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:01.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:01.949 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:57:02.006 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:57:02.007 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:57:02.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:02.041+0000 7fb16a580780 1 bucket sync caught up with source: 2023-10-18T19:57:02.044 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.8.11, , , 00000000002.6.11, 00000000011.26.6, ] 2023-10-18T19:57:02.045 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.8.11, , , 00000000002.6.11, 00000000011.26.6, ] 2023-10-18T19:57:02.045 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:02.041+0000 7fb16a580780 0 bucket checkpoint complete 2023-10-18T19:57:02.056 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-12 2023-10-18T19:57:02.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:02.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:02.057 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:57:02.113 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:57:02.113 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:57:02.148 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:02.145+0000 7fc9cdb64780 1 bucket sync caught up with source: 2023-10-18T19:57:02.148 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000011.38.6, 00000000002.15.11, ] 2023-10-18T19:57:02.148 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000011.38.6, 00000000002.15.11, ] 2023-10-18T19:57:02.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:02.145+0000 7fc9cdb64780 0 bucket checkpoint complete 2023-10-18T19:57:02.161 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-12 zones={a2, a1} 2023-10-18T19:57:02.162 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.163 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.163 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.164 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:57:02.165 DEBUG:boto:Data: 2023-10-18T19:57:02.166 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.167 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.167 DEBUG:boto:Params: {} 2023-10-18T19:57:02.168 DEBUG:boto:Token: None 2023-10-18T19:57:02.168 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.169 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ES3ikzB6tMesyd96f8OZMKR3iTc= 2023-10-18T19:57:02.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ES3ikzB6tMesyd96f8OZMKR3iTc=', 'Content-Length': '0'} 2023-10-18T19:57:02.172 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013b6c64e3a4fd410-006530388e-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.173 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.173 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.173 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.174 DEBUG:boto:Path: /bofrnf-12/ 2023-10-18T19:57:02.174 DEBUG:boto:Data: 2023-10-18T19:57:02.175 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.175 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.175 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.176 DEBUG:boto:Params: {} 2023-10-18T19:57:02.176 DEBUG:boto:Token: None 2023-10-18T19:57:02.177 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.177 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ES3ikzB6tMesyd96f8OZMKR3iTc= 2023-10-18T19:57:02.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ES3ikzB6tMesyd96f8OZMKR3iTc=', 'Content-Length': '0'} 2023-10-18T19:57:02.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003a446c11db17c2c-006530388e-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.180 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:57:02.181 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.181 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.182 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.182 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.183 DEBUG:boto:Method: GET 2023-10-18T19:57:02.183 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.183 DEBUG:boto:Data: 2023-10-18T19:57:02.184 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.184 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.185 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.185 DEBUG:boto:Params: {} 2023-10-18T19:57:02.185 DEBUG:boto:Token: None 2023-10-18T19:57:02.186 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/?versions 2023-10-18T19:57:02.187 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ueqYhHGlVgWu1gkN5j+tpFMv5aA= 2023-10-18T19:57:02.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ueqYhHGlVgWu1gkN5j+tpFMv5aA=', 'Content-Length': '0'} 2023-10-18T19:57:02.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a112c3972c2f1e-006530388e-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.190 DEBUG:boto:b'bofrnf-12999falsedummyytzJEO2gslxPWNYVg6CTpsims4XGqOgtrue2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1toth3ilqud.qHjmYoshz5E--Nxv2rQYtrue2023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2yipaUQxzrtbVwuxaybttOFpenBTsbAbtrue2023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.191 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:57:02.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-18T19:57:02.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-18T19:57:02.192 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:57:02.192 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.193 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.193 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.193 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.194 DEBUG:boto:Method: GET 2023-10-18T19:57:02.194 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.195 DEBUG:boto:Data: 2023-10-18T19:57:02.195 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.195 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.196 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.196 DEBUG:boto:Params: {} 2023-10-18T19:57:02.197 DEBUG:boto:Token: None 2023-10-18T19:57:02.197 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/?versions 2023-10-18T19:57:02.198 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ueqYhHGlVgWu1gkN5j+tpFMv5aA= 2023-10-18T19:57:02.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ueqYhHGlVgWu1gkN5j+tpFMv5aA=', 'Content-Length': '0'} 2023-10-18T19:57:02.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002659a705f663045e-006530388e-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.201 DEBUG:boto:b'bofrnf-12999falsedummyytzJEO2gslxPWNYVg6CTpsims4XGqOgtrue2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1toth3ilqud.qHjmYoshz5E--Nxv2rQYtrue2023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2yipaUQxzrtbVwuxaybttOFpenBTsbAbtrue2023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.202 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:57:02.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-18T19:57:02.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-18T19:57:02.203 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.204 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.204 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.204 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.205 DEBUG:boto:Method: GET 2023-10-18T19:57:02.205 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.206 DEBUG:boto:Data: 2023-10-18T19:57:02.206 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.206 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.207 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.207 DEBUG:boto:Params: {} 2023-10-18T19:57:02.208 DEBUG:boto:Token: None 2023-10-18T19:57:02.208 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/?versions 2023-10-18T19:57:02.209 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ueqYhHGlVgWu1gkN5j+tpFMv5aA= 2023-10-18T19:57:02.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ueqYhHGlVgWu1gkN5j+tpFMv5aA=', 'Content-Length': '0'} 2023-10-18T19:57:02.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066997f8eb4a67c2c-006530388e-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.212 DEBUG:boto:b'bofrnf-12999falsedummyytzJEO2gslxPWNYVg6CTpsims4XGqOgtrue2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1toth3ilqud.qHjmYoshz5E--Nxv2rQYtrue2023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2yipaUQxzrtbVwuxaybttOFpenBTsbAbtrue2023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.213 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.213 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.214 DEBUG:boto:path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.215 DEBUG:boto:auth_path=/bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.216 DEBUG:boto:Method: GET 2023-10-18T19:57:02.216 DEBUG:boto:Path: /bofrnf-12/?versions&max-keys=999 2023-10-18T19:57:02.217 DEBUG:boto:Data: 2023-10-18T19:57:02.218 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.219 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.219 DEBUG:boto:Params: {} 2023-10-18T19:57:02.220 DEBUG:boto:Token: None 2023-10-18T19:57:02.221 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/?versions 2023-10-18T19:57:02.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ueqYhHGlVgWu1gkN5j+tpFMv5aA= 2023-10-18T19:57:02.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ueqYhHGlVgWu1gkN5j+tpFMv5aA=', 'Content-Length': '0'} 2023-10-18T19:57:02.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f9f2c6c17c5ab7-006530388e-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.225 DEBUG:boto:b'bofrnf-12999falsedummyytzJEO2gslxPWNYVg6CTpsims4XGqOgtrue2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1toth3ilqud.qHjmYoshz5E--Nxv2rQYtrue2023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2yipaUQxzrtbVwuxaybttOFpenBTsbAbtrue2023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:57:02.226 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.227 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.227 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.228 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.228 DEBUG:boto:Method: GET 2023-10-18T19:57:02.229 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.229 DEBUG:boto:Data: 2023-10-18T19:57:02.230 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.231 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.231 DEBUG:boto:Params: {} 2023-10-18T19:57:02.232 DEBUG:boto:Token: None 2023-10-18T19:57:02.232 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.233 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM= 2023-10-18T19:57:02.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM=', 'Content-Length': '0'} 2023-10-18T19:57:02.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059b9249c2d420d5e-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.236 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.237 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.237 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.238 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.239 DEBUG:boto:Method: GET 2023-10-18T19:57:02.239 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.240 DEBUG:boto:Data: 2023-10-18T19:57:02.241 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.242 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.243 DEBUG:boto:Params: {} 2023-10-18T19:57:02.244 DEBUG:boto:Token: None 2023-10-18T19:57:02.245 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.245 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM= 2023-10-18T19:57:02.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM=', 'Content-Length': '0'} 2023-10-18T19:57:02.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4a6a47e1e5b609f-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.249 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.249 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.250 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.250 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.250 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.251 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.251 DEBUG:boto:Data: 2023-10-18T19:57:02.252 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.252 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.252 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.253 DEBUG:boto:Params: {} 2023-10-18T19:57:02.253 DEBUG:boto:Token: None 2023-10-18T19:57:02.254 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.254 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Juk8EtyCWHXwvP/bybzRlfZGqio= 2023-10-18T19:57:02.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Juk8EtyCWHXwvP/bybzRlfZGqio=', 'Content-Length': '0'} 2023-10-18T19:57:02.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093bd28ff80aafb70-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.257 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.257 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.258 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.258 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.259 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.259 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.259 DEBUG:boto:Data: 2023-10-18T19:57:02.260 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.260 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.261 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.261 DEBUG:boto:Params: {} 2023-10-18T19:57:02.261 DEBUG:boto:Token: None 2023-10-18T19:57:02.262 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.262 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Juk8EtyCWHXwvP/bybzRlfZGqio= 2023-10-18T19:57:02.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Juk8EtyCWHXwvP/bybzRlfZGqio=', 'Content-Length': '0'} 2023-10-18T19:57:02.265 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ded93b051402552a-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:57:02.267 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.268 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.269 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.270 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.270 DEBUG:boto:Method: GET 2023-10-18T19:57:02.271 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.271 DEBUG:boto:Data: 2023-10-18T19:57:02.272 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.272 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.273 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.273 DEBUG:boto:Params: {} 2023-10-18T19:57:02.274 DEBUG:boto:Token: None 2023-10-18T19:57:02.274 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.275 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM= 2023-10-18T19:57:02.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM=', 'Content-Length': '0'} 2023-10-18T19:57:02.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bfe1d09b8508f31b-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.277 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.278 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.279 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.280 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.281 DEBUG:boto:Method: GET 2023-10-18T19:57:02.281 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.282 DEBUG:boto:Data: 2023-10-18T19:57:02.283 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.284 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.284 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.285 DEBUG:boto:Params: {} 2023-10-18T19:57:02.286 DEBUG:boto:Token: None 2023-10-18T19:57:02.286 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.287 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM= 2023-10-18T19:57:02.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM=', 'Content-Length': '0'} 2023-10-18T19:57:02.289 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e526b11089ce5d74-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.290 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.290 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.291 DEBUG:boto:path=/bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.291 DEBUG:boto:auth_path=/bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.292 DEBUG:boto:Method: GET 2023-10-18T19:57:02.292 DEBUG:boto:Path: /bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.292 DEBUG:boto:Data: 2023-10-18T19:57:02.293 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.294 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.295 DEBUG:boto:Params: {} 2023-10-18T19:57:02.297 DEBUG:boto:Token: None 2023-10-18T19:57:02.298 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.299 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013596f7a5be63e68-006530388e-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.304 DEBUG:boto:b'bofrnf-12dummy1000falsedummy2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.305 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.305 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.306 DEBUG:boto:path=/bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.306 DEBUG:boto:auth_path=/bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.307 DEBUG:boto:Method: GET 2023-10-18T19:57:02.307 DEBUG:boto:Path: /bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.307 DEBUG:boto:Data: 2023-10-18T19:57:02.308 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.308 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.309 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.309 DEBUG:boto:Params: {} 2023-10-18T19:57:02.310 DEBUG:boto:Token: None 2023-10-18T19:57:02.310 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.311 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000269403ade4ab8384-006530388e-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.314 DEBUG:boto:b'bofrnf-12dummy1000falsedummy2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.315 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.315 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.316 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.316 DEBUG:boto:Path: /bofrnf-12/dummy 2023-10-18T19:57:02.316 DEBUG:boto:Data: 2023-10-18T19:57:02.317 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.317 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.318 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.318 DEBUG:boto:Params: {} 2023-10-18T19:57:02.318 DEBUG:boto:Token: None 2023-10-18T19:57:02.319 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy 2023-10-18T19:57:02.319 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ePp0p+/850QXKeyTe/wYLrnc7YM= 2023-10-18T19:57:02.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ePp0p+/850QXKeyTe/wYLrnc7YM=', 'Content-Length': '0'} 2023-10-18T19:57:02.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008fc108c2554a6eb8-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.323 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.323 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.324 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.324 DEBUG:boto:Path: /bofrnf-12/dummy 2023-10-18T19:57:02.324 DEBUG:boto:Data: 2023-10-18T19:57:02.325 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.325 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.325 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.326 DEBUG:boto:Params: {} 2023-10-18T19:57:02.326 DEBUG:boto:Token: None 2023-10-18T19:57:02.327 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy 2023-10-18T19:57:02.327 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ePp0p+/850QXKeyTe/wYLrnc7YM= 2023-10-18T19:57:02.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ePp0p+/850QXKeyTe/wYLrnc7YM=', 'Content-Length': '0'} 2023-10-18T19:57:02.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d76b598c33de3591-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:57:02.331 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.332 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.332 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.333 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.333 DEBUG:boto:Method: GET 2023-10-18T19:57:02.333 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.334 DEBUG:boto:Data: 2023-10-18T19:57:02.334 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.335 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.335 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.335 DEBUG:boto:Params: {} 2023-10-18T19:57:02.336 DEBUG:boto:Token: None 2023-10-18T19:57:02.336 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM= 2023-10-18T19:57:02.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM=', 'Content-Length': '0'} 2023-10-18T19:57:02.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7991ccd0c02977b-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.340 DEBUG:boto:path=/bofrnf-12/dummy 2023-10-18T19:57:02.340 DEBUG:boto:auth_path=/bofrnf-12/dummy 2023-10-18T19:57:02.341 DEBUG:boto:path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.341 DEBUG:boto:auth_path=/bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.342 DEBUG:boto:Method: GET 2023-10-18T19:57:02.342 DEBUG:boto:Path: /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.343 DEBUG:boto:Data: 2023-10-18T19:57:02.343 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.343 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.344 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.344 DEBUG:boto:Params: {} 2023-10-18T19:57:02.345 DEBUG:boto:Token: None 2023-10-18T19:57:02.345 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/dummy?versionId=ytzJEO2gslxPWNYVg6CTpsims4XGqOg 2023-10-18T19:57:02.346 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM= 2023-10-18T19:57:02.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UEMf7z4iY7DFXNU21/3fsVBiESM=', 'Content-Length': '0'} 2023-10-18T19:57:02.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:18 GMT'), ('x-amz-version-id', 'ytzJEO2gslxPWNYVg6CTpsims4XGqOg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf052802daf9eafe-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.349 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.349 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.350 DEBUG:boto:path=/bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.350 DEBUG:boto:auth_path=/bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.351 DEBUG:boto:Method: GET 2023-10-18T19:57:02.351 DEBUG:boto:Path: /bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.351 DEBUG:boto:Data: 2023-10-18T19:57:02.352 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.353 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.353 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.354 DEBUG:boto:Params: {} 2023-10-18T19:57:02.354 DEBUG:boto:Token: None 2023-10-18T19:57:02.355 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.355 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021bb1a567e908666-006530388e-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.402 DEBUG:boto:b'bofrnf-12dummy1000falsedummy2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.403 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.404 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.404 DEBUG:boto:path=/bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.405 DEBUG:boto:auth_path=/bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.405 DEBUG:boto:Method: GET 2023-10-18T19:57:02.406 DEBUG:boto:Path: /bofrnf-12/?prefix=dummy 2023-10-18T19:57:02.406 DEBUG:boto:Data: 2023-10-18T19:57:02.407 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.407 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.407 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.408 DEBUG:boto:Params: {} 2023-10-18T19:57:02.408 DEBUG:boto:Token: None 2023-10-18T19:57:02.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e77cbeabc148ab-006530388e-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.414 DEBUG:boto:b'bofrnf-12dummy1000falsedummy2023-10-18T19:55:18.552Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:57:02.415 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.416 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.416 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.417 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.417 DEBUG:boto:Method: GET 2023-10-18T19:57:02.418 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.419 DEBUG:boto:Data: 2023-10-18T19:57:02.419 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.419 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.420 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.420 DEBUG:boto:Params: {} 2023-10-18T19:57:02.421 DEBUG:boto:Token: None 2023-10-18T19:57:02.421 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.422 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8= 2023-10-18T19:57:02.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8=', 'Content-Length': '0'} 2023-10-18T19:57:02.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbc5aaba6d839734-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.424 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.425 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.425 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.426 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.427 DEBUG:boto:Method: GET 2023-10-18T19:57:02.427 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.428 DEBUG:boto:Data: 2023-10-18T19:57:02.428 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.428 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.429 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.429 DEBUG:boto:Params: {} 2023-10-18T19:57:02.430 DEBUG:boto:Token: None 2023-10-18T19:57:02.431 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.431 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8= 2023-10-18T19:57:02.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8=', 'Content-Length': '0'} 2023-10-18T19:57:02.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e487c6d90a044246-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.434 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.434 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.435 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.435 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.435 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.436 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.436 DEBUG:boto:Data: 2023-10-18T19:57:02.437 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.437 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.438 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.438 DEBUG:boto:Params: {} 2023-10-18T19:57:02.438 DEBUG:boto:Token: None 2023-10-18T19:57:02.439 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BAq3B+XebEeC+6vb9bj82lNHlz8= 2023-10-18T19:57:02.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BAq3B+XebEeC+6vb9bj82lNHlz8=', 'Content-Length': '0'} 2023-10-18T19:57:02.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbe35c99481e70d9-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.442 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.443 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.443 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.444 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.444 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.444 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.445 DEBUG:boto:Data: 2023-10-18T19:57:02.445 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.446 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.446 DEBUG:boto:Params: {} 2023-10-18T19:57:02.447 DEBUG:boto:Token: None 2023-10-18T19:57:02.447 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BAq3B+XebEeC+6vb9bj82lNHlz8= 2023-10-18T19:57:02.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BAq3B+XebEeC+6vb9bj82lNHlz8=', 'Content-Length': '0'} 2023-10-18T19:57:02.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000327e6f654b9aef-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:57:02.451 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.452 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.453 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.454 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.454 DEBUG:boto:Method: GET 2023-10-18T19:57:02.455 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.456 DEBUG:boto:Data: 2023-10-18T19:57:02.456 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.458 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.458 DEBUG:boto:Params: {} 2023-10-18T19:57:02.459 DEBUG:boto:Token: None 2023-10-18T19:57:02.460 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.461 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8= 2023-10-18T19:57:02.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8=', 'Content-Length': '0'} 2023-10-18T19:57:02.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5ea4850b8bdc653-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.464 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.464 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.465 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.465 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.466 DEBUG:boto:Method: GET 2023-10-18T19:57:02.466 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.467 DEBUG:boto:Data: 2023-10-18T19:57:02.467 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.467 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.468 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.468 DEBUG:boto:Params: {} 2023-10-18T19:57:02.469 DEBUG:boto:Token: None 2023-10-18T19:57:02.469 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.470 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8= 2023-10-18T19:57:02.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8=', 'Content-Length': '0'} 2023-10-18T19:57:02.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb32b42d8ede2adb-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.473 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.474 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.474 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.475 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.475 DEBUG:boto:Method: GET 2023-10-18T19:57:02.476 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.476 DEBUG:boto:Data: 2023-10-18T19:57:02.477 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.478 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.478 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.479 DEBUG:boto:Params: {} 2023-10-18T19:57:02.480 DEBUG:boto:Token: None 2023-10-18T19:57:02.481 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.482 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006100880f4fcc028-006530388e-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.486 DEBUG:boto:b'bofrnf-12obj-a11000falseobj-a12023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.487 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.487 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.488 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.488 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.489 DEBUG:boto:Method: GET 2023-10-18T19:57:02.489 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.490 DEBUG:boto:Data: 2023-10-18T19:57:02.490 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.491 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.491 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.492 DEBUG:boto:Params: {} 2023-10-18T19:57:02.492 DEBUG:boto:Token: None 2023-10-18T19:57:02.493 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.493 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1390dc9cc8fd1e6-006530388e-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.497 DEBUG:boto:b'bofrnf-12obj-a11000falseobj-a12023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.497 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.498 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.499 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.499 DEBUG:boto:Path: /bofrnf-12/obj-a1 2023-10-18T19:57:02.499 DEBUG:boto:Data: 2023-10-18T19:57:02.501 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.502 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.503 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.503 DEBUG:boto:Params: {} 2023-10-18T19:57:02.503 DEBUG:boto:Token: None 2023-10-18T19:57:02.504 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1 2023-10-18T19:57:02.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1BSz1u3doJTVm28yfgkvcpU3K/I= 2023-10-18T19:57:02.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1BSz1u3doJTVm28yfgkvcpU3K/I=', 'Content-Length': '0'} 2023-10-18T19:57:02.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a79beb7d3ca5a87-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.508 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.509 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.509 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.510 DEBUG:boto:Path: /bofrnf-12/obj-a1 2023-10-18T19:57:02.510 DEBUG:boto:Data: 2023-10-18T19:57:02.511 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.511 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.511 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.512 DEBUG:boto:Params: {} 2023-10-18T19:57:02.513 DEBUG:boto:Token: None 2023-10-18T19:57:02.513 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1 2023-10-18T19:57:02.513 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1BSz1u3doJTVm28yfgkvcpU3K/I= 2023-10-18T19:57:02.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1BSz1u3doJTVm28yfgkvcpU3K/I=', 'Content-Length': '0'} 2023-10-18T19:57:02.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c19d00ed4edee297-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-18T19:57:02.517 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.518 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.518 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.519 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.519 DEBUG:boto:Method: GET 2023-10-18T19:57:02.520 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.520 DEBUG:boto:Data: 2023-10-18T19:57:02.521 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.521 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.522 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.522 DEBUG:boto:Params: {} 2023-10-18T19:57:02.523 DEBUG:boto:Token: None 2023-10-18T19:57:02.523 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8= 2023-10-18T19:57:02.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8=', 'Content-Length': '0'} 2023-10-18T19:57:02.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f65ddff540c9e45-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.526 DEBUG:boto:path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.527 DEBUG:boto:auth_path=/bofrnf-12/obj-a1 2023-10-18T19:57:02.528 DEBUG:boto:path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.528 DEBUG:boto:auth_path=/bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.528 DEBUG:boto:Method: GET 2023-10-18T19:57:02.529 DEBUG:boto:Path: /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.529 DEBUG:boto:Data: 2023-10-18T19:57:02.530 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.531 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.531 DEBUG:boto:Params: {} 2023-10-18T19:57:02.532 DEBUG:boto:Token: None 2023-10-18T19:57:02.533 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a1?versionId=toth3ilqud.qHjmYoshz5E--Nxv2rQY 2023-10-18T19:57:02.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8= 2023-10-18T19:57:02.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:woSgfB3SVPbuIyvS2A/9SDpjmk8=', 'Content-Length': '0'} 2023-10-18T19:57:02.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:38 GMT'), ('x-amz-version-id', 'toth3ilqud.qHjmYoshz5E--Nxv2rQY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028fba0979bea9fa0-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.536 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.537 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.537 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.537 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.538 DEBUG:boto:Method: GET 2023-10-18T19:57:02.538 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.539 DEBUG:boto:Data: 2023-10-18T19:57:02.539 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.541 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.541 DEBUG:boto:Params: {} 2023-10-18T19:57:02.541 DEBUG:boto:Token: None 2023-10-18T19:57:02.542 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ac8297beb17316f-006530388e-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.546 DEBUG:boto:b'bofrnf-12obj-a11000falseobj-a12023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.547 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.547 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.548 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.548 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.549 DEBUG:boto:Method: GET 2023-10-18T19:57:02.549 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a1 2023-10-18T19:57:02.549 DEBUG:boto:Data: 2023-10-18T19:57:02.550 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.551 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.552 DEBUG:boto:Params: {} 2023-10-18T19:57:02.552 DEBUG:boto:Token: None 2023-10-18T19:57:02.552 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eaab8fa7a026b7c-006530388e-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.556 DEBUG:boto:b'bofrnf-12obj-a11000falseobj-a12023-10-18T19:55:38.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:57:02.557 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.558 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.558 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.559 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.559 DEBUG:boto:Method: GET 2023-10-18T19:57:02.560 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.560 DEBUG:boto:Data: 2023-10-18T19:57:02.561 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.561 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.562 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.562 DEBUG:boto:Params: {} 2023-10-18T19:57:02.563 DEBUG:boto:Token: None 2023-10-18T19:57:02.563 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE= 2023-10-18T19:57:02.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE=', 'Content-Length': '0'} 2023-10-18T19:57:02.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e0d5101004d4dc3-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.568 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.569 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.570 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.571 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.572 DEBUG:boto:Method: GET 2023-10-18T19:57:02.572 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.572 DEBUG:boto:Data: 2023-10-18T19:57:02.573 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.573 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.574 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.574 DEBUG:boto:Params: {} 2023-10-18T19:57:02.575 DEBUG:boto:Token: None 2023-10-18T19:57:02.575 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE= 2023-10-18T19:57:02.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE=', 'Content-Length': '0'} 2023-10-18T19:57:02.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c16ce73923d0585e-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.579 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.579 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.580 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.580 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.581 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.581 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.582 DEBUG:boto:Data: 2023-10-18T19:57:02.582 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.583 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.583 DEBUG:boto:Params: {} 2023-10-18T19:57:02.584 DEBUG:boto:Token: None 2023-10-18T19:57:02.584 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FazeBhMpodlVvhN8il5N74TbYEE= 2023-10-18T19:57:02.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FazeBhMpodlVvhN8il5N74TbYEE=', 'Content-Length': '0'} 2023-10-18T19:57:02.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000eef893de478b6e4-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.588 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.589 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.589 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.590 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.591 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.592 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.592 DEBUG:boto:Data: 2023-10-18T19:57:02.593 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.593 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.594 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.594 DEBUG:boto:Params: {} 2023-10-18T19:57:02.595 DEBUG:boto:Token: None 2023-10-18T19:57:02.595 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.596 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FazeBhMpodlVvhN8il5N74TbYEE= 2023-10-18T19:57:02.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FazeBhMpodlVvhN8il5N74TbYEE=', 'Content-Length': '0'} 2023-10-18T19:57:02.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d9c34a49e191db9-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:57:02.599 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.599 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.600 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.600 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.601 DEBUG:boto:Method: GET 2023-10-18T19:57:02.601 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.601 DEBUG:boto:Data: 2023-10-18T19:57:02.602 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.603 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.603 DEBUG:boto:Params: {} 2023-10-18T19:57:02.604 DEBUG:boto:Token: None 2023-10-18T19:57:02.604 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE= 2023-10-18T19:57:02.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE=', 'Content-Length': '0'} 2023-10-18T19:57:02.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000979df72ca927e112-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.607 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.608 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.608 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.609 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.609 DEBUG:boto:Method: GET 2023-10-18T19:57:02.610 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.610 DEBUG:boto:Data: 2023-10-18T19:57:02.611 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.611 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.611 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.612 DEBUG:boto:Params: {} 2023-10-18T19:57:02.612 DEBUG:boto:Token: None 2023-10-18T19:57:02.613 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE= 2023-10-18T19:57:02.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE=', 'Content-Length': '0'} 2023-10-18T19:57:02.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f83413f06545a5bf-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.658 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.659 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.659 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.660 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.660 DEBUG:boto:Method: GET 2023-10-18T19:57:02.661 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.661 DEBUG:boto:Data: 2023-10-18T19:57:02.661 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.662 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.663 DEBUG:boto:Params: {} 2023-10-18T19:57:02.663 DEBUG:boto:Token: None 2023-10-18T19:57:02.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4b2c019d3fbee1e-006530388e-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.668 DEBUG:boto:b'bofrnf-12obj-a21000falseobj-a22023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.668 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.669 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.669 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.669 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.670 DEBUG:boto:Method: GET 2023-10-18T19:57:02.670 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.671 DEBUG:boto:Data: 2023-10-18T19:57:02.671 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.672 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.672 DEBUG:boto:Params: {} 2023-10-18T19:57:02.672 DEBUG:boto:Token: None 2023-10-18T19:57:02.673 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6d377f6af872d2f-006530388e-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.677 DEBUG:boto:b'bofrnf-12obj-a21000falseobj-a22023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.677 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.678 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.678 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.678 DEBUG:boto:Path: /bofrnf-12/obj-a2 2023-10-18T19:57:02.679 DEBUG:boto:Data: 2023-10-18T19:57:02.679 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.680 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.680 DEBUG:boto:Params: {} 2023-10-18T19:57:02.681 DEBUG:boto:Token: None 2023-10-18T19:57:02.681 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2 2023-10-18T19:57:02.682 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+Im7HvHu+IlykhLek+hFLE3hvT4= 2023-10-18T19:57:02.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+Im7HvHu+IlykhLek+hFLE3hvT4=', 'Content-Length': '0'} 2023-10-18T19:57:02.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f4e36523269641d-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.686 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.686 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.687 DEBUG:boto:Method: HEAD 2023-10-18T19:57:02.687 DEBUG:boto:Path: /bofrnf-12/obj-a2 2023-10-18T19:57:02.687 DEBUG:boto:Data: 2023-10-18T19:57:02.688 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.688 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.688 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.689 DEBUG:boto:Params: {} 2023-10-18T19:57:02.689 DEBUG:boto:Token: None 2023-10-18T19:57:02.690 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2 2023-10-18T19:57:02.690 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+Im7HvHu+IlykhLek+hFLE3hvT4= 2023-10-18T19:57:02.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+Im7HvHu+IlykhLek+hFLE3hvT4=', 'Content-Length': '0'} 2023-10-18T19:57:02.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed5c2ac5468af18c-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-18T19:57:02.694 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.694 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.695 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.695 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.696 DEBUG:boto:Method: GET 2023-10-18T19:57:02.696 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.696 DEBUG:boto:Data: 2023-10-18T19:57:02.697 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.697 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.698 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.698 DEBUG:boto:Params: {} 2023-10-18T19:57:02.698 DEBUG:boto:Token: None 2023-10-18T19:57:02.699 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.699 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE= 2023-10-18T19:57:02.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE=', 'Content-Length': '0'} 2023-10-18T19:57:02.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f79ba1be36e8f37-006530388e-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.746 DEBUG:boto:path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.747 DEBUG:boto:auth_path=/bofrnf-12/obj-a2 2023-10-18T19:57:02.747 DEBUG:boto:path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.748 DEBUG:boto:auth_path=/bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.748 DEBUG:boto:Method: GET 2023-10-18T19:57:02.749 DEBUG:boto:Path: /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.749 DEBUG:boto:Data: 2023-10-18T19:57:02.750 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.750 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.750 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.751 DEBUG:boto:Params: {} 2023-10-18T19:57:02.751 DEBUG:boto:Token: None 2023-10-18T19:57:02.752 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/obj-a2?versionId=yipaUQxzrtbVwuxaybttOFpenBTsbAb 2023-10-18T19:57:02.752 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE= 2023-10-18T19:57:02.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ERclzgWuBFt7UduphwY6pxTTzrE=', 'Content-Length': '0'} 2023-10-18T19:57:02.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:55:39 GMT'), ('x-amz-version-id', 'yipaUQxzrtbVwuxaybttOFpenBTsbAb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a3166fa79fba914-006530388e-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.756 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.756 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.757 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.757 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.757 DEBUG:boto:Method: GET 2023-10-18T19:57:02.758 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.758 DEBUG:boto:Data: 2023-10-18T19:57:02.759 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.759 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:02.760 DEBUG:boto:Port: 8001 2023-10-18T19:57:02.760 DEBUG:boto:Params: {} 2023-10-18T19:57:02.760 DEBUG:boto:Token: None 2023-10-18T19:57:02.761 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.761 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc8b23f173a06b04-006530388e-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.765 DEBUG:boto:b'bofrnf-12obj-a21000falseobj-a22023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.765 DEBUG:boto:path=/bofrnf-12/ 2023-10-18T19:57:02.766 DEBUG:boto:auth_path=/bofrnf-12/ 2023-10-18T19:57:02.766 DEBUG:boto:path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.767 DEBUG:boto:auth_path=/bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.767 DEBUG:boto:Method: GET 2023-10-18T19:57:02.767 DEBUG:boto:Path: /bofrnf-12/?prefix=obj-a2 2023-10-18T19:57:02.768 DEBUG:boto:Data: 2023-10-18T19:57:02.768 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.769 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.770 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.770 DEBUG:boto:Params: {} 2023-10-18T19:57:02.771 DEBUG:boto:Token: None 2023-10-18T19:57:02.772 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-12/ 2023-10-18T19:57:02.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw= 2023-10-18T19:57:02.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRda5aVxz+MM3kPzMpVEi9N11Dw=', 'Content-Length': '0'} 2023-10-18T19:57:02.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016a9ff208ab1b9ca-006530388e-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.776 DEBUG:boto:b'bofrnf-12obj-a21000falseobj-a22023-10-18T19:55:39.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:02.776 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-12 zones={a2, a1} 2023-10-18T19:57:02.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-10-18T19:57:02.778 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:02.778 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:02.779 DEBUG:boto:Method: PUT 2023-10-18T19:57:02.779 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:02.780 DEBUG:boto:Data: 2023-10-18T19:57:02.780 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.781 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.782 DEBUG:boto:Params: {} 2023-10-18T19:57:02.782 DEBUG:boto:Token: None 2023-10-18T19:57:02.783 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-13/ 2023-10-18T19:57:02.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KQ8WbG+S5wmlLjn+IgzE0mCD8ZI= 2023-10-18T19:57:02.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KQ8WbG+S5wmlLjn+IgzE0mCD8ZI=', 'Content-Length': '0'} 2023-10-18T19:57:02.796 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000628f2847beffe881-006530388e-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.797 DEBUG:rgw_multi.tests:created bucket=bofrnf-13 2023-10-18T19:57:02.797 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:02.797 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:02.798 DEBUG:boto:path=/bofrnf-13/?versioning 2023-10-18T19:57:02.798 DEBUG:boto:auth_path=/bofrnf-13/?versioning 2023-10-18T19:57:02.799 DEBUG:boto:Method: PUT 2023-10-18T19:57:02.799 DEBUG:boto:Path: /bofrnf-13/?versioning 2023-10-18T19:57:02.799 DEBUG:boto:Data: Enabled Disabled 2023-10-18T19:57:02.800 DEBUG:boto:Headers: {} 2023-10-18T19:57:02.801 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:02.801 DEBUG:boto:Port: 8000 2023-10-18T19:57:02.802 DEBUG:boto:Params: {} 2023-10-18T19:57:02.802 DEBUG:boto:Token: None 2023-10-18T19:57:02.803 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:57:02 GMT /bofrnf-13/?versioning 2023-10-18T19:57:02.804 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3dyeQGLdNdtMeK8bgqitLou+kAw= 2023-10-18T19:57:02.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3dyeQGLdNdtMeK8bgqitLou+kAw=', 'Content-Length': '228'} 2023-10-18T19:57:02.810 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f0e64fd2fd6fd53a-006530388e-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:02.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:57:02.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:57:02.811 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:57:02.862 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:57:02.862 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:57:02.931 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:57:02.942 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:57:02.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:02.943 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659022.808590_191.1", 2023-10-18T19:57:02.943 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:57:02.808590Z" 2023-10-18T19:57:02.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:02.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:02.944 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:57:02.944 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T19:57:02.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:02.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:02.944 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:57:02.945 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:57:02.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:02.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:02.945 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:57:02.946 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:57:02.946 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:02.946 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:57:02.947 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659022.808590_191.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:57:02.947 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:57:02.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:57:02.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:57:02.949 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:57:02.997 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:57:02.998 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:57:03.024 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:03.022+0000 7f4121ad2780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:57:03.025 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:03.023+0000 7f4121ad2780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:57:03.036 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:57:03.036 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:57:03.036 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:57:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:57:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:57:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:57:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:03.038 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:57:03.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:03.038 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:57:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:57:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:57:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659022.788248_181.1", 2023-10-18T19:57:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:57:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:57:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:57:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:02.788248Z", 2023-10-18T19:57:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:03.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:03.041 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:57:03.041 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:57:03.041 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:57:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:57:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:57:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:57:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:57:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T19:57:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:57:03.044 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:57:03.044 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:57:03.044 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:57:03.044 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:57:03.044 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:57:03.045 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:57:03.045 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:57:03.045 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:03.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:03.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:03.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:03.046 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:57:03.046 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:57:03.047 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:57:03.047 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:57:03.047 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:57:03.047 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:57:03.047 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:57:03.048 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:57:03.048 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:03.048 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:03.048 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:03.048 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:57:03.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:03.049 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:57:03.049 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:57:03.049 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:57:03.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:03.050 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:57:03Z" 2023-10-18T19:57:03.050 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:57:03.051 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659022.788248_181.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:57:02.788248Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:57:03Z" } 2023-10-18T19:57:03.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659022.788248_181.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:57:02.788248Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:57:03.052 DEBUG:rgw_multi.tests:log_status={0: '1_1697659022.808590_191.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:57:03.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659022.788248_181.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:57:03.053 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659022.808590_191.1 target=1_1697659022.788248_181.1 2023-10-18T19:57:08.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:57:08.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:57:08.055 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:57:08.113 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:57:08.114 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:57:08.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:08.144+0000 7feb79c04780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:57:08.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:08.145+0000 7feb79c04780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:57:08.159 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:57:08.159 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:57:08.159 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:57:08.160 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:57:08.160 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:57:08.160 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:57:08.160 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:08.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:57:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:57:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:57:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:57:08.162 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659022.808590_191.1", 2023-10-18T19:57:08.162 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:57:08.162 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:57:08.162 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:57:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:02.808590Z", 2023-10-18T19:57:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:57:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:57:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:57:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:57:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:57:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:57:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:57:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T19:57:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:57:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:57:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:57:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:57:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:57:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:57:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:57:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:57:08.168 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:08.168 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:08.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:08.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:57:08.168 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:57:08.169 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:57:08.169 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:57:08.169 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:57:08.169 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:57:08.169 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:57:08.170 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:57:08.170 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:57:08.170 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:57:08.170 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:08.170 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:57:08.171 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:57:08.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:08.171 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:57:08.171 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:57:08.171 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:57:08.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:57:08.172 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:57:08Z" 2023-10-18T19:57:08.172 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:57:08.173 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659022.808590_191.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:57:02.808590Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:57:08Z" } 2023-10-18T19:57:08.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659022.808590_191.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:57:02.808590Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:57:08.174 DEBUG:rgw_multi.tests:log_status={0: '1_1697659022.808590_191.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:57:08.175 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659022.808590_191.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:57:08.175 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:57:08.176 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:08.176 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:08.176 DEBUG:boto:Method: HEAD 2023-10-18T19:57:08.177 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:08.177 DEBUG:boto:Data: 2023-10-18T19:57:08.178 DEBUG:boto:Headers: {} 2023-10-18T19:57:08.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:08.178 DEBUG:boto:Port: 8000 2023-10-18T19:57:08.179 DEBUG:boto:Params: {} 2023-10-18T19:57:08.179 DEBUG:boto:Token: None 2023-10-18T19:57:08.180 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:08 GMT /bofrnf-13/ 2023-10-18T19:57:08.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ACnxrpah533nqI8IToAX1nWCnNE= 2023-10-18T19:57:08.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ACnxrpah533nqI8IToAX1nWCnNE=', 'Content-Length': '0'} 2023-10-18T19:57:08.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcc62186f47094d8-0065303894-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:08.184 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:08.184 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:08.185 DEBUG:boto:Method: PUT 2023-10-18T19:57:08.185 DEBUG:boto:Path: /bofrnf-13/dummy 2023-10-18T19:57:08.186 DEBUG:boto:Data: 2023-10-18T19:57:08.186 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T19:57:08.186 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:08.187 DEBUG:boto:Port: 8000 2023-10-18T19:57:08.187 DEBUG:boto:Params: {} 2023-10-18T19:57:08.188 DEBUG:boto:Token: None 2023-10-18T19:57:08.188 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 19:57:08 GMT /bofrnf-13/dummy 2023-10-18T19:57:08.188 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pCQ91h87Jq/BPWvBH5+vEBkvXWU= 2023-10-18T19:57:08.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pCQ91h87Jq/BPWvBH5+vEBkvXWU='} 2023-10-18T19:57:08.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-amz-request-id', 'tx00000bbf35271d788f8ee-0065303894-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:08.200 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-13 2023-10-18T19:57:08.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:08.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:08.201 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:57:08.260 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:57:08.260 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:57:08.292 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:08.290+0000 7fbce933a780 1 waiting to reach incremental sync.. 2023-10-18T19:57:13.293 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:13.291+0000 7fbce933a780 1 waiting to reach incremental sync.. 2023-10-18T19:57:18.294 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:18.293+0000 7fbce933a780 1 bucket sync caught up with source: 2023-10-18T19:57:18.295 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2023-10-18T19:57:18.295 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.5.11, , , , , ] 2023-10-18T19:57:18.295 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:18.293+0000 7fbce933a780 0 bucket checkpoint complete 2023-10-18T19:57:18.310 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-13 2023-10-18T19:57:18.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:18.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:18.311 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:57:18.373 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:57:18.374 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:57:18.411 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:18.409+0000 7f44ab5d1780 1 waiting to reach incremental sync.. 2023-10-18T19:57:23.411 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:23.410+0000 7f44ab5d1780 1 waiting to reach incremental sync.. 2023-10-18T19:57:28.412 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:28.410+0000 7f44ab5d1780 1 waiting to reach incremental sync.. 2023-10-18T19:57:33.413 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:33.411+0000 7f44ab5d1780 1 waiting to reach incremental sync.. 2023-10-18T19:57:38.414 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:38.413+0000 7f44ab5d1780 1 bucket sync caught up with source: 2023-10-18T19:57:38.415 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , , , ] 2023-10-18T19:57:38.415 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , , , ] 2023-10-18T19:57:38.415 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:38.413+0000 7f44ab5d1780 0 bucket checkpoint complete 2023-10-18T19:57:38.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-13 zones={a2, a1} 2023-10-18T19:57:38.431 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.431 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.431 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.432 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.432 DEBUG:boto:Data: 2023-10-18T19:57:38.433 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.433 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.434 DEBUG:boto:Params: {} 2023-10-18T19:57:38.434 DEBUG:boto:Token: None 2023-10-18T19:57:38.435 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.435 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000338ac85f6aa78245-00653038b2-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.439 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.440 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.440 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.440 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.441 DEBUG:boto:Data: 2023-10-18T19:57:38.442 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.442 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.443 DEBUG:boto:Params: {} 2023-10-18T19:57:38.443 DEBUG:boto:Token: None 2023-10-18T19:57:38.444 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.444 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.447 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c48f0a188386ceb0-00653038b2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.448 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:57:38.448 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.449 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.449 DEBUG:boto:path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.450 DEBUG:boto:auth_path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.450 DEBUG:boto:Method: GET 2023-10-18T19:57:38.450 DEBUG:boto:Path: /bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.451 DEBUG:boto:Data: 2023-10-18T19:57:38.451 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.452 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.453 DEBUG:boto:Params: {} 2023-10-18T19:57:38.453 DEBUG:boto:Token: None 2023-10-18T19:57:38.454 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/?versions 2023-10-18T19:57:38.454 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vBh8JxLvKKq87eH9COlKHGCPGic= 2023-10-18T19:57:38.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vBh8JxLvKKq87eH9COlKHGCPGic=', 'Content-Length': '0'} 2023-10-18T19:57:38.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736cdc4892328518-00653038b2-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.458 DEBUG:boto:b'bofrnf-13999falsedummyILg3ajpYX4M9rwAnxDpJcCATEIMbxgBtrue2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:38.459 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:57:38.459 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:57:38.460 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.460 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.460 DEBUG:boto:path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.461 DEBUG:boto:auth_path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.461 DEBUG:boto:Method: GET 2023-10-18T19:57:38.462 DEBUG:boto:Path: /bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.462 DEBUG:boto:Data: 2023-10-18T19:57:38.462 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.463 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.463 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.464 DEBUG:boto:Params: {} 2023-10-18T19:57:38.464 DEBUG:boto:Token: None 2023-10-18T19:57:38.465 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/?versions 2023-10-18T19:57:38.465 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vBh8JxLvKKq87eH9COlKHGCPGic= 2023-10-18T19:57:38.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vBh8JxLvKKq87eH9COlKHGCPGic=', 'Content-Length': '0'} 2023-10-18T19:57:38.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a11dc440ce0c3a2-00653038b2-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.469 DEBUG:boto:b'bofrnf-13999falsedummyILg3ajpYX4M9rwAnxDpJcCATEIMbxgBtrue2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:38.470 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:57:38.470 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.471 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.471 DEBUG:boto:path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.471 DEBUG:boto:auth_path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.472 DEBUG:boto:Method: GET 2023-10-18T19:57:38.472 DEBUG:boto:Path: /bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.473 DEBUG:boto:Data: 2023-10-18T19:57:38.473 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.474 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.474 DEBUG:boto:Params: {} 2023-10-18T19:57:38.475 DEBUG:boto:Token: None 2023-10-18T19:57:38.475 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/?versions 2023-10-18T19:57:38.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vBh8JxLvKKq87eH9COlKHGCPGic= 2023-10-18T19:57:38.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vBh8JxLvKKq87eH9COlKHGCPGic=', 'Content-Length': '0'} 2023-10-18T19:57:38.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005540d835d3637fa7-00653038b2-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.479 DEBUG:boto:b'bofrnf-13999falsedummyILg3ajpYX4M9rwAnxDpJcCATEIMbxgBtrue2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:38.480 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.481 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.481 DEBUG:boto:path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.482 DEBUG:boto:auth_path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.482 DEBUG:boto:Method: GET 2023-10-18T19:57:38.482 DEBUG:boto:Path: /bofrnf-13/?versions&max-keys=999 2023-10-18T19:57:38.483 DEBUG:boto:Data: 2023-10-18T19:57:38.483 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.484 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.484 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.484 DEBUG:boto:Params: {} 2023-10-18T19:57:38.485 DEBUG:boto:Token: None 2023-10-18T19:57:38.485 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/?versions 2023-10-18T19:57:38.486 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vBh8JxLvKKq87eH9COlKHGCPGic= 2023-10-18T19:57:38.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vBh8JxLvKKq87eH9COlKHGCPGic=', 'Content-Length': '0'} 2023-10-18T19:57:38.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f1d047477aca164-00653038b2-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.490 DEBUG:boto:b'bofrnf-13999falsedummyILg3ajpYX4M9rwAnxDpJcCATEIMbxgBtrue2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:38.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:57:38.491 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.491 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.492 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.492 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.493 DEBUG:boto:Method: GET 2023-10-18T19:57:38.493 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.494 DEBUG:boto:Data: 2023-10-18T19:57:38.494 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.495 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.495 DEBUG:boto:Params: {} 2023-10-18T19:57:38.496 DEBUG:boto:Token: None 2023-10-18T19:57:38.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM= 2023-10-18T19:57:38.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM=', 'Content-Length': '0'} 2023-10-18T19:57:38.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b1d092a9217c005-00653038b2-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.501 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.501 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.501 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.502 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.503 DEBUG:boto:Method: GET 2023-10-18T19:57:38.503 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.504 DEBUG:boto:Data: 2023-10-18T19:57:38.504 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.504 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.505 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.505 DEBUG:boto:Params: {} 2023-10-18T19:57:38.506 DEBUG:boto:Token: None 2023-10-18T19:57:38.506 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.507 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM= 2023-10-18T19:57:38.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM=', 'Content-Length': '0'} 2023-10-18T19:57:38.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ed738156be98796-00653038b2-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.510 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.510 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.511 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.511 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.512 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.512 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.513 DEBUG:boto:Data: 2023-10-18T19:57:38.513 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.513 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.514 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.514 DEBUG:boto:Params: {} 2023-10-18T19:57:38.515 DEBUG:boto:Token: None 2023-10-18T19:57:38.515 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NykTWUwHx4Qi0tRCUVgenUtnsKs= 2023-10-18T19:57:38.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NykTWUwHx4Qi0tRCUVgenUtnsKs=', 'Content-Length': '0'} 2023-10-18T19:57:38.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d5e620b6975f240-00653038b2-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.519 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.519 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.520 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.520 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.520 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.521 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.521 DEBUG:boto:Data: 2023-10-18T19:57:38.522 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.522 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.523 DEBUG:boto:Params: {} 2023-10-18T19:57:38.523 DEBUG:boto:Token: None 2023-10-18T19:57:38.524 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NykTWUwHx4Qi0tRCUVgenUtnsKs= 2023-10-18T19:57:38.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NykTWUwHx4Qi0tRCUVgenUtnsKs=', 'Content-Length': '0'} 2023-10-18T19:57:38.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e979101e704205ef-00653038b2-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:57:38.528 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.528 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.529 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.529 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.530 DEBUG:boto:Method: GET 2023-10-18T19:57:38.530 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.531 DEBUG:boto:Data: 2023-10-18T19:57:38.531 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.532 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.532 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.532 DEBUG:boto:Params: {} 2023-10-18T19:57:38.533 DEBUG:boto:Token: None 2023-10-18T19:57:38.533 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.534 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM= 2023-10-18T19:57:38.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM=', 'Content-Length': '0'} 2023-10-18T19:57:38.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013dc385d6e80798c-00653038b2-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.537 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.538 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.538 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.539 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.539 DEBUG:boto:Method: GET 2023-10-18T19:57:38.539 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.540 DEBUG:boto:Data: 2023-10-18T19:57:38.540 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.541 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.542 DEBUG:boto:Params: {} 2023-10-18T19:57:38.542 DEBUG:boto:Token: None 2023-10-18T19:57:38.543 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM= 2023-10-18T19:57:38.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM=', 'Content-Length': '0'} 2023-10-18T19:57:38.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a8bfcf516e8b5c57-00653038b2-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.549 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.549 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.550 DEBUG:boto:path=/bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.550 DEBUG:boto:auth_path=/bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.550 DEBUG:boto:Method: GET 2023-10-18T19:57:38.551 DEBUG:boto:Path: /bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.551 DEBUG:boto:Data: 2023-10-18T19:57:38.552 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.553 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.553 DEBUG:boto:Params: {} 2023-10-18T19:57:38.554 DEBUG:boto:Token: None 2023-10-18T19:57:38.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6/YTL96Kbqk6tooNufIYCDlx5Kg= 2023-10-18T19:57:38.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6/YTL96Kbqk6tooNufIYCDlx5Kg=', 'Content-Length': '0'} 2023-10-18T19:57:38.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c57a270d974e8d73-00653038b2-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.598 DEBUG:boto:b'bofrnf-13dummy1000falsedummy2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:38.599 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.600 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.600 DEBUG:boto:path=/bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.601 DEBUG:boto:auth_path=/bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.601 DEBUG:boto:Method: GET 2023-10-18T19:57:38.602 DEBUG:boto:Path: /bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.602 DEBUG:boto:Data: 2023-10-18T19:57:38.603 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.603 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.603 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.604 DEBUG:boto:Params: {} 2023-10-18T19:57:38.604 DEBUG:boto:Token: None 2023-10-18T19:57:38.605 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.606 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6/YTL96Kbqk6tooNufIYCDlx5Kg= 2023-10-18T19:57:38.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6/YTL96Kbqk6tooNufIYCDlx5Kg=', 'Content-Length': '0'} 2023-10-18T19:57:38.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005227a37f08209a93-00653038b2-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.609 DEBUG:boto:b'bofrnf-13dummy1000falsedummy2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:38.610 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.610 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.611 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.611 DEBUG:boto:Path: /bofrnf-13/dummy 2023-10-18T19:57:38.612 DEBUG:boto:Data: 2023-10-18T19:57:38.612 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.612 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.613 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.613 DEBUG:boto:Params: {} 2023-10-18T19:57:38.614 DEBUG:boto:Token: None 2023-10-18T19:57:38.614 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy 2023-10-18T19:57:38.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F7vFHbrkBems1xTFh1prOsXwPs4= 2023-10-18T19:57:38.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F7vFHbrkBems1xTFh1prOsXwPs4=', 'Content-Length': '0'} 2023-10-18T19:57:38.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f001cc5ecc1d78c8-00653038b2-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.618 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.619 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.619 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.620 DEBUG:boto:Path: /bofrnf-13/dummy 2023-10-18T19:57:38.620 DEBUG:boto:Data: 2023-10-18T19:57:38.621 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.622 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.622 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.623 DEBUG:boto:Params: {} 2023-10-18T19:57:38.623 DEBUG:boto:Token: None 2023-10-18T19:57:38.623 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy 2023-10-18T19:57:38.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F7vFHbrkBems1xTFh1prOsXwPs4= 2023-10-18T19:57:38.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F7vFHbrkBems1xTFh1prOsXwPs4=', 'Content-Length': '0'} 2023-10-18T19:57:38.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8e4d8f14d2db946-00653038b2-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:57:38.628 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.629 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.629 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.629 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.630 DEBUG:boto:Method: GET 2023-10-18T19:57:38.630 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.631 DEBUG:boto:Data: 2023-10-18T19:57:38.631 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.632 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.632 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.633 DEBUG:boto:Params: {} 2023-10-18T19:57:38.633 DEBUG:boto:Token: None 2023-10-18T19:57:38.634 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.634 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM= 2023-10-18T19:57:38.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM=', 'Content-Length': '0'} 2023-10-18T19:57:38.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004201f60e157b9459-00653038b2-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.637 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:57:38.638 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:57:38.638 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.639 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.639 DEBUG:boto:Method: GET 2023-10-18T19:57:38.640 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.640 DEBUG:boto:Data: 2023-10-18T19:57:38.640 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.641 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.642 DEBUG:boto:Params: {} 2023-10-18T19:57:38.642 DEBUG:boto:Token: None 2023-10-18T19:57:38.643 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:57:38.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM= 2023-10-18T19:57:38.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yiy8DKEhsz59gxNY4YKLJ2QByUM=', 'Content-Length': '0'} 2023-10-18T19:57:38.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a74c2534530f15d2-00653038b2-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.647 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.648 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.648 DEBUG:boto:path=/bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.648 DEBUG:boto:auth_path=/bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.649 DEBUG:boto:Method: GET 2023-10-18T19:57:38.649 DEBUG:boto:Path: /bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.650 DEBUG:boto:Data: 2023-10-18T19:57:38.650 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.651 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.652 DEBUG:boto:Params: {} 2023-10-18T19:57:38.652 DEBUG:boto:Token: None 2023-10-18T19:57:38.653 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6/YTL96Kbqk6tooNufIYCDlx5Kg= 2023-10-18T19:57:38.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6/YTL96Kbqk6tooNufIYCDlx5Kg=', 'Content-Length': '0'} 2023-10-18T19:57:38.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca6c61caf96285cd-00653038b2-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.657 DEBUG:boto:b'bofrnf-13dummy1000falsedummy2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:38.658 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.658 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.658 DEBUG:boto:path=/bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.659 DEBUG:boto:auth_path=/bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.659 DEBUG:boto:Method: GET 2023-10-18T19:57:38.660 DEBUG:boto:Path: /bofrnf-13/?prefix=dummy 2023-10-18T19:57:38.660 DEBUG:boto:Data: 2023-10-18T19:57:38.661 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.662 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.662 DEBUG:boto:Params: {} 2023-10-18T19:57:38.663 DEBUG:boto:Token: None 2023-10-18T19:57:38.663 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6/YTL96Kbqk6tooNufIYCDlx5Kg= 2023-10-18T19:57:38.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6/YTL96Kbqk6tooNufIYCDlx5Kg=', 'Content-Length': '0'} 2023-10-18T19:57:38.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5a2c11c29dfdd2e-00653038b2-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.667 DEBUG:boto:b'bofrnf-13dummy1000falsedummy2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:57:38.668 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-13 zones={a2, a1} 2023-10-18T19:57:38.668 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.669 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.669 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.670 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.670 DEBUG:boto:Data: 2023-10-18T19:57:38.670 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.671 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.672 DEBUG:boto:Params: {} 2023-10-18T19:57:38.672 DEBUG:boto:Token: None 2023-10-18T19:57:38.672 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003077a8d0bdfa32c7-00653038b2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.676 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:38.677 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:38.677 DEBUG:boto:Method: PUT 2023-10-18T19:57:38.678 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:38.678 DEBUG:boto:Data: 2023-10-18T19:57:38.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:38.679 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.679 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.680 DEBUG:boto:Params: {} 2023-10-18T19:57:38.680 DEBUG:boto:Token: None 2023-10-18T19:57:38.681 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/obj 2023-10-18T19:57:38.681 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo= 2023-10-18T19:57:38.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo='} 2023-10-18T19:57:38.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v'), ('x-amz-request-id', 'tx00000053bae088d4eeef9-00653038b2-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.697 DEBUG:rgw_multi.tests:zone=a1 version=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:57:38.697 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.697 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.698 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.698 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.699 DEBUG:boto:Data: 2023-10-18T19:57:38.699 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.699 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.700 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.700 DEBUG:boto:Params: {} 2023-10-18T19:57:38.701 DEBUG:boto:Token: None 2023-10-18T19:57:38.701 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.702 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83b75220e24022c-00653038b2-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.705 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:38.705 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:38.706 DEBUG:boto:Method: PUT 2023-10-18T19:57:38.706 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:38.707 DEBUG:boto:Data: 2023-10-18T19:57:38.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:38.708 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.708 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.708 DEBUG:boto:Params: {} 2023-10-18T19:57:38.709 DEBUG:boto:Token: None 2023-10-18T19:57:38.709 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/obj 2023-10-18T19:57:38.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo= 2023-10-18T19:57:38.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo='} 2023-10-18T19:57:38.723 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz'), ('x-amz-request-id', 'tx00000b1ab5e788ce3bbc3-00653038b2-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.724 DEBUG:rgw_multi.tests:zone=a2 version=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:57:38.724 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.725 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.725 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.725 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.726 DEBUG:boto:Data: 2023-10-18T19:57:38.726 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.727 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.727 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.728 DEBUG:boto:Params: {} 2023-10-18T19:57:38.728 DEBUG:boto:Token: None 2023-10-18T19:57:38.728 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.729 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039d8dd26b1dba0e8-00653038b2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.732 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:38.733 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:38.733 DEBUG:boto:Method: PUT 2023-10-18T19:57:38.734 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:38.734 DEBUG:boto:Data: 2023-10-18T19:57:38.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:38.735 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.735 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.736 DEBUG:boto:Params: {} 2023-10-18T19:57:38.736 DEBUG:boto:Token: None 2023-10-18T19:57:38.737 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/obj 2023-10-18T19:57:38.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo= 2023-10-18T19:57:38.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo='} 2023-10-18T19:57:38.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg'), ('x-amz-request-id', 'tx000005b1afd59a573ae10-00653038b2-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.783 DEBUG:rgw_multi.tests:zone=a1 version=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:57:38.784 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.784 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.785 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.785 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.785 DEBUG:boto:Data: 2023-10-18T19:57:38.786 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.786 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.787 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.787 DEBUG:boto:Params: {} 2023-10-18T19:57:38.787 DEBUG:boto:Token: None 2023-10-18T19:57:38.788 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.788 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7ffe791237b2e7c-00653038b2-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.792 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:38.792 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:38.793 DEBUG:boto:Method: PUT 2023-10-18T19:57:38.793 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:38.794 DEBUG:boto:Data: 2023-10-18T19:57:38.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:38.795 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.795 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.795 DEBUG:boto:Params: {} 2023-10-18T19:57:38.796 DEBUG:boto:Token: None 2023-10-18T19:57:38.796 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/obj 2023-10-18T19:57:38.797 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo= 2023-10-18T19:57:38.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo='} 2023-10-18T19:57:38.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'oyyerD918wRExgeuwLQNphank4bD2k3'), ('x-amz-request-id', 'tx00000b5559aedb90959fd-00653038b2-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.809 DEBUG:rgw_multi.tests:zone=a2 version=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:57:38.809 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.809 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.810 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.810 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.811 DEBUG:boto:Data: 2023-10-18T19:57:38.811 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.811 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.812 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.812 DEBUG:boto:Params: {} 2023-10-18T19:57:38.813 DEBUG:boto:Token: None 2023-10-18T19:57:38.813 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.814 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002827fa8490c0e449-00653038b2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.818 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:38.818 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:38.819 DEBUG:boto:Method: PUT 2023-10-18T19:57:38.819 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:38.819 DEBUG:boto:Data: 2023-10-18T19:57:38.820 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:38.820 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.821 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.821 DEBUG:boto:Params: {} 2023-10-18T19:57:38.821 DEBUG:boto:Token: None 2023-10-18T19:57:38.822 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/obj 2023-10-18T19:57:38.822 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo= 2023-10-18T19:57:38.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo='} 2023-10-18T19:57:38.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6s7LnG-qRPtK-UCaUqeTxtDibOflrUd'), ('x-amz-request-id', 'tx00000650da2583005a555-00653038b2-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.834 DEBUG:rgw_multi.tests:zone=a1 version=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:57:38.835 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.835 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.835 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.836 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.836 DEBUG:boto:Data: 2023-10-18T19:57:38.837 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.837 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.838 DEBUG:boto:Params: {} 2023-10-18T19:57:38.838 DEBUG:boto:Token: None 2023-10-18T19:57:38.839 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.839 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ea2bc2b8155fe54-00653038b2-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.843 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:38.843 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:38.843 DEBUG:boto:Method: PUT 2023-10-18T19:57:38.844 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:38.844 DEBUG:boto:Data: 2023-10-18T19:57:38.844 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:38.845 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.845 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.846 DEBUG:boto:Params: {} 2023-10-18T19:57:38.846 DEBUG:boto:Token: None 2023-10-18T19:57:38.847 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/obj 2023-10-18T19:57:38.847 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo= 2023-10-18T19:57:38.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo='} 2023-10-18T19:57:38.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NTvDWcfE9kVoyznfhCGU6ELa04H.jwL'), ('x-amz-request-id', 'tx00000d7169b693e13561c-00653038b2-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.891 DEBUG:rgw_multi.tests:zone=a2 version=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:57:38.892 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.892 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.893 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.893 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.894 DEBUG:boto:Data: 2023-10-18T19:57:38.894 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.894 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.895 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.895 DEBUG:boto:Params: {} 2023-10-18T19:57:38.896 DEBUG:boto:Token: None 2023-10-18T19:57:38.896 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.897 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081d2e94374570712-00653038b2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.900 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:38.900 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:38.900 DEBUG:boto:Method: PUT 2023-10-18T19:57:38.901 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:38.901 DEBUG:boto:Data: 2023-10-18T19:57:38.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:38.902 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.903 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.903 DEBUG:boto:Params: {} 2023-10-18T19:57:38.903 DEBUG:boto:Token: None 2023-10-18T19:57:38.904 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/obj 2023-10-18T19:57:38.904 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo= 2023-10-18T19:57:38.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo='} 2023-10-18T19:57:38.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'egd1qNK-ZKDNN5bN4A1VaSs97JSE297'), ('x-amz-request-id', 'tx00000bac278cd000a9f03-00653038b2-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.915 DEBUG:rgw_multi.tests:zone=a1 version=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:57:38.915 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.915 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.916 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.916 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.917 DEBUG:boto:Data: 2023-10-18T19:57:38.917 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.918 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.918 DEBUG:boto:Params: {} 2023-10-18T19:57:38.919 DEBUG:boto:Token: None 2023-10-18T19:57:38.919 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d261df8ee5c7b1e-00653038b2-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.923 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:38.923 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:38.924 DEBUG:boto:Method: PUT 2023-10-18T19:57:38.924 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:38.925 DEBUG:boto:Data: 2023-10-18T19:57:38.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:38.926 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.926 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.926 DEBUG:boto:Params: {} 2023-10-18T19:57:38.927 DEBUG:boto:Token: None 2023-10-18T19:57:38.927 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/obj 2023-10-18T19:57:38.928 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo= 2023-10-18T19:57:38.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo='} 2023-10-18T19:57:38.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n'), ('x-amz-request-id', 'tx000000ad6058ba6b44996-00653038b2-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.938 DEBUG:rgw_multi.tests:zone=a2 version=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:57:38.939 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.939 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.939 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.940 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.940 DEBUG:boto:Data: 2023-10-18T19:57:38.941 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.941 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.942 DEBUG:boto:Params: {} 2023-10-18T19:57:38.942 DEBUG:boto:Token: None 2023-10-18T19:57:38.943 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/ 2023-10-18T19:57:38.943 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ= 2023-10-18T19:57:38.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0nbWZDyBxvVYVjaJfsUNUO4ZyBQ=', 'Content-Length': '0'} 2023-10-18T19:57:38.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b5743a48ed81f56-00653038b2-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.947 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:38.947 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:38.947 DEBUG:boto:Method: PUT 2023-10-18T19:57:38.948 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:38.948 DEBUG:boto:Data: 2023-10-18T19:57:38.949 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:38.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:38.950 DEBUG:boto:Port: 8000 2023-10-18T19:57:38.950 DEBUG:boto:Params: {} 2023-10-18T19:57:38.950 DEBUG:boto:Token: None 2023-10-18T19:57:38.951 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:38 GMT /bofrnf-13/obj 2023-10-18T19:57:38.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo= 2023-10-18T19:57:38.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kt4yvCZBHr68p89Cte+tURpjzlo='} 2023-10-18T19:57:38.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'U42vKDwj1WFjI-bMrhPe5O7B-9dabl9'), ('x-amz-request-id', 'tx00000f2ff464c5ec74300-00653038b2-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:38.995 DEBUG:rgw_multi.tests:zone=a1 version=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:57:38.996 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:38.996 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:38.996 DEBUG:boto:Method: HEAD 2023-10-18T19:57:38.997 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:38.997 DEBUG:boto:Data: 2023-10-18T19:57:38.998 DEBUG:boto:Headers: {} 2023-10-18T19:57:38.998 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:38.998 DEBUG:boto:Port: 8001 2023-10-18T19:57:38.999 DEBUG:boto:Params: {} 2023-10-18T19:57:38.999 DEBUG:boto:Token: None 2023-10-18T19:57:39.000 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.000 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044ca5661ecb229d6-00653038b3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.003 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.004 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.004 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.005 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.005 DEBUG:boto:Data: 2023-10-18T19:57:39.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.006 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.007 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.007 DEBUG:boto:Params: {} 2023-10-18T19:57:39.007 DEBUG:boto:Token: None 2023-10-18T19:57:39.008 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.008 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v'), ('x-amz-request-id', 'tx0000072e2221cecd47177-00653038b3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.059 DEBUG:rgw_multi.tests:zone=a2 version=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:57:39.060 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.060 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.060 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.061 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.061 DEBUG:boto:Data: 2023-10-18T19:57:39.062 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.062 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.062 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.063 DEBUG:boto:Params: {} 2023-10-18T19:57:39.063 DEBUG:boto:Token: None 2023-10-18T19:57:39.064 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.064 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b911713f6c0105e0-00653038b3-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.068 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.068 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.069 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.069 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.069 DEBUG:boto:Data: 2023-10-18T19:57:39.070 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.070 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.071 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.071 DEBUG:boto:Params: {} 2023-10-18T19:57:39.072 DEBUG:boto:Token: None 2023-10-18T19:57:39.072 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.072 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6'), ('x-amz-request-id', 'tx00000ad7499fbce8335f4-00653038b3-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.082 DEBUG:rgw_multi.tests:zone=a1 version=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:57:39.083 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.083 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.083 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.084 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.084 DEBUG:boto:Data: 2023-10-18T19:57:39.085 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.085 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.085 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.086 DEBUG:boto:Params: {} 2023-10-18T19:57:39.086 DEBUG:boto:Token: None 2023-10-18T19:57:39.087 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.087 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb5558f2f333704c-00653038b3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.091 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.091 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.091 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.092 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.092 DEBUG:boto:Data: 2023-10-18T19:57:39.093 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.093 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.093 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.094 DEBUG:boto:Params: {} 2023-10-18T19:57:39.094 DEBUG:boto:Token: None 2023-10-18T19:57:39.095 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.095 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'm--RJO4V9Vrua4FSRFh-ASpvQgROG6k'), ('x-amz-request-id', 'tx00000e2417ef26ae8422a-00653038b3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.107 DEBUG:rgw_multi.tests:zone=a2 version=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:57:39.107 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.107 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.108 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.108 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.109 DEBUG:boto:Data: 2023-10-18T19:57:39.109 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.110 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.110 DEBUG:boto:Params: {} 2023-10-18T19:57:39.111 DEBUG:boto:Token: None 2023-10-18T19:57:39.111 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.112 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0af504868c562f1-00653038b3-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.115 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.115 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.116 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.116 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.116 DEBUG:boto:Data: 2023-10-18T19:57:39.117 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.117 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.118 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.118 DEBUG:boto:Params: {} 2023-10-18T19:57:39.118 DEBUG:boto:Token: None 2023-10-18T19:57:39.119 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.119 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'GBljJZwndU8OOxk7KJyahMn7aN2joxp'), ('x-amz-request-id', 'tx00000fc7d25af067946b5-00653038b3-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.167 DEBUG:rgw_multi.tests:zone=a1 version=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:57:39.168 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.168 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.168 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.169 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.169 DEBUG:boto:Data: 2023-10-18T19:57:39.170 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.170 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.170 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.171 DEBUG:boto:Params: {} 2023-10-18T19:57:39.171 DEBUG:boto:Token: None 2023-10-18T19:57:39.172 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.172 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.175 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c50f14452dbe8590-00653038b3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.175 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.176 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.176 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.177 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.177 DEBUG:boto:Data: 2023-10-18T19:57:39.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.178 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.179 DEBUG:boto:Params: {} 2023-10-18T19:57:39.179 DEBUG:boto:Token: None 2023-10-18T19:57:39.180 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.193 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ'), ('x-amz-request-id', 'tx000009cbba00dd125da5b-00653038b3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.193 DEBUG:rgw_multi.tests:zone=a2 version=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:57:39.194 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.194 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.195 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.195 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.195 DEBUG:boto:Data: 2023-10-18T19:57:39.196 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.196 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.196 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.197 DEBUG:boto:Params: {} 2023-10-18T19:57:39.197 DEBUG:boto:Token: None 2023-10-18T19:57:39.198 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.198 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.201 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000559a7e334d3e33ca-00653038b3-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.202 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.202 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.203 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.203 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.204 DEBUG:boto:Data: 2023-10-18T19:57:39.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.204 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.205 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.205 DEBUG:boto:Params: {} 2023-10-18T19:57:39.206 DEBUG:boto:Token: None 2023-10-18T19:57:39.206 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.207 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj'), ('x-amz-request-id', 'tx000000aa90d4116a6645f-00653038b3-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.217 DEBUG:rgw_multi.tests:zone=a1 version=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:57:39.217 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.218 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.218 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.219 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.219 DEBUG:boto:Data: 2023-10-18T19:57:39.220 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.220 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.221 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.221 DEBUG:boto:Params: {} 2023-10-18T19:57:39.222 DEBUG:boto:Token: None 2023-10-18T19:57:39.222 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.223 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c54c41778d8e6de8-00653038b3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.226 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.227 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.227 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.228 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.228 DEBUG:boto:Data: 2023-10-18T19:57:39.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.229 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.230 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.230 DEBUG:boto:Params: {} 2023-10-18T19:57:39.231 DEBUG:boto:Token: None 2023-10-18T19:57:39.231 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'obRs9QrejEnMKy2WaXOLHr.iYfq1qDe'), ('x-amz-request-id', 'tx00000fbeb8d385b5e905f-00653038b3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.275 DEBUG:rgw_multi.tests:zone=a2 version=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:57:39.275 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.276 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.276 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.277 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.277 DEBUG:boto:Data: 2023-10-18T19:57:39.277 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.278 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.278 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.279 DEBUG:boto:Params: {} 2023-10-18T19:57:39.279 DEBUG:boto:Token: None 2023-10-18T19:57:39.280 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.280 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047788e9b59d3d9b5-00653038b3-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.283 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.284 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.284 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.285 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.285 DEBUG:boto:Data: 2023-10-18T19:57:39.286 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.286 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.287 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.287 DEBUG:boto:Params: {} 2023-10-18T19:57:39.287 DEBUG:boto:Token: None 2023-10-18T19:57:39.288 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.288 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn'), ('x-amz-request-id', 'tx0000048e6ee23be9e5222-00653038b3-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.300 DEBUG:rgw_multi.tests:zone=a1 version=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:57:39.300 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.301 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.301 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.302 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.302 DEBUG:boto:Data: 2023-10-18T19:57:39.302 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.303 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.303 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.304 DEBUG:boto:Params: {} 2023-10-18T19:57:39.304 DEBUG:boto:Token: None 2023-10-18T19:57:39.304 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.309 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049e9483cbd672965-00653038b3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.310 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.310 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.311 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.311 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.311 DEBUG:boto:Data: 2023-10-18T19:57:39.312 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.313 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.313 DEBUG:boto:Params: {} 2023-10-18T19:57:39.314 DEBUG:boto:Token: None 2023-10-18T19:57:39.314 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X'), ('x-amz-request-id', 'tx00000092e182e3db2d25c-00653038b3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.325 DEBUG:rgw_multi.tests:zone=a2 version=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:57:39.326 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.326 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.326 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.327 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.327 DEBUG:boto:Data: 2023-10-18T19:57:39.327 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.328 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.329 DEBUG:boto:Params: {} 2023-10-18T19:57:39.329 DEBUG:boto:Token: None 2023-10-18T19:57:39.330 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1fb5f5394740c63-00653038b3-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.334 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.334 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.334 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.335 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.335 DEBUG:boto:Data: 2023-10-18T19:57:39.336 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.336 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:57:39.336 DEBUG:boto:Port: 8000 2023-10-18T19:57:39.337 DEBUG:boto:Params: {} 2023-10-18T19:57:39.337 DEBUG:boto:Token: None 2023-10-18T19:57:39.338 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.338 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o'), ('x-amz-request-id', 'tx000000ab870f5fb1ed5c1-00653038b3-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.387 DEBUG:rgw_multi.tests:zone=a1 version=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:57:39.388 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:57:39.388 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:57:39.388 DEBUG:boto:Method: HEAD 2023-10-18T19:57:39.389 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:57:39.389 DEBUG:boto:Data: 2023-10-18T19:57:39.390 DEBUG:boto:Headers: {} 2023-10-18T19:57:39.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.390 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.391 DEBUG:boto:Params: {} 2023-10-18T19:57:39.391 DEBUG:boto:Token: None 2023-10-18T19:57:39.392 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/ 2023-10-18T19:57:39.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ= 2023-10-18T19:57:39.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jNua6OJRh/Q3r+TVy4H8238McaQ=', 'Content-Length': '0'} 2023-10-18T19:57:39.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041c8579b08beede0-00653038b3-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.395 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:57:39.396 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:57:39.396 DEBUG:boto:Method: PUT 2023-10-18T19:57:39.397 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:57:39.397 DEBUG:boto:Data: 2023-10-18T19:57:39.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-18T19:57:39.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:57:39.398 DEBUG:boto:Port: 8001 2023-10-18T19:57:39.399 DEBUG:boto:Params: {} 2023-10-18T19:57:39.399 DEBUG:boto:Token: None 2023-10-18T19:57:39.399 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 18 Oct 2023 19:57:39 GMT /bofrnf-13/obj 2023-10-18T19:57:39.400 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc= 2023-10-18T19:57:39.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:57:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4D5wHGLpwELoJN8wzG9JyhOvDTc='} 2023-10-18T19:57:39.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-amz-request-id', 'tx00000613708dd437352d7-00653038b3-1116-a2'), ('Date', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:57:39.409 DEBUG:rgw_multi.tests:zone=a2 version=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:57:39.409 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-13 2023-10-18T19:57:39.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:39.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:57:39.411 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:57:39.467 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:57:39.468 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:57:39.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:39.499+0000 7fbafb567780 1 waiting for incremental sync to catch up: 2023-10-18T19:57:39.502 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2023-10-18T19:57:39.502 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.46.12, , ] 2023-10-18T19:57:44.502 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:44.501+0000 7fbafb567780 1 waiting for incremental sync to catch up: 2023-10-18T19:57:44.505 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2023-10-18T19:57:44.505 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.46.12, , ] 2023-10-18T19:57:49.503 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:49.501+0000 7fbafb567780 1 waiting for incremental sync to catch up: 2023-10-18T19:57:49.504 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2023-10-18T19:57:49.504 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.46.12, , ] 2023-10-18T19:57:54.505 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:54.502+0000 7fbafb567780 1 waiting for incremental sync to catch up: 2023-10-18T19:57:54.505 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2023-10-18T19:57:54.505 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.46.12, , ] 2023-10-18T19:57:59.506 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:57:59.503+0000 7fbafb567780 1 waiting for incremental sync to catch up: 2023-10-18T19:57:59.507 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000011.29.6, , ] 2023-10-18T19:57:59.507 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.46.12, , ] 2023-10-18T19:58:04.507 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:04.506+0000 7fbafb567780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:04.508 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000011.29.6, , ] 2023-10-18T19:58:04.508 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.46.12, , ] 2023-10-18T19:58:09.509 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:09.507+0000 7fbafb567780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:09.510 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000011.29.6, , ] 2023-10-18T19:58:09.510 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.46.12, , ] 2023-10-18T19:58:14.510 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:14.508+0000 7fbafb567780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:14.511 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000011.29.6, , ] 2023-10-18T19:58:14.511 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.46.12, , ] 2023-10-18T19:58:19.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:19.510+0000 7fbafb567780 1 bucket sync caught up with source: 2023-10-18T19:58:19.512 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000031.77.12, , ] 2023-10-18T19:58:19.512 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.46.12, , ] 2023-10-18T19:58:19.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:19.510+0000 7fbafb567780 0 bucket checkpoint complete 2023-10-18T19:58:19.527 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-13 2023-10-18T19:58:19.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:58:19.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:58:19.529 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:58:19.591 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:58:19.592 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:58:19.629 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:19.627+0000 7f024217b780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:19.629 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , 00000000026.72.6, , ] 2023-10-18T19:58:19.629 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:24.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:24.628+0000 7f024217b780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:24.631 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , 00000000026.72.6, , ] 2023-10-18T19:58:24.631 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:29.631 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:29.630+0000 7f024217b780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:29.632 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , 00000000026.72.6, , ] 2023-10-18T19:58:29.632 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:34.632 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:34.631+0000 7f024217b780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:34.633 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , 00000000026.72.6, , ] 2023-10-18T19:58:34.633 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:39.634 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:39.631+0000 7f024217b780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:39.634 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , 00000000026.72.6, , ] 2023-10-18T19:58:39.634 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:44.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:44.632+0000 7f024217b780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:44.635 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , 00000000026.72.6, , ] 2023-10-18T19:58:44.636 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:49.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:49.633+0000 7f024217b780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:49.637 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , 00000000026.72.6, , ] 2023-10-18T19:58:49.637 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:54.637 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:54.636+0000 7f024217b780 1 waiting for incremental sync to catch up: 2023-10-18T19:58:54.638 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , 00000000026.72.6, , ] 2023-10-18T19:58:54.638 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:59.638 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:59.637+0000 7f024217b780 1 bucket sync caught up with source: 2023-10-18T19:58:59.639 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:59.639 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000002.13.11, , , 00000000031.85.9, , ] 2023-10-18T19:58:59.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:58:59.637+0000 7f024217b780 0 bucket checkpoint complete 2023-10-18T19:58:59.654 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-13 zones={a2, a1} 2023-10-18T19:58:59.655 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.655 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.656 DEBUG:boto:Method: HEAD 2023-10-18T19:58:59.656 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:58:59.657 DEBUG:boto:Data: 2023-10-18T19:58:59.657 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.657 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.658 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.658 DEBUG:boto:Params: {} 2023-10-18T19:58:59.659 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T19:58:59.660 DEBUG:boto:Token: None 2023-10-18T19:58:59.660 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/ 2023-10-18T19:58:59.661 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dYfy9/hOpPh7cIOv+dmtAoz3iFo= 2023-10-18T19:58:59.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dYfy9/hOpPh7cIOv+dmtAoz3iFo=', 'Content-Length': '0'} 2023-10-18T19:58:59.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022d9c079807076aa-0065303903-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.667 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.667 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.668 DEBUG:boto:Method: HEAD 2023-10-18T19:58:59.668 DEBUG:boto:Path: /bofrnf-13/ 2023-10-18T19:58:59.669 DEBUG:boto:Data: 2023-10-18T19:58:59.669 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.669 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.670 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.670 DEBUG:boto:Params: {} 2023-10-18T19:58:59.671 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T19:58:59.671 DEBUG:boto:Token: None 2023-10-18T19:58:59.672 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/ 2023-10-18T19:58:59.672 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dYfy9/hOpPh7cIOv+dmtAoz3iFo= 2023-10-18T19:58:59.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dYfy9/hOpPh7cIOv+dmtAoz3iFo=', 'Content-Length': '0'} 2023-10-18T19:58:59.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcb98ef301cd809b-0065303903-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.677 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:58:59.677 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.678 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.678 DEBUG:boto:path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.679 DEBUG:boto:auth_path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.679 DEBUG:boto:Method: GET 2023-10-18T19:58:59.679 DEBUG:boto:Path: /bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.680 DEBUG:boto:Data: 2023-10-18T19:58:59.680 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.681 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.681 DEBUG:boto:Params: {} 2023-10-18T19:58:59.682 DEBUG:boto:Token: None 2023-10-18T19:58:59.682 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/?versions 2023-10-18T19:58:59.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:crzKNezoCOmUG8W9MoYgDqM0Om0= 2023-10-18T19:58:59.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:crzKNezoCOmUG8W9MoYgDqM0Om0=', 'Content-Length': '0'} 2023-10-18T19:58:59.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d22889d3fe42bb56-0065303903-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.687 DEBUG:boto:b'bofrnf-13999falsedummyILg3ajpYX4M9rwAnxDpJcCATEIMbxgBtrue2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ6voUTCwzJD04f4D04wP01totCTFrWPtrue2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz9BErq0CUM4rDKx8n4IUCoMdTT-iq6ofalse2023-10-18T19:57:39.339Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHx.Z0UF.X7M99L16W2LmnrMzRT52wSnfalse2023-10-18T19:57:39.290Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOZ8sYbxHfpd4TmnYec31NLGZCLDDZ9Xfalse2023-10-18T19:57:39.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXwY9HQwGVPJQ92uscYuFYsIdbQgXEYjfalse2023-10-18T19:57:39.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobRs9QrejEnMKy2WaXOLHr.iYfq1qDefalse2023-10-18T19:57:39.233Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGBljJZwndU8OOxk7KJyahMn7aN2joxpfalse2023-10-18T19:57:39.121Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqx9JNJ-cAf2CB7fwdF8n.40VRiX85wJfalse2023-10-18T19:57:39.182Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6false2023-10-18T19:57:39.074Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm--RJO4V9Vrua4FSRFh-ASpvQgROG6kfalse2023-10-18T19:57:39.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU42vKDwj1WFjI-bMrhPe5O7B-9dabl9false2023-10-18T19:57:38.953Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqjJoJ0JywLBVngkF4NqtHZRMQQpSi7vfalse2023-10-18T19:57:39.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8nfalse2023-10-18T19:57:38.929Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjegd1qNK-ZKDNN5bN4A1VaSs97JSE297false2023-10-18T19:57:38.906Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6s7LnG-qRPtK-UCaUqeTxtDibOflrUdfalse2023-10-18T19:57:38.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNTvDWcfE9kVoyznfhCGU6ELa04H.jwLfalse2023-10-18T19:57:38.848Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG59arbhnbpCc-hwP0YJ8EXQMYoW5oYgfalse2023-10-18T19:57:38.739Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoyyerD918wRExgeuwLQNphank4bD2k3false2023-10-18T19:57:38.798Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBH2jE4UqtGp8OfC1t9VV-x6wQuT5PZzfalse2023-10-18T19:57:38.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNAVmL-UuMH5Lyboppq2UUDWFWO-Ud0vfalse2023-10-18T19:57:38.682Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:58:59.688 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:58:59.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.696 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:58:59.696 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.697 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.697 DEBUG:boto:path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.698 DEBUG:boto:auth_path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.698 DEBUG:boto:Method: GET 2023-10-18T19:58:59.699 DEBUG:boto:Path: /bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.699 DEBUG:boto:Data: 2023-10-18T19:58:59.700 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.700 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.700 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.701 DEBUG:boto:Params: {} 2023-10-18T19:58:59.701 DEBUG:boto:Token: None 2023-10-18T19:58:59.702 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/?versions 2023-10-18T19:58:59.702 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:crzKNezoCOmUG8W9MoYgDqM0Om0= 2023-10-18T19:58:59.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:crzKNezoCOmUG8W9MoYgDqM0Om0=', 'Content-Length': '0'} 2023-10-18T19:58:59.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000512d3e811e116e26-0065303903-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.706 DEBUG:boto:b'bofrnf-13999falsedummyILg3ajpYX4M9rwAnxDpJcCATEIMbxgBtrue2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ6voUTCwzJD04f4D04wP01totCTFrWPtrue2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz9BErq0CUM4rDKx8n4IUCoMdTT-iq6ofalse2023-10-18T19:57:39.339Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHx.Z0UF.X7M99L16W2LmnrMzRT52wSnfalse2023-10-18T19:57:39.290Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOZ8sYbxHfpd4TmnYec31NLGZCLDDZ9Xfalse2023-10-18T19:57:39.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXwY9HQwGVPJQ92uscYuFYsIdbQgXEYjfalse2023-10-18T19:57:39.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobRs9QrejEnMKy2WaXOLHr.iYfq1qDefalse2023-10-18T19:57:39.233Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGBljJZwndU8OOxk7KJyahMn7aN2joxpfalse2023-10-18T19:57:39.121Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqx9JNJ-cAf2CB7fwdF8n.40VRiX85wJfalse2023-10-18T19:57:39.182Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6false2023-10-18T19:57:39.074Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm--RJO4V9Vrua4FSRFh-ASpvQgROG6kfalse2023-10-18T19:57:39.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU42vKDwj1WFjI-bMrhPe5O7B-9dabl9false2023-10-18T19:57:38.953Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqjJoJ0JywLBVngkF4NqtHZRMQQpSi7vfalse2023-10-18T19:57:39.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8nfalse2023-10-18T19:57:38.929Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjegd1qNK-ZKDNN5bN4A1VaSs97JSE297false2023-10-18T19:57:38.906Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6s7LnG-qRPtK-UCaUqeTxtDibOflrUdfalse2023-10-18T19:57:38.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNTvDWcfE9kVoyznfhCGU6ELa04H.jwLfalse2023-10-18T19:57:38.848Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG59arbhnbpCc-hwP0YJ8EXQMYoW5oYgfalse2023-10-18T19:57:38.739Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoyyerD918wRExgeuwLQNphank4bD2k3false2023-10-18T19:57:38.798Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBH2jE4UqtGp8OfC1t9VV-x6wQuT5PZzfalse2023-10-18T19:57:38.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNAVmL-UuMH5Lyboppq2UUDWFWO-Ud0vfalse2023-10-18T19:57:38.682Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:58:59.707 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T19:58:59.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:58:59.715 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.716 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.716 DEBUG:boto:path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.717 DEBUG:boto:auth_path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.717 DEBUG:boto:Method: GET 2023-10-18T19:58:59.717 DEBUG:boto:Path: /bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.718 DEBUG:boto:Data: 2023-10-18T19:58:59.718 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.719 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.719 DEBUG:boto:Params: {} 2023-10-18T19:58:59.720 DEBUG:boto:Token: None 2023-10-18T19:58:59.721 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/?versions 2023-10-18T19:58:59.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:crzKNezoCOmUG8W9MoYgDqM0Om0= 2023-10-18T19:58:59.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:crzKNezoCOmUG8W9MoYgDqM0Om0=', 'Content-Length': '0'} 2023-10-18T19:58:59.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eae86816d48f01e-0065303903-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.725 DEBUG:boto:b'bofrnf-13999falsedummyILg3ajpYX4M9rwAnxDpJcCATEIMbxgBtrue2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ6voUTCwzJD04f4D04wP01totCTFrWPtrue2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz9BErq0CUM4rDKx8n4IUCoMdTT-iq6ofalse2023-10-18T19:57:39.339Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHx.Z0UF.X7M99L16W2LmnrMzRT52wSnfalse2023-10-18T19:57:39.290Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOZ8sYbxHfpd4TmnYec31NLGZCLDDZ9Xfalse2023-10-18T19:57:39.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXwY9HQwGVPJQ92uscYuFYsIdbQgXEYjfalse2023-10-18T19:57:39.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobRs9QrejEnMKy2WaXOLHr.iYfq1qDefalse2023-10-18T19:57:39.233Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGBljJZwndU8OOxk7KJyahMn7aN2joxpfalse2023-10-18T19:57:39.121Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqx9JNJ-cAf2CB7fwdF8n.40VRiX85wJfalse2023-10-18T19:57:39.182Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6false2023-10-18T19:57:39.074Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm--RJO4V9Vrua4FSRFh-ASpvQgROG6kfalse2023-10-18T19:57:39.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU42vKDwj1WFjI-bMrhPe5O7B-9dabl9false2023-10-18T19:57:38.953Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqjJoJ0JywLBVngkF4NqtHZRMQQpSi7vfalse2023-10-18T19:57:39.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8nfalse2023-10-18T19:57:38.929Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjegd1qNK-ZKDNN5bN4A1VaSs97JSE297false2023-10-18T19:57:38.906Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6s7LnG-qRPtK-UCaUqeTxtDibOflrUdfalse2023-10-18T19:57:38.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNTvDWcfE9kVoyznfhCGU6ELa04H.jwLfalse2023-10-18T19:57:38.848Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG59arbhnbpCc-hwP0YJ8EXQMYoW5oYgfalse2023-10-18T19:57:38.739Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoyyerD918wRExgeuwLQNphank4bD2k3false2023-10-18T19:57:38.798Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBH2jE4UqtGp8OfC1t9VV-x6wQuT5PZzfalse2023-10-18T19:57:38.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNAVmL-UuMH5Lyboppq2UUDWFWO-Ud0vfalse2023-10-18T19:57:38.682Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:58:59.727 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.727 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.728 DEBUG:boto:path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.728 DEBUG:boto:auth_path=/bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.728 DEBUG:boto:Method: GET 2023-10-18T19:58:59.729 DEBUG:boto:Path: /bofrnf-13/?versions&max-keys=999 2023-10-18T19:58:59.729 DEBUG:boto:Data: 2023-10-18T19:58:59.729 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.730 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.730 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.731 DEBUG:boto:Params: {} 2023-10-18T19:58:59.731 DEBUG:boto:Token: None 2023-10-18T19:58:59.732 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/?versions 2023-10-18T19:58:59.732 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:crzKNezoCOmUG8W9MoYgDqM0Om0= 2023-10-18T19:58:59.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:crzKNezoCOmUG8W9MoYgDqM0Om0=', 'Content-Length': '0'} 2023-10-18T19:58:59.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c33f89ab73d3e6d6-0065303903-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.736 DEBUG:boto:b'bofrnf-13999falsedummyILg3ajpYX4M9rwAnxDpJcCATEIMbxgBtrue2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ6voUTCwzJD04f4D04wP01totCTFrWPtrue2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz9BErq0CUM4rDKx8n4IUCoMdTT-iq6ofalse2023-10-18T19:57:39.339Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHx.Z0UF.X7M99L16W2LmnrMzRT52wSnfalse2023-10-18T19:57:39.290Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOZ8sYbxHfpd4TmnYec31NLGZCLDDZ9Xfalse2023-10-18T19:57:39.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXwY9HQwGVPJQ92uscYuFYsIdbQgXEYjfalse2023-10-18T19:57:39.208Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjobRs9QrejEnMKy2WaXOLHr.iYfq1qDefalse2023-10-18T19:57:39.233Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGBljJZwndU8OOxk7KJyahMn7aN2joxpfalse2023-10-18T19:57:39.121Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqx9JNJ-cAf2CB7fwdF8n.40VRiX85wJfalse2023-10-18T19:57:39.182Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6false2023-10-18T19:57:39.074Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm--RJO4V9Vrua4FSRFh-ASpvQgROG6kfalse2023-10-18T19:57:39.097Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU42vKDwj1WFjI-bMrhPe5O7B-9dabl9false2023-10-18T19:57:38.953Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqjJoJ0JywLBVngkF4NqtHZRMQQpSi7vfalse2023-10-18T19:57:39.010Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8nfalse2023-10-18T19:57:38.929Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjegd1qNK-ZKDNN5bN4A1VaSs97JSE297false2023-10-18T19:57:38.906Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6s7LnG-qRPtK-UCaUqeTxtDibOflrUdfalse2023-10-18T19:57:38.824Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNTvDWcfE9kVoyznfhCGU6ELa04H.jwLfalse2023-10-18T19:57:38.848Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjG59arbhnbpCc-hwP0YJ8EXQMYoW5oYgfalse2023-10-18T19:57:38.739Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoyyerD918wRExgeuwLQNphank4bD2k3false2023-10-18T19:57:38.798Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBH2jE4UqtGp8OfC1t9VV-x6wQuT5PZzfalse2023-10-18T19:57:38.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNAVmL-UuMH5Lyboppq2UUDWFWO-Ud0vfalse2023-10-18T19:57:38.682Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:58:59.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:58:59.738 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.739 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.739 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.740 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.740 DEBUG:boto:Method: GET 2023-10-18T19:58:59.740 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.741 DEBUG:boto:Data: 2023-10-18T19:58:59.741 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.742 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.742 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.743 DEBUG:boto:Params: {} 2023-10-18T19:58:59.743 DEBUG:boto:Token: None 2023-10-18T19:58:59.744 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.744 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU= 2023-10-18T19:58:59.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU=', 'Content-Length': '0'} 2023-10-18T19:58:59.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076743620354c08a4-0065303903-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.748 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.748 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.749 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.749 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.749 DEBUG:boto:Method: GET 2023-10-18T19:58:59.750 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.750 DEBUG:boto:Data: 2023-10-18T19:58:59.751 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.751 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.752 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.752 DEBUG:boto:Params: {} 2023-10-18T19:58:59.752 DEBUG:boto:Token: None 2023-10-18T19:58:59.753 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.753 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU= 2023-10-18T19:58:59.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU=', 'Content-Length': '0'} 2023-10-18T19:58:59.756 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000092042bc2579c95a-0065303903-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.757 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.757 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.757 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.758 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.758 DEBUG:boto:Method: HEAD 2023-10-18T19:58:59.759 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.759 DEBUG:boto:Data: 2023-10-18T19:58:59.760 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.760 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.760 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.761 DEBUG:boto:Params: {} 2023-10-18T19:58:59.761 DEBUG:boto:Token: None 2023-10-18T19:58:59.762 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.762 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:alOZIsw8EXHO8V1+AvqSRLggjU0= 2023-10-18T19:58:59.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:alOZIsw8EXHO8V1+AvqSRLggjU0=', 'Content-Length': '0'} 2023-10-18T19:58:59.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ac10d4ee59f9fb4-0065303903-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.765 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.765 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.766 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.766 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.767 DEBUG:boto:Method: HEAD 2023-10-18T19:58:59.767 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.767 DEBUG:boto:Data: 2023-10-18T19:58:59.768 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.768 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.769 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.769 DEBUG:boto:Params: {} 2023-10-18T19:58:59.769 DEBUG:boto:Token: None 2023-10-18T19:58:59.770 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.770 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:alOZIsw8EXHO8V1+AvqSRLggjU0= 2023-10-18T19:58:59.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:alOZIsw8EXHO8V1+AvqSRLggjU0=', 'Content-Length': '0'} 2023-10-18T19:58:59.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001c9c7fd45aa899d-0065303903-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:58:59.774 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.774 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.775 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.775 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.775 DEBUG:boto:Method: GET 2023-10-18T19:58:59.776 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.776 DEBUG:boto:Data: 2023-10-18T19:58:59.777 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.777 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.778 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.778 DEBUG:boto:Params: {} 2023-10-18T19:58:59.779 DEBUG:boto:Token: None 2023-10-18T19:58:59.779 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU= 2023-10-18T19:58:59.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU=', 'Content-Length': '0'} 2023-10-18T19:58:59.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a83f45b10308655-0065303903-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.783 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.784 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.784 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.784 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.785 DEBUG:boto:Method: GET 2023-10-18T19:58:59.785 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.785 DEBUG:boto:Data: 2023-10-18T19:58:59.786 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.786 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.787 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.787 DEBUG:boto:Params: {} 2023-10-18T19:58:59.788 DEBUG:boto:Token: None 2023-10-18T19:58:59.788 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU= 2023-10-18T19:58:59.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU=', 'Content-Length': '0'} 2023-10-18T19:58:59.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065c99b01969a4395-0065303903-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.792 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.792 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.793 DEBUG:boto:path=/bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.793 DEBUG:boto:auth_path=/bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.793 DEBUG:boto:Method: GET 2023-10-18T19:58:59.794 DEBUG:boto:Path: /bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.794 DEBUG:boto:Data: 2023-10-18T19:58:59.795 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.795 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.796 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.796 DEBUG:boto:Params: {} 2023-10-18T19:58:59.797 DEBUG:boto:Token: None 2023-10-18T19:58:59.797 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/ 2023-10-18T19:58:59.797 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B6fkEe31aWV3SHLCH/FLT/s7jzY= 2023-10-18T19:58:59.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B6fkEe31aWV3SHLCH/FLT/s7jzY=', 'Content-Length': '0'} 2023-10-18T19:58:59.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ba28590a7553381-0065303903-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.842 DEBUG:boto:b'bofrnf-13dummy1000falsedummy2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:58:59.843 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.844 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.844 DEBUG:boto:path=/bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.844 DEBUG:boto:auth_path=/bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.845 DEBUG:boto:Method: GET 2023-10-18T19:58:59.845 DEBUG:boto:Path: /bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.846 DEBUG:boto:Data: 2023-10-18T19:58:59.846 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.847 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.847 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.848 DEBUG:boto:Params: {} 2023-10-18T19:58:59.848 DEBUG:boto:Token: None 2023-10-18T19:58:59.849 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/ 2023-10-18T19:58:59.849 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B6fkEe31aWV3SHLCH/FLT/s7jzY= 2023-10-18T19:58:59.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B6fkEe31aWV3SHLCH/FLT/s7jzY=', 'Content-Length': '0'} 2023-10-18T19:58:59.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000259d618ae868bb69-0065303903-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.853 DEBUG:boto:b'bofrnf-13dummy1000falsedummy2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:58:59.854 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.854 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.855 DEBUG:boto:Method: HEAD 2023-10-18T19:58:59.855 DEBUG:boto:Path: /bofrnf-13/dummy 2023-10-18T19:58:59.856 DEBUG:boto:Data: 2023-10-18T19:58:59.856 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.856 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.857 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.857 DEBUG:boto:Params: {} 2023-10-18T19:58:59.858 DEBUG:boto:Token: None 2023-10-18T19:58:59.858 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy 2023-10-18T19:58:59.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C1+idVRYQneY2KLk6cNJxH/p2qs= 2023-10-18T19:58:59.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C1+idVRYQneY2KLk6cNJxH/p2qs=', 'Content-Length': '0'} 2023-10-18T19:58:59.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a87e10d0370e0624-0065303903-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.863 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.863 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.864 DEBUG:boto:Method: HEAD 2023-10-18T19:58:59.865 DEBUG:boto:Path: /bofrnf-13/dummy 2023-10-18T19:58:59.865 DEBUG:boto:Data: 2023-10-18T19:58:59.866 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.866 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.866 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.867 DEBUG:boto:Params: {} 2023-10-18T19:58:59.867 DEBUG:boto:Token: None 2023-10-18T19:58:59.868 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy 2023-10-18T19:58:59.868 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C1+idVRYQneY2KLk6cNJxH/p2qs= 2023-10-18T19:58:59.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C1+idVRYQneY2KLk6cNJxH/p2qs=', 'Content-Length': '0'} 2023-10-18T19:58:59.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000741d1ea5771b4baf-0065303903-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T19:58:59.872 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.872 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.873 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.873 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.874 DEBUG:boto:Method: GET 2023-10-18T19:58:59.874 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.875 DEBUG:boto:Data: 2023-10-18T19:58:59.875 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.876 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.876 DEBUG:boto:Params: {} 2023-10-18T19:58:59.877 DEBUG:boto:Token: None 2023-10-18T19:58:59.877 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.878 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU= 2023-10-18T19:58:59.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU=', 'Content-Length': '0'} 2023-10-18T19:58:59.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d744135b8f98d83e-0065303903-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.881 DEBUG:boto:path=/bofrnf-13/dummy 2023-10-18T19:58:59.881 DEBUG:boto:auth_path=/bofrnf-13/dummy 2023-10-18T19:58:59.882 DEBUG:boto:path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.882 DEBUG:boto:auth_path=/bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.883 DEBUG:boto:Method: GET 2023-10-18T19:58:59.883 DEBUG:boto:Path: /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.883 DEBUG:boto:Data: 2023-10-18T19:58:59.884 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.884 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.885 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.885 DEBUG:boto:Params: {} 2023-10-18T19:58:59.886 DEBUG:boto:Token: None 2023-10-18T19:58:59.886 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/dummy?versionId=ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB 2023-10-18T19:58:59.887 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU= 2023-10-18T19:58:59.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CKOVL06uX/o32QP8i5bfO4iKOPU=', 'Content-Length': '0'} 2023-10-18T19:58:59.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:08 GMT'), ('x-amz-version-id', 'ILg3ajpYX4M9rwAnxDpJcCATEIMbxgB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000272de18dcdaf46e2-0065303903-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.890 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.890 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.891 DEBUG:boto:path=/bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.891 DEBUG:boto:auth_path=/bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.891 DEBUG:boto:Method: GET 2023-10-18T19:58:59.892 DEBUG:boto:Path: /bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.892 DEBUG:boto:Data: 2023-10-18T19:58:59.893 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.893 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.894 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.894 DEBUG:boto:Params: {} 2023-10-18T19:58:59.894 DEBUG:boto:Token: None 2023-10-18T19:58:59.895 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/ 2023-10-18T19:58:59.895 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B6fkEe31aWV3SHLCH/FLT/s7jzY= 2023-10-18T19:58:59.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B6fkEe31aWV3SHLCH/FLT/s7jzY=', 'Content-Length': '0'} 2023-10-18T19:58:59.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095008d4fda7b2d81-0065303903-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.899 DEBUG:boto:b'bofrnf-13dummy1000falsedummy2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:58:59.900 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:58:59.900 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:58:59.901 DEBUG:boto:path=/bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.901 DEBUG:boto:auth_path=/bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.901 DEBUG:boto:Method: GET 2023-10-18T19:58:59.902 DEBUG:boto:Path: /bofrnf-13/?prefix=dummy 2023-10-18T19:58:59.902 DEBUG:boto:Data: 2023-10-18T19:58:59.903 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.903 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.904 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.904 DEBUG:boto:Params: {} 2023-10-18T19:58:59.905 DEBUG:boto:Token: None 2023-10-18T19:58:59.905 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/ 2023-10-18T19:58:59.905 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B6fkEe31aWV3SHLCH/FLT/s7jzY= 2023-10-18T19:58:59.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B6fkEe31aWV3SHLCH/FLT/s7jzY=', 'Content-Length': '0'} 2023-10-18T19:58:59.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018f04032264cc154-0065303903-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.909 DEBUG:boto:b'bofrnf-13dummy1000falsedummy2023-10-18T19:57:08.190Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:58:59.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:58:59.910 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:58:59.911 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:58:59.911 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.911 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.912 DEBUG:boto:Method: GET 2023-10-18T19:58:59.912 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.913 DEBUG:boto:Data: 2023-10-18T19:58:59.913 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.914 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.914 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.915 DEBUG:boto:Params: {} 2023-10-18T19:58:59.915 DEBUG:boto:Token: None 2023-10-18T19:58:59.916 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.916 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AdcymsIVtCjXnA5ib04HFJi3bug= 2023-10-18T19:58:59.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AdcymsIVtCjXnA5ib04HFJi3bug=', 'Content-Length': '0'} 2023-10-18T19:58:59.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e4dc1580c6a8dd1-0065303903-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.919 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:58:59.920 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:58:59.920 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.920 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.921 DEBUG:boto:Method: GET 2023-10-18T19:58:59.921 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.922 DEBUG:boto:Data: 2023-10-18T19:58:59.922 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.923 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.923 DEBUG:boto:Params: {} 2023-10-18T19:58:59.924 DEBUG:boto:Token: None 2023-10-18T19:58:59.924 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AdcymsIVtCjXnA5ib04HFJi3bug= 2023-10-18T19:58:59.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AdcymsIVtCjXnA5ib04HFJi3bug=', 'Content-Length': '0'} 2023-10-18T19:58:59.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cfb7279ac43f50e-0065303903-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.929 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:58:59.930 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:58:59.930 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.931 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.931 DEBUG:boto:Method: HEAD 2023-10-18T19:58:59.932 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.932 DEBUG:boto:Data: 2023-10-18T19:58:59.932 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.933 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.933 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.934 DEBUG:boto:Params: {} 2023-10-18T19:58:59.934 DEBUG:boto:Token: None 2023-10-18T19:58:59.935 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.935 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8+IScv1f+beBAsB7UYnH0V+AHe4= 2023-10-18T19:58:59.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8+IScv1f+beBAsB7UYnH0V+AHe4=', 'Content-Length': '0'} 2023-10-18T19:58:59.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009760b05d3c353a58-0065303903-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.938 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:58:59.938 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:58:59.939 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.939 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.939 DEBUG:boto:Method: HEAD 2023-10-18T19:58:59.940 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.940 DEBUG:boto:Data: 2023-10-18T19:58:59.941 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:58:59.941 DEBUG:boto:Port: 8000 2023-10-18T19:58:59.942 DEBUG:boto:Params: {} 2023-10-18T19:58:59.942 DEBUG:boto:Token: None 2023-10-18T19:58:59.943 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.943 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8+IScv1f+beBAsB7UYnH0V+AHe4= 2023-10-18T19:58:59.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8+IScv1f+beBAsB7UYnH0V+AHe4=', 'Content-Length': '0'} 2023-10-18T19:58:59.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e7ca8781a3fc3d9-0065303903-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:58:59.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:58:59.947 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:58:59.947 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:58:59.947 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.948 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.948 DEBUG:boto:Method: GET 2023-10-18T19:58:59.949 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.949 DEBUG:boto:Data: 2023-10-18T19:58:59.949 DEBUG:boto:Headers: {} 2023-10-18T19:58:59.950 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:58:59.950 DEBUG:boto:Port: 8001 2023-10-18T19:58:59.951 DEBUG:boto:Params: {} 2023-10-18T19:58:59.951 DEBUG:boto:Token: None 2023-10-18T19:58:59.952 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:58:59 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:58:59.952 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AdcymsIVtCjXnA5ib04HFJi3bug= 2023-10-18T19:58:59.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:58:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AdcymsIVtCjXnA5ib04HFJi3bug=', 'Content-Length': '0'} 2023-10-18T19:58:59.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000597fe60e51b30c7-0065303903-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:58:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.005 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.006 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.007 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.007 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.008 DEBUG:boto:Method: GET 2023-10-18T19:59:00.008 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.008 DEBUG:boto:Data: 2023-10-18T19:59:00.009 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.009 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.010 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.010 DEBUG:boto:Params: {} 2023-10-18T19:59:00.011 DEBUG:boto:Token: None 2023-10-18T19:59:00.011 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68= 2023-10-18T19:59:00.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68=', 'Content-Length': '0'} 2023-10-18T19:59:00.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025f616c166c80105-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.054 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.055 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.055 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.056 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.056 DEBUG:boto:Method: GET 2023-10-18T19:59:00.056 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.057 DEBUG:boto:Data: 2023-10-18T19:59:00.057 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.058 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.058 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.059 DEBUG:boto:Params: {} 2023-10-18T19:59:00.059 DEBUG:boto:Token: None 2023-10-18T19:59:00.060 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.060 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d845387bf0023d5-0065303904-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.064 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.065 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.065 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.066 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.066 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.067 DEBUG:boto:Method: GET 2023-10-18T19:59:00.067 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.067 DEBUG:boto:Data: 2023-10-18T19:59:00.068 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.068 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.069 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.069 DEBUG:boto:Params: {} 2023-10-18T19:59:00.069 DEBUG:boto:Token: None 2023-10-18T19:59:00.070 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.070 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a11c9e0a37277f56-0065303904-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.118 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.119 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.120 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.120 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.121 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:00.121 DEBUG:boto:Data: 2023-10-18T19:59:00.121 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.122 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.123 DEBUG:boto:Params: {} 2023-10-18T19:59:00.123 DEBUG:boto:Token: None 2023-10-18T19:59:00.124 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj 2023-10-18T19:59:00.124 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg= 2023-10-18T19:59:00.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg=', 'Content-Length': '0'} 2023-10-18T19:59:00.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006f67313a02086ed-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.128 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.129 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.129 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.130 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:00.130 DEBUG:boto:Data: 2023-10-18T19:59:00.130 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.131 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.131 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.132 DEBUG:boto:Params: {} 2023-10-18T19:59:00.132 DEBUG:boto:Token: None 2023-10-18T19:59:00.132 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj 2023-10-18T19:59:00.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg= 2023-10-18T19:59:00.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg=', 'Content-Length': '0'} 2023-10-18T19:59:00.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb246003d5a92e04-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.137 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.138 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.138 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.139 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.139 DEBUG:boto:Method: GET 2023-10-18T19:59:00.139 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.140 DEBUG:boto:Data: 2023-10-18T19:59:00.140 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.141 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.142 DEBUG:boto:Params: {} 2023-10-18T19:59:00.142 DEBUG:boto:Token: None 2023-10-18T19:59:00.143 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.143 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68= 2023-10-18T19:59:00.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68=', 'Content-Length': '0'} 2023-10-18T19:59:00.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ce81bd821583562-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.146 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.147 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.147 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.148 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.148 DEBUG:boto:Method: GET 2023-10-18T19:59:00.148 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.149 DEBUG:boto:Data: 2023-10-18T19:59:00.149 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.150 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.151 DEBUG:boto:Params: {} 2023-10-18T19:59:00.151 DEBUG:boto:Token: None 2023-10-18T19:59:00.152 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68= 2023-10-18T19:59:00.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68=', 'Content-Length': '0'} 2023-10-18T19:59:00.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f84b311a3ff7230-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.155 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.156 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.156 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.157 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.157 DEBUG:boto:Method: GET 2023-10-18T19:59:00.157 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.158 DEBUG:boto:Data: 2023-10-18T19:59:00.158 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.159 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.159 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.159 DEBUG:boto:Params: {} 2023-10-18T19:59:00.160 DEBUG:boto:Token: None 2023-10-18T19:59:00.160 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096c9ef00ebaeff80-0065303904-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.164 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.165 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.166 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.166 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.167 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.167 DEBUG:boto:Method: GET 2023-10-18T19:59:00.167 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.168 DEBUG:boto:Data: 2023-10-18T19:59:00.168 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.169 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.169 DEBUG:boto:Params: {} 2023-10-18T19:59:00.170 DEBUG:boto:Token: None 2023-10-18T19:59:00.170 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d244f746127e44e-0065303904-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.214 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.216 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.216 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.216 DEBUG:boto:path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.217 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.217 DEBUG:boto:Method: GET 2023-10-18T19:59:00.218 DEBUG:boto:Path: /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.218 DEBUG:boto:Data: 2023-10-18T19:59:00.219 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.219 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.220 DEBUG:boto:Params: {} 2023-10-18T19:59:00.220 DEBUG:boto:Token: None 2023-10-18T19:59:00.221 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JGX0nNjJjcg1NgRTfAaLwtwwiBU= 2023-10-18T19:59:00.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JGX0nNjJjcg1NgRTfAaLwtwwiBU=', 'Content-Length': '0'} 2023-10-18T19:59:00.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e15bb45d91d1b4d2-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.225 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.226 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.226 DEBUG:boto:path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.226 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.227 DEBUG:boto:Method: GET 2023-10-18T19:59:00.227 DEBUG:boto:Path: /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.228 DEBUG:boto:Data: 2023-10-18T19:59:00.228 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.229 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.229 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.230 DEBUG:boto:Params: {} 2023-10-18T19:59:00.230 DEBUG:boto:Token: None 2023-10-18T19:59:00.231 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JGX0nNjJjcg1NgRTfAaLwtwwiBU= 2023-10-18T19:59:00.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JGX0nNjJjcg1NgRTfAaLwtwwiBU=', 'Content-Length': '0'} 2023-10-18T19:59:00.234 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1dc10c3063364fe-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.234 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.235 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.235 DEBUG:boto:path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.236 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.236 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.236 DEBUG:boto:Path: /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.237 DEBUG:boto:Data: 2023-10-18T19:59:00.237 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.238 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.238 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.238 DEBUG:boto:Params: {} 2023-10-18T19:59:00.239 DEBUG:boto:Token: None 2023-10-18T19:59:00.239 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kziP2uUKLZpHB+T/YmFQFXpyGxw= 2023-10-18T19:59:00.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kziP2uUKLZpHB+T/YmFQFXpyGxw=', 'Content-Length': '0'} 2023-10-18T19:59:00.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000919387a49310c3ea-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.243 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.243 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.244 DEBUG:boto:path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.244 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.244 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.245 DEBUG:boto:Path: /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.245 DEBUG:boto:Data: 2023-10-18T19:59:00.245 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.246 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.246 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.247 DEBUG:boto:Params: {} 2023-10-18T19:59:00.247 DEBUG:boto:Token: None 2023-10-18T19:59:00.247 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.248 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kziP2uUKLZpHB+T/YmFQFXpyGxw= 2023-10-18T19:59:00.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kziP2uUKLZpHB+T/YmFQFXpyGxw=', 'Content-Length': '0'} 2023-10-18T19:59:00.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eae12d381f3e7d3f-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.251 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.252 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.252 DEBUG:boto:path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.252 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.253 DEBUG:boto:Method: GET 2023-10-18T19:59:00.253 DEBUG:boto:Path: /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.254 DEBUG:boto:Data: 2023-10-18T19:59:00.254 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.255 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.255 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.255 DEBUG:boto:Params: {} 2023-10-18T19:59:00.256 DEBUG:boto:Token: None 2023-10-18T19:59:00.256 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.257 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JGX0nNjJjcg1NgRTfAaLwtwwiBU= 2023-10-18T19:59:00.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JGX0nNjJjcg1NgRTfAaLwtwwiBU=', 'Content-Length': '0'} 2023-10-18T19:59:00.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000598733f990af150c-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.260 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.260 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.261 DEBUG:boto:path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.261 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.261 DEBUG:boto:Method: GET 2023-10-18T19:59:00.262 DEBUG:boto:Path: /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.262 DEBUG:boto:Data: 2023-10-18T19:59:00.263 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.263 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.264 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.264 DEBUG:boto:Params: {} 2023-10-18T19:59:00.265 DEBUG:boto:Token: None 2023-10-18T19:59:00.265 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o 2023-10-18T19:59:00.266 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JGX0nNjJjcg1NgRTfAaLwtwwiBU= 2023-10-18T19:59:00.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JGX0nNjJjcg1NgRTfAaLwtwwiBU=', 'Content-Length': '0'} 2023-10-18T19:59:00.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'z9BErq0CUM4rDKx8n4IUCoMdTT-iq6o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000690e6ece40e1258c-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.269 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.269 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.269 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.270 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.270 DEBUG:boto:Method: GET 2023-10-18T19:59:00.271 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.271 DEBUG:boto:Data: 2023-10-18T19:59:00.272 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.272 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.272 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.273 DEBUG:boto:Params: {} 2023-10-18T19:59:00.273 DEBUG:boto:Token: None 2023-10-18T19:59:00.273 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.274 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000598ecd35f64b5611-0065303904-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.278 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.278 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.279 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.279 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.279 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.280 DEBUG:boto:Method: GET 2023-10-18T19:59:00.280 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.281 DEBUG:boto:Data: 2023-10-18T19:59:00.281 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.281 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.282 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.282 DEBUG:boto:Params: {} 2023-10-18T19:59:00.283 DEBUG:boto:Token: None 2023-10-18T19:59:00.283 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008691f1cd94ee637a-0065303904-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.287 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.288 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.288 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.289 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.289 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:00.290 DEBUG:boto:Data: 2023-10-18T19:59:00.290 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.290 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.291 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.291 DEBUG:boto:Params: {} 2023-10-18T19:59:00.292 DEBUG:boto:Token: None 2023-10-18T19:59:00.292 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj 2023-10-18T19:59:00.293 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg= 2023-10-18T19:59:00.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg=', 'Content-Length': '0'} 2023-10-18T19:59:00.296 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097bfe4600f578af2-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.297 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.297 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.298 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.298 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:00.298 DEBUG:boto:Data: 2023-10-18T19:59:00.299 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.300 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.300 DEBUG:boto:Params: {} 2023-10-18T19:59:00.300 DEBUG:boto:Token: None 2023-10-18T19:59:00.301 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj 2023-10-18T19:59:00.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg= 2023-10-18T19:59:00.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg=', 'Content-Length': '0'} 2023-10-18T19:59:00.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ce6c4eb717ad476-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.306 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.307 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.307 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.307 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.308 DEBUG:boto:Method: GET 2023-10-18T19:59:00.308 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.309 DEBUG:boto:Data: 2023-10-18T19:59:00.309 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.309 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.310 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.310 DEBUG:boto:Params: {} 2023-10-18T19:59:00.311 DEBUG:boto:Token: None 2023-10-18T19:59:00.311 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68= 2023-10-18T19:59:00.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68=', 'Content-Length': '0'} 2023-10-18T19:59:00.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e30a33e95986f75e-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.358 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.359 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.359 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.360 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.360 DEBUG:boto:Method: GET 2023-10-18T19:59:00.361 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.361 DEBUG:boto:Data: 2023-10-18T19:59:00.362 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.362 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.362 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.363 DEBUG:boto:Params: {} 2023-10-18T19:59:00.363 DEBUG:boto:Token: None 2023-10-18T19:59:00.364 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68= 2023-10-18T19:59:00.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68=', 'Content-Length': '0'} 2023-10-18T19:59:00.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002403c4f756e1fca2-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.368 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.368 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.369 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.370 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.371 DEBUG:boto:Method: GET 2023-10-18T19:59:00.371 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.372 DEBUG:boto:Data: 2023-10-18T19:59:00.372 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.372 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.373 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.373 DEBUG:boto:Params: {} 2023-10-18T19:59:00.374 DEBUG:boto:Token: None 2023-10-18T19:59:00.374 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.375 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b230cb0279ffba30-0065303904-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.378 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.379 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.380 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.380 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.381 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.381 DEBUG:boto:Method: GET 2023-10-18T19:59:00.381 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.382 DEBUG:boto:Data: 2023-10-18T19:59:00.382 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.383 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.383 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.384 DEBUG:boto:Params: {} 2023-10-18T19:59:00.384 DEBUG:boto:Token: None 2023-10-18T19:59:00.385 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.385 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001309936ae734b31b-0065303904-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.430 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.432 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.432 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.433 DEBUG:boto:path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.433 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.433 DEBUG:boto:Method: GET 2023-10-18T19:59:00.434 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.434 DEBUG:boto:Data: 2023-10-18T19:59:00.435 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.435 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.436 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.436 DEBUG:boto:Params: {} 2023-10-18T19:59:00.437 DEBUG:boto:Token: None 2023-10-18T19:59:00.437 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.438 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OdC3g3dSTAdB7t5TYr6OGhc0XqY= 2023-10-18T19:59:00.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OdC3g3dSTAdB7t5TYr6OGhc0XqY=', 'Content-Length': '0'} 2023-10-18T19:59:00.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000def3d2b2b74ff5ec-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.441 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.442 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.443 DEBUG:boto:path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.443 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.443 DEBUG:boto:Method: GET 2023-10-18T19:59:00.444 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.444 DEBUG:boto:Data: 2023-10-18T19:59:00.445 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.445 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.446 DEBUG:boto:Params: {} 2023-10-18T19:59:00.446 DEBUG:boto:Token: None 2023-10-18T19:59:00.447 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.447 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OdC3g3dSTAdB7t5TYr6OGhc0XqY= 2023-10-18T19:59:00.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OdC3g3dSTAdB7t5TYr6OGhc0XqY=', 'Content-Length': '0'} 2023-10-18T19:59:00.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6b24daa6d4b6af6-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.451 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.451 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.452 DEBUG:boto:path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.452 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.453 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.453 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.453 DEBUG:boto:Data: 2023-10-18T19:59:00.454 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.455 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.455 DEBUG:boto:Params: {} 2023-10-18T19:59:00.455 DEBUG:boto:Token: None 2023-10-18T19:59:00.456 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bnr9JbN3sLYGAGvwbxCMKQ6up3w= 2023-10-18T19:59:00.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bnr9JbN3sLYGAGvwbxCMKQ6up3w=', 'Content-Length': '0'} 2023-10-18T19:59:00.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ba36e951ed6e2e7-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.460 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.460 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.460 DEBUG:boto:path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.461 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.461 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.461 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.462 DEBUG:boto:Data: 2023-10-18T19:59:00.462 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.463 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.463 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.463 DEBUG:boto:Params: {} 2023-10-18T19:59:00.464 DEBUG:boto:Token: None 2023-10-18T19:59:00.464 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.465 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bnr9JbN3sLYGAGvwbxCMKQ6up3w= 2023-10-18T19:59:00.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bnr9JbN3sLYGAGvwbxCMKQ6up3w=', 'Content-Length': '0'} 2023-10-18T19:59:00.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c26c6b1bae29605d-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.468 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.468 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.469 DEBUG:boto:path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.469 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.470 DEBUG:boto:Method: GET 2023-10-18T19:59:00.470 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.471 DEBUG:boto:Data: 2023-10-18T19:59:00.471 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.472 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.472 DEBUG:boto:Params: {} 2023-10-18T19:59:00.473 DEBUG:boto:Token: None 2023-10-18T19:59:00.473 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.474 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OdC3g3dSTAdB7t5TYr6OGhc0XqY= 2023-10-18T19:59:00.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OdC3g3dSTAdB7t5TYr6OGhc0XqY=', 'Content-Length': '0'} 2023-10-18T19:59:00.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe9a71cd20e4e532-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.477 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.477 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.478 DEBUG:boto:path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.478 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.479 DEBUG:boto:Method: GET 2023-10-18T19:59:00.479 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.479 DEBUG:boto:Data: 2023-10-18T19:59:00.480 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.480 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.481 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.481 DEBUG:boto:Params: {} 2023-10-18T19:59:00.482 DEBUG:boto:Token: None 2023-10-18T19:59:00.482 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn 2023-10-18T19:59:00.483 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OdC3g3dSTAdB7t5TYr6OGhc0XqY= 2023-10-18T19:59:00.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OdC3g3dSTAdB7t5TYr6OGhc0XqY=', 'Content-Length': '0'} 2023-10-18T19:59:00.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Hx.Z0UF.X7M99L16W2LmnrMzRT52wSn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f62479cd50e32c84-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.530 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.531 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.532 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.532 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.532 DEBUG:boto:Method: GET 2023-10-18T19:59:00.533 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.533 DEBUG:boto:Data: 2023-10-18T19:59:00.534 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.534 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.535 DEBUG:boto:Params: {} 2023-10-18T19:59:00.536 DEBUG:boto:Token: None 2023-10-18T19:59:00.536 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.537 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca642b75c338e94-0065303904-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.542 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.542 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.543 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.543 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.544 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.544 DEBUG:boto:Method: GET 2023-10-18T19:59:00.544 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.545 DEBUG:boto:Data: 2023-10-18T19:59:00.545 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.546 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.546 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.547 DEBUG:boto:Params: {} 2023-10-18T19:59:00.547 DEBUG:boto:Token: None 2023-10-18T19:59:00.548 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.548 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000354e70b08c579e7e-0065303904-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.552 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.552 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.553 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.553 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.553 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:00.554 DEBUG:boto:Data: 2023-10-18T19:59:00.554 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.555 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.555 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.555 DEBUG:boto:Params: {} 2023-10-18T19:59:00.556 DEBUG:boto:Token: None 2023-10-18T19:59:00.556 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj 2023-10-18T19:59:00.557 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg= 2023-10-18T19:59:00.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg=', 'Content-Length': '0'} 2023-10-18T19:59:00.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007accd262cedf3fd2-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.561 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.561 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.562 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.562 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:00.563 DEBUG:boto:Data: 2023-10-18T19:59:00.563 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.563 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.564 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.564 DEBUG:boto:Params: {} 2023-10-18T19:59:00.565 DEBUG:boto:Token: None 2023-10-18T19:59:00.565 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj 2023-10-18T19:59:00.566 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg= 2023-10-18T19:59:00.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg=', 'Content-Length': '0'} 2023-10-18T19:59:00.570 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a21a44d64cedc02-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.571 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.571 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.572 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.572 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.573 DEBUG:boto:Method: GET 2023-10-18T19:59:00.573 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.573 DEBUG:boto:Data: 2023-10-18T19:59:00.574 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.574 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.575 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.575 DEBUG:boto:Params: {} 2023-10-18T19:59:00.576 DEBUG:boto:Token: None 2023-10-18T19:59:00.576 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68= 2023-10-18T19:59:00.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68=', 'Content-Length': '0'} 2023-10-18T19:59:00.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041227a88f8ac8e5c-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.622 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.623 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.623 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.624 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.624 DEBUG:boto:Method: GET 2023-10-18T19:59:00.625 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.625 DEBUG:boto:Data: 2023-10-18T19:59:00.626 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.626 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.626 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.627 DEBUG:boto:Params: {} 2023-10-18T19:59:00.627 DEBUG:boto:Token: None 2023-10-18T19:59:00.628 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.628 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68= 2023-10-18T19:59:00.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68=', 'Content-Length': '0'} 2023-10-18T19:59:00.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f670547337b352ce-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.632 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.632 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.632 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.633 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.633 DEBUG:boto:Method: GET 2023-10-18T19:59:00.634 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.634 DEBUG:boto:Data: 2023-10-18T19:59:00.635 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.635 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.635 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.636 DEBUG:boto:Params: {} 2023-10-18T19:59:00.636 DEBUG:boto:Token: None 2023-10-18T19:59:00.637 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.637 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebce7a3bd23fcd5b-0065303904-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.641 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.641 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.642 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.642 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.643 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.643 DEBUG:boto:Method: GET 2023-10-18T19:59:00.644 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.644 DEBUG:boto:Data: 2023-10-18T19:59:00.644 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.645 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.645 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.646 DEBUG:boto:Params: {} 2023-10-18T19:59:00.646 DEBUG:boto:Token: None 2023-10-18T19:59:00.647 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.647 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045ecdc496b329811-0065303904-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.690 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.692 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.692 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.692 DEBUG:boto:path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.693 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.693 DEBUG:boto:Method: GET 2023-10-18T19:59:00.693 DEBUG:boto:Path: /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.694 DEBUG:boto:Data: 2023-10-18T19:59:00.694 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.695 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.695 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.695 DEBUG:boto:Params: {} 2023-10-18T19:59:00.696 DEBUG:boto:Token: None 2023-10-18T19:59:00.696 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.697 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lbs2zzpFMVchJuGXGqzJ8siuLPw= 2023-10-18T19:59:00.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lbs2zzpFMVchJuGXGqzJ8siuLPw=', 'Content-Length': '0'} 2023-10-18T19:59:00.699 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005281780119d68e83-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.700 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.700 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.701 DEBUG:boto:path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.701 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.702 DEBUG:boto:Method: GET 2023-10-18T19:59:00.702 DEBUG:boto:Path: /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.702 DEBUG:boto:Data: 2023-10-18T19:59:00.703 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.703 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.704 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.704 DEBUG:boto:Params: {} 2023-10-18T19:59:00.705 DEBUG:boto:Token: None 2023-10-18T19:59:00.705 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.705 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lbs2zzpFMVchJuGXGqzJ8siuLPw= 2023-10-18T19:59:00.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lbs2zzpFMVchJuGXGqzJ8siuLPw=', 'Content-Length': '0'} 2023-10-18T19:59:00.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000237d15c7444dc838-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.709 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.709 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.710 DEBUG:boto:path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.710 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.710 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.711 DEBUG:boto:Path: /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.711 DEBUG:boto:Data: 2023-10-18T19:59:00.712 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.712 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.712 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.713 DEBUG:boto:Params: {} 2023-10-18T19:59:00.713 DEBUG:boto:Token: None 2023-10-18T19:59:00.714 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.714 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H0ck9wVPy8BRZf2uztSb/1I/Pxw= 2023-10-18T19:59:00.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H0ck9wVPy8BRZf2uztSb/1I/Pxw=', 'Content-Length': '0'} 2023-10-18T19:59:00.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070117dea587f207b-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.717 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.718 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.718 DEBUG:boto:path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.719 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.719 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.719 DEBUG:boto:Path: /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.720 DEBUG:boto:Data: 2023-10-18T19:59:00.720 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.721 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.721 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.721 DEBUG:boto:Params: {} 2023-10-18T19:59:00.722 DEBUG:boto:Token: None 2023-10-18T19:59:00.722 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.723 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H0ck9wVPy8BRZf2uztSb/1I/Pxw= 2023-10-18T19:59:00.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H0ck9wVPy8BRZf2uztSb/1I/Pxw=', 'Content-Length': '0'} 2023-10-18T19:59:00.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb33463379c2064e-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.726 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.726 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.727 DEBUG:boto:path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.727 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.728 DEBUG:boto:Method: GET 2023-10-18T19:59:00.728 DEBUG:boto:Path: /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.729 DEBUG:boto:Data: 2023-10-18T19:59:00.729 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.729 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.730 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.730 DEBUG:boto:Params: {} 2023-10-18T19:59:00.731 DEBUG:boto:Token: None 2023-10-18T19:59:00.731 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.732 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lbs2zzpFMVchJuGXGqzJ8siuLPw= 2023-10-18T19:59:00.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lbs2zzpFMVchJuGXGqzJ8siuLPw=', 'Content-Length': '0'} 2023-10-18T19:59:00.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000505412703dbdfd5d-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.735 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.735 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.735 DEBUG:boto:path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.736 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.736 DEBUG:boto:Method: GET 2023-10-18T19:59:00.737 DEBUG:boto:Path: /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.737 DEBUG:boto:Data: 2023-10-18T19:59:00.737 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.738 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.739 DEBUG:boto:Params: {} 2023-10-18T19:59:00.739 DEBUG:boto:Token: None 2023-10-18T19:59:00.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X 2023-10-18T19:59:00.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lbs2zzpFMVchJuGXGqzJ8siuLPw= 2023-10-18T19:59:00.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lbs2zzpFMVchJuGXGqzJ8siuLPw=', 'Content-Length': '0'} 2023-10-18T19:59:00.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'OZ8sYbxHfpd4TmnYec31NLGZCLDDZ9X'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c274d3effe3c6b53-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.743 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.744 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.744 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.744 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.745 DEBUG:boto:Method: GET 2023-10-18T19:59:00.745 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.746 DEBUG:boto:Data: 2023-10-18T19:59:00.746 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.749 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.750 DEBUG:boto:Params: {} 2023-10-18T19:59:00.750 DEBUG:boto:Token: None 2023-10-18T19:59:00.751 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.751 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6c7163815eed30a-0065303904-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.755 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.756 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.756 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.756 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.757 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.757 DEBUG:boto:Method: GET 2023-10-18T19:59:00.758 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.758 DEBUG:boto:Data: 2023-10-18T19:59:00.759 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.759 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.759 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.760 DEBUG:boto:Params: {} 2023-10-18T19:59:00.760 DEBUG:boto:Token: None 2023-10-18T19:59:00.761 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.761 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f06cf0d63ebf2a41-0065303904-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.765 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.765 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.766 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.766 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.767 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:00.767 DEBUG:boto:Data: 2023-10-18T19:59:00.767 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.768 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.768 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.769 DEBUG:boto:Params: {} 2023-10-18T19:59:00.769 DEBUG:boto:Token: None 2023-10-18T19:59:00.770 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj 2023-10-18T19:59:00.770 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg= 2023-10-18T19:59:00.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg=', 'Content-Length': '0'} 2023-10-18T19:59:00.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf686cb863d53c11-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.774 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.775 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.775 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.776 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:00.776 DEBUG:boto:Data: 2023-10-18T19:59:00.776 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.777 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.777 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.777 DEBUG:boto:Params: {} 2023-10-18T19:59:00.778 DEBUG:boto:Token: None 2023-10-18T19:59:00.778 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj 2023-10-18T19:59:00.779 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg= 2023-10-18T19:59:00.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+EACv+Djget1cD+ioFRIlPT/hEg=', 'Content-Length': '0'} 2023-10-18T19:59:00.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004079ab43e5d054bd-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.783 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.784 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.784 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.785 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.785 DEBUG:boto:Method: GET 2023-10-18T19:59:00.786 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.786 DEBUG:boto:Data: 2023-10-18T19:59:00.786 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.787 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.788 DEBUG:boto:Params: {} 2023-10-18T19:59:00.788 DEBUG:boto:Token: None 2023-10-18T19:59:00.789 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68= 2023-10-18T19:59:00.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68=', 'Content-Length': '0'} 2023-10-18T19:59:00.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a39a3f1338e6545-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.834 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.835 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.835 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.836 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.836 DEBUG:boto:Method: GET 2023-10-18T19:59:00.837 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.837 DEBUG:boto:Data: 2023-10-18T19:59:00.838 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.838 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.839 DEBUG:boto:Params: {} 2023-10-18T19:59:00.839 DEBUG:boto:Token: None 2023-10-18T19:59:00.840 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:00.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68= 2023-10-18T19:59:00.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPYZLuYWxGtf/SgvluJ8zBmbe68=', 'Content-Length': '0'} 2023-10-18T19:59:00.843 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a280268e8cec69b-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.844 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.844 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.845 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.845 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.845 DEBUG:boto:Method: GET 2023-10-18T19:59:00.846 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.846 DEBUG:boto:Data: 2023-10-18T19:59:00.847 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.847 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.848 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.848 DEBUG:boto:Params: {} 2023-10-18T19:59:00.848 DEBUG:boto:Token: None 2023-10-18T19:59:00.850 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.850 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c64c86087ad153b-0065303904-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.854 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.855 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:00.855 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:00.855 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.856 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:00.856 DEBUG:boto:Method: GET 2023-10-18T19:59:00.857 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:00.857 DEBUG:boto:Data: 2023-10-18T19:59:00.857 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.858 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.859 DEBUG:boto:Params: {} 2023-10-18T19:59:00.859 DEBUG:boto:Token: None 2023-10-18T19:59:00.860 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/ 2023-10-18T19:59:00.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M= 2023-10-18T19:59:00.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01VV/4+x917Li5/OlTnMsF/k6/M=', 'Content-Length': '0'} 2023-10-18T19:59:00.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003817dcaa6cd1c59e-0065303904-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.906 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:00.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.908 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.908 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.909 DEBUG:boto:path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.909 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.909 DEBUG:boto:Method: GET 2023-10-18T19:59:00.910 DEBUG:boto:Path: /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.910 DEBUG:boto:Data: 2023-10-18T19:59:00.911 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.911 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.911 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.912 DEBUG:boto:Params: {} 2023-10-18T19:59:00.912 DEBUG:boto:Token: None 2023-10-18T19:59:00.913 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.913 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z+tiXyw7UcT9WgHE26cOF63uujc= 2023-10-18T19:59:00.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z+tiXyw7UcT9WgHE26cOF63uujc=', 'Content-Length': '0'} 2023-10-18T19:59:00.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000380a2dc1685bfb36-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.917 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.917 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.917 DEBUG:boto:path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.918 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.918 DEBUG:boto:Method: GET 2023-10-18T19:59:00.919 DEBUG:boto:Path: /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.919 DEBUG:boto:Data: 2023-10-18T19:59:00.919 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.920 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.920 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.921 DEBUG:boto:Params: {} 2023-10-18T19:59:00.921 DEBUG:boto:Token: None 2023-10-18T19:59:00.922 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z+tiXyw7UcT9WgHE26cOF63uujc= 2023-10-18T19:59:00.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z+tiXyw7UcT9WgHE26cOF63uujc=', 'Content-Length': '0'} 2023-10-18T19:59:00.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073ba1e4fb1517213-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.925 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.926 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.926 DEBUG:boto:path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.926 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.927 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.927 DEBUG:boto:Path: /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.928 DEBUG:boto:Data: 2023-10-18T19:59:00.928 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.929 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.929 DEBUG:boto:Params: {} 2023-10-18T19:59:00.930 DEBUG:boto:Token: None 2023-10-18T19:59:00.930 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.931 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0uHC4emDj2zQjWNUQh2DSGnkPyU= 2023-10-18T19:59:00.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0uHC4emDj2zQjWNUQh2DSGnkPyU=', 'Content-Length': '0'} 2023-10-18T19:59:00.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009805cac868739eef-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.934 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.934 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.935 DEBUG:boto:path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.935 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.935 DEBUG:boto:Method: HEAD 2023-10-18T19:59:00.936 DEBUG:boto:Path: /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.936 DEBUG:boto:Data: 2023-10-18T19:59:00.937 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.937 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.938 DEBUG:boto:Params: {} 2023-10-18T19:59:00.938 DEBUG:boto:Token: None 2023-10-18T19:59:00.939 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0uHC4emDj2zQjWNUQh2DSGnkPyU= 2023-10-18T19:59:00.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0uHC4emDj2zQjWNUQh2DSGnkPyU=', 'Content-Length': '0'} 2023-10-18T19:59:00.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000382b308e5655b19a-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:00.942 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.942 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.943 DEBUG:boto:path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.943 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.944 DEBUG:boto:Method: GET 2023-10-18T19:59:00.944 DEBUG:boto:Path: /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.944 DEBUG:boto:Data: 2023-10-18T19:59:00.945 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:00.946 DEBUG:boto:Port: 8001 2023-10-18T19:59:00.946 DEBUG:boto:Params: {} 2023-10-18T19:59:00.947 DEBUG:boto:Token: None 2023-10-18T19:59:00.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z+tiXyw7UcT9WgHE26cOF63uujc= 2023-10-18T19:59:00.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z+tiXyw7UcT9WgHE26cOF63uujc=', 'Content-Length': '0'} 2023-10-18T19:59:00.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b75f38833b7469cb-0065303904-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.990 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:00.991 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:00.991 DEBUG:boto:path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.992 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.992 DEBUG:boto:Method: GET 2023-10-18T19:59:00.992 DEBUG:boto:Path: /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.993 DEBUG:boto:Data: 2023-10-18T19:59:00.993 DEBUG:boto:Headers: {} 2023-10-18T19:59:00.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:00.994 DEBUG:boto:Port: 8000 2023-10-18T19:59:00.995 DEBUG:boto:Params: {} 2023-10-18T19:59:00.995 DEBUG:boto:Token: None 2023-10-18T19:59:00.996 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:00 GMT /bofrnf-13/obj?versionId=XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj 2023-10-18T19:59:00.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z+tiXyw7UcT9WgHE26cOF63uujc= 2023-10-18T19:59:00.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z+tiXyw7UcT9WgHE26cOF63uujc=', 'Content-Length': '0'} 2023-10-18T19:59:00.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'XwY9HQwGVPJQ92uscYuFYsIdbQgXEYj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004752d7ce02621b4d-0065303904-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:00.999 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.000 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.000 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.000 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.001 DEBUG:boto:Method: GET 2023-10-18T19:59:01.001 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.002 DEBUG:boto:Data: 2023-10-18T19:59:01.002 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.003 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.003 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.003 DEBUG:boto:Params: {} 2023-10-18T19:59:01.004 DEBUG:boto:Token: None 2023-10-18T19:59:01.004 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.005 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2dc6f57e0b5b0a0-0065303905-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.008 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.009 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.009 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.010 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.010 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.011 DEBUG:boto:Method: GET 2023-10-18T19:59:01.011 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.011 DEBUG:boto:Data: 2023-10-18T19:59:01.012 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.012 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.013 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.013 DEBUG:boto:Params: {} 2023-10-18T19:59:01.013 DEBUG:boto:Token: None 2023-10-18T19:59:01.014 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.014 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c953574457c264a6-0065303905-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.018 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.019 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.019 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.020 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.020 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.020 DEBUG:boto:Data: 2023-10-18T19:59:01.021 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.021 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.022 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.022 DEBUG:boto:Params: {} 2023-10-18T19:59:01.022 DEBUG:boto:Token: None 2023-10-18T19:59:01.023 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.023 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d8e078e38ee87d8-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.027 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.028 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.028 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.028 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.029 DEBUG:boto:Data: 2023-10-18T19:59:01.029 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.030 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.030 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.030 DEBUG:boto:Params: {} 2023-10-18T19:59:01.031 DEBUG:boto:Token: None 2023-10-18T19:59:01.031 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006095d0f7153e77b6-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.036 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.037 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.037 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.038 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.038 DEBUG:boto:Method: GET 2023-10-18T19:59:01.038 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.039 DEBUG:boto:Data: 2023-10-18T19:59:01.039 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.040 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.040 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.041 DEBUG:boto:Params: {} 2023-10-18T19:59:01.041 DEBUG:boto:Token: None 2023-10-18T19:59:01.041 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.042 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs= 2023-10-18T19:59:01.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs=', 'Content-Length': '0'} 2023-10-18T19:59:01.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b127db919ae3af76-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.086 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.087 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.087 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.088 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.088 DEBUG:boto:Method: GET 2023-10-18T19:59:01.089 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.089 DEBUG:boto:Data: 2023-10-18T19:59:01.089 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.090 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.090 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.091 DEBUG:boto:Params: {} 2023-10-18T19:59:01.091 DEBUG:boto:Token: None 2023-10-18T19:59:01.092 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.092 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs= 2023-10-18T19:59:01.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs=', 'Content-Length': '0'} 2023-10-18T19:59:01.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f9d4053fb60ec8a-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.095 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.096 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.096 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.097 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.097 DEBUG:boto:Method: GET 2023-10-18T19:59:01.097 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.098 DEBUG:boto:Data: 2023-10-18T19:59:01.098 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.099 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.099 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.100 DEBUG:boto:Params: {} 2023-10-18T19:59:01.100 DEBUG:boto:Token: None 2023-10-18T19:59:01.101 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.101 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afd1e157b8fdab53-0065303905-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.105 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.105 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.106 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.106 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.107 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.107 DEBUG:boto:Method: GET 2023-10-18T19:59:01.108 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.108 DEBUG:boto:Data: 2023-10-18T19:59:01.108 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.109 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.110 DEBUG:boto:Params: {} 2023-10-18T19:59:01.110 DEBUG:boto:Token: None 2023-10-18T19:59:01.111 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.111 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009944df5b0c368a0f-0065303905-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.162 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.164 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.164 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.164 DEBUG:boto:path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.165 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.165 DEBUG:boto:Method: GET 2023-10-18T19:59:01.166 DEBUG:boto:Path: /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.166 DEBUG:boto:Data: 2023-10-18T19:59:01.166 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.167 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.167 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.168 DEBUG:boto:Params: {} 2023-10-18T19:59:01.168 DEBUG:boto:Token: None 2023-10-18T19:59:01.169 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.169 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tazcyqB8AH/M5PTeTp0g+eLsp3U= 2023-10-18T19:59:01.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tazcyqB8AH/M5PTeTp0g+eLsp3U=', 'Content-Length': '0'} 2023-10-18T19:59:01.172 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'obRs9QrejEnMKy2WaXOLHr.iYfq1qDe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d34631267d7451d4-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.173 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.173 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.174 DEBUG:boto:path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.174 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.174 DEBUG:boto:Method: GET 2023-10-18T19:59:01.175 DEBUG:boto:Path: /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.175 DEBUG:boto:Data: 2023-10-18T19:59:01.176 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.176 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.176 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.177 DEBUG:boto:Params: {} 2023-10-18T19:59:01.177 DEBUG:boto:Token: None 2023-10-18T19:59:01.178 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.178 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tazcyqB8AH/M5PTeTp0g+eLsp3U= 2023-10-18T19:59:01.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tazcyqB8AH/M5PTeTp0g+eLsp3U=', 'Content-Length': '0'} 2023-10-18T19:59:01.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'obRs9QrejEnMKy2WaXOLHr.iYfq1qDe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea2d3787d4802601-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.182 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.182 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.182 DEBUG:boto:path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.183 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.183 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.184 DEBUG:boto:Path: /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.184 DEBUG:boto:Data: 2023-10-18T19:59:01.184 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.185 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.185 DEBUG:boto:Params: {} 2023-10-18T19:59:01.186 DEBUG:boto:Token: None 2023-10-18T19:59:01.186 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.187 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XRZk6T4E65ywGikOLGnkVniyILE= 2023-10-18T19:59:01.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XRZk6T4E65ywGikOLGnkVniyILE=', 'Content-Length': '0'} 2023-10-18T19:59:01.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'obRs9QrejEnMKy2WaXOLHr.iYfq1qDe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d17453ba927c0abe-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.190 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.190 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.191 DEBUG:boto:path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.191 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.191 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.192 DEBUG:boto:Path: /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.192 DEBUG:boto:Data: 2023-10-18T19:59:01.192 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.193 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.193 DEBUG:boto:Params: {} 2023-10-18T19:59:01.194 DEBUG:boto:Token: None 2023-10-18T19:59:01.194 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.195 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XRZk6T4E65ywGikOLGnkVniyILE= 2023-10-18T19:59:01.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XRZk6T4E65ywGikOLGnkVniyILE=', 'Content-Length': '0'} 2023-10-18T19:59:01.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'obRs9QrejEnMKy2WaXOLHr.iYfq1qDe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b4ecbc0f70c6f83-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.198 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.199 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.199 DEBUG:boto:path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.200 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.200 DEBUG:boto:Method: GET 2023-10-18T19:59:01.200 DEBUG:boto:Path: /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.201 DEBUG:boto:Data: 2023-10-18T19:59:01.201 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.202 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.202 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.202 DEBUG:boto:Params: {} 2023-10-18T19:59:01.203 DEBUG:boto:Token: None 2023-10-18T19:59:01.203 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.204 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tazcyqB8AH/M5PTeTp0g+eLsp3U= 2023-10-18T19:59:01.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tazcyqB8AH/M5PTeTp0g+eLsp3U=', 'Content-Length': '0'} 2023-10-18T19:59:01.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'obRs9QrejEnMKy2WaXOLHr.iYfq1qDe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046a90b4beda6770b-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.207 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.207 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.208 DEBUG:boto:path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.208 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.208 DEBUG:boto:Method: GET 2023-10-18T19:59:01.209 DEBUG:boto:Path: /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.209 DEBUG:boto:Data: 2023-10-18T19:59:01.210 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.211 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.211 DEBUG:boto:Params: {} 2023-10-18T19:59:01.211 DEBUG:boto:Token: None 2023-10-18T19:59:01.212 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=obRs9QrejEnMKy2WaXOLHr.iYfq1qDe 2023-10-18T19:59:01.212 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tazcyqB8AH/M5PTeTp0g+eLsp3U= 2023-10-18T19:59:01.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tazcyqB8AH/M5PTeTp0g+eLsp3U=', 'Content-Length': '0'} 2023-10-18T19:59:01.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'obRs9QrejEnMKy2WaXOLHr.iYfq1qDe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001245536bc1bab59c-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.216 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.216 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.217 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.217 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.217 DEBUG:boto:Method: GET 2023-10-18T19:59:01.218 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.218 DEBUG:boto:Data: 2023-10-18T19:59:01.219 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.219 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.220 DEBUG:boto:Params: {} 2023-10-18T19:59:01.220 DEBUG:boto:Token: None 2023-10-18T19:59:01.221 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e1319ba17b156ea-0065303905-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.225 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.225 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.226 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.226 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.227 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.227 DEBUG:boto:Method: GET 2023-10-18T19:59:01.227 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.228 DEBUG:boto:Data: 2023-10-18T19:59:01.228 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.229 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.229 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.230 DEBUG:boto:Params: {} 2023-10-18T19:59:01.230 DEBUG:boto:Token: None 2023-10-18T19:59:01.231 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000ebda46f4df6e78-0065303905-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.235 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.235 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.236 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.236 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.237 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.237 DEBUG:boto:Data: 2023-10-18T19:59:01.237 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.238 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.238 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.238 DEBUG:boto:Params: {} 2023-10-18T19:59:01.239 DEBUG:boto:Token: None 2023-10-18T19:59:01.239 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a58456ff10c859ad-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.244 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.244 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.245 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.245 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.245 DEBUG:boto:Data: 2023-10-18T19:59:01.246 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.246 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.247 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.247 DEBUG:boto:Params: {} 2023-10-18T19:59:01.248 DEBUG:boto:Token: None 2023-10-18T19:59:01.248 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.249 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df61a40009291f44-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.253 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.254 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.254 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.255 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.255 DEBUG:boto:Method: GET 2023-10-18T19:59:01.256 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.256 DEBUG:boto:Data: 2023-10-18T19:59:01.256 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.257 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.257 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.258 DEBUG:boto:Params: {} 2023-10-18T19:59:01.258 DEBUG:boto:Token: None 2023-10-18T19:59:01.259 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.259 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs= 2023-10-18T19:59:01.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs=', 'Content-Length': '0'} 2023-10-18T19:59:01.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d168883305f8278-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.306 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.307 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.307 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.308 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.308 DEBUG:boto:Method: GET 2023-10-18T19:59:01.309 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.309 DEBUG:boto:Data: 2023-10-18T19:59:01.310 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.310 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.311 DEBUG:boto:Params: {} 2023-10-18T19:59:01.311 DEBUG:boto:Token: None 2023-10-18T19:59:01.312 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs= 2023-10-18T19:59:01.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs=', 'Content-Length': '0'} 2023-10-18T19:59:01.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000773129589c62901-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.316 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.316 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.317 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.317 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.317 DEBUG:boto:Method: GET 2023-10-18T19:59:01.318 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.318 DEBUG:boto:Data: 2023-10-18T19:59:01.319 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.320 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.320 DEBUG:boto:Params: {} 2023-10-18T19:59:01.321 DEBUG:boto:Token: None 2023-10-18T19:59:01.321 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000913045d12bb081e1-0065303905-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.325 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.326 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.326 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.327 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.327 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.327 DEBUG:boto:Method: GET 2023-10-18T19:59:01.328 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.328 DEBUG:boto:Data: 2023-10-18T19:59:01.329 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.329 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.330 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.330 DEBUG:boto:Params: {} 2023-10-18T19:59:01.330 DEBUG:boto:Token: None 2023-10-18T19:59:01.331 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.331 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096ac465c35d01255-0065303905-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.335 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.336 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.336 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.337 DEBUG:boto:path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.337 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.338 DEBUG:boto:Method: GET 2023-10-18T19:59:01.338 DEBUG:boto:Path: /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.338 DEBUG:boto:Data: 2023-10-18T19:59:01.339 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.339 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.340 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.340 DEBUG:boto:Params: {} 2023-10-18T19:59:01.341 DEBUG:boto:Token: None 2023-10-18T19:59:01.341 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.342 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XH8qofvHNWv2rCeH7lcIJbNDi6E= 2023-10-18T19:59:01.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XH8qofvHNWv2rCeH7lcIJbNDi6E=', 'Content-Length': '0'} 2023-10-18T19:59:01.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'GBljJZwndU8OOxk7KJyahMn7aN2joxp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048b051bebccd6db8-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.345 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.346 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.346 DEBUG:boto:path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.347 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.347 DEBUG:boto:Method: GET 2023-10-18T19:59:01.348 DEBUG:boto:Path: /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.348 DEBUG:boto:Data: 2023-10-18T19:59:01.349 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.349 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.350 DEBUG:boto:Params: {} 2023-10-18T19:59:01.350 DEBUG:boto:Token: None 2023-10-18T19:59:01.351 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XH8qofvHNWv2rCeH7lcIJbNDi6E= 2023-10-18T19:59:01.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XH8qofvHNWv2rCeH7lcIJbNDi6E=', 'Content-Length': '0'} 2023-10-18T19:59:01.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'GBljJZwndU8OOxk7KJyahMn7aN2joxp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013e8117582048288-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.355 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.355 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.356 DEBUG:boto:path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.356 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.356 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.357 DEBUG:boto:Path: /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.357 DEBUG:boto:Data: 2023-10-18T19:59:01.358 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.359 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.359 DEBUG:boto:Params: {} 2023-10-18T19:59:01.359 DEBUG:boto:Token: None 2023-10-18T19:59:01.360 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cJUsUpWJMD4ztDaMXaCxUiozvyM= 2023-10-18T19:59:01.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cJUsUpWJMD4ztDaMXaCxUiozvyM=', 'Content-Length': '0'} 2023-10-18T19:59:01.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'GBljJZwndU8OOxk7KJyahMn7aN2joxp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e8f3a4fe03f952f-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.363 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.363 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.364 DEBUG:boto:path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.364 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.365 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.365 DEBUG:boto:Path: /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.366 DEBUG:boto:Data: 2023-10-18T19:59:01.366 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.366 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.367 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.367 DEBUG:boto:Params: {} 2023-10-18T19:59:01.368 DEBUG:boto:Token: None 2023-10-18T19:59:01.368 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cJUsUpWJMD4ztDaMXaCxUiozvyM= 2023-10-18T19:59:01.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cJUsUpWJMD4ztDaMXaCxUiozvyM=', 'Content-Length': '0'} 2023-10-18T19:59:01.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'GBljJZwndU8OOxk7KJyahMn7aN2joxp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f99cf9e6fb4543f2-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.372 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.372 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.373 DEBUG:boto:path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.373 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.374 DEBUG:boto:Method: GET 2023-10-18T19:59:01.374 DEBUG:boto:Path: /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.375 DEBUG:boto:Data: 2023-10-18T19:59:01.375 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.375 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.376 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.376 DEBUG:boto:Params: {} 2023-10-18T19:59:01.377 DEBUG:boto:Token: None 2023-10-18T19:59:01.377 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.378 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XH8qofvHNWv2rCeH7lcIJbNDi6E= 2023-10-18T19:59:01.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XH8qofvHNWv2rCeH7lcIJbNDi6E=', 'Content-Length': '0'} 2023-10-18T19:59:01.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'GBljJZwndU8OOxk7KJyahMn7aN2joxp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfaac38c0970e207-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.426 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.427 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.428 DEBUG:boto:path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.428 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.428 DEBUG:boto:Method: GET 2023-10-18T19:59:01.429 DEBUG:boto:Path: /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.429 DEBUG:boto:Data: 2023-10-18T19:59:01.430 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.430 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.431 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.431 DEBUG:boto:Params: {} 2023-10-18T19:59:01.432 DEBUG:boto:Token: None 2023-10-18T19:59:01.432 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=GBljJZwndU8OOxk7KJyahMn7aN2joxp 2023-10-18T19:59:01.433 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XH8qofvHNWv2rCeH7lcIJbNDi6E= 2023-10-18T19:59:01.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XH8qofvHNWv2rCeH7lcIJbNDi6E=', 'Content-Length': '0'} 2023-10-18T19:59:01.435 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'GBljJZwndU8OOxk7KJyahMn7aN2joxp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004109711a219c48d2-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.436 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.436 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.437 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.437 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.437 DEBUG:boto:Method: GET 2023-10-18T19:59:01.438 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.438 DEBUG:boto:Data: 2023-10-18T19:59:01.439 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.439 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.440 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.440 DEBUG:boto:Params: {} 2023-10-18T19:59:01.441 DEBUG:boto:Token: None 2023-10-18T19:59:01.441 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.442 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acf1469ef4e4858b-0065303905-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.490 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.491 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.492 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.492 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.493 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.493 DEBUG:boto:Method: GET 2023-10-18T19:59:01.493 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.494 DEBUG:boto:Data: 2023-10-18T19:59:01.494 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.495 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.495 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.496 DEBUG:boto:Params: {} 2023-10-18T19:59:01.496 DEBUG:boto:Token: None 2023-10-18T19:59:01.497 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2adb44309782f6d-0065303905-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.501 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.502 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.502 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.503 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.503 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.503 DEBUG:boto:Data: 2023-10-18T19:59:01.504 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.504 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.505 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.505 DEBUG:boto:Params: {} 2023-10-18T19:59:01.505 DEBUG:boto:Token: None 2023-10-18T19:59:01.506 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.506 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8696ed142330e00-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.511 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.511 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.512 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.512 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.512 DEBUG:boto:Data: 2023-10-18T19:59:01.513 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.513 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.514 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.514 DEBUG:boto:Params: {} 2023-10-18T19:59:01.514 DEBUG:boto:Token: None 2023-10-18T19:59:01.515 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.515 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab13d17517ec277d-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.520 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.521 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.521 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.522 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.522 DEBUG:boto:Method: GET 2023-10-18T19:59:01.522 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.523 DEBUG:boto:Data: 2023-10-18T19:59:01.523 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.524 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.524 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.524 DEBUG:boto:Params: {} 2023-10-18T19:59:01.525 DEBUG:boto:Token: None 2023-10-18T19:59:01.525 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.526 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs= 2023-10-18T19:59:01.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs=', 'Content-Length': '0'} 2023-10-18T19:59:01.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005189a0067eacef9c-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.529 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.529 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.530 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.530 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.531 DEBUG:boto:Method: GET 2023-10-18T19:59:01.531 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.532 DEBUG:boto:Data: 2023-10-18T19:59:01.532 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.532 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.533 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.533 DEBUG:boto:Params: {} 2023-10-18T19:59:01.534 DEBUG:boto:Token: None 2023-10-18T19:59:01.534 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.535 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs= 2023-10-18T19:59:01.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs=', 'Content-Length': '0'} 2023-10-18T19:59:01.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf36267d7b5c2a19-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.578 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.579 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.579 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.580 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.580 DEBUG:boto:Method: GET 2023-10-18T19:59:01.581 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.581 DEBUG:boto:Data: 2023-10-18T19:59:01.581 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.582 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.583 DEBUG:boto:Params: {} 2023-10-18T19:59:01.583 DEBUG:boto:Token: None 2023-10-18T19:59:01.584 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.584 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040edd394f8056f9e-0065303905-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.588 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.589 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.589 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.590 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.590 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.591 DEBUG:boto:Method: GET 2023-10-18T19:59:01.591 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.591 DEBUG:boto:Data: 2023-10-18T19:59:01.592 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.592 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.593 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.593 DEBUG:boto:Params: {} 2023-10-18T19:59:01.594 DEBUG:boto:Token: None 2023-10-18T19:59:01.594 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009df01ac9851f484a-0065303905-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.598 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.599 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.600 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.600 DEBUG:boto:path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.600 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.601 DEBUG:boto:Method: GET 2023-10-18T19:59:01.601 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.602 DEBUG:boto:Data: 2023-10-18T19:59:01.602 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.603 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.603 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.604 DEBUG:boto:Params: {} 2023-10-18T19:59:01.604 DEBUG:boto:Token: None 2023-10-18T19:59:01.605 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5SIPT7Gep2q2g2gMLIFlB5Fgm4M= 2023-10-18T19:59:01.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5SIPT7Gep2q2g2gMLIFlB5Fgm4M=', 'Content-Length': '0'} 2023-10-18T19:59:01.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d84bb8c745882a2-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.650 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.651 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.651 DEBUG:boto:path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.652 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.652 DEBUG:boto:Method: GET 2023-10-18T19:59:01.653 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.653 DEBUG:boto:Data: 2023-10-18T19:59:01.654 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.655 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.655 DEBUG:boto:Params: {} 2023-10-18T19:59:01.655 DEBUG:boto:Token: None 2023-10-18T19:59:01.656 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5SIPT7Gep2q2g2gMLIFlB5Fgm4M= 2023-10-18T19:59:01.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5SIPT7Gep2q2g2gMLIFlB5Fgm4M=', 'Content-Length': '0'} 2023-10-18T19:59:01.659 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055da8ed0a843586d-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.660 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.660 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.661 DEBUG:boto:path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.661 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.661 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.662 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.662 DEBUG:boto:Data: 2023-10-18T19:59:01.663 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.663 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.663 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.664 DEBUG:boto:Params: {} 2023-10-18T19:59:01.664 DEBUG:boto:Token: None 2023-10-18T19:59:01.665 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:axUgUh08W/BNzDI96M6N+S69yVM= 2023-10-18T19:59:01.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:axUgUh08W/BNzDI96M6N+S69yVM=', 'Content-Length': '0'} 2023-10-18T19:59:01.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b9dc438f704a3ec-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.668 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.669 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.669 DEBUG:boto:path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.669 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.670 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.670 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.671 DEBUG:boto:Data: 2023-10-18T19:59:01.671 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.672 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.672 DEBUG:boto:Params: {} 2023-10-18T19:59:01.673 DEBUG:boto:Token: None 2023-10-18T19:59:01.673 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:axUgUh08W/BNzDI96M6N+S69yVM= 2023-10-18T19:59:01.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:axUgUh08W/BNzDI96M6N+S69yVM=', 'Content-Length': '0'} 2023-10-18T19:59:01.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000532fda070867f6f2-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.677 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.677 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.678 DEBUG:boto:path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.678 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.679 DEBUG:boto:Method: GET 2023-10-18T19:59:01.679 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.679 DEBUG:boto:Data: 2023-10-18T19:59:01.680 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.681 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.681 DEBUG:boto:Params: {} 2023-10-18T19:59:01.682 DEBUG:boto:Token: None 2023-10-18T19:59:01.682 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5SIPT7Gep2q2g2gMLIFlB5Fgm4M= 2023-10-18T19:59:01.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5SIPT7Gep2q2g2gMLIFlB5Fgm4M=', 'Content-Length': '0'} 2023-10-18T19:59:01.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033ff8aa8cbd3487c-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.685 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.686 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.686 DEBUG:boto:path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.687 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.687 DEBUG:boto:Method: GET 2023-10-18T19:59:01.688 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.688 DEBUG:boto:Data: 2023-10-18T19:59:01.688 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.689 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.690 DEBUG:boto:Params: {} 2023-10-18T19:59:01.690 DEBUG:boto:Token: None 2023-10-18T19:59:01.691 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ 2023-10-18T19:59:01.691 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5SIPT7Gep2q2g2gMLIFlB5Fgm4M= 2023-10-18T19:59:01.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5SIPT7Gep2q2g2gMLIFlB5Fgm4M=', 'Content-Length': '0'} 2023-10-18T19:59:01.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qx9JNJ-cAf2CB7fwdF8n.40VRiX85wJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a6d2550dda68f5a-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.694 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.695 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.695 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.696 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.696 DEBUG:boto:Method: GET 2023-10-18T19:59:01.696 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.697 DEBUG:boto:Data: 2023-10-18T19:59:01.697 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.698 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.699 DEBUG:boto:Params: {} 2023-10-18T19:59:01.699 DEBUG:boto:Token: None 2023-10-18T19:59:01.699 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.700 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020fb94a2c5c8c399-0065303905-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.703 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.704 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.705 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.705 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.705 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.706 DEBUG:boto:Method: GET 2023-10-18T19:59:01.706 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.707 DEBUG:boto:Data: 2023-10-18T19:59:01.707 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.708 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.708 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.708 DEBUG:boto:Params: {} 2023-10-18T19:59:01.709 DEBUG:boto:Token: None 2023-10-18T19:59:01.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f80717f05350cd91-0065303905-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.713 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.714 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.714 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.715 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.715 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.716 DEBUG:boto:Data: 2023-10-18T19:59:01.716 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.716 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.717 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.717 DEBUG:boto:Params: {} 2023-10-18T19:59:01.718 DEBUG:boto:Token: None 2023-10-18T19:59:01.718 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.718 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ac370c7ff36b7fa-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.723 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.723 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.724 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.724 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.724 DEBUG:boto:Data: 2023-10-18T19:59:01.725 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.726 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.726 DEBUG:boto:Params: {} 2023-10-18T19:59:01.726 DEBUG:boto:Token: None 2023-10-18T19:59:01.727 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.727 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9ad55b6defa378b-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.732 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.732 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.733 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.733 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.733 DEBUG:boto:Method: GET 2023-10-18T19:59:01.734 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.734 DEBUG:boto:Data: 2023-10-18T19:59:01.735 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.735 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.735 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.736 DEBUG:boto:Params: {} 2023-10-18T19:59:01.736 DEBUG:boto:Token: None 2023-10-18T19:59:01.737 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs= 2023-10-18T19:59:01.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs=', 'Content-Length': '0'} 2023-10-18T19:59:01.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fa838c71bae903b-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.786 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.787 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.787 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.788 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.788 DEBUG:boto:Method: GET 2023-10-18T19:59:01.789 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.789 DEBUG:boto:Data: 2023-10-18T19:59:01.789 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.790 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.791 DEBUG:boto:Params: {} 2023-10-18T19:59:01.791 DEBUG:boto:Token: None 2023-10-18T19:59:01.792 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs= 2023-10-18T19:59:01.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8OeemoWPVYiKRKw5cwcdjKraFs=', 'Content-Length': '0'} 2023-10-18T19:59:01.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ea70cb937cdafbf-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.795 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.796 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.796 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.797 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.797 DEBUG:boto:Method: GET 2023-10-18T19:59:01.797 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.798 DEBUG:boto:Data: 2023-10-18T19:59:01.798 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.799 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.799 DEBUG:boto:Params: {} 2023-10-18T19:59:01.800 DEBUG:boto:Token: None 2023-10-18T19:59:01.800 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000941d9f29f904e39c-0065303905-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.805 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.805 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.806 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.806 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.807 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.807 DEBUG:boto:Method: GET 2023-10-18T19:59:01.807 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.808 DEBUG:boto:Data: 2023-10-18T19:59:01.808 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.809 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.810 DEBUG:boto:Params: {} 2023-10-18T19:59:01.810 DEBUG:boto:Token: None 2023-10-18T19:59:01.811 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000a6bb3902008bcb-0065303905-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.854 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.856 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.856 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.857 DEBUG:boto:path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.857 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.857 DEBUG:boto:Method: GET 2023-10-18T19:59:01.858 DEBUG:boto:Path: /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.858 DEBUG:boto:Data: 2023-10-18T19:59:01.859 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.859 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.860 DEBUG:boto:Params: {} 2023-10-18T19:59:01.860 DEBUG:boto:Token: None 2023-10-18T19:59:01.861 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.861 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4dVVXe73rEEqHars+GLtvaQ4qrM= 2023-10-18T19:59:01.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4dVVXe73rEEqHars+GLtvaQ4qrM=', 'Content-Length': '0'} 2023-10-18T19:59:01.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ecd0599d122d45a-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.866 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.867 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.867 DEBUG:boto:path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.868 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.868 DEBUG:boto:Method: GET 2023-10-18T19:59:01.869 DEBUG:boto:Path: /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.869 DEBUG:boto:Data: 2023-10-18T19:59:01.870 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.871 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.871 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.872 DEBUG:boto:Params: {} 2023-10-18T19:59:01.873 DEBUG:boto:Token: None 2023-10-18T19:59:01.873 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.874 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4dVVXe73rEEqHars+GLtvaQ4qrM= 2023-10-18T19:59:01.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4dVVXe73rEEqHars+GLtvaQ4qrM=', 'Content-Length': '0'} 2023-10-18T19:59:01.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000401896a01c30b297-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.877 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.878 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.878 DEBUG:boto:path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.879 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.879 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.880 DEBUG:boto:Path: /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.880 DEBUG:boto:Data: 2023-10-18T19:59:01.881 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.881 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.881 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.882 DEBUG:boto:Params: {} 2023-10-18T19:59:01.882 DEBUG:boto:Token: None 2023-10-18T19:59:01.883 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.883 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+7FMMTlsfG7j5kbWg6bn7AAwPaw= 2023-10-18T19:59:01.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+7FMMTlsfG7j5kbWg6bn7AAwPaw=', 'Content-Length': '0'} 2023-10-18T19:59:01.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b89a110fb346e92b-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.887 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.887 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.887 DEBUG:boto:path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.888 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.888 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.889 DEBUG:boto:Path: /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.889 DEBUG:boto:Data: 2023-10-18T19:59:01.890 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.890 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.891 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.891 DEBUG:boto:Params: {} 2023-10-18T19:59:01.891 DEBUG:boto:Token: None 2023-10-18T19:59:01.892 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.892 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+7FMMTlsfG7j5kbWg6bn7AAwPaw= 2023-10-18T19:59:01.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+7FMMTlsfG7j5kbWg6bn7AAwPaw=', 'Content-Length': '0'} 2023-10-18T19:59:01.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000969697be9730be92-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.896 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.896 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.897 DEBUG:boto:path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.897 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.897 DEBUG:boto:Method: GET 2023-10-18T19:59:01.898 DEBUG:boto:Path: /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.898 DEBUG:boto:Data: 2023-10-18T19:59:01.899 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.899 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.899 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.900 DEBUG:boto:Params: {} 2023-10-18T19:59:01.900 DEBUG:boto:Token: None 2023-10-18T19:59:01.901 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.901 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4dVVXe73rEEqHars+GLtvaQ4qrM= 2023-10-18T19:59:01.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4dVVXe73rEEqHars+GLtvaQ4qrM=', 'Content-Length': '0'} 2023-10-18T19:59:01.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dec074075f04d5a4-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.946 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.947 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.947 DEBUG:boto:path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.948 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.948 DEBUG:boto:Method: GET 2023-10-18T19:59:01.949 DEBUG:boto:Path: /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.949 DEBUG:boto:Data: 2023-10-18T19:59:01.949 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.950 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.950 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.951 DEBUG:boto:Params: {} 2023-10-18T19:59:01.951 DEBUG:boto:Token: None 2023-10-18T19:59:01.952 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj?versionId=WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6 2023-10-18T19:59:01.952 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4dVVXe73rEEqHars+GLtvaQ4qrM= 2023-10-18T19:59:01.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4dVVXe73rEEqHars+GLtvaQ4qrM=', 'Content-Length': '0'} 2023-10-18T19:59:01.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'WPed-jyVQTR-HWvNwOCST8Ob9Ta4vw6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa48b3c36d03bcde-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.955 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.956 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.956 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.957 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.957 DEBUG:boto:Method: GET 2023-10-18T19:59:01.958 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.958 DEBUG:boto:Data: 2023-10-18T19:59:01.958 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.959 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.959 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.960 DEBUG:boto:Params: {} 2023-10-18T19:59:01.960 DEBUG:boto:Token: None 2023-10-18T19:59:01.961 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.961 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef1ce7fb6914889a-0065303905-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.965 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.965 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:01.966 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:01.966 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.967 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:01.967 DEBUG:boto:Method: GET 2023-10-18T19:59:01.968 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:01.968 DEBUG:boto:Data: 2023-10-18T19:59:01.968 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.969 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.970 DEBUG:boto:Params: {} 2023-10-18T19:59:01.970 DEBUG:boto:Token: None 2023-10-18T19:59:01.970 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/ 2023-10-18T19:59:01.971 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI= 2023-10-18T19:59:01.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kfpf6CYsD5r3/y2M11zDk+O6aZI=', 'Content-Length': '0'} 2023-10-18T19:59:01.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052b56c243ec94b7c-0065303905-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.977 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:01.978 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.978 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.979 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.979 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.980 DEBUG:boto:Data: 2023-10-18T19:59:01.980 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.980 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:01.981 DEBUG:boto:Port: 8001 2023-10-18T19:59:01.981 DEBUG:boto:Params: {} 2023-10-18T19:59:01.982 DEBUG:boto:Token: None 2023-10-18T19:59:01.982 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d629d4cd50001811-0065303905-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.987 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.987 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.988 DEBUG:boto:Method: HEAD 2023-10-18T19:59:01.988 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:01.988 DEBUG:boto:Data: 2023-10-18T19:59:01.989 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.989 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:01.989 DEBUG:boto:Port: 8000 2023-10-18T19:59:01.990 DEBUG:boto:Params: {} 2023-10-18T19:59:01.990 DEBUG:boto:Token: None 2023-10-18T19:59:01.991 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:01 GMT /bofrnf-13/obj 2023-10-18T19:59:01.991 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I= 2023-10-18T19:59:01.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pbio6gzGV1xRbBbNHcTk2MT5N9I=', 'Content-Length': '0'} 2023-10-18T19:59:01.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b6b38a0aaf8cd41-0065303905-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:01.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:01.996 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:01.996 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:01.997 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.997 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.998 DEBUG:boto:Method: GET 2023-10-18T19:59:01.998 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:01.998 DEBUG:boto:Data: 2023-10-18T19:59:01.999 DEBUG:boto:Headers: {} 2023-10-18T19:59:01.999 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.000 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.000 DEBUG:boto:Params: {} 2023-10-18T19:59:02.001 DEBUG:boto:Token: None 2023-10-18T19:59:02.001 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.001 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s= 2023-10-18T19:59:02.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s=', 'Content-Length': '0'} 2023-10-18T19:59:02.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d608ba4c55fe79b-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.005 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.005 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.005 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.006 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.006 DEBUG:boto:Method: GET 2023-10-18T19:59:02.007 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.007 DEBUG:boto:Data: 2023-10-18T19:59:02.008 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.008 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.008 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.009 DEBUG:boto:Params: {} 2023-10-18T19:59:02.009 DEBUG:boto:Token: None 2023-10-18T19:59:02.010 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.010 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s= 2023-10-18T19:59:02.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s=', 'Content-Length': '0'} 2023-10-18T19:59:02.013 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e3123943f6a5b94-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.058 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.059 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.059 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.060 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.060 DEBUG:boto:Method: GET 2023-10-18T19:59:02.061 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.061 DEBUG:boto:Data: 2023-10-18T19:59:02.062 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.062 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.062 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.063 DEBUG:boto:Params: {} 2023-10-18T19:59:02.063 DEBUG:boto:Token: None 2023-10-18T19:59:02.064 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.064 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000852de7e4fcf149b-0065303906-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.068 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.069 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.069 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.070 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.070 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.070 DEBUG:boto:Method: GET 2023-10-18T19:59:02.071 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.071 DEBUG:boto:Data: 2023-10-18T19:59:02.072 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.072 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.072 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.073 DEBUG:boto:Params: {} 2023-10-18T19:59:02.073 DEBUG:boto:Token: None 2023-10-18T19:59:02.074 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.074 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef38676c033df0bb-0065303906-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.078 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.079 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.080 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.080 DEBUG:boto:path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.080 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.081 DEBUG:boto:Method: GET 2023-10-18T19:59:02.081 DEBUG:boto:Path: /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.082 DEBUG:boto:Data: 2023-10-18T19:59:02.082 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.083 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.083 DEBUG:boto:Params: {} 2023-10-18T19:59:02.084 DEBUG:boto:Token: None 2023-10-18T19:59:02.084 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4bcyJOvnUcVHKjLRg+AXIdNFuZI= 2023-10-18T19:59:02.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4bcyJOvnUcVHKjLRg+AXIdNFuZI=', 'Content-Length': '0'} 2023-10-18T19:59:02.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'm--RJO4V9Vrua4FSRFh-ASpvQgROG6k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000419dd9cebe16eb17-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.130 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.131 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.131 DEBUG:boto:path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.132 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.132 DEBUG:boto:Method: GET 2023-10-18T19:59:02.133 DEBUG:boto:Path: /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.133 DEBUG:boto:Data: 2023-10-18T19:59:02.134 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.134 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.134 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.135 DEBUG:boto:Params: {} 2023-10-18T19:59:02.135 DEBUG:boto:Token: None 2023-10-18T19:59:02.136 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.136 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4bcyJOvnUcVHKjLRg+AXIdNFuZI= 2023-10-18T19:59:02.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4bcyJOvnUcVHKjLRg+AXIdNFuZI=', 'Content-Length': '0'} 2023-10-18T19:59:02.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'm--RJO4V9Vrua4FSRFh-ASpvQgROG6k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9e9eacfd0c8c9f4-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.140 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.140 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.140 DEBUG:boto:path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.141 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.141 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.142 DEBUG:boto:Path: /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.142 DEBUG:boto:Data: 2023-10-18T19:59:02.142 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.143 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.143 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.144 DEBUG:boto:Params: {} 2023-10-18T19:59:02.144 DEBUG:boto:Token: None 2023-10-18T19:59:02.145 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.145 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Irh7EdCSJW9OMhqMGVt/cW/ds+Y= 2023-10-18T19:59:02.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Irh7EdCSJW9OMhqMGVt/cW/ds+Y=', 'Content-Length': '0'} 2023-10-18T19:59:02.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'm--RJO4V9Vrua4FSRFh-ASpvQgROG6k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095feaf2f2f4657a1-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.148 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.148 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.149 DEBUG:boto:path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.149 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.150 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.150 DEBUG:boto:Path: /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.150 DEBUG:boto:Data: 2023-10-18T19:59:02.151 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.151 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.152 DEBUG:boto:Params: {} 2023-10-18T19:59:02.152 DEBUG:boto:Token: None 2023-10-18T19:59:02.153 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Irh7EdCSJW9OMhqMGVt/cW/ds+Y= 2023-10-18T19:59:02.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Irh7EdCSJW9OMhqMGVt/cW/ds+Y=', 'Content-Length': '0'} 2023-10-18T19:59:02.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'm--RJO4V9Vrua4FSRFh-ASpvQgROG6k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eab478e7b20d2c4c-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.157 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.157 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.158 DEBUG:boto:path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.158 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.159 DEBUG:boto:Method: GET 2023-10-18T19:59:02.159 DEBUG:boto:Path: /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.159 DEBUG:boto:Data: 2023-10-18T19:59:02.160 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.160 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.161 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.161 DEBUG:boto:Params: {} 2023-10-18T19:59:02.162 DEBUG:boto:Token: None 2023-10-18T19:59:02.162 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.162 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4bcyJOvnUcVHKjLRg+AXIdNFuZI= 2023-10-18T19:59:02.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4bcyJOvnUcVHKjLRg+AXIdNFuZI=', 'Content-Length': '0'} 2023-10-18T19:59:02.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'm--RJO4V9Vrua4FSRFh-ASpvQgROG6k'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f728589217001fdb-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.165 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.166 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.166 DEBUG:boto:path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.167 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.167 DEBUG:boto:Method: GET 2023-10-18T19:59:02.168 DEBUG:boto:Path: /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.168 DEBUG:boto:Data: 2023-10-18T19:59:02.168 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.169 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.170 DEBUG:boto:Params: {} 2023-10-18T19:59:02.170 DEBUG:boto:Token: None 2023-10-18T19:59:02.171 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=m--RJO4V9Vrua4FSRFh-ASpvQgROG6k 2023-10-18T19:59:02.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4bcyJOvnUcVHKjLRg+AXIdNFuZI= 2023-10-18T19:59:02.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4bcyJOvnUcVHKjLRg+AXIdNFuZI=', 'Content-Length': '0'} 2023-10-18T19:59:02.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'm--RJO4V9Vrua4FSRFh-ASpvQgROG6k'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008609e70cd01a8424-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.174 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.175 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.175 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.176 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.176 DEBUG:boto:Method: GET 2023-10-18T19:59:02.176 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.177 DEBUG:boto:Data: 2023-10-18T19:59:02.177 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.178 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.179 DEBUG:boto:Params: {} 2023-10-18T19:59:02.179 DEBUG:boto:Token: None 2023-10-18T19:59:02.179 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005270f55da23720af-0065303906-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.186 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.186 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.187 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.187 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.188 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.188 DEBUG:boto:Method: GET 2023-10-18T19:59:02.188 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.189 DEBUG:boto:Data: 2023-10-18T19:59:02.189 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.190 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.190 DEBUG:boto:Params: {} 2023-10-18T19:59:02.191 DEBUG:boto:Token: None 2023-10-18T19:59:02.191 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004364e8759e209e30-0065303906-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.195 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.196 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.196 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.197 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.197 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:02.198 DEBUG:boto:Data: 2023-10-18T19:59:02.198 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.198 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.199 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.199 DEBUG:boto:Params: {} 2023-10-18T19:59:02.200 DEBUG:boto:Token: None 2023-10-18T19:59:02.200 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj 2023-10-18T19:59:02.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ= 2023-10-18T19:59:02.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ=', 'Content-Length': '0'} 2023-10-18T19:59:02.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002764106717521e79-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.205 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.205 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.206 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.206 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:02.206 DEBUG:boto:Data: 2023-10-18T19:59:02.207 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.207 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.208 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.208 DEBUG:boto:Params: {} 2023-10-18T19:59:02.208 DEBUG:boto:Token: None 2023-10-18T19:59:02.209 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj 2023-10-18T19:59:02.209 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ= 2023-10-18T19:59:02.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ=', 'Content-Length': '0'} 2023-10-18T19:59:02.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d1f05b64ae676f6-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.214 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.214 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.215 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.215 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.216 DEBUG:boto:Method: GET 2023-10-18T19:59:02.216 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.217 DEBUG:boto:Data: 2023-10-18T19:59:02.217 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.218 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.219 DEBUG:boto:Params: {} 2023-10-18T19:59:02.219 DEBUG:boto:Token: None 2023-10-18T19:59:02.220 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s= 2023-10-18T19:59:02.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s=', 'Content-Length': '0'} 2023-10-18T19:59:02.223 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000778f8e5b8559bbd4-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.266 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.267 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.268 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.268 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.268 DEBUG:boto:Method: GET 2023-10-18T19:59:02.269 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.269 DEBUG:boto:Data: 2023-10-18T19:59:02.270 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.270 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.271 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.271 DEBUG:boto:Params: {} 2023-10-18T19:59:02.272 DEBUG:boto:Token: None 2023-10-18T19:59:02.272 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.273 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s= 2023-10-18T19:59:02.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s=', 'Content-Length': '0'} 2023-10-18T19:59:02.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e181f1b0ae57790-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.276 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.276 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.277 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.277 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.278 DEBUG:boto:Method: GET 2023-10-18T19:59:02.278 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.278 DEBUG:boto:Data: 2023-10-18T19:59:02.279 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.280 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.280 DEBUG:boto:Params: {} 2023-10-18T19:59:02.281 DEBUG:boto:Token: None 2023-10-18T19:59:02.281 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.282 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068febef2355abbe4-0065303906-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.285 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.286 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.286 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.287 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.287 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.287 DEBUG:boto:Method: GET 2023-10-18T19:59:02.288 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.288 DEBUG:boto:Data: 2023-10-18T19:59:02.289 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.289 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.289 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.290 DEBUG:boto:Params: {} 2023-10-18T19:59:02.290 DEBUG:boto:Token: None 2023-10-18T19:59:02.291 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ea890d29ed63717-0065303906-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.295 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.296 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.296 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.297 DEBUG:boto:path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.297 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.297 DEBUG:boto:Method: GET 2023-10-18T19:59:02.298 DEBUG:boto:Path: /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.298 DEBUG:boto:Data: 2023-10-18T19:59:02.299 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.300 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.300 DEBUG:boto:Params: {} 2023-10-18T19:59:02.300 DEBUG:boto:Token: None 2023-10-18T19:59:02.301 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bA/bFIgBe/cGchcF+qKuQjL17Uk= 2023-10-18T19:59:02.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bA/bFIgBe/cGchcF+qKuQjL17Uk=', 'Content-Length': '0'} 2023-10-18T19:59:02.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'U42vKDwj1WFjI-bMrhPe5O7B-9dabl9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfb7fa4d1046d4bf-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.305 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.305 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.306 DEBUG:boto:path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.306 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.307 DEBUG:boto:Method: GET 2023-10-18T19:59:02.307 DEBUG:boto:Path: /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.307 DEBUG:boto:Data: 2023-10-18T19:59:02.308 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.308 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.309 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.309 DEBUG:boto:Params: {} 2023-10-18T19:59:02.309 DEBUG:boto:Token: None 2023-10-18T19:59:02.310 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.311 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bA/bFIgBe/cGchcF+qKuQjL17Uk= 2023-10-18T19:59:02.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bA/bFIgBe/cGchcF+qKuQjL17Uk=', 'Content-Length': '0'} 2023-10-18T19:59:02.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'U42vKDwj1WFjI-bMrhPe5O7B-9dabl9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004434be23b48c6d51-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.314 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.314 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.315 DEBUG:boto:path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.315 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.315 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.316 DEBUG:boto:Path: /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.316 DEBUG:boto:Data: 2023-10-18T19:59:02.317 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.317 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.317 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.318 DEBUG:boto:Params: {} 2023-10-18T19:59:02.318 DEBUG:boto:Token: None 2023-10-18T19:59:02.319 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.319 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RDOMw8iRe4xmD13qn06AJGXCa9w= 2023-10-18T19:59:02.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RDOMw8iRe4xmD13qn06AJGXCa9w=', 'Content-Length': '0'} 2023-10-18T19:59:02.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'U42vKDwj1WFjI-bMrhPe5O7B-9dabl9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a80b0723b7d9f451-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.322 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.323 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.323 DEBUG:boto:path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.323 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.324 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.324 DEBUG:boto:Path: /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.325 DEBUG:boto:Data: 2023-10-18T19:59:02.325 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.325 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.326 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.326 DEBUG:boto:Params: {} 2023-10-18T19:59:02.327 DEBUG:boto:Token: None 2023-10-18T19:59:02.327 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RDOMw8iRe4xmD13qn06AJGXCa9w= 2023-10-18T19:59:02.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RDOMw8iRe4xmD13qn06AJGXCa9w=', 'Content-Length': '0'} 2023-10-18T19:59:02.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'U42vKDwj1WFjI-bMrhPe5O7B-9dabl9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003986f5c8b0ccb93-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.331 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.331 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.332 DEBUG:boto:path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.332 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.333 DEBUG:boto:Method: GET 2023-10-18T19:59:02.333 DEBUG:boto:Path: /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.333 DEBUG:boto:Data: 2023-10-18T19:59:02.334 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.334 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.335 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.335 DEBUG:boto:Params: {} 2023-10-18T19:59:02.336 DEBUG:boto:Token: None 2023-10-18T19:59:02.336 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bA/bFIgBe/cGchcF+qKuQjL17Uk= 2023-10-18T19:59:02.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bA/bFIgBe/cGchcF+qKuQjL17Uk=', 'Content-Length': '0'} 2023-10-18T19:59:02.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'U42vKDwj1WFjI-bMrhPe5O7B-9dabl9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c70cfad12e3467b-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.386 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.387 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.387 DEBUG:boto:path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.388 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.388 DEBUG:boto:Method: GET 2023-10-18T19:59:02.389 DEBUG:boto:Path: /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.389 DEBUG:boto:Data: 2023-10-18T19:59:02.389 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.390 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.391 DEBUG:boto:Params: {} 2023-10-18T19:59:02.391 DEBUG:boto:Token: None 2023-10-18T19:59:02.392 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=U42vKDwj1WFjI-bMrhPe5O7B-9dabl9 2023-10-18T19:59:02.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bA/bFIgBe/cGchcF+qKuQjL17Uk= 2023-10-18T19:59:02.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bA/bFIgBe/cGchcF+qKuQjL17Uk=', 'Content-Length': '0'} 2023-10-18T19:59:02.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'U42vKDwj1WFjI-bMrhPe5O7B-9dabl9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3ff78efa2be3923-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.395 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.396 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.396 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.397 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.397 DEBUG:boto:Method: GET 2023-10-18T19:59:02.398 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.398 DEBUG:boto:Data: 2023-10-18T19:59:02.398 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.399 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.400 DEBUG:boto:Params: {} 2023-10-18T19:59:02.400 DEBUG:boto:Token: None 2023-10-18T19:59:02.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8f8b6649d42020-0065303906-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.405 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.405 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.406 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.406 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.407 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.407 DEBUG:boto:Method: GET 2023-10-18T19:59:02.407 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.408 DEBUG:boto:Data: 2023-10-18T19:59:02.408 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.409 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.410 DEBUG:boto:Params: {} 2023-10-18T19:59:02.410 DEBUG:boto:Token: None 2023-10-18T19:59:02.411 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000368e8e22b7f3c79-0065303906-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.458 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.459 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.460 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.460 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.460 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:02.461 DEBUG:boto:Data: 2023-10-18T19:59:02.461 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.462 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.463 DEBUG:boto:Params: {} 2023-10-18T19:59:02.463 DEBUG:boto:Token: None 2023-10-18T19:59:02.463 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj 2023-10-18T19:59:02.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ= 2023-10-18T19:59:02.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ=', 'Content-Length': '0'} 2023-10-18T19:59:02.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f0e9d71f4cb3a75-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.468 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.469 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.469 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.469 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:02.470 DEBUG:boto:Data: 2023-10-18T19:59:02.470 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.471 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.471 DEBUG:boto:Params: {} 2023-10-18T19:59:02.472 DEBUG:boto:Token: None 2023-10-18T19:59:02.472 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj 2023-10-18T19:59:02.473 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ= 2023-10-18T19:59:02.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ=', 'Content-Length': '0'} 2023-10-18T19:59:02.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c72e4c21445a4155-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.478 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.478 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.478 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.479 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.479 DEBUG:boto:Method: GET 2023-10-18T19:59:02.480 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.480 DEBUG:boto:Data: 2023-10-18T19:59:02.481 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.482 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.482 DEBUG:boto:Params: {} 2023-10-18T19:59:02.482 DEBUG:boto:Token: None 2023-10-18T19:59:02.483 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.483 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s= 2023-10-18T19:59:02.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s=', 'Content-Length': '0'} 2023-10-18T19:59:02.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd76059828fd771c-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.526 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.527 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.527 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.528 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.528 DEBUG:boto:Method: GET 2023-10-18T19:59:02.529 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.529 DEBUG:boto:Data: 2023-10-18T19:59:02.530 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.530 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.531 DEBUG:boto:Params: {} 2023-10-18T19:59:02.531 DEBUG:boto:Token: None 2023-10-18T19:59:02.532 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s= 2023-10-18T19:59:02.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s=', 'Content-Length': '0'} 2023-10-18T19:59:02.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081d1ec0df27b751a-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.584 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.585 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.585 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.585 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.586 DEBUG:boto:Method: GET 2023-10-18T19:59:02.586 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.587 DEBUG:boto:Data: 2023-10-18T19:59:02.587 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.588 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.588 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.588 DEBUG:boto:Params: {} 2023-10-18T19:59:02.589 DEBUG:boto:Token: None 2023-10-18T19:59:02.589 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.590 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d18076ed4c0a4a3-0065303906-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.594 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.595 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.595 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.596 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.596 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.597 DEBUG:boto:Method: GET 2023-10-18T19:59:02.597 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.597 DEBUG:boto:Data: 2023-10-18T19:59:02.598 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.598 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.599 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.599 DEBUG:boto:Params: {} 2023-10-18T19:59:02.600 DEBUG:boto:Token: None 2023-10-18T19:59:02.600 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.601 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c9e9cb25009f849-0065303906-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.604 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.605 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.606 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.606 DEBUG:boto:path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.607 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.607 DEBUG:boto:Method: GET 2023-10-18T19:59:02.608 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.608 DEBUG:boto:Data: 2023-10-18T19:59:02.608 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.609 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.609 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.610 DEBUG:boto:Params: {} 2023-10-18T19:59:02.610 DEBUG:boto:Token: None 2023-10-18T19:59:02.611 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.611 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdVg7IFVgcnTf+PwWSC0WfDLXTw= 2023-10-18T19:59:02.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdVg7IFVgcnTf+PwWSC0WfDLXTw=', 'Content-Length': '0'} 2023-10-18T19:59:02.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d41fad8b871fa93-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.615 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.616 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.616 DEBUG:boto:path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.617 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.617 DEBUG:boto:Method: GET 2023-10-18T19:59:02.617 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.618 DEBUG:boto:Data: 2023-10-18T19:59:02.618 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.619 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.619 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.619 DEBUG:boto:Params: {} 2023-10-18T19:59:02.620 DEBUG:boto:Token: None 2023-10-18T19:59:02.620 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.621 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdVg7IFVgcnTf+PwWSC0WfDLXTw= 2023-10-18T19:59:02.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdVg7IFVgcnTf+PwWSC0WfDLXTw=', 'Content-Length': '0'} 2023-10-18T19:59:02.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005311d348a2468842-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.670 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.671 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.671 DEBUG:boto:path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.672 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.672 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.673 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.673 DEBUG:boto:Data: 2023-10-18T19:59:02.673 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.674 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.674 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.675 DEBUG:boto:Params: {} 2023-10-18T19:59:02.675 DEBUG:boto:Token: None 2023-10-18T19:59:02.676 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:snrhqBjTTw6pJDsTpWyfBSsP/hc= 2023-10-18T19:59:02.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:snrhqBjTTw6pJDsTpWyfBSsP/hc=', 'Content-Length': '0'} 2023-10-18T19:59:02.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011f20c0d8c22b875-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.679 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.680 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.680 DEBUG:boto:path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.681 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.681 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.681 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.682 DEBUG:boto:Data: 2023-10-18T19:59:02.682 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.683 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.683 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.683 DEBUG:boto:Params: {} 2023-10-18T19:59:02.684 DEBUG:boto:Token: None 2023-10-18T19:59:02.684 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.685 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:snrhqBjTTw6pJDsTpWyfBSsP/hc= 2023-10-18T19:59:02.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:snrhqBjTTw6pJDsTpWyfBSsP/hc=', 'Content-Length': '0'} 2023-10-18T19:59:02.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004715ec32c4356d9b-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.688 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.689 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.689 DEBUG:boto:path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.690 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.690 DEBUG:boto:Method: GET 2023-10-18T19:59:02.691 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.691 DEBUG:boto:Data: 2023-10-18T19:59:02.691 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.692 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.692 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.693 DEBUG:boto:Params: {} 2023-10-18T19:59:02.693 DEBUG:boto:Token: None 2023-10-18T19:59:02.694 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.694 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdVg7IFVgcnTf+PwWSC0WfDLXTw= 2023-10-18T19:59:02.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdVg7IFVgcnTf+PwWSC0WfDLXTw=', 'Content-Length': '0'} 2023-10-18T19:59:02.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc3987c8591fa7b7-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.698 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.698 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.698 DEBUG:boto:path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.699 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.699 DEBUG:boto:Method: GET 2023-10-18T19:59:02.700 DEBUG:boto:Path: /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.700 DEBUG:boto:Data: 2023-10-18T19:59:02.701 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.701 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.701 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.702 DEBUG:boto:Params: {} 2023-10-18T19:59:02.702 DEBUG:boto:Token: None 2023-10-18T19:59:02.703 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v 2023-10-18T19:59:02.703 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdVg7IFVgcnTf+PwWSC0WfDLXTw= 2023-10-18T19:59:02.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdVg7IFVgcnTf+PwWSC0WfDLXTw=', 'Content-Length': '0'} 2023-10-18T19:59:02.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'qjJoJ0JywLBVngkF4NqtHZRMQQpSi7v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007533e00e6ca7f027-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.746 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.747 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.747 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.748 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.748 DEBUG:boto:Method: GET 2023-10-18T19:59:02.749 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.749 DEBUG:boto:Data: 2023-10-18T19:59:02.750 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.750 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.751 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.751 DEBUG:boto:Params: {} 2023-10-18T19:59:02.751 DEBUG:boto:Token: None 2023-10-18T19:59:02.752 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.752 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9e59e0d9c445a74-0065303906-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.756 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.757 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.757 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.758 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.758 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.759 DEBUG:boto:Method: GET 2023-10-18T19:59:02.759 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.759 DEBUG:boto:Data: 2023-10-18T19:59:02.760 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.760 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.761 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.761 DEBUG:boto:Params: {} 2023-10-18T19:59:02.762 DEBUG:boto:Token: None 2023-10-18T19:59:02.762 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f17fd6b839f0e2e-0065303906-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.806 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.807 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.808 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.808 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.809 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:02.809 DEBUG:boto:Data: 2023-10-18T19:59:02.810 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.810 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.811 DEBUG:boto:Params: {} 2023-10-18T19:59:02.811 DEBUG:boto:Token: None 2023-10-18T19:59:02.812 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj 2023-10-18T19:59:02.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ= 2023-10-18T19:59:02.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ=', 'Content-Length': '0'} 2023-10-18T19:59:02.816 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057e584ee5b5fd932-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.817 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.817 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.818 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.818 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:02.819 DEBUG:boto:Data: 2023-10-18T19:59:02.819 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.819 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.820 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.820 DEBUG:boto:Params: {} 2023-10-18T19:59:02.820 DEBUG:boto:Token: None 2023-10-18T19:59:02.821 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj 2023-10-18T19:59:02.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ= 2023-10-18T19:59:02.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zcrdjH89YWIwsGts9vezX7DpHKQ=', 'Content-Length': '0'} 2023-10-18T19:59:02.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037c991eac5f02b73-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.826 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.826 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.827 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.827 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.828 DEBUG:boto:Method: GET 2023-10-18T19:59:02.828 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.828 DEBUG:boto:Data: 2023-10-18T19:59:02.829 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.829 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.830 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.830 DEBUG:boto:Params: {} 2023-10-18T19:59:02.831 DEBUG:boto:Token: None 2023-10-18T19:59:02.831 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s= 2023-10-18T19:59:02.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s=', 'Content-Length': '0'} 2023-10-18T19:59:02.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007219616df8333113-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.874 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.875 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.875 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.876 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.876 DEBUG:boto:Method: GET 2023-10-18T19:59:02.877 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.877 DEBUG:boto:Data: 2023-10-18T19:59:02.878 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.878 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.878 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.879 DEBUG:boto:Params: {} 2023-10-18T19:59:02.879 DEBUG:boto:Token: None 2023-10-18T19:59:02.880 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:02.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s= 2023-10-18T19:59:02.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E5aIx0JX5r15BD8qm7U921hRN9s=', 'Content-Length': '0'} 2023-10-18T19:59:02.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a21810cb2ba57ce-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.930 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.931 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.931 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.932 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.932 DEBUG:boto:Method: GET 2023-10-18T19:59:02.933 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.933 DEBUG:boto:Data: 2023-10-18T19:59:02.934 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.934 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.934 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.935 DEBUG:boto:Params: {} 2023-10-18T19:59:02.935 DEBUG:boto:Token: None 2023-10-18T19:59:02.936 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070829e125ca6e47f-0065303906-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.940 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.941 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:02.941 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:02.942 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.942 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:02.943 DEBUG:boto:Method: GET 2023-10-18T19:59:02.943 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:02.944 DEBUG:boto:Data: 2023-10-18T19:59:02.944 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.944 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.945 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.945 DEBUG:boto:Params: {} 2023-10-18T19:59:02.946 DEBUG:boto:Token: None 2023-10-18T19:59:02.946 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/ 2023-10-18T19:59:02.947 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE= 2023-10-18T19:59:02.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OMTZO0Yx+7xPFUL3H+lpNePLZXE=', 'Content-Length': '0'} 2023-10-18T19:59:02.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a7ad35281a6b758-0065303906-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.950 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:02.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.951 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.952 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.952 DEBUG:boto:path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.953 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.953 DEBUG:boto:Method: GET 2023-10-18T19:59:02.954 DEBUG:boto:Path: /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.954 DEBUG:boto:Data: 2023-10-18T19:59:02.954 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.955 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.955 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.956 DEBUG:boto:Params: {} 2023-10-18T19:59:02.956 DEBUG:boto:Token: None 2023-10-18T19:59:02.957 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.957 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0SdjOOu7g+6SIYbkIMef/JJFCVM= 2023-10-18T19:59:02.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0SdjOOu7g+6SIYbkIMef/JJFCVM=', 'Content-Length': '0'} 2023-10-18T19:59:02.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008135ff977b68b85f-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.960 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.961 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.961 DEBUG:boto:path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.961 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.962 DEBUG:boto:Method: GET 2023-10-18T19:59:02.962 DEBUG:boto:Path: /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.963 DEBUG:boto:Data: 2023-10-18T19:59:02.963 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.964 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.964 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.964 DEBUG:boto:Params: {} 2023-10-18T19:59:02.965 DEBUG:boto:Token: None 2023-10-18T19:59:02.965 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.966 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0SdjOOu7g+6SIYbkIMef/JJFCVM= 2023-10-18T19:59:02.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0SdjOOu7g+6SIYbkIMef/JJFCVM=', 'Content-Length': '0'} 2023-10-18T19:59:02.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d09f52f455847ea8-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.969 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.970 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.970 DEBUG:boto:path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.971 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.971 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.971 DEBUG:boto:Path: /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.972 DEBUG:boto:Data: 2023-10-18T19:59:02.972 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.973 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.973 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.973 DEBUG:boto:Params: {} 2023-10-18T19:59:02.974 DEBUG:boto:Token: None 2023-10-18T19:59:02.974 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.975 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J2bhyZU+QX7IjJQdunYY/EFc7co= 2023-10-18T19:59:02.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J2bhyZU+QX7IjJQdunYY/EFc7co=', 'Content-Length': '0'} 2023-10-18T19:59:02.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000294fd179c944674a-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.978 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.978 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.979 DEBUG:boto:path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.979 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.979 DEBUG:boto:Method: HEAD 2023-10-18T19:59:02.980 DEBUG:boto:Path: /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.980 DEBUG:boto:Data: 2023-10-18T19:59:02.980 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.981 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.982 DEBUG:boto:Params: {} 2023-10-18T19:59:02.982 DEBUG:boto:Token: None 2023-10-18T19:59:02.983 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J2bhyZU+QX7IjJQdunYY/EFc7co= 2023-10-18T19:59:02.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J2bhyZU+QX7IjJQdunYY/EFc7co=', 'Content-Length': '0'} 2023-10-18T19:59:02.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b81c00820c5d759d-0065303906-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:02.987 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.987 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.987 DEBUG:boto:path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.988 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.988 DEBUG:boto:Method: GET 2023-10-18T19:59:02.989 DEBUG:boto:Path: /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.989 DEBUG:boto:Data: 2023-10-18T19:59:02.990 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.990 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:02.990 DEBUG:boto:Port: 8001 2023-10-18T19:59:02.991 DEBUG:boto:Params: {} 2023-10-18T19:59:02.991 DEBUG:boto:Token: None 2023-10-18T19:59:02.992 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:02 GMT /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.992 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0SdjOOu7g+6SIYbkIMef/JJFCVM= 2023-10-18T19:59:02.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0SdjOOu7g+6SIYbkIMef/JJFCVM=', 'Content-Length': '0'} 2023-10-18T19:59:02.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e25cb75360a03150-0065303906-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:02.995 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:02.996 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:02.996 DEBUG:boto:path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.997 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.997 DEBUG:boto:Method: GET 2023-10-18T19:59:02.997 DEBUG:boto:Path: /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:02.998 DEBUG:boto:Data: 2023-10-18T19:59:02.998 DEBUG:boto:Headers: {} 2023-10-18T19:59:02.999 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:02.999 DEBUG:boto:Port: 8000 2023-10-18T19:59:02.999 DEBUG:boto:Params: {} 2023-10-18T19:59:03.001 DEBUG:boto:Token: None 2023-10-18T19:59:03.001 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n 2023-10-18T19:59:03.002 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cVX1oTD9godKsMwaFduAaM0zjBo= 2023-10-18T19:59:03.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cVX1oTD9godKsMwaFduAaM0zjBo=', 'Content-Length': '0'} 2023-10-18T19:59:03.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'bOVsRIIYKWn1jCbW9KOeNyU.Y-2yd8n'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2493bf522dd0b98-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.046 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.047 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.048 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.048 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.048 DEBUG:boto:Method: GET 2023-10-18T19:59:03.049 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.049 DEBUG:boto:Data: 2023-10-18T19:59:03.050 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.051 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.051 DEBUG:boto:Params: {} 2023-10-18T19:59:03.051 DEBUG:boto:Token: None 2023-10-18T19:59:03.052 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.053 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002beaadb948bc92d1-0065303907-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.056 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.057 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.058 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.058 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.058 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.059 DEBUG:boto:Method: GET 2023-10-18T19:59:03.059 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.060 DEBUG:boto:Data: 2023-10-18T19:59:03.060 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.061 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.061 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.061 DEBUG:boto:Params: {} 2023-10-18T19:59:03.062 DEBUG:boto:Token: None 2023-10-18T19:59:03.062 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.063 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000310263ba63b9f8fa-0065303907-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.106 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.107 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.108 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.108 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.109 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:03.109 DEBUG:boto:Data: 2023-10-18T19:59:03.109 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.110 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.110 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.111 DEBUG:boto:Params: {} 2023-10-18T19:59:03.111 DEBUG:boto:Token: None 2023-10-18T19:59:03.112 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj 2023-10-18T19:59:03.112 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY= 2023-10-18T19:59:03.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY=', 'Content-Length': '0'} 2023-10-18T19:59:03.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e58b7ea4bc2486a3-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.116 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.117 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.117 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.118 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:03.118 DEBUG:boto:Data: 2023-10-18T19:59:03.118 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.119 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.119 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.120 DEBUG:boto:Params: {} 2023-10-18T19:59:03.120 DEBUG:boto:Token: None 2023-10-18T19:59:03.120 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj 2023-10-18T19:59:03.121 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY= 2023-10-18T19:59:03.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY=', 'Content-Length': '0'} 2023-10-18T19:59:03.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae0fef0077eb1760-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:03.126 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.126 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.126 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.127 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.127 DEBUG:boto:Method: GET 2023-10-18T19:59:03.128 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.128 DEBUG:boto:Data: 2023-10-18T19:59:03.129 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.129 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.129 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.130 DEBUG:boto:Params: {} 2023-10-18T19:59:03.130 DEBUG:boto:Token: None 2023-10-18T19:59:03.131 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.131 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA= 2023-10-18T19:59:03.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA=', 'Content-Length': '0'} 2023-10-18T19:59:03.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7aaed3b0ed1943b-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.174 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.175 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.175 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.176 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.176 DEBUG:boto:Method: GET 2023-10-18T19:59:03.177 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.177 DEBUG:boto:Data: 2023-10-18T19:59:03.178 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.178 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.179 DEBUG:boto:Params: {} 2023-10-18T19:59:03.179 DEBUG:boto:Token: None 2023-10-18T19:59:03.180 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA= 2023-10-18T19:59:03.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA=', 'Content-Length': '0'} 2023-10-18T19:59:03.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fb8c711b94929fc-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.226 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.227 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.228 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.228 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.229 DEBUG:boto:Method: GET 2023-10-18T19:59:03.229 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.230 DEBUG:boto:Data: 2023-10-18T19:59:03.230 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.231 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.231 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.232 DEBUG:boto:Params: {} 2023-10-18T19:59:03.232 DEBUG:boto:Token: None 2023-10-18T19:59:03.233 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.233 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043326557366e1bab-0065303907-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.237 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.238 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.238 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.238 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.239 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.239 DEBUG:boto:Method: GET 2023-10-18T19:59:03.240 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.240 DEBUG:boto:Data: 2023-10-18T19:59:03.241 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.241 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.241 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.242 DEBUG:boto:Params: {} 2023-10-18T19:59:03.242 DEBUG:boto:Token: None 2023-10-18T19:59:03.243 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.243 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ced01676c3a88660-0065303907-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.247 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:03.248 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.248 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.249 DEBUG:boto:path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.249 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.250 DEBUG:boto:Method: GET 2023-10-18T19:59:03.250 DEBUG:boto:Path: /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.251 DEBUG:boto:Data: 2023-10-18T19:59:03.251 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.252 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.252 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.252 DEBUG:boto:Params: {} 2023-10-18T19:59:03.253 DEBUG:boto:Token: None 2023-10-18T19:59:03.253 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.254 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nfw65PKZazM/Rvwz2oSaNMB8QQE= 2023-10-18T19:59:03.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nfw65PKZazM/Rvwz2oSaNMB8QQE=', 'Content-Length': '0'} 2023-10-18T19:59:03.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'egd1qNK-ZKDNN5bN4A1VaSs97JSE297'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d12aec296d5c3d8-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.257 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.258 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.258 DEBUG:boto:path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.259 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.259 DEBUG:boto:Method: GET 2023-10-18T19:59:03.260 DEBUG:boto:Path: /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.260 DEBUG:boto:Data: 2023-10-18T19:59:03.260 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.261 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.261 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.262 DEBUG:boto:Params: {} 2023-10-18T19:59:03.262 DEBUG:boto:Token: None 2023-10-18T19:59:03.263 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nfw65PKZazM/Rvwz2oSaNMB8QQE= 2023-10-18T19:59:03.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nfw65PKZazM/Rvwz2oSaNMB8QQE=', 'Content-Length': '0'} 2023-10-18T19:59:03.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'egd1qNK-ZKDNN5bN4A1VaSs97JSE297'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d324e3f76954f51c-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.306 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.307 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.307 DEBUG:boto:path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.308 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.308 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.309 DEBUG:boto:Path: /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.309 DEBUG:boto:Data: 2023-10-18T19:59:03.309 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.310 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.311 DEBUG:boto:Params: {} 2023-10-18T19:59:03.311 DEBUG:boto:Token: None 2023-10-18T19:59:03.312 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Cf7d6FtlQddvUx3A6d+s/12nYnA= 2023-10-18T19:59:03.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Cf7d6FtlQddvUx3A6d+s/12nYnA=', 'Content-Length': '0'} 2023-10-18T19:59:03.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'egd1qNK-ZKDNN5bN4A1VaSs97JSE297'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d38929b05616a14-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.315 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.315 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.316 DEBUG:boto:path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.316 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.317 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.317 DEBUG:boto:Path: /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.317 DEBUG:boto:Data: 2023-10-18T19:59:03.318 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.319 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.319 DEBUG:boto:Params: {} 2023-10-18T19:59:03.320 DEBUG:boto:Token: None 2023-10-18T19:59:03.320 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Cf7d6FtlQddvUx3A6d+s/12nYnA= 2023-10-18T19:59:03.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Cf7d6FtlQddvUx3A6d+s/12nYnA=', 'Content-Length': '0'} 2023-10-18T19:59:03.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'egd1qNK-ZKDNN5bN4A1VaSs97JSE297'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b7dacfe41da9aea-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:03.324 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.324 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.325 DEBUG:boto:path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.325 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.326 DEBUG:boto:Method: GET 2023-10-18T19:59:03.326 DEBUG:boto:Path: /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.327 DEBUG:boto:Data: 2023-10-18T19:59:03.327 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.327 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.328 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.328 DEBUG:boto:Params: {} 2023-10-18T19:59:03.329 DEBUG:boto:Token: None 2023-10-18T19:59:03.329 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nfw65PKZazM/Rvwz2oSaNMB8QQE= 2023-10-18T19:59:03.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nfw65PKZazM/Rvwz2oSaNMB8QQE=', 'Content-Length': '0'} 2023-10-18T19:59:03.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'egd1qNK-ZKDNN5bN4A1VaSs97JSE297'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e281eaa1f4c387f-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.374 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.375 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.376 DEBUG:boto:path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.376 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.376 DEBUG:boto:Method: GET 2023-10-18T19:59:03.377 DEBUG:boto:Path: /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.377 DEBUG:boto:Data: 2023-10-18T19:59:03.378 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.378 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.378 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.379 DEBUG:boto:Params: {} 2023-10-18T19:59:03.379 DEBUG:boto:Token: None 2023-10-18T19:59:03.380 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=egd1qNK-ZKDNN5bN4A1VaSs97JSE297 2023-10-18T19:59:03.380 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nfw65PKZazM/Rvwz2oSaNMB8QQE= 2023-10-18T19:59:03.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nfw65PKZazM/Rvwz2oSaNMB8QQE=', 'Content-Length': '0'} 2023-10-18T19:59:03.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'egd1qNK-ZKDNN5bN4A1VaSs97JSE297'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fa53a9bf2c29044-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.384 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.384 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.385 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.385 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.385 DEBUG:boto:Method: GET 2023-10-18T19:59:03.386 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.386 DEBUG:boto:Data: 2023-10-18T19:59:03.386 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.387 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.387 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.388 DEBUG:boto:Params: {} 2023-10-18T19:59:03.388 DEBUG:boto:Token: None 2023-10-18T19:59:03.388 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.389 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000870b8d36f29c4cf6-0065303907-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.392 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.393 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.394 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.394 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.394 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.395 DEBUG:boto:Method: GET 2023-10-18T19:59:03.395 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.396 DEBUG:boto:Data: 2023-10-18T19:59:03.396 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.397 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.397 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.397 DEBUG:boto:Params: {} 2023-10-18T19:59:03.398 DEBUG:boto:Token: None 2023-10-18T19:59:03.398 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.399 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0cb6c29b3294307-0065303907-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.402 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.403 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.403 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.404 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.404 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:03.404 DEBUG:boto:Data: 2023-10-18T19:59:03.405 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.405 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.406 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.406 DEBUG:boto:Params: {} 2023-10-18T19:59:03.406 DEBUG:boto:Token: None 2023-10-18T19:59:03.407 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj 2023-10-18T19:59:03.407 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY= 2023-10-18T19:59:03.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY=', 'Content-Length': '0'} 2023-10-18T19:59:03.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c2f237244c43528-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.414 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.414 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.414 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.415 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:03.415 DEBUG:boto:Data: 2023-10-18T19:59:03.416 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.416 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.416 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.417 DEBUG:boto:Params: {} 2023-10-18T19:59:03.417 DEBUG:boto:Token: None 2023-10-18T19:59:03.418 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj 2023-10-18T19:59:03.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY= 2023-10-18T19:59:03.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY=', 'Content-Length': '0'} 2023-10-18T19:59:03.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9fb286aa84ae9d4-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:03.423 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.423 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.424 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.424 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.425 DEBUG:boto:Method: GET 2023-10-18T19:59:03.425 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.425 DEBUG:boto:Data: 2023-10-18T19:59:03.426 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.427 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.427 DEBUG:boto:Params: {} 2023-10-18T19:59:03.428 DEBUG:boto:Token: None 2023-10-18T19:59:03.428 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.429 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA= 2023-10-18T19:59:03.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA=', 'Content-Length': '0'} 2023-10-18T19:59:03.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000632158cf381746c2-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.478 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.479 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.479 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.480 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.480 DEBUG:boto:Method: GET 2023-10-18T19:59:03.481 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.481 DEBUG:boto:Data: 2023-10-18T19:59:03.482 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.482 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.483 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.483 DEBUG:boto:Params: {} 2023-10-18T19:59:03.483 DEBUG:boto:Token: None 2023-10-18T19:59:03.484 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.484 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA= 2023-10-18T19:59:03.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA=', 'Content-Length': '0'} 2023-10-18T19:59:03.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7c7abdc0cacaf6a-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.530 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.531 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.532 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.532 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.532 DEBUG:boto:Method: GET 2023-10-18T19:59:03.533 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.533 DEBUG:boto:Data: 2023-10-18T19:59:03.534 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.535 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.535 DEBUG:boto:Params: {} 2023-10-18T19:59:03.536 DEBUG:boto:Token: None 2023-10-18T19:59:03.536 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.537 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089fae7c347222070-0065303907-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.540 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.541 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.541 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.542 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.542 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.543 DEBUG:boto:Method: GET 2023-10-18T19:59:03.543 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.544 DEBUG:boto:Data: 2023-10-18T19:59:03.544 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.545 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.545 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.545 DEBUG:boto:Params: {} 2023-10-18T19:59:03.546 DEBUG:boto:Token: None 2023-10-18T19:59:03.546 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.547 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d264f5ec7b25713-0065303907-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.590 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:03.592 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.592 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.593 DEBUG:boto:path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.593 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.594 DEBUG:boto:Method: GET 2023-10-18T19:59:03.594 DEBUG:boto:Path: /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.595 DEBUG:boto:Data: 2023-10-18T19:59:03.595 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.595 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.596 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.596 DEBUG:boto:Params: {} 2023-10-18T19:59:03.597 DEBUG:boto:Token: None 2023-10-18T19:59:03.597 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.598 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gv7Kp3A6yoEROabfhMz7q4MjsEI= 2023-10-18T19:59:03.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gv7Kp3A6yoEROabfhMz7q4MjsEI=', 'Content-Length': '0'} 2023-10-18T19:59:03.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', '6s7LnG-qRPtK-UCaUqeTxtDibOflrUd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f16f554c42d89766-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.602 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.602 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.603 DEBUG:boto:path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.603 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.603 DEBUG:boto:Method: GET 2023-10-18T19:59:03.604 DEBUG:boto:Path: /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.604 DEBUG:boto:Data: 2023-10-18T19:59:03.605 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.605 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.606 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.606 DEBUG:boto:Params: {} 2023-10-18T19:59:03.606 DEBUG:boto:Token: None 2023-10-18T19:59:03.607 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.607 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gv7Kp3A6yoEROabfhMz7q4MjsEI= 2023-10-18T19:59:03.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gv7Kp3A6yoEROabfhMz7q4MjsEI=', 'Content-Length': '0'} 2023-10-18T19:59:03.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', '6s7LnG-qRPtK-UCaUqeTxtDibOflrUd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8d7f5853113b801-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.611 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.612 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.612 DEBUG:boto:path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.612 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.613 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.613 DEBUG:boto:Path: /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.614 DEBUG:boto:Data: 2023-10-18T19:59:03.614 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.614 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.617 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.617 DEBUG:boto:Params: {} 2023-10-18T19:59:03.618 DEBUG:boto:Token: None 2023-10-18T19:59:03.618 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.619 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MpTh11zMTFkAqaroZ4O9pfHDoV8= 2023-10-18T19:59:03.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MpTh11zMTFkAqaroZ4O9pfHDoV8=', 'Content-Length': '0'} 2023-10-18T19:59:03.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', '6s7LnG-qRPtK-UCaUqeTxtDibOflrUd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec3c099002433813-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.622 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.623 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.623 DEBUG:boto:path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.623 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.624 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.624 DEBUG:boto:Path: /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.625 DEBUG:boto:Data: 2023-10-18T19:59:03.625 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.626 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.626 DEBUG:boto:Params: {} 2023-10-18T19:59:03.627 DEBUG:boto:Token: None 2023-10-18T19:59:03.627 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.628 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MpTh11zMTFkAqaroZ4O9pfHDoV8= 2023-10-18T19:59:03.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MpTh11zMTFkAqaroZ4O9pfHDoV8=', 'Content-Length': '0'} 2023-10-18T19:59:03.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', '6s7LnG-qRPtK-UCaUqeTxtDibOflrUd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b1e5f04c1691f0a-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:03.631 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.631 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.632 DEBUG:boto:path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.632 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.633 DEBUG:boto:Method: GET 2023-10-18T19:59:03.633 DEBUG:boto:Path: /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.633 DEBUG:boto:Data: 2023-10-18T19:59:03.634 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.634 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.635 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.635 DEBUG:boto:Params: {} 2023-10-18T19:59:03.636 DEBUG:boto:Token: None 2023-10-18T19:59:03.636 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.637 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gv7Kp3A6yoEROabfhMz7q4MjsEI= 2023-10-18T19:59:03.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gv7Kp3A6yoEROabfhMz7q4MjsEI=', 'Content-Length': '0'} 2023-10-18T19:59:03.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', '6s7LnG-qRPtK-UCaUqeTxtDibOflrUd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054c5848993ec09cb-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.686 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.687 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.687 DEBUG:boto:path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.688 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.688 DEBUG:boto:Method: GET 2023-10-18T19:59:03.689 DEBUG:boto:Path: /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.689 DEBUG:boto:Data: 2023-10-18T19:59:03.690 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.691 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.691 DEBUG:boto:Params: {} 2023-10-18T19:59:03.691 DEBUG:boto:Token: None 2023-10-18T19:59:03.692 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=6s7LnG-qRPtK-UCaUqeTxtDibOflrUd 2023-10-18T19:59:03.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gv7Kp3A6yoEROabfhMz7q4MjsEI= 2023-10-18T19:59:03.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gv7Kp3A6yoEROabfhMz7q4MjsEI=', 'Content-Length': '0'} 2023-10-18T19:59:03.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', '6s7LnG-qRPtK-UCaUqeTxtDibOflrUd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081d85c8fbfea80b5-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.696 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.696 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.697 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.697 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.698 DEBUG:boto:Method: GET 2023-10-18T19:59:03.698 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.698 DEBUG:boto:Data: 2023-10-18T19:59:03.699 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.699 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.700 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.700 DEBUG:boto:Params: {} 2023-10-18T19:59:03.701 DEBUG:boto:Token: None 2023-10-18T19:59:03.701 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.702 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5688f38d18606af-0065303907-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.705 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.706 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.707 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.707 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.708 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.708 DEBUG:boto:Method: GET 2023-10-18T19:59:03.708 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.709 DEBUG:boto:Data: 2023-10-18T19:59:03.709 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.710 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.710 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.710 DEBUG:boto:Params: {} 2023-10-18T19:59:03.711 DEBUG:boto:Token: None 2023-10-18T19:59:03.711 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d230da4bc1d7660f-0065303907-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.758 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.759 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.760 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.760 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.760 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:03.761 DEBUG:boto:Data: 2023-10-18T19:59:03.761 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.762 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.762 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.762 DEBUG:boto:Params: {} 2023-10-18T19:59:03.763 DEBUG:boto:Token: None 2023-10-18T19:59:03.763 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj 2023-10-18T19:59:03.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY= 2023-10-18T19:59:03.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY=', 'Content-Length': '0'} 2023-10-18T19:59:03.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4ae104cccabea5f-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.768 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.769 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.769 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.769 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:03.770 DEBUG:boto:Data: 2023-10-18T19:59:03.770 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.771 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.771 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.771 DEBUG:boto:Params: {} 2023-10-18T19:59:03.772 DEBUG:boto:Token: None 2023-10-18T19:59:03.772 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj 2023-10-18T19:59:03.773 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY= 2023-10-18T19:59:03.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mN4Rui7hrvFlSmjvoc3XV1kl/rY=', 'Content-Length': '0'} 2023-10-18T19:59:03.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0f790ac969b7e32-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:03.777 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.777 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.778 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.778 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.779 DEBUG:boto:Method: GET 2023-10-18T19:59:03.779 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.780 DEBUG:boto:Data: 2023-10-18T19:59:03.780 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.780 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.781 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.781 DEBUG:boto:Params: {} 2023-10-18T19:59:03.782 DEBUG:boto:Token: None 2023-10-18T19:59:03.782 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA= 2023-10-18T19:59:03.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA=', 'Content-Length': '0'} 2023-10-18T19:59:03.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083862110a7b352b8-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.830 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.831 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.831 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.832 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.832 DEBUG:boto:Method: GET 2023-10-18T19:59:03.833 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.833 DEBUG:boto:Data: 2023-10-18T19:59:03.834 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.834 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.834 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.835 DEBUG:boto:Params: {} 2023-10-18T19:59:03.835 DEBUG:boto:Token: None 2023-10-18T19:59:03.836 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:03.836 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA= 2023-10-18T19:59:03.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w/DALZyxANXIgIe6PJVOzEVB0QA=', 'Content-Length': '0'} 2023-10-18T19:59:03.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e990ff42b6a21ce-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.882 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.883 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.883 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.884 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.884 DEBUG:boto:Method: GET 2023-10-18T19:59:03.885 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.885 DEBUG:boto:Data: 2023-10-18T19:59:03.886 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.886 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.887 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.887 DEBUG:boto:Params: {} 2023-10-18T19:59:03.887 DEBUG:boto:Token: None 2023-10-18T19:59:03.888 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.889 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000511d7550058d32ec-0065303907-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.893 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.893 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:03.894 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:03.894 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.895 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:03.895 DEBUG:boto:Method: GET 2023-10-18T19:59:03.896 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:03.896 DEBUG:boto:Data: 2023-10-18T19:59:03.896 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.897 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.897 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.898 DEBUG:boto:Params: {} 2023-10-18T19:59:03.898 DEBUG:boto:Token: None 2023-10-18T19:59:03.899 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/ 2023-10-18T19:59:03.899 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc= 2023-10-18T19:59:03.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmk3g9be7cWzI/Su2wXhn0WhgOc=', 'Content-Length': '0'} 2023-10-18T19:59:03.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000100707283ab1dba6-0065303907-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.903 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:03.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:03.905 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.905 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.905 DEBUG:boto:path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.906 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.906 DEBUG:boto:Method: GET 2023-10-18T19:59:03.907 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.907 DEBUG:boto:Data: 2023-10-18T19:59:03.908 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.908 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.908 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.909 DEBUG:boto:Params: {} 2023-10-18T19:59:03.909 DEBUG:boto:Token: None 2023-10-18T19:59:03.910 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.910 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/psUjh90Cun+1ohZASkFpJHZkF4= 2023-10-18T19:59:03.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/psUjh90Cun+1ohZASkFpJHZkF4=', 'Content-Length': '0'} 2023-10-18T19:59:03.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NTvDWcfE9kVoyznfhCGU6ELa04H.jwL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cb9533b03298ec1-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.913 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.914 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.914 DEBUG:boto:path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.915 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.915 DEBUG:boto:Method: GET 2023-10-18T19:59:03.916 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.916 DEBUG:boto:Data: 2023-10-18T19:59:03.916 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.917 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.918 DEBUG:boto:Params: {} 2023-10-18T19:59:03.918 DEBUG:boto:Token: None 2023-10-18T19:59:03.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.919 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/psUjh90Cun+1ohZASkFpJHZkF4= 2023-10-18T19:59:03.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/psUjh90Cun+1ohZASkFpJHZkF4=', 'Content-Length': '0'} 2023-10-18T19:59:03.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NTvDWcfE9kVoyznfhCGU6ELa04H.jwL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e33404724b535a23-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.962 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.963 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.963 DEBUG:boto:path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.964 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.964 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.965 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.965 DEBUG:boto:Data: 2023-10-18T19:59:03.965 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.966 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.967 DEBUG:boto:Params: {} 2023-10-18T19:59:03.967 DEBUG:boto:Token: None 2023-10-18T19:59:03.968 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sEF7pfglu8eol2kCwjpPhRu5vpY= 2023-10-18T19:59:03.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sEF7pfglu8eol2kCwjpPhRu5vpY=', 'Content-Length': '0'} 2023-10-18T19:59:03.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NTvDWcfE9kVoyznfhCGU6ELa04H.jwL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ce3f1963e85a15a-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.971 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.972 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.972 DEBUG:boto:path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.973 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.973 DEBUG:boto:Method: HEAD 2023-10-18T19:59:03.974 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.974 DEBUG:boto:Data: 2023-10-18T19:59:03.974 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.975 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.975 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.976 DEBUG:boto:Params: {} 2023-10-18T19:59:03.976 DEBUG:boto:Token: None 2023-10-18T19:59:03.977 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.977 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sEF7pfglu8eol2kCwjpPhRu5vpY= 2023-10-18T19:59:03.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sEF7pfglu8eol2kCwjpPhRu5vpY=', 'Content-Length': '0'} 2023-10-18T19:59:03.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NTvDWcfE9kVoyznfhCGU6ELa04H.jwL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e331f97bbf200b60-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:03.980 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.981 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.981 DEBUG:boto:path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.981 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.982 DEBUG:boto:Method: GET 2023-10-18T19:59:03.982 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.983 DEBUG:boto:Data: 2023-10-18T19:59:03.983 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.984 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:03.984 DEBUG:boto:Port: 8001 2023-10-18T19:59:03.985 DEBUG:boto:Params: {} 2023-10-18T19:59:03.985 DEBUG:boto:Token: None 2023-10-18T19:59:03.985 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.986 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/psUjh90Cun+1ohZASkFpJHZkF4= 2023-10-18T19:59:03.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/psUjh90Cun+1ohZASkFpJHZkF4=', 'Content-Length': '0'} 2023-10-18T19:59:03.988 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NTvDWcfE9kVoyznfhCGU6ELa04H.jwL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004f2f4c3c17b11bc-0065303907-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:03.989 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:03.989 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:03.990 DEBUG:boto:path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.990 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.991 DEBUG:boto:Method: GET 2023-10-18T19:59:03.991 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.992 DEBUG:boto:Data: 2023-10-18T19:59:03.992 DEBUG:boto:Headers: {} 2023-10-18T19:59:03.992 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:03.993 DEBUG:boto:Port: 8000 2023-10-18T19:59:03.993 DEBUG:boto:Params: {} 2023-10-18T19:59:03.994 DEBUG:boto:Token: None 2023-10-18T19:59:03.994 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:03 GMT /bofrnf-13/obj?versionId=NTvDWcfE9kVoyznfhCGU6ELa04H.jwL 2023-10-18T19:59:03.995 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/psUjh90Cun+1ohZASkFpJHZkF4= 2023-10-18T19:59:03.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/psUjh90Cun+1ohZASkFpJHZkF4=', 'Content-Length': '0'} 2023-10-18T19:59:03.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NTvDWcfE9kVoyznfhCGU6ELa04H.jwL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033ef74238ba67554-0065303907-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.042 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.043 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.043 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.044 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.044 DEBUG:boto:Method: GET 2023-10-18T19:59:04.044 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.045 DEBUG:boto:Data: 2023-10-18T19:59:04.045 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.045 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.046 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.046 DEBUG:boto:Params: {} 2023-10-18T19:59:04.047 DEBUG:boto:Token: None 2023-10-18T19:59:04.047 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.048 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2d1a106960237b3-0065303908-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.052 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.052 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.053 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.053 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.054 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.054 DEBUG:boto:Method: GET 2023-10-18T19:59:04.054 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.055 DEBUG:boto:Data: 2023-10-18T19:59:04.055 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.056 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.057 DEBUG:boto:Params: {} 2023-10-18T19:59:04.057 DEBUG:boto:Token: None 2023-10-18T19:59:04.058 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.058 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f8a8717a12cc75a-0065303908-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.062 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.062 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.063 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.063 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.064 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:04.064 DEBUG:boto:Data: 2023-10-18T19:59:04.065 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.065 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.065 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.066 DEBUG:boto:Params: {} 2023-10-18T19:59:04.066 DEBUG:boto:Token: None 2023-10-18T19:59:04.067 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj 2023-10-18T19:59:04.067 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU= 2023-10-18T19:59:04.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU=', 'Content-Length': '0'} 2023-10-18T19:59:04.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c2bacca3e0d59a5-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.072 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.072 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.072 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.073 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:04.073 DEBUG:boto:Data: 2023-10-18T19:59:04.074 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.074 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.074 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.075 DEBUG:boto:Params: {} 2023-10-18T19:59:04.075 DEBUG:boto:Token: None 2023-10-18T19:59:04.076 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj 2023-10-18T19:59:04.076 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU= 2023-10-18T19:59:04.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU=', 'Content-Length': '0'} 2023-10-18T19:59:04.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000225984263c804b83-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.080 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.081 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.081 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.082 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.082 DEBUG:boto:Method: GET 2023-10-18T19:59:04.083 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.083 DEBUG:boto:Data: 2023-10-18T19:59:04.083 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.084 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.084 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.085 DEBUG:boto:Params: {} 2023-10-18T19:59:04.085 DEBUG:boto:Token: None 2023-10-18T19:59:04.086 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.086 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw= 2023-10-18T19:59:04.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw=', 'Content-Length': '0'} 2023-10-18T19:59:04.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000189d1915013f59f7-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.134 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.135 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.136 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.136 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.136 DEBUG:boto:Method: GET 2023-10-18T19:59:04.137 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.137 DEBUG:boto:Data: 2023-10-18T19:59:04.138 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.138 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.138 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.139 DEBUG:boto:Params: {} 2023-10-18T19:59:04.139 DEBUG:boto:Token: None 2023-10-18T19:59:04.140 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.140 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw= 2023-10-18T19:59:04.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw=', 'Content-Length': '0'} 2023-10-18T19:59:04.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ca2321406847ddd-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.186 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.187 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.187 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.188 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.188 DEBUG:boto:Method: GET 2023-10-18T19:59:04.189 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.189 DEBUG:boto:Data: 2023-10-18T19:59:04.190 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.190 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.191 DEBUG:boto:Params: {} 2023-10-18T19:59:04.191 DEBUG:boto:Token: None 2023-10-18T19:59:04.192 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002be01124d78b93ff-0065303908-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.197 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.197 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.198 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.198 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.199 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.199 DEBUG:boto:Method: GET 2023-10-18T19:59:04.199 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.200 DEBUG:boto:Data: 2023-10-18T19:59:04.200 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.201 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.201 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.202 DEBUG:boto:Params: {} 2023-10-18T19:59:04.202 DEBUG:boto:Token: None 2023-10-18T19:59:04.203 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.203 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008126c04fa99e957f-0065303908-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.254 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.256 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.256 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.257 DEBUG:boto:path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.257 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.257 DEBUG:boto:Method: GET 2023-10-18T19:59:04.258 DEBUG:boto:Path: /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.258 DEBUG:boto:Data: 2023-10-18T19:59:04.259 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.259 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.259 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.260 DEBUG:boto:Params: {} 2023-10-18T19:59:04.260 DEBUG:boto:Token: None 2023-10-18T19:59:04.261 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.262 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/UgVvPWzxyY1jguPjEj8DiSJqfg= 2023-10-18T19:59:04.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/UgVvPWzxyY1jguPjEj8DiSJqfg=', 'Content-Length': '0'} 2023-10-18T19:59:04.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016e862848996a0f2-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.265 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.265 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.266 DEBUG:boto:path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.266 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.267 DEBUG:boto:Method: GET 2023-10-18T19:59:04.267 DEBUG:boto:Path: /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.267 DEBUG:boto:Data: 2023-10-18T19:59:04.268 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.268 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.269 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.269 DEBUG:boto:Params: {} 2023-10-18T19:59:04.270 DEBUG:boto:Token: None 2023-10-18T19:59:04.270 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.271 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/UgVvPWzxyY1jguPjEj8DiSJqfg= 2023-10-18T19:59:04.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/UgVvPWzxyY1jguPjEj8DiSJqfg=', 'Content-Length': '0'} 2023-10-18T19:59:04.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dad14072549075a-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.274 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.274 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.275 DEBUG:boto:path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.275 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.276 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.276 DEBUG:boto:Path: /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.276 DEBUG:boto:Data: 2023-10-18T19:59:04.277 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.277 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.278 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.278 DEBUG:boto:Params: {} 2023-10-18T19:59:04.278 DEBUG:boto:Token: None 2023-10-18T19:59:04.279 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.279 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LNY6tNXfxTLsUDouLklrGNY7+fg= 2023-10-18T19:59:04.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LNY6tNXfxTLsUDouLklrGNY7+fg=', 'Content-Length': '0'} 2023-10-18T19:59:04.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004aa3c1ce5f840459-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.283 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.283 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.284 DEBUG:boto:path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.284 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.284 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.285 DEBUG:boto:Path: /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.285 DEBUG:boto:Data: 2023-10-18T19:59:04.286 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.286 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.286 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.287 DEBUG:boto:Params: {} 2023-10-18T19:59:04.287 DEBUG:boto:Token: None 2023-10-18T19:59:04.288 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.288 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LNY6tNXfxTLsUDouLklrGNY7+fg= 2023-10-18T19:59:04.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LNY6tNXfxTLsUDouLklrGNY7+fg=', 'Content-Length': '0'} 2023-10-18T19:59:04.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0d1a08981c21a84-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.292 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.292 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.293 DEBUG:boto:path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.293 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.293 DEBUG:boto:Method: GET 2023-10-18T19:59:04.294 DEBUG:boto:Path: /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.294 DEBUG:boto:Data: 2023-10-18T19:59:04.295 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.295 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.295 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.296 DEBUG:boto:Params: {} 2023-10-18T19:59:04.296 DEBUG:boto:Token: None 2023-10-18T19:59:04.297 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.297 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/UgVvPWzxyY1jguPjEj8DiSJqfg= 2023-10-18T19:59:04.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/UgVvPWzxyY1jguPjEj8DiSJqfg=', 'Content-Length': '0'} 2023-10-18T19:59:04.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006183900402bb596c-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.342 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.343 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.343 DEBUG:boto:path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.344 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.344 DEBUG:boto:Method: GET 2023-10-18T19:59:04.345 DEBUG:boto:Path: /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.345 DEBUG:boto:Data: 2023-10-18T19:59:04.346 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.346 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.347 DEBUG:boto:Params: {} 2023-10-18T19:59:04.347 DEBUG:boto:Token: None 2023-10-18T19:59:04.348 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg 2023-10-18T19:59:04.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/UgVvPWzxyY1jguPjEj8DiSJqfg= 2023-10-18T19:59:04.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/UgVvPWzxyY1jguPjEj8DiSJqfg=', 'Content-Length': '0'} 2023-10-18T19:59:04.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'G59arbhnbpCc-hwP0YJ8EXQMYoW5oYg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a97eaee55220783d-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.352 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.352 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.352 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.353 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.353 DEBUG:boto:Method: GET 2023-10-18T19:59:04.354 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.354 DEBUG:boto:Data: 2023-10-18T19:59:04.354 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.355 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.355 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.356 DEBUG:boto:Params: {} 2023-10-18T19:59:04.356 DEBUG:boto:Token: None 2023-10-18T19:59:04.357 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.357 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ec7069bbdec2b80-0065303908-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.402 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.403 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.404 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.404 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.405 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.405 DEBUG:boto:Method: GET 2023-10-18T19:59:04.406 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.406 DEBUG:boto:Data: 2023-10-18T19:59:04.406 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.407 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.407 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.408 DEBUG:boto:Params: {} 2023-10-18T19:59:04.408 DEBUG:boto:Token: None 2023-10-18T19:59:04.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.409 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fe7003b6de6f342-0065303908-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.413 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.414 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.414 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.415 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.415 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:04.415 DEBUG:boto:Data: 2023-10-18T19:59:04.416 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.416 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.417 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.417 DEBUG:boto:Params: {} 2023-10-18T19:59:04.418 DEBUG:boto:Token: None 2023-10-18T19:59:04.418 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj 2023-10-18T19:59:04.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU= 2023-10-18T19:59:04.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU=', 'Content-Length': '0'} 2023-10-18T19:59:04.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065a1d2aaaf95d546-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.423 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.423 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.424 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.424 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:04.424 DEBUG:boto:Data: 2023-10-18T19:59:04.425 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.426 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.426 DEBUG:boto:Params: {} 2023-10-18T19:59:04.426 DEBUG:boto:Token: None 2023-10-18T19:59:04.427 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj 2023-10-18T19:59:04.429 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU= 2023-10-18T19:59:04.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU=', 'Content-Length': '0'} 2023-10-18T19:59:04.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003772e3a851b5c461-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.434 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.434 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.435 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.435 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.436 DEBUG:boto:Method: GET 2023-10-18T19:59:04.436 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.437 DEBUG:boto:Data: 2023-10-18T19:59:04.437 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.438 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.438 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.438 DEBUG:boto:Params: {} 2023-10-18T19:59:04.439 DEBUG:boto:Token: None 2023-10-18T19:59:04.439 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw= 2023-10-18T19:59:04.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw=', 'Content-Length': '0'} 2023-10-18T19:59:04.443 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9555c2d40dab8f2-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.443 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.444 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.444 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.445 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.445 DEBUG:boto:Method: GET 2023-10-18T19:59:04.446 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.446 DEBUG:boto:Data: 2023-10-18T19:59:04.447 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.447 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.448 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.448 DEBUG:boto:Params: {} 2023-10-18T19:59:04.449 DEBUG:boto:Token: None 2023-10-18T19:59:04.449 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.450 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw= 2023-10-18T19:59:04.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw=', 'Content-Length': '0'} 2023-10-18T19:59:04.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000170e46f7f871c4fc-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.494 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.495 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.496 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.496 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.496 DEBUG:boto:Method: GET 2023-10-18T19:59:04.497 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.497 DEBUG:boto:Data: 2023-10-18T19:59:04.498 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.499 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.499 DEBUG:boto:Params: {} 2023-10-18T19:59:04.500 DEBUG:boto:Token: None 2023-10-18T19:59:04.500 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.501 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b73279edc902ec4-0065303908-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.504 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.505 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.506 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.506 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.507 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.507 DEBUG:boto:Method: GET 2023-10-18T19:59:04.507 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.508 DEBUG:boto:Data: 2023-10-18T19:59:04.508 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.509 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.509 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.510 DEBUG:boto:Params: {} 2023-10-18T19:59:04.510 DEBUG:boto:Token: None 2023-10-18T19:59:04.511 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.511 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090362942e6c795d5-0065303908-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.515 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.516 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.516 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.517 DEBUG:boto:path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.517 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.518 DEBUG:boto:Method: GET 2023-10-18T19:59:04.518 DEBUG:boto:Path: /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.518 DEBUG:boto:Data: 2023-10-18T19:59:04.519 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.519 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.520 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.520 DEBUG:boto:Params: {} 2023-10-18T19:59:04.521 DEBUG:boto:Token: None 2023-10-18T19:59:04.521 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.522 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YmASl/8SKvwJ31Uq31H3hHm2vUY= 2023-10-18T19:59:04.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YmASl/8SKvwJ31Uq31H3hHm2vUY=', 'Content-Length': '0'} 2023-10-18T19:59:04.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'oyyerD918wRExgeuwLQNphank4bD2k3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f29636c2fd756855-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.525 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.525 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.526 DEBUG:boto:path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.526 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.527 DEBUG:boto:Method: GET 2023-10-18T19:59:04.527 DEBUG:boto:Path: /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.528 DEBUG:boto:Data: 2023-10-18T19:59:04.528 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.528 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.529 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.529 DEBUG:boto:Params: {} 2023-10-18T19:59:04.530 DEBUG:boto:Token: None 2023-10-18T19:59:04.530 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.531 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YmASl/8SKvwJ31Uq31H3hHm2vUY= 2023-10-18T19:59:04.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YmASl/8SKvwJ31Uq31H3hHm2vUY=', 'Content-Length': '0'} 2023-10-18T19:59:04.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'oyyerD918wRExgeuwLQNphank4bD2k3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e926b8da35ccbe4d-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.578 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.579 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.579 DEBUG:boto:path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.580 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.580 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.581 DEBUG:boto:Path: /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.581 DEBUG:boto:Data: 2023-10-18T19:59:04.582 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.583 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.583 DEBUG:boto:Params: {} 2023-10-18T19:59:04.584 DEBUG:boto:Token: None 2023-10-18T19:59:04.584 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Y2F3ki2G/m4Zax75MKqWruVLe40= 2023-10-18T19:59:04.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Y2F3ki2G/m4Zax75MKqWruVLe40=', 'Content-Length': '0'} 2023-10-18T19:59:04.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'oyyerD918wRExgeuwLQNphank4bD2k3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7bba48da1760b4b-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.588 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.589 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.589 DEBUG:boto:path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.589 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.590 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.590 DEBUG:boto:Path: /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.591 DEBUG:boto:Data: 2023-10-18T19:59:04.591 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.592 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.592 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.593 DEBUG:boto:Params: {} 2023-10-18T19:59:04.593 DEBUG:boto:Token: None 2023-10-18T19:59:04.594 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.594 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Y2F3ki2G/m4Zax75MKqWruVLe40= 2023-10-18T19:59:04.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Y2F3ki2G/m4Zax75MKqWruVLe40=', 'Content-Length': '0'} 2023-10-18T19:59:04.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'oyyerD918wRExgeuwLQNphank4bD2k3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074a10f16a1489fe0-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.598 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.599 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.599 DEBUG:boto:path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.599 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.600 DEBUG:boto:Method: GET 2023-10-18T19:59:04.600 DEBUG:boto:Path: /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.601 DEBUG:boto:Data: 2023-10-18T19:59:04.601 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.602 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.603 DEBUG:boto:Params: {} 2023-10-18T19:59:04.603 DEBUG:boto:Token: None 2023-10-18T19:59:04.604 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YmASl/8SKvwJ31Uq31H3hHm2vUY= 2023-10-18T19:59:04.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YmASl/8SKvwJ31Uq31H3hHm2vUY=', 'Content-Length': '0'} 2023-10-18T19:59:04.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'oyyerD918wRExgeuwLQNphank4bD2k3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003993b50fba5a44fb-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.607 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.608 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.608 DEBUG:boto:path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.608 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.609 DEBUG:boto:Method: GET 2023-10-18T19:59:04.609 DEBUG:boto:Path: /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.610 DEBUG:boto:Data: 2023-10-18T19:59:04.610 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.610 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.611 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.611 DEBUG:boto:Params: {} 2023-10-18T19:59:04.612 DEBUG:boto:Token: None 2023-10-18T19:59:04.612 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=oyyerD918wRExgeuwLQNphank4bD2k3 2023-10-18T19:59:04.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YmASl/8SKvwJ31Uq31H3hHm2vUY= 2023-10-18T19:59:04.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YmASl/8SKvwJ31Uq31H3hHm2vUY=', 'Content-Length': '0'} 2023-10-18T19:59:04.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'oyyerD918wRExgeuwLQNphank4bD2k3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d2ecd9b42be0ef9-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.662 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.664 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.664 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.665 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.665 DEBUG:boto:Method: GET 2023-10-18T19:59:04.666 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.667 DEBUG:boto:Data: 2023-10-18T19:59:04.667 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.668 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.669 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.669 DEBUG:boto:Params: {} 2023-10-18T19:59:04.670 DEBUG:boto:Token: None 2023-10-18T19:59:04.670 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.671 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678d1a85235d6be7-0065303908-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.674 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.675 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.676 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.677 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.677 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.678 DEBUG:boto:Method: GET 2023-10-18T19:59:04.679 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.679 DEBUG:boto:Data: 2023-10-18T19:59:04.680 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.682 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.682 DEBUG:boto:Params: {} 2023-10-18T19:59:04.683 DEBUG:boto:Token: None 2023-10-18T19:59:04.684 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.685 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f05b07280f04c20-0065303908-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.689 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.690 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.690 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.690 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.691 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:04.691 DEBUG:boto:Data: 2023-10-18T19:59:04.691 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.692 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.692 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.693 DEBUG:boto:Params: {} 2023-10-18T19:59:04.693 DEBUG:boto:Token: None 2023-10-18T19:59:04.694 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj 2023-10-18T19:59:04.694 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU= 2023-10-18T19:59:04.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU=', 'Content-Length': '0'} 2023-10-18T19:59:04.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2d64b2a3a6aec5b-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.698 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.699 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.699 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.700 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:04.700 DEBUG:boto:Data: 2023-10-18T19:59:04.700 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.701 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.701 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.702 DEBUG:boto:Params: {} 2023-10-18T19:59:04.702 DEBUG:boto:Token: None 2023-10-18T19:59:04.703 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj 2023-10-18T19:59:04.703 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU= 2023-10-18T19:59:04.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU=', 'Content-Length': '0'} 2023-10-18T19:59:04.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f9340febad40b38-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.708 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.708 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.709 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.709 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.710 DEBUG:boto:Method: GET 2023-10-18T19:59:04.710 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.710 DEBUG:boto:Data: 2023-10-18T19:59:04.711 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.711 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.712 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.712 DEBUG:boto:Params: {} 2023-10-18T19:59:04.713 DEBUG:boto:Token: None 2023-10-18T19:59:04.713 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.714 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw= 2023-10-18T19:59:04.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw=', 'Content-Length': '0'} 2023-10-18T19:59:04.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e41cd554b186b7d-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.717 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.717 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.718 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.718 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.719 DEBUG:boto:Method: GET 2023-10-18T19:59:04.719 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.719 DEBUG:boto:Data: 2023-10-18T19:59:04.720 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.720 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.721 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.721 DEBUG:boto:Params: {} 2023-10-18T19:59:04.722 DEBUG:boto:Token: None 2023-10-18T19:59:04.722 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.723 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw= 2023-10-18T19:59:04.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw=', 'Content-Length': '0'} 2023-10-18T19:59:04.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000708e3b2b516618c5-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.766 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.767 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.768 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.768 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.768 DEBUG:boto:Method: GET 2023-10-18T19:59:04.769 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.769 DEBUG:boto:Data: 2023-10-18T19:59:04.770 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.771 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.771 DEBUG:boto:Params: {} 2023-10-18T19:59:04.772 DEBUG:boto:Token: None 2023-10-18T19:59:04.772 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.773 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c673e06ea96d7fd-0065303908-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.776 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.778 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.778 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.779 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.779 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.779 DEBUG:boto:Method: GET 2023-10-18T19:59:04.780 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.780 DEBUG:boto:Data: 2023-10-18T19:59:04.781 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.782 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.782 DEBUG:boto:Params: {} 2023-10-18T19:59:04.783 DEBUG:boto:Token: None 2023-10-18T19:59:04.783 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000846a421d268e46bf-0065303908-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.787 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.788 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.788 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.788 DEBUG:boto:path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.789 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.789 DEBUG:boto:Method: GET 2023-10-18T19:59:04.790 DEBUG:boto:Path: /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.790 DEBUG:boto:Data: 2023-10-18T19:59:04.791 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.791 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.791 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.792 DEBUG:boto:Params: {} 2023-10-18T19:59:04.792 DEBUG:boto:Token: None 2023-10-18T19:59:04.793 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.793 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cwclop7GDC0D/VTrGePZu1ac3Gg= 2023-10-18T19:59:04.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cwclop7GDC0D/VTrGePZu1ac3Gg=', 'Content-Length': '0'} 2023-10-18T19:59:04.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b41e3191041a4cdb-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.842 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.843 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.843 DEBUG:boto:path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.844 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.844 DEBUG:boto:Method: GET 2023-10-18T19:59:04.845 DEBUG:boto:Path: /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.845 DEBUG:boto:Data: 2023-10-18T19:59:04.846 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.846 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.847 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.847 DEBUG:boto:Params: {} 2023-10-18T19:59:04.847 DEBUG:boto:Token: None 2023-10-18T19:59:04.848 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.849 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cwclop7GDC0D/VTrGePZu1ac3Gg= 2023-10-18T19:59:04.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cwclop7GDC0D/VTrGePZu1ac3Gg=', 'Content-Length': '0'} 2023-10-18T19:59:04.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d39ef1ba01a77752-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.852 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.852 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.853 DEBUG:boto:path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.853 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.853 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.854 DEBUG:boto:Path: /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.854 DEBUG:boto:Data: 2023-10-18T19:59:04.855 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.856 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.856 DEBUG:boto:Params: {} 2023-10-18T19:59:04.856 DEBUG:boto:Token: None 2023-10-18T19:59:04.857 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.857 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ks1/s+xxT6u1k2HDMDDfdZsvRNI= 2023-10-18T19:59:04.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ks1/s+xxT6u1k2HDMDDfdZsvRNI=', 'Content-Length': '0'} 2023-10-18T19:59:04.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e557f6b44eca167-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.860 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.861 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.861 DEBUG:boto:path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.861 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.862 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.862 DEBUG:boto:Path: /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.863 DEBUG:boto:Data: 2023-10-18T19:59:04.863 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.863 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.864 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.864 DEBUG:boto:Params: {} 2023-10-18T19:59:04.865 DEBUG:boto:Token: None 2023-10-18T19:59:04.865 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.865 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ks1/s+xxT6u1k2HDMDDfdZsvRNI= 2023-10-18T19:59:04.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ks1/s+xxT6u1k2HDMDDfdZsvRNI=', 'Content-Length': '0'} 2023-10-18T19:59:04.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c929973bee6cca2c-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.869 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.869 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.870 DEBUG:boto:path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.870 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.870 DEBUG:boto:Method: GET 2023-10-18T19:59:04.871 DEBUG:boto:Path: /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.871 DEBUG:boto:Data: 2023-10-18T19:59:04.872 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.872 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.872 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.873 DEBUG:boto:Params: {} 2023-10-18T19:59:04.873 DEBUG:boto:Token: None 2023-10-18T19:59:04.874 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.874 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cwclop7GDC0D/VTrGePZu1ac3Gg= 2023-10-18T19:59:04.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cwclop7GDC0D/VTrGePZu1ac3Gg=', 'Content-Length': '0'} 2023-10-18T19:59:04.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000450717a495275073-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.877 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.878 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.878 DEBUG:boto:path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.878 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.879 DEBUG:boto:Method: GET 2023-10-18T19:59:04.879 DEBUG:boto:Path: /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.880 DEBUG:boto:Data: 2023-10-18T19:59:04.880 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.880 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.881 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.881 DEBUG:boto:Params: {} 2023-10-18T19:59:04.882 DEBUG:boto:Token: None 2023-10-18T19:59:04.882 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz 2023-10-18T19:59:04.883 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cwclop7GDC0D/VTrGePZu1ac3Gg= 2023-10-18T19:59:04.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cwclop7GDC0D/VTrGePZu1ac3Gg=', 'Content-Length': '0'} 2023-10-18T19:59:04.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'BH2jE4UqtGp8OfC1t9VV-x6wQuT5PZz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cf0dbb9fe110748-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.926 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.927 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.927 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.928 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.928 DEBUG:boto:Method: GET 2023-10-18T19:59:04.928 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.929 DEBUG:boto:Data: 2023-10-18T19:59:04.929 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.930 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.930 DEBUG:boto:Params: {} 2023-10-18T19:59:04.931 DEBUG:boto:Token: None 2023-10-18T19:59:04.931 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.932 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f050149c74fce5-0065303908-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.935 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.936 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:04.936 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:04.937 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.937 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:04.938 DEBUG:boto:Method: GET 2023-10-18T19:59:04.938 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:04.939 DEBUG:boto:Data: 2023-10-18T19:59:04.939 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.939 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.940 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.940 DEBUG:boto:Params: {} 2023-10-18T19:59:04.941 DEBUG:boto:Token: None 2023-10-18T19:59:04.941 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/ 2023-10-18T19:59:04.942 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU= 2023-10-18T19:59:04.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdWlLOczjNahJeRkyahVCX6aIiU=', 'Content-Length': '0'} 2023-10-18T19:59:04.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055629ceb24cefb1b-0065303908-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.945 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:04.946 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.946 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.947 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.947 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:04.947 DEBUG:boto:Data: 2023-10-18T19:59:04.948 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.948 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.949 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.949 DEBUG:boto:Params: {} 2023-10-18T19:59:04.950 DEBUG:boto:Token: None 2023-10-18T19:59:04.950 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj 2023-10-18T19:59:04.950 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU= 2023-10-18T19:59:04.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU=', 'Content-Length': '0'} 2023-10-18T19:59:04.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba7e48a007adaabb-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.955 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.955 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.955 DEBUG:boto:Method: HEAD 2023-10-18T19:59:04.956 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:04.956 DEBUG:boto:Data: 2023-10-18T19:59:04.957 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.957 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.958 DEBUG:boto:Params: {} 2023-10-18T19:59:04.958 DEBUG:boto:Token: None 2023-10-18T19:59:04.959 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj 2023-10-18T19:59:04.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU= 2023-10-18T19:59:04.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obCpoO2hCjBZBXkiGplna0sn0AU=', 'Content-Length': '0'} 2023-10-18T19:59:04.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000360b2117d12793ab-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:04.964 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.964 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.965 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.965 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.966 DEBUG:boto:Method: GET 2023-10-18T19:59:04.966 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.967 DEBUG:boto:Data: 2023-10-18T19:59:04.967 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.967 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:04.968 DEBUG:boto:Port: 8001 2023-10-18T19:59:04.968 DEBUG:boto:Params: {} 2023-10-18T19:59:04.969 DEBUG:boto:Token: None 2023-10-18T19:59:04.969 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.970 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw= 2023-10-18T19:59:04.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw=', 'Content-Length': '0'} 2023-10-18T19:59:04.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000455098804756e2b1-0065303908-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:04.973 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:04.973 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:04.974 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.974 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.975 DEBUG:boto:Method: GET 2023-10-18T19:59:04.975 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.975 DEBUG:boto:Data: 2023-10-18T19:59:04.976 DEBUG:boto:Headers: {} 2023-10-18T19:59:04.976 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:04.977 DEBUG:boto:Port: 8000 2023-10-18T19:59:04.977 DEBUG:boto:Params: {} 2023-10-18T19:59:04.977 DEBUG:boto:Token: None 2023-10-18T19:59:04.978 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:04 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:04.978 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw= 2023-10-18T19:59:04.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/zAAKBfcL5PaM1t0A06lL216Qw=', 'Content-Length': '0'} 2023-10-18T19:59:04.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b26a2a66e665baf8-0065303908-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.022 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:05.023 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:05.023 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.024 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.024 DEBUG:boto:Method: GET 2023-10-18T19:59:05.025 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:05.025 DEBUG:boto:Data: 2023-10-18T19:59:05.026 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.026 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:05.027 DEBUG:boto:Port: 8001 2023-10-18T19:59:05.027 DEBUG:boto:Params: {} 2023-10-18T19:59:05.027 DEBUG:boto:Token: None 2023-10-18T19:59:05.028 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/ 2023-10-18T19:59:05.029 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA= 2023-10-18T19:59:05.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA=', 'Content-Length': '0'} 2023-10-18T19:59:05.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0fc6c4cd46577fd-0065303909-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.032 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:05.033 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:05.034 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:05.034 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.034 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.035 DEBUG:boto:Method: GET 2023-10-18T19:59:05.035 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:05.036 DEBUG:boto:Data: 2023-10-18T19:59:05.036 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.037 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:05.037 DEBUG:boto:Port: 8000 2023-10-18T19:59:05.037 DEBUG:boto:Params: {} 2023-10-18T19:59:05.038 DEBUG:boto:Token: None 2023-10-18T19:59:05.039 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/ 2023-10-18T19:59:05.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA= 2023-10-18T19:59:05.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA=', 'Content-Length': '0'} 2023-10-18T19:59:05.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076c12754aaf79fa0-0065303909-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.086 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:05.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:05.088 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.088 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.089 DEBUG:boto:path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.089 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.090 DEBUG:boto:Method: GET 2023-10-18T19:59:05.090 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.091 DEBUG:boto:Data: 2023-10-18T19:59:05.091 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.092 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:05.092 DEBUG:boto:Port: 8001 2023-10-18T19:59:05.092 DEBUG:boto:Params: {} 2023-10-18T19:59:05.093 DEBUG:boto:Token: None 2023-10-18T19:59:05.093 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.094 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+oAHWzkkS9y9I+VSEai3GQEuU4A= 2023-10-18T19:59:05.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+oAHWzkkS9y9I+VSEai3GQEuU4A=', 'Content-Length': '0'} 2023-10-18T19:59:05.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048f201156c83a222-0065303909-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.142 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.144 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.144 DEBUG:boto:path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.145 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.146 DEBUG:boto:Method: GET 2023-10-18T19:59:05.147 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.147 DEBUG:boto:Data: 2023-10-18T19:59:05.148 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.149 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:05.149 DEBUG:boto:Port: 8000 2023-10-18T19:59:05.150 DEBUG:boto:Params: {} 2023-10-18T19:59:05.150 DEBUG:boto:Token: None 2023-10-18T19:59:05.151 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.151 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+oAHWzkkS9y9I+VSEai3GQEuU4A= 2023-10-18T19:59:05.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+oAHWzkkS9y9I+VSEai3GQEuU4A=', 'Content-Length': '0'} 2023-10-18T19:59:05.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025581fe663cc9415-0065303909-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.155 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.155 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.155 DEBUG:boto:path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.156 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.156 DEBUG:boto:Method: HEAD 2023-10-18T19:59:05.157 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.157 DEBUG:boto:Data: 2023-10-18T19:59:05.157 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:05.158 DEBUG:boto:Port: 8001 2023-10-18T19:59:05.158 DEBUG:boto:Params: {} 2023-10-18T19:59:05.159 DEBUG:boto:Token: None 2023-10-18T19:59:05.159 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.160 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uaGF9qhdrwE0plK68i+lDH1fPko= 2023-10-18T19:59:05.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uaGF9qhdrwE0plK68i+lDH1fPko=', 'Content-Length': '0'} 2023-10-18T19:59:05.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000756682f264a5b243-0065303909-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.163 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.163 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.164 DEBUG:boto:path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.164 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.164 DEBUG:boto:Method: HEAD 2023-10-18T19:59:05.165 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.165 DEBUG:boto:Data: 2023-10-18T19:59:05.166 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:05.166 DEBUG:boto:Port: 8000 2023-10-18T19:59:05.167 DEBUG:boto:Params: {} 2023-10-18T19:59:05.167 DEBUG:boto:Token: None 2023-10-18T19:59:05.168 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uaGF9qhdrwE0plK68i+lDH1fPko= 2023-10-18T19:59:05.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uaGF9qhdrwE0plK68i+lDH1fPko=', 'Content-Length': '0'} 2023-10-18T19:59:05.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4cb90c92df26182-0065303909-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:05.171 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.172 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.172 DEBUG:boto:path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.173 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.173 DEBUG:boto:Method: GET 2023-10-18T19:59:05.174 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.174 DEBUG:boto:Data: 2023-10-18T19:59:05.175 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.175 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:05.175 DEBUG:boto:Port: 8001 2023-10-18T19:59:05.176 DEBUG:boto:Params: {} 2023-10-18T19:59:05.177 DEBUG:boto:Token: None 2023-10-18T19:59:05.177 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.178 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+oAHWzkkS9y9I+VSEai3GQEuU4A= 2023-10-18T19:59:05.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+oAHWzkkS9y9I+VSEai3GQEuU4A=', 'Content-Length': '0'} 2023-10-18T19:59:05.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000338b9768d4d53785-0065303909-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.222 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.223 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.223 DEBUG:boto:path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.224 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.224 DEBUG:boto:Method: GET 2023-10-18T19:59:05.225 DEBUG:boto:Path: /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.225 DEBUG:boto:Data: 2023-10-18T19:59:05.226 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:05.227 DEBUG:boto:Port: 8000 2023-10-18T19:59:05.227 DEBUG:boto:Params: {} 2023-10-18T19:59:05.227 DEBUG:boto:Token: None 2023-10-18T19:59:05.228 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj?versionId=NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v 2023-10-18T19:59:05.229 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+oAHWzkkS9y9I+VSEai3GQEuU4A= 2023-10-18T19:59:05.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+oAHWzkkS9y9I+VSEai3GQEuU4A=', 'Content-Length': '0'} 2023-10-18T19:59:05.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:38 GMT'), ('x-amz-version-id', 'NAVmL-UuMH5Lyboppq2UUDWFWO-Ud0v'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e31c4e6d226d90c-0065303909-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.232 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:05.232 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:05.233 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.233 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.234 DEBUG:boto:Method: GET 2023-10-18T19:59:05.234 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:05.235 DEBUG:boto:Data: 2023-10-18T19:59:05.235 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.236 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:05.236 DEBUG:boto:Port: 8001 2023-10-18T19:59:05.237 DEBUG:boto:Params: {} 2023-10-18T19:59:05.237 DEBUG:boto:Token: None 2023-10-18T19:59:05.238 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/ 2023-10-18T19:59:05.238 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA= 2023-10-18T19:59:05.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA=', 'Content-Length': '0'} 2023-10-18T19:59:05.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001d7e58802dd34da-0065303909-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.286 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:05.287 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:05.288 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:05.288 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.289 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.289 DEBUG:boto:Method: GET 2023-10-18T19:59:05.289 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:05.290 DEBUG:boto:Data: 2023-10-18T19:59:05.290 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.291 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:05.291 DEBUG:boto:Port: 8000 2023-10-18T19:59:05.292 DEBUG:boto:Params: {} 2023-10-18T19:59:05.292 DEBUG:boto:Token: None 2023-10-18T19:59:05.293 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/ 2023-10-18T19:59:05.293 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA= 2023-10-18T19:59:05.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA=', 'Content-Length': '0'} 2023-10-18T19:59:05.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c4e1729846d3cc4-0065303909-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.297 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:05.298 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.298 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.298 DEBUG:boto:Method: HEAD 2023-10-18T19:59:05.299 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:05.299 DEBUG:boto:Data: 2023-10-18T19:59:05.300 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.300 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:05.301 DEBUG:boto:Port: 8001 2023-10-18T19:59:05.301 DEBUG:boto:Params: {} 2023-10-18T19:59:05.301 DEBUG:boto:Token: None 2023-10-18T19:59:05.302 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj 2023-10-18T19:59:05.303 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V5GxfcrI6mjFB3D/SS/MjZssAuo= 2023-10-18T19:59:05.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V5GxfcrI6mjFB3D/SS/MjZssAuo=', 'Content-Length': '0'} 2023-10-18T19:59:05.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026eb5c49b042f4f6-0065303909-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.308 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.309 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.309 DEBUG:boto:Method: HEAD 2023-10-18T19:59:05.310 DEBUG:boto:Path: /bofrnf-13/obj 2023-10-18T19:59:05.310 DEBUG:boto:Data: 2023-10-18T19:59:05.311 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.311 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:05.312 DEBUG:boto:Port: 8000 2023-10-18T19:59:05.312 DEBUG:boto:Params: {} 2023-10-18T19:59:05.313 DEBUG:boto:Token: None 2023-10-18T19:59:05.313 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj 2023-10-18T19:59:05.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V5GxfcrI6mjFB3D/SS/MjZssAuo= 2023-10-18T19:59:05.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V5GxfcrI6mjFB3D/SS/MjZssAuo=', 'Content-Length': '0'} 2023-10-18T19:59:05.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e96f6027788c206-0065303909-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:05.319 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.319 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.319 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:05.320 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:05.320 DEBUG:boto:Method: GET 2023-10-18T19:59:05.321 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:05.321 DEBUG:boto:Data: 2023-10-18T19:59:05.322 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:05.323 DEBUG:boto:Port: 8001 2023-10-18T19:59:05.323 DEBUG:boto:Params: {} 2023-10-18T19:59:05.324 DEBUG:boto:Token: None 2023-10-18T19:59:05.325 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:05.326 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nYyzAGEqDjrNe9w+cqlIPyVQFc4= 2023-10-18T19:59:05.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nYyzAGEqDjrNe9w+cqlIPyVQFc4=', 'Content-Length': '0'} 2023-10-18T19:59:05.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000233021004ccd3163-0065303909-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.330 DEBUG:boto:path=/bofrnf-13/obj 2023-10-18T19:59:05.330 DEBUG:boto:auth_path=/bofrnf-13/obj 2023-10-18T19:59:05.331 DEBUG:boto:path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:05.331 DEBUG:boto:auth_path=/bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:05.332 DEBUG:boto:Method: GET 2023-10-18T19:59:05.332 DEBUG:boto:Path: /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:05.333 DEBUG:boto:Data: 2023-10-18T19:59:05.333 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:05.334 DEBUG:boto:Port: 8000 2023-10-18T19:59:05.334 DEBUG:boto:Params: {} 2023-10-18T19:59:05.335 DEBUG:boto:Token: None 2023-10-18T19:59:05.335 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/obj?versionId=Z6voUTCwzJD04f4D04wP01totCTFrWP 2023-10-18T19:59:05.336 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nYyzAGEqDjrNe9w+cqlIPyVQFc4= 2023-10-18T19:59:05.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nYyzAGEqDjrNe9w+cqlIPyVQFc4=', 'Content-Length': '0'} 2023-10-18T19:59:05.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:57:39 GMT'), ('x-amz-version-id', 'Z6voUTCwzJD04f4D04wP01totCTFrWP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac4a0257ac9a31b4-0065303909-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.378 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:05.379 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:05.379 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.380 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.380 DEBUG:boto:Method: GET 2023-10-18T19:59:05.381 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:05.381 DEBUG:boto:Data: 2023-10-18T19:59:05.382 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.382 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:05.383 DEBUG:boto:Port: 8001 2023-10-18T19:59:05.383 DEBUG:boto:Params: {} 2023-10-18T19:59:05.384 DEBUG:boto:Token: None 2023-10-18T19:59:05.384 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/ 2023-10-18T19:59:05.384 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA= 2023-10-18T19:59:05.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA=', 'Content-Length': '0'} 2023-10-18T19:59:05.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2b4a2db77c6b92-0065303909-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.388 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:05.389 DEBUG:boto:path=/bofrnf-13/ 2023-10-18T19:59:05.389 DEBUG:boto:auth_path=/bofrnf-13/ 2023-10-18T19:59:05.390 DEBUG:boto:path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.390 DEBUG:boto:auth_path=/bofrnf-13/?prefix=obj 2023-10-18T19:59:05.391 DEBUG:boto:Method: GET 2023-10-18T19:59:05.391 DEBUG:boto:Path: /bofrnf-13/?prefix=obj 2023-10-18T19:59:05.392 DEBUG:boto:Data: 2023-10-18T19:59:05.392 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.393 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:05.393 DEBUG:boto:Port: 8000 2023-10-18T19:59:05.393 DEBUG:boto:Params: {} 2023-10-18T19:59:05.394 DEBUG:boto:Token: None 2023-10-18T19:59:05.394 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-13/ 2023-10-18T19:59:05.395 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA= 2023-10-18T19:59:05.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lWA7sH2e1WV/spkMJNlU8o9h4GA=', 'Content-Length': '0'} 2023-10-18T19:59:05.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f31fa227f7232314-0065303909-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.399 DEBUG:boto:b'bofrnf-13obj1000falseobj2023-10-18T19:57:39.401Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:05.399 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-13 zones={a2, a1} 2023-10-18T19:59:05.400 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-18T19:59:05.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.401 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:59:05.454 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:05.455 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:05.501 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:59:05.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.502 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-18T19:59:05.502 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:57:53.289654Z" 2023-10-18T19:59:05.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.503 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-18T19:59:05.503 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:56.622781Z" 2023-10-18T19:59:05.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.504 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T19:59:05.504 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:50:16.613109Z" 2023-10-18T19:59:05.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.505 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-10-18T19:59:05.505 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:58:15.292795Z" 2023-10-18T19:59:05.505 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.505 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:59:05.506 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000001632'} 2023-10-18T19:59:05.507 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T19:59:05.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.508 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:59:05.559 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:05.559 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:05.600 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:59:05.601 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:59:05.601 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:59:05.601 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:59:05.601 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:59:05.602 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 2477610155421060198 2023-10-18T19:59:05.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.602 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:59:05.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.602 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:59:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T19:59:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-18T19:59:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:05.604 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:05.604 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:53.289654Z" 2023-10-18T19:59:05.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:59:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T19:59:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-18T19:59:05.606 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:05.606 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:05.606 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:05.606 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.622781Z" 2023-10-18T19:59:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:59:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:05.608 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T19:59:05.608 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T19:59:05.608 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:05.608 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:05.608 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:05.609 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:50:16.613109Z" 2023-10-18T19:59:05.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.609 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:59:05.610 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:05.610 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T19:59:05.610 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-10-18T19:59:05.610 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:05.611 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:05.611 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:05.611 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:58:15.292795Z" 2023-10-18T19:59:05.611 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.611 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.612 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:59:05.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.612 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:59:05.612 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T19:59:05.612 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T19:59:05.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.613 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:59:05Z" 2023-10-18T19:59:05.613 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:59:05.614 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2477610155421060198 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:53.289654Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.622781Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:50:16.613109Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:58:15.292795Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T19:59:05Z" } 2023-10-18T19:59:05.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:53.289654Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.622781Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:50:16.613109Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:58:15.292795Z'}}] 2023-10-18T19:59:05.615 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000001632'} 2023-10-18T19:59:05.616 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000001632'} 2023-10-18T19:59:05.616 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T19:59:05.616 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-18T19:59:05.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.618 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:59:05.666 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:05.666 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:05.713 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:59:05.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.713 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-18T19:59:05.713 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:57:16.726511Z" 2023-10-18T19:59:05.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.714 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-18T19:59:05.714 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:50:16.637795Z" 2023-10-18T19:59:05.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.715 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T19:59:05.715 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:56.644123Z" 2023-10-18T19:59:05.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.716 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002478", 2023-10-18T19:59:05.716 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:58:37.434579Z" 2023-10-18T19:59:05.716 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.716 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:59:05.717 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002478'} 2023-10-18T19:59:05.718 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T19:59:05.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.719 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:59:05.770 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:05.770 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:05.812 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:59:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:59:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:59:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:59:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:59:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T19:59:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:59:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:59:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T19:59:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-18T19:59:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:16.726511Z" 2023-10-18T19:59:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:59:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T19:59:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-18T19:59:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:05.817 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:50:16.637795Z" 2023-10-18T19:59:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:59:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T19:59:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T19:59:05.819 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.644123Z" 2023-10-18T19:59:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.821 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:59:05.821 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:05.821 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T19:59:05.821 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002478", 2023-10-18T19:59:05.822 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:05.822 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:05.822 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:05.822 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:58:37.434579Z" 2023-10-18T19:59:05.822 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.823 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:59:05.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.823 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:59:05.823 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T19:59:05.824 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T19:59:05.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.824 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:59:05Z" 2023-10-18T19:59:05.824 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:59:05.825 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:16.726511Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:50:16.637795Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.644123Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002478", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:58:37.434579Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T19:59:05Z" } 2023-10-18T19:59:05.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:16.726511Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:50:16.637795Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.644123Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002478', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:58:37.434579Z'}}] 2023-10-18T19:59:05.826 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002478'} 2023-10-18T19:59:05.827 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000002478'} 2023-10-18T19:59:05.827 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T19:59:05.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-10-18T19:59:05.829 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:05.829 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:05.830 DEBUG:boto:Method: PUT 2023-10-18T19:59:05.830 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T19:59:05.831 DEBUG:boto:Data: 2023-10-18T19:59:05.831 DEBUG:boto:Headers: {} 2023-10-18T19:59:05.831 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:05.832 DEBUG:boto:Port: 8000 2023-10-18T19:59:05.832 DEBUG:boto:Params: {} 2023-10-18T19:59:05.833 DEBUG:boto:Token: None 2023-10-18T19:59:05.833 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:59:05 GMT /bofrnf-14/ 2023-10-18T19:59:05.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j86nslhUB1xo1dcgRxilHn7RXWc= 2023-10-18T19:59:05.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j86nslhUB1xo1dcgRxilHn7RXWc=', 'Content-Length': '0'} 2023-10-18T19:59:05.847 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000099abdcb75da4ad3-0065303909-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:59:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:05.847 DEBUG:rgw_multi.tests:created bucket=bofrnf-14 2023-10-18T19:59:05.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.848 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:59:05.898 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:05.898 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:05.927 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:59:05.946 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:59:05.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.946 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659145.844758_221.1", 2023-10-18T19:59:05.947 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:59:05.844758Z" 2023-10-18T19:59:05.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.947 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:59:05.947 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T19:59:05.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.948 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:59:05.948 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:59:05.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:05.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:05.949 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:59:05.949 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:59:05.949 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:05.949 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:59:05.951 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659145.844758_221.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:59:05.951 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:59:05.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:05.953 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:59:06.002 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:06.002 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:06.031 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:06.029+0000 7f080f7c3780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:59:06.031 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:06.030+0000 7f080f7c3780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:59:06.044 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:59:06.044 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:59:06.044 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:59:06.044 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:59:06.045 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:59:06.045 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:59:06.045 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:06.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:06.045 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:59:06.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:06.046 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:59:06.046 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:06.046 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:59:06.047 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659145.844758_221.1", 2023-10-18T19:59:06.047 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:06.047 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:59:06.047 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:06.047 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:59:05.844758Z", 2023-10-18T19:59:06.048 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:06.048 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:06.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:06.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:06.048 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:59:06.049 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:06.049 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:59:06.049 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:59:06.049 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:06.049 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:06.050 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:06.050 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T19:59:06.050 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:06.050 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:06.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:06.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:06.051 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:59:06.051 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:06.051 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:59:06.051 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:59:06.052 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:06.052 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:06.052 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:06.052 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:59:06.053 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:06.053 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:06.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:06.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:06.053 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:59:06.053 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:06.054 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:59:06.054 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:59:06.054 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:06.054 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:06.054 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:06.055 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:59:06.055 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:06.055 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:06.055 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:06.055 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:59:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:59:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:59:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:59:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:06.057 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:59:06Z" 2023-10-18T19:59:06.057 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:59:06.058 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659145.844758_221.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:59:05.844758Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:59:06Z" } 2023-10-18T19:59:06.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659145.844758_221.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:59:05.844758Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:59:06.059 DEBUG:rgw_multi.tests:log_status={0: '1_1697659145.844758_221.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:59:06.059 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659145.844758_221.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:59:06.059 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:59:06.060 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:06.060 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:06.061 DEBUG:boto:Method: HEAD 2023-10-18T19:59:06.061 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T19:59:06.061 DEBUG:boto:Data: 2023-10-18T19:59:06.062 DEBUG:boto:Headers: {} 2023-10-18T19:59:06.062 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:06.063 DEBUG:boto:Port: 8000 2023-10-18T19:59:06.063 DEBUG:boto:Params: {} 2023-10-18T19:59:06.063 DEBUG:boto:Token: None 2023-10-18T19:59:06.064 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:06 GMT /bofrnf-14/ 2023-10-18T19:59:06.064 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L4HdxV5C2N72gi2hM4tELLdqPRU= 2023-10-18T19:59:06.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L4HdxV5C2N72gi2hM4tELLdqPRU=', 'Content-Length': '0'} 2023-10-18T19:59:06.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d748bdca742e4bf-006530390a-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:06.068 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:06.068 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:06.068 DEBUG:boto:Method: PUT 2023-10-18T19:59:06.069 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T19:59:06.069 DEBUG:boto:Data: 2023-10-18T19:59:06.070 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T19:59:06.070 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:06.070 DEBUG:boto:Port: 8000 2023-10-18T19:59:06.071 DEBUG:boto:Params: {} 2023-10-18T19:59:06.071 DEBUG:boto:Token: None 2023-10-18T19:59:06.072 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 19:59:06 GMT /bofrnf-14/obj 2023-10-18T19:59:06.072 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rV27gSCy6ySdC6CQFqZRrc14eF8= 2023-10-18T19:59:06.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:59:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rV27gSCy6ySdC6CQFqZRrc14eF8='} 2023-10-18T19:59:06.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd878788638b7aad-006530390a-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:06.115 DEBUG:rgw_multi.tests:created initial version id=None 2023-10-18T19:59:06.115 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-14 2023-10-18T19:59:06.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:59:06.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:59:06.117 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:59:06.173 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:06.173 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:06.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:06.206+0000 7ff2cc06b780 1 waiting to reach incremental sync.. 2023-10-18T19:59:11.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:11.207+0000 7ff2cc06b780 1 waiting to reach incremental sync.. 2023-10-18T19:59:16.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:16.208+0000 7ff2cc06b780 1 waiting to reach incremental sync.. 2023-10-18T19:59:21.211 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:21.209+0000 7ff2cc06b780 1 bucket sync caught up with source: 2023-10-18T19:59:21.211 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.12.6, , ] 2023-10-18T19:59:21.212 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.12.6, , ] 2023-10-18T19:59:21.212 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:21.209+0000 7ff2cc06b780 0 bucket checkpoint complete 2023-10-18T19:59:21.227 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-14 2023-10-18T19:59:21.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:59:21.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:59:21.228 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:59:21.289 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:21.290 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:21.325 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:21.324+0000 7f7ac1ed4780 1 waiting to reach incremental sync.. 2023-10-18T19:59:26.326 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:26.325+0000 7f7ac1ed4780 1 waiting to reach incremental sync.. 2023-10-18T19:59:31.327 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:31.324+0000 7f7ac1ed4780 1 waiting to reach incremental sync.. 2023-10-18T19:59:36.327 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:36.325+0000 7f7ac1ed4780 1 waiting to reach incremental sync.. 2023-10-18T19:59:41.329 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:41.326+0000 7f7ac1ed4780 1 bucket sync caught up with source: 2023-10-18T19:59:41.329 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.33.6, , ] 2023-10-18T19:59:41.329 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.33.6, , ] 2023-10-18T19:59:41.330 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:41.326+0000 7f7ac1ed4780 0 bucket checkpoint complete 2023-10-18T19:59:41.345 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-14 zones={a2, a1} 2023-10-18T19:59:41.345 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.346 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.346 DEBUG:boto:Method: HEAD 2023-10-18T19:59:41.347 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T19:59:41.347 DEBUG:boto:Data: 2023-10-18T19:59:41.347 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.348 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.348 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.349 DEBUG:boto:Params: {} 2023-10-18T19:59:41.349 DEBUG:boto:Token: None 2023-10-18T19:59:41.350 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/ 2023-10-18T19:59:41.350 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wIGNyrABEhwQBcBKeBADK608St4= 2023-10-18T19:59:41.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wIGNyrABEhwQBcBKeBADK608St4=', 'Content-Length': '0'} 2023-10-18T19:59:41.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd17c83dd45a6b12-006530392d-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.354 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.355 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.355 DEBUG:boto:Method: HEAD 2023-10-18T19:59:41.356 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T19:59:41.356 DEBUG:boto:Data: 2023-10-18T19:59:41.357 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.357 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.358 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.358 DEBUG:boto:Params: {} 2023-10-18T19:59:41.359 DEBUG:boto:Token: None 2023-10-18T19:59:41.359 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/ 2023-10-18T19:59:41.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wIGNyrABEhwQBcBKeBADK608St4= 2023-10-18T19:59:41.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wIGNyrABEhwQBcBKeBADK608St4=', 'Content-Length': '0'} 2023-10-18T19:59:41.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000117eb58d658836ff-006530392d-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.363 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T19:59:41.363 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.364 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.364 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.365 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.365 DEBUG:boto:Method: GET 2023-10-18T19:59:41.366 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.366 DEBUG:boto:Data: 2023-10-18T19:59:41.367 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.367 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.368 DEBUG:boto:Params: {} 2023-10-18T19:59:41.369 DEBUG:boto:Token: None 2023-10-18T19:59:41.369 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/?versions 2023-10-18T19:59:41.370 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WNKQ55pm61AsdhRnj+T2cbZX8qc= 2023-10-18T19:59:41.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WNKQ55pm61AsdhRnj+T2cbZX8qc=', 'Content-Length': '0'} 2023-10-18T19:59:41.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e168ce8f4a029bae-006530392d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.373 DEBUG:boto:b'bofrnf-14999falseobjnulltrue2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:41.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:59:41.374 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T19:59:41.375 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.375 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.376 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.376 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.377 DEBUG:boto:Method: GET 2023-10-18T19:59:41.377 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.377 DEBUG:boto:Data: 2023-10-18T19:59:41.378 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.378 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.379 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.379 DEBUG:boto:Params: {} 2023-10-18T19:59:41.380 DEBUG:boto:Token: None 2023-10-18T19:59:41.380 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/?versions 2023-10-18T19:59:41.380 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WNKQ55pm61AsdhRnj+T2cbZX8qc= 2023-10-18T19:59:41.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WNKQ55pm61AsdhRnj+T2cbZX8qc=', 'Content-Length': '0'} 2023-10-18T19:59:41.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000002b086791bda63d-006530392d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.384 DEBUG:boto:b'bofrnf-14999falseobjnulltrue2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:41.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T19:59:41.385 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.385 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.386 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.386 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.386 DEBUG:boto:Method: GET 2023-10-18T19:59:41.387 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.387 DEBUG:boto:Data: 2023-10-18T19:59:41.388 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.388 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.388 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.389 DEBUG:boto:Params: {} 2023-10-18T19:59:41.389 DEBUG:boto:Token: None 2023-10-18T19:59:41.390 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/?versions 2023-10-18T19:59:41.390 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WNKQ55pm61AsdhRnj+T2cbZX8qc= 2023-10-18T19:59:41.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WNKQ55pm61AsdhRnj+T2cbZX8qc=', 'Content-Length': '0'} 2023-10-18T19:59:41.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028595c74642503dc-006530392d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.393 DEBUG:boto:b'bofrnf-14999falseobjnulltrue2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:41.394 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.395 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.395 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.396 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.396 DEBUG:boto:Method: GET 2023-10-18T19:59:41.396 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T19:59:41.397 DEBUG:boto:Data: 2023-10-18T19:59:41.397 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.398 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.399 DEBUG:boto:Params: {} 2023-10-18T19:59:41.399 DEBUG:boto:Token: None 2023-10-18T19:59:41.399 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/?versions 2023-10-18T19:59:41.400 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WNKQ55pm61AsdhRnj+T2cbZX8qc= 2023-10-18T19:59:41.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WNKQ55pm61AsdhRnj+T2cbZX8qc=', 'Content-Length': '0'} 2023-10-18T19:59:41.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd7e5889adfe20d-006530392d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.403 DEBUG:boto:b'bofrnf-14999falseobjnulltrue2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:41.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:41.405 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.405 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.406 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.406 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.406 DEBUG:boto:Method: GET 2023-10-18T19:59:41.407 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.407 DEBUG:boto:Data: 2023-10-18T19:59:41.408 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.409 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.409 DEBUG:boto:Params: {} 2023-10-18T19:59:41.409 DEBUG:boto:Token: None 2023-10-18T19:59:41.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mzl0b0HqHD927CXYcW2FLbFYLHI= 2023-10-18T19:59:41.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mzl0b0HqHD927CXYcW2FLbFYLHI=', 'Content-Length': '0'} 2023-10-18T19:59:41.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000091576c4c3fb2d43d-006530392d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.413 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.414 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.414 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.415 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.415 DEBUG:boto:Method: GET 2023-10-18T19:59:41.416 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.416 DEBUG:boto:Data: 2023-10-18T19:59:41.417 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.417 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.418 DEBUG:boto:Params: {} 2023-10-18T19:59:41.418 DEBUG:boto:Token: None 2023-10-18T19:59:41.419 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mzl0b0HqHD927CXYcW2FLbFYLHI= 2023-10-18T19:59:41.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mzl0b0HqHD927CXYcW2FLbFYLHI=', 'Content-Length': '0'} 2023-10-18T19:59:41.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa411ebf2992b4b0-006530392d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.422 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.423 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.423 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.424 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.424 DEBUG:boto:Method: HEAD 2023-10-18T19:59:41.424 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.425 DEBUG:boto:Data: 2023-10-18T19:59:41.425 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.426 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.427 DEBUG:boto:Params: {} 2023-10-18T19:59:41.427 DEBUG:boto:Token: None 2023-10-18T19:59:41.427 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SLt30s3fSnMBW2lWc5MJbOPYW44= 2023-10-18T19:59:41.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SLt30s3fSnMBW2lWc5MJbOPYW44=', 'Content-Length': '0'} 2023-10-18T19:59:41.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3da65a27f26f244-006530392d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.431 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.431 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.432 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.432 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.433 DEBUG:boto:Method: HEAD 2023-10-18T19:59:41.433 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.434 DEBUG:boto:Data: 2023-10-18T19:59:41.434 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.434 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.435 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.435 DEBUG:boto:Params: {} 2023-10-18T19:59:41.436 DEBUG:boto:Token: None 2023-10-18T19:59:41.436 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.437 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SLt30s3fSnMBW2lWc5MJbOPYW44= 2023-10-18T19:59:41.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SLt30s3fSnMBW2lWc5MJbOPYW44=', 'Content-Length': '0'} 2023-10-18T19:59:41.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ac8bc84667e6b60e-006530392d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:41.440 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.441 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.441 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.442 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.442 DEBUG:boto:Method: GET 2023-10-18T19:59:41.443 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.443 DEBUG:boto:Data: 2023-10-18T19:59:41.444 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.444 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.444 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.445 DEBUG:boto:Params: {} 2023-10-18T19:59:41.445 DEBUG:boto:Token: None 2023-10-18T19:59:41.446 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.446 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mzl0b0HqHD927CXYcW2FLbFYLHI= 2023-10-18T19:59:41.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mzl0b0HqHD927CXYcW2FLbFYLHI=', 'Content-Length': '0'} 2023-10-18T19:59:41.449 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ff60d768605e6f6-006530392d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.449 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.450 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.450 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.451 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.451 DEBUG:boto:Method: GET 2023-10-18T19:59:41.452 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.452 DEBUG:boto:Data: 2023-10-18T19:59:41.452 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.453 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.454 DEBUG:boto:Params: {} 2023-10-18T19:59:41.454 DEBUG:boto:Token: None 2023-10-18T19:59:41.455 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj?versionId=null 2023-10-18T19:59:41.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mzl0b0HqHD927CXYcW2FLbFYLHI= 2023-10-18T19:59:41.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mzl0b0HqHD927CXYcW2FLbFYLHI=', 'Content-Length': '0'} 2023-10-18T19:59:41.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d57fb87e125739e7-006530392d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.459 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.459 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.460 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T19:59:41.460 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T19:59:41.461 DEBUG:boto:Method: GET 2023-10-18T19:59:41.461 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T19:59:41.461 DEBUG:boto:Data: 2023-10-18T19:59:41.462 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.463 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.463 DEBUG:boto:Params: {} 2023-10-18T19:59:41.464 DEBUG:boto:Token: None 2023-10-18T19:59:41.464 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/ 2023-10-18T19:59:41.465 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zFY9qOC36A/NhuCwECwzKQe+gD0= 2023-10-18T19:59:41.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zFY9qOC36A/NhuCwECwzKQe+gD0=', 'Content-Length': '0'} 2023-10-18T19:59:41.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fe68db9332c4d16-006530392d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.514 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:41.515 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.516 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.516 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T19:59:41.516 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T19:59:41.517 DEBUG:boto:Method: GET 2023-10-18T19:59:41.517 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T19:59:41.518 DEBUG:boto:Data: 2023-10-18T19:59:41.518 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.519 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.519 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.519 DEBUG:boto:Params: {} 2023-10-18T19:59:41.520 DEBUG:boto:Token: None 2023-10-18T19:59:41.520 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/ 2023-10-18T19:59:41.521 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zFY9qOC36A/NhuCwECwzKQe+gD0= 2023-10-18T19:59:41.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zFY9qOC36A/NhuCwECwzKQe+gD0=', 'Content-Length': '0'} 2023-10-18T19:59:41.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7fbb77f7dd07d5-006530392d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.525 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:41.526 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.526 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.527 DEBUG:boto:Method: HEAD 2023-10-18T19:59:41.527 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T19:59:41.527 DEBUG:boto:Data: 2023-10-18T19:59:41.528 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.528 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.528 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.529 DEBUG:boto:Params: {} 2023-10-18T19:59:41.529 DEBUG:boto:Token: None 2023-10-18T19:59:41.530 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj 2023-10-18T19:59:41.530 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g/RAHCgxGVTvAt2w/Cv3Yv6Cp4o= 2023-10-18T19:59:41.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g/RAHCgxGVTvAt2w/Cv3Yv6Cp4o=', 'Content-Length': '0'} 2023-10-18T19:59:41.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e932619730843bfb-006530392d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.533 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.533 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.534 DEBUG:boto:Method: HEAD 2023-10-18T19:59:41.534 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T19:59:41.535 DEBUG:boto:Data: 2023-10-18T19:59:41.535 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.535 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.536 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.536 DEBUG:boto:Params: {} 2023-10-18T19:59:41.537 DEBUG:boto:Token: None 2023-10-18T19:59:41.537 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj 2023-10-18T19:59:41.538 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g/RAHCgxGVTvAt2w/Cv3Yv6Cp4o= 2023-10-18T19:59:41.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g/RAHCgxGVTvAt2w/Cv3Yv6Cp4o=', 'Content-Length': '0'} 2023-10-18T19:59:41.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001abf1906850bec76-006530392d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T19:59:41.541 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.541 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.542 DEBUG:boto:Method: GET 2023-10-18T19:59:41.542 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T19:59:41.543 DEBUG:boto:Data: 2023-10-18T19:59:41.543 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.544 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.544 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.544 DEBUG:boto:Params: {} 2023-10-18T19:59:41.545 DEBUG:boto:Token: None 2023-10-18T19:59:41.545 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj 2023-10-18T19:59:41.546 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kqT/HZjL69CYG4jkiP7bjenQ42o= 2023-10-18T19:59:41.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kqT/HZjL69CYG4jkiP7bjenQ42o=', 'Content-Length': '0'} 2023-10-18T19:59:41.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000028032e6ab1ac969f-006530392d-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.549 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.549 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.550 DEBUG:boto:Method: GET 2023-10-18T19:59:41.550 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T19:59:41.550 DEBUG:boto:Data: 2023-10-18T19:59:41.551 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.551 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.551 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.552 DEBUG:boto:Params: {} 2023-10-18T19:59:41.552 DEBUG:boto:Token: None 2023-10-18T19:59:41.553 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj 2023-10-18T19:59:41.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kqT/HZjL69CYG4jkiP7bjenQ42o= 2023-10-18T19:59:41.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kqT/HZjL69CYG4jkiP7bjenQ42o=', 'Content-Length': '0'} 2023-10-18T19:59:41.556 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000508977f5a020e85-006530392d-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.556 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.557 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.557 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T19:59:41.558 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T19:59:41.558 DEBUG:boto:Method: GET 2023-10-18T19:59:41.558 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T19:59:41.559 DEBUG:boto:Data: 2023-10-18T19:59:41.559 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.560 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T19:59:41.560 DEBUG:boto:Port: 8001 2023-10-18T19:59:41.560 DEBUG:boto:Params: {} 2023-10-18T19:59:41.561 DEBUG:boto:Token: None 2023-10-18T19:59:41.561 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/ 2023-10-18T19:59:41.562 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zFY9qOC36A/NhuCwECwzKQe+gD0= 2023-10-18T19:59:41.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zFY9qOC36A/NhuCwECwzKQe+gD0=', 'Content-Length': '0'} 2023-10-18T19:59:41.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d275012eb593ad2-006530392d-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.565 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:41.566 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.566 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.567 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T19:59:41.567 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T19:59:41.568 DEBUG:boto:Method: GET 2023-10-18T19:59:41.568 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T19:59:41.568 DEBUG:boto:Data: 2023-10-18T19:59:41.569 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.569 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.570 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.570 DEBUG:boto:Params: {} 2023-10-18T19:59:41.570 DEBUG:boto:Token: None 2023-10-18T19:59:41.571 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/ 2023-10-18T19:59:41.571 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zFY9qOC36A/NhuCwECwzKQe+gD0= 2023-10-18T19:59:41.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zFY9qOC36A/NhuCwECwzKQe+gD0=', 'Content-Length': '0'} 2023-10-18T19:59:41.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7129e76f27b7c5f-006530392d-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.575 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T19:59:41.575 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-14 zones={a2, a1} 2023-10-18T19:59:41.576 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.576 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.577 DEBUG:boto:path=/bofrnf-14/?versioning 2023-10-18T19:59:41.577 DEBUG:boto:auth_path=/bofrnf-14/?versioning 2023-10-18T19:59:41.577 DEBUG:boto:Method: PUT 2023-10-18T19:59:41.578 DEBUG:boto:Path: /bofrnf-14/?versioning 2023-10-18T19:59:41.578 DEBUG:boto:Data: Enabled Disabled 2023-10-18T19:59:41.578 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.579 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.579 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.580 DEBUG:boto:Params: {} 2023-10-18T19:59:41.580 DEBUG:boto:Token: None 2023-10-18T19:59:41.580 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/?versioning 2023-10-18T19:59:41.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/YE/2pQKC6Y4x5cMX1lyiRh8nxg= 2023-10-18T19:59:41.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/YE/2pQKC6Y4x5cMX1lyiRh8nxg=', 'Content-Length': '228'} 2023-10-18T19:59:41.587 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe341da0263d88c7-006530392d-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:41.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:41.589 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:59:41.643 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:41.643 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:41.674 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T19:59:41.686 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T19:59:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659181.586125_229.1", 2023-10-18T19:59:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:59:41.586125Z" 2023-10-18T19:59:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:59:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T19:59:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:59:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:59:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:59:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T19:59:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:41.690 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T19:59:41.691 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659181.586125_229.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:59:41.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T19:59:41.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:41.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T19:59:41.693 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T19:59:41.743 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:41.743 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:41.770 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:41.767+0000 7f640754b780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:59:41.770 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:41.768+0000 7f640754b780 20 RGW-SYNC:meta: read sync status 2023-10-18T19:59:41.782 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T19:59:41.782 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T19:59:41.783 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T19:59:41.783 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T19:59:41.783 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T19:59:41.783 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T19:59:41.784 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:41.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:41.784 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T19:59:41.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:41.784 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T19:59:41.785 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:41.785 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:59:41.785 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659181.586125_229.1", 2023-10-18T19:59:41.785 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:41.785 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T19:59:41.786 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:41.786 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:59:41.586125Z", 2023-10-18T19:59:41.786 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:41.786 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:41.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:41.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:41.787 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T19:59:41.787 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:41.787 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:59:41.788 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T19:59:41.788 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:41.788 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:41.788 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:41.788 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T19:59:41.789 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:41.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:41.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:41.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:41.789 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T19:59:41.790 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:41.790 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:59:41.790 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:59:41.790 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:41.791 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:41.791 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:41.791 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:59:41.791 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:41.791 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:41.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:41.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T19:59:41.792 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T19:59:41.792 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T19:59:41.792 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T19:59:41.793 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T19:59:41.793 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T19:59:41.793 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T19:59:41.793 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T19:59:41.794 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T19:59:41.794 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T19:59:41.794 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:41.794 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T19:59:41.795 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T19:59:41.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:41.795 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T19:59:41.795 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T19:59:41.795 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T19:59:41.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T19:59:41.796 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T19:59:41Z" 2023-10-18T19:59:41.796 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T19:59:41.797 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659181.586125_229.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T19:59:41.586125Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T19:59:41Z" } 2023-10-18T19:59:41.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659181.586125_229.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T19:59:41.586125Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T19:59:41.798 DEBUG:rgw_multi.tests:log_status={0: '1_1697659181.586125_229.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:59:41.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659181.586125_229.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T19:59:41.799 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T19:59:41.800 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T19:59:41.800 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T19:59:41.800 DEBUG:boto:Method: HEAD 2023-10-18T19:59:41.801 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T19:59:41.801 DEBUG:boto:Data: 2023-10-18T19:59:41.802 DEBUG:boto:Headers: {} 2023-10-18T19:59:41.802 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.803 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.803 DEBUG:boto:Params: {} 2023-10-18T19:59:41.804 DEBUG:boto:Token: None 2023-10-18T19:59:41.804 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/ 2023-10-18T19:59:41.804 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wIGNyrABEhwQBcBKeBADK608St4= 2023-10-18T19:59:41.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wIGNyrABEhwQBcBKeBADK608St4=', 'Content-Length': '0'} 2023-10-18T19:59:41.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006af483ff0a3b5aa9-006530392d-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.808 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T19:59:41.808 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T19:59:41.808 DEBUG:boto:Method: PUT 2023-10-18T19:59:41.809 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T19:59:41.809 DEBUG:boto:Data: 2023-10-18T19:59:41.810 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T19:59:41.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T19:59:41.810 DEBUG:boto:Port: 8000 2023-10-18T19:59:41.811 DEBUG:boto:Params: {} 2023-10-18T19:59:41.811 DEBUG:boto:Token: None 2023-10-18T19:59:41.812 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 19:59:41 GMT /bofrnf-14/obj 2023-10-18T19:59:41.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:drlPQ0HC0K4QEOWlmd280JV050s= 2023-10-18T19:59:41.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 19:59:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:drlPQ0HC0K4QEOWlmd280JV050s='} 2023-10-18T19:59:41.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-amz-request-id', 'tx00000d69de493559ef9c4-006530392d-1112-a1'), ('Date', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T19:59:41.826 DEBUG:rgw_multi.tests:created new version id=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T19:59:41.827 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-14 2023-10-18T19:59:41.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:59:41.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:59:41.828 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:59:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:41.884 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:41.919 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:41.916+0000 7f309f314780 1 waiting for incremental sync to catch up: 2023-10-18T19:59:41.919 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.12.6, , ] 2023-10-18T19:59:41.919 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.15.13, , ] 2023-10-18T19:59:46.920 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:46.918+0000 7f309f314780 1 waiting for incremental sync to catch up: 2023-10-18T19:59:46.920 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.12.6, , ] 2023-10-18T19:59:46.921 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.15.13, , ] 2023-10-18T19:59:51.921 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:51.920+0000 7f309f314780 1 waiting for incremental sync to catch up: 2023-10-18T19:59:51.922 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.12.6, , ] 2023-10-18T19:59:51.922 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.15.13, , ] 2023-10-18T19:59:56.922 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:56.921+0000 7f309f314780 1 bucket sync caught up with source: 2023-10-18T19:59:56.923 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.15.13, , ] 2023-10-18T19:59:56.923 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.15.13, , ] 2023-10-18T19:59:56.923 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:56.921+0000 7f309f314780 0 bucket checkpoint complete 2023-10-18T19:59:56.938 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-14 2023-10-18T19:59:56.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:59:56.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T19:59:56.940 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T19:59:57.002 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T19:59:57.003 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T19:59:57.039 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T19:59:57.038+0000 7f930e155780 1 waiting for incremental sync to catch up: 2023-10-18T19:59:57.040 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.33.6, , ] 2023-10-18T19:59:57.040 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:02.041 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:02.039+0000 7f930e155780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:02.041 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.33.6, , ] 2023-10-18T20:00:02.042 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:07.042 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:07.040+0000 7f930e155780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:07.043 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.33.6, , ] 2023-10-18T20:00:07.043 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:12.043 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:12.042+0000 7f930e155780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:12.044 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.33.6, , ] 2023-10-18T20:00:12.044 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:17.045 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:17.043+0000 7f930e155780 1 bucket sync caught up with source: 2023-10-18T20:00:17.045 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:17.046 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:17.046 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:17.043+0000 7f930e155780 0 bucket checkpoint complete 2023-10-18T20:00:17.061 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-14 zones={a2, a1} 2023-10-18T20:00:17.061 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.062 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.062 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.063 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T20:00:17.063 DEBUG:boto:Data: 2023-10-18T20:00:17.063 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.064 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.065 DEBUG:boto:Params: {} 2023-10-18T20:00:17.065 DEBUG:boto:Token: None 2023-10-18T20:00:17.066 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.066 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:km9kJK1l7UdASlntD96jlApbPTs= 2023-10-18T20:00:17.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:km9kJK1l7UdASlntD96jlApbPTs=', 'Content-Length': '0'} 2023-10-18T20:00:17.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077c5c24ed19a5aef-0065303951-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.070 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.070 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.071 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.071 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T20:00:17.071 DEBUG:boto:Data: 2023-10-18T20:00:17.072 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.072 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.073 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.073 DEBUG:boto:Params: {} 2023-10-18T20:00:17.073 DEBUG:boto:Token: None 2023-10-18T20:00:17.074 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.074 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:km9kJK1l7UdASlntD96jlApbPTs= 2023-10-18T20:00:17.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:km9kJK1l7UdASlntD96jlApbPTs=', 'Content-Length': '0'} 2023-10-18T20:00:17.077 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d197198e67c4b62a-0065303951-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.078 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:00:17.078 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.079 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.079 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.080 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.080 DEBUG:boto:Method: GET 2023-10-18T20:00:17.081 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.081 DEBUG:boto:Data: 2023-10-18T20:00:17.081 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.082 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.083 DEBUG:boto:Params: {} 2023-10-18T20:00:17.083 DEBUG:boto:Token: None 2023-10-18T20:00:17.084 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/?versions 2023-10-18T20:00:17.084 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WD2Sq/zs2MUSGx0gP93TbqHTF9c= 2023-10-18T20:00:17.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WD2Sq/zs2MUSGx0gP93TbqHTF9c=', 'Content-Length': '0'} 2023-10-18T20:00:17.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004925d29affb5836e-0065303951-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.088 DEBUG:boto:b'bofrnf-14999falseobjD1pKGGRpyp.sGsRu5ysoaixV1tu4nFBtrue2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:00:17.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:00:17.089 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:00:17.090 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.090 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.091 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.091 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.092 DEBUG:boto:Method: GET 2023-10-18T20:00:17.092 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.092 DEBUG:boto:Data: 2023-10-18T20:00:17.093 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.093 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.093 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.094 DEBUG:boto:Params: {} 2023-10-18T20:00:17.094 DEBUG:boto:Token: None 2023-10-18T20:00:17.095 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/?versions 2023-10-18T20:00:17.095 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WD2Sq/zs2MUSGx0gP93TbqHTF9c= 2023-10-18T20:00:17.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WD2Sq/zs2MUSGx0gP93TbqHTF9c=', 'Content-Length': '0'} 2023-10-18T20:00:17.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099feaf32d748273e-0065303951-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.099 DEBUG:boto:b'bofrnf-14999falseobjD1pKGGRpyp.sGsRu5ysoaixV1tu4nFBtrue2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:00:17.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:00:17.100 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.100 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.101 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.101 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.102 DEBUG:boto:Method: GET 2023-10-18T20:00:17.102 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.103 DEBUG:boto:Data: 2023-10-18T20:00:17.103 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.103 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.104 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.104 DEBUG:boto:Params: {} 2023-10-18T20:00:17.105 DEBUG:boto:Token: None 2023-10-18T20:00:17.105 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/?versions 2023-10-18T20:00:17.106 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WD2Sq/zs2MUSGx0gP93TbqHTF9c= 2023-10-18T20:00:17.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WD2Sq/zs2MUSGx0gP93TbqHTF9c=', 'Content-Length': '0'} 2023-10-18T20:00:17.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9290f2948a40dbc-0065303951-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.109 DEBUG:boto:b'bofrnf-14999falseobjD1pKGGRpyp.sGsRu5ysoaixV1tu4nFBtrue2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.109 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.110 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.110 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.111 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.111 DEBUG:boto:Method: GET 2023-10-18T20:00:17.111 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:17.112 DEBUG:boto:Data: 2023-10-18T20:00:17.112 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.112 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.113 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.113 DEBUG:boto:Params: {} 2023-10-18T20:00:17.114 DEBUG:boto:Token: None 2023-10-18T20:00:17.114 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/?versions 2023-10-18T20:00:17.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WD2Sq/zs2MUSGx0gP93TbqHTF9c= 2023-10-18T20:00:17.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WD2Sq/zs2MUSGx0gP93TbqHTF9c=', 'Content-Length': '0'} 2023-10-18T20:00:17.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f1c51d0bc0f640a-0065303951-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.158 DEBUG:boto:b'bofrnf-14999falseobjD1pKGGRpyp.sGsRu5ysoaixV1tu4nFBtrue2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-18T19:59:06.074Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:17.159 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.160 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.160 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.161 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.161 DEBUG:boto:Method: GET 2023-10-18T20:00:17.161 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.162 DEBUG:boto:Data: 2023-10-18T20:00:17.162 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.163 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.163 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.163 DEBUG:boto:Params: {} 2023-10-18T20:00:17.164 DEBUG:boto:Token: None 2023-10-18T20:00:17.164 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.165 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs= 2023-10-18T20:00:17.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs=', 'Content-Length': '0'} 2023-10-18T20:00:17.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d293f33c945a9a8-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.168 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.168 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.169 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.169 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.170 DEBUG:boto:Method: GET 2023-10-18T20:00:17.170 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.171 DEBUG:boto:Data: 2023-10-18T20:00:17.171 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.171 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.172 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.172 DEBUG:boto:Params: {} 2023-10-18T20:00:17.173 DEBUG:boto:Token: None 2023-10-18T20:00:17.173 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.174 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs= 2023-10-18T20:00:17.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs=', 'Content-Length': '0'} 2023-10-18T20:00:17.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0c0e8faeefede1c-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.177 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.177 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.177 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.178 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.178 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.179 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.179 DEBUG:boto:Data: 2023-10-18T20:00:17.179 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.180 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.180 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.180 DEBUG:boto:Params: {} 2023-10-18T20:00:17.181 DEBUG:boto:Token: None 2023-10-18T20:00:17.181 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.182 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B7344awGIz8Rk7Z+py8bTgoujnk= 2023-10-18T20:00:17.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B7344awGIz8Rk7Z+py8bTgoujnk=', 'Content-Length': '0'} 2023-10-18T20:00:17.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbde35aec3b7839a-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.184 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.185 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.185 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.185 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.186 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.186 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.187 DEBUG:boto:Data: 2023-10-18T20:00:17.187 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.188 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.188 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.188 DEBUG:boto:Params: {} 2023-10-18T20:00:17.189 DEBUG:boto:Token: None 2023-10-18T20:00:17.189 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.190 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B7344awGIz8Rk7Z+py8bTgoujnk= 2023-10-18T20:00:17.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B7344awGIz8Rk7Z+py8bTgoujnk=', 'Content-Length': '0'} 2023-10-18T20:00:17.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a6e32a367c5a0b4-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:17.193 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.194 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.194 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.195 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.195 DEBUG:boto:Method: GET 2023-10-18T20:00:17.196 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.196 DEBUG:boto:Data: 2023-10-18T20:00:17.196 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.197 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.198 DEBUG:boto:Params: {} 2023-10-18T20:00:17.198 DEBUG:boto:Token: None 2023-10-18T20:00:17.199 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.199 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs= 2023-10-18T20:00:17.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs=', 'Content-Length': '0'} 2023-10-18T20:00:17.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcc964342081aa1b-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.202 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.203 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.203 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.204 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.204 DEBUG:boto:Method: GET 2023-10-18T20:00:17.205 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.205 DEBUG:boto:Data: 2023-10-18T20:00:17.205 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.206 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.206 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.207 DEBUG:boto:Params: {} 2023-10-18T20:00:17.207 DEBUG:boto:Token: None 2023-10-18T20:00:17.208 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.208 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs= 2023-10-18T20:00:17.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs=', 'Content-Length': '0'} 2023-10-18T20:00:17.211 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f03ee74ea28fd0c1-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.211 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.212 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.212 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.212 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.213 DEBUG:boto:Method: GET 2023-10-18T20:00:17.213 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:17.214 DEBUG:boto:Data: 2023-10-18T20:00:17.214 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.214 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.215 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.215 DEBUG:boto:Params: {} 2023-10-18T20:00:17.216 DEBUG:boto:Token: None 2023-10-18T20:00:17.216 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.217 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0= 2023-10-18T20:00:17.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0=', 'Content-Length': '0'} 2023-10-18T20:00:17.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c62dbf64b241e1d-0065303951-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.262 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.263 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.263 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.264 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.264 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.265 DEBUG:boto:Method: GET 2023-10-18T20:00:17.265 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:17.265 DEBUG:boto:Data: 2023-10-18T20:00:17.266 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.266 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.267 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.267 DEBUG:boto:Params: {} 2023-10-18T20:00:17.268 DEBUG:boto:Token: None 2023-10-18T20:00:17.268 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.269 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0= 2023-10-18T20:00:17.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0=', 'Content-Length': '0'} 2023-10-18T20:00:17.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031d627fdfe5f3fd6-0065303951-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.272 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.273 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.273 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.273 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.274 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:17.274 DEBUG:boto:Data: 2023-10-18T20:00:17.275 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.275 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.275 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.276 DEBUG:boto:Params: {} 2023-10-18T20:00:17.276 DEBUG:boto:Token: None 2023-10-18T20:00:17.276 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj 2023-10-18T20:00:17.277 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DHhz+N84QaYKtI1b2nvIzNFBX9k= 2023-10-18T20:00:17.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DHhz+N84QaYKtI1b2nvIzNFBX9k=', 'Content-Length': '0'} 2023-10-18T20:00:17.280 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004c2eac20e793f947-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.281 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.281 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.281 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.282 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:17.282 DEBUG:boto:Data: 2023-10-18T20:00:17.283 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.283 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.283 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.284 DEBUG:boto:Params: {} 2023-10-18T20:00:17.284 DEBUG:boto:Token: None 2023-10-18T20:00:17.285 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj 2023-10-18T20:00:17.285 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DHhz+N84QaYKtI1b2nvIzNFBX9k= 2023-10-18T20:00:17.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DHhz+N84QaYKtI1b2nvIzNFBX9k=', 'Content-Length': '0'} 2023-10-18T20:00:17.288 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb814e3d5e6ce3cc-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:17.289 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.289 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.290 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.290 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.291 DEBUG:boto:Method: GET 2023-10-18T20:00:17.291 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.291 DEBUG:boto:Data: 2023-10-18T20:00:17.292 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.292 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.292 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.293 DEBUG:boto:Params: {} 2023-10-18T20:00:17.293 DEBUG:boto:Token: None 2023-10-18T20:00:17.294 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.294 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs= 2023-10-18T20:00:17.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs=', 'Content-Length': '0'} 2023-10-18T20:00:17.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003037a20f68037bbc-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.297 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.297 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.298 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.298 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.299 DEBUG:boto:Method: GET 2023-10-18T20:00:17.299 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.299 DEBUG:boto:Data: 2023-10-18T20:00:17.300 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.300 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.301 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.301 DEBUG:boto:Params: {} 2023-10-18T20:00:17.302 DEBUG:boto:Token: None 2023-10-18T20:00:17.302 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.303 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs= 2023-10-18T20:00:17.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs=', 'Content-Length': '0'} 2023-10-18T20:00:17.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004eeff01c993b554d-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.306 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.306 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.307 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.307 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.308 DEBUG:boto:Method: GET 2023-10-18T20:00:17.308 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:17.308 DEBUG:boto:Data: 2023-10-18T20:00:17.309 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.309 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.310 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.310 DEBUG:boto:Params: {} 2023-10-18T20:00:17.311 DEBUG:boto:Token: None 2023-10-18T20:00:17.311 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.311 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0= 2023-10-18T20:00:17.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0=', 'Content-Length': '0'} 2023-10-18T20:00:17.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d36b435f807ec174-0065303951-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.315 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.315 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.316 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.316 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.316 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.317 DEBUG:boto:Method: GET 2023-10-18T20:00:17.317 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:17.318 DEBUG:boto:Data: 2023-10-18T20:00:17.318 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.319 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.319 DEBUG:boto:Params: {} 2023-10-18T20:00:17.320 DEBUG:boto:Token: None 2023-10-18T20:00:17.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0= 2023-10-18T20:00:17.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0=', 'Content-Length': '0'} 2023-10-18T20:00:17.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7fd91b4f0de873c-0065303951-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.324 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:17.325 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.325 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.326 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.326 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.327 DEBUG:boto:Method: GET 2023-10-18T20:00:17.327 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.327 DEBUG:boto:Data: 2023-10-18T20:00:17.328 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.328 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.329 DEBUG:boto:Params: {} 2023-10-18T20:00:17.329 DEBUG:boto:Token: None 2023-10-18T20:00:17.330 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTvQ3k0MyVL0Eap/PWRcYsZlegw= 2023-10-18T20:00:17.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTvQ3k0MyVL0Eap/PWRcYsZlegw=', 'Content-Length': '0'} 2023-10-18T20:00:17.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036797b2d69269ed3-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.333 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.333 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.334 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.334 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.335 DEBUG:boto:Method: GET 2023-10-18T20:00:17.335 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.335 DEBUG:boto:Data: 2023-10-18T20:00:17.336 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.336 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.336 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.337 DEBUG:boto:Params: {} 2023-10-18T20:00:17.337 DEBUG:boto:Token: None 2023-10-18T20:00:17.338 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.338 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTvQ3k0MyVL0Eap/PWRcYsZlegw= 2023-10-18T20:00:17.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTvQ3k0MyVL0Eap/PWRcYsZlegw=', 'Content-Length': '0'} 2023-10-18T20:00:17.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd57bc9fc0e587b8-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.341 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.342 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.342 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.342 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.343 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.343 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.343 DEBUG:boto:Data: 2023-10-18T20:00:17.344 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.344 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.345 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.345 DEBUG:boto:Params: {} 2023-10-18T20:00:17.345 DEBUG:boto:Token: None 2023-10-18T20:00:17.346 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.346 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4H+pR0tAgVLGb3lnj3Vqv9VxT1k= 2023-10-18T20:00:17.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4H+pR0tAgVLGb3lnj3Vqv9VxT1k=', 'Content-Length': '0'} 2023-10-18T20:00:17.349 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd7c7437b575b3dd-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.349 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.350 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.350 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.350 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.351 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.351 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.352 DEBUG:boto:Data: 2023-10-18T20:00:17.352 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.352 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.353 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.353 DEBUG:boto:Params: {} 2023-10-18T20:00:17.354 DEBUG:boto:Token: None 2023-10-18T20:00:17.354 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.355 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4H+pR0tAgVLGb3lnj3Vqv9VxT1k= 2023-10-18T20:00:17.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4H+pR0tAgVLGb3lnj3Vqv9VxT1k=', 'Content-Length': '0'} 2023-10-18T20:00:17.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b770bad3b63346a2-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:17.358 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.358 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.359 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.359 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.360 DEBUG:boto:Method: GET 2023-10-18T20:00:17.360 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.361 DEBUG:boto:Data: 2023-10-18T20:00:17.361 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.362 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.362 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.363 DEBUG:boto:Params: {} 2023-10-18T20:00:17.363 DEBUG:boto:Token: None 2023-10-18T20:00:17.364 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTvQ3k0MyVL0Eap/PWRcYsZlegw= 2023-10-18T20:00:17.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTvQ3k0MyVL0Eap/PWRcYsZlegw=', 'Content-Length': '0'} 2023-10-18T20:00:17.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bfdda149dfa37d8-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.367 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.367 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.368 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.368 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.369 DEBUG:boto:Method: GET 2023-10-18T20:00:17.369 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.370 DEBUG:boto:Data: 2023-10-18T20:00:17.370 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.371 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.371 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.372 DEBUG:boto:Params: {} 2023-10-18T20:00:17.372 DEBUG:boto:Token: None 2023-10-18T20:00:17.373 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:17.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTvQ3k0MyVL0Eap/PWRcYsZlegw= 2023-10-18T20:00:17.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTvQ3k0MyVL0Eap/PWRcYsZlegw=', 'Content-Length': '0'} 2023-10-18T20:00:17.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8017b26d2760cbd-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.377 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.377 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.377 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.378 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.378 DEBUG:boto:Method: GET 2023-10-18T20:00:17.379 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:17.379 DEBUG:boto:Data: 2023-10-18T20:00:17.380 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.380 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.381 DEBUG:boto:Params: {} 2023-10-18T20:00:17.381 DEBUG:boto:Token: None 2023-10-18T20:00:17.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0= 2023-10-18T20:00:17.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0=', 'Content-Length': '0'} 2023-10-18T20:00:17.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fbdc40448295902-0065303951-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.426 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.427 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.428 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.428 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.429 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.429 DEBUG:boto:Method: GET 2023-10-18T20:00:17.429 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:17.430 DEBUG:boto:Data: 2023-10-18T20:00:17.430 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.431 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.431 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.432 DEBUG:boto:Params: {} 2023-10-18T20:00:17.432 DEBUG:boto:Token: None 2023-10-18T20:00:17.433 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.433 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0= 2023-10-18T20:00:17.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0=', 'Content-Length': '0'} 2023-10-18T20:00:17.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccc3cdcf62ee0f01-0065303951-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.437 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.437 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.438 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.438 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.439 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:17.439 DEBUG:boto:Data: 2023-10-18T20:00:17.439 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.440 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.440 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.441 DEBUG:boto:Params: {} 2023-10-18T20:00:17.441 DEBUG:boto:Token: None 2023-10-18T20:00:17.442 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj 2023-10-18T20:00:17.442 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DHhz+N84QaYKtI1b2nvIzNFBX9k= 2023-10-18T20:00:17.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DHhz+N84QaYKtI1b2nvIzNFBX9k=', 'Content-Length': '0'} 2023-10-18T20:00:17.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aaa33b28da77d112-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.446 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.446 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.447 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.447 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:17.447 DEBUG:boto:Data: 2023-10-18T20:00:17.448 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.448 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.449 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.449 DEBUG:boto:Params: {} 2023-10-18T20:00:17.449 DEBUG:boto:Token: None 2023-10-18T20:00:17.450 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj 2023-10-18T20:00:17.450 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DHhz+N84QaYKtI1b2nvIzNFBX9k= 2023-10-18T20:00:17.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DHhz+N84QaYKtI1b2nvIzNFBX9k=', 'Content-Length': '0'} 2023-10-18T20:00:17.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7a23b1699256825-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:17.454 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.455 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.455 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.456 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.456 DEBUG:boto:Method: GET 2023-10-18T20:00:17.456 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.457 DEBUG:boto:Data: 2023-10-18T20:00:17.457 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.458 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.458 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.459 DEBUG:boto:Params: {} 2023-10-18T20:00:17.459 DEBUG:boto:Token: None 2023-10-18T20:00:17.460 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs= 2023-10-18T20:00:17.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs=', 'Content-Length': '0'} 2023-10-18T20:00:17.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c50584aec699e06-0065303951-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.463 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.464 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.464 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.464 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.465 DEBUG:boto:Method: GET 2023-10-18T20:00:17.465 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.466 DEBUG:boto:Data: 2023-10-18T20:00:17.466 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.466 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.467 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.467 DEBUG:boto:Params: {} 2023-10-18T20:00:17.468 DEBUG:boto:Token: None 2023-10-18T20:00:17.468 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:17.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs= 2023-10-18T20:00:17.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGRXfOMmUlg+0Q5eyxd7ykmrRDs=', 'Content-Length': '0'} 2023-10-18T20:00:17.471 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a97d478c19f6e08-0065303951-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.472 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.472 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.473 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.473 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.474 DEBUG:boto:Method: GET 2023-10-18T20:00:17.474 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:17.475 DEBUG:boto:Data: 2023-10-18T20:00:17.475 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.475 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:17.476 DEBUG:boto:Port: 8001 2023-10-18T20:00:17.476 DEBUG:boto:Params: {} 2023-10-18T20:00:17.477 DEBUG:boto:Token: None 2023-10-18T20:00:17.477 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.478 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0= 2023-10-18T20:00:17.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0=', 'Content-Length': '0'} 2023-10-18T20:00:17.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076fd154c67100fe8-0065303951-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.482 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.482 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.483 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.483 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.484 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:17.484 DEBUG:boto:Method: GET 2023-10-18T20:00:17.485 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:17.485 DEBUG:boto:Data: 2023-10-18T20:00:17.485 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.486 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.486 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.487 DEBUG:boto:Params: {} 2023-10-18T20:00:17.487 DEBUG:boto:Token: None 2023-10-18T20:00:17.488 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.488 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0= 2023-10-18T20:00:17.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gudIQsXf6EDT3PNBE1akpvA1VK0=', 'Content-Length': '0'} 2023-10-18T20:00:17.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029eabc03bc10d1cd-0065303951-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.492 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:17.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-14 zones={a2, a1} 2023-10-18T20:00:17.493 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.493 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.494 DEBUG:boto:path=/bofrnf-14/?versioning 2023-10-18T20:00:17.494 DEBUG:boto:auth_path=/bofrnf-14/?versioning 2023-10-18T20:00:17.494 DEBUG:boto:Method: PUT 2023-10-18T20:00:17.495 DEBUG:boto:Path: /bofrnf-14/?versioning 2023-10-18T20:00:17.495 DEBUG:boto:Data: Suspended Disabled 2023-10-18T20:00:17.496 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.496 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.497 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.497 DEBUG:boto:Params: {} 2023-10-18T20:00:17.498 DEBUG:boto:Token: None 2023-10-18T20:00:17.498 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/?versioning 2023-10-18T20:00:17.498 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GOytx3cQNj2I3GIvCWIAndP92gs= 2023-10-18T20:00:17.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GOytx3cQNj2I3GIvCWIAndP92gs=', 'Content-Length': '230'} 2023-10-18T20:00:17.505 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001765c9cf59b1de21-0065303951-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:00:17.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:00:17.506 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:00:17.563 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:00:17.563 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:00:17.600 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:00:17.611 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:00:17.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:17.612 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659217.503470_236.1", 2023-10-18T20:00:17.612 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:00:17.503470Z" 2023-10-18T20:00:17.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:17.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:17.613 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:00:17.613 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:00:17.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:17.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:17.614 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:00:17.614 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:00:17.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:17.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:17.615 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:00:17.615 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:00:17.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:17.615 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:00:17.616 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659217.503470_236.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:00:17.617 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:00:17.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:00:17.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:00:17.618 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:00:17.668 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:00:17.668 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:00:17.696 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:17.695+0000 7fb12d655780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:00:17.697 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:17.695+0000 7fb12d655780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:00:17.709 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:00:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:00:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:00:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:00:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:00:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:00:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:00:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:00:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:00:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:00:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659217.503470_236.1", 2023-10-18T20:00:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:00:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:00:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:00:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:00:17.503470Z", 2023-10-18T20:00:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:00:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:00:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:00:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:00:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:00:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:00:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:00:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:00:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:00:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:00:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:00:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:00:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:00:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:00:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:00:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:00:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:00:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:00:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:00:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:00:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:00:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:00:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:00:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:00:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:00:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:00:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:00:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:00:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:00:17Z" 2023-10-18T20:00:17.723 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:00:17.723 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659217.503470_236.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:00:17.503470Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:00:17Z" } 2023-10-18T20:00:17.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659217.503470_236.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:00:17.503470Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:00:17.724 DEBUG:rgw_multi.tests:log_status={0: '1_1697659217.503470_236.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:00:17.725 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659217.503470_236.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:00:17.725 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:00:17.726 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:17.726 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:17.726 DEBUG:boto:Method: HEAD 2023-10-18T20:00:17.727 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T20:00:17.727 DEBUG:boto:Data: 2023-10-18T20:00:17.728 DEBUG:boto:Headers: {} 2023-10-18T20:00:17.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.728 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.729 DEBUG:boto:Params: {} 2023-10-18T20:00:17.729 DEBUG:boto:Token: None 2023-10-18T20:00:17.730 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/ 2023-10-18T20:00:17.730 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:km9kJK1l7UdASlntD96jlApbPTs= 2023-10-18T20:00:17.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:km9kJK1l7UdASlntD96jlApbPTs=', 'Content-Length': '0'} 2023-10-18T20:00:17.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083d00a020f443d53-0065303951-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.734 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:17.734 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:17.734 DEBUG:boto:Method: PUT 2023-10-18T20:00:17.735 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:17.735 DEBUG:boto:Data: 2023-10-18T20:00:17.736 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T20:00:17.736 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:17.737 DEBUG:boto:Port: 8000 2023-10-18T20:00:17.737 DEBUG:boto:Params: {} 2023-10-18T20:00:17.737 DEBUG:boto:Token: None 2023-10-18T20:00:17.738 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 20:00:17 GMT /bofrnf-14/obj 2023-10-18T20:00:17.738 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:k6BRnFpTIjesr2ywujdvI9jp/3E= 2023-10-18T20:00:17.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:00:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:k6BRnFpTIjesr2ywujdvI9jp/3E='} 2023-10-18T20:00:17.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5fbb317676a99be-0065303951-1112-a1'), ('Date', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:17.750 DEBUG:rgw_multi.tests:created null version id=None 2023-10-18T20:00:17.751 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-14 2023-10-18T20:00:17.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:00:17.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:00:17.752 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:00:17.808 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:00:17.808 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:00:17.841 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:17.840+0000 7f2061544780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:17.841 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.15.13, , ] 2023-10-18T20:00:17.842 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.19.18, , ] 2023-10-18T20:00:22.843 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:22.840+0000 7f2061544780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:22.843 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.15.13, , ] 2023-10-18T20:00:22.843 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.19.18, , ] 2023-10-18T20:00:27.844 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:27.841+0000 7f2061544780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:27.844 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.15.13, , ] 2023-10-18T20:00:27.845 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.19.18, , ] 2023-10-18T20:00:32.845 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:32.843+0000 7f2061544780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:32.846 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.15.13, , ] 2023-10-18T20:00:32.846 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.19.18, , ] 2023-10-18T20:00:37.846 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:37.845+0000 7f2061544780 1 bucket sync caught up with source: 2023-10-18T20:00:37.847 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000005.19.18, , ] 2023-10-18T20:00:37.847 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.19.18, , ] 2023-10-18T20:00:37.847 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:37.845+0000 7f2061544780 0 bucket checkpoint complete 2023-10-18T20:00:37.862 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-14 2023-10-18T20:00:37.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:00:37.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:00:37.864 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:00:37.925 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:00:37.925 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:00:37.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:37.960+0000 7f923b44d780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:37.962 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:37.962 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.42.18, , ] 2023-10-18T20:00:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:42.961+0000 7f923b44d780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:42.963 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:42.964 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.42.18, , ] 2023-10-18T20:00:47.964 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:47.962+0000 7f923b44d780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:47.964 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:47.965 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.42.18, , ] 2023-10-18T20:00:52.965 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:52.964+0000 7f923b44d780 1 waiting for incremental sync to catch up: 2023-10-18T20:00:52.966 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.37.13, , ] 2023-10-18T20:00:52.966 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.42.18, , ] 2023-10-18T20:00:57.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:57.965+0000 7f923b44d780 1 bucket sync caught up with source: 2023-10-18T20:00:57.967 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000005.42.18, , ] 2023-10-18T20:00:57.967 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000005.42.18, , ] 2023-10-18T20:00:57.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:57.965+0000 7f923b44d780 0 bucket checkpoint complete 2023-10-18T20:00:57.982 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-14 zones={a2, a1} 2023-10-18T20:00:57.983 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:57.984 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:57.984 DEBUG:boto:Method: HEAD 2023-10-18T20:00:57.985 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T20:00:57.986 DEBUG:boto:Data: 2023-10-18T20:00:57.986 DEBUG:boto:Headers: {} 2023-10-18T20:00:57.987 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:57.987 DEBUG:boto:Port: 8001 2023-10-18T20:00:57.988 DEBUG:boto:Params: {} 2023-10-18T20:00:57.989 DEBUG:boto:Token: None 2023-10-18T20:00:57.989 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:57 GMT /bofrnf-14/ 2023-10-18T20:00:57.990 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:znr9U7R/sreEcVLi2GT0kON6i/c= 2023-10-18T20:00:57.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:znr9U7R/sreEcVLi2GT0kON6i/c=', 'Content-Length': '0'} 2023-10-18T20:00:57.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6dae878a0404934-0065303979-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:57.994 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:57.994 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:57.994 DEBUG:boto:Method: HEAD 2023-10-18T20:00:57.995 DEBUG:boto:Path: /bofrnf-14/ 2023-10-18T20:00:57.995 DEBUG:boto:Data: 2023-10-18T20:00:57.996 DEBUG:boto:Headers: {} 2023-10-18T20:00:57.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:57.997 DEBUG:boto:Port: 8000 2023-10-18T20:00:57.997 DEBUG:boto:Params: {} 2023-10-18T20:00:57.998 DEBUG:boto:Token: None 2023-10-18T20:00:57.998 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:57 GMT /bofrnf-14/ 2023-10-18T20:00:57.998 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:znr9U7R/sreEcVLi2GT0kON6i/c= 2023-10-18T20:00:57.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:znr9U7R/sreEcVLi2GT0kON6i/c=', 'Content-Length': '0'} 2023-10-18T20:00:58.001 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac6e56524e13a7c2-0065303979-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.002 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:00:58.002 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.003 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.003 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.003 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.004 DEBUG:boto:Method: GET 2023-10-18T20:00:58.004 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.005 DEBUG:boto:Data: 2023-10-18T20:00:58.005 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.006 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.006 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.006 DEBUG:boto:Params: {} 2023-10-18T20:00:58.007 DEBUG:boto:Token: None 2023-10-18T20:00:58.007 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/?versions 2023-10-18T20:00:58.008 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1dGyaC4VQDqBRN9zAn3DSaSFoBE= 2023-10-18T20:00:58.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1dGyaC4VQDqBRN9zAn3DSaSFoBE=', 'Content-Length': '0'} 2023-10-18T20:00:58.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f4a470b6dd4c3c0-006530397a-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.011 DEBUG:boto:b'bofrnf-14999falseobjnulltrue2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD1pKGGRpyp.sGsRu5ysoaixV1tu4nFBfalse2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:00:58.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:00:58.013 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:00:58.013 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.014 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.014 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.015 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.015 DEBUG:boto:Method: GET 2023-10-18T20:00:58.016 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.016 DEBUG:boto:Data: 2023-10-18T20:00:58.016 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.017 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.017 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.018 DEBUG:boto:Params: {} 2023-10-18T20:00:58.018 DEBUG:boto:Token: None 2023-10-18T20:00:58.019 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/?versions 2023-10-18T20:00:58.019 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1dGyaC4VQDqBRN9zAn3DSaSFoBE= 2023-10-18T20:00:58.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1dGyaC4VQDqBRN9zAn3DSaSFoBE=', 'Content-Length': '0'} 2023-10-18T20:00:58.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dcad4b0677a39ea-006530397a-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.023 DEBUG:boto:b'bofrnf-14999falseobjnulltrue2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD1pKGGRpyp.sGsRu5ysoaixV1tu4nFBfalse2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:00:58.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:00:58.024 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.025 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.025 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.026 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.026 DEBUG:boto:Method: GET 2023-10-18T20:00:58.026 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.027 DEBUG:boto:Data: 2023-10-18T20:00:58.027 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.028 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.028 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.028 DEBUG:boto:Params: {} 2023-10-18T20:00:58.029 DEBUG:boto:Token: None 2023-10-18T20:00:58.029 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/?versions 2023-10-18T20:00:58.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1dGyaC4VQDqBRN9zAn3DSaSFoBE= 2023-10-18T20:00:58.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1dGyaC4VQDqBRN9zAn3DSaSFoBE=', 'Content-Length': '0'} 2023-10-18T20:00:58.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d50ce3edae6017d2-006530397a-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.074 DEBUG:boto:b'bofrnf-14999falseobjnulltrue2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD1pKGGRpyp.sGsRu5ysoaixV1tu4nFBfalse2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.076 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.077 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.078 DEBUG:boto:path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.079 DEBUG:boto:auth_path=/bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.079 DEBUG:boto:Method: GET 2023-10-18T20:00:58.080 DEBUG:boto:Path: /bofrnf-14/?versions&max-keys=999 2023-10-18T20:00:58.081 DEBUG:boto:Data: 2023-10-18T20:00:58.081 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.083 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.083 DEBUG:boto:Params: {} 2023-10-18T20:00:58.084 DEBUG:boto:Token: None 2023-10-18T20:00:58.085 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/?versions 2023-10-18T20:00:58.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1dGyaC4VQDqBRN9zAn3DSaSFoBE= 2023-10-18T20:00:58.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1dGyaC4VQDqBRN9zAn3DSaSFoBE=', 'Content-Length': '0'} 2023-10-18T20:00:58.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079350b6aae2ed280-006530397a-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.090 DEBUG:boto:b'bofrnf-14999falseobjnulltrue2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD1pKGGRpyp.sGsRu5ysoaixV1tu4nFBfalse2023-10-18T19:59:41.814Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:58.091 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.091 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.092 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.092 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.093 DEBUG:boto:Method: GET 2023-10-18T20:00:58.093 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.094 DEBUG:boto:Data: 2023-10-18T20:00:58.094 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.094 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.095 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.095 DEBUG:boto:Params: {} 2023-10-18T20:00:58.096 DEBUG:boto:Token: None 2023-10-18T20:00:58.096 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.097 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wbRVLJ2BlPxshmwi6dwCIUBqtwU= 2023-10-18T20:00:58.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wbRVLJ2BlPxshmwi6dwCIUBqtwU=', 'Content-Length': '0'} 2023-10-18T20:00:58.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bdc7495fd6325a5-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.100 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.101 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.101 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.101 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.102 DEBUG:boto:Method: GET 2023-10-18T20:00:58.102 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.103 DEBUG:boto:Data: 2023-10-18T20:00:58.103 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.104 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.104 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.104 DEBUG:boto:Params: {} 2023-10-18T20:00:58.105 DEBUG:boto:Token: None 2023-10-18T20:00:58.105 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.106 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wbRVLJ2BlPxshmwi6dwCIUBqtwU= 2023-10-18T20:00:58.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wbRVLJ2BlPxshmwi6dwCIUBqtwU=', 'Content-Length': '0'} 2023-10-18T20:00:58.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a0060f1b5b93ede-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.110 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.110 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.111 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.112 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.113 DEBUG:boto:Method: HEAD 2023-10-18T20:00:58.113 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.114 DEBUG:boto:Data: 2023-10-18T20:00:58.115 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.116 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.116 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.117 DEBUG:boto:Params: {} 2023-10-18T20:00:58.118 DEBUG:boto:Token: None 2023-10-18T20:00:58.119 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.120 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TXI0yYEj1oxzPSGFSskKsYyJcKA= 2023-10-18T20:00:58.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TXI0yYEj1oxzPSGFSskKsYyJcKA=', 'Content-Length': '0'} 2023-10-18T20:00:58.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e6d7ca6ba6f9447e-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.124 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.125 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.125 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.126 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.127 DEBUG:boto:Method: HEAD 2023-10-18T20:00:58.128 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.128 DEBUG:boto:Data: 2023-10-18T20:00:58.129 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.130 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.131 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.132 DEBUG:boto:Params: {} 2023-10-18T20:00:58.132 DEBUG:boto:Token: None 2023-10-18T20:00:58.133 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.134 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TXI0yYEj1oxzPSGFSskKsYyJcKA= 2023-10-18T20:00:58.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TXI0yYEj1oxzPSGFSskKsYyJcKA=', 'Content-Length': '0'} 2023-10-18T20:00:58.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6df00a6f9cf74a2-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:58.139 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.140 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.140 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.141 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.142 DEBUG:boto:Method: GET 2023-10-18T20:00:58.142 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.143 DEBUG:boto:Data: 2023-10-18T20:00:58.144 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.144 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.145 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.145 DEBUG:boto:Params: {} 2023-10-18T20:00:58.146 DEBUG:boto:Token: None 2023-10-18T20:00:58.146 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.147 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wbRVLJ2BlPxshmwi6dwCIUBqtwU= 2023-10-18T20:00:58.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wbRVLJ2BlPxshmwi6dwCIUBqtwU=', 'Content-Length': '0'} 2023-10-18T20:00:58.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b5da6e7757addcc4-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.151 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.151 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.152 DEBUG:boto:path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.152 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.153 DEBUG:boto:Method: GET 2023-10-18T20:00:58.153 DEBUG:boto:Path: /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.154 DEBUG:boto:Data: 2023-10-18T20:00:58.154 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.154 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.155 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.155 DEBUG:boto:Params: {} 2023-10-18T20:00:58.156 DEBUG:boto:Token: None 2023-10-18T20:00:58.156 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=null 2023-10-18T20:00:58.157 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wbRVLJ2BlPxshmwi6dwCIUBqtwU= 2023-10-18T20:00:58.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wbRVLJ2BlPxshmwi6dwCIUBqtwU=', 'Content-Length': '0'} 2023-10-18T20:00:58.159 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054f589633448ff01-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.160 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.161 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.161 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.162 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.162 DEBUG:boto:Method: GET 2023-10-18T20:00:58.162 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:58.163 DEBUG:boto:Data: 2023-10-18T20:00:58.164 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.164 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.165 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.165 DEBUG:boto:Params: {} 2023-10-18T20:00:58.166 DEBUG:boto:Token: None 2023-10-18T20:00:58.166 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/ 2023-10-18T20:00:58.167 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4= 2023-10-18T20:00:58.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4=', 'Content-Length': '0'} 2023-10-18T20:00:58.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fa352f178a852e1-006530397a-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.170 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.171 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.172 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.173 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.174 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.174 DEBUG:boto:Method: GET 2023-10-18T20:00:58.175 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:58.176 DEBUG:boto:Data: 2023-10-18T20:00:58.176 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.177 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.177 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.178 DEBUG:boto:Params: {} 2023-10-18T20:00:58.179 DEBUG:boto:Token: None 2023-10-18T20:00:58.179 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/ 2023-10-18T20:00:58.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4= 2023-10-18T20:00:58.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4=', 'Content-Length': '0'} 2023-10-18T20:00:58.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6b1946e2e4d429f-006530397a-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.184 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.185 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.185 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.186 DEBUG:boto:Method: HEAD 2023-10-18T20:00:58.186 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:58.187 DEBUG:boto:Data: 2023-10-18T20:00:58.187 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.188 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.188 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.189 DEBUG:boto:Params: {} 2023-10-18T20:00:58.189 DEBUG:boto:Token: None 2023-10-18T20:00:58.190 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj 2023-10-18T20:00:58.190 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RQHZ1anymjOQt8vTgVuRKJ7u48A= 2023-10-18T20:00:58.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RQHZ1anymjOQt8vTgVuRKJ7u48A=', 'Content-Length': '0'} 2023-10-18T20:00:58.193 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000922d86678cf4c26-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.193 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.194 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.194 DEBUG:boto:Method: HEAD 2023-10-18T20:00:58.195 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:58.195 DEBUG:boto:Data: 2023-10-18T20:00:58.195 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.196 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.196 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.197 DEBUG:boto:Params: {} 2023-10-18T20:00:58.197 DEBUG:boto:Token: None 2023-10-18T20:00:58.198 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj 2023-10-18T20:00:58.198 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RQHZ1anymjOQt8vTgVuRKJ7u48A= 2023-10-18T20:00:58.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RQHZ1anymjOQt8vTgVuRKJ7u48A=', 'Content-Length': '0'} 2023-10-18T20:00:58.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e59515a927317b7a-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:58.202 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.202 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.203 DEBUG:boto:Method: GET 2023-10-18T20:00:58.203 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:58.203 DEBUG:boto:Data: 2023-10-18T20:00:58.204 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.204 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.205 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.205 DEBUG:boto:Params: {} 2023-10-18T20:00:58.206 DEBUG:boto:Token: None 2023-10-18T20:00:58.206 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj 2023-10-18T20:00:58.207 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bddNYS+7xj7P/P0cPZ+3XAfBzBE= 2023-10-18T20:00:58.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bddNYS+7xj7P/P0cPZ+3XAfBzBE=', 'Content-Length': '0'} 2023-10-18T20:00:58.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f9547ab44d39a78f-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.211 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.211 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.212 DEBUG:boto:Method: GET 2023-10-18T20:00:58.212 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:58.213 DEBUG:boto:Data: 2023-10-18T20:00:58.213 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.214 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.214 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.215 DEBUG:boto:Params: {} 2023-10-18T20:00:58.215 DEBUG:boto:Token: None 2023-10-18T20:00:58.216 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj 2023-10-18T20:00:58.216 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bddNYS+7xj7P/P0cPZ+3XAfBzBE= 2023-10-18T20:00:58.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bddNYS+7xj7P/P0cPZ+3XAfBzBE=', 'Content-Length': '0'} 2023-10-18T20:00:58.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092f54fa582ad189f-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.220 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.221 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.221 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.222 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.223 DEBUG:boto:Method: GET 2023-10-18T20:00:58.223 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:58.224 DEBUG:boto:Data: 2023-10-18T20:00:58.224 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.225 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.225 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.226 DEBUG:boto:Params: {} 2023-10-18T20:00:58.226 DEBUG:boto:Token: None 2023-10-18T20:00:58.227 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/ 2023-10-18T20:00:58.227 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4= 2023-10-18T20:00:58.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4=', 'Content-Length': '0'} 2023-10-18T20:00:58.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f3611b0f0da893-006530397a-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.231 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.232 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.232 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.232 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.233 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.233 DEBUG:boto:Method: GET 2023-10-18T20:00:58.234 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:58.234 DEBUG:boto:Data: 2023-10-18T20:00:58.235 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.235 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.236 DEBUG:boto:Params: {} 2023-10-18T20:00:58.236 DEBUG:boto:Token: None 2023-10-18T20:00:58.237 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/ 2023-10-18T20:00:58.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4= 2023-10-18T20:00:58.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4=', 'Content-Length': '0'} 2023-10-18T20:00:58.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c9de84c9385f2db-006530397a-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.287 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:58.288 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.288 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.289 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.290 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.290 DEBUG:boto:Method: GET 2023-10-18T20:00:58.291 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.291 DEBUG:boto:Data: 2023-10-18T20:00:58.292 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.292 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.292 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.293 DEBUG:boto:Params: {} 2023-10-18T20:00:58.293 DEBUG:boto:Token: None 2023-10-18T20:00:58.294 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.294 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PVEu0XZVLhYbLCM6k+w8U2TOwn4= 2023-10-18T20:00:58.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PVEu0XZVLhYbLCM6k+w8U2TOwn4=', 'Content-Length': '0'} 2023-10-18T20:00:58.297 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057bcc760769963fc-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.298 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.298 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.299 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.299 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.300 DEBUG:boto:Method: GET 2023-10-18T20:00:58.300 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.301 DEBUG:boto:Data: 2023-10-18T20:00:58.301 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.301 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.302 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.302 DEBUG:boto:Params: {} 2023-10-18T20:00:58.303 DEBUG:boto:Token: None 2023-10-18T20:00:58.303 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PVEu0XZVLhYbLCM6k+w8U2TOwn4= 2023-10-18T20:00:58.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PVEu0XZVLhYbLCM6k+w8U2TOwn4=', 'Content-Length': '0'} 2023-10-18T20:00:58.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd274dae0a1ff1a1-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.307 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.307 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.308 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.308 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.309 DEBUG:boto:Method: HEAD 2023-10-18T20:00:58.309 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.309 DEBUG:boto:Data: 2023-10-18T20:00:58.310 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.311 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.311 DEBUG:boto:Params: {} 2023-10-18T20:00:58.312 DEBUG:boto:Token: None 2023-10-18T20:00:58.312 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uK2Xgn5GnA9tY15KVXzcqmu+hxE= 2023-10-18T20:00:58.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uK2Xgn5GnA9tY15KVXzcqmu+hxE=', 'Content-Length': '0'} 2023-10-18T20:00:58.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000029d020f99709e2ce-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.316 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.316 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.317 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.317 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.317 DEBUG:boto:Method: HEAD 2023-10-18T20:00:58.318 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.318 DEBUG:boto:Data: 2023-10-18T20:00:58.319 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.319 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.320 DEBUG:boto:Params: {} 2023-10-18T20:00:58.320 DEBUG:boto:Token: None 2023-10-18T20:00:58.321 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uK2Xgn5GnA9tY15KVXzcqmu+hxE= 2023-10-18T20:00:58.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uK2Xgn5GnA9tY15KVXzcqmu+hxE=', 'Content-Length': '0'} 2023-10-18T20:00:58.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000124211a79622254b-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:58.325 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.326 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.326 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.327 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.330 DEBUG:boto:Method: GET 2023-10-18T20:00:58.331 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.331 DEBUG:boto:Data: 2023-10-18T20:00:58.332 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.332 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.333 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.333 DEBUG:boto:Params: {} 2023-10-18T20:00:58.333 DEBUG:boto:Token: None 2023-10-18T20:00:58.334 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.335 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PVEu0XZVLhYbLCM6k+w8U2TOwn4= 2023-10-18T20:00:58.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PVEu0XZVLhYbLCM6k+w8U2TOwn4=', 'Content-Length': '0'} 2023-10-18T20:00:58.337 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000995fddc364338de0-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.338 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.339 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.340 DEBUG:boto:path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.340 DEBUG:boto:auth_path=/bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.341 DEBUG:boto:Method: GET 2023-10-18T20:00:58.341 DEBUG:boto:Path: /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.342 DEBUG:boto:Data: 2023-10-18T20:00:58.342 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.343 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.343 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.343 DEBUG:boto:Params: {} 2023-10-18T20:00:58.344 DEBUG:boto:Token: None 2023-10-18T20:00:58.344 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj?versionId=D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB 2023-10-18T20:00:58.345 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PVEu0XZVLhYbLCM6k+w8U2TOwn4= 2023-10-18T20:00:58.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PVEu0XZVLhYbLCM6k+w8U2TOwn4=', 'Content-Length': '0'} 2023-10-18T20:00:58.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 19:59:41 GMT'), ('x-amz-version-id', 'D1pKGGRpyp.sGsRu5ysoaixV1tu4nFB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007e01092ef404d38-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.348 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.349 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.349 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.350 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.350 DEBUG:boto:Method: GET 2023-10-18T20:00:58.350 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:58.351 DEBUG:boto:Data: 2023-10-18T20:00:58.351 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.352 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.352 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.353 DEBUG:boto:Params: {} 2023-10-18T20:00:58.353 DEBUG:boto:Token: None 2023-10-18T20:00:58.354 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/ 2023-10-18T20:00:58.354 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4= 2023-10-18T20:00:58.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4=', 'Content-Length': '0'} 2023-10-18T20:00:58.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c1a0f51155f90f2-006530397a-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.398 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.399 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.400 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.400 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.400 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.401 DEBUG:boto:Method: GET 2023-10-18T20:00:58.401 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:58.402 DEBUG:boto:Data: 2023-10-18T20:00:58.402 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.403 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.403 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.403 DEBUG:boto:Params: {} 2023-10-18T20:00:58.404 DEBUG:boto:Token: None 2023-10-18T20:00:58.404 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/ 2023-10-18T20:00:58.405 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4= 2023-10-18T20:00:58.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4=', 'Content-Length': '0'} 2023-10-18T20:00:58.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000984174b181a4f786-006530397a-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.408 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.409 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.410 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.410 DEBUG:boto:Method: HEAD 2023-10-18T20:00:58.410 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:58.411 DEBUG:boto:Data: 2023-10-18T20:00:58.411 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.411 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.412 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.412 DEBUG:boto:Params: {} 2023-10-18T20:00:58.413 DEBUG:boto:Token: None 2023-10-18T20:00:58.413 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj 2023-10-18T20:00:58.413 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RQHZ1anymjOQt8vTgVuRKJ7u48A= 2023-10-18T20:00:58.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RQHZ1anymjOQt8vTgVuRKJ7u48A=', 'Content-Length': '0'} 2023-10-18T20:00:58.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000467c405674d82b66-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.416 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.417 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.417 DEBUG:boto:Method: HEAD 2023-10-18T20:00:58.418 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:58.418 DEBUG:boto:Data: 2023-10-18T20:00:58.419 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.419 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.419 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.420 DEBUG:boto:Params: {} 2023-10-18T20:00:58.420 DEBUG:boto:Token: None 2023-10-18T20:00:58.421 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj 2023-10-18T20:00:58.421 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RQHZ1anymjOQt8vTgVuRKJ7u48A= 2023-10-18T20:00:58.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RQHZ1anymjOQt8vTgVuRKJ7u48A=', 'Content-Length': '0'} 2023-10-18T20:00:58.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051c9aab256a60996-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:00:58.424 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.425 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.425 DEBUG:boto:Method: GET 2023-10-18T20:00:58.426 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:58.426 DEBUG:boto:Data: 2023-10-18T20:00:58.427 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.427 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.427 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.428 DEBUG:boto:Params: {} 2023-10-18T20:00:58.428 DEBUG:boto:Token: None 2023-10-18T20:00:58.429 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj 2023-10-18T20:00:58.429 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bddNYS+7xj7P/P0cPZ+3XAfBzBE= 2023-10-18T20:00:58.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bddNYS+7xj7P/P0cPZ+3XAfBzBE=', 'Content-Length': '0'} 2023-10-18T20:00:58.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3f5a184b6aa853a-006530397a-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.432 DEBUG:boto:path=/bofrnf-14/obj 2023-10-18T20:00:58.433 DEBUG:boto:auth_path=/bofrnf-14/obj 2023-10-18T20:00:58.433 DEBUG:boto:Method: GET 2023-10-18T20:00:58.434 DEBUG:boto:Path: /bofrnf-14/obj 2023-10-18T20:00:58.434 DEBUG:boto:Data: 2023-10-18T20:00:58.434 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.435 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.435 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.436 DEBUG:boto:Params: {} 2023-10-18T20:00:58.436 DEBUG:boto:Token: None 2023-10-18T20:00:58.437 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/obj 2023-10-18T20:00:58.437 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bddNYS+7xj7P/P0cPZ+3XAfBzBE= 2023-10-18T20:00:58.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bddNYS+7xj7P/P0cPZ+3XAfBzBE=', 'Content-Length': '0'} 2023-10-18T20:00:58.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:00:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000041b0ed9a86bc0823-006530397a-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.440 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.441 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.441 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.442 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.442 DEBUG:boto:Method: GET 2023-10-18T20:00:58.442 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:58.443 DEBUG:boto:Data: 2023-10-18T20:00:58.443 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.444 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.444 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.445 DEBUG:boto:Params: {} 2023-10-18T20:00:58.445 DEBUG:boto:Token: None 2023-10-18T20:00:58.445 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/ 2023-10-18T20:00:58.446 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4= 2023-10-18T20:00:58.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4=', 'Content-Length': '0'} 2023-10-18T20:00:58.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4f4c1c1ede8fe78-006530397a-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.449 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.450 DEBUG:boto:path=/bofrnf-14/ 2023-10-18T20:00:58.451 DEBUG:boto:auth_path=/bofrnf-14/ 2023-10-18T20:00:58.451 DEBUG:boto:path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.451 DEBUG:boto:auth_path=/bofrnf-14/?prefix=obj 2023-10-18T20:00:58.452 DEBUG:boto:Method: GET 2023-10-18T20:00:58.452 DEBUG:boto:Path: /bofrnf-14/?prefix=obj 2023-10-18T20:00:58.453 DEBUG:boto:Data: 2023-10-18T20:00:58.453 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.454 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.454 DEBUG:boto:Params: {} 2023-10-18T20:00:58.455 DEBUG:boto:Token: None 2023-10-18T20:00:58.455 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-14/ 2023-10-18T20:00:58.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4= 2023-10-18T20:00:58.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cuqkLwYb3fJlE/h8Yw07jNXP3e4=', 'Content-Length': '0'} 2023-10-18T20:00:58.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f991982524f4fd78-006530397a-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.502 DEBUG:boto:b'bofrnf-14obj1000falseobj2023-10-18T20:00:17.740Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:00:58.503 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-14 zones={a2, a1} 2023-10-18T20:00:58.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-10-18T20:00:58.505 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-15 2023-10-18T20:00:58.505 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:00:58.505 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:00:58.506 DEBUG:boto:Method: PUT 2023-10-18T20:00:58.506 DEBUG:boto:Path: /bofrnf-15/ 2023-10-18T20:00:58.507 DEBUG:boto:Data: 2023-10-18T20:00:58.507 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.508 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.508 DEBUG:boto:Params: {} 2023-10-18T20:00:58.509 DEBUG:boto:Token: None 2023-10-18T20:00:58.509 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-15/ 2023-10-18T20:00:58.510 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KzfrymyBOmWaIGdnM5Cyb84lM1E= 2023-10-18T20:00:58.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KzfrymyBOmWaIGdnM5Cyb84lM1E=', 'Content-Length': '0'} 2023-10-18T20:00:58.530 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000069039f05356d3a67-006530397a-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.530 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-16 2023-10-18T20:00:58.531 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:00:58.531 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:00:58.532 DEBUG:boto:Method: PUT 2023-10-18T20:00:58.533 DEBUG:boto:Path: /bofrnf-16/ 2023-10-18T20:00:58.534 DEBUG:boto:Data: 2023-10-18T20:00:58.534 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.535 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.535 DEBUG:boto:Params: {} 2023-10-18T20:00:58.536 DEBUG:boto:Token: None 2023-10-18T20:00:58.536 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-16/ 2023-10-18T20:00:58.537 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wJDQGed+qQfRyeCl5fJxFYD3Sec= 2023-10-18T20:00:58.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wJDQGed+qQfRyeCl5fJxFYD3Sec=', 'Content-Length': '0'} 2023-10-18T20:00:58.568 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009305afb968931a78-006530397a-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.569 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:00:58.569 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:00:58.570 DEBUG:boto:path=/bofrnf-15/?versioning 2023-10-18T20:00:58.570 DEBUG:boto:auth_path=/bofrnf-15/?versioning 2023-10-18T20:00:58.571 DEBUG:boto:Method: PUT 2023-10-18T20:00:58.571 DEBUG:boto:Path: /bofrnf-15/?versioning 2023-10-18T20:00:58.571 DEBUG:boto:Data: Enabled Disabled 2023-10-18T20:00:58.572 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.572 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:00:58.573 DEBUG:boto:Port: 8000 2023-10-18T20:00:58.573 DEBUG:boto:Params: {} 2023-10-18T20:00:58.573 DEBUG:boto:Token: None 2023-10-18T20:00:58.574 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-15/?versioning 2023-10-18T20:00:58.574 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vzf2oXmtevfXP9B8LqgDHrpqQLY= 2023-10-18T20:00:58.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vzf2oXmtevfXP9B8LqgDHrpqQLY=', 'Content-Length': '228'} 2023-10-18T20:00:58.580 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000074e89b4d52b74d10-006530397a-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.580 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:00:58.581 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:00:58.581 DEBUG:boto:path=/bofrnf-16/?versioning 2023-10-18T20:00:58.582 DEBUG:boto:auth_path=/bofrnf-16/?versioning 2023-10-18T20:00:58.582 DEBUG:boto:Method: PUT 2023-10-18T20:00:58.582 DEBUG:boto:Path: /bofrnf-16/?versioning 2023-10-18T20:00:58.583 DEBUG:boto:Data: Enabled Disabled 2023-10-18T20:00:58.583 DEBUG:boto:Headers: {} 2023-10-18T20:00:58.584 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:00:58.584 DEBUG:boto:Port: 8001 2023-10-18T20:00:58.584 DEBUG:boto:Params: {} 2023-10-18T20:00:58.585 DEBUG:boto:Token: None 2023-10-18T20:00:58.585 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:00:58 GMT /bofrnf-16/?versioning 2023-10-18T20:00:58.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PradNrQ/6n5ryTakCQ1/6FbEGRE= 2023-10-18T20:00:58.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:00:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PradNrQ/6n5ryTakCQ1/6FbEGRE=', 'Content-Length': '228'} 2023-10-18T20:00:58.595 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000094cda590bf2d7317-006530397a-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:00:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:00:58.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:00:58.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:00:58.597 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:00:58.647 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:00:58.647 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:00:58.675 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:00:58.687 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:00:58.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:58.688 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659258.590699_252.1", 2023-10-18T20:00:58.688 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:00:58.590699Z" 2023-10-18T20:00:58.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:58.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:58.689 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:00:58.689 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:00:58.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:58.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:58.689 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:00:58.690 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:00:58.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:58.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:58.690 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:00:58.690 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:00:58.691 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:58.691 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:00:58.692 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659258.590699_252.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:00:58.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:00:58.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:00:58.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:00:58.694 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:00:58.744 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:00:58.744 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:00:58.778 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:58.776+0000 7f58a203e780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:00:58.779 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:00:58.777+0000 7f58a203e780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:00:58.795 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:00:58.795 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:00:58.795 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:00:58.796 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:00:58.796 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:00:58.796 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:00:58.796 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:58.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:58.797 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:00:58.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:58.797 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:00:58.797 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:00:58.798 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:00:58.798 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659258.548823_245.1", 2023-10-18T20:00:58.798 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:00:58.798 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:00:58.798 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:00:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:00:58.548823Z", 2023-10-18T20:00:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:58.800 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:00:58.800 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:00:58.800 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:00:58.800 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:00:58.800 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:00:58.801 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:00:58.801 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:00:58.801 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:00:58.801 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:58.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:00:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:00:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:00:58.803 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:00:58.803 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:00:58.803 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:00:58.803 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:00:58.803 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:00:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:00:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:00:58.805 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:00:58.805 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:00:58.805 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:00:58.805 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:00:58.806 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:00:58.806 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:00:58.806 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:00:58.806 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:00:58.806 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:58.807 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:00:58.807 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:00:58.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:58.807 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:00:58.807 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:00:58.808 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:00:58.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:00:58.808 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:00:58Z" 2023-10-18T20:00:58.808 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:00:58.809 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659258.548823_245.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:00:58.548823Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:00:58Z" } 2023-10-18T20:00:58.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659258.548823_245.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:00:58.548823Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:00:58.810 DEBUG:rgw_multi.tests:log_status={0: '1_1697659258.590699_252.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:00:58.810 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659258.548823_245.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:00:58.811 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659258.590699_252.1 target=1_1697659258.548823_245.1 2023-10-18T20:01:03.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:01:03.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:01:03.813 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:01:03.872 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:03.873 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:03.904 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:03.903+0000 7fee07056780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:01:03.905 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:03.904+0000 7fee07056780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:01:03.917 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:01:03.918 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:01:03.918 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:01:03.918 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:01:03.918 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:01:03.919 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:01:03.919 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:03.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:03.919 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:01:03.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:03.920 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:01:03.920 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:03.920 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:03.920 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659258.590699_252.1", 2023-10-18T20:01:03.921 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:03.921 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:01:03.921 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:03.921 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:00:58.590699Z", 2023-10-18T20:01:03.921 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:03.922 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:03.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:03.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:03.922 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:01:03.922 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:03.923 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:03.923 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:01:03.923 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:03.924 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:01:03.924 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:03.924 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:01:03.924 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:03.925 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:03.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:03.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:03.925 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:01:03.925 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:03.926 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:03.926 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:01:03.926 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:03.926 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:01:03.927 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:03.927 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:01:03.927 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:03.927 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:03.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:03.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:03.928 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:01:03.928 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:03.929 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:03.929 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:01:03.930 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:03.930 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:01:03.930 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:03.930 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:01:03.931 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:03.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:03.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:03.932 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:01:03.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:03.932 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:01:03.932 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:01:03.932 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:01:03.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:03.933 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:01:03Z" 2023-10-18T20:01:03.933 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:01:03.934 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659258.590699_252.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:00:58.590699Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:01:03Z" } 2023-10-18T20:01:03.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659258.590699_252.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:00:58.590699Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:01:03.936 DEBUG:rgw_multi.tests:log_status={0: '1_1697659258.590699_252.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:01:03.936 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659258.590699_252.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:01:03.936 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:01:03.937 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:03.937 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:03.938 DEBUG:boto:Method: HEAD 2023-10-18T20:01:03.938 DEBUG:boto:Path: /bofrnf-15/ 2023-10-18T20:01:03.939 DEBUG:boto:Data: 2023-10-18T20:01:03.939 DEBUG:boto:Headers: {} 2023-10-18T20:01:03.939 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:03.940 DEBUG:boto:Port: 8000 2023-10-18T20:01:03.940 DEBUG:boto:Params: {} 2023-10-18T20:01:03.941 DEBUG:boto:Token: None 2023-10-18T20:01:03.941 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:03 GMT /bofrnf-15/ 2023-10-18T20:01:03.942 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GoVP7BUjaMUKYhvlud53+82ZCKY= 2023-10-18T20:01:03.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GoVP7BUjaMUKYhvlud53+82ZCKY=', 'Content-Length': '0'} 2023-10-18T20:01:03.945 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007477d0237a261478-006530397f-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:03.945 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:03.946 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:03.946 DEBUG:boto:Method: PUT 2023-10-18T20:01:03.947 DEBUG:boto:Path: /bofrnf-15/obj 2023-10-18T20:01:03.947 DEBUG:boto:Data: 2023-10-18T20:01:03.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T20:01:03.948 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:03.948 DEBUG:boto:Port: 8000 2023-10-18T20:01:03.949 DEBUG:boto:Params: {} 2023-10-18T20:01:03.949 DEBUG:boto:Token: None 2023-10-18T20:01:03.950 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 20:01:03 GMT /bofrnf-15/obj 2023-10-18T20:01:03.950 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9KhZH5g/rGhHN+IMhpzHfLKdkBM= 2023-10-18T20:01:03.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:01:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9KhZH5g/rGhHN+IMhpzHfLKdkBM='} 2023-10-18T20:01:03.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk'), ('x-amz-request-id', 'tx000004d2778ed004b913e-006530397f-1112-a1'), ('Date', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:03.965 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:03.966 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:03.967 DEBUG:boto:Method: HEAD 2023-10-18T20:01:03.967 DEBUG:boto:Path: /bofrnf-15/ 2023-10-18T20:01:03.968 DEBUG:boto:Data: 2023-10-18T20:01:03.968 DEBUG:boto:Headers: {} 2023-10-18T20:01:03.968 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:03.969 DEBUG:boto:Port: 8000 2023-10-18T20:01:03.969 DEBUG:boto:Params: {} 2023-10-18T20:01:03.970 DEBUG:boto:Token: None 2023-10-18T20:01:03.971 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:03 GMT /bofrnf-15/ 2023-10-18T20:01:03.971 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GoVP7BUjaMUKYhvlud53+82ZCKY= 2023-10-18T20:01:03.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GoVP7BUjaMUKYhvlud53+82ZCKY=', 'Content-Length': '0'} 2023-10-18T20:01:03.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d95e5ac71dbdc39f-006530397f-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:03.975 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:03.975 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:03.976 DEBUG:boto:Method: DELETE 2023-10-18T20:01:03.976 DEBUG:boto:Path: /bofrnf-15/obj 2023-10-18T20:01:03.976 DEBUG:boto:Data: 2023-10-18T20:01:03.977 DEBUG:boto:Headers: {} 2023-10-18T20:01:03.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:03.978 DEBUG:boto:Port: 8000 2023-10-18T20:01:03.978 DEBUG:boto:Params: {} 2023-10-18T20:01:03.978 DEBUG:boto:Token: None 2023-10-18T20:01:03.979 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 20:01:03 GMT /bofrnf-15/obj 2023-10-18T20:01:03.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dRvtkPeLnoLL/ojiN9jmlWDCg68= 2023-10-18T20:01:03.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dRvtkPeLnoLL/ojiN9jmlWDCg68=', 'Content-Length': '0'} 2023-10-18T20:01:03.986 DEBUG:boto:Response headers: [('x-amz-version-id', 'Wl7GTYACsBSHxHzQxT-6CpM8Zg-tUyd'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000050e0b314dc98dca0-006530397f-1112-a1'), ('Date', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:03.987 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:03.987 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:03.988 DEBUG:boto:Method: HEAD 2023-10-18T20:01:03.988 DEBUG:boto:Path: /bofrnf-16/ 2023-10-18T20:01:03.989 DEBUG:boto:Data: 2023-10-18T20:01:03.989 DEBUG:boto:Headers: {} 2023-10-18T20:01:03.989 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:03.990 DEBUG:boto:Port: 8001 2023-10-18T20:01:03.990 DEBUG:boto:Params: {} 2023-10-18T20:01:03.991 DEBUG:boto:Token: None 2023-10-18T20:01:03.991 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:03 GMT /bofrnf-16/ 2023-10-18T20:01:03.992 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4TEfE/RcaTlz80WTVHe5ESVWU9I= 2023-10-18T20:01:03.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4TEfE/RcaTlz80WTVHe5ESVWU9I=', 'Content-Length': '0'} 2023-10-18T20:01:03.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a83a0f5956a5b478-006530397f-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:03.995 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:03.996 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:03.996 DEBUG:boto:Method: PUT 2023-10-18T20:01:03.996 DEBUG:boto:Path: /bofrnf-16/obj 2023-10-18T20:01:03.997 DEBUG:boto:Data: 2023-10-18T20:01:03.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T20:01:03.998 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:03.998 DEBUG:boto:Port: 8001 2023-10-18T20:01:03.999 DEBUG:boto:Params: {} 2023-10-18T20:01:03.999 DEBUG:boto:Token: None 2023-10-18T20:01:04.000 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 20:01:04 GMT /bofrnf-16/obj 2023-10-18T20:01:04.000 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c9WFC8nYcXplxZvhorjz+7W9prc= 2023-10-18T20:01:04.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:01:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c9WFC8nYcXplxZvhorjz+7W9prc='} 2023-10-18T20:01:04.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dNZsMCWPZgrqoYBBpKsJustaw7SI.bg'), ('x-amz-request-id', 'tx0000044c7d00021336d25-0065303980-1116-a2'), ('Date', 'Wed, 18 Oct 2023 20:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:04.047 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:04.048 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:04.048 DEBUG:boto:Method: HEAD 2023-10-18T20:01:04.048 DEBUG:boto:Path: /bofrnf-16/ 2023-10-18T20:01:04.049 DEBUG:boto:Data: 2023-10-18T20:01:04.049 DEBUG:boto:Headers: {} 2023-10-18T20:01:04.049 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:04.050 DEBUG:boto:Port: 8001 2023-10-18T20:01:04.050 DEBUG:boto:Params: {} 2023-10-18T20:01:04.051 DEBUG:boto:Token: None 2023-10-18T20:01:04.051 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:04 GMT /bofrnf-16/ 2023-10-18T20:01:04.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mVR1pazlLCAq1A4kiWUwOMRYM6k= 2023-10-18T20:01:04.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mVR1pazlLCAq1A4kiWUwOMRYM6k=', 'Content-Length': '0'} 2023-10-18T20:01:04.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6af89f48259bcef-0065303980-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:04.055 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:04.055 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:04.055 DEBUG:boto:Method: DELETE 2023-10-18T20:01:04.056 DEBUG:boto:Path: /bofrnf-16/obj 2023-10-18T20:01:04.056 DEBUG:boto:Data: 2023-10-18T20:01:04.057 DEBUG:boto:Headers: {} 2023-10-18T20:01:04.057 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:04.057 DEBUG:boto:Port: 8001 2023-10-18T20:01:04.058 DEBUG:boto:Params: {} 2023-10-18T20:01:04.058 DEBUG:boto:Token: None 2023-10-18T20:01:04.059 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 20:01:04 GMT /bofrnf-16/obj 2023-10-18T20:01:04.059 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qwx5wMi3mfiedI85cUHruPUqaS4= 2023-10-18T20:01:04.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qwx5wMi3mfiedI85cUHruPUqaS4=', 'Content-Length': '0'} 2023-10-18T20:01:04.067 DEBUG:boto:Response headers: [('x-amz-version-id', 'aP8w7BOTresKDZ81w3dOBuWm7gb9wuE'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000bbe9f46209b19844-0065303980-1116-a2'), ('Date', 'Wed, 18 Oct 2023 20:01:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:04.070 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-15 2023-10-18T20:01:04.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:04.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:04.071 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:01:04.127 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:04.128 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:04.160 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:04.159+0000 7f0591017780 1 waiting to reach incremental sync.. 2023-10-18T20:01:09.161 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:09.160+0000 7f0591017780 1 waiting to reach incremental sync.. 2023-10-18T20:01:14.162 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:14.159+0000 7f0591017780 1 waiting to reach incremental sync.. 2023-10-18T20:01:19.163 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:19.161+0000 7f0591017780 1 bucket sync caught up with source: 2023-10-18T20:01:19.164 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.8.12, , ] 2023-10-18T20:01:19.164 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.8.12, , ] 2023-10-18T20:01:19.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:19.161+0000 7f0591017780 0 bucket checkpoint complete 2023-10-18T20:01:19.179 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-15 2023-10-18T20:01:19.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:19.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:19.181 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:01:19.244 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:19.244 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:19.279 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:19.277+0000 7f0bcd83d780 1 bucket sync caught up with source: 2023-10-18T20:01:19.280 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000002.20.6, , ] 2023-10-18T20:01:19.280 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000002.20.6, , ] 2023-10-18T20:01:19.280 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:19.277+0000 7f0bcd83d780 0 bucket checkpoint complete 2023-10-18T20:01:19.292 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-15 zones={a2, a1} 2023-10-18T20:01:19.293 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:19.293 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:19.294 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.294 DEBUG:boto:Path: /bofrnf-15/ 2023-10-18T20:01:19.295 DEBUG:boto:Data: 2023-10-18T20:01:19.295 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.295 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.296 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.296 DEBUG:boto:Params: {} 2023-10-18T20:01:19.297 DEBUG:boto:Token: None 2023-10-18T20:01:19.298 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/ 2023-10-18T20:01:19.298 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eJ9OiCy3iKBBTX778XG1sBSktLg= 2023-10-18T20:01:19.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eJ9OiCy3iKBBTX778XG1sBSktLg=', 'Content-Length': '0'} 2023-10-18T20:01:19.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea200d7314297e87-006530398f-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.302 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:19.302 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:19.303 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.303 DEBUG:boto:Path: /bofrnf-15/ 2023-10-18T20:01:19.303 DEBUG:boto:Data: 2023-10-18T20:01:19.304 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.304 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.305 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.305 DEBUG:boto:Params: {} 2023-10-18T20:01:19.305 DEBUG:boto:Token: None 2023-10-18T20:01:19.306 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/ 2023-10-18T20:01:19.306 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eJ9OiCy3iKBBTX778XG1sBSktLg= 2023-10-18T20:01:19.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eJ9OiCy3iKBBTX778XG1sBSktLg=', 'Content-Length': '0'} 2023-10-18T20:01:19.309 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052b4e1cd77d6c955-006530398f-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.310 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:01:19.310 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:19.311 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:19.311 DEBUG:boto:path=/bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.311 DEBUG:boto:auth_path=/bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.312 DEBUG:boto:Method: GET 2023-10-18T20:01:19.312 DEBUG:boto:Path: /bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.313 DEBUG:boto:Data: 2023-10-18T20:01:19.313 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.313 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.314 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.314 DEBUG:boto:Params: {} 2023-10-18T20:01:19.315 DEBUG:boto:Token: None 2023-10-18T20:01:19.315 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/?versions 2023-10-18T20:01:19.316 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7vMuxAeyrSxT1YTrRG69eTdOAw= 2023-10-18T20:01:19.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7vMuxAeyrSxT1YTrRG69eTdOAw=', 'Content-Length': '0'} 2023-10-18T20:01:19.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000069c21cece6eb99e-006530398f-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.319 DEBUG:boto:b'bofrnf-15999falseobjWl7GTYACsBSHxHzQxT-6CpM8Zg-tUydtrue2023-10-18T20:01:03.981Zrgw-multisite-test-userMultisite Test UserNormalobj4vzXnjF2u2HrYlpGYR9nPKJasOfcPRkfalse2023-10-18T20:01:03.952Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:01:19.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:01:19.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:01:19.320 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:01:19.321 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:19.321 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:19.322 DEBUG:boto:path=/bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.322 DEBUG:boto:auth_path=/bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.323 DEBUG:boto:Method: GET 2023-10-18T20:01:19.323 DEBUG:boto:Path: /bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.323 DEBUG:boto:Data: 2023-10-18T20:01:19.324 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.324 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.325 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.325 DEBUG:boto:Params: {} 2023-10-18T20:01:19.326 DEBUG:boto:Token: None 2023-10-18T20:01:19.326 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/?versions 2023-10-18T20:01:19.327 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7vMuxAeyrSxT1YTrRG69eTdOAw= 2023-10-18T20:01:19.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7vMuxAeyrSxT1YTrRG69eTdOAw=', 'Content-Length': '0'} 2023-10-18T20:01:19.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007029c18fca205eeb-006530398f-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.331 DEBUG:boto:b'bofrnf-15999falseobjWl7GTYACsBSHxHzQxT-6CpM8Zg-tUydtrue2023-10-18T20:01:03.981Zrgw-multisite-test-userMultisite Test UserNormalobj4vzXnjF2u2HrYlpGYR9nPKJasOfcPRkfalse2023-10-18T20:01:03.952Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:01:19.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:01:19.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:01:19.332 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:19.333 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:19.333 DEBUG:boto:path=/bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.334 DEBUG:boto:auth_path=/bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.334 DEBUG:boto:Method: GET 2023-10-18T20:01:19.334 DEBUG:boto:Path: /bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.335 DEBUG:boto:Data: 2023-10-18T20:01:19.335 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.336 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.336 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.336 DEBUG:boto:Params: {} 2023-10-18T20:01:19.337 DEBUG:boto:Token: None 2023-10-18T20:01:19.337 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/?versions 2023-10-18T20:01:19.338 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7vMuxAeyrSxT1YTrRG69eTdOAw= 2023-10-18T20:01:19.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7vMuxAeyrSxT1YTrRG69eTdOAw=', 'Content-Length': '0'} 2023-10-18T20:01:19.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007647c26a693c73da-006530398f-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.341 DEBUG:boto:b'bofrnf-15999falseobjWl7GTYACsBSHxHzQxT-6CpM8Zg-tUydtrue2023-10-18T20:01:03.981Zrgw-multisite-test-userMultisite Test UserNormalobj4vzXnjF2u2HrYlpGYR9nPKJasOfcPRkfalse2023-10-18T20:01:03.952Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:01:19.342 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:19.342 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:19.343 DEBUG:boto:path=/bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.343 DEBUG:boto:auth_path=/bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.344 DEBUG:boto:Method: GET 2023-10-18T20:01:19.344 DEBUG:boto:Path: /bofrnf-15/?versions&max-keys=999 2023-10-18T20:01:19.344 DEBUG:boto:Data: 2023-10-18T20:01:19.345 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.345 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.346 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.346 DEBUG:boto:Params: {} 2023-10-18T20:01:19.347 DEBUG:boto:Token: None 2023-10-18T20:01:19.347 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/?versions 2023-10-18T20:01:19.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7vMuxAeyrSxT1YTrRG69eTdOAw= 2023-10-18T20:01:19.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7vMuxAeyrSxT1YTrRG69eTdOAw=', 'Content-Length': '0'} 2023-10-18T20:01:19.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ebb06b696820442-006530398f-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.394 DEBUG:boto:b'bofrnf-15999falseobjWl7GTYACsBSHxHzQxT-6CpM8Zg-tUydtrue2023-10-18T20:01:03.981Zrgw-multisite-test-userMultisite Test UserNormalobj4vzXnjF2u2HrYlpGYR9nPKJasOfcPRkfalse2023-10-18T20:01:03.952Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:01:19.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:01:19.396 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.396 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.397 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.397 DEBUG:boto:Path: /bofrnf-15/obj 2023-10-18T20:01:19.397 DEBUG:boto:Data: 2023-10-18T20:01:19.398 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.399 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.399 DEBUG:boto:Params: {} 2023-10-18T20:01:19.400 DEBUG:boto:Token: None 2023-10-18T20:01:19.400 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj 2023-10-18T20:01:19.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:epMocOxGnHjblHXxJlOYLqT3/xQ= 2023-10-18T20:01:19.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:epMocOxGnHjblHXxJlOYLqT3/xQ=', 'Content-Length': '0'} 2023-10-18T20:01:19.403 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002b4447417233cd1c-006530398f-1116-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.404 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.404 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.405 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.405 DEBUG:boto:Path: /bofrnf-15/obj 2023-10-18T20:01:19.406 DEBUG:boto:Data: 2023-10-18T20:01:19.406 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.406 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.407 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.407 DEBUG:boto:Params: {} 2023-10-18T20:01:19.408 DEBUG:boto:Token: None 2023-10-18T20:01:19.408 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj 2023-10-18T20:01:19.408 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:epMocOxGnHjblHXxJlOYLqT3/xQ= 2023-10-18T20:01:19.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:epMocOxGnHjblHXxJlOYLqT3/xQ=', 'Content-Length': '0'} 2023-10-18T20:01:19.411 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000203bd574ef86f588-006530398f-1112-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:01:19.412 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.412 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.412 DEBUG:boto:path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.413 DEBUG:boto:auth_path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.413 DEBUG:boto:Method: GET 2023-10-18T20:01:19.414 DEBUG:boto:Path: /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.414 DEBUG:boto:Data: 2023-10-18T20:01:19.415 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.416 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.416 DEBUG:boto:Params: {} 2023-10-18T20:01:19.417 DEBUG:boto:Token: None 2023-10-18T20:01:19.417 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tQTru24k2ghf0qYcDPVl4a93XJE= 2023-10-18T20:01:19.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tQTru24k2ghf0qYcDPVl4a93XJE=', 'Content-Length': '0'} 2023-10-18T20:01:19.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('x-amz-version-id', '4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000338720340a8aa32e-006530398f-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.421 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.421 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.422 DEBUG:boto:path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.422 DEBUG:boto:auth_path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.423 DEBUG:boto:Method: GET 2023-10-18T20:01:19.423 DEBUG:boto:Path: /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.423 DEBUG:boto:Data: 2023-10-18T20:01:19.424 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.424 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.425 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.425 DEBUG:boto:Params: {} 2023-10-18T20:01:19.426 DEBUG:boto:Token: None 2023-10-18T20:01:19.426 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.427 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tQTru24k2ghf0qYcDPVl4a93XJE= 2023-10-18T20:01:19.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tQTru24k2ghf0qYcDPVl4a93XJE=', 'Content-Length': '0'} 2023-10-18T20:01:19.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('x-amz-version-id', '4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1fe59235b2656a6-006530398f-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.430 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.430 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.431 DEBUG:boto:path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.431 DEBUG:boto:auth_path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.431 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.432 DEBUG:boto:Path: /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.432 DEBUG:boto:Data: 2023-10-18T20:01:19.432 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.433 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.434 DEBUG:boto:Params: {} 2023-10-18T20:01:19.434 DEBUG:boto:Token: None 2023-10-18T20:01:19.435 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.435 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3QKoCoT40jWw7am/9fbpzbH3Nhs= 2023-10-18T20:01:19.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3QKoCoT40jWw7am/9fbpzbH3Nhs=', 'Content-Length': '0'} 2023-10-18T20:01:19.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('x-amz-version-id', '4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f0fc9c7e6d1dc164-006530398f-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.438 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.438 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.439 DEBUG:boto:path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.439 DEBUG:boto:auth_path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.440 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.440 DEBUG:boto:Path: /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.440 DEBUG:boto:Data: 2023-10-18T20:01:19.441 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.441 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.442 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.442 DEBUG:boto:Params: {} 2023-10-18T20:01:19.442 DEBUG:boto:Token: None 2023-10-18T20:01:19.443 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.443 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3QKoCoT40jWw7am/9fbpzbH3Nhs= 2023-10-18T20:01:19.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3QKoCoT40jWw7am/9fbpzbH3Nhs=', 'Content-Length': '0'} 2023-10-18T20:01:19.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('x-amz-version-id', '4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000616f35eb12be831d-006530398f-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:01:19.447 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.447 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.447 DEBUG:boto:path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.448 DEBUG:boto:auth_path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.448 DEBUG:boto:Method: GET 2023-10-18T20:01:19.449 DEBUG:boto:Path: /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.449 DEBUG:boto:Data: 2023-10-18T20:01:19.450 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.450 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.451 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.451 DEBUG:boto:Params: {} 2023-10-18T20:01:19.451 DEBUG:boto:Token: None 2023-10-18T20:01:19.454 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tQTru24k2ghf0qYcDPVl4a93XJE= 2023-10-18T20:01:19.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tQTru24k2ghf0qYcDPVl4a93XJE=', 'Content-Length': '0'} 2023-10-18T20:01:19.457 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('x-amz-version-id', '4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089dce526631f4995-006530398f-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.458 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.459 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.459 DEBUG:boto:path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.459 DEBUG:boto:auth_path=/bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.460 DEBUG:boto:Method: GET 2023-10-18T20:01:19.460 DEBUG:boto:Path: /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.461 DEBUG:boto:Data: 2023-10-18T20:01:19.461 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.461 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.462 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.462 DEBUG:boto:Params: {} 2023-10-18T20:01:19.463 DEBUG:boto:Token: None 2023-10-18T20:01:19.463 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj?versionId=4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk 2023-10-18T20:01:19.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tQTru24k2ghf0qYcDPVl4a93XJE= 2023-10-18T20:01:19.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tQTru24k2ghf0qYcDPVl4a93XJE=', 'Content-Length': '0'} 2023-10-18T20:01:19.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:03 GMT'), ('x-amz-version-id', '4vzXnjF2u2HrYlpGYR9nPKJasOfcPRk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002aa099fb0138a293-006530398f-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.467 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:19.467 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:19.468 DEBUG:boto:path=/bofrnf-15/?prefix=obj 2023-10-18T20:01:19.468 DEBUG:boto:auth_path=/bofrnf-15/?prefix=obj 2023-10-18T20:01:19.468 DEBUG:boto:Method: GET 2023-10-18T20:01:19.469 DEBUG:boto:Path: /bofrnf-15/?prefix=obj 2023-10-18T20:01:19.470 DEBUG:boto:Data: 2023-10-18T20:01:19.471 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.472 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.473 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.473 DEBUG:boto:Params: {} 2023-10-18T20:01:19.474 DEBUG:boto:Token: None 2023-10-18T20:01:19.475 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/ 2023-10-18T20:01:19.475 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EIt4Iu7gNZGkaX2EnJCfzZ8JB+w= 2023-10-18T20:01:19.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EIt4Iu7gNZGkaX2EnJCfzZ8JB+w=', 'Content-Length': '0'} 2023-10-18T20:01:19.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea024c8251d465be-006530398f-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.518 DEBUG:boto:b'bofrnf-15obj1000false' 2023-10-18T20:01:19.519 DEBUG:boto:path=/bofrnf-15/ 2023-10-18T20:01:19.520 DEBUG:boto:auth_path=/bofrnf-15/ 2023-10-18T20:01:19.520 DEBUG:boto:path=/bofrnf-15/?prefix=obj 2023-10-18T20:01:19.521 DEBUG:boto:auth_path=/bofrnf-15/?prefix=obj 2023-10-18T20:01:19.521 DEBUG:boto:Method: GET 2023-10-18T20:01:19.522 DEBUG:boto:Path: /bofrnf-15/?prefix=obj 2023-10-18T20:01:19.522 DEBUG:boto:Data: 2023-10-18T20:01:19.522 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.523 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.523 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.524 DEBUG:boto:Params: {} 2023-10-18T20:01:19.524 DEBUG:boto:Token: None 2023-10-18T20:01:19.525 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/ 2023-10-18T20:01:19.525 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EIt4Iu7gNZGkaX2EnJCfzZ8JB+w= 2023-10-18T20:01:19.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EIt4Iu7gNZGkaX2EnJCfzZ8JB+w=', 'Content-Length': '0'} 2023-10-18T20:01:19.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a738fb2fff04e497-006530398f-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.529 DEBUG:boto:b'bofrnf-15obj1000false' 2023-10-18T20:01:19.530 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.530 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.531 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.531 DEBUG:boto:Path: /bofrnf-15/obj 2023-10-18T20:01:19.532 DEBUG:boto:Data: 2023-10-18T20:01:19.532 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.532 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.533 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.533 DEBUG:boto:Params: {} 2023-10-18T20:01:19.534 DEBUG:boto:Token: None 2023-10-18T20:01:19.534 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj 2023-10-18T20:01:19.534 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:epMocOxGnHjblHXxJlOYLqT3/xQ= 2023-10-18T20:01:19.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:epMocOxGnHjblHXxJlOYLqT3/xQ=', 'Content-Length': '0'} 2023-10-18T20:01:19.537 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000065be999b71b4aa0e-006530398f-1116-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.538 DEBUG:boto:path=/bofrnf-15/obj 2023-10-18T20:01:19.539 DEBUG:boto:auth_path=/bofrnf-15/obj 2023-10-18T20:01:19.540 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.541 DEBUG:boto:Path: /bofrnf-15/obj 2023-10-18T20:01:19.541 DEBUG:boto:Data: 2023-10-18T20:01:19.541 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.542 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.543 DEBUG:boto:Params: {} 2023-10-18T20:01:19.543 DEBUG:boto:Token: None 2023-10-18T20:01:19.544 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-15/obj 2023-10-18T20:01:19.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:epMocOxGnHjblHXxJlOYLqT3/xQ= 2023-10-18T20:01:19.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:epMocOxGnHjblHXxJlOYLqT3/xQ=', 'Content-Length': '0'} 2023-10-18T20:01:19.547 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000005020f69407b65b91-006530398f-1112-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.547 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-15 zones={a2, a1} 2023-10-18T20:01:19.548 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-16 2023-10-18T20:01:19.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:19.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:19.549 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:01:19.605 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:19.606 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:19.640 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:19.638+0000 7fe1e6275780 1 bucket sync caught up with source: 2023-10-18T20:01:19.641 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000002.10.6, , ] 2023-10-18T20:01:19.641 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000002.10.6, , ] 2023-10-18T20:01:19.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:19.638+0000 7fe1e6275780 0 bucket checkpoint complete 2023-10-18T20:01:19.653 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-16 2023-10-18T20:01:19.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:19.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:19.654 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:01:19.710 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:19.710 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:19.746 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:19.743+0000 7ff0b972a780 1 bucket sync caught up with source: 2023-10-18T20:01:19.746 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.21.12, , ] 2023-10-18T20:01:19.746 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.21.12, , ] 2023-10-18T20:01:19.746 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:19.743+0000 7ff0b972a780 0 bucket checkpoint complete 2023-10-18T20:01:19.757 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-16 zones={a2, a1} 2023-10-18T20:01:19.758 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:19.758 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:19.759 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.759 DEBUG:boto:Path: /bofrnf-16/ 2023-10-18T20:01:19.760 DEBUG:boto:Data: 2023-10-18T20:01:19.760 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.761 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.761 DEBUG:boto:Params: {} 2023-10-18T20:01:19.762 DEBUG:boto:Token: None 2023-10-18T20:01:19.763 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/ 2023-10-18T20:01:19.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YVMh8DIqdamoTy/tzVKLREMtCzg= 2023-10-18T20:01:19.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YVMh8DIqdamoTy/tzVKLREMtCzg=', 'Content-Length': '0'} 2023-10-18T20:01:19.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b04188ba7bebb764-006530398f-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.766 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:19.767 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:19.767 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.768 DEBUG:boto:Path: /bofrnf-16/ 2023-10-18T20:01:19.768 DEBUG:boto:Data: 2023-10-18T20:01:19.769 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.769 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.770 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.770 DEBUG:boto:Params: {} 2023-10-18T20:01:19.771 DEBUG:boto:Token: None 2023-10-18T20:01:19.771 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/ 2023-10-18T20:01:19.771 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YVMh8DIqdamoTy/tzVKLREMtCzg= 2023-10-18T20:01:19.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YVMh8DIqdamoTy/tzVKLREMtCzg=', 'Content-Length': '0'} 2023-10-18T20:01:19.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095e0c8b8d9b315c1-006530398f-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.775 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:01:19.775 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:19.775 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:19.776 DEBUG:boto:path=/bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.776 DEBUG:boto:auth_path=/bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.777 DEBUG:boto:Method: GET 2023-10-18T20:01:19.777 DEBUG:boto:Path: /bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.777 DEBUG:boto:Data: 2023-10-18T20:01:19.778 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.778 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.779 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.779 DEBUG:boto:Params: {} 2023-10-18T20:01:19.780 DEBUG:boto:Token: None 2023-10-18T20:01:19.780 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/?versions 2023-10-18T20:01:19.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pW3B+sppjy+2G7GQbmcSn3875D0= 2023-10-18T20:01:19.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pW3B+sppjy+2G7GQbmcSn3875D0=', 'Content-Length': '0'} 2023-10-18T20:01:19.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a926dedd494a3d93-006530398f-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.784 DEBUG:boto:b'bofrnf-16999falseobjaP8w7BOTresKDZ81w3dOBuWm7gb9wuEtrue2023-10-18T20:01:04.061Zrgw-multisite-test-userMultisite Test UserNormalobjdNZsMCWPZgrqoYBBpKsJustaw7SI.bgfalse2023-10-18T20:01:04.001Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:01:19.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:01:19.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:01:19.785 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:01:19.785 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:19.786 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:19.786 DEBUG:boto:path=/bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.787 DEBUG:boto:auth_path=/bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.787 DEBUG:boto:Method: GET 2023-10-18T20:01:19.788 DEBUG:boto:Path: /bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.788 DEBUG:boto:Data: 2023-10-18T20:01:19.788 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.789 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.790 DEBUG:boto:Params: {} 2023-10-18T20:01:19.790 DEBUG:boto:Token: None 2023-10-18T20:01:19.791 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/?versions 2023-10-18T20:01:19.791 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pW3B+sppjy+2G7GQbmcSn3875D0= 2023-10-18T20:01:19.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pW3B+sppjy+2G7GQbmcSn3875D0=', 'Content-Length': '0'} 2023-10-18T20:01:19.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b008700733c9494-006530398f-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.795 DEBUG:boto:b'bofrnf-16999falseobjaP8w7BOTresKDZ81w3dOBuWm7gb9wuEtrue2023-10-18T20:01:04.061Zrgw-multisite-test-userMultisite Test UserNormalobjdNZsMCWPZgrqoYBBpKsJustaw7SI.bgfalse2023-10-18T20:01:04.001Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:01:19.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:01:19.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:01:19.796 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:19.796 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:19.797 DEBUG:boto:path=/bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.797 DEBUG:boto:auth_path=/bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.798 DEBUG:boto:Method: GET 2023-10-18T20:01:19.798 DEBUG:boto:Path: /bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.799 DEBUG:boto:Data: 2023-10-18T20:01:19.799 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.800 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.800 DEBUG:boto:Params: {} 2023-10-18T20:01:19.801 DEBUG:boto:Token: None 2023-10-18T20:01:19.801 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/?versions 2023-10-18T20:01:19.802 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pW3B+sppjy+2G7GQbmcSn3875D0= 2023-10-18T20:01:19.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pW3B+sppjy+2G7GQbmcSn3875D0=', 'Content-Length': '0'} 2023-10-18T20:01:19.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4898b999235b3d6-006530398f-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.805 DEBUG:boto:b'bofrnf-16999falseobjaP8w7BOTresKDZ81w3dOBuWm7gb9wuEtrue2023-10-18T20:01:04.061Zrgw-multisite-test-userMultisite Test UserNormalobjdNZsMCWPZgrqoYBBpKsJustaw7SI.bgfalse2023-10-18T20:01:04.001Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:01:19.806 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:19.806 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:19.807 DEBUG:boto:path=/bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.807 DEBUG:boto:auth_path=/bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.808 DEBUG:boto:Method: GET 2023-10-18T20:01:19.808 DEBUG:boto:Path: /bofrnf-16/?versions&max-keys=999 2023-10-18T20:01:19.808 DEBUG:boto:Data: 2023-10-18T20:01:19.809 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.810 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.810 DEBUG:boto:Params: {} 2023-10-18T20:01:19.811 DEBUG:boto:Token: None 2023-10-18T20:01:19.811 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/?versions 2023-10-18T20:01:19.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pW3B+sppjy+2G7GQbmcSn3875D0= 2023-10-18T20:01:19.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pW3B+sppjy+2G7GQbmcSn3875D0=', 'Content-Length': '0'} 2023-10-18T20:01:19.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0c185f17f7f04fc-006530398f-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.858 DEBUG:boto:b'bofrnf-16999falseobjaP8w7BOTresKDZ81w3dOBuWm7gb9wuEtrue2023-10-18T20:01:04.061Zrgw-multisite-test-userMultisite Test UserNormalobjdNZsMCWPZgrqoYBBpKsJustaw7SI.bgfalse2023-10-18T20:01:04.001Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:01:19.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:01:19.860 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:19.861 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:19.861 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.862 DEBUG:boto:Path: /bofrnf-16/obj 2023-10-18T20:01:19.863 DEBUG:boto:Data: 2023-10-18T20:01:19.863 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.866 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.866 DEBUG:boto:Params: {} 2023-10-18T20:01:19.867 DEBUG:boto:Token: None 2023-10-18T20:01:19.867 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/obj 2023-10-18T20:01:19.868 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sQYE4GtTAMBfCWUDzpUaERUDK6U= 2023-10-18T20:01:19.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sQYE4GtTAMBfCWUDzpUaERUDK6U=', 'Content-Length': '0'} 2023-10-18T20:01:19.870 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000cd5dd6cbe20dde1c-006530398f-1116-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.871 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:19.871 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:19.872 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.872 DEBUG:boto:Path: /bofrnf-16/obj 2023-10-18T20:01:19.873 DEBUG:boto:Data: 2023-10-18T20:01:19.873 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.874 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.874 DEBUG:boto:Params: {} 2023-10-18T20:01:19.875 DEBUG:boto:Token: None 2023-10-18T20:01:19.875 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/obj 2023-10-18T20:01:19.876 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sQYE4GtTAMBfCWUDzpUaERUDK6U= 2023-10-18T20:01:19.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sQYE4GtTAMBfCWUDzpUaERUDK6U=', 'Content-Length': '0'} 2023-10-18T20:01:19.878 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f77a6b2f2e401a37-006530398f-1112-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:01:19.879 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:19.879 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:19.880 DEBUG:boto:path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.880 DEBUG:boto:auth_path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.881 DEBUG:boto:Method: GET 2023-10-18T20:01:19.881 DEBUG:boto:Path: /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.882 DEBUG:boto:Data: 2023-10-18T20:01:19.882 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.883 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.883 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.883 DEBUG:boto:Params: {} 2023-10-18T20:01:19.884 DEBUG:boto:Token: None 2023-10-18T20:01:19.884 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.885 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E42ldxZeBzHeNSmzC6ZH57IGQSE= 2023-10-18T20:01:19.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E42ldxZeBzHeNSmzC6ZH57IGQSE=', 'Content-Length': '0'} 2023-10-18T20:01:19.887 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:04 GMT'), ('x-amz-version-id', 'dNZsMCWPZgrqoYBBpKsJustaw7SI.bg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002753570f03a482f8-006530398f-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.888 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:19.888 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:19.889 DEBUG:boto:path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.889 DEBUG:boto:auth_path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.889 DEBUG:boto:Method: GET 2023-10-18T20:01:19.890 DEBUG:boto:Path: /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.890 DEBUG:boto:Data: 2023-10-18T20:01:19.891 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.891 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.892 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.892 DEBUG:boto:Params: {} 2023-10-18T20:01:19.892 DEBUG:boto:Token: None 2023-10-18T20:01:19.893 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.893 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E42ldxZeBzHeNSmzC6ZH57IGQSE= 2023-10-18T20:01:19.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E42ldxZeBzHeNSmzC6ZH57IGQSE=', 'Content-Length': '0'} 2023-10-18T20:01:19.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:04 GMT'), ('x-amz-version-id', 'dNZsMCWPZgrqoYBBpKsJustaw7SI.bg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c260aeb7a95f8af-006530398f-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.896 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:19.897 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:19.897 DEBUG:boto:path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.897 DEBUG:boto:auth_path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.898 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.898 DEBUG:boto:Path: /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.899 DEBUG:boto:Data: 2023-10-18T20:01:19.899 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.900 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.900 DEBUG:boto:Params: {} 2023-10-18T20:01:19.901 DEBUG:boto:Token: None 2023-10-18T20:01:19.901 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.902 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OKfGNbqvxFyvKtRCXsB2UFRF1Nk= 2023-10-18T20:01:19.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OKfGNbqvxFyvKtRCXsB2UFRF1Nk=', 'Content-Length': '0'} 2023-10-18T20:01:19.904 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:04 GMT'), ('x-amz-version-id', 'dNZsMCWPZgrqoYBBpKsJustaw7SI.bg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044e3b4056ef22759-006530398f-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.904 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:19.905 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:19.905 DEBUG:boto:path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.905 DEBUG:boto:auth_path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.906 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.906 DEBUG:boto:Path: /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.907 DEBUG:boto:Data: 2023-10-18T20:01:19.907 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.908 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.908 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.909 DEBUG:boto:Params: {} 2023-10-18T20:01:19.909 DEBUG:boto:Token: None 2023-10-18T20:01:19.910 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.910 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OKfGNbqvxFyvKtRCXsB2UFRF1Nk= 2023-10-18T20:01:19.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OKfGNbqvxFyvKtRCXsB2UFRF1Nk=', 'Content-Length': '0'} 2023-10-18T20:01:19.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:04 GMT'), ('x-amz-version-id', 'dNZsMCWPZgrqoYBBpKsJustaw7SI.bg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e3180a46f24b3da-006530398f-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:01:19.913 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:19.914 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:19.914 DEBUG:boto:path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.915 DEBUG:boto:auth_path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.916 DEBUG:boto:Method: GET 2023-10-18T20:01:19.916 DEBUG:boto:Path: /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.916 DEBUG:boto:Data: 2023-10-18T20:01:19.917 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.918 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.918 DEBUG:boto:Params: {} 2023-10-18T20:01:19.919 DEBUG:boto:Token: None 2023-10-18T20:01:19.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E42ldxZeBzHeNSmzC6ZH57IGQSE= 2023-10-18T20:01:19.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E42ldxZeBzHeNSmzC6ZH57IGQSE=', 'Content-Length': '0'} 2023-10-18T20:01:19.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:04 GMT'), ('x-amz-version-id', 'dNZsMCWPZgrqoYBBpKsJustaw7SI.bg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec5dcdc6536dd377-006530398f-1116-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.922 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:19.923 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:19.923 DEBUG:boto:path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.924 DEBUG:boto:auth_path=/bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.924 DEBUG:boto:Method: GET 2023-10-18T20:01:19.925 DEBUG:boto:Path: /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.925 DEBUG:boto:Data: 2023-10-18T20:01:19.926 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.926 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.927 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.927 DEBUG:boto:Params: {} 2023-10-18T20:01:19.927 DEBUG:boto:Token: None 2023-10-18T20:01:19.928 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/obj?versionId=dNZsMCWPZgrqoYBBpKsJustaw7SI.bg 2023-10-18T20:01:19.928 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E42ldxZeBzHeNSmzC6ZH57IGQSE= 2023-10-18T20:01:19.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E42ldxZeBzHeNSmzC6ZH57IGQSE=', 'Content-Length': '0'} 2023-10-18T20:01:19.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:01:04 GMT'), ('x-amz-version-id', 'dNZsMCWPZgrqoYBBpKsJustaw7SI.bg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000014acffdc00efe073-006530398f-1112-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.931 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:19.932 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:19.932 DEBUG:boto:path=/bofrnf-16/?prefix=obj 2023-10-18T20:01:19.932 DEBUG:boto:auth_path=/bofrnf-16/?prefix=obj 2023-10-18T20:01:19.933 DEBUG:boto:Method: GET 2023-10-18T20:01:19.933 DEBUG:boto:Path: /bofrnf-16/?prefix=obj 2023-10-18T20:01:19.934 DEBUG:boto:Data: 2023-10-18T20:01:19.934 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.935 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.935 DEBUG:boto:Params: {} 2023-10-18T20:01:19.936 DEBUG:boto:Token: None 2023-10-18T20:01:19.936 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/ 2023-10-18T20:01:19.937 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gOJA4OkQLRKVFx/WyqyIrukqZow= 2023-10-18T20:01:19.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gOJA4OkQLRKVFx/WyqyIrukqZow=', 'Content-Length': '0'} 2023-10-18T20:01:19.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d3c2df2bee900e1-006530398f-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.982 DEBUG:boto:b'bofrnf-16obj1000false' 2023-10-18T20:01:19.983 DEBUG:boto:path=/bofrnf-16/ 2023-10-18T20:01:19.984 DEBUG:boto:auth_path=/bofrnf-16/ 2023-10-18T20:01:19.984 DEBUG:boto:path=/bofrnf-16/?prefix=obj 2023-10-18T20:01:19.985 DEBUG:boto:auth_path=/bofrnf-16/?prefix=obj 2023-10-18T20:01:19.985 DEBUG:boto:Method: GET 2023-10-18T20:01:19.985 DEBUG:boto:Path: /bofrnf-16/?prefix=obj 2023-10-18T20:01:19.986 DEBUG:boto:Data: 2023-10-18T20:01:19.986 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.987 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:19.987 DEBUG:boto:Port: 8000 2023-10-18T20:01:19.988 DEBUG:boto:Params: {} 2023-10-18T20:01:19.988 DEBUG:boto:Token: None 2023-10-18T20:01:19.989 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/ 2023-10-18T20:01:19.989 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gOJA4OkQLRKVFx/WyqyIrukqZow= 2023-10-18T20:01:19.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gOJA4OkQLRKVFx/WyqyIrukqZow=', 'Content-Length': '0'} 2023-10-18T20:01:19.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dec1caec0edf22d8-006530398f-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:19.992 DEBUG:boto:b'bofrnf-16obj1000false' 2023-10-18T20:01:19.993 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:19.993 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:19.993 DEBUG:boto:Method: HEAD 2023-10-18T20:01:19.994 DEBUG:boto:Path: /bofrnf-16/obj 2023-10-18T20:01:19.994 DEBUG:boto:Data: 2023-10-18T20:01:19.995 DEBUG:boto:Headers: {} 2023-10-18T20:01:19.995 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:19.996 DEBUG:boto:Port: 8001 2023-10-18T20:01:19.996 DEBUG:boto:Params: {} 2023-10-18T20:01:19.996 DEBUG:boto:Token: None 2023-10-18T20:01:19.997 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:19 GMT /bofrnf-16/obj 2023-10-18T20:01:19.997 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sQYE4GtTAMBfCWUDzpUaERUDK6U= 2023-10-18T20:01:19.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sQYE4GtTAMBfCWUDzpUaERUDK6U=', 'Content-Length': '0'} 2023-10-18T20:01:20.000 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ae234d58e6cba689-006530398f-1116-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:20.000 DEBUG:boto:path=/bofrnf-16/obj 2023-10-18T20:01:20.001 DEBUG:boto:auth_path=/bofrnf-16/obj 2023-10-18T20:01:20.001 DEBUG:boto:Method: HEAD 2023-10-18T20:01:20.002 DEBUG:boto:Path: /bofrnf-16/obj 2023-10-18T20:01:20.002 DEBUG:boto:Data: 2023-10-18T20:01:20.002 DEBUG:boto:Headers: {} 2023-10-18T20:01:20.003 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:20.003 DEBUG:boto:Port: 8000 2023-10-18T20:01:20.004 DEBUG:boto:Params: {} 2023-10-18T20:01:20.004 DEBUG:boto:Token: None 2023-10-18T20:01:20.005 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:20 GMT /bofrnf-16/obj 2023-10-18T20:01:20.005 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:K/nqLg6ONMxb9SfKOqPS/z1mO2E= 2023-10-18T20:01:20.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:K/nqLg6ONMxb9SfKOqPS/z1mO2E=', 'Content-Length': '0'} 2023-10-18T20:01:20.008 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000192f1a2b62022e8e-0065303990-1112-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:01:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:20.009 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-16 zones={a2, a1} 2023-10-18T20:01:20.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-10-18T20:01:20.011 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-17 2023-10-18T20:01:20.011 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:01:20.012 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:01:20.012 DEBUG:boto:Method: PUT 2023-10-18T20:01:20.013 DEBUG:boto:Path: /bofrnf-17/ 2023-10-18T20:01:20.013 DEBUG:boto:Data: 2023-10-18T20:01:20.013 DEBUG:boto:Headers: {} 2023-10-18T20:01:20.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:20.014 DEBUG:boto:Port: 8000 2023-10-18T20:01:20.015 DEBUG:boto:Params: {} 2023-10-18T20:01:20.015 DEBUG:boto:Token: None 2023-10-18T20:01:20.015 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:01:20 GMT /bofrnf-17/ 2023-10-18T20:01:20.016 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fq0KWXTz3CtRMVJut1S3mhWVeAw= 2023-10-18T20:01:20.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fq0KWXTz3CtRMVJut1S3mhWVeAw=', 'Content-Length': '0'} 2023-10-18T20:01:20.029 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6fce287cd07dc33-0065303990-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:20.030 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-18 2023-10-18T20:01:20.030 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:01:20.031 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:01:20.031 DEBUG:boto:Method: PUT 2023-10-18T20:01:20.032 DEBUG:boto:Path: /bofrnf-18/ 2023-10-18T20:01:20.032 DEBUG:boto:Data: 2023-10-18T20:01:20.032 DEBUG:boto:Headers: {} 2023-10-18T20:01:20.033 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:20.033 DEBUG:boto:Port: 8001 2023-10-18T20:01:20.034 DEBUG:boto:Params: {} 2023-10-18T20:01:20.034 DEBUG:boto:Token: None 2023-10-18T20:01:20.035 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:01:20 GMT /bofrnf-18/ 2023-10-18T20:01:20.035 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXt20CQUH88W4GoFdsgaV7jQFR0= 2023-10-18T20:01:20.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXt20CQUH88W4GoFdsgaV7jQFR0=', 'Content-Length': '0'} 2023-10-18T20:01:20.057 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8716b9adadd3cc7-0065303990-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:20.058 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:01:20.058 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:01:20.059 DEBUG:boto:path=/bofrnf-17/?versioning 2023-10-18T20:01:20.059 DEBUG:boto:auth_path=/bofrnf-17/?versioning 2023-10-18T20:01:20.060 DEBUG:boto:Method: PUT 2023-10-18T20:01:20.060 DEBUG:boto:Path: /bofrnf-17/?versioning 2023-10-18T20:01:20.060 DEBUG:boto:Data: Enabled Disabled 2023-10-18T20:01:20.061 DEBUG:boto:Headers: {} 2023-10-18T20:01:20.061 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:20.062 DEBUG:boto:Port: 8000 2023-10-18T20:01:20.062 DEBUG:boto:Params: {} 2023-10-18T20:01:20.063 DEBUG:boto:Token: None 2023-10-18T20:01:20.063 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:01:20 GMT /bofrnf-17/?versioning 2023-10-18T20:01:20.064 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tQ8QwU+TxHed84GacMyQ3tivrRw= 2023-10-18T20:01:20.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tQ8QwU+TxHed84GacMyQ3tivrRw=', 'Content-Length': '228'} 2023-10-18T20:01:20.069 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000632760564faf88ce-0065303990-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:20.070 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:01:20.070 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:01:20.071 DEBUG:boto:path=/bofrnf-17/?versioning 2023-10-18T20:01:20.071 DEBUG:boto:auth_path=/bofrnf-17/?versioning 2023-10-18T20:01:20.072 DEBUG:boto:Method: PUT 2023-10-18T20:01:20.072 DEBUG:boto:Path: /bofrnf-17/?versioning 2023-10-18T20:01:20.072 DEBUG:boto:Data: Suspended Disabled 2023-10-18T20:01:20.073 DEBUG:boto:Headers: {} 2023-10-18T20:01:20.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:20.074 DEBUG:boto:Port: 8000 2023-10-18T20:01:20.074 DEBUG:boto:Params: {} 2023-10-18T20:01:20.075 DEBUG:boto:Token: None 2023-10-18T20:01:20.075 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:01:20 GMT /bofrnf-17/?versioning 2023-10-18T20:01:20.075 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tQ8QwU+TxHed84GacMyQ3tivrRw= 2023-10-18T20:01:20.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tQ8QwU+TxHed84GacMyQ3tivrRw=', 'Content-Length': '230'} 2023-10-18T20:01:20.081 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cf783d4f8024ccc7-0065303990-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:20.081 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:01:20.081 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:01:20.082 DEBUG:boto:path=/bofrnf-18/?versioning 2023-10-18T20:01:20.082 DEBUG:boto:auth_path=/bofrnf-18/?versioning 2023-10-18T20:01:20.083 DEBUG:boto:Method: PUT 2023-10-18T20:01:20.083 DEBUG:boto:Path: /bofrnf-18/?versioning 2023-10-18T20:01:20.083 DEBUG:boto:Data: Enabled Disabled 2023-10-18T20:01:20.084 DEBUG:boto:Headers: {} 2023-10-18T20:01:20.084 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:20.085 DEBUG:boto:Port: 8001 2023-10-18T20:01:20.085 DEBUG:boto:Params: {} 2023-10-18T20:01:20.085 DEBUG:boto:Token: None 2023-10-18T20:01:20.086 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:01:20 GMT /bofrnf-18/?versioning 2023-10-18T20:01:20.087 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gw8aHp6CBDW3M2hDLXzL+BcTBto= 2023-10-18T20:01:20.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gw8aHp6CBDW3M2hDLXzL+BcTBto=', 'Content-Length': '228'} 2023-10-18T20:01:20.094 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000efb1a826b26a16a0-0065303990-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:20.095 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:01:20.095 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:01:20.096 DEBUG:boto:path=/bofrnf-18/?versioning 2023-10-18T20:01:20.096 DEBUG:boto:auth_path=/bofrnf-18/?versioning 2023-10-18T20:01:20.096 DEBUG:boto:Method: PUT 2023-10-18T20:01:20.097 DEBUG:boto:Path: /bofrnf-18/?versioning 2023-10-18T20:01:20.097 DEBUG:boto:Data: Suspended Disabled 2023-10-18T20:01:20.097 DEBUG:boto:Headers: {} 2023-10-18T20:01:20.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:20.098 DEBUG:boto:Port: 8001 2023-10-18T20:01:20.099 DEBUG:boto:Params: {} 2023-10-18T20:01:20.099 DEBUG:boto:Token: None 2023-10-18T20:01:20.099 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:01:20 GMT /bofrnf-18/?versioning 2023-10-18T20:01:20.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gw8aHp6CBDW3M2hDLXzL+BcTBto= 2023-10-18T20:01:20.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gw8aHp6CBDW3M2hDLXzL+BcTBto=', 'Content-Length': '230'} 2023-10-18T20:01:20.107 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cabf5a31067a859e-0065303990-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:20.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:01:20.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:01:20.109 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:01:20.159 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:20.159 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:20.190 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:01:20.204 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:01:20.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:20.205 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659280.104653_275.1", 2023-10-18T20:01:20.205 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:01:20.104653Z" 2023-10-18T20:01:20.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:20.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:20.206 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:01:20.206 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:01:20.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:20.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:20.207 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:01:20.207 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:01:20.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:20.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:20.208 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:01:20.208 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:01:20.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:20.209 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:01:20.210 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659280.104653_275.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:01:20.210 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:01:20.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:01:20.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:01:20.212 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:01:20.261 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:20.262 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:20.290 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:20.287+0000 7fc664f38780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:01:20.291 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:20.288+0000 7fc664f38780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:01:20.302 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:01:20.302 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:01:20.302 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:01:20.302 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:01:20.303 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:01:20.303 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:01:20.303 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:20.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:20.303 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:01:20.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:20.304 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:01:20.304 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:20.305 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:20.305 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659280.021986_256.1", 2023-10-18T20:01:20.305 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:20.305 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:01:20.305 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:20.306 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:01:20.021986Z", 2023-10-18T20:01:20.306 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:20.306 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:20.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:20.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:20.307 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:01:20.307 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:20.307 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:20.307 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:01:20.308 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:20.308 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:01:20.308 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:20.308 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:01:20.309 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:20.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:20.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:20.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:20.310 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:01:20.310 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:20.310 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:20.311 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:01:20.311 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:20.311 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:01:20.311 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:20.311 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:01:20.312 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:20.312 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:20.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:20.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:20.312 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:01:20.313 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:20.313 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:20.313 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:01:20.313 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:20.314 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:01:20.314 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:20.314 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:01:20.314 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:20.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:20.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:20.315 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:01:20.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:20.316 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:01:20.317 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:01:20.317 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:01:20.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:20.318 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:01:20Z" 2023-10-18T20:01:20.318 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:01:20.319 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659280.021986_256.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:01:20.021986Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:01:20Z" } 2023-10-18T20:01:20.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659280.021986_256.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:01:20.021986Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:01:20.321 DEBUG:rgw_multi.tests:log_status={0: '1_1697659280.104653_275.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:01:20.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659280.021986_256.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:01:20.321 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659280.104653_275.1 target=1_1697659280.021986_256.1 2023-10-18T20:01:25.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:01:25.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:01:25.324 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:01:25.381 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:25.381 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:25.413 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:25.410+0000 7fca02a58780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:01:25.413 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:25.411+0000 7fca02a58780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:01:25.426 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:01:25.426 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:01:25.426 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:01:25.427 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:01:25.427 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:01:25.427 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:01:25.427 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:25.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:25.428 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:01:25.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:25.428 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:01:25.428 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:25.429 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:25.429 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659280.104653_275.1", 2023-10-18T20:01:25.429 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:25.429 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:01:25.430 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:25.430 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:01:20.104653Z", 2023-10-18T20:01:25.430 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:25.430 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:25.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:25.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:25.431 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:01:25.431 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:25.431 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:25.432 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:01:25.432 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:25.432 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:01:25.432 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:25.432 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:01:25.433 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:25.433 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:25.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:25.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:25.433 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:01:25.434 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:25.434 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:25.434 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:01:25.434 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:25.435 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:01:25.435 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:25.435 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:01:25.435 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:25.435 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:25.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:25.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:01:25.436 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:01:25.436 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:01:25.436 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:01:25.437 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:01:25.437 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:01:25.437 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:01:25.437 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:01:25.437 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:01:25.438 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:01:25.438 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:25.438 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:01:25.439 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:01:25.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:25.439 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:01:25.439 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:01:25.439 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:01:25.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:01:25.440 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:01:25Z" 2023-10-18T20:01:25.440 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:01:25.441 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659280.104653_275.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:01:20.104653Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:01:25Z" } 2023-10-18T20:01:25.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659280.104653_275.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:01:20.104653Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:01:25.442 DEBUG:rgw_multi.tests:log_status={0: '1_1697659280.104653_275.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:01:25.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659280.104653_275.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:01:25.443 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:01:25.444 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:01:25.444 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:01:25.444 DEBUG:boto:Method: HEAD 2023-10-18T20:01:25.445 DEBUG:boto:Path: /bofrnf-17/ 2023-10-18T20:01:25.445 DEBUG:boto:Data: 2023-10-18T20:01:25.446 DEBUG:boto:Headers: {} 2023-10-18T20:01:25.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:25.446 DEBUG:boto:Port: 8000 2023-10-18T20:01:25.447 DEBUG:boto:Params: {} 2023-10-18T20:01:25.447 DEBUG:boto:Token: None 2023-10-18T20:01:25.448 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:25 GMT /bofrnf-17/ 2023-10-18T20:01:25.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tf9dc8lCSmaJbOzPjUatRmIHpSM= 2023-10-18T20:01:25.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tf9dc8lCSmaJbOzPjUatRmIHpSM=', 'Content-Length': '0'} 2023-10-18T20:01:25.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ad30d2820bf9f80-0065303995-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:25.452 DEBUG:boto:path=/bofrnf-17/obj 2023-10-18T20:01:25.452 DEBUG:boto:auth_path=/bofrnf-17/obj 2023-10-18T20:01:25.453 DEBUG:boto:Method: PUT 2023-10-18T20:01:25.453 DEBUG:boto:Path: /bofrnf-17/obj 2023-10-18T20:01:25.454 DEBUG:boto:Data: 2023-10-18T20:01:25.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T20:01:25.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:25.455 DEBUG:boto:Port: 8000 2023-10-18T20:01:25.455 DEBUG:boto:Params: {} 2023-10-18T20:01:25.456 DEBUG:boto:Token: None 2023-10-18T20:01:25.456 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 20:01:25 GMT /bofrnf-17/obj 2023-10-18T20:01:25.457 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:su77XJUry9t8Yo3JSIMa4ED2UBU= 2023-10-18T20:01:25.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:01:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:su77XJUry9t8Yo3JSIMa4ED2UBU='} 2023-10-18T20:01:25.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4eae18f9d54fc74-0065303995-1112-a1'), ('Date', 'Wed, 18 Oct 2023 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:25.464 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:01:25.464 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:01:25.465 DEBUG:boto:Method: HEAD 2023-10-18T20:01:25.465 DEBUG:boto:Path: /bofrnf-17/ 2023-10-18T20:01:25.466 DEBUG:boto:Data: 2023-10-18T20:01:25.466 DEBUG:boto:Headers: {} 2023-10-18T20:01:25.467 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:25.467 DEBUG:boto:Port: 8000 2023-10-18T20:01:25.468 DEBUG:boto:Params: {} 2023-10-18T20:01:25.468 DEBUG:boto:Token: None 2023-10-18T20:01:25.469 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:25 GMT /bofrnf-17/ 2023-10-18T20:01:25.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tf9dc8lCSmaJbOzPjUatRmIHpSM= 2023-10-18T20:01:25.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tf9dc8lCSmaJbOzPjUatRmIHpSM=', 'Content-Length': '0'} 2023-10-18T20:01:25.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003592d769a0b0cd6-0065303995-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:25.472 DEBUG:boto:path=/bofrnf-17/obj 2023-10-18T20:01:25.473 DEBUG:boto:auth_path=/bofrnf-17/obj 2023-10-18T20:01:25.473 DEBUG:boto:Method: DELETE 2023-10-18T20:01:25.474 DEBUG:boto:Path: /bofrnf-17/obj 2023-10-18T20:01:25.474 DEBUG:boto:Data: 2023-10-18T20:01:25.474 DEBUG:boto:Headers: {} 2023-10-18T20:01:25.475 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:01:25.475 DEBUG:boto:Port: 8000 2023-10-18T20:01:25.476 DEBUG:boto:Params: {} 2023-10-18T20:01:25.476 DEBUG:boto:Token: None 2023-10-18T20:01:25.477 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 20:01:25 GMT /bofrnf-17/obj 2023-10-18T20:01:25.477 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iKQsfSocT6qtwMnlW7idEVEiG6I= 2023-10-18T20:01:25.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iKQsfSocT6qtwMnlW7idEVEiG6I=', 'Content-Length': '0'} 2023-10-18T20:01:25.486 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000d02f42ad5ccb74af-0065303995-1112-a1'), ('Date', 'Wed, 18 Oct 2023 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:25.486 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:01:25.487 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:01:25.488 DEBUG:boto:Method: HEAD 2023-10-18T20:01:25.488 DEBUG:boto:Path: /bofrnf-18/ 2023-10-18T20:01:25.488 DEBUG:boto:Data: 2023-10-18T20:01:25.489 DEBUG:boto:Headers: {} 2023-10-18T20:01:25.489 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:25.490 DEBUG:boto:Port: 8001 2023-10-18T20:01:25.490 DEBUG:boto:Params: {} 2023-10-18T20:01:25.491 DEBUG:boto:Token: None 2023-10-18T20:01:25.491 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:25 GMT /bofrnf-18/ 2023-10-18T20:01:25.492 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hqLPsyQRvOeBXqrnKE5tpAsqeEg= 2023-10-18T20:01:25.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hqLPsyQRvOeBXqrnKE5tpAsqeEg=', 'Content-Length': '0'} 2023-10-18T20:01:25.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2fd8778999f2ad4-0065303995-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:25.495 DEBUG:boto:path=/bofrnf-18/obj 2023-10-18T20:01:25.496 DEBUG:boto:auth_path=/bofrnf-18/obj 2023-10-18T20:01:25.496 DEBUG:boto:Method: PUT 2023-10-18T20:01:25.496 DEBUG:boto:Path: /bofrnf-18/obj 2023-10-18T20:01:25.497 DEBUG:boto:Data: 2023-10-18T20:01:25.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-18T20:01:25.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:25.498 DEBUG:boto:Port: 8001 2023-10-18T20:01:25.499 DEBUG:boto:Params: {} 2023-10-18T20:01:25.499 DEBUG:boto:Token: None 2023-10-18T20:01:25.500 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 18 Oct 2023 20:01:25 GMT /bofrnf-18/obj 2023-10-18T20:01:25.500 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wmaq6MRMhPqsT4fnLNkmTMYRy5k= 2023-10-18T20:01:25.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:01:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wmaq6MRMhPqsT4fnLNkmTMYRy5k='} 2023-10-18T20:01:25.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8d1835b62cb39d6-0065303995-1116-a2'), ('Date', 'Wed, 18 Oct 2023 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:25.507 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:01:25.507 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:01:25.508 DEBUG:boto:Method: HEAD 2023-10-18T20:01:25.508 DEBUG:boto:Path: /bofrnf-18/ 2023-10-18T20:01:25.509 DEBUG:boto:Data: 2023-10-18T20:01:25.509 DEBUG:boto:Headers: {} 2023-10-18T20:01:25.510 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:25.510 DEBUG:boto:Port: 8001 2023-10-18T20:01:25.511 DEBUG:boto:Params: {} 2023-10-18T20:01:25.511 DEBUG:boto:Token: None 2023-10-18T20:01:25.511 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:01:25 GMT /bofrnf-18/ 2023-10-18T20:01:25.512 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hqLPsyQRvOeBXqrnKE5tpAsqeEg= 2023-10-18T20:01:25.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hqLPsyQRvOeBXqrnKE5tpAsqeEg=', 'Content-Length': '0'} 2023-10-18T20:01:25.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f745b145a3dfe002-0065303995-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:25.515 DEBUG:boto:path=/bofrnf-18/obj 2023-10-18T20:01:25.516 DEBUG:boto:auth_path=/bofrnf-18/obj 2023-10-18T20:01:25.516 DEBUG:boto:Method: DELETE 2023-10-18T20:01:25.517 DEBUG:boto:Path: /bofrnf-18/obj 2023-10-18T20:01:25.517 DEBUG:boto:Data: 2023-10-18T20:01:25.517 DEBUG:boto:Headers: {} 2023-10-18T20:01:25.518 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:01:25.518 DEBUG:boto:Port: 8001 2023-10-18T20:01:25.519 DEBUG:boto:Params: {} 2023-10-18T20:01:25.519 DEBUG:boto:Token: None 2023-10-18T20:01:25.520 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 20:01:25 GMT /bofrnf-18/obj 2023-10-18T20:01:25.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:K/BThaplvCADzRfNz3gVWpH4lJw= 2023-10-18T20:01:25.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:01:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:K/BThaplvCADzRfNz3gVWpH4lJw=', 'Content-Length': '0'} 2023-10-18T20:01:25.529 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000010843ee1f26825eb-0065303995-1116-a2'), ('Date', 'Wed, 18 Oct 2023 20:01:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:01:25.529 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-17 2023-10-18T20:01:25.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:25.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:25.531 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:01:25.587 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:25.587 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:25.621 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:25.619+0000 7fc74f7cb780 1 waiting to reach incremental sync.. 2023-10-18T20:01:30.622 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:30.621+0000 7fc74f7cb780 1 waiting to reach incremental sync.. 2023-10-18T20:01:35.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:35.621+0000 7fc74f7cb780 1 waiting to reach incremental sync.. 2023-10-18T20:01:40.625 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:40.623+0000 7fc74f7cb780 1 bucket sync caught up with source: 2023-10-18T20:01:40.625 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.7.7, , ] 2023-10-18T20:01:40.625 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.7.7, , ] 2023-10-18T20:01:40.626 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:40.623+0000 7fc74f7cb780 0 bucket checkpoint complete 2023-10-18T20:01:40.641 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-17 2023-10-18T20:01:40.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:40.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:01:40.642 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:01:40.705 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:01:40.705 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:01:40.741 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:40.740+0000 7f8d789db780 1 waiting to reach incremental sync.. 2023-10-18T20:01:45.742 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:45.740+0000 7f8d789db780 1 waiting to reach incremental sync.. 2023-10-18T20:01:50.742 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:50.741+0000 7f8d789db780 1 waiting to reach incremental sync.. 2023-10-18T20:01:55.743 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:01:55.741+0000 7f8d789db780 1 waiting to reach incremental sync.. 2023-10-18T20:02:00.745 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:00.743+0000 7f8d789db780 1 bucket sync caught up with source: 2023-10-18T20:02:00.745 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.10.14, , ] 2023-10-18T20:02:00.746 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.10.14, , ] 2023-10-18T20:02:00.746 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:00.743+0000 7f8d789db780 0 bucket checkpoint complete 2023-10-18T20:02:00.761 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-17 zones={a2, a1} 2023-10-18T20:02:00.762 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:02:00.762 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:02:00.763 DEBUG:boto:Method: HEAD 2023-10-18T20:02:00.763 DEBUG:boto:Path: /bofrnf-17/ 2023-10-18T20:02:00.764 DEBUG:boto:Data: 2023-10-18T20:02:00.764 DEBUG:boto:Headers: {} 2023-10-18T20:02:00.764 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:00.765 DEBUG:boto:Port: 8001 2023-10-18T20:02:00.765 DEBUG:boto:Params: {} 2023-10-18T20:02:00.766 DEBUG:boto:Token: None 2023-10-18T20:02:00.766 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:00 GMT /bofrnf-17/ 2023-10-18T20:02:00.767 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J8yNOdEDSYpYrn/mmcmYX1wC1zE= 2023-10-18T20:02:00.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J8yNOdEDSYpYrn/mmcmYX1wC1zE=', 'Content-Length': '0'} 2023-10-18T20:02:00.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011529a9fc700e5ab-00653039b8-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:00.771 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:02:00.771 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:02:00.772 DEBUG:boto:Method: HEAD 2023-10-18T20:02:00.772 DEBUG:boto:Path: /bofrnf-17/ 2023-10-18T20:02:00.772 DEBUG:boto:Data: 2023-10-18T20:02:00.773 DEBUG:boto:Headers: {} 2023-10-18T20:02:00.773 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:00.774 DEBUG:boto:Port: 8000 2023-10-18T20:02:00.774 DEBUG:boto:Params: {} 2023-10-18T20:02:00.774 DEBUG:boto:Token: None 2023-10-18T20:02:00.775 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:00 GMT /bofrnf-17/ 2023-10-18T20:02:00.775 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J8yNOdEDSYpYrn/mmcmYX1wC1zE= 2023-10-18T20:02:00.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J8yNOdEDSYpYrn/mmcmYX1wC1zE=', 'Content-Length': '0'} 2023-10-18T20:02:00.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6b1e1d855ffdaab-00653039b8-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:00.780 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:02:00.780 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:02:00.781 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:02:00.781 DEBUG:boto:path=/bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.781 DEBUG:boto:auth_path=/bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.782 DEBUG:boto:Method: GET 2023-10-18T20:02:00.782 DEBUG:boto:Path: /bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.783 DEBUG:boto:Data: 2023-10-18T20:02:00.783 DEBUG:boto:Headers: {} 2023-10-18T20:02:00.783 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:00.784 DEBUG:boto:Port: 8001 2023-10-18T20:02:00.784 DEBUG:boto:Params: {} 2023-10-18T20:02:00.785 DEBUG:boto:Token: None 2023-10-18T20:02:00.785 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:00 GMT /bofrnf-17/?versions 2023-10-18T20:02:00.786 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gqhFdgBf1sANIL6Q0ukpO9qut4c= 2023-10-18T20:02:00.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gqhFdgBf1sANIL6Q0ukpO9qut4c=', 'Content-Length': '0'} 2023-10-18T20:02:00.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099e9f796b7564011-00653039b8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:00.789 DEBUG:boto:b'bofrnf-17999falseobjnulltrue2023-10-18T20:01:25.479Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:02:00.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:02:00.790 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:02:00.791 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:02:00.791 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:02:00.791 DEBUG:boto:path=/bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.792 DEBUG:boto:auth_path=/bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.792 DEBUG:boto:Method: GET 2023-10-18T20:02:00.793 DEBUG:boto:Path: /bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.793 DEBUG:boto:Data: 2023-10-18T20:02:00.794 DEBUG:boto:Headers: {} 2023-10-18T20:02:00.794 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:00.794 DEBUG:boto:Port: 8000 2023-10-18T20:02:00.795 DEBUG:boto:Params: {} 2023-10-18T20:02:00.795 DEBUG:boto:Token: None 2023-10-18T20:02:00.796 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:00 GMT /bofrnf-17/?versions 2023-10-18T20:02:00.796 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gqhFdgBf1sANIL6Q0ukpO9qut4c= 2023-10-18T20:02:00.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gqhFdgBf1sANIL6Q0ukpO9qut4c=', 'Content-Length': '0'} 2023-10-18T20:02:00.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006135e6adaea1229b-00653039b8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:00.842 DEBUG:boto:b'bofrnf-17999falseobjnulltrue2023-10-18T20:01:25.479Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:02:00.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:02:00.844 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:02:00.844 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:02:00.845 DEBUG:boto:path=/bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.845 DEBUG:boto:auth_path=/bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.845 DEBUG:boto:Method: GET 2023-10-18T20:02:00.846 DEBUG:boto:Path: /bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.846 DEBUG:boto:Data: 2023-10-18T20:02:00.847 DEBUG:boto:Headers: {} 2023-10-18T20:02:00.847 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:00.848 DEBUG:boto:Port: 8001 2023-10-18T20:02:00.848 DEBUG:boto:Params: {} 2023-10-18T20:02:00.848 DEBUG:boto:Token: None 2023-10-18T20:02:00.849 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:00 GMT /bofrnf-17/?versions 2023-10-18T20:02:00.849 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gqhFdgBf1sANIL6Q0ukpO9qut4c= 2023-10-18T20:02:00.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gqhFdgBf1sANIL6Q0ukpO9qut4c=', 'Content-Length': '0'} 2023-10-18T20:02:00.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aacfe327c1579c50-00653039b8-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:00.853 DEBUG:boto:b'bofrnf-17999falseobjnulltrue2023-10-18T20:01:25.479Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:02:00.854 DEBUG:boto:path=/bofrnf-17/ 2023-10-18T20:02:00.854 DEBUG:boto:auth_path=/bofrnf-17/ 2023-10-18T20:02:00.855 DEBUG:boto:path=/bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.855 DEBUG:boto:auth_path=/bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.855 DEBUG:boto:Method: GET 2023-10-18T20:02:00.856 DEBUG:boto:Path: /bofrnf-17/?versions&max-keys=999 2023-10-18T20:02:00.856 DEBUG:boto:Data: 2023-10-18T20:02:00.857 DEBUG:boto:Headers: {} 2023-10-18T20:02:00.857 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:00.857 DEBUG:boto:Port: 8000 2023-10-18T20:02:00.858 DEBUG:boto:Params: {} 2023-10-18T20:02:00.858 DEBUG:boto:Token: None 2023-10-18T20:02:00.859 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:00 GMT /bofrnf-17/?versions 2023-10-18T20:02:00.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gqhFdgBf1sANIL6Q0ukpO9qut4c= 2023-10-18T20:02:00.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gqhFdgBf1sANIL6Q0ukpO9qut4c=', 'Content-Length': '0'} 2023-10-18T20:02:00.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bca29008b6b50c48-00653039b8-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:00.863 DEBUG:boto:b'bofrnf-17999falseobjnulltrue2023-10-18T20:01:25.479Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:02:00.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:02:00.864 DEBUG:boto:path=/bofrnf-17/obj 2023-10-18T20:02:00.864 DEBUG:boto:auth_path=/bofrnf-17/obj 2023-10-18T20:02:00.865 DEBUG:boto:Method: HEAD 2023-10-18T20:02:00.865 DEBUG:boto:Path: /bofrnf-17/obj 2023-10-18T20:02:00.865 DEBUG:boto:Data: 2023-10-18T20:02:00.866 DEBUG:boto:Headers: {} 2023-10-18T20:02:00.866 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:00.867 DEBUG:boto:Port: 8001 2023-10-18T20:02:00.867 DEBUG:boto:Params: {} 2023-10-18T20:02:00.867 DEBUG:boto:Token: None 2023-10-18T20:02:00.868 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:00 GMT /bofrnf-17/obj 2023-10-18T20:02:00.868 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fwuczoGqknXV83vBVmnxsoFhqZg= 2023-10-18T20:02:00.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fwuczoGqknXV83vBVmnxsoFhqZg=', 'Content-Length': '0'} 2023-10-18T20:02:00.871 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ed5d26b8c7c16070-00653039b8-1116-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:00.871 DEBUG:boto:path=/bofrnf-17/obj 2023-10-18T20:02:00.871 DEBUG:boto:auth_path=/bofrnf-17/obj 2023-10-18T20:02:00.872 DEBUG:boto:Method: HEAD 2023-10-18T20:02:00.872 DEBUG:boto:Path: /bofrnf-17/obj 2023-10-18T20:02:00.873 DEBUG:boto:Data: 2023-10-18T20:02:00.873 DEBUG:boto:Headers: {} 2023-10-18T20:02:00.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:00.874 DEBUG:boto:Port: 8000 2023-10-18T20:02:00.874 DEBUG:boto:Params: {} 2023-10-18T20:02:00.875 DEBUG:boto:Token: None 2023-10-18T20:02:00.875 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:00 GMT /bofrnf-17/obj 2023-10-18T20:02:00.876 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fwuczoGqknXV83vBVmnxsoFhqZg= 2023-10-18T20:02:00.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fwuczoGqknXV83vBVmnxsoFhqZg=', 'Content-Length': '0'} 2023-10-18T20:02:00.878 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006f8b42c8016f6078-00653039b8-1112-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:00.878 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-17 zones={a2, a1} 2023-10-18T20:02:00.879 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-18 2023-10-18T20:02:00.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:02:00.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:02:00.880 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:02:00.944 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:00.944 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:00.984 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:00.982+0000 7f1eb6942780 1 bucket sync caught up with source: 2023-10-18T20:02:00.984 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.8.14, , ] 2023-10-18T20:02:00.984 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.8.14, , ] 2023-10-18T20:02:00.985 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:00.982+0000 7f1eb6942780 0 bucket checkpoint complete 2023-10-18T20:02:00.997 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-18 2023-10-18T20:02:00.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:02:00.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:02:00.998 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:02:01.055 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:01.055 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:01.090 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:01.088+0000 7f8d77bda780 1 bucket sync caught up with source: 2023-10-18T20:02:01.090 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000003.40.7, , ] 2023-10-18T20:02:01.091 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000003.40.7, , ] 2023-10-18T20:02:01.091 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:01.088+0000 7f8d77bda780 0 bucket checkpoint complete 2023-10-18T20:02:01.102 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-18 zones={a2, a1} 2023-10-18T20:02:01.103 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:02:01.103 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:02:01.103 DEBUG:boto:Method: HEAD 2023-10-18T20:02:01.104 DEBUG:boto:Path: /bofrnf-18/ 2023-10-18T20:02:01.104 DEBUG:boto:Data: 2023-10-18T20:02:01.105 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.105 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.105 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.106 DEBUG:boto:Params: {} 2023-10-18T20:02:01.106 DEBUG:boto:Token: None 2023-10-18T20:02:01.107 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-18/ 2023-10-18T20:02:01.107 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w4FA3YKklNZfV36/kS1YtTT02Zc= 2023-10-18T20:02:01.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w4FA3YKklNZfV36/kS1YtTT02Zc=', 'Content-Length': '0'} 2023-10-18T20:02:01.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021921644759fe162-00653039b9-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.111 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:02:01.111 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:02:01.111 DEBUG:boto:Method: HEAD 2023-10-18T20:02:01.112 DEBUG:boto:Path: /bofrnf-18/ 2023-10-18T20:02:01.112 DEBUG:boto:Data: 2023-10-18T20:02:01.113 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.113 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.113 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.114 DEBUG:boto:Params: {} 2023-10-18T20:02:01.114 DEBUG:boto:Token: None 2023-10-18T20:02:01.115 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-18/ 2023-10-18T20:02:01.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w4FA3YKklNZfV36/kS1YtTT02Zc= 2023-10-18T20:02:01.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w4FA3YKklNZfV36/kS1YtTT02Zc=', 'Content-Length': '0'} 2023-10-18T20:02:01.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002898fffa07379934-00653039b9-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.118 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:02:01.119 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:02:01.119 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:02:01.119 DEBUG:boto:path=/bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.120 DEBUG:boto:auth_path=/bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.120 DEBUG:boto:Method: GET 2023-10-18T20:02:01.121 DEBUG:boto:Path: /bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.121 DEBUG:boto:Data: 2023-10-18T20:02:01.121 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.122 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.123 DEBUG:boto:Params: {} 2023-10-18T20:02:01.123 DEBUG:boto:Token: None 2023-10-18T20:02:01.124 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-18/?versions 2023-10-18T20:02:01.124 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:101DUv652t8Aj+14XqpUvAK7qeU= 2023-10-18T20:02:01.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:101DUv652t8Aj+14XqpUvAK7qeU=', 'Content-Length': '0'} 2023-10-18T20:02:01.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018bf050e66ce3cb9-00653039b9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.127 DEBUG:boto:b'bofrnf-18999falseobjnulltrue2023-10-18T20:01:25.522Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:02:01.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:02:01.129 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:02:01.129 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:02:01.129 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:02:01.130 DEBUG:boto:path=/bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.130 DEBUG:boto:auth_path=/bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.131 DEBUG:boto:Method: GET 2023-10-18T20:02:01.131 DEBUG:boto:Path: /bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.131 DEBUG:boto:Data: 2023-10-18T20:02:01.132 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.132 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.133 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.133 DEBUG:boto:Params: {} 2023-10-18T20:02:01.133 DEBUG:boto:Token: None 2023-10-18T20:02:01.134 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-18/?versions 2023-10-18T20:02:01.134 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:101DUv652t8Aj+14XqpUvAK7qeU= 2023-10-18T20:02:01.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:101DUv652t8Aj+14XqpUvAK7qeU=', 'Content-Length': '0'} 2023-10-18T20:02:01.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a369631c2a30aee6-00653039b9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.138 DEBUG:boto:b'bofrnf-18999falseobjnulltrue2023-10-18T20:01:25.522Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:02:01.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-18T20:02:01.139 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:02:01.139 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:02:01.139 DEBUG:boto:path=/bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.140 DEBUG:boto:auth_path=/bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.140 DEBUG:boto:Method: GET 2023-10-18T20:02:01.141 DEBUG:boto:Path: /bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.141 DEBUG:boto:Data: 2023-10-18T20:02:01.141 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.142 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.143 DEBUG:boto:Params: {} 2023-10-18T20:02:01.143 DEBUG:boto:Token: None 2023-10-18T20:02:01.144 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-18/?versions 2023-10-18T20:02:01.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:101DUv652t8Aj+14XqpUvAK7qeU= 2023-10-18T20:02:01.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:101DUv652t8Aj+14XqpUvAK7qeU=', 'Content-Length': '0'} 2023-10-18T20:02:01.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000751ea500094c7260-00653039b9-1116-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.147 DEBUG:boto:b'bofrnf-18999falseobjnulltrue2023-10-18T20:01:25.522Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:02:01.148 DEBUG:boto:path=/bofrnf-18/ 2023-10-18T20:02:01.148 DEBUG:boto:auth_path=/bofrnf-18/ 2023-10-18T20:02:01.149 DEBUG:boto:path=/bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.149 DEBUG:boto:auth_path=/bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.150 DEBUG:boto:Method: GET 2023-10-18T20:02:01.150 DEBUG:boto:Path: /bofrnf-18/?versions&max-keys=999 2023-10-18T20:02:01.150 DEBUG:boto:Data: 2023-10-18T20:02:01.151 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.152 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.152 DEBUG:boto:Params: {} 2023-10-18T20:02:01.152 DEBUG:boto:Token: None 2023-10-18T20:02:01.153 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-18/?versions 2023-10-18T20:02:01.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:101DUv652t8Aj+14XqpUvAK7qeU= 2023-10-18T20:02:01.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:101DUv652t8Aj+14XqpUvAK7qeU=', 'Content-Length': '0'} 2023-10-18T20:02:01.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040104622d89935c7-00653039b9-1112-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.157 DEBUG:boto:b'bofrnf-18999falseobjnulltrue2023-10-18T20:01:25.522Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:02:01.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-18T20:02:01.158 DEBUG:boto:path=/bofrnf-18/obj 2023-10-18T20:02:01.158 DEBUG:boto:auth_path=/bofrnf-18/obj 2023-10-18T20:02:01.158 DEBUG:boto:Method: HEAD 2023-10-18T20:02:01.159 DEBUG:boto:Path: /bofrnf-18/obj 2023-10-18T20:02:01.159 DEBUG:boto:Data: 2023-10-18T20:02:01.160 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.160 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.160 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.161 DEBUG:boto:Params: {} 2023-10-18T20:02:01.161 DEBUG:boto:Token: None 2023-10-18T20:02:01.162 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-18/obj 2023-10-18T20:02:01.162 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yuvp7vw37yGuVxG6tahblHrjGVk= 2023-10-18T20:02:01.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yuvp7vw37yGuVxG6tahblHrjGVk=', 'Content-Length': '0'} 2023-10-18T20:02:01.164 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000a95ac0ba64dea54-00653039b9-1116-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.165 DEBUG:boto:path=/bofrnf-18/obj 2023-10-18T20:02:01.165 DEBUG:boto:auth_path=/bofrnf-18/obj 2023-10-18T20:02:01.166 DEBUG:boto:Method: HEAD 2023-10-18T20:02:01.166 DEBUG:boto:Path: /bofrnf-18/obj 2023-10-18T20:02:01.166 DEBUG:boto:Data: 2023-10-18T20:02:01.167 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.167 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.168 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.168 DEBUG:boto:Params: {} 2023-10-18T20:02:01.168 DEBUG:boto:Token: None 2023-10-18T20:02:01.169 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-18/obj 2023-10-18T20:02:01.169 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yuvp7vw37yGuVxG6tahblHrjGVk= 2023-10-18T20:02:01.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yuvp7vw37yGuVxG6tahblHrjGVk=', 'Content-Length': '0'} 2023-10-18T20:02:01.172 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c49ef1972b9574d1-00653039b9-1112-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-18 zones={a2, a1} 2023-10-18T20:02:01.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-10-18T20:02:01.174 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-19 2023-10-18T20:02:01.174 DEBUG:boto:path=/bofrnf-19/ 2023-10-18T20:02:01.175 DEBUG:boto:auth_path=/bofrnf-19/ 2023-10-18T20:02:01.175 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.175 DEBUG:boto:Path: /bofrnf-19/ 2023-10-18T20:02:01.176 DEBUG:boto:Data: 2023-10-18T20:02:01.176 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.177 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.177 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.177 DEBUG:boto:Params: {} 2023-10-18T20:02:01.178 DEBUG:boto:Token: None 2023-10-18T20:02:01.178 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-19/ 2023-10-18T20:02:01.179 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/gHtED2tNNH1Y3Ya77b1vXwIhZU= 2023-10-18T20:02:01.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/gHtED2tNNH1Y3Ya77b1vXwIhZU=', 'Content-Length': '0'} 2023-10-18T20:02:01.196 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a000d1ad2b8ad451-00653039b9-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.197 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-20 2023-10-18T20:02:01.197 DEBUG:boto:path=/bofrnf-20/ 2023-10-18T20:02:01.197 DEBUG:boto:auth_path=/bofrnf-20/ 2023-10-18T20:02:01.198 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.198 DEBUG:boto:Path: /bofrnf-20/ 2023-10-18T20:02:01.199 DEBUG:boto:Data: 2023-10-18T20:02:01.199 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.200 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.200 DEBUG:boto:Params: {} 2023-10-18T20:02:01.201 DEBUG:boto:Token: None 2023-10-18T20:02:01.201 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-20/ 2023-10-18T20:02:01.202 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QTT8JAHNNJ7uG/pj8kXCv2rOIr4= 2023-10-18T20:02:01.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QTT8JAHNNJ7uG/pj8kXCv2rOIr4=', 'Content-Length': '0'} 2023-10-18T20:02:01.229 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000045f1214f1bc05c2c-00653039b9-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.230 DEBUG:boto:Using access key provided by client. 2023-10-18T20:02:01.231 DEBUG:boto:Using secret key provided by client. 2023-10-18T20:02:01.232 DEBUG:boto:Using access key provided by client. 2023-10-18T20:02:01.232 DEBUG:boto:Using secret key provided by client. 2023-10-18T20:02:01.233 DEBUG:boto:Using access key provided by client. 2023-10-18T20:02:01.233 DEBUG:boto:Using secret key provided by client. 2023-10-18T20:02:01.234 DEBUG:boto:Using access key provided by client. 2023-10-18T20:02:01.234 DEBUG:boto:Using secret key provided by client. 2023-10-18T20:02:01.235 INFO:rgw_multi.tests:create bucket zone=b1 name=bofrnf-21 2023-10-18T20:02:01.235 DEBUG:boto:path=/bofrnf-21/ 2023-10-18T20:02:01.236 DEBUG:boto:auth_path=/bofrnf-21/ 2023-10-18T20:02:01.236 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.237 DEBUG:boto:Path: /bofrnf-21/ 2023-10-18T20:02:01.237 DEBUG:boto:Data: 2023-10-18T20:02:01.237 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.238 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:01.238 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.239 DEBUG:boto:Params: {} 2023-10-18T20:02:01.239 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:02:01.240 DEBUG:boto:Token: None 2023-10-18T20:02:01.240 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-21/ 2023-10-18T20:02:01.241 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f8gHT8sWJ8OXtWs/IhJewFotekU= 2023-10-18T20:02:01.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f8gHT8sWJ8OXtWs/IhJewFotekU=', 'Content-Length': '0'} 2023-10-18T20:02:01.268 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa0a7efa06055ddf-00653039b9-10fc-b1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.269 INFO:rgw_multi.tests:create bucket zone=b2 name=bofrnf-22 2023-10-18T20:02:01.269 DEBUG:boto:path=/bofrnf-22/ 2023-10-18T20:02:01.270 DEBUG:boto:auth_path=/bofrnf-22/ 2023-10-18T20:02:01.270 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.270 DEBUG:boto:Path: /bofrnf-22/ 2023-10-18T20:02:01.271 DEBUG:boto:Data: 2023-10-18T20:02:01.271 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.272 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:01.272 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.272 DEBUG:boto:Params: {} 2023-10-18T20:02:01.273 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:02:01.273 DEBUG:boto:Token: None 2023-10-18T20:02:01.274 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-22/ 2023-10-18T20:02:01.274 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3dqKeU1WhlEB2xzYgYXZ/3thSok= 2023-10-18T20:02:01.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3dqKeU1WhlEB2xzYgYXZ/3thSok=', 'Content-Length': '0'} 2023-10-18T20:02:01.302 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000027bf92dd85887898-00653039b9-1102-b2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.303 DEBUG:boto:path=/bofrnf-19/ 2023-10-18T20:02:01.303 DEBUG:boto:auth_path=/bofrnf-19/ 2023-10-18T20:02:01.303 DEBUG:boto:path=/bofrnf-19/?versioning 2023-10-18T20:02:01.304 DEBUG:boto:auth_path=/bofrnf-19/?versioning 2023-10-18T20:02:01.304 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.305 DEBUG:boto:Path: /bofrnf-19/?versioning 2023-10-18T20:02:01.305 DEBUG:boto:Data: Enabled Disabled 2023-10-18T20:02:01.305 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.306 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.306 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.306 DEBUG:boto:Params: {} 2023-10-18T20:02:01.307 DEBUG:boto:Token: None 2023-10-18T20:02:01.307 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-19/?versioning 2023-10-18T20:02:01.308 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WQG2WY5AYuatLU6wpDKOQxulFEw= 2023-10-18T20:02:01.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WQG2WY5AYuatLU6wpDKOQxulFEw=', 'Content-Length': '228'} 2023-10-18T20:02:01.313 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000078ef51556bce0781-00653039b9-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.313 DEBUG:boto:path=/bofrnf-19/ 2023-10-18T20:02:01.314 DEBUG:boto:auth_path=/bofrnf-19/ 2023-10-18T20:02:01.314 DEBUG:boto:path=/bofrnf-19/?versioning 2023-10-18T20:02:01.315 DEBUG:boto:auth_path=/bofrnf-19/?versioning 2023-10-18T20:02:01.315 DEBUG:boto:Method: GET 2023-10-18T20:02:01.315 DEBUG:boto:Path: /bofrnf-19/?versioning 2023-10-18T20:02:01.316 DEBUG:boto:Data: 2023-10-18T20:02:01.316 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.317 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.317 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.317 DEBUG:boto:Params: {} 2023-10-18T20:02:01.318 DEBUG:boto:Token: None 2023-10-18T20:02:01.318 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-19/?versioning 2023-10-18T20:02:01.319 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n7grzU6mgYrme9ZPQ9AFrCWHXFM= 2023-10-18T20:02:01.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n7grzU6mgYrme9ZPQ9AFrCWHXFM=', 'Content-Length': '0'} 2023-10-18T20:02:01.320 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cfe4fc47454ce8e7-00653039b9-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.362 DEBUG:boto:EnabledDisabled 2023-10-18T20:02:01.363 DEBUG:boto:path=/bofrnf-20/ 2023-10-18T20:02:01.364 DEBUG:boto:auth_path=/bofrnf-20/ 2023-10-18T20:02:01.364 DEBUG:boto:path=/bofrnf-20/?versioning 2023-10-18T20:02:01.364 DEBUG:boto:auth_path=/bofrnf-20/?versioning 2023-10-18T20:02:01.365 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.365 DEBUG:boto:Path: /bofrnf-20/?versioning 2023-10-18T20:02:01.366 DEBUG:boto:Data: Enabled Disabled 2023-10-18T20:02:01.366 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.366 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.367 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.367 DEBUG:boto:Params: {} 2023-10-18T20:02:01.368 DEBUG:boto:Token: None 2023-10-18T20:02:01.368 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-20/?versioning 2023-10-18T20:02:01.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xq1MjaZQoqsm0QowmZ/mtKepq2o= 2023-10-18T20:02:01.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xq1MjaZQoqsm0QowmZ/mtKepq2o=', 'Content-Length': '228'} 2023-10-18T20:02:01.376 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8551eae5751facd-00653039b9-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.377 DEBUG:boto:path=/bofrnf-20/ 2023-10-18T20:02:01.377 DEBUG:boto:auth_path=/bofrnf-20/ 2023-10-18T20:02:01.377 DEBUG:boto:path=/bofrnf-20/?versioning 2023-10-18T20:02:01.378 DEBUG:boto:auth_path=/bofrnf-20/?versioning 2023-10-18T20:02:01.378 DEBUG:boto:Method: GET 2023-10-18T20:02:01.379 DEBUG:boto:Path: /bofrnf-20/?versioning 2023-10-18T20:02:01.379 DEBUG:boto:Data: 2023-10-18T20:02:01.379 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.380 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.380 DEBUG:boto:Params: {} 2023-10-18T20:02:01.381 DEBUG:boto:Token: None 2023-10-18T20:02:01.381 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-20/?versioning 2023-10-18T20:02:01.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qBT4Vvw4iMpIJQdJ5hrf8A8q6UI= 2023-10-18T20:02:01.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qBT4Vvw4iMpIJQdJ5hrf8A8q6UI=', 'Content-Length': '0'} 2023-10-18T20:02:01.384 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086a191d118546f0d-00653039b9-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.384 DEBUG:boto:EnabledDisabled 2023-10-18T20:02:01.385 DEBUG:boto:path=/bofrnf-21/ 2023-10-18T20:02:01.385 DEBUG:boto:auth_path=/bofrnf-21/ 2023-10-18T20:02:01.386 DEBUG:boto:path=/bofrnf-21/?versioning 2023-10-18T20:02:01.386 DEBUG:boto:auth_path=/bofrnf-21/?versioning 2023-10-18T20:02:01.386 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.387 DEBUG:boto:Path: /bofrnf-21/?versioning 2023-10-18T20:02:01.387 DEBUG:boto:Data: Enabled Disabled 2023-10-18T20:02:01.388 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.388 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:01.388 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.389 DEBUG:boto:Params: {} 2023-10-18T20:02:01.389 DEBUG:boto:Token: None 2023-10-18T20:02:01.390 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-21/?versioning 2023-10-18T20:02:01.390 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZwnFGrfl3TqbDP1/AyExqm0xg/o= 2023-10-18T20:02:01.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZwnFGrfl3TqbDP1/AyExqm0xg/o=', 'Content-Length': '228'} 2023-10-18T20:02:01.405 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4a1ac03d1c43991-00653039b9-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.405 DEBUG:boto:path=/bofrnf-21/ 2023-10-18T20:02:01.406 DEBUG:boto:auth_path=/bofrnf-21/ 2023-10-18T20:02:01.406 DEBUG:boto:path=/bofrnf-21/?versioning 2023-10-18T20:02:01.407 DEBUG:boto:auth_path=/bofrnf-21/?versioning 2023-10-18T20:02:01.407 DEBUG:boto:Method: GET 2023-10-18T20:02:01.407 DEBUG:boto:Path: /bofrnf-21/?versioning 2023-10-18T20:02:01.408 DEBUG:boto:Data: 2023-10-18T20:02:01.408 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.409 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:01.409 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.409 DEBUG:boto:Params: {} 2023-10-18T20:02:01.410 DEBUG:boto:Token: None 2023-10-18T20:02:01.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-21/?versioning 2023-10-18T20:02:01.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:u5b8HXF1Va5KHIkrem7jvBGQsv8= 2023-10-18T20:02:01.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:u5b8HXF1Va5KHIkrem7jvBGQsv8=', 'Content-Length': '0'} 2023-10-18T20:02:01.413 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c41bc7dbe7507197-00653039b9-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.413 DEBUG:boto:EnabledDisabled 2023-10-18T20:02:01.414 DEBUG:boto:path=/bofrnf-22/ 2023-10-18T20:02:01.414 DEBUG:boto:auth_path=/bofrnf-22/ 2023-10-18T20:02:01.414 DEBUG:boto:path=/bofrnf-22/?versioning 2023-10-18T20:02:01.415 DEBUG:boto:auth_path=/bofrnf-22/?versioning 2023-10-18T20:02:01.415 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.416 DEBUG:boto:Path: /bofrnf-22/?versioning 2023-10-18T20:02:01.416 DEBUG:boto:Data: Enabled Disabled 2023-10-18T20:02:01.417 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.417 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:01.417 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.418 DEBUG:boto:Params: {} 2023-10-18T20:02:01.418 DEBUG:boto:Token: None 2023-10-18T20:02:01.419 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-22/?versioning 2023-10-18T20:02:01.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:K143EtmjBvAtfpGyZfa0AGt2GZ8= 2023-10-18T20:02:01.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:K143EtmjBvAtfpGyZfa0AGt2GZ8=', 'Content-Length': '228'} 2023-10-18T20:02:01.438 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014b5ad6ff444c944-00653039b9-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.439 DEBUG:boto:path=/bofrnf-22/ 2023-10-18T20:02:01.439 DEBUG:boto:auth_path=/bofrnf-22/ 2023-10-18T20:02:01.440 DEBUG:boto:path=/bofrnf-22/?versioning 2023-10-18T20:02:01.440 DEBUG:boto:auth_path=/bofrnf-22/?versioning 2023-10-18T20:02:01.441 DEBUG:boto:Method: GET 2023-10-18T20:02:01.441 DEBUG:boto:Path: /bofrnf-22/?versioning 2023-10-18T20:02:01.441 DEBUG:boto:Data: 2023-10-18T20:02:01.442 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.442 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:01.442 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.443 DEBUG:boto:Params: {} 2023-10-18T20:02:01.443 DEBUG:boto:Token: None 2023-10-18T20:02:01.444 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-22/?versioning 2023-10-18T20:02:01.444 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5AQ22x1Kml4x1iFN+mB3V/yjqto= 2023-10-18T20:02:01.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5AQ22x1Kml4x1iFN+mB3V/yjqto=', 'Content-Length': '0'} 2023-10-18T20:02:01.446 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008bf0a6e518f9c847-00653039b9-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.447 DEBUG:boto:EnabledDisabled 2023-10-18T20:02:01.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-10-18T20:02:01.448 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-23 2023-10-18T20:02:01.448 DEBUG:boto:path=/bofrnf-23/ 2023-10-18T20:02:01.449 DEBUG:boto:auth_path=/bofrnf-23/ 2023-10-18T20:02:01.449 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.450 DEBUG:boto:Path: /bofrnf-23/ 2023-10-18T20:02:01.450 DEBUG:boto:Data: 2023-10-18T20:02:01.450 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.451 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.451 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.452 DEBUG:boto:Params: {} 2023-10-18T20:02:01.452 DEBUG:boto:Token: None 2023-10-18T20:02:01.453 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-23/ 2023-10-18T20:02:01.453 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/2bEhB6mPPnxH1sznLhmvQ/pDa8= 2023-10-18T20:02:01.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/2bEhB6mPPnxH1sznLhmvQ/pDa8=', 'Content-Length': '0'} 2023-10-18T20:02:01.470 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003e5519e190d8e490-00653039b9-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.470 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-24 2023-10-18T20:02:01.471 DEBUG:boto:path=/bofrnf-24/ 2023-10-18T20:02:01.471 DEBUG:boto:auth_path=/bofrnf-24/ 2023-10-18T20:02:01.471 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.472 DEBUG:boto:Path: /bofrnf-24/ 2023-10-18T20:02:01.472 DEBUG:boto:Data: 2023-10-18T20:02:01.473 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.473 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.473 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.474 DEBUG:boto:Params: {} 2023-10-18T20:02:01.474 DEBUG:boto:Token: None 2023-10-18T20:02:01.475 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-24/ 2023-10-18T20:02:01.475 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PobAia41/i9KxbU452xzsPAgG68= 2023-10-18T20:02:01.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PobAia41/i9KxbU452xzsPAgG68=', 'Content-Length': '0'} 2023-10-18T20:02:01.497 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042be8a654be6fb40-00653039b9-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.498 INFO:rgw_multi.tests:create bucket zone=b1 name=bofrnf-25 2023-10-18T20:02:01.498 DEBUG:boto:path=/bofrnf-25/ 2023-10-18T20:02:01.499 DEBUG:boto:auth_path=/bofrnf-25/ 2023-10-18T20:02:01.499 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.499 DEBUG:boto:Path: /bofrnf-25/ 2023-10-18T20:02:01.500 DEBUG:boto:Data: 2023-10-18T20:02:01.500 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.501 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:01.501 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.501 DEBUG:boto:Params: {} 2023-10-18T20:02:01.502 DEBUG:boto:Token: None 2023-10-18T20:02:01.502 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-25/ 2023-10-18T20:02:01.503 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gG/eZ+Wvy5RerS6WBSMwhTReiDQ= 2023-10-18T20:02:01.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gG/eZ+Wvy5RerS6WBSMwhTReiDQ=', 'Content-Length': '0'} 2023-10-18T20:02:01.527 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f82bdfbb302f6f65-00653039b9-10fc-b1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.528 INFO:rgw_multi.tests:create bucket zone=b2 name=bofrnf-26 2023-10-18T20:02:01.528 DEBUG:boto:path=/bofrnf-26/ 2023-10-18T20:02:01.528 DEBUG:boto:auth_path=/bofrnf-26/ 2023-10-18T20:02:01.529 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.529 DEBUG:boto:Path: /bofrnf-26/ 2023-10-18T20:02:01.530 DEBUG:boto:Data: 2023-10-18T20:02:01.530 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.531 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:01.531 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.531 DEBUG:boto:Params: {} 2023-10-18T20:02:01.532 DEBUG:boto:Token: None 2023-10-18T20:02:01.532 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-26/ 2023-10-18T20:02:01.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jKO4aK7y6N6/K1Kp1YoO+0owcjw= 2023-10-18T20:02:01.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jKO4aK7y6N6/K1Kp1YoO+0owcjw=', 'Content-Length': '0'} 2023-10-18T20:02:01.564 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000167adbef5e0f7a7-00653039b9-1102-b2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.565 DEBUG:boto:path=/bofrnf-23/ 2023-10-18T20:02:01.565 DEBUG:boto:auth_path=/bofrnf-23/ 2023-10-18T20:02:01.565 DEBUG:boto:path=/bofrnf-23/?acl 2023-10-18T20:02:01.566 DEBUG:boto:auth_path=/bofrnf-23/?acl 2023-10-18T20:02:01.566 DEBUG:boto:Method: GET 2023-10-18T20:02:01.567 DEBUG:boto:Path: /bofrnf-23/?acl 2023-10-18T20:02:01.567 DEBUG:boto:Data: 2023-10-18T20:02:01.567 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.568 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.568 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.569 DEBUG:boto:Params: {} 2023-10-18T20:02:01.569 DEBUG:boto:Token: None 2023-10-18T20:02:01.569 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-23/?acl 2023-10-18T20:02:01.570 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pxrp6k4bUulcxhNP//EiYbsynco= 2023-10-18T20:02:01.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pxrp6k4bUulcxhNP//EiYbsynco=', 'Content-Length': '0'} 2023-10-18T20:02:01.572 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be40416c60103f33-00653039b9-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.573 DEBUG:boto:path=/bofrnf-23/ 2023-10-18T20:02:01.573 DEBUG:boto:auth_path=/bofrnf-23/ 2023-10-18T20:02:01.573 DEBUG:boto:path=/bofrnf-23/?acl 2023-10-18T20:02:01.574 DEBUG:boto:auth_path=/bofrnf-23/?acl 2023-10-18T20:02:01.574 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.575 DEBUG:boto:Path: /bofrnf-23/?acl 2023-10-18T20:02:01.575 DEBUG:boto:Data: 2023-10-18T20:02:01.575 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-18T20:02:01.576 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.576 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.577 DEBUG:boto:Params: {} 2023-10-18T20:02:01.577 DEBUG:boto:Token: None 2023-10-18T20:02:01.577 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT x-amz-acl:public-read /bofrnf-23/?acl 2023-10-18T20:02:01.578 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Vh1hIp+Dv/YCe7A5Hwifdot5zxQ= 2023-10-18T20:02:01.578 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vh1hIp+Dv/YCe7A5Hwifdot5zxQ=', 'Content-Length': '0'} 2023-10-18T20:02:01.591 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a6728f843c90c013-00653039b9-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.591 DEBUG:boto:path=/bofrnf-23/ 2023-10-18T20:02:01.592 DEBUG:boto:auth_path=/bofrnf-23/ 2023-10-18T20:02:01.592 DEBUG:boto:path=/bofrnf-23/?acl 2023-10-18T20:02:01.592 DEBUG:boto:auth_path=/bofrnf-23/?acl 2023-10-18T20:02:01.593 DEBUG:boto:Method: GET 2023-10-18T20:02:01.593 DEBUG:boto:Path: /bofrnf-23/?acl 2023-10-18T20:02:01.594 DEBUG:boto:Data: 2023-10-18T20:02:01.594 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.594 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.595 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.595 DEBUG:boto:Params: {} 2023-10-18T20:02:01.596 DEBUG:boto:Token: None 2023-10-18T20:02:01.596 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-23/?acl 2023-10-18T20:02:01.596 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pxrp6k4bUulcxhNP//EiYbsynco= 2023-10-18T20:02:01.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pxrp6k4bUulcxhNP//EiYbsynco=', 'Content-Length': '0'} 2023-10-18T20:02:01.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054eee3c1cb9b79cb-00653039b9-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.599 DEBUG:boto:path=/bofrnf-24/ 2023-10-18T20:02:01.599 DEBUG:boto:auth_path=/bofrnf-24/ 2023-10-18T20:02:01.600 DEBUG:boto:path=/bofrnf-24/?acl 2023-10-18T20:02:01.600 DEBUG:boto:auth_path=/bofrnf-24/?acl 2023-10-18T20:02:01.600 DEBUG:boto:Method: GET 2023-10-18T20:02:01.601 DEBUG:boto:Path: /bofrnf-24/?acl 2023-10-18T20:02:01.601 DEBUG:boto:Data: 2023-10-18T20:02:01.602 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.603 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.603 DEBUG:boto:Params: {} 2023-10-18T20:02:01.603 DEBUG:boto:Token: None 2023-10-18T20:02:01.604 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-24/?acl 2023-10-18T20:02:01.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1VU/+46HmPJyDT2Mo6duysqf9qk= 2023-10-18T20:02:01.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1VU/+46HmPJyDT2Mo6duysqf9qk=', 'Content-Length': '0'} 2023-10-18T20:02:01.606 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000edb11651539fee01-00653039b9-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.606 DEBUG:boto:path=/bofrnf-24/ 2023-10-18T20:02:01.607 DEBUG:boto:auth_path=/bofrnf-24/ 2023-10-18T20:02:01.607 DEBUG:boto:path=/bofrnf-24/?acl 2023-10-18T20:02:01.608 DEBUG:boto:auth_path=/bofrnf-24/?acl 2023-10-18T20:02:01.608 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.609 DEBUG:boto:Path: /bofrnf-24/?acl 2023-10-18T20:02:01.609 DEBUG:boto:Data: 2023-10-18T20:02:01.609 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-18T20:02:01.610 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.610 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.611 DEBUG:boto:Params: {} 2023-10-18T20:02:01.611 DEBUG:boto:Token: None 2023-10-18T20:02:01.611 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT x-amz-acl:public-read /bofrnf-24/?acl 2023-10-18T20:02:01.612 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bCvTmv7tILR1eUgGK23CpR8j7aU= 2023-10-18T20:02:01.612 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bCvTmv7tILR1eUgGK23CpR8j7aU=', 'Content-Length': '0'} 2023-10-18T20:02:01.619 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003006fa5d7422ec16-00653039b9-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.620 DEBUG:boto:path=/bofrnf-24/ 2023-10-18T20:02:01.620 DEBUG:boto:auth_path=/bofrnf-24/ 2023-10-18T20:02:01.621 DEBUG:boto:path=/bofrnf-24/?acl 2023-10-18T20:02:01.621 DEBUG:boto:auth_path=/bofrnf-24/?acl 2023-10-18T20:02:01.622 DEBUG:boto:Method: GET 2023-10-18T20:02:01.622 DEBUG:boto:Path: /bofrnf-24/?acl 2023-10-18T20:02:01.622 DEBUG:boto:Data: 2023-10-18T20:02:01.623 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.623 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.624 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.624 DEBUG:boto:Params: {} 2023-10-18T20:02:01.624 DEBUG:boto:Token: None 2023-10-18T20:02:01.625 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-24/?acl 2023-10-18T20:02:01.625 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1VU/+46HmPJyDT2Mo6duysqf9qk= 2023-10-18T20:02:01.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1VU/+46HmPJyDT2Mo6duysqf9qk=', 'Content-Length': '0'} 2023-10-18T20:02:01.627 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000020c1ef8f35a6113b-00653039b9-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.627 DEBUG:boto:path=/bofrnf-25/ 2023-10-18T20:02:01.628 DEBUG:boto:auth_path=/bofrnf-25/ 2023-10-18T20:02:01.628 DEBUG:boto:path=/bofrnf-25/?acl 2023-10-18T20:02:01.629 DEBUG:boto:auth_path=/bofrnf-25/?acl 2023-10-18T20:02:01.629 DEBUG:boto:Method: GET 2023-10-18T20:02:01.629 DEBUG:boto:Path: /bofrnf-25/?acl 2023-10-18T20:02:01.630 DEBUG:boto:Data: 2023-10-18T20:02:01.630 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.631 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:01.631 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.631 DEBUG:boto:Params: {} 2023-10-18T20:02:01.633 DEBUG:boto:Token: None 2023-10-18T20:02:01.634 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-25/?acl 2023-10-18T20:02:01.635 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7O/PulbIavpoCgCmMaWLtmwusiI= 2023-10-18T20:02:01.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7O/PulbIavpoCgCmMaWLtmwusiI=', 'Content-Length': '0'} 2023-10-18T20:02:01.637 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007d7442f341dfbdd0-00653039b9-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.637 DEBUG:boto:path=/bofrnf-25/ 2023-10-18T20:02:01.638 DEBUG:boto:auth_path=/bofrnf-25/ 2023-10-18T20:02:01.638 DEBUG:boto:path=/bofrnf-25/?acl 2023-10-18T20:02:01.639 DEBUG:boto:auth_path=/bofrnf-25/?acl 2023-10-18T20:02:01.639 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.639 DEBUG:boto:Path: /bofrnf-25/?acl 2023-10-18T20:02:01.640 DEBUG:boto:Data: 2023-10-18T20:02:01.640 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-18T20:02:01.641 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:01.641 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.641 DEBUG:boto:Params: {} 2023-10-18T20:02:01.642 DEBUG:boto:Token: None 2023-10-18T20:02:01.642 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT x-amz-acl:public-read /bofrnf-25/?acl 2023-10-18T20:02:01.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YvwWEljWk/pL74kM+sBDqYsWOo0= 2023-10-18T20:02:01.643 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YvwWEljWk/pL74kM+sBDqYsWOo0=', 'Content-Length': '0'} 2023-10-18T20:02:01.651 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000519f25de0862e67d-00653039b9-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.651 DEBUG:boto:path=/bofrnf-25/ 2023-10-18T20:02:01.652 DEBUG:boto:auth_path=/bofrnf-25/ 2023-10-18T20:02:01.652 DEBUG:boto:path=/bofrnf-25/?acl 2023-10-18T20:02:01.653 DEBUG:boto:auth_path=/bofrnf-25/?acl 2023-10-18T20:02:01.653 DEBUG:boto:Method: GET 2023-10-18T20:02:01.653 DEBUG:boto:Path: /bofrnf-25/?acl 2023-10-18T20:02:01.654 DEBUG:boto:Data: 2023-10-18T20:02:01.654 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.654 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:01.655 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.655 DEBUG:boto:Params: {} 2023-10-18T20:02:01.656 DEBUG:boto:Token: None 2023-10-18T20:02:01.656 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-25/?acl 2023-10-18T20:02:01.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7O/PulbIavpoCgCmMaWLtmwusiI= 2023-10-18T20:02:01.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7O/PulbIavpoCgCmMaWLtmwusiI=', 'Content-Length': '0'} 2023-10-18T20:02:01.658 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002c1b7988cf22ab7e-00653039b9-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.659 DEBUG:boto:path=/bofrnf-26/ 2023-10-18T20:02:01.659 DEBUG:boto:auth_path=/bofrnf-26/ 2023-10-18T20:02:01.660 DEBUG:boto:path=/bofrnf-26/?acl 2023-10-18T20:02:01.660 DEBUG:boto:auth_path=/bofrnf-26/?acl 2023-10-18T20:02:01.660 DEBUG:boto:Method: GET 2023-10-18T20:02:01.661 DEBUG:boto:Path: /bofrnf-26/?acl 2023-10-18T20:02:01.661 DEBUG:boto:Data: 2023-10-18T20:02:01.662 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.662 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:01.662 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.663 DEBUG:boto:Params: {} 2023-10-18T20:02:01.663 DEBUG:boto:Token: None 2023-10-18T20:02:01.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-26/?acl 2023-10-18T20:02:01.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tQK+XeBW9do0LB+DG3avH66m6Oo= 2023-10-18T20:02:01.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tQK+XeBW9do0LB+DG3avH66m6Oo=', 'Content-Length': '0'} 2023-10-18T20:02:01.666 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000749abcd4e35d961-00653039b9-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.667 DEBUG:boto:path=/bofrnf-26/ 2023-10-18T20:02:01.667 DEBUG:boto:auth_path=/bofrnf-26/ 2023-10-18T20:02:01.667 DEBUG:boto:path=/bofrnf-26/?acl 2023-10-18T20:02:01.668 DEBUG:boto:auth_path=/bofrnf-26/?acl 2023-10-18T20:02:01.668 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.669 DEBUG:boto:Path: /bofrnf-26/?acl 2023-10-18T20:02:01.669 DEBUG:boto:Data: 2023-10-18T20:02:01.669 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-18T20:02:01.670 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:01.670 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.671 DEBUG:boto:Params: {} 2023-10-18T20:02:01.671 DEBUG:boto:Token: None 2023-10-18T20:02:01.672 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT x-amz-acl:public-read /bofrnf-26/?acl 2023-10-18T20:02:01.672 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EUJyQhQFtuwpldrwFTOXiiDxBcc= 2023-10-18T20:02:01.672 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EUJyQhQFtuwpldrwFTOXiiDxBcc=', 'Content-Length': '0'} 2023-10-18T20:02:01.680 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c00c05762a484b96-00653039b9-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.681 DEBUG:boto:path=/bofrnf-26/ 2023-10-18T20:02:01.681 DEBUG:boto:auth_path=/bofrnf-26/ 2023-10-18T20:02:01.681 DEBUG:boto:path=/bofrnf-26/?acl 2023-10-18T20:02:01.682 DEBUG:boto:auth_path=/bofrnf-26/?acl 2023-10-18T20:02:01.682 DEBUG:boto:Method: GET 2023-10-18T20:02:01.683 DEBUG:boto:Path: /bofrnf-26/?acl 2023-10-18T20:02:01.683 DEBUG:boto:Data: 2023-10-18T20:02:01.683 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.684 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:01.684 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.684 DEBUG:boto:Params: {} 2023-10-18T20:02:01.685 DEBUG:boto:Token: None 2023-10-18T20:02:01.685 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-26/?acl 2023-10-18T20:02:01.686 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tQK+XeBW9do0LB+DG3avH66m6Oo= 2023-10-18T20:02:01.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tQK+XeBW9do0LB+DG3avH66m6Oo=', 'Content-Length': '0'} 2023-10-18T20:02:01.688 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cd6dbd228984b6ad-00653039b9-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-10-18T20:02:01.731 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-27 2023-10-18T20:02:01.732 DEBUG:boto:path=/bofrnf-27/ 2023-10-18T20:02:01.732 DEBUG:boto:auth_path=/bofrnf-27/ 2023-10-18T20:02:01.733 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.786 DEBUG:boto:Path: /bofrnf-27/ 2023-10-18T20:02:01.788 DEBUG:boto:Data: 2023-10-18T20:02:01.788 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.790 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.791 DEBUG:boto:Params: {} 2023-10-18T20:02:01.792 DEBUG:boto:Token: None 2023-10-18T20:02:01.792 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-27/ 2023-10-18T20:02:01.793 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HSiCNxGt00XfDCLWSKNdP4NNxi4= 2023-10-18T20:02:01.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HSiCNxGt00XfDCLWSKNdP4NNxi4=', 'Content-Length': '0'} 2023-10-18T20:02:01.809 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000881a64cdd938a30c-00653039b9-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.810 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-28 2023-10-18T20:02:01.810 DEBUG:boto:path=/bofrnf-28/ 2023-10-18T20:02:01.811 DEBUG:boto:auth_path=/bofrnf-28/ 2023-10-18T20:02:01.811 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.812 DEBUG:boto:Path: /bofrnf-28/ 2023-10-18T20:02:01.812 DEBUG:boto:Data: 2023-10-18T20:02:01.813 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.813 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.813 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.814 DEBUG:boto:Params: {} 2023-10-18T20:02:01.814 DEBUG:boto:Token: None 2023-10-18T20:02:01.815 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-28/ 2023-10-18T20:02:01.815 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RGq3etByBC+RS/objh8aDVTAYTM= 2023-10-18T20:02:01.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RGq3etByBC+RS/objh8aDVTAYTM=', 'Content-Length': '0'} 2023-10-18T20:02:01.847 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d3256319aa881af0-00653039b9-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.848 INFO:rgw_multi.tests:create bucket zone=b1 name=bofrnf-29 2023-10-18T20:02:01.848 DEBUG:boto:path=/bofrnf-29/ 2023-10-18T20:02:01.849 DEBUG:boto:auth_path=/bofrnf-29/ 2023-10-18T20:02:01.849 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.850 DEBUG:boto:Path: /bofrnf-29/ 2023-10-18T20:02:01.850 DEBUG:boto:Data: 2023-10-18T20:02:01.850 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.851 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:01.851 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.852 DEBUG:boto:Params: {} 2023-10-18T20:02:01.852 DEBUG:boto:Token: None 2023-10-18T20:02:01.853 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-29/ 2023-10-18T20:02:01.853 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bBRzikRLIjN4ycXQGsksu5nDpWQ= 2023-10-18T20:02:01.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bBRzikRLIjN4ycXQGsksu5nDpWQ=', 'Content-Length': '0'} 2023-10-18T20:02:01.892 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f3cb9463a5e1a93-00653039b9-10fc-b1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.893 INFO:rgw_multi.tests:create bucket zone=b2 name=bofrnf-30 2023-10-18T20:02:01.893 DEBUG:boto:path=/bofrnf-30/ 2023-10-18T20:02:01.894 DEBUG:boto:auth_path=/bofrnf-30/ 2023-10-18T20:02:01.894 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.895 DEBUG:boto:Path: /bofrnf-30/ 2023-10-18T20:02:01.895 DEBUG:boto:Data: 2023-10-18T20:02:01.895 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.896 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:01.896 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.897 DEBUG:boto:Params: {} 2023-10-18T20:02:01.897 DEBUG:boto:Token: None 2023-10-18T20:02:01.898 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-30/ 2023-10-18T20:02:01.898 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zcrDotGEdj+dsO8KYKYva6T5ou0= 2023-10-18T20:02:01.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zcrDotGEdj+dsO8KYKYva6T5ou0=', 'Content-Length': '0'} 2023-10-18T20:02:01.929 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe2256c88859c7ea-00653039b9-1102-b2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.930 DEBUG:boto:path=/bofrnf-27/ 2023-10-18T20:02:01.930 DEBUG:boto:auth_path=/bofrnf-27/ 2023-10-18T20:02:01.931 DEBUG:boto:path=/bofrnf-27/?cors 2023-10-18T20:02:01.931 DEBUG:boto:auth_path=/bofrnf-27/?cors 2023-10-18T20:02:01.931 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.932 DEBUG:boto:Path: /bofrnf-27/?cors 2023-10-18T20:02:01.932 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-18T20:02:01.932 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-18T20:02:01.933 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.933 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.934 DEBUG:boto:Params: {} 2023-10-18T20:02:01.934 DEBUG:boto:Token: None 2023-10-18T20:02:01.935 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-27/?cors 2023-10-18T20:02:01.935 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QC5mnE30GpGS74A9PApXfd8dJ7U= 2023-10-18T20:02:01.936 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QC5mnE30GpGS74A9PApXfd8dJ7U=', 'Content-Length': '218'} 2023-10-18T20:02:01.941 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d3dccaa33987a791-00653039b9-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.941 DEBUG:boto:path=/bofrnf-27/ 2023-10-18T20:02:01.942 DEBUG:boto:auth_path=/bofrnf-27/ 2023-10-18T20:02:01.942 DEBUG:boto:path=/bofrnf-27/?cors 2023-10-18T20:02:01.942 DEBUG:boto:auth_path=/bofrnf-27/?cors 2023-10-18T20:02:01.943 DEBUG:boto:Method: GET 2023-10-18T20:02:01.943 DEBUG:boto:Path: /bofrnf-27/?cors 2023-10-18T20:02:01.943 DEBUG:boto:Data: 2023-10-18T20:02:01.944 DEBUG:boto:Headers: {} 2023-10-18T20:02:01.944 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:01.945 DEBUG:boto:Port: 8000 2023-10-18T20:02:01.945 DEBUG:boto:Params: {} 2023-10-18T20:02:01.945 DEBUG:boto:Token: None 2023-10-18T20:02:01.946 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-27/?cors 2023-10-18T20:02:01.946 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ql531Am6YodXGyjtGl05YdQYhW4= 2023-10-18T20:02:01.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ql531Am6YodXGyjtGl05YdQYhW4=', 'Content-Length': '0'} 2023-10-18T20:02:01.948 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d2324bb37f4a247b-00653039b9-1112-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 18 Oct 2023 20:02:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:01.949 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-18T20:02:01.949 DEBUG:boto:path=/bofrnf-28/ 2023-10-18T20:02:01.950 DEBUG:boto:auth_path=/bofrnf-28/ 2023-10-18T20:02:01.950 DEBUG:boto:path=/bofrnf-28/?cors 2023-10-18T20:02:01.950 DEBUG:boto:auth_path=/bofrnf-28/?cors 2023-10-18T20:02:01.951 DEBUG:boto:Method: PUT 2023-10-18T20:02:01.951 DEBUG:boto:Path: /bofrnf-28/?cors 2023-10-18T20:02:01.952 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-18T20:02:01.952 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-18T20:02:01.952 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:01.953 DEBUG:boto:Port: 8001 2023-10-18T20:02:01.953 DEBUG:boto:Params: {} 2023-10-18T20:02:01.954 DEBUG:boto:Token: None 2023-10-18T20:02:01.954 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 18 Oct 2023 20:02:01 GMT /bofrnf-28/?cors 2023-10-18T20:02:01.955 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A9V1dDzf0oJWg/A7fCQd+T17T7g= 2023-10-18T20:02:01.955 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A9V1dDzf0oJWg/A7fCQd+T17T7g=', 'Content-Length': '218'} 2023-10-18T20:02:02.965 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099d53d58160a2cc5-00653039b9-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:02.966 DEBUG:boto:path=/bofrnf-28/ 2023-10-18T20:02:02.967 DEBUG:boto:auth_path=/bofrnf-28/ 2023-10-18T20:02:02.967 DEBUG:boto:path=/bofrnf-28/?cors 2023-10-18T20:02:02.967 DEBUG:boto:auth_path=/bofrnf-28/?cors 2023-10-18T20:02:02.968 DEBUG:boto:Method: GET 2023-10-18T20:02:02.968 DEBUG:boto:Path: /bofrnf-28/?cors 2023-10-18T20:02:02.969 DEBUG:boto:Data: 2023-10-18T20:02:02.969 DEBUG:boto:Headers: {} 2023-10-18T20:02:02.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:02.970 DEBUG:boto:Port: 8001 2023-10-18T20:02:02.970 DEBUG:boto:Params: {} 2023-10-18T20:02:02.971 DEBUG:boto:Token: None 2023-10-18T20:02:02.971 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:02 GMT /bofrnf-28/?cors 2023-10-18T20:02:02.972 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KNnJN7XfMmoK4YCBUUbPJ5ryGe8= 2023-10-18T20:02:02.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KNnJN7XfMmoK4YCBUUbPJ5ryGe8=', 'Content-Length': '0'} 2023-10-18T20:02:02.974 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e92bedb4e15ce8c-00653039ba-1116-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 18 Oct 2023 20:02:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:03.014 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-18T20:02:03.015 DEBUG:boto:path=/bofrnf-29/ 2023-10-18T20:02:03.016 DEBUG:boto:auth_path=/bofrnf-29/ 2023-10-18T20:02:03.016 DEBUG:boto:path=/bofrnf-29/?cors 2023-10-18T20:02:03.016 DEBUG:boto:auth_path=/bofrnf-29/?cors 2023-10-18T20:02:03.017 DEBUG:boto:Method: PUT 2023-10-18T20:02:03.017 DEBUG:boto:Path: /bofrnf-29/?cors 2023-10-18T20:02:03.018 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-18T20:02:03.018 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-18T20:02:03.018 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:03.019 DEBUG:boto:Port: 8000 2023-10-18T20:02:03.019 DEBUG:boto:Params: {} 2023-10-18T20:02:03.020 DEBUG:boto:Token: None 2023-10-18T20:02:03.020 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 18 Oct 2023 20:02:03 GMT /bofrnf-29/?cors 2023-10-18T20:02:03.021 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mWW/Q2gYqZLPWvNe3xpgtm9EMCg= 2023-10-18T20:02:03.021 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mWW/Q2gYqZLPWvNe3xpgtm9EMCg=', 'Content-Length': '218'} 2023-10-18T20:02:04.032 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004082d8619a8db973-00653039bb-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:04.033 DEBUG:boto:path=/bofrnf-29/ 2023-10-18T20:02:04.034 DEBUG:boto:auth_path=/bofrnf-29/ 2023-10-18T20:02:04.034 DEBUG:boto:path=/bofrnf-29/?cors 2023-10-18T20:02:04.034 DEBUG:boto:auth_path=/bofrnf-29/?cors 2023-10-18T20:02:04.035 DEBUG:boto:Method: GET 2023-10-18T20:02:04.035 DEBUG:boto:Path: /bofrnf-29/?cors 2023-10-18T20:02:04.036 DEBUG:boto:Data: 2023-10-18T20:02:04.036 DEBUG:boto:Headers: {} 2023-10-18T20:02:04.036 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:02:04.037 DEBUG:boto:Port: 8000 2023-10-18T20:02:04.037 DEBUG:boto:Params: {} 2023-10-18T20:02:04.038 DEBUG:boto:Token: None 2023-10-18T20:02:04.038 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:04 GMT /bofrnf-29/?cors 2023-10-18T20:02:04.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WiB3C3Qurob9hNdSfsgHTdy6ytE= 2023-10-18T20:02:04.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WiB3C3Qurob9hNdSfsgHTdy6ytE=', 'Content-Length': '0'} 2023-10-18T20:02:04.041 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000565ae03df705eb18-00653039bc-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 18 Oct 2023 20:02:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:04.082 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-18T20:02:04.083 DEBUG:boto:path=/bofrnf-30/ 2023-10-18T20:02:04.083 DEBUG:boto:auth_path=/bofrnf-30/ 2023-10-18T20:02:04.084 DEBUG:boto:path=/bofrnf-30/?cors 2023-10-18T20:02:04.084 DEBUG:boto:auth_path=/bofrnf-30/?cors 2023-10-18T20:02:04.085 DEBUG:boto:Method: PUT 2023-10-18T20:02:04.085 DEBUG:boto:Path: /bofrnf-30/?cors 2023-10-18T20:02:04.085 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-18T20:02:04.086 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-18T20:02:04.086 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:04.087 DEBUG:boto:Port: 8001 2023-10-18T20:02:04.087 DEBUG:boto:Params: {} 2023-10-18T20:02:04.088 DEBUG:boto:Token: None 2023-10-18T20:02:04.088 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 18 Oct 2023 20:02:04 GMT /bofrnf-30/?cors 2023-10-18T20:02:04.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9R/nxMAapP3Xf0Y6yveEFHAsk+0= 2023-10-18T20:02:04.089 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9R/nxMAapP3Xf0Y6yveEFHAsk+0=', 'Content-Length': '218'} 2023-10-18T20:02:05.100 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000792060dc14ee5814-00653039bc-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.101 DEBUG:boto:path=/bofrnf-30/ 2023-10-18T20:02:05.101 DEBUG:boto:auth_path=/bofrnf-30/ 2023-10-18T20:02:05.101 DEBUG:boto:path=/bofrnf-30/?cors 2023-10-18T20:02:05.102 DEBUG:boto:auth_path=/bofrnf-30/?cors 2023-10-18T20:02:05.102 DEBUG:boto:Method: GET 2023-10-18T20:02:05.103 DEBUG:boto:Path: /bofrnf-30/?cors 2023-10-18T20:02:05.103 DEBUG:boto:Data: 2023-10-18T20:02:05.103 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.104 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:02:05.104 DEBUG:boto:Port: 8001 2023-10-18T20:02:05.104 DEBUG:boto:Params: {} 2023-10-18T20:02:05.105 DEBUG:boto:Token: None 2023-10-18T20:02:05.105 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-30/?cors 2023-10-18T20:02:05.106 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YawyJT8wje4GEi9aZtuRXmZufqY= 2023-10-18T20:02:05.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YawyJT8wje4GEi9aZtuRXmZufqY=', 'Content-Length': '0'} 2023-10-18T20:02:05.108 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d72844474c97c05a-00653039bd-1102-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.108 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-18T20:02:05.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-10-18T20:02:05.110 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-31 2023-10-18T20:02:05.110 DEBUG:boto:path=/bofrnf-31/ 2023-10-18T20:02:05.111 DEBUG:boto:auth_path=/bofrnf-31/ 2023-10-18T20:02:05.111 DEBUG:boto:Method: PUT 2023-10-18T20:02:05.111 DEBUG:boto:Path: /bofrnf-31/ 2023-10-18T20:02:05.112 DEBUG:boto:Data: 2023-10-18T20:02:05.112 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.113 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:05.113 DEBUG:boto:Port: 8000 2023-10-18T20:02:05.113 DEBUG:boto:Params: {} 2023-10-18T20:02:05.114 DEBUG:boto:Token: None 2023-10-18T20:02:05.114 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-31/ 2023-10-18T20:02:05.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gPIMnLA/wgGuQrujapM+7JLktR4= 2023-10-18T20:02:05.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gPIMnLA/wgGuQrujapM+7JLktR4=', 'Content-Length': '0'} 2023-10-18T20:02:05.133 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009842d415b5daf815-00653039bd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.134 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-32 2023-10-18T20:02:05.135 DEBUG:boto:path=/bofrnf-32/ 2023-10-18T20:02:05.135 DEBUG:boto:auth_path=/bofrnf-32/ 2023-10-18T20:02:05.135 DEBUG:boto:Method: PUT 2023-10-18T20:02:05.136 DEBUG:boto:Path: /bofrnf-32/ 2023-10-18T20:02:05.136 DEBUG:boto:Data: 2023-10-18T20:02:05.137 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.137 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:05.137 DEBUG:boto:Port: 8001 2023-10-18T20:02:05.138 DEBUG:boto:Params: {} 2023-10-18T20:02:05.138 DEBUG:boto:Token: None 2023-10-18T20:02:05.139 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-32/ 2023-10-18T20:02:05.140 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sFMrCz97y415awSpUPKKsKLrEgA= 2023-10-18T20:02:05.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sFMrCz97y415awSpUPKKsKLrEgA=', 'Content-Length': '0'} 2023-10-18T20:02:05.177 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000512267908ef23b16-00653039bd-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.179 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:05.236 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:05.236 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:05.265 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:02:05.277 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:02:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.158635_361.1", 2023-10-18T20:02:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:05.158635Z" 2023-10-18T20:02:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:02:05.279 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:02:05.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.279 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:02:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:02:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:02:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:02:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.281 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:02:05.282 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659325.158635_361.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:02:05.282 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:02:05.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.284 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:05.335 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:05.335 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:05.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:05.361+0000 7f81d506b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:02:05.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:05.362+0000 7f81d506b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:02:05.376 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:02:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:02:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.158635_361.1", 2023-10-18T20:02:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:02:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.158635Z", 2023-10-18T20:02:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:02:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:02:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:02:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:02:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:02:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:02:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:02:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:02:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:02:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:05.389 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:02:05.389 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:02:05.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.389 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:05Z" 2023-10-18T20:02:05.390 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:05.391 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.158635_361.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.158635Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:02:05Z" } 2023-10-18T20:02:05.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.158635_361.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.158635Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:02:05.392 DEBUG:rgw_multi.tests:log_status={0: '1_1697659325.158635_361.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:02:05.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.158635_361.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:02:05.392 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:02:05.393 DEBUG:boto:path=/bofrnf-31/ 2023-10-18T20:02:05.393 DEBUG:boto:auth_path=/bofrnf-31/ 2023-10-18T20:02:05.393 DEBUG:boto:Method: HEAD 2023-10-18T20:02:05.394 DEBUG:boto:Path: /bofrnf-31/ 2023-10-18T20:02:05.394 DEBUG:boto:Data: 2023-10-18T20:02:05.395 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.395 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:05.396 DEBUG:boto:Port: 8000 2023-10-18T20:02:05.396 DEBUG:boto:Params: {} 2023-10-18T20:02:05.396 DEBUG:boto:Token: None 2023-10-18T20:02:05.397 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-31/ 2023-10-18T20:02:05.397 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LN2nsZpU6TeCl+CxnfPBGmRNRWA= 2023-10-18T20:02:05.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LN2nsZpU6TeCl+CxnfPBGmRNRWA=', 'Content-Length': '0'} 2023-10-18T20:02:05.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002ef07ecaa269485-00653039bd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.401 DEBUG:boto:path=/bofrnf-31/foo 2023-10-18T20:02:05.401 DEBUG:boto:auth_path=/bofrnf-31/foo 2023-10-18T20:02:05.402 DEBUG:boto:Method: PUT 2023-10-18T20:02:05.402 DEBUG:boto:Path: /bofrnf-31/foo 2023-10-18T20:02:05.402 DEBUG:boto:Data: 2023-10-18T20:02:05.403 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:02:05.403 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:05.404 DEBUG:boto:Port: 8000 2023-10-18T20:02:05.404 DEBUG:boto:Params: {} 2023-10-18T20:02:05.404 DEBUG:boto:Token: None 2023-10-18T20:02:05.405 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-31/foo 2023-10-18T20:02:05.405 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xdsJUjfnsRTXvzxj5J05n1yjR2A= 2023-10-18T20:02:05.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xdsJUjfnsRTXvzxj5J05n1yjR2A='} 2023-10-18T20:02:05.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000009b37c89a69fbba-00653039bd-1112-a1'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.413 DEBUG:boto:path=/bofrnf-31/ 2023-10-18T20:02:05.413 DEBUG:boto:auth_path=/bofrnf-31/ 2023-10-18T20:02:05.414 DEBUG:boto:Method: DELETE 2023-10-18T20:02:05.414 DEBUG:boto:Path: /bofrnf-31/ 2023-10-18T20:02:05.414 DEBUG:boto:Data: 2023-10-18T20:02:05.415 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:05.416 DEBUG:boto:Port: 8000 2023-10-18T20:02:05.416 DEBUG:boto:Params: {} 2023-10-18T20:02:05.416 DEBUG:boto:Token: None 2023-10-18T20:02:05.417 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-31/ 2023-10-18T20:02:05.417 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:elgHfCtLoti4pPsgjDsgfLHK9Ds= 2023-10-18T20:02:05.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:elgHfCtLoti4pPsgjDsgfLHK9Ds=', 'Content-Length': '0'} 2023-10-18T20:02:05.424 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000e1bf4523a1aee074-00653039bd-1112-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.467 DEBUG:boto:path=/bofrnf-32/ 2023-10-18T20:02:05.467 DEBUG:boto:auth_path=/bofrnf-32/ 2023-10-18T20:02:05.468 DEBUG:boto:Method: HEAD 2023-10-18T20:02:05.468 DEBUG:boto:Path: /bofrnf-32/ 2023-10-18T20:02:05.468 DEBUG:boto:Data: 2023-10-18T20:02:05.469 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.469 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:05.470 DEBUG:boto:Port: 8001 2023-10-18T20:02:05.470 DEBUG:boto:Params: {} 2023-10-18T20:02:05.470 DEBUG:boto:Token: None 2023-10-18T20:02:05.471 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-32/ 2023-10-18T20:02:05.471 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KhjQoY7xg6Ap23psNfN4hQy5MZ4= 2023-10-18T20:02:05.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KhjQoY7xg6Ap23psNfN4hQy5MZ4=', 'Content-Length': '0'} 2023-10-18T20:02:05.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068378a3a8bcb978d-00653039bd-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.475 DEBUG:boto:path=/bofrnf-32/foo 2023-10-18T20:02:05.475 DEBUG:boto:auth_path=/bofrnf-32/foo 2023-10-18T20:02:05.476 DEBUG:boto:Method: PUT 2023-10-18T20:02:05.476 DEBUG:boto:Path: /bofrnf-32/foo 2023-10-18T20:02:05.477 DEBUG:boto:Data: 2023-10-18T20:02:05.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:02:05.478 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:05.478 DEBUG:boto:Port: 8001 2023-10-18T20:02:05.478 DEBUG:boto:Params: {} 2023-10-18T20:02:05.479 DEBUG:boto:Token: None 2023-10-18T20:02:05.479 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-32/foo 2023-10-18T20:02:05.480 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qMl3WafoUF4ikh4caDCnS4khFy8= 2023-10-18T20:02:05.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qMl3WafoUF4ikh4caDCnS4khFy8='} 2023-10-18T20:02:05.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000075cd0971f91d9d9-00653039bd-1116-a2'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.486 DEBUG:boto:path=/bofrnf-32/ 2023-10-18T20:02:05.486 DEBUG:boto:auth_path=/bofrnf-32/ 2023-10-18T20:02:05.487 DEBUG:boto:Method: DELETE 2023-10-18T20:02:05.487 DEBUG:boto:Path: /bofrnf-32/ 2023-10-18T20:02:05.488 DEBUG:boto:Data: 2023-10-18T20:02:05.488 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.488 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:05.489 DEBUG:boto:Port: 8001 2023-10-18T20:02:05.489 DEBUG:boto:Params: {} 2023-10-18T20:02:05.489 DEBUG:boto:Token: None 2023-10-18T20:02:05.490 DEBUG:boto:StringToSign: DELETE Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-32/ 2023-10-18T20:02:05.490 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SR19aBFiNCCFz1g7KVW0XmXF428= 2023-10-18T20:02:05.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SR19aBFiNCCFz1g7KVW0XmXF428=', 'Content-Length': '0'} 2023-10-18T20:02:05.497 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx000008886541533dd64a6-00653039bd-1116-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.497 DEBUG:boto:path=/bofrnf-31/ 2023-10-18T20:02:05.498 DEBUG:boto:auth_path=/bofrnf-31/ 2023-10-18T20:02:05.498 DEBUG:boto:Method: HEAD 2023-10-18T20:02:05.499 DEBUG:boto:Path: /bofrnf-31/ 2023-10-18T20:02:05.499 DEBUG:boto:Data: 2023-10-18T20:02:05.499 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.500 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:05.500 DEBUG:boto:Port: 8000 2023-10-18T20:02:05.500 DEBUG:boto:Params: {} 2023-10-18T20:02:05.501 DEBUG:boto:Token: None 2023-10-18T20:02:05.501 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-31/ 2023-10-18T20:02:05.502 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LN2nsZpU6TeCl+CxnfPBGmRNRWA= 2023-10-18T20:02:05.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LN2nsZpU6TeCl+CxnfPBGmRNRWA=', 'Content-Length': '0'} 2023-10-18T20:02:05.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c009ddc88f5b0aed-00653039bd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.505 DEBUG:boto:path=/bofrnf-32/ 2023-10-18T20:02:05.505 DEBUG:boto:auth_path=/bofrnf-32/ 2023-10-18T20:02:05.505 DEBUG:boto:Method: HEAD 2023-10-18T20:02:05.506 DEBUG:boto:Path: /bofrnf-32/ 2023-10-18T20:02:05.506 DEBUG:boto:Data: 2023-10-18T20:02:05.507 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:05.507 DEBUG:boto:Port: 8000 2023-10-18T20:02:05.508 DEBUG:boto:Params: {} 2023-10-18T20:02:05.508 DEBUG:boto:Token: None 2023-10-18T20:02:05.509 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-32/ 2023-10-18T20:02:05.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KhjQoY7xg6Ap23psNfN4hQy5MZ4= 2023-10-18T20:02:05.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KhjQoY7xg6Ap23psNfN4hQy5MZ4=', 'Content-Length': '0'} 2023-10-18T20:02:05.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4c56071c16c0429-00653039bd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-10-18T20:02:05.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-10-18T20:02:05.534 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-33 2023-10-18T20:02:05.535 DEBUG:boto:path=/bofrnf-33/ 2023-10-18T20:02:05.535 DEBUG:boto:auth_path=/bofrnf-33/ 2023-10-18T20:02:05.535 DEBUG:boto:Method: PUT 2023-10-18T20:02:05.536 DEBUG:boto:Path: /bofrnf-33/ 2023-10-18T20:02:05.536 DEBUG:boto:Data: 2023-10-18T20:02:05.537 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:05.537 DEBUG:boto:Port: 8000 2023-10-18T20:02:05.538 DEBUG:boto:Params: {} 2023-10-18T20:02:05.538 DEBUG:boto:Token: None 2023-10-18T20:02:05.539 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-33/ 2023-10-18T20:02:05.539 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LEUZV0J0tr236DUPtqOJCPFoRyA= 2023-10-18T20:02:05.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LEUZV0J0tr236DUPtqOJCPFoRyA=', 'Content-Length': '0'} 2023-10-18T20:02:05.555 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000de0a37936967ff61-00653039bd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.556 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-34 2023-10-18T20:02:05.556 DEBUG:boto:path=/bofrnf-34/ 2023-10-18T20:02:05.556 DEBUG:boto:auth_path=/bofrnf-34/ 2023-10-18T20:02:05.557 DEBUG:boto:Method: PUT 2023-10-18T20:02:05.557 DEBUG:boto:Path: /bofrnf-34/ 2023-10-18T20:02:05.557 DEBUG:boto:Data: 2023-10-18T20:02:05.558 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:05.559 DEBUG:boto:Port: 8001 2023-10-18T20:02:05.559 DEBUG:boto:Params: {} 2023-10-18T20:02:05.559 DEBUG:boto:Token: None 2023-10-18T20:02:05.560 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-34/ 2023-10-18T20:02:05.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VFWN3O9TYFW8TFFFT+h9/SLBykw= 2023-10-18T20:02:05.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VFWN3O9TYFW8TFFFT+h9/SLBykw=', 'Content-Length': '0'} 2023-10-18T20:02:05.587 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e260068013191b56-00653039bd-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.587 DEBUG:boto:path=/bofrnf-33/ 2023-10-18T20:02:05.587 DEBUG:boto:auth_path=/bofrnf-33/ 2023-10-18T20:02:05.588 DEBUG:boto:Method: HEAD 2023-10-18T20:02:05.588 DEBUG:boto:Path: /bofrnf-33/ 2023-10-18T20:02:05.589 DEBUG:boto:Data: 2023-10-18T20:02:05.589 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.589 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:05.590 DEBUG:boto:Port: 8000 2023-10-18T20:02:05.590 DEBUG:boto:Params: {} 2023-10-18T20:02:05.591 DEBUG:boto:Token: None 2023-10-18T20:02:05.591 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-33/ 2023-10-18T20:02:05.592 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EY0zFvuH1uQoJfAL/UqcClEv8ds= 2023-10-18T20:02:05.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EY0zFvuH1uQoJfAL/UqcClEv8ds=', 'Content-Length': '0'} 2023-10-18T20:02:05.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000222d858eac812d0b-00653039bd-1112-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.595 DEBUG:boto:path=/bofrnf-33/key 2023-10-18T20:02:05.595 DEBUG:boto:auth_path=/bofrnf-33/key 2023-10-18T20:02:05.596 DEBUG:boto:Method: PUT 2023-10-18T20:02:05.596 DEBUG:boto:Path: /bofrnf-33/key 2023-10-18T20:02:05.596 DEBUG:boto:Data: 2023-10-18T20:02:05.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-10-18T20:02:05.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:02:05.597 DEBUG:boto:Port: 8000 2023-10-18T20:02:05.598 DEBUG:boto:Params: {} 2023-10-18T20:02:05.598 DEBUG:boto:Token: None 2023-10-18T20:02:05.599 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-33/key 2023-10-18T20:02:05.599 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:87V62CVKcZfDJNMy/OVs1dylBPY= 2023-10-18T20:02:05.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:87V62CVKcZfDJNMy/OVs1dylBPY='} 2023-10-18T20:02:05.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f9c0fa69f20745a-00653039bd-1112-a1'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.643 DEBUG:boto:path=/bofrnf-34/ 2023-10-18T20:02:05.643 DEBUG:boto:auth_path=/bofrnf-34/ 2023-10-18T20:02:05.644 DEBUG:boto:Method: HEAD 2023-10-18T20:02:05.644 DEBUG:boto:Path: /bofrnf-34/ 2023-10-18T20:02:05.644 DEBUG:boto:Data: 2023-10-18T20:02:05.645 DEBUG:boto:Headers: {} 2023-10-18T20:02:05.645 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:05.646 DEBUG:boto:Port: 8001 2023-10-18T20:02:05.646 DEBUG:boto:Params: {} 2023-10-18T20:02:05.646 DEBUG:boto:Token: None 2023-10-18T20:02:05.647 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-34/ 2023-10-18T20:02:05.647 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m+aPfmUO7X9hH8oJ8K6LIKG2s+E= 2023-10-18T20:02:05.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m+aPfmUO7X9hH8oJ8K6LIKG2s+E=', 'Content-Length': '0'} 2023-10-18T20:02:05.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b12a10f5bcd98c06-00653039bd-1116-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.650 DEBUG:boto:path=/bofrnf-34/key 2023-10-18T20:02:05.651 DEBUG:boto:auth_path=/bofrnf-34/key 2023-10-18T20:02:05.651 DEBUG:boto:Method: PUT 2023-10-18T20:02:05.651 DEBUG:boto:Path: /bofrnf-34/key 2023-10-18T20:02:05.652 DEBUG:boto:Data: 2023-10-18T20:02:05.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-10-18T20:02:05.653 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:02:05.653 DEBUG:boto:Port: 8001 2023-10-18T20:02:05.653 DEBUG:boto:Params: {} 2023-10-18T20:02:05.654 DEBUG:boto:Token: None 2023-10-18T20:02:05.654 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Wed, 18 Oct 2023 20:02:05 GMT /bofrnf-34/key 2023-10-18T20:02:05.655 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LzYmV3HQBRYpWvXHGcFI9ccv3Yg= 2023-10-18T20:02:05.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:02:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LzYmV3HQBRYpWvXHGcFI9ccv3Yg='} 2023-10-18T20:02:05.661 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012757141eee7d7a9-00653039bd-1116-a2'), ('Date', 'Wed, 18 Oct 2023 20:02:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:02:05.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.662 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:05.710 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:05.710 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:05.742 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:02:05.753 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:02:05.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.754 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:02:05.754 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:05.574459Z" 2023-10-18T20:02:05.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.755 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:02:05.755 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:02:05.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.756 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:02:05.756 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:02:05.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.756 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:02:05.757 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:02:05.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.757 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:02:05.758 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:02:05.758 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:02:05.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.760 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:05.808 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:05.808 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:05.835 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:05.833+0000 7f97a010d780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:02:05.836 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:02:05.833+0000 7f97a010d780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:02:05.847 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:05.847 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:05.848 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:05.848 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:05.848 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:05.848 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:02:05.848 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.849 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:05.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.849 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:05.849 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:05.850 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:02:05.850 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:02:05.850 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:05.850 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:02:05.850 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:05.851 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:02:05.851 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.851 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.852 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:05.852 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:05.852 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:02:05.852 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:02:05.853 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:05.853 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:05.853 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:05.853 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:02:05.853 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.853 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.854 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:05.854 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:05.855 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:02:05.855 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:02:05.855 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:05.855 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:05.855 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:05.856 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:02:05.856 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.856 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.857 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:05.857 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:05.857 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:02:05.857 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:02:05.858 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:05.858 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:05.858 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:05.858 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:02:05.858 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:05.859 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.859 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.859 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:05.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.859 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:05.860 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:02:05.860 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:02:05.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.860 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:05Z" 2023-10-18T20:02:05.860 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:05.861 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:02:05Z" } 2023-10-18T20:02:05.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:02:05.862 DEBUG:rgw_multi.tests:log_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:02:05.863 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:02:05.863 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:02:05.864 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-18T20:02:05.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.865 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:05.912 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:05.913 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:05.957 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:02:05.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.958 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-18T20:02:05.958 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:57:53.289654Z" 2023-10-18T20:02:05.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.959 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-18T20:02:05.959 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:05.410700Z" 2023-10-18T20:02:05.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.959 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:05.960 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:50:16.613109Z" 2023-10-18T20:02:05.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:05.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:05.960 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-18T20:02:05.960 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:05.604352Z" 2023-10-18T20:02:05.961 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:05.961 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:02:05.962 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-18T20:02:05.962 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:02:05.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:05.964 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:06.015 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:06.015 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:06.058 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:06.058 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:06.058 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:06.058 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 2477610155421060198 2023-10-18T20:02:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-18T20:02:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:53.289654Z" 2023-10-18T20:02:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:06.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:06.062 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:06.062 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:06.062 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-18T20:02:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.622781Z" 2023-10-18T20:02:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:50:16.613109Z" 2023-10-18T20:02:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:06.067 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:06.067 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:06.067 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-10-18T20:02:06.067 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:06.067 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:06.068 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:06.068 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:01:37.769269Z" 2023-10-18T20:02:06.068 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:06.068 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:06.068 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:06.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:06.069 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:06.069 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:02:06.069 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:02:06.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:06.070 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:06Z" 2023-10-18T20:02:06.070 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:06.071 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2477610155421060198 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:53.289654Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.622781Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:50:16.613109Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003183", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:01:37.769269Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:02:06Z" } 2023-10-18T20:02:06.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:53.289654Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.622781Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:50:16.613109Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003183', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:01:37.769269Z'}}] 2023-10-18T20:02:06.072 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-18T20:02:06.072 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003183'} 2023-10-18T20:02:06.073 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=3 master=00000000000000000000:00000000000000003324 target=00000000000000000000:00000000000000003183 2023-10-18T20:02:11.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:11.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:11.075 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:11.135 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:11.136 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:11.181 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 2477610155421060198 2023-10-18T20:02:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-18T20:02:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:53.289654Z" 2023-10-18T20:02:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-18T20:02:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.622781Z" 2023-10-18T20:02:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:50:16.613109Z" 2023-10-18T20:02:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-10-18T20:02:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:01:37.769269Z" 2023-10-18T20:02:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:02:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:02:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:11Z" 2023-10-18T20:02:11.194 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:11.195 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2477610155421060198 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:53.289654Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.622781Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:50:16.613109Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003183", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:01:37.769269Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:02:11Z" } 2023-10-18T20:02:11.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:53.289654Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.622781Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:50:16.613109Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003183', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:01:37.769269Z'}}] 2023-10-18T20:02:11.196 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-18T20:02:11.197 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003183'} 2023-10-18T20:02:11.197 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=3 master=00000000000000000000:00000000000000003324 target=00000000000000000000:00000000000000003183 2023-10-18T20:02:16.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:16.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:16.200 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:16.259 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:16.260 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:16.305 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:16.305 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:16.305 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:16.305 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:16.306 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:16.306 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 2477610155421060198 2023-10-18T20:02:16.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:16.306 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:16.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:16.307 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:16.307 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:16.307 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:16.307 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-18T20:02:16.308 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:16.308 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:16.308 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:16.308 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:53.289654Z" 2023-10-18T20:02:16.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:16.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:16.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:16.309 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:16.309 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:16.310 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:16.310 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-18T20:02:16.310 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:16.310 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:16.310 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:16.311 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.622781Z" 2023-10-18T20:02:16.311 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:16.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:16.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:16.312 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:16.312 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:16.312 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:16.312 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:16.313 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:16.313 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:16.313 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:16.313 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:50:16.613109Z" 2023-10-18T20:02:16.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:16.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:16.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:16.314 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:16.314 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:16.315 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:16.315 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-10-18T20:02:16.315 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:16.315 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:16.315 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:16.316 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:01:37.769269Z" 2023-10-18T20:02:16.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:16.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:16.316 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:16.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:16.317 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:16.317 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:02:16.317 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:02:16.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:16.318 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:16Z" 2023-10-18T20:02:16.318 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:16.319 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2477610155421060198 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:53.289654Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.622781Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:50:16.613109Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003183", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:01:37.769269Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:02:16Z" } 2023-10-18T20:02:16.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:53.289654Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.622781Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:50:16.613109Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003183', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:01:37.769269Z'}}] 2023-10-18T20:02:16.320 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-18T20:02:16.321 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000000929', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003183'} 2023-10-18T20:02:16.321 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=3 master=00000000000000000000:00000000000000003324 target=00000000000000000000:00000000000000003183 2023-10-18T20:02:21.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:21.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:21.323 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:21.383 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:21.384 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:21.428 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:21.429 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:21.429 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:21.429 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:21.429 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:21.430 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 2477610155421060198 2023-10-18T20:02:21.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.430 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:21.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.431 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:21.431 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:21.431 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:21.431 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-18T20:02:21.431 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:21.432 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:21.432 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:21.432 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:53.289654Z" 2023-10-18T20:02:21.432 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.433 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:21.433 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:21.433 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:21.433 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-18T20:02:21.434 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:21.434 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:21.434 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:21.434 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.410700Z" 2023-10-18T20:02:21.435 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.435 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:21.436 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:21.436 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:21.436 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:21.436 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:21.436 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:21.437 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:21.437 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:50:16.613109Z" 2023-10-18T20:02:21.437 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.438 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:21.438 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:21.438 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:21.438 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-18T20:02:21.438 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:21.439 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:21.439 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:21.439 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.604352Z" 2023-10-18T20:02:21.439 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.439 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.440 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:21.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.440 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:21.440 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:02:21.440 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:02:21.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.441 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:21Z" 2023-10-18T20:02:21.441 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:21.442 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2477610155421060198 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:53.289654Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:05.410700Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:50:16.613109Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:05.604352Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:02:21Z" } 2023-10-18T20:02:21.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:53.289654Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.410700Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:50:16.613109Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.604352Z'}}] 2023-10-18T20:02:21.443 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-18T20:02:21.444 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002204', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003324'} 2023-10-18T20:02:21.444 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:02:21.444 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-18T20:02:21.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:21.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:21.446 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:21.495 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:21.495 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:21.540 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:02:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-18T20:02:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:57:16.726511Z" 2023-10-18T20:02:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-18T20:02:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:16.697848Z" 2023-10-18T20:02:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:56.644123Z" 2023-10-18T20:02:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-10-18T20:02:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:17.857801Z" 2023-10-18T20:02:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.544 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:02:21.545 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:02:21.546 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:02:21.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:21.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:21.547 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:21.598 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:21.598 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:21.641 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:21.642 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:21.642 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:21.642 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:21.642 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:21.643 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:02:21.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.643 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:21.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.643 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:21.644 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:21.644 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:21.644 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-18T20:02:21.644 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:21.644 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:21.645 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:21.645 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:16.726511Z" 2023-10-18T20:02:21.645 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.646 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:21.646 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:21.646 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:21.646 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-18T20:02:21.647 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:21.647 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:21.647 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:21.647 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.484762Z" 2023-10-18T20:02:21.647 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.648 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:21.648 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:21.648 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:21.649 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:21.649 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:21.649 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:21.649 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:21.649 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.644123Z" 2023-10-18T20:02:21.650 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:21.650 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:21.650 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:21.651 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:21.651 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-10-18T20:02:21.651 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:21.651 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:21.651 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:21.652 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.659811Z" 2023-10-18T20:02:21.652 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.652 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:21.652 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:21.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.653 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:21.653 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:02:21.653 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:02:21.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:21.653 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:21Z" 2023-10-18T20:02:21.654 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:21.655 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:16.726511Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:05.484762Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.644123Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:05.659811Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:02:21Z" } 2023-10-18T20:02:21.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:16.726511Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.484762Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.644123Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.659811Z'}}] 2023-10-18T20:02:21.656 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:02:21.656 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-18T20:02:21.657 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=3 master=00000000000000000000:00000000000000004311 target=00000000000000000000:00000000000000004170 2023-10-18T20:02:26.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:26.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:26.659 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:26.717 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:26.718 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:26.762 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:26.763 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:26.763 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:26.763 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:26.763 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:26.764 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:02:26.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:26.764 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:26.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:26.764 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:26.765 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:26.765 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:26.765 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-18T20:02:26.765 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:26.765 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:26.766 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:26.766 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:16.726511Z" 2023-10-18T20:02:26.766 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:26.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:26.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:26.767 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:26.767 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:26.767 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:26.767 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-18T20:02:26.767 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:26.768 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:26.768 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:26.768 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.484762Z" 2023-10-18T20:02:26.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:26.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:26.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:26.769 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:26.769 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:26.769 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:26.769 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:26.770 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:26.770 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:26.770 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:26.770 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.644123Z" 2023-10-18T20:02:26.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:26.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:26.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:26.771 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:26.771 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:26.772 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:26.772 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-10-18T20:02:26.772 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:26.772 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:26.772 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:26.773 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.659811Z" 2023-10-18T20:02:26.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:26.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:26.773 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:26.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:26.774 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:26.774 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:02:26.774 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:02:26.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:26.775 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:26Z" 2023-10-18T20:02:26.775 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:26.776 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:16.726511Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:05.484762Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.644123Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:05.659811Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:02:26Z" } 2023-10-18T20:02:26.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:16.726511Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.484762Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.644123Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.659811Z'}}] 2023-10-18T20:02:26.777 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:02:26.777 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-18T20:02:26.778 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=3 master=00000000000000000000:00000000000000004311 target=00000000000000000000:00000000000000004170 2023-10-18T20:02:31.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:31.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:31.780 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:31.839 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:31.840 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:31.885 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:31.886 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:31.886 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:31.886 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:31.886 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:31.886 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:02:31.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:31.887 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:31.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:31.887 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:31.888 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:31.888 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:31.888 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-18T20:02:31.888 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:31.888 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:31.889 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:31.889 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:16.726511Z" 2023-10-18T20:02:31.889 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:31.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:31.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:31.890 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:31.890 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:31.890 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:31.890 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-18T20:02:31.891 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:31.891 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:31.891 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:31.891 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.484762Z" 2023-10-18T20:02:31.891 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:31.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:31.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:31.892 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:31.892 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:31.892 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:31.893 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:31.893 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:31.893 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:31.893 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:31.894 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.644123Z" 2023-10-18T20:02:31.894 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:31.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:31.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:31.895 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:31.895 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:31.895 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:31.895 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-10-18T20:02:31.895 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:31.896 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:31.896 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:31.896 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.659811Z" 2023-10-18T20:02:31.896 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:31.896 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:31.897 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:31.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:31.897 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:31.897 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:02:31.897 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:02:31.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:31.898 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:31Z" 2023-10-18T20:02:31.898 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:31.899 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:16.726511Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:05.484762Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.644123Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:05.659811Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:02:31Z" } 2023-10-18T20:02:31.900 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:16.726511Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.484762Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.644123Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.659811Z'}}] 2023-10-18T20:02:31.900 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:02:31.901 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001068', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-18T20:02:31.901 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=3 master=00000000000000000000:00000000000000004311 target=00000000000000000000:00000000000000004170 2023-10-18T20:02:36.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:36.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:36.903 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:36.959 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:36.959 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:37.002 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:37.003 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:37.003 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:37.003 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:37.003 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:37.004 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:02:37.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:37.004 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:37.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:37.005 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:37.005 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:37.005 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:37.005 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-18T20:02:37.005 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:37.006 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:37.006 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:37.006 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:16.726511Z" 2023-10-18T20:02:37.006 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:37.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:37.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:37.007 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:37.007 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:37.007 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:37.007 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-18T20:02:37.008 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:37.008 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:37.008 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:37.008 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:16.697848Z" 2023-10-18T20:02:37.008 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:37.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:37.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:37.009 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:37.009 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:37.009 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:37.009 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:37.010 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:37.010 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:37.010 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:37.010 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.644123Z" 2023-10-18T20:02:37.010 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:37.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:37.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:37.011 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:37.011 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:37.011 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:37.012 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-10-18T20:02:37.012 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:37.012 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:37.012 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:37.012 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.659811Z" 2023-10-18T20:02:37.013 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:37.013 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:37.013 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:37.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:37.013 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:37.014 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:02:37.014 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:02:37.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:37.014 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:36Z" 2023-10-18T20:02:37.014 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:37.015 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:16.726511Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:16.697848Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.644123Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:05.659811Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:02:36Z" } 2023-10-18T20:02:37.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:16.726511Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:16.697848Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.644123Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.659811Z'}}] 2023-10-18T20:02:37.016 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:02:37.017 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004170'} 2023-10-18T20:02:37.017 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=3 master=00000000000000000000:00000000000000004311 target=00000000000000000000:00000000000000004170 2023-10-18T20:02:42.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.022 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:42.082 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.083 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:42.130 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:42.131 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:02:42.131 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:02:42.131 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:02:42.131 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:02:42.132 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:02:42.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.132 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:02:42.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.133 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:02:42.133 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:42.133 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:42.133 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-10-18T20:02:42.133 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:42.134 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:42.134 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:42.134 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:57:16.726511Z" 2023-10-18T20:02:42.134 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:42.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.135 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:02:42.135 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:42.135 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:42.135 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-18T20:02:42.136 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:42.136 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:42.136 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:42.136 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:16.697848Z" 2023-10-18T20:02:42.137 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:42.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.137 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:02:42.137 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:42.138 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:42.138 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:42.138 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:42.138 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:42.138 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:42.139 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.644123Z" 2023-10-18T20:02:42.139 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:42.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.139 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:02:42.140 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:42.140 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:02:42.140 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-10-18T20:02:42.140 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:02:42.140 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:02:42.141 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:02:42.141 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:17.857801Z" 2023-10-18T20:02:42.141 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:42.141 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:42.141 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:42.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.142 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:02:42.142 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:02:42.142 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:02:42.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.143 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:02:42Z" 2023-10-18T20:02:42.143 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:42.144 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:57:16.726511Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:16.697848Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.644123Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:17.857801Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:02:42Z" } 2023-10-18T20:02:42.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:57:16.726511Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:16.697848Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.644123Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:17.857801Z'}}] 2023-10-18T20:02:42.145 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:02:42.145 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001640', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:02:42.145 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:02:42.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.147 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:42.193 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.194 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:42.239 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:02:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-18T20:02:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:57:53.289654Z" 2023-10-18T20:02:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-18T20:02:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:16.711504Z" 2023-10-18T20:02:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:50:16.613109Z" 2023-10-18T20:02:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003465", 2023-10-18T20:02:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:17.768198Z" 2023-10-18T20:02:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:42.243 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:02:42.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:02:42.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:02:42.245 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:02:42.290 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.290 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:42.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.347 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:42.396 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.396 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:42.438 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:02:42.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.440 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:42.489 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.489 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:42.528 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:02:42.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.531 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:42.580 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.580 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:42.622 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:02:42.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.624 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:42.673 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.674 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:42.715 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:02:42.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.718 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:42.763 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.763 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:42.807 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:02:42.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.808 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002204", 2023-10-18T20:02:42.808 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:57:53.289654Z" 2023-10-18T20:02:42.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.809 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-18T20:02:42.809 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:16.711504Z" 2023-10-18T20:02:42.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.810 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:02:42.810 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:50:16.613109Z" 2023-10-18T20:02:42.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:42.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:42.811 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003465", 2023-10-18T20:02:42.811 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:17.768198Z" 2023-10-18T20:02:42.811 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:42.811 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:02:42.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:02:42.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:02:42.813 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:02:42.857 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.858 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:42.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:42.913 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:42.962 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:43.003 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:02:43.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:43.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:43.006 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:43.055 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:43.055 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:43.095 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:02:43.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:43.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:43.098 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:43.148 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:43.148 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:43.188 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:02:43.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:43.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:02:43.191 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:02:43.242 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:43.245 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:43.281 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:02:43.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2023-10-18T20:02:43.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:02:43.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:02:43.285 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:02:43.336 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:43.336 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:43.360 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:43.361 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:02:43.361 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:02:43.361 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-18T20:02:43.361 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "a2.rgw.control", 2023-10-18T20:02:43.361 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-18T20:02:43.362 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-18T20:02:43.362 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "a2.rgw.log", 2023-10-18T20:02:43.362 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-18T20:02:43.362 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-18T20:02:43.362 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-18T20:02:43.363 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-18T20:02:43.363 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-18T20:02:43.363 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-18T20:02:43.363 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-18T20:02:43.363 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-18T20:02:43.364 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "a2.rgw.otp", 2023-10-18T20:02:43.364 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T20:02:43.364 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T20:02:43.364 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T20:02:43.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.365 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T20:02:43.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.365 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T20:02:43.365 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:02:43.365 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-18T20:02:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T20:02:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T20:02:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-18T20:02:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: "compression_type": "random" 2023-10-18T20:02:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.367 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-18T20:02:43.367 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T20:02:43.367 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T20:02:43.367 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.368 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.368 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:02:43.368 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-18T20:02:43.368 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:43.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:02:43.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:02:43.370 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:02:43.419 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:02:43.419 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:02:43.461 INFO:teuthology.orchestra.run.smithi082.stderr:Sending period to new master zone 403a921b-fcfc-4ab2-b434-bf29d35a51b6 2023-10-18T20:02:43.485 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:02:43.485 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:02:43.485 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5, 2023-10-18T20:02:43.486 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T20:02:43.486 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T20:02:43.486 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T20:02:43.486 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:02:43.487 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T20:02:43.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.487 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T20:02:43.487 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T20:02:43.487 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:02:43.488 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T20:02:43.488 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:02:43.488 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:02:43.488 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.488 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:02:43.489 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:02:43.489 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:02:43.489 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:02:43.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.489 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:02:43.490 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T20:02:43.490 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:02:43.490 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:02:43.490 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.491 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:02:43.491 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:02:43.491 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:02:43.491 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:02:43.491 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:02:43.492 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:02:43.492 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:02:43.492 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:02:43.492 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:02:43.492 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:02:43.493 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:02:43.493 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:43.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.493 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:02:43.494 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:02:43.494 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:02:43.494 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T20:02:43.494 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.494 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:02:43.495 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:02:43.495 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:02:43.495 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:02:43.495 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:02:43.496 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": false, 2023-10-18T20:02:43.496 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:02:43.496 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:02:43.496 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:02:43.496 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:02:43.497 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:02:43.497 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:43.497 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.497 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.497 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:02:43.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.498 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:02:43.498 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:02:43.498 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:02:43.498 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:02:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:02:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:02:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:02:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:02:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:02:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:02:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T20:02:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T20:02:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:02:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:02:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:02:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:02:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:02:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:02:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:02:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:02:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:02:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T20:02:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:02:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:02:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:02:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:02:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:02:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:02:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:02:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:02:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:02:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:02:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:02:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:02:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:02:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:02:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T20:02:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:02:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T20:02:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:02:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:02:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:02:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:02:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:02:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:02:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:02:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:02:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:02:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:02:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:02:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:02:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:02:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:02:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:02:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:02:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:02:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:02:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:02:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:02:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:02:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:02:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:02:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T20:02:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:02:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T20:02:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:02:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T20:02:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:02:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T20:02:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:02:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:02:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T20:02:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:02:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T20:02:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:02:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T20:02:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T20:02:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:02:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:02:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:02:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:02:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:02:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T20:02:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:02:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:02:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:02:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:02:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:02:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T20:02:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:02:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:02:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:02:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:02:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:02:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T20:02:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:02:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:02:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:02:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:02:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:02:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T20:02:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:02:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:02:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:02:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:02:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:02:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:02:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:02:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:02:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T20:02:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:02:43.529 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:02:43.530 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-10-18T20:03:43.531 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-35 2023-10-18T20:03:43.532 DEBUG:boto:path=/bofrnf-35/ 2023-10-18T20:03:43.533 DEBUG:boto:auth_path=/bofrnf-35/ 2023-10-18T20:03:43.533 DEBUG:boto:Method: PUT 2023-10-18T20:03:43.534 DEBUG:boto:Path: /bofrnf-35/ 2023-10-18T20:03:43.534 DEBUG:boto:Data: 2023-10-18T20:03:43.534 DEBUG:boto:Headers: {} 2023-10-18T20:03:43.535 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:03:43.535 DEBUG:boto:Port: 8000 2023-10-18T20:03:43.536 DEBUG:boto:Params: {} 2023-10-18T20:03:43.536 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:03:43.537 DEBUG:boto:Token: None 2023-10-18T20:03:43.537 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:03:43 GMT /bofrnf-35/ 2023-10-18T20:03:43.538 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Qr1ZAMGSrE7R5RbhwEnqvdJ7Wgw= 2023-10-18T20:03:43.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:03:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Qr1ZAMGSrE7R5RbhwEnqvdJ7Wgw=', 'Content-Length': '0'} 2023-10-18T20:03:43.562 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000876410345739a172-0065303a1f-1525-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:03:43.563 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:03:43.564 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:03:43.564 DEBUG:boto:Method: PUT 2023-10-18T20:03:43.565 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:03:43.565 DEBUG:boto:Data: 2023-10-18T20:03:43.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-18T20:03:43.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:03:43.566 DEBUG:boto:Port: 8000 2023-10-18T20:03:43.567 DEBUG:boto:Params: {} 2023-10-18T20:03:43.567 DEBUG:boto:Token: None 2023-10-18T20:03:43.567 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 18 Oct 2023 20:03:43 GMT /bofrnf-35/testredirect 2023-10-18T20:03:43.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:N2GwS7GF0kOv62jVQtb1zh9tpKk= 2023-10-18T20:03:43.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:03:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:N2GwS7GF0kOv62jVQtb1zh9tpKk='} 2023-10-18T20:03:43.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a5176ad2b492e6d-0065303a1f-1525-a1'), ('Date', 'Wed, 18 Oct 2023 20:03:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:03:43.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:43.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:43.579 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:03:43.633 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:03:43.633 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:03:43.668 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:03:43.680 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:03:43.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:43.681 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659423.560083_443.1", 2023-10-18T20:03:43.682 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:03:43.560083Z" 2023-10-18T20:03:43.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:43.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:43.682 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:03:43.683 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:03:43.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:43.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:43.683 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:43.683 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:03:43.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:43.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:43.684 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:43.684 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:03:43.684 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:43.684 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:03:43.685 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:03:43.686 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:03:43.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:43.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:43.687 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:03:43.737 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:03:43.737 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:03:43.767 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:03:43.765+0000 7f2b903e3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:03:43.767 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:03:43.766+0000 7f2b903e3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:03:43.779 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:03:43.779 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:03:43.779 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:03:43.779 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:03:43.779 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:03:43.780 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:03:43.780 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:43.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:43.780 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:03:43.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:43.781 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:03:43.781 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:43.781 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:43.781 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:03:43.782 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:43.782 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:03:43.782 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:43.782 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:03:43.782 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:43.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:43.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:43.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:43.783 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:03:43.783 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:43.784 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:43.784 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:03:43.784 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:43.784 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:43.784 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:43.785 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:03:43.785 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:43.785 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:43.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:43.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:43.786 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:03:43.786 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:43.786 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:43.786 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:43.787 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:43.787 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:43.787 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:43.787 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:03:43.787 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:43.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:43.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:43.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:43.788 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:03:43.788 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:43.789 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:43.789 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:43.789 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:43.789 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:43.789 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:43.789 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:03:43.790 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:43.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:43.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:43.790 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:03:43.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:43.791 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:03:43.791 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:03:43.791 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:03:43.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:43.792 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:03:43Z" 2023-10-18T20:03:43.792 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:03:43.793 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:03:43Z" } 2023-10-18T20:03:43.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:03:43.794 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:03:43.794 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:03:43.794 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:03:48.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:48.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:48.797 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:03:48.855 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:03:48.855 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:03:48.886 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:03:48.884+0000 7f6906c5c780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:03:48.887 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:03:48.885+0000 7f6906c5c780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:03:48.899 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:03:48.900 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:03:48.900 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:03:48.900 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:03:48.900 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:03:48.901 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:03:48.901 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:48.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:48.901 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:03:48.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:48.902 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:03:48.902 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:48.902 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:48.902 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:03:48.903 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:48.903 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:03:48.903 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:48.903 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:03:48.903 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:48.903 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:48.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:48.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:48.904 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:03:48.904 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:48.904 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:48.905 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:03:48.905 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:48.905 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:48.905 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:48.906 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:03:48.906 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:48.906 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:48.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:48.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:48.907 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:03:48.907 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:48.907 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:48.907 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:48.907 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:48.908 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:48.908 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:48.908 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:03:48.908 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:48.908 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:48.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:48.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:48.909 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:03:48.909 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:48.909 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:48.910 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:48.910 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:48.910 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:48.910 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:48.911 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:03:48.911 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:48.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:48.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:48.911 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:03:48.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:48.912 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:03:48.912 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:03:48.912 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:03:48.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:48.913 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:03:48Z" 2023-10-18T20:03:48.913 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:03:48.914 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:03:48Z" } 2023-10-18T20:03:48.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:03:48.915 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:03:48.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:03:48.916 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:03:53.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:53.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:53.918 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:03:53.975 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:03:53.975 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:03:54.005 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:03:54.003+0000 7f706c6be780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:03:54.006 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:03:54.004+0000 7f706c6be780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:03:54.018 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:03:54.019 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:03:54.019 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:03:54.019 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:03:54.019 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:03:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:03:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:03:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:03:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:03:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:54.022 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:03:54.022 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:54.022 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:03:54.022 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:03:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:03:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:03:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:03:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:54.027 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:54.027 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:54.027 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:03:54.027 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:54.027 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:54.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:54.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:54.028 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:03:54.028 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:54.028 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:54.029 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:54.029 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:54.029 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:54.029 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:54.029 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:03:54.030 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:54.030 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:54.030 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:54.030 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:03:54.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:54.031 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:03:54.031 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:03:54.031 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:03:54.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:54.031 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:03:54Z" 2023-10-18T20:03:54.032 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:03:54.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:03:54Z" } 2023-10-18T20:03:54.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:03:54.034 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:03:54.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:03:54.034 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:03:59.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:59.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:03:59.037 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:03:59.094 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:03:59.095 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:03:59.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:03:59.124+0000 7f7766ef3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:03:59.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:03:59.125+0000 7f7766ef3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:03:59.139 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:03:59.139 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:03:59.140 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:03:59.140 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:03:59.140 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:03:59.140 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:03:59.140 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:59.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:59.141 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:03:59.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:59.141 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:03:59.142 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:59.142 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:59.142 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:03:59.142 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:59.142 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:03:59.143 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:59.143 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:03:59.143 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:59.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:59.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:59.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:59.144 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:03:59.144 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:59.144 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:59.144 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:03:59.145 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:59.145 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:59.145 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:59.145 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:03:59.146 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:59.146 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:59.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:59.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:59.147 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:03:59.147 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:59.147 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:59.147 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:59.147 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:59.148 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:59.148 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:59.148 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:03:59.148 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:59.148 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:59.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:59.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:03:59.149 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:03:59.149 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:03:59.149 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:03:59.149 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:03:59.150 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:03:59.150 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:03:59.150 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:03:59.150 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:03:59.151 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:03:59.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:59.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:03:59.151 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:03:59.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:59.152 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:03:59.152 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:03:59.152 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:03:59.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:03:59.153 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:03:59Z" 2023-10-18T20:03:59.153 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:03:59.154 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:03:59Z" } 2023-10-18T20:03:59.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:03:59.155 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:03:59.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:03:59.156 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:04:04.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:04.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:04.159 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:04:04.215 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:04.215 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:04.246 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:04.244+0000 7f2223a44780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:04.247 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:04.246+0000 7f2223a44780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:04.259 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:04.259 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:04:04.259 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:04:04.260 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:04:04.260 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:04:04.260 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:04.260 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:04.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:04.261 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:04:04.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:04.261 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:04:04.261 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:04.261 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:04.262 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:04:04.262 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:04.262 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:04:04.262 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:04.262 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:04:04.263 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:04.263 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:04.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:04.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:04.263 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:04:04.264 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:04.264 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:04.264 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:04:04.264 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:04.264 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:04.265 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:04.265 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:04:04.265 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:04.265 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:04.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:04.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:04.266 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:04:04.266 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:04.266 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:04.266 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:04.267 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:04.267 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:04.267 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:04.267 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:04.268 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:04.268 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:04.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:04.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:04.268 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:04:04.268 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:04.269 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:04.269 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:04.269 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:04.269 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:04.269 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:04.270 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:04.270 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:04.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:04.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:04.270 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:04.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:04.271 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:04:04.271 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:04:04.271 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:04:04.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:04.272 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:04:04Z" 2023-10-18T20:04:04.272 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:04.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:04:04Z" } 2023-10-18T20:04:04.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:04:04.274 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:04.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:04.275 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:04:09.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:09.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:09.278 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:04:09.335 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:09.335 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:09.369 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:09.368+0000 7f5a0024f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:09.370 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:09.369+0000 7f5a0024f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:09.382 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:09.382 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:04:09.383 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:04:09.383 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:04:09.383 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:04:09.383 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:09.383 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:09.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:09.384 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:04:09.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:09.384 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:04:09.384 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:09.385 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:09.385 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:04:09.385 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:09.385 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:04:09.386 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:09.386 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:04:09.386 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:09.386 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:09.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:09.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:09.387 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:04:09.387 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:09.387 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:09.387 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:04:09.388 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:09.388 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:09.388 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:09.388 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:04:09.388 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:09.389 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:09.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:09.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:09.389 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:04:09.389 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:09.389 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:09.390 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:09.390 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:09.390 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:09.390 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:09.391 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:09.391 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:09.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:09.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:09.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:09.392 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:04:09.392 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:09.392 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:09.392 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:09.392 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:09.393 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:09.393 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:09.393 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:09.393 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:09.393 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:09.394 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:09.394 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:09.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:09.394 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:04:09.394 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:04:09.395 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:04:09.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:09.395 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:04:09Z" 2023-10-18T20:04:09.395 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:09.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:04:09Z" } 2023-10-18T20:04:09.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:04:09.397 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:09.397 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:09.398 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:04:14.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:14.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:14.400 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:04:14.456 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:14.457 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:14.489 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:14.487+0000 7f0b4905c780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:14.490 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:14.488+0000 7f0b4905c780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:14.503 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:14.503 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:04:14.503 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:04:14.504 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:04:14.504 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:04:14.504 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:14.504 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:14.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:14.505 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:04:14.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:14.505 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:04:14.505 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:14.506 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:14.506 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:04:14.506 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:14.506 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:04:14.506 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:14.507 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:04:14.507 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:14.507 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:14.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:14.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:14.508 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:04:14.508 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:14.508 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:14.508 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:04:14.508 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:14.509 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:14.509 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:14.509 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:04:14.509 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:14.509 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:14.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:14.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:14.510 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:04:14.510 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:14.510 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:14.511 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:14.511 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:14.511 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:14.511 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:14.512 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:14.512 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:14.512 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:14.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:14.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:14.513 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:04:14.513 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:14.513 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:14.513 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:14.513 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:14.513 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:14.514 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:14.514 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:14.514 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:14.514 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:14.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:14.515 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:14.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:14.515 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:04:14.515 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:04:14.516 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:04:14.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:14.516 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:04:14Z" 2023-10-18T20:04:14.516 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:14.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:04:14Z" } 2023-10-18T20:04:14.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:04:14.518 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:14.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:14.519 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:04:19.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:19.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:19.522 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:04:19.579 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:19.579 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:19.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:19.608+0000 7f11435e5780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:19.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:19.609+0000 7f11435e5780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:19.623 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:19.623 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:04:19.623 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:04:19.623 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:04:19.624 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:04:19.624 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:19.624 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:19.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:19.624 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:04:19.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:19.625 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:04:19.625 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:19.625 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:19.626 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:04:19.626 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:19.626 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:04:19.626 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:19.627 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:04:19.627 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:19.627 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:19.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:19.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:19.628 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:04:19.628 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:19.628 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:19.629 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:04:19.629 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:19.629 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:19.629 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:19.630 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:04:19.630 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:19.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:19.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:19.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:19.631 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:04:19.631 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:19.631 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:19.631 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:19.631 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:19.632 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:19.632 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:19.632 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:19.632 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:19.632 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:19.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:19.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:19.633 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:04:19.633 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:19.634 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:19.634 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:19.634 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:19.634 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:19.634 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:19.635 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:19.635 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:19.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:19.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:19.635 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:19.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:19.636 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:04:19.636 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:04:19.636 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:04:19.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:19.637 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:04:19Z" 2023-10-18T20:04:19.637 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:19.638 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:04:19Z" } 2023-10-18T20:04:19.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:04:19.639 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:19.640 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:19.640 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:04:24.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:24.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:24.642 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:04:24.699 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:24.700 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:24.735 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:24.733+0000 7f58fc390780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:24.736 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:24.735+0000 7f58fc390780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:24.748 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:24.748 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:04:24.749 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:04:24.749 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:04:24.749 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:04:24.749 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:24.749 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:24.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:24.750 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:04:24.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:24.750 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:04:24.751 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:24.751 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:24.751 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:04:24.751 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:24.751 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:04:24.752 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:24.752 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:04:24.752 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:24.752 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:24.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:24.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:24.753 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:04:24.753 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:24.753 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:24.753 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:04:24.754 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:24.754 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:24.754 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:24.754 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:04:24.755 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:24.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:24.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:24.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:24.755 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:04:24.756 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:24.756 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:24.756 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:24.756 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:24.756 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:24.757 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:24.757 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:24.757 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:24.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:24.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:24.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:24.758 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:04:24.758 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:24.759 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:24.759 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:24.759 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:24.759 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:24.759 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:24.760 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:24.760 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:24.760 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:24.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:24.761 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:24.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:24.761 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:04:24.761 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:04:24.762 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:04:24.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:24.762 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:04:24Z" 2023-10-18T20:04:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:24.763 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:04:24Z" } 2023-10-18T20:04:24.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:04:24.764 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:24.765 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:24.765 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:04:29.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:29.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:29.768 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:04:29.827 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:29.827 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:29.859 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:29.857+0000 7f555c498780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:29.859 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:29.858+0000 7f555c498780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:29.872 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:29.872 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:04:29.872 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:04:29.873 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:04:29.873 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:04:29.873 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:29.873 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:29.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:29.874 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:04:29.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:29.874 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:04:29.874 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:29.875 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:29.875 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:04:29.875 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:29.875 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:04:29.876 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:29.876 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:04:29.876 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:29.876 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:29.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:29.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:29.877 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:04:29.878 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:29.878 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:29.878 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:04:29.878 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:29.878 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:29.879 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:29.879 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:04:29.879 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:29.879 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:29.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:29.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:29.880 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:04:29.880 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:29.880 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:29.881 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:29.881 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:29.881 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:29.881 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:29.881 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:29.882 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:29.882 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:29.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:29.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:29.883 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:04:29.883 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:29.883 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:29.883 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:29.883 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:29.884 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:29.884 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:29.884 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:29.884 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:29.884 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:29.885 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:29.885 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:29.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:29.885 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:04:29.886 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:04:29.886 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:04:29.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:29.886 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:04:29Z" 2023-10-18T20:04:29.886 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:29.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:04:29Z" } 2023-10-18T20:04:29.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:04:29.888 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:29.889 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:29.889 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:04:34.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:34.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:34.891 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:04:34.948 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:34.948 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:34.978+0000 7f6ffce03780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:34.979+0000 7f6ffce03780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:34.993 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:34.993 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:04:34.993 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:04:34.993 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:04:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:04:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:04:34.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:34.995 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:04:34.995 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:34.995 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:34.995 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:04:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:04:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:04:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:34.997 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:34.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:34.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:34.997 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:04:34.997 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:04:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:34.999 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:04:34.999 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:34.999 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:34.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:34.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:04:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:04:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:04:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:04:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:04:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:04:34Z" 2023-10-18T20:04:35.006 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:35.007 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:04:34Z" } 2023-10-18T20:04:35.007 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:04:35.007 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:35.008 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:35.008 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:04:40.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:40.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:40.011 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:04:40.069 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:40.069 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:40.100+0000 7f3f07496780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:40.103 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:40.101+0000 7f3f07496780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:40.116 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:40.116 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:04:40.116 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:04:40.117 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:04:40.117 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:04:40.117 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:40.117 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:40.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:40.118 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:04:40.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:40.118 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:04:40.118 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:40.119 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:40.119 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659325.574459_371.1", 2023-10-18T20:04:40.119 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:40.119 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:04:40.120 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:40.120 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:05.574459Z", 2023-10-18T20:04:40.120 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:40.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:40.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:40.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:40.121 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:04:40.121 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:40.121 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:40.122 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:04:40.122 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:40.122 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:40.122 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:40.122 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:04:40.123 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:40.123 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:40.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:40.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:40.123 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:04:40.124 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:40.124 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:40.124 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:40.124 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:40.124 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:40.125 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:40.125 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:40.125 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:40.125 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:40.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:40.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:40.126 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:04:40.126 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:40.126 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:40.127 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:40.127 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:40.127 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:40.127 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:40.127 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:40.128 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:40.128 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:40.128 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:40.128 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:40.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:40.129 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:04:40.129 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:04:40.129 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:04:40.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:40.129 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:04:40Z" 2023-10-18T20:04:40.130 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:40.131 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659325.574459_371.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:02:05.574459Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:04:40Z" } 2023-10-18T20:04:40.131 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659325.574459_371.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:02:05.574459Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:04:40.132 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:40.132 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659325.574459_371.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:40.132 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659423.560083_443.1 target=1_1697659325.574459_371.1 2023-10-18T20:04:45.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:45.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:04:45.135 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:04:45.192 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:45.192 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:45.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:45.222+0000 7fd200b2d780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:45.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:04:45.223+0000 7fd200b2d780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:04:45.238 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:45.238 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:04:45.238 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:04:45.238 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:04:45.239 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:04:45.239 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:45.239 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:45.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.239 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:04:45.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.240 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:04:45.240 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:45.240 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:45.241 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659423.560083_443.1", 2023-10-18T20:04:45.241 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:45.241 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:04:45.241 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:45.241 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:03:43.560083Z", 2023-10-18T20:04:45.242 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:45.242 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.243 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:04:45.243 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:45.243 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:45.243 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:04:45.243 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:45.244 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:45.244 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:45.244 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:04:45.244 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:45.244 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.245 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:04:45.245 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:45.246 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:45.246 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:45.246 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:45.246 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:45.246 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:45.247 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:45.247 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:45.247 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.248 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:04:45.248 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:45.248 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:04:45.248 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:04:45.248 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:04:45.249 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:04:45.249 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:04:45.249 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:04:45.249 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:45.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.250 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.250 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.250 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:04:45.251 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:04:45.251 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:04:45.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.251 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:04:45Z" 2023-10-18T20:04:45.251 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:45.252 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659423.560083_443.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:03:43.560083Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:04:45Z" } 2023-10-18T20:04:45.253 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659423.560083_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:03:43.560083Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:04:45.253 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:45.254 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:04:45.254 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:04:45.255 DEBUG:boto:path=/bofrnf-35/ 2023-10-18T20:04:45.255 DEBUG:boto:auth_path=/bofrnf-35/ 2023-10-18T20:04:45.255 DEBUG:boto:Method: HEAD 2023-10-18T20:04:45.256 DEBUG:boto:Path: /bofrnf-35/ 2023-10-18T20:04:45.256 DEBUG:boto:Data: 2023-10-18T20:04:45.257 DEBUG:boto:Headers: {} 2023-10-18T20:04:45.257 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:04:45.257 DEBUG:boto:Port: 8001 2023-10-18T20:04:45.258 DEBUG:boto:Params: {} 2023-10-18T20:04:45.258 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:04:45.259 DEBUG:boto:Token: None 2023-10-18T20:04:45.259 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:04:45 GMT /bofrnf-35/ 2023-10-18T20:04:45.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2pMFtWgYag37DuWyZ5k18FJ+kD8= 2023-10-18T20:04:45.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:04:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2pMFtWgYag37DuWyZ5k18FJ+kD8=', 'Content-Length': '0'} 2023-10-18T20:04:45.266 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005358a173f23e639e-0065303a5d-1521-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:04:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:04:45.266 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:04:45.267 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:04:45.267 DEBUG:boto:Method: HEAD 2023-10-18T20:04:45.268 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:04:45.268 DEBUG:boto:Data: 2023-10-18T20:04:45.269 DEBUG:boto:Headers: {} 2023-10-18T20:04:45.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:04:45.269 DEBUG:boto:Port: 8001 2023-10-18T20:04:45.270 DEBUG:boto:Params: {} 2023-10-18T20:04:45.270 DEBUG:boto:Token: None 2023-10-18T20:04:45.271 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:04:45 GMT /bofrnf-35/testredirect 2023-10-18T20:04:45.271 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Sep6lg7HjKQkFSC8ZH/bIueWDxU= 2023-10-18T20:04:45.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:04:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Sep6lg7HjKQkFSC8ZH/bIueWDxU=', 'Content-Length': '0'} 2023-10-18T20:04:45.273 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d34dc4f1e0462d2f-0065303a5d-1521-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:04:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:04:45.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:04:45.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:04:45.275 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:04:45.325 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:45.325 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:45.346 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:04:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:04:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-18T20:04:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "a2.rgw.control", 2023-10-18T20:04:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-18T20:04:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-18T20:04:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "a2.rgw.log", 2023-10-18T20:04:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-18T20:04:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-18T20:04:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-18T20:04:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-18T20:04:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-18T20:04:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-18T20:04:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-18T20:04:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-18T20:04:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "a2.rgw.otp", 2023-10-18T20:04:45.350 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T20:04:45.350 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T20:04:45.350 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T20:04:45.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.350 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T20:04:45.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.351 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T20:04:45.351 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:04:45.351 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-18T20:04:45.351 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T20:04:45.352 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T20:04:45.352 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-18T20:04:45.352 INFO:teuthology.orchestra.run.smithi082.stdout: "compression_type": "random" 2023-10-18T20:04:45.352 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.353 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-18T20:04:45.353 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T20:04:45.353 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T20:04:45.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.354 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.354 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:04:45.354 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-18T20:04:45.354 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:45.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:04:45.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:04:45.356 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:04:45.405 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:04:45.405 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:04:45.447 INFO:teuthology.orchestra.run.smithi082.stderr:Sending period to new master zone 403a921b-fcfc-4ab2-b434-bf29d35a51b6 2023-10-18T20:04:45.470 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:04:45.471 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:45.471 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6, 2023-10-18T20:04:45.471 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T20:04:45.471 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T20:04:45.472 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T20:04:45.472 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:04:45.472 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T20:04:45.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.472 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T20:04:45.473 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T20:04:45.473 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:04:45.473 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T20:04:45.473 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:04:45.474 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:04:45.474 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.474 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:04:45.474 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:04:45.474 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:04:45.475 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:04:45.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.475 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:04:45.475 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T20:04:45.475 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:04:45.476 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:04:45.476 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.476 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:04:45.476 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:04:45.477 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:04:45.477 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:04:45.477 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:04:45.477 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:04:45.477 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:04:45.478 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:04:45.478 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:04:45.478 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:04:45.478 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:04:45.478 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.479 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:04:45.479 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:04:45.480 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:04:45.480 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T20:04:45.480 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.480 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:04:45.480 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:04:45.481 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:04:45.481 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:04:45.481 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:04:45.481 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": false, 2023-10-18T20:04:45.481 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:04:45.482 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:04:45.482 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:04:45.482 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:04:45.482 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:04:45.482 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.483 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.483 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:04:45.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.485 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:04:45.485 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:04:45.485 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:04:45.485 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:04:45.486 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.486 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.486 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.486 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:04:45.487 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:04:45.487 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:04:45.487 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:04:45.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.487 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:04:45.488 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:04:45.488 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.488 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T20:04:45.489 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T20:04:45.489 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:04:45.489 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:04:45.489 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:04:45.489 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:04:45.490 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.490 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:04:45.490 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:04:45.490 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:04:45.491 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:04:45.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.491 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:04:45.491 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T20:04:45.491 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:04:45.492 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:04:45.492 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.492 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:04:45.492 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:04:45.492 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:04:45.493 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:04:45.493 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:04:45.493 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:04:45.493 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:04:45.493 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:04:45.494 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:04:45.495 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:04:45.495 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:04:45.495 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.496 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:04:45.496 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T20:04:45.496 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:04:45.497 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T20:04:45.497 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.497 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:04:45.497 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:04:45.497 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:04:45.498 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:04:45.498 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:04:45.498 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:04:45.498 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:04:45.499 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:04:45.499 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:04:45.499 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:04:45.499 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:04:45.499 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.500 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.500 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.500 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:04:45.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.500 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:04:45.501 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:04:45.501 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:04:45.501 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:04:45.501 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.501 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.502 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.502 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:04:45.502 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:04:45.503 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:04:45.503 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:04:45.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.503 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:04:45.503 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:04:45.504 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.504 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.504 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:04:45.504 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T20:04:45.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.505 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:04:45.505 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T20:04:45.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.506 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:04:45.506 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T20:04:45.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.506 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:04:45.507 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T20:04:45.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:04:45.507 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:04:45.507 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T20:04:45.508 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.508 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:04:45.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.508 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T20:04:45.509 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:04:45.509 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T20:04:45.509 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T20:04:45.509 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:04:45.509 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:04:45.510 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:04:45.510 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:04:45.510 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:04:45.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.510 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T20:04:45.511 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:04:45.511 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:04:45.511 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:04:45.511 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:04:45.511 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:04:45.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.512 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T20:04:45.512 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:04:45.512 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:04:45.513 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:04:45.513 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:04:45.513 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:04:45.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.513 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T20:04:45.514 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:04:45.514 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:04:45.514 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:04:45.514 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:04:45.514 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:04:45.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.515 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T20:04:45.515 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:04:45.515 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:04:45.515 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:04:45.516 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:04:45.516 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:04:45.516 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:04:45.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:04:45.517 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:04:45.517 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T20:04:45.517 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:04:45.517 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:04:45.518 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T20:05:45.519 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.520 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.521 DEBUG:boto:Method: HEAD 2023-10-18T20:05:45.521 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.522 DEBUG:boto:Data: 2023-10-18T20:05:45.522 DEBUG:boto:Headers: {} 2023-10-18T20:05:45.523 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:05:45.523 DEBUG:boto:Port: 8001 2023-10-18T20:05:45.523 DEBUG:boto:Params: {} 2023-10-18T20:05:45.524 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:05:45.525 DEBUG:boto:Token: None 2023-10-18T20:05:45.525 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.526 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/xdRN3CFwif1fg1Xz6LMJ0+JUwA= 2023-10-18T20:05:45.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/xdRN3CFwif1fg1Xz6LMJ0+JUwA=', 'Content-Length': '0'} 2023-10-18T20:05:45.532 DEBUG:boto:Response headers: [('Location', 'http://smithi082.front.sepia.ceph.com:8000/bofrnf-35/testredirect?'), ('x-amz-request-id', 'tx00000152c83a1e63a4a32-0065303a99-15ad-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.533 DEBUG:boto:Redirecting: http://smithi082.front.sepia.ceph.com/bofrnf-35/testredirect 2023-10-18T20:05:45.533 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-10-18T20:05:45.534 DEBUG:boto:Token: None 2023-10-18T20:05:45.534 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.535 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/xdRN3CFwif1fg1Xz6LMJ0+JUwA= 2023-10-18T20:05:45.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/xdRN3CFwif1fg1Xz6LMJ0+JUwA=', 'Content-Length': '0'} 2023-10-18T20:05:45.540 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000006270fa00c49832dc-0065303a99-15a9-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.541 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.541 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.542 DEBUG:boto:Method: GET 2023-10-18T20:05:45.542 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.543 DEBUG:boto:Data: 2023-10-18T20:05:45.543 DEBUG:boto:Headers: {} 2023-10-18T20:05:45.543 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:05:45.544 DEBUG:boto:Port: 8001 2023-10-18T20:05:45.544 DEBUG:boto:Params: {} 2023-10-18T20:05:45.545 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:05:45.545 DEBUG:boto:Token: None 2023-10-18T20:05:45.546 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.546 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.550 DEBUG:boto:Response headers: [('Location', 'http://smithi082.front.sepia.ceph.com:8000/bofrnf-35/testredirect?'), ('x-amz-request-id', 'tx000005db2661f5fd50a4f-0065303a99-15ad-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.550 DEBUG:boto:Redirecting: http://smithi082.front.sepia.ceph.com/bofrnf-35/testredirect 2023-10-18T20:05:45.551 DEBUG:boto:Token: None 2023-10-18T20:05:45.551 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.552 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.555 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:03:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000cac12a40f478da40-0065303a99-15a9-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.555 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.556 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.556 DEBUG:boto:Method: PUT 2023-10-18T20:05:45.557 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.557 DEBUG:boto:Data: 2023-10-18T20:05:45.557 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-18T20:05:45.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:05:45.558 DEBUG:boto:Port: 8000 2023-10-18T20:05:45.559 DEBUG:boto:Params: {} 2023-10-18T20:05:45.559 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:05:45.560 DEBUG:boto:Token: None 2023-10-18T20:05:45.560 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.561 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:scudn4Yp45sQ9M8/jEPXR4zRHUY= 2023-10-18T20:05:45.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:scudn4Yp45sQ9M8/jEPXR4zRHUY='} 2023-10-18T20:05:45.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047eb098a0fd3b041-0065303a99-15a9-a1'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.573 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.574 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.574 DEBUG:boto:Method: GET 2023-10-18T20:05:45.575 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.575 DEBUG:boto:Data: 2023-10-18T20:05:45.575 DEBUG:boto:Headers: {} 2023-10-18T20:05:45.576 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:05:45.576 DEBUG:boto:Port: 8001 2023-10-18T20:05:45.577 DEBUG:boto:Params: {} 2023-10-18T20:05:45.577 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:05:45.577 DEBUG:boto:Token: None 2023-10-18T20:05:45.578 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.578 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.582 DEBUG:boto:Response headers: [('Location', 'http://smithi082.front.sepia.ceph.com:8000/bofrnf-35/testredirect?'), ('x-amz-request-id', 'tx000003d3e2f5d063709ff-0065303a99-15ad-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.582 DEBUG:boto:Redirecting: http://smithi082.front.sepia.ceph.com/bofrnf-35/testredirect 2023-10-18T20:05:45.583 DEBUG:boto:Token: None 2023-10-18T20:05:45.583 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.584 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.589 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx0000089e5039a8d0d7325-0065303a99-15a9-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.590 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.590 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.591 DEBUG:boto:Method: PUT 2023-10-18T20:05:45.591 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.592 DEBUG:boto:Data: 2023-10-18T20:05:45.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-18T20:05:45.593 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:05:45.593 DEBUG:boto:Port: 8000 2023-10-18T20:05:45.593 DEBUG:boto:Params: {} 2023-10-18T20:05:45.594 DEBUG:boto:Token: None 2023-10-18T20:05:45.594 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1YOkRtSaU6eFuCyp6NlrTWy1nw8= 2023-10-18T20:05:45.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1YOkRtSaU6eFuCyp6NlrTWy1nw8='} 2023-10-18T20:05:45.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7f009c2e06454d1-0065303a99-15a9-a1'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.602 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.602 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.603 DEBUG:boto:Method: GET 2023-10-18T20:05:45.603 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.604 DEBUG:boto:Data: 2023-10-18T20:05:45.604 DEBUG:boto:Headers: {} 2023-10-18T20:05:45.604 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:05:45.605 DEBUG:boto:Port: 8001 2023-10-18T20:05:45.605 DEBUG:boto:Params: {} 2023-10-18T20:05:45.606 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:05:45.606 DEBUG:boto:Token: None 2023-10-18T20:05:45.606 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.607 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.610 DEBUG:boto:Response headers: [('Location', 'http://smithi082.front.sepia.ceph.com:8000/bofrnf-35/testredirect?'), ('x-amz-request-id', 'tx000000b849ae0af95f3db-0065303a99-15ad-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.611 DEBUG:boto:Redirecting: http://smithi082.front.sepia.ceph.com/bofrnf-35/testredirect 2023-10-18T20:05:45.611 DEBUG:boto:Token: None 2023-10-18T20:05:45.611 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.612 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.617 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx0000097569e8a922d1a4f-0065303a99-15a9-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.658 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.659 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.659 DEBUG:boto:Method: PUT 2023-10-18T20:05:45.660 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.660 DEBUG:boto:Data: 2023-10-18T20:05:45.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-18T20:05:45.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:05:45.661 DEBUG:boto:Port: 8000 2023-10-18T20:05:45.662 DEBUG:boto:Params: {} 2023-10-18T20:05:45.662 DEBUG:boto:Token: None 2023-10-18T20:05:45.662 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.663 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EKcs6lSj/GqHBitp2mlBT13O+XY= 2023-10-18T20:05:45.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EKcs6lSj/GqHBitp2mlBT13O+XY='} 2023-10-18T20:05:45.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f85b5c648abe355-0065303a99-15a9-a1'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.670 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.671 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.671 DEBUG:boto:Method: GET 2023-10-18T20:05:45.671 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.672 DEBUG:boto:Data: 2023-10-18T20:05:45.672 DEBUG:boto:Headers: {} 2023-10-18T20:05:45.673 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:05:45.673 DEBUG:boto:Port: 8001 2023-10-18T20:05:45.674 DEBUG:boto:Params: {} 2023-10-18T20:05:45.674 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:05:45.675 DEBUG:boto:Token: None 2023-10-18T20:05:45.675 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.679 DEBUG:boto:Response headers: [('Location', 'http://smithi082.front.sepia.ceph.com:8000/bofrnf-35/testredirect?'), ('x-amz-request-id', 'tx000007009a2b3d10cdf32-0065303a99-15ad-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.680 DEBUG:boto:Redirecting: http://smithi082.front.sepia.ceph.com/bofrnf-35/testredirect 2023-10-18T20:05:45.680 DEBUG:boto:Token: None 2023-10-18T20:05:45.681 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.681 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.684 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000daf9ca719ed04c0e-0065303a99-15a9-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.684 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.685 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.685 DEBUG:boto:Method: PUT 2023-10-18T20:05:45.686 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.686 DEBUG:boto:Data: 2023-10-18T20:05:45.687 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-18T20:05:45.687 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:05:45.687 DEBUG:boto:Port: 8000 2023-10-18T20:05:45.688 DEBUG:boto:Params: {} 2023-10-18T20:05:45.688 DEBUG:boto:Token: None 2023-10-18T20:05:45.689 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.689 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ytrxUW6HxV0+YF/znVZM6bJqqsM= 2023-10-18T20:05:45.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ytrxUW6HxV0+YF/znVZM6bJqqsM='} 2023-10-18T20:05:45.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098eea33cd285cf0b-0065303a99-15a9-a1'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.697 DEBUG:boto:path=/bofrnf-35/testredirect 2023-10-18T20:05:45.697 DEBUG:boto:auth_path=/bofrnf-35/testredirect 2023-10-18T20:05:45.698 DEBUG:boto:Method: GET 2023-10-18T20:05:45.699 DEBUG:boto:Path: /bofrnf-35/testredirect 2023-10-18T20:05:45.699 DEBUG:boto:Data: 2023-10-18T20:05:45.700 DEBUG:boto:Headers: {} 2023-10-18T20:05:45.701 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:05:45.701 DEBUG:boto:Port: 8001 2023-10-18T20:05:45.702 DEBUG:boto:Params: {} 2023-10-18T20:05:45.703 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:05:45.703 DEBUG:boto:Token: None 2023-10-18T20:05:45.704 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.705 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.708 DEBUG:boto:Response headers: [('Location', 'http://smithi082.front.sepia.ceph.com:8000/bofrnf-35/testredirect?'), ('x-amz-request-id', 'tx00000b3445fcdcecce09e-0065303a99-15ad-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.709 DEBUG:boto:Redirecting: http://smithi082.front.sepia.ceph.com/bofrnf-35/testredirect 2023-10-18T20:05:45.709 DEBUG:boto:Token: None 2023-10-18T20:05:45.710 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:05:45 GMT /bofrnf-35/testredirect 2023-10-18T20:05:45.711 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY= 2023-10-18T20:05:45.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:05:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16dEA+dCT9q0Ww0oP4LTisyLOPY=', 'Content-Length': '0'} 2023-10-18T20:05:45.714 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx000000999a135e3e7871b-0065303a99-15a9-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:05:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:05:45.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:05:45.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:05:45.716 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:05:45.771 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:05:45.771 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:05:45.792 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:05:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:05:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:05:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-18T20:05:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "a2.rgw.control", 2023-10-18T20:05:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-18T20:05:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-18T20:05:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "a2.rgw.log", 2023-10-18T20:05:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-18T20:05:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-18T20:05:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-18T20:05:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-18T20:05:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-18T20:05:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-18T20:05:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-18T20:05:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-18T20:05:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "a2.rgw.otp", 2023-10-18T20:05:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T20:05:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T20:05:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T20:05:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T20:05:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T20:05:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:05:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-18T20:05:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T20:05:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T20:05:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-18T20:05:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "compression_type": "random" 2023-10-18T20:05:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-18T20:05:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T20:05:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T20:05:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:05:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-18T20:05:45.802 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:05:45.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:05:45.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:05:45.804 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:05:45.854 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:05:45.854 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:05:45.895 INFO:teuthology.orchestra.run.smithi082.stderr:Sending period to new master zone 403a921b-fcfc-4ab2-b434-bf29d35a51b6 2023-10-18T20:05:45.921 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:05:45.921 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:05:45.921 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7, 2023-10-18T20:05:45.921 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T20:05:45.922 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T20:05:45.922 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T20:05:45.922 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:05:45.922 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T20:05:45.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.923 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T20:05:45.923 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T20:05:45.923 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:05:45.923 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T20:05:45.924 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:05:45.924 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:05:45.924 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.924 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:05:45.924 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:05:45.925 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:05:45.925 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:05:45.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.925 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:05:45.925 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T20:05:45.925 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:05:45.926 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:05:45.926 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.926 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:05:45.926 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:05:45.927 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:05:45.927 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:05:45.927 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:05:45.927 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:05:45.927 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:05:45.928 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:05:45.928 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:05:45.928 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:05:45.928 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:05:45.929 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:05:45.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.929 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:05:45.929 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:05:45.930 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:05:45.930 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T20:05:45.930 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.930 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:05:45.930 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:05:45.931 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:05:45.931 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:05:45.931 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:05:45.931 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:05:45.931 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:05:45.932 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:05:45.932 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:05:45.932 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:05:45.932 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:05:45.932 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:05:45.933 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.933 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.933 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:05:45.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.934 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:05:45.934 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:05:45.934 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:05:45.934 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:05:45.934 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:05:45.935 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.935 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.935 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:05:45.935 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:05:45.935 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:05:45.936 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:05:45.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.936 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:05:45.936 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:05:45.936 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:05:45.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.937 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T20:05:45.937 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T20:05:45.937 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:05:45.938 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:05:45.938 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:05:45.938 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:05:45.938 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.939 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:05:45.939 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:05:45.939 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:05:45.939 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:05:45.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.940 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:05:45.940 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T20:05:45.940 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:05:45.940 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:05:45.940 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.941 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:05:45.941 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:05:45.941 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:05:45.941 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:05:45.941 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:05:45.942 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:05:45.942 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:05:45.942 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:05:45.942 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:05:45.943 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:05:45.943 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:05:45.944 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:05:45.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.944 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:05:45.945 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T20:05:45.945 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:05:45.945 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T20:05:45.945 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.945 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:05:45.946 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:05:45.946 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:05:45.946 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:05:45.946 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:05:45.946 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:05:45.947 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:05:45.947 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:05:45.947 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:05:45.947 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:05:45.947 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:05:45.948 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:05:45.948 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.948 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.948 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:05:45.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.949 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:05:45.949 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:05:45.949 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:05:45.949 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:05:45.949 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:05:45.950 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.950 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.950 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:05:45.950 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:05:45.950 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:05:45.951 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:05:45.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.951 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:05:45.951 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:05:45.951 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:05:45.952 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.952 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:05:45.952 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T20:05:45.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.952 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:05:45.953 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T20:05:45.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.953 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:05:45.954 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T20:05:45.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.954 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:05:45.954 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T20:05:45.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:05:45.955 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:05:45.955 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T20:05:45.955 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.956 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:05:45.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.956 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T20:05:45.956 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:05:45.956 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T20:05:45.957 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T20:05:45.957 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:05:45.957 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:05:45.957 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:05:45.957 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:05:45.958 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:05:45.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.958 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T20:05:45.958 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:05:45.958 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:05:45.959 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:05:45.959 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:05:45.959 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:05:45.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.959 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T20:05:45.960 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:05:45.960 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:05:45.960 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:05:45.960 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:05:45.960 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:05:45.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.961 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T20:05:45.961 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:05:45.961 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:05:45.961 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:05:45.962 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:05:45.962 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:05:45.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.962 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T20:05:45.962 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:05:45.963 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:05:45.963 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:05:45.963 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:05:45.963 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:05:45.963 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:05:45.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:05:45.964 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:05:45.964 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T20:05:45.964 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:05:45.964 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:05:45.965 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-10-18T20:06:45.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:06:45.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:06:45.969 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:06:46.026 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:06:46.027 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:06:46.048 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:06:46.048 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:06:46.049 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:06:46.049 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-18T20:06:46.049 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "a2.rgw.control", 2023-10-18T20:06:46.049 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-18T20:06:46.049 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-18T20:06:46.050 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "a2.rgw.log", 2023-10-18T20:06:46.050 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-18T20:06:46.050 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-18T20:06:46.050 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-18T20:06:46.051 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-18T20:06:46.051 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-18T20:06:46.051 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-18T20:06:46.051 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-18T20:06:46.051 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-18T20:06:46.052 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "a2.rgw.otp", 2023-10-18T20:06:46.052 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T20:06:46.052 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T20:06:46.052 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T20:06:46.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.053 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T20:06:46.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.053 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T20:06:46.053 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:06:46.054 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-18T20:06:46.054 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T20:06:46.054 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T20:06:46.054 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-18T20:06:46.054 INFO:teuthology.orchestra.run.smithi082.stdout: "compression_type": "random" 2023-10-18T20:06:46.055 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.055 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-18T20:06:46.055 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T20:06:46.055 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T20:06:46.056 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.056 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.056 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.056 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:06:46.056 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-18T20:06:46.057 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:06:46.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:06:46.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:06:46.059 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:06:46.109 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:06:46.109 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:06:46.151 INFO:teuthology.orchestra.run.smithi082.stderr:Sending period to new master zone 403a921b-fcfc-4ab2-b434-bf29d35a51b6 2023-10-18T20:06:46.176 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:06:46.176 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:06:46.177 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8, 2023-10-18T20:06:46.177 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T20:06:46.177 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T20:06:46.177 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T20:06:46.178 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:06:46.178 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T20:06:46.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.178 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T20:06:46.178 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T20:06:46.179 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:06:46.179 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T20:06:46.179 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:06:46.179 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:06:46.179 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.180 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:06:46.180 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:06:46.180 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:06:46.180 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:06:46.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.181 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:06:46.181 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T20:06:46.181 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:06:46.181 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:06:46.181 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.182 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:06:46.182 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:06:46.182 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:06:46.182 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:06:46.182 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:06:46.183 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:06:46.183 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:06:46.183 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:06:46.183 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:06:46.183 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:06:46.184 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:06:46.184 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:06:46.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.184 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:06:46.185 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:06:46.185 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:06:46.185 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T20:06:46.185 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.185 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:06:46.186 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:06:46.186 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:06:46.186 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:06:46.186 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:06:46.186 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:06:46.187 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:06:46.187 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:06:46.187 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:06:46.187 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:06:46.187 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:06:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:06:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:06:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:06:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:06:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:06:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:06:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:06:46.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.190 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.190 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:06:46.190 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:06:46.190 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:06:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:06:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:06:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:06:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:06:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T20:06:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T20:06:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:06:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:06:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:06:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:06:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:06:46.194 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:06:46.194 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:06:46.194 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:06:46.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:06:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T20:06:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:06:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:06:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.196 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:06:46.196 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:06:46.196 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:06:46.196 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:06:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:06:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:06:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:06:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:06:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:06:46.198 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:06:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:06:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:06:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:06:46.200 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T20:06:46.200 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:06:46.200 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T20:06:46.200 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.201 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:06:46.201 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:06:46.201 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:06:46.201 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:06:46.201 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:06:46.202 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:06:46.202 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:06:46.202 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:06:46.202 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:06:46.202 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:06:46.203 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:06:46.203 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:06:46.203 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.203 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.203 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:06:46.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.204 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:06:46.204 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:06:46.204 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:06:46.205 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:06:46.205 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:06:46.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.205 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.205 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:06:46.206 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:06:46.206 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:06:46.206 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:06:46.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.206 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:06:46.207 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:06:46.207 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:06:46.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.207 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:06:46.208 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T20:06:46.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.208 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:06:46.208 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T20:06:46.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.209 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:06:46.209 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T20:06:46.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.210 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:06:46.210 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T20:06:46.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:06:46.210 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:06:46.211 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T20:06:46.211 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.211 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:06:46.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.211 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T20:06:46.212 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:06:46.212 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T20:06:46.212 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T20:06:46.212 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:06:46.212 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:06:46.213 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:06:46.213 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:06:46.213 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:06:46.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.213 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T20:06:46.214 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:06:46.214 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:06:46.214 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:06:46.214 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:06:46.215 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:06:46.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.215 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T20:06:46.215 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:06:46.215 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:06:46.215 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:06:46.216 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:06:46.216 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:06:46.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.216 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T20:06:46.216 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:06:46.217 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:06:46.217 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:06:46.217 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:06:46.217 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:06:46.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.218 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T20:06:46.218 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:06:46.218 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:06:46.218 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:06:46.219 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:06:46.219 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:06:46.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:06:46.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:06:46.219 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:06:46.220 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T20:06:46.220 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:06:46.220 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:06:46.220 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-10-18T20:07:46.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-10-18T20:07:46.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:07:46.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:07:46.225 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:07:46.282 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:46.283 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:46.315 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:07:46.328 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:07:46.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:46.328 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659423.560083_443.1", 2023-10-18T20:07:46.329 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:03:43.560083Z" 2023-10-18T20:07:46.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:46.330 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:07:46.330 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:07:46.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:46.331 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:07:46.332 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:07:46.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:46.333 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:07:46.333 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:07:46.333 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:46.334 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:07:46.335 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:07:46.335 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:07:46.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:07:46.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:07:46.337 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:07:46.387 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:46.388 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:46.415 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:07:46.414+0000 7fda3a210780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:07:46.416 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:07:46.415+0000 7fda3a210780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:07:46.430 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:07:46.431 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:07:46.431 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:07:46.431 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:07:46.431 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:07:46.432 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:07:46.432 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:07:46.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.432 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:07:46.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:46.433 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:07:46.433 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:07:46.433 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:07:46.433 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659423.560083_443.1", 2023-10-18T20:07:46.434 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:07:46.434 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:07:46.434 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:07:46.434 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:03:43.560083Z", 2023-10-18T20:07:46.435 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:07:46.435 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:46.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:46.435 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:07:46.436 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:07:46.436 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:07:46.436 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:07:46.437 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:07:46.437 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:07:46.437 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:07:46.437 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:07:46.437 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:07:46.438 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:46.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:46.438 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:07:46.439 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:07:46.439 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:07:46.439 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:07:46.439 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:07:46.439 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:07:46.441 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:07:46.441 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:07:46.441 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:07:46.442 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:46.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:46.442 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:07:46.443 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:07:46.443 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:07:46.443 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:07:46.443 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:07:46.443 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:07:46.444 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:07:46.444 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:07:46.444 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:07:46.444 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:46.445 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:46.445 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:46.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.446 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:07:46.446 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:07:46.446 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:07:46.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.447 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:07:46Z" 2023-10-18T20:07:46.447 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:07:46.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659423.560083_443.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:03:43.560083Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:07:46Z" } 2023-10-18T20:07:46.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659423.560083_443.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:03:43.560083Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:07:46.451 DEBUG:rgw_multi.tests:log_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:07:46.452 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659423.560083_443.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:07:46.452 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:07:46.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:46.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'HD6X1XK4CX1C8MO8', '--secret', 'iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:46.455 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key HD6X1XK4CX1C8MO8 --secret iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:46.506 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:46.507 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:46.520 INFO:teuthology.orchestra.run.smithi082.stderr:NOTICE: set zone's realm_id=7565323d-5a72-4e51-af8d-a123f7dfc36a 2023-10-18T20:07:46.534 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:07:46.535 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "050fb0dc-6343-47dc-864d-a77a88ca1576", 2023-10-18T20:07:46.535 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "remove", 2023-10-18T20:07:46.535 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "remove.rgw.meta:root", 2023-10-18T20:07:46.535 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "remove.rgw.control", 2023-10-18T20:07:46.535 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "remove.rgw.log:gc", 2023-10-18T20:07:46.536 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "remove.rgw.log:lc", 2023-10-18T20:07:46.536 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "remove.rgw.log", 2023-10-18T20:07:46.536 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-10-18T20:07:46.536 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-10-18T20:07:46.536 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-10-18T20:07:46.537 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-10-18T20:07:46.537 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-10-18T20:07:46.537 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-10-18T20:07:46.537 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-10-18T20:07:46.538 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-10-18T20:07:46.538 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "remove.rgw.otp", 2023-10-18T20:07:46.538 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T20:07:46.538 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "HD6X1XK4CX1C8MO8", 2023-10-18T20:07:46.539 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "iWKHPt2qfpJuJ4YJ8XxxzFLNUlj85VEX" 2023-10-18T20:07:46.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.539 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T20:07:46.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:46.539 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T20:07:46.540 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:07:46.540 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "remove.rgw.buckets.index", 2023-10-18T20:07:46.540 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T20:07:46.540 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T20:07:46.540 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "remove.rgw.buckets.data" 2023-10-18T20:07:46.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:46.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:46.541 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-10-18T20:07:46.541 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T20:07:46.541 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T20:07:46.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:46.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:46.542 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:46.542 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:46.543 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "remove.rgw.log:notif" 2023-10-18T20:07:46.543 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:07:46.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:46.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:46.545 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:46.593 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:46.594 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:46.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:07:46.604+0000 7faa8f7bf780 0 period (67bf1060-aa43-46d9-8036-efd54ef55d7d does not have zone 050fb0dc-6343-47dc-864d-a77a88ca1576 configured 2023-10-18T20:07:53.480 INFO:teuthology.orchestra.run.smithi082.stderr:Sending period to new master zone 403a921b-fcfc-4ab2-b434-bf29d35a51b6 2023-10-18T20:07:53.506 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:07:53.506 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:07:53.507 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9, 2023-10-18T20:07:53.507 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T20:07:53.507 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T20:07:53.507 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T20:07:53.507 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:07:53.508 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T20:07:53.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.508 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T20:07:53.508 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T20:07:53.509 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:07:53.509 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T20:07:53.509 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.509 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:53.509 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.510 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:53.510 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:53.510 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:53.510 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:53.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.511 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "050fb0dc-6343-47dc-864d-a77a88ca1576", 2023-10-18T20:07:53.511 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "remove", 2023-10-18T20:07:53.511 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [], 2023-10-18T20:07:53.511 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:53.511 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:53.512 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:53.512 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:53.512 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:53.512 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:53.512 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:53.513 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:53.513 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:53.513 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:53.513 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.513 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.514 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:53.514 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T20:07:53.514 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.515 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:53.515 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.515 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:53.515 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:53.515 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:53.515 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:53.516 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:53.516 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:53.516 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:53.516 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:53.516 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:53.517 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:53.517 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.517 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.518 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:07:53.518 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:07:53.518 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.518 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T20:07:53.519 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.519 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:53.519 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:53.519 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:53.519 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:53.520 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:53.520 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:53.520 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:53.520 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:53.520 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:53.520 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:53.521 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.521 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.521 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.521 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.521 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:53.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.522 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:53.522 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:53.522 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:53.522 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:53.523 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.523 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.523 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.523 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:53.523 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:53.524 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:53.524 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:53.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.524 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:53.524 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.525 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.525 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T20:07:53.525 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T20:07:53.525 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:07:53.526 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:07:53.526 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.526 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:07:53.526 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.526 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:53.527 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:53.527 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:53.527 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:53.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.527 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:53.528 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T20:07:53.528 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.528 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:07:53.528 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.528 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:53.529 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:53.529 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:53.529 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:53.529 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:53.529 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:53.530 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:53.530 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:53.530 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:53.530 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:53.530 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.531 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.531 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:07:53.531 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T20:07:53.532 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.532 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T20:07:53.532 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.532 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:53.532 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:53.533 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:53.533 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:53.533 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:53.533 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:53.533 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:53.534 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:53.534 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:53.534 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:53.534 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.534 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.535 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.535 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.535 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:53.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.535 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:53.536 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:53.536 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:53.536 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:53.536 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.537 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.537 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:53.537 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:53.537 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:53.537 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:53.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.538 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:53.538 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.538 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.539 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.539 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T20:07:53.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.539 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "050fb0dc-6343-47dc-864d-a77a88ca1576", 2023-10-18T20:07:53.540 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3212349176 2023-10-18T20:07:53.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.540 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:53.540 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T20:07:53.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.541 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:07:53.541 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T20:07:53.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.542 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:53.542 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T20:07:53.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.542 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:07:53.543 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T20:07:53.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.543 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.543 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T20:07:53.543 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:53.544 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T20:07:53.544 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T20:07:53.544 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:07:53.544 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:07:53.544 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:07:53.545 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:07:53.545 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:07:53.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.545 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T20:07:53.545 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:07:53.546 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:07:53.546 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:07:53.546 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:07:53.546 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:07:53.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.547 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T20:07:53.547 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:53.547 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:53.547 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:53.547 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:53.547 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:53.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.548 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T20:07:53.548 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:53.548 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:53.548 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:53.549 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:53.549 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:53.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.549 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T20:07:53.549 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:53.550 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:53.550 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:53.550 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:53.550 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:53.550 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.551 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:53.552 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T20:07:53.552 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:07:53.552 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:07:53.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:53.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:53.554 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:53.601 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:53.602 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:53.617 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:07:53.618 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T20:07:53.618 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T20:07:53.618 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:07:53.618 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T20:07:53.619 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.619 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:53.619 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.619 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:53.619 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:53.620 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:53.620 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:53.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.620 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:53.621 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T20:07:53.621 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.621 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:53.621 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.621 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:53.622 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:53.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:53.622 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:53.622 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:53.623 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:53.623 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:53.623 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:53.623 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:53.623 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:53.624 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.624 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.624 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:07:53.625 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:07:53.625 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.625 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T20:07:53.625 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.625 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:53.626 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:53.626 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:53.626 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:53.626 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:53.626 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:53.627 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:53.627 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:53.627 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:53.627 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:53.627 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.628 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.628 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.628 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.628 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:53.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.629 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:53.629 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:53.629 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:53.629 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:53.630 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.630 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.630 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:53.630 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:53.631 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:53.631 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:53.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.631 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:53.631 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.632 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.632 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:07:53.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:53.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:53.634 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:53.687 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:53.688 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:53.707 INFO:teuthology.orchestra.run.smithi082.stderr:zone name remove not found in zonegroup a 2023-10-18T20:07:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-18T20:07:53.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:53.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:53.714 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:53.770 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:53.770 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:53.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:53.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:53.814 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:53.884 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:53.884 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:53.908 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:07:53.907+0000 7f1a52b9f780 0 WARNING: can't generate connection for zone 050fb0dc-6343-47dc-864d-a77a88ca1576 id remove: no endpoints defined 2023-10-18T20:07:53.982 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:07:53.982 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:07:53.982 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10, 2023-10-18T20:07:53.982 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T20:07:53.983 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T20:07:53.983 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T20:07:53.983 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:07:53.983 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T20:07:53.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.984 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T20:07:53.984 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T20:07:53.984 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:07:53.984 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T20:07:53.985 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.985 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:53.985 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.985 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:53.985 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:53.986 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:53.986 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:53.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.986 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:53.986 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T20:07:53.987 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.987 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:53.987 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.987 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:53.987 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:53.988 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:53.988 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:53.988 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:53.988 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:53.988 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:53.989 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:53.989 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:53.989 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:53.989 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.989 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.990 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:07:53.990 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:07:53.991 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.991 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T20:07:53.991 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.991 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:53.991 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:53.992 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:53.992 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:53.992 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:53.992 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:53.992 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:53.993 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:53.993 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:53.993 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:53.993 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.993 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.994 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.994 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.994 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:53.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.994 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:53.995 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:53.995 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:53.995 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:53.995 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.995 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:53.996 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.996 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:53.996 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:53.996 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:53.996 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:53.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.997 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:53.997 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:53.997 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:53.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:53.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:53.998 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T20:07:53.998 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T20:07:53.998 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:07:53.999 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:07:53.999 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:53.999 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:07:53.999 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:53.999 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:54.000 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:54.000 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:54.000 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:54.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.000 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:54.001 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T20:07:54.001 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.001 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:07:54.001 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.002 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.002 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.002 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.002 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.002 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.003 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.003 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.003 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.003 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.003 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.006 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.006 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.007 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:07:54.007 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T20:07:54.007 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.007 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T20:07:54.008 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.008 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.008 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.008 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.008 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.009 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.009 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.009 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.009 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.009 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.010 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.010 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.010 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.010 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.011 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.011 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:54.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.011 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:54.011 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:54.012 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:54.012 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:54.012 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.012 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.012 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.013 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:54.013 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.013 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:54.013 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:54.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.014 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:54.014 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.014 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.014 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.014 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.015 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T20:07:54.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.015 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.015 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T20:07:54.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.016 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:07:54.016 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T20:07:54.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.017 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:54.017 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T20:07:54.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.017 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:07:54.018 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T20:07:54.018 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.018 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.018 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T20:07:54.019 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.019 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T20:07:54.019 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T20:07:54.019 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:07:54.019 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:07:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:07:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:07:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:07:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.020 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T20:07:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:07:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:07:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:07:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:07:54.021 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:07:54.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.022 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T20:07:54.022 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:54.022 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.023 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T20:07:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:54.024 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T20:07:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:54.025 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.026 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.027 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T20:07:54.027 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:07:54.027 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:07:54.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-10-18T20:07:54.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.030 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:54.088 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:54.089 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:54.116 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:07:54.116 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "f7d9dd23-1412-40c2-bf2c-973f21da18be", 2023-10-18T20:07:54.116 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "remove_zg", 2023-10-18T20:07:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "remove_zg", 2023-10-18T20:07:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:07:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [], 2023-10-18T20:07:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:54.118 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "", 2023-10-18T20:07:54.118 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [], 2023-10-18T20:07:54.118 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:54.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [] 2023-10-18T20:07:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.120 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:54.120 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.120 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:54.120 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:54.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.121 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:54.121 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.121 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.121 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:07:54.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.123 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:54.183 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:54.183 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:54.228 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:07:54.226+0000 7f6505d1f780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-10-18T20:07:54.232 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:07:54.232 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "f040972e-89ae-4174-beab-79dc74c3fc06", 2023-10-18T20:07:54.232 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "remove", 2023-10-18T20:07:54.233 INFO:teuthology.orchestra.run.smithi082.stdout: "domain_root": "remove.rgw.meta:root", 2023-10-18T20:07:54.233 INFO:teuthology.orchestra.run.smithi082.stdout: "control_pool": "remove.rgw.control", 2023-10-18T20:07:54.233 INFO:teuthology.orchestra.run.smithi082.stdout: "gc_pool": "remove.rgw.log:gc", 2023-10-18T20:07:54.233 INFO:teuthology.orchestra.run.smithi082.stdout: "lc_pool": "remove.rgw.log:lc", 2023-10-18T20:07:54.233 INFO:teuthology.orchestra.run.smithi082.stdout: "log_pool": "remove.rgw.log", 2023-10-18T20:07:54.234 INFO:teuthology.orchestra.run.smithi082.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-10-18T20:07:54.234 INFO:teuthology.orchestra.run.smithi082.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-10-18T20:07:54.234 INFO:teuthology.orchestra.run.smithi082.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-10-18T20:07:54.234 INFO:teuthology.orchestra.run.smithi082.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-10-18T20:07:54.235 INFO:teuthology.orchestra.run.smithi082.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-10-18T20:07:54.235 INFO:teuthology.orchestra.run.smithi082.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-10-18T20:07:54.235 INFO:teuthology.orchestra.run.smithi082.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-10-18T20:07:54.235 INFO:teuthology.orchestra.run.smithi082.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-10-18T20:07:54.235 INFO:teuthology.orchestra.run.smithi082.stdout: "otp_pool": "remove.rgw.otp", 2023-10-18T20:07:54.236 INFO:teuthology.orchestra.run.smithi082.stdout: "system_key": { 2023-10-18T20:07:54.236 INFO:teuthology.orchestra.run.smithi082.stdout: "access_key": "", 2023-10-18T20:07:54.236 INFO:teuthology.orchestra.run.smithi082.stdout: "secret_key": "" 2023-10-18T20:07:54.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.236 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_pools": [ 2023-10-18T20:07:54.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.237 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "default-placement", 2023-10-18T20:07:54.237 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:07:54.237 INFO:teuthology.orchestra.run.smithi082.stdout: "index_pool": "remove.rgw.buckets.index", 2023-10-18T20:07:54.237 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": { 2023-10-18T20:07:54.238 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD": { 2023-10-18T20:07:54.238 INFO:teuthology.orchestra.run.smithi082.stdout: "data_pool": "remove.rgw.buckets.data" 2023-10-18T20:07:54.238 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.238 INFO:teuthology.orchestra.run.smithi082.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-10-18T20:07:54.239 INFO:teuthology.orchestra.run.smithi082.stdout: "index_type": 0, 2023-10-18T20:07:54.239 INFO:teuthology.orchestra.run.smithi082.stdout: "inline_data": true 2023-10-18T20:07:54.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.239 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.240 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.240 INFO:teuthology.orchestra.run.smithi082.stdout: "notif_pool": "remove.rgw.log:notif" 2023-10-18T20:07:54.240 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:07:54.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.242 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:54.303 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:54.303 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:54.379 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:07:54.379 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:07:54.380 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11, 2023-10-18T20:07:54.380 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T20:07:54.380 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T20:07:54.381 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T20:07:54.381 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:07:54.382 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T20:07:54.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.382 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T20:07:54.383 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T20:07:54.383 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:07:54.384 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T20:07:54.384 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.384 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:54.385 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.385 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:54.385 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:54.385 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.386 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:54.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.386 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.386 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T20:07:54.387 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.387 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:54.387 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.387 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.387 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.388 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.388 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.388 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.388 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.389 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.389 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.389 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.389 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.389 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.390 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.390 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:07:54.390 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:07:54.391 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.391 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T20:07:54.391 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.391 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.391 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.392 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.392 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.392 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.392 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.393 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.393 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.393 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.394 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.394 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.394 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.394 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.395 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.395 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:54.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.396 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:54.396 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:54.396 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:54.397 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:54.397 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.397 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.398 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.398 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:54.398 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.399 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:54.399 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:54.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.400 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:54.400 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.400 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.401 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T20:07:54.401 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T20:07:54.402 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:07:54.402 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:07:54.402 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.403 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:07:54.403 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.403 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:54.403 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:54.404 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:54.404 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:54.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.404 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:54.404 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T20:07:54.405 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.405 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:07:54.405 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.406 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.406 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.406 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.406 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.407 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.407 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.407 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.407 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.408 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.408 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.408 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.408 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.409 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:07:54.409 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T20:07:54.409 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.410 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T20:07:54.410 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.410 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.410 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.410 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.411 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.411 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.411 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.411 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.411 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.412 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.412 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.412 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.412 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.412 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.413 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.413 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:54.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.414 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:54.414 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:54.414 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:54.414 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:54.415 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.415 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.415 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.415 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:54.415 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.416 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:54.416 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:54.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.416 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:54.416 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.417 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.417 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "f7d9dd23-1412-40c2-bf2c-973f21da18be", 2023-10-18T20:07:54.417 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "remove_zg", 2023-10-18T20:07:54.418 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "remove_zg", 2023-10-18T20:07:54.418 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:07:54.418 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [], 2023-10-18T20:07:54.418 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:54.418 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:54.419 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "f040972e-89ae-4174-beab-79dc74c3fc06", 2023-10-18T20:07:54.419 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:54.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.419 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "f040972e-89ae-4174-beab-79dc74c3fc06", 2023-10-18T20:07:54.419 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "remove", 2023-10-18T20:07:54.420 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [], 2023-10-18T20:07:54.420 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.420 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": false, 2023-10-18T20:07:54.420 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.420 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.421 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.421 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.421 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.421 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.421 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.422 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.422 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.422 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.423 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.423 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.423 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:54.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.424 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:54.424 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:54.424 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:54.425 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:54.425 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.426 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.426 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:54.426 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.427 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:54.427 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:54.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.428 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:54.428 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.428 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.429 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.429 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.429 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T20:07:54.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.430 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.430 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T20:07:54.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.431 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:07:54.432 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T20:07:54.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.433 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:54.433 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T20:07:54.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.434 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:07:54.434 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T20:07:54.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.435 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "f040972e-89ae-4174-beab-79dc74c3fc06", 2023-10-18T20:07:54.435 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3177956342 2023-10-18T20:07:54.436 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.436 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.437 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T20:07:54.437 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.437 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T20:07:54.439 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T20:07:54.439 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:07:54.439 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:07:54.439 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:07:54.440 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:07:54.440 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:07:54.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.440 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T20:07:54.440 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:07:54.441 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:07:54.441 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:07:54.441 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:07:54.441 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:07:54.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.442 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T20:07:54.442 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:54.442 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:54.442 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:54.443 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:54.443 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:54.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.443 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T20:07:54.444 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:54.444 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:54.444 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:54.444 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:54.445 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:54.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.445 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T20:07:54.445 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:54.445 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:54.446 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:54.446 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:54.446 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:54.446 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.447 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.447 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T20:07:54.447 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:07:54.448 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:07:54.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.449 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:54.501 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:54.501 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:54.528 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:07:54.527+0000 7fb15bbd3780 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-10-18T20:07:54.529 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:07:54.528+0000 7fb15bbd3780 0 Removed zone from zonegroup remove_zg 2023-10-18T20:07:54.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.540 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:54.588 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:54.588 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:54.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:07:54.608+0000 7f39e525e780 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-10-18T20:07:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-18T20:07:54.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.618 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:54.663 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:54.664 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:54.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:07:54.684 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:07:54.732 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:07:54.732 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:07:54.779 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:07:54.780 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:07:54.780 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12, 2023-10-18T20:07:54.780 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T20:07:54.780 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T20:07:54.781 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T20:07:54.781 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:07:54.781 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T20:07:54.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.782 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T20:07:54.782 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T20:07:54.782 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:07:54.782 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T20:07:54.782 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.783 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:54.783 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.783 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:54.783 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:54.783 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.784 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:54.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.784 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.784 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T20:07:54.785 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.785 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T20:07:54.785 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.785 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.785 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.786 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.786 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.786 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.786 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.786 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.787 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.787 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.787 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.787 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.788 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.788 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:07:54.788 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T20:07:54.789 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.789 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T20:07:54.789 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.789 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.789 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.790 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.790 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.790 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.790 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.791 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.791 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.791 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.791 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.791 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.792 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.792 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.792 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:54.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.793 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:54.793 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:54.793 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:54.793 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:54.793 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.794 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.794 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.794 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:54.794 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.794 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:54.795 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:54.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.795 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:54.795 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.795 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.796 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T20:07:54.796 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T20:07:54.796 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T20:07:54.797 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T20:07:54.797 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.797 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:07:54.797 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.798 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T20:07:54.798 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T20:07:54.798 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:54.798 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T20:07:54.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.799 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:54.799 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T20:07:54.799 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.799 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T20:07:54.799 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.800 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.800 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.800 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.800 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.800 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:07:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T20:07:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T20:07:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T20:07:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T20:07:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T20:07:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T20:07:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T20:07:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T20:07:54.806 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T20:07:54.806 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T20:07:54.806 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T20:07:54.806 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T20:07:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T20:07:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T20:07:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T20:07:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T20:07:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T20:07:54.809 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T20:07:54.809 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.809 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.809 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.810 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T20:07:54.810 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.810 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T20:07:54.810 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T20:07:54.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T20:07:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T20:07:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T20:07:54.812 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T20:07:54.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.812 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.812 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T20:07:54.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.813 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:07:54.813 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T20:07:54.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.814 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T20:07:54.814 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T20:07:54.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:07:54.815 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T20:07:54.815 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T20:07:54.815 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.815 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:07:54.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.816 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T20:07:54.816 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:07:54.816 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T20:07:54.816 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T20:07:54.817 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:07:54.817 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:07:54.817 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:07:54.817 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:07:54.817 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:07:54.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.818 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T20:07:54.818 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T20:07:54.818 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T20:07:54.819 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T20:07:54.819 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T20:07:54.819 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T20:07:54.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.819 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T20:07:54.820 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:54.820 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:54.820 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:54.820 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:54.820 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:54.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.821 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T20:07:54.821 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:54.821 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:54.821 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:54.822 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:54.822 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:54.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.823 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T20:07:54.823 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T20:07:54.823 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T20:07:54.823 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T20:07:54.823 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T20:07:54.824 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T20:07:54.824 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:07:54.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:07:54.824 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T20:07:54.825 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T20:07:54.825 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:07:54.825 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:07:54.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-10-18T20:07:54.826 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-36 2023-10-18T20:07:54.827 DEBUG:boto:path=/bofrnf-36/ 2023-10-18T20:07:54.827 DEBUG:boto:auth_path=/bofrnf-36/ 2023-10-18T20:07:54.828 DEBUG:boto:Method: PUT 2023-10-18T20:07:54.828 DEBUG:boto:Path: /bofrnf-36/ 2023-10-18T20:07:54.829 DEBUG:boto:Data: 2023-10-18T20:07:54.829 DEBUG:boto:Headers: {} 2023-10-18T20:07:54.829 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:07:54.830 DEBUG:boto:Port: 8000 2023-10-18T20:07:54.830 DEBUG:boto:Params: {} 2023-10-18T20:07:54.831 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:07:54.831 DEBUG:boto:Token: None 2023-10-18T20:07:54.832 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:07:54 GMT /bofrnf-36/ 2023-10-18T20:07:54.833 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw= 2023-10-18T20:07:54.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:07:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw=', 'Content-Length': '0'} 2023-10-18T20:09:04.836 DEBUG:boto:encountered timeout exception, reconnecting 2023-10-18T20:09:04.889 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:09:05.044 DEBUG:boto:Token: None 2023-10-18T20:09:05.045 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:07:54 GMT /bofrnf-36/ 2023-10-18T20:09:05.045 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw= 2023-10-18T20:09:05.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:07:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw=', 'Content-Length': '0'} 2023-10-18T20:10:15.049 DEBUG:boto:encountered timeout exception, reconnecting 2023-10-18T20:10:15.050 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:10:15.304 DEBUG:boto:Token: None 2023-10-18T20:10:15.305 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:07:54 GMT /bofrnf-36/ 2023-10-18T20:10:15.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw= 2023-10-18T20:10:15.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:07:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw=', 'Content-Length': '0'} 2023-10-18T20:11:25.310 DEBUG:boto:encountered timeout exception, reconnecting 2023-10-18T20:11:25.312 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:11:26.334 DEBUG:boto:Token: None 2023-10-18T20:11:26.335 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:07:54 GMT /bofrnf-36/ 2023-10-18T20:11:26.336 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw= 2023-10-18T20:11:26.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:07:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw=', 'Content-Length': '0'} 2023-10-18T20:12:36.340 DEBUG:boto:encountered timeout exception, reconnecting 2023-10-18T20:12:36.341 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:12:40.469 DEBUG:boto:Token: None 2023-10-18T20:12:40.470 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:07:54 GMT /bofrnf-36/ 2023-10-18T20:12:40.471 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw= 2023-10-18T20:12:40.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:07:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p5W4M4XTqUvAsxU+ZIDCI3ihnSw=', 'Content-Length': '0'} 2023-10-18T20:12:55.518 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ba876491b39c94b9-0065303c47-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:55.519 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-37 2023-10-18T20:12:55.519 DEBUG:boto:path=/bofrnf-37/ 2023-10-18T20:12:55.520 DEBUG:boto:auth_path=/bofrnf-37/ 2023-10-18T20:12:55.520 DEBUG:boto:Method: PUT 2023-10-18T20:12:55.521 DEBUG:boto:Path: /bofrnf-37/ 2023-10-18T20:12:55.521 DEBUG:boto:Data: 2023-10-18T20:12:55.522 DEBUG:boto:Headers: {} 2023-10-18T20:12:55.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:12:55.522 DEBUG:boto:Port: 8001 2023-10-18T20:12:55.523 DEBUG:boto:Params: {} 2023-10-18T20:12:55.523 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:12:55.524 DEBUG:boto:Token: None 2023-10-18T20:12:55.524 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:55 GMT /bofrnf-37/ 2023-10-18T20:12:55.525 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QTFGBDHqpf49G7Nzn0k3vEHSLSM= 2023-10-18T20:12:55.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QTFGBDHqpf49G7Nzn0k3vEHSLSM=', 'Content-Length': '0'} 2023-10-18T20:12:55.759 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a18cb1bc31a1473d-0065303c47-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:55.760 INFO:rgw_multi.tests:create bucket zone=b1 name=bofrnf-38 2023-10-18T20:12:55.760 DEBUG:boto:path=/bofrnf-38/ 2023-10-18T20:12:55.761 DEBUG:boto:auth_path=/bofrnf-38/ 2023-10-18T20:12:55.761 DEBUG:boto:Method: PUT 2023-10-18T20:12:55.762 DEBUG:boto:Path: /bofrnf-38/ 2023-10-18T20:12:55.762 DEBUG:boto:Data: 2023-10-18T20:12:55.763 DEBUG:boto:Headers: {} 2023-10-18T20:12:55.763 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:12:55.763 DEBUG:boto:Port: 8000 2023-10-18T20:12:55.764 DEBUG:boto:Params: {} 2023-10-18T20:12:55.764 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:12:55.765 DEBUG:boto:Token: None 2023-10-18T20:12:55.765 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:55 GMT /bofrnf-38/ 2023-10-18T20:12:55.766 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHFyvWifKRpl34FQOoZDlWnS5xc= 2023-10-18T20:12:55.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHFyvWifKRpl34FQOoZDlWnS5xc=', 'Content-Length': '0'} 2023-10-18T20:12:55.851 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000024cb9192ce56b4e2-0065303c47-115c-b1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:55.851 INFO:rgw_multi.tests:create bucket zone=b2 name=bofrnf-39 2023-10-18T20:12:55.852 DEBUG:boto:path=/bofrnf-39/ 2023-10-18T20:12:55.852 DEBUG:boto:auth_path=/bofrnf-39/ 2023-10-18T20:12:55.853 DEBUG:boto:Method: PUT 2023-10-18T20:12:55.853 DEBUG:boto:Path: /bofrnf-39/ 2023-10-18T20:12:55.853 DEBUG:boto:Data: 2023-10-18T20:12:55.854 DEBUG:boto:Headers: {} 2023-10-18T20:12:55.854 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:12:55.855 DEBUG:boto:Port: 8001 2023-10-18T20:12:55.855 DEBUG:boto:Params: {} 2023-10-18T20:12:55.856 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:12:55.856 DEBUG:boto:Token: None 2023-10-18T20:12:55.857 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:55 GMT /bofrnf-39/ 2023-10-18T20:12:55.857 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KBtwI40z3vIKkDI7WASIetTKO2E= 2023-10-18T20:12:55.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KBtwI40z3vIKkDI7WASIetTKO2E=', 'Content-Length': '0'} 2023-10-18T20:12:55.891 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000034e615a1079770e-0065303c47-1158-b2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:55.891 DEBUG:boto:path=/bofrnf-36/ 2023-10-18T20:12:55.892 DEBUG:boto:auth_path=/bofrnf-36/ 2023-10-18T20:12:55.892 DEBUG:boto:path=/bofrnf-36/?website 2023-10-18T20:12:55.893 DEBUG:boto:auth_path=/bofrnf-36/?website 2023-10-18T20:12:55.893 DEBUG:boto:Method: PUT 2023-10-18T20:12:55.894 DEBUG:boto:Path: /bofrnf-36/?website 2023-10-18T20:12:55.894 DEBUG:boto:Data: index.htmlerror.html 2023-10-18T20:12:55.894 DEBUG:boto:Headers: {} 2023-10-18T20:12:55.895 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:12:55.895 DEBUG:boto:Port: 8000 2023-10-18T20:12:55.896 DEBUG:boto:Params: {} 2023-10-18T20:12:55.896 DEBUG:boto:Token: None 2023-10-18T20:12:55.897 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:55 GMT /bofrnf-36/?website 2023-10-18T20:12:55.897 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4TkcS1mSbj6qsjihO+VFOvENtGM= 2023-10-18T20:12:55.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4TkcS1mSbj6qsjihO+VFOvENtGM=', 'Content-Length': '241'} 2023-10-18T20:12:55.899 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx0000031506fd9ea7d98e5-0065303c47-1664-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:55.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-10-18T20:12:55.901 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-40 2023-10-18T20:12:55.901 DEBUG:boto:path=/bofrnf-40/ 2023-10-18T20:12:55.902 DEBUG:boto:auth_path=/bofrnf-40/ 2023-10-18T20:12:55.902 DEBUG:boto:Method: PUT 2023-10-18T20:12:55.902 DEBUG:boto:Path: /bofrnf-40/ 2023-10-18T20:12:55.903 DEBUG:boto:Data: 2023-10-18T20:12:55.903 DEBUG:boto:Headers: {} 2023-10-18T20:12:55.904 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:12:55.904 DEBUG:boto:Port: 8000 2023-10-18T20:12:55.905 DEBUG:boto:Params: {} 2023-10-18T20:12:55.905 DEBUG:boto:Token: None 2023-10-18T20:12:55.906 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:55 GMT /bofrnf-40/ 2023-10-18T20:12:55.906 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wVVTOOdh0HTk1aB6bFMj9z0Nx70= 2023-10-18T20:12:55.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wVVTOOdh0HTk1aB6bFMj9z0Nx70=', 'Content-Length': '0'} 2023-10-18T20:12:55.921 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005bdf582fd8b172b5-0065303c47-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:55.922 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-41 2023-10-18T20:12:55.922 DEBUG:boto:path=/bofrnf-41/ 2023-10-18T20:12:55.923 DEBUG:boto:auth_path=/bofrnf-41/ 2023-10-18T20:12:55.923 DEBUG:boto:Method: PUT 2023-10-18T20:12:55.924 DEBUG:boto:Path: /bofrnf-41/ 2023-10-18T20:12:55.924 DEBUG:boto:Data: 2023-10-18T20:12:55.924 DEBUG:boto:Headers: {} 2023-10-18T20:12:55.925 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:12:55.925 DEBUG:boto:Port: 8001 2023-10-18T20:12:55.926 DEBUG:boto:Params: {} 2023-10-18T20:12:55.926 DEBUG:boto:Token: None 2023-10-18T20:12:55.927 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:55 GMT /bofrnf-41/ 2023-10-18T20:12:55.927 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:glyKgsqAtOmtXhBTIlvdMGt57UA= 2023-10-18T20:12:55.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:glyKgsqAtOmtXhBTIlvdMGt57UA=', 'Content-Length': '0'} 2023-10-18T20:12:55.953 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000665aad7cfcfece5d-0065303c47-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:55.953 INFO:rgw_multi.tests:create bucket zone=b1 name=bofrnf-42 2023-10-18T20:12:55.954 DEBUG:boto:path=/bofrnf-42/ 2023-10-18T20:12:55.954 DEBUG:boto:auth_path=/bofrnf-42/ 2023-10-18T20:12:55.955 DEBUG:boto:Method: PUT 2023-10-18T20:12:55.955 DEBUG:boto:Path: /bofrnf-42/ 2023-10-18T20:12:55.956 DEBUG:boto:Data: 2023-10-18T20:12:55.956 DEBUG:boto:Headers: {} 2023-10-18T20:12:55.956 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:12:55.957 DEBUG:boto:Port: 8000 2023-10-18T20:12:55.958 DEBUG:boto:Params: {} 2023-10-18T20:12:55.958 DEBUG:boto:Token: None 2023-10-18T20:12:55.959 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:55 GMT /bofrnf-42/ 2023-10-18T20:12:55.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A5EodXwpsygLEAHj2SACItqI/+I= 2023-10-18T20:12:55.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A5EodXwpsygLEAHj2SACItqI/+I=', 'Content-Length': '0'} 2023-10-18T20:12:55.987 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005dee5dcca0e86b34-0065303c47-115c-b1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:55.988 INFO:rgw_multi.tests:create bucket zone=b2 name=bofrnf-43 2023-10-18T20:12:55.988 DEBUG:boto:path=/bofrnf-43/ 2023-10-18T20:12:55.989 DEBUG:boto:auth_path=/bofrnf-43/ 2023-10-18T20:12:55.989 DEBUG:boto:Method: PUT 2023-10-18T20:12:55.989 DEBUG:boto:Path: /bofrnf-43/ 2023-10-18T20:12:55.990 DEBUG:boto:Data: 2023-10-18T20:12:55.990 DEBUG:boto:Headers: {} 2023-10-18T20:12:55.991 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:12:55.991 DEBUG:boto:Port: 8001 2023-10-18T20:12:55.991 DEBUG:boto:Params: {} 2023-10-18T20:12:55.992 DEBUG:boto:Token: None 2023-10-18T20:12:55.992 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:55 GMT /bofrnf-43/ 2023-10-18T20:12:55.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IyjtjDpIln4HHd1CB22nCdJCoUc= 2023-10-18T20:12:55.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IyjtjDpIln4HHd1CB22nCdJCoUc=', 'Content-Length': '0'} 2023-10-18T20:12:56.020 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000df6647f7ea682b53-0065303c47-1158-b2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:56.021 DEBUG:boto:path=/bofrnf-40/ 2023-10-18T20:12:56.021 DEBUG:boto:auth_path=/bofrnf-40/ 2023-10-18T20:12:56.021 DEBUG:boto:path=/bofrnf-40/?policy 2023-10-18T20:12:56.022 DEBUG:boto:auth_path=/bofrnf-40/?policy 2023-10-18T20:12:56.022 DEBUG:boto:Method: PUT 2023-10-18T20:12:56.023 DEBUG:boto:Path: /bofrnf-40/?policy 2023-10-18T20:12:56.023 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-18T20:12:56.023 DEBUG:boto:Headers: {} 2023-10-18T20:12:56.024 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:12:56.024 DEBUG:boto:Port: 8000 2023-10-18T20:12:56.025 DEBUG:boto:Params: {} 2023-10-18T20:12:56.025 DEBUG:boto:Token: None 2023-10-18T20:12:56.025 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:56 GMT /bofrnf-40/?policy 2023-10-18T20:12:56.026 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aEsPyXZfo3XPaDiBQlG0VT1k7nc= 2023-10-18T20:12:56.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aEsPyXZfo3XPaDiBQlG0VT1k7nc=', 'Content-Length': '97'} 2023-10-18T20:12:56.032 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000694a3b0c6bae2b5b-0065303c48-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:12:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:56.032 DEBUG:boto:path=/bofrnf-40/ 2023-10-18T20:12:56.033 DEBUG:boto:auth_path=/bofrnf-40/ 2023-10-18T20:12:56.033 DEBUG:boto:path=/bofrnf-40/?policy 2023-10-18T20:12:56.033 DEBUG:boto:auth_path=/bofrnf-40/?policy 2023-10-18T20:12:56.034 DEBUG:boto:Method: GET 2023-10-18T20:12:56.034 DEBUG:boto:Path: /bofrnf-40/?policy 2023-10-18T20:12:56.035 DEBUG:boto:Data: 2023-10-18T20:12:56.035 DEBUG:boto:Headers: {} 2023-10-18T20:12:56.035 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:12:56.036 DEBUG:boto:Port: 8000 2023-10-18T20:12:56.036 DEBUG:boto:Params: {} 2023-10-18T20:12:56.037 DEBUG:boto:Token: None 2023-10-18T20:12:56.037 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:12:56 GMT /bofrnf-40/?policy 2023-10-18T20:12:56.038 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2K8B501CBC3absxEQzW+wfXFao4= 2023-10-18T20:12:56.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2K8B501CBC3absxEQzW+wfXFao4=', 'Content-Length': '0'} 2023-10-18T20:12:56.040 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001c56c2f716d960c3-0065303c48-1664-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 18 Oct 2023 20:12:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:56.040 DEBUG:boto:path=/bofrnf-41/ 2023-10-18T20:12:56.041 DEBUG:boto:auth_path=/bofrnf-41/ 2023-10-18T20:12:56.041 DEBUG:boto:path=/bofrnf-41/?policy 2023-10-18T20:12:56.041 DEBUG:boto:auth_path=/bofrnf-41/?policy 2023-10-18T20:12:56.042 DEBUG:boto:Method: PUT 2023-10-18T20:12:56.042 DEBUG:boto:Path: /bofrnf-41/?policy 2023-10-18T20:12:56.043 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-18T20:12:56.043 DEBUG:boto:Headers: {} 2023-10-18T20:12:56.044 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:12:56.044 DEBUG:boto:Port: 8001 2023-10-18T20:12:56.044 DEBUG:boto:Params: {} 2023-10-18T20:12:56.045 DEBUG:boto:Token: None 2023-10-18T20:12:56.045 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:56 GMT /bofrnf-41/?policy 2023-10-18T20:12:56.046 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H+qJvrHFyeU61+nzOEMH3Groxps= 2023-10-18T20:12:56.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H+qJvrHFyeU61+nzOEMH3Groxps=', 'Content-Length': '97'} 2023-10-18T20:12:57.055 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000081cc546094401613-0065303c48-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:12:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:57.056 DEBUG:boto:path=/bofrnf-41/ 2023-10-18T20:12:57.056 DEBUG:boto:auth_path=/bofrnf-41/ 2023-10-18T20:12:57.057 DEBUG:boto:path=/bofrnf-41/?policy 2023-10-18T20:12:57.057 DEBUG:boto:auth_path=/bofrnf-41/?policy 2023-10-18T20:12:57.057 DEBUG:boto:Method: GET 2023-10-18T20:12:57.058 DEBUG:boto:Path: /bofrnf-41/?policy 2023-10-18T20:12:57.058 DEBUG:boto:Data: 2023-10-18T20:12:57.059 DEBUG:boto:Headers: {} 2023-10-18T20:12:57.059 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:12:57.059 DEBUG:boto:Port: 8001 2023-10-18T20:12:57.060 DEBUG:boto:Params: {} 2023-10-18T20:12:57.060 DEBUG:boto:Token: None 2023-10-18T20:12:57.061 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:12:57 GMT /bofrnf-41/?policy 2023-10-18T20:12:57.061 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WKc2V+jaduDu7+eqEuAGp4768PI= 2023-10-18T20:12:57.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WKc2V+jaduDu7+eqEuAGp4768PI=', 'Content-Length': '0'} 2023-10-18T20:12:57.063 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000411abbcb7fd5fcf-0065303c49-1668-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 18 Oct 2023 20:12:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:57.064 DEBUG:boto:path=/bofrnf-42/ 2023-10-18T20:12:57.064 DEBUG:boto:auth_path=/bofrnf-42/ 2023-10-18T20:12:57.064 DEBUG:boto:path=/bofrnf-42/?policy 2023-10-18T20:12:57.065 DEBUG:boto:auth_path=/bofrnf-42/?policy 2023-10-18T20:12:57.065 DEBUG:boto:Method: PUT 2023-10-18T20:12:57.065 DEBUG:boto:Path: /bofrnf-42/?policy 2023-10-18T20:12:57.066 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-18T20:12:57.066 DEBUG:boto:Headers: {} 2023-10-18T20:12:57.067 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:12:57.067 DEBUG:boto:Port: 8000 2023-10-18T20:12:57.067 DEBUG:boto:Params: {} 2023-10-18T20:12:57.068 DEBUG:boto:Token: None 2023-10-18T20:12:57.068 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:57 GMT /bofrnf-42/?policy 2023-10-18T20:12:57.069 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ud48EbY5PS7Y4a9xZwD0zDHa3SI= 2023-10-18T20:12:57.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ud48EbY5PS7Y4a9xZwD0zDHa3SI=', 'Content-Length': '97'} 2023-10-18T20:12:58.079 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000802378ab3102b168-0065303c49-115c-b1'), ('Date', 'Wed, 18 Oct 2023 20:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:58.080 DEBUG:boto:path=/bofrnf-42/ 2023-10-18T20:12:58.080 DEBUG:boto:auth_path=/bofrnf-42/ 2023-10-18T20:12:58.081 DEBUG:boto:path=/bofrnf-42/?policy 2023-10-18T20:12:58.081 DEBUG:boto:auth_path=/bofrnf-42/?policy 2023-10-18T20:12:58.081 DEBUG:boto:Method: GET 2023-10-18T20:12:58.082 DEBUG:boto:Path: /bofrnf-42/?policy 2023-10-18T20:12:58.082 DEBUG:boto:Data: 2023-10-18T20:12:58.083 DEBUG:boto:Headers: {} 2023-10-18T20:12:58.083 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8000 2023-10-18T20:12:58.083 DEBUG:boto:Port: 8000 2023-10-18T20:12:58.084 DEBUG:boto:Params: {} 2023-10-18T20:12:58.084 DEBUG:boto:Token: None 2023-10-18T20:12:58.085 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:12:58 GMT /bofrnf-42/?policy 2023-10-18T20:12:58.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GjD8i13uGFPk3lYouwa6nZ1ec5I= 2023-10-18T20:12:58.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GjD8i13uGFPk3lYouwa6nZ1ec5I=', 'Content-Length': '0'} 2023-10-18T20:12:58.087 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f7ce9bf15b8978a1-0065303c4a-115c-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 18 Oct 2023 20:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:58.088 DEBUG:boto:path=/bofrnf-43/ 2023-10-18T20:12:58.088 DEBUG:boto:auth_path=/bofrnf-43/ 2023-10-18T20:12:58.088 DEBUG:boto:path=/bofrnf-43/?policy 2023-10-18T20:12:58.089 DEBUG:boto:auth_path=/bofrnf-43/?policy 2023-10-18T20:12:58.089 DEBUG:boto:Method: PUT 2023-10-18T20:12:58.089 DEBUG:boto:Path: /bofrnf-43/?policy 2023-10-18T20:12:58.090 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-18T20:12:58.090 DEBUG:boto:Headers: {} 2023-10-18T20:12:58.091 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:12:58.091 DEBUG:boto:Port: 8001 2023-10-18T20:12:58.091 DEBUG:boto:Params: {} 2023-10-18T20:12:58.092 DEBUG:boto:Token: None 2023-10-18T20:12:58.092 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:58 GMT /bofrnf-43/?policy 2023-10-18T20:12:58.093 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/zxOybPEOwowhM3nlF2vCYPpITw= 2023-10-18T20:12:58.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/zxOybPEOwowhM3nlF2vCYPpITw=', 'Content-Length': '97'} 2023-10-18T20:12:59.102 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000003c532bbfff723dc-0065303c4a-1158-b2'), ('Date', 'Wed, 18 Oct 2023 20:12:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:59.103 DEBUG:boto:path=/bofrnf-43/ 2023-10-18T20:12:59.103 DEBUG:boto:auth_path=/bofrnf-43/ 2023-10-18T20:12:59.104 DEBUG:boto:path=/bofrnf-43/?policy 2023-10-18T20:12:59.104 DEBUG:boto:auth_path=/bofrnf-43/?policy 2023-10-18T20:12:59.104 DEBUG:boto:Method: GET 2023-10-18T20:12:59.105 DEBUG:boto:Path: /bofrnf-43/?policy 2023-10-18T20:12:59.105 DEBUG:boto:Data: 2023-10-18T20:12:59.106 DEBUG:boto:Headers: {} 2023-10-18T20:12:59.106 DEBUG:boto:Host: smithi156.front.sepia.ceph.com:8001 2023-10-18T20:12:59.106 DEBUG:boto:Port: 8001 2023-10-18T20:12:59.107 DEBUG:boto:Params: {} 2023-10-18T20:12:59.107 DEBUG:boto:Token: None 2023-10-18T20:12:59.108 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:12:59 GMT /bofrnf-43/?policy 2023-10-18T20:12:59.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jWZceiBLghVAIPizjrcd3TBXhBg= 2023-10-18T20:12:59.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jWZceiBLghVAIPizjrcd3TBXhBg=', 'Content-Length': '0'} 2023-10-18T20:12:59.110 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000441aca1565287585-0065303c4b-1158-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 18 Oct 2023 20:12:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:59.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-10-18T20:12:59.112 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-44 2023-10-18T20:12:59.112 DEBUG:boto:path=/bofrnf-44/ 2023-10-18T20:12:59.112 DEBUG:boto:auth_path=/bofrnf-44/ 2023-10-18T20:12:59.113 DEBUG:boto:Method: PUT 2023-10-18T20:12:59.113 DEBUG:boto:Path: /bofrnf-44/ 2023-10-18T20:12:59.114 DEBUG:boto:Data: 2023-10-18T20:12:59.114 DEBUG:boto:Headers: {} 2023-10-18T20:12:59.115 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:12:59.115 DEBUG:boto:Port: 8000 2023-10-18T20:12:59.116 DEBUG:boto:Params: {} 2023-10-18T20:12:59.116 DEBUG:boto:Token: None 2023-10-18T20:12:59.116 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:59 GMT /bofrnf-44/ 2023-10-18T20:12:59.117 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iXrdt5r6MKtR311sfLn1yvpCCmc= 2023-10-18T20:12:59.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iXrdt5r6MKtR311sfLn1yvpCCmc=', 'Content-Length': '0'} 2023-10-18T20:12:59.137 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000007ebf2918754dc55-0065303c4b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:59.138 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-45 2023-10-18T20:12:59.141 DEBUG:boto:path=/bofrnf-45/ 2023-10-18T20:12:59.142 DEBUG:boto:auth_path=/bofrnf-45/ 2023-10-18T20:12:59.142 DEBUG:boto:Method: PUT 2023-10-18T20:12:59.143 DEBUG:boto:Path: /bofrnf-45/ 2023-10-18T20:12:59.143 DEBUG:boto:Data: 2023-10-18T20:12:59.143 DEBUG:boto:Headers: {} 2023-10-18T20:12:59.144 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:12:59.144 DEBUG:boto:Port: 8001 2023-10-18T20:12:59.145 DEBUG:boto:Params: {} 2023-10-18T20:12:59.145 DEBUG:boto:Token: None 2023-10-18T20:12:59.145 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:12:59 GMT /bofrnf-45/ 2023-10-18T20:12:59.146 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:N8ffrHVK2mYDV6kl+kjGk0nyHRk= 2023-10-18T20:12:59.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:12:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:N8ffrHVK2mYDV6kl+kjGk0nyHRk=', 'Content-Length': '0'} 2023-10-18T20:12:59.184 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085dc4523690f7b9c-0065303c4b-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:12:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:12:59.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:12:59.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:12:59.186 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:12:59.235 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:12:59.236 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:12:59.264 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:12:59.277 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:12:59.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:12:59.277 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659979.175406_651.1", 2023-10-18T20:12:59.277 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:12:59.175406Z" 2023-10-18T20:12:59.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:12:59.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:12:59.278 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:12:59.278 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:12:59.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:12:59.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:12:59.279 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:12:59.279 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:12:59.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:12:59.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:12:59.280 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:12:59.280 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:12:59.280 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:12:59.281 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:12:59.282 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:12:59.282 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:12:59.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:12:59.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:12:59.284 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:12:59.347 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:12:59.347 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:12:59.400 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:12:59.399+0000 7fa080768780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:12:59.401 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:12:59.400+0000 7fa080768780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:12:59.414 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:12:59.414 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:12:59.414 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:12:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:12:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:12:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:12:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:12:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:12:59.416 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:12:59.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:12:59.416 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:12:59.416 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:12:59.417 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:12:59.417 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.497973_578.1", 2023-10-18T20:12:59.417 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:12:59.417 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:12:59.418 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:12:59.418 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.497973Z", 2023-10-18T20:12:59.418 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:12:59.418 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:12:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:12:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:12:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:12:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:12:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:12:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:12:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:12:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:12:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:12:59.421 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:12:59.421 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:12:59.421 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:12:59.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:12:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:12:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:12:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:12:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:12:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:12:59.423 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:12:59.423 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:12:59.423 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:12:59.423 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:12:59.423 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:12:59.424 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:12:59.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:12:59.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:12:59.424 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:12:59.424 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:12:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:12:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:12:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:12:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:12:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:12:59.426 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:12:59.426 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:12:59.426 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:12:59.426 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:12:59.427 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:12:59.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:12:59.427 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:12:59.427 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:12:59.427 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:12:59.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:12:59.428 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:12:59Z" 2023-10-18T20:12:59.428 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:12:59.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.497973_578.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.497973Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:12:59Z" } 2023-10-18T20:12:59.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.497973_578.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.497973Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:12:59.430 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:12:59.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.497973_578.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:12:59.431 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.497973_578.1 2023-10-18T20:13:04.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:04.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:04.434 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:04.486 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:04.486 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:04.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:04.512+0000 7f44178ef780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:04.514 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:04.513+0000 7f44178ef780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:04.525 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.499891_579.1", 2023-10-18T20:13:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.499891Z", 2023-10-18T20:13:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:04.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:04.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:04.530 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:04.530 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:04.531 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:04.531 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:04.531 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:04.531 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:04.531 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:04Z" 2023-10-18T20:13:04.539 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:04.540 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.499891_579.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.499891Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:04Z" } 2023-10-18T20:13:04.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.499891_579.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.499891Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:04.541 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:04.541 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.499891_579.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:04.542 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.499891_579.1 2023-10-18T20:13:09.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:09.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:09.545 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:09.616 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:09.617 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:09.668 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:09.665+0000 7fbc88816780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:09.669 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:09.666+0000 7fbc88816780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:09.681 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.503236_582.1", 2023-10-18T20:13:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.503236Z", 2023-10-18T20:13:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:09Z" 2023-10-18T20:13:09.695 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:09.696 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.503236_582.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.503236Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:09Z" } 2023-10-18T20:13:09.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.503236_582.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.503236Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:09.697 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:09.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.503236_582.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:09.698 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.503236_582.1 2023-10-18T20:13:14.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:14.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:14.700 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:14.776 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:14.777 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:14.816 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:14.814+0000 7ff911ffc780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:14.817 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:14.815+0000 7ff911ffc780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:14.829 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:14.830 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:14.830 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:14.830 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:14.830 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:14.831 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:14.831 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:14.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:14.831 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:14.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:14.832 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:14.832 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:14.832 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:14.832 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.935659_624.1", 2023-10-18T20:13:14.833 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:14.833 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:14.833 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:14.833 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.935659Z", 2023-10-18T20:13:14.834 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:14.834 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:14.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:14.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:14.835 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:14.835 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:14.835 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:14.835 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:14.835 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:14.836 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:14.836 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:14.836 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:14.836 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:14.836 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:14.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:14.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:14.837 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:14.837 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:14.837 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:14.838 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:14.838 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:14.838 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:14.838 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:14.838 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:14.839 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:14.839 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:14.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:14.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:14.839 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:14.840 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:14.840 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:14.840 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:14.840 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:14.840 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:14.841 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:14.841 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:14.841 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:14.841 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:14.841 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:14.842 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:14.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:14.842 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:14.842 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:14.842 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:14.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:14.843 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:14Z" 2023-10-18T20:13:14.843 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:14.844 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.935659_624.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.935659Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:14Z" } 2023-10-18T20:13:14.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.935659_624.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.935659Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:14.845 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:14.845 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.935659_624.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:14.846 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.935659_624.1 2023-10-18T20:13:19.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:19.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:19.848 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:19.905 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:19.905 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:19.935 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:19.933+0000 7f7d19782780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:19.936 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:19.934+0000 7f7d19782780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:19.949 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.520085_594.1", 2023-10-18T20:13:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.520085Z", 2023-10-18T20:13:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:19.957 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:19.957 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:19.957 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:19.957 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:19.957 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:19.958 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:19.958 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:19.958 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:19.958 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:19.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:19.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:19.959 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:19.959 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:19.959 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:19.960 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:19.960 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:19.960 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:19.960 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:19.960 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:19.961 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:19.961 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:19.961 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:19.961 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:19.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:19.962 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:19.962 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:19.962 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:19.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:19.963 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:19Z" 2023-10-18T20:13:19.964 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:19.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.520085_594.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.520085Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:19Z" } 2023-10-18T20:13:19.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.520085_594.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.520085Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:19.967 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:19.968 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.520085_594.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:19.968 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.520085_594.1 2023-10-18T20:13:24.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:24.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:24.971 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:25.027 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:25.027 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:25.058 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:25.056+0000 7fd7f71fe780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:25.058 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:25.057+0000 7fd7f71fe780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:25.071 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:25.072 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:25.072 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:25.072 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:25.072 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:25.072 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:25.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:25.073 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:25.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:25.073 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:25.073 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:25.074 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:25.074 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.501182_580.1", 2023-10-18T20:13:25.074 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:25.074 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:25.074 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:25.075 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.501182Z", 2023-10-18T20:13:25.075 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:25.075 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:25.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:25.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:25.076 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:25.076 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:25.076 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:25.076 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:25.076 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:25.077 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:25.077 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:25.077 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:25.077 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:25.077 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:25.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:25.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:25.078 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:25.078 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:25.078 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:25.079 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:25.079 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:25.079 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:25.079 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:25.079 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:25.079 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:25.080 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:25.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:25.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:25.080 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:25.080 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:25.081 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:25.081 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:25.081 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:25.081 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:25.081 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:25.082 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:25.082 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:25.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:25.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:25.083 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:25.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:25.083 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:25.083 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:25.083 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:25.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:25.084 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:25Z" 2023-10-18T20:13:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:25.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.501182_580.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.501182Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:25Z" } 2023-10-18T20:13:25.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.501182_580.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.501182Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:25.086 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:25.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.501182_580.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:25.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.501182_580.1 2023-10-18T20:13:30.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:30.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:30.088 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:30.143 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:30.143 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:30.172 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:30.171+0000 7f4d89b5b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:30.174 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:30.172+0000 7f4d89b5b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:30.185 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:30.186 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:30.186 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:30.186 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:30.187 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:30.187 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:30.187 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:30.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:30.188 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:30.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:30.188 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:30.188 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:30.188 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:30.189 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.506517_587.1", 2023-10-18T20:13:30.189 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:30.189 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:30.189 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:30.189 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.506517Z", 2023-10-18T20:13:30.190 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:30.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:30.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:30.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:30.191 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:30.191 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:30.191 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:30.191 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:30.191 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:30.192 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:30.192 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:30.192 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:30.192 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:30.192 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:30.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:30.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:30.193 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:30.193 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:30.194 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:30.194 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:30.194 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:30.194 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:30.195 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:30.195 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:30.195 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:30.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:30.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:30.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:30.196 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:30.196 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:30.196 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:30.196 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:30.197 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:30.197 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:30.197 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:30.197 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:30.198 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:30.198 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:30.198 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:30.198 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:30.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:30.199 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:30.199 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:30.199 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:30.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:30.200 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:30Z" 2023-10-18T20:13:30.200 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:30.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.506517_587.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.506517Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:30Z" } 2023-10-18T20:13:30.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.506517_587.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.506517Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:30.202 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:30.202 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.506517_587.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:30.202 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.506517_587.1 2023-10-18T20:13:35.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:35.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:35.205 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:35.259 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:35.259 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:35.290 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:35.289+0000 7f1a00e67780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:35.291 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:35.290+0000 7f1a00e67780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:35.303 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:35.304 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:35.304 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:35.304 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:35.304 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:35.304 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:35.305 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:35.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:35.305 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:35.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:35.306 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:35.306 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:35.306 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:35.306 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.724211_606.1", 2023-10-18T20:13:35.307 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:35.307 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:35.307 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:35.307 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.724211Z", 2023-10-18T20:13:35.307 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:35.308 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:35.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:35.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:35.309 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:35.309 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:35.310 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:35.310 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:35.311 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:35.311 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:35.311 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:35.311 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:35.312 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:35.312 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:35.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:35.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:35.312 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:35.313 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:35.313 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:35.313 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:35.313 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:35.313 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:35.314 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:35.314 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:35.314 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:35.314 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:35.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:35.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:35.315 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:35.315 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:35.315 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:35.315 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:35.316 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:35.316 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:35.316 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:35.316 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:35.316 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:35.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:35.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:35.317 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:35.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:35.318 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:35.318 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:35.318 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:35.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:35.318 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:35Z" 2023-10-18T20:13:35.319 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:35.320 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.724211_606.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.724211Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:35Z" } 2023-10-18T20:13:35.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.724211_606.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.724211Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:35.321 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:35.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.724211_606.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:35.322 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.724211_606.1 2023-10-18T20:13:40.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:40.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:40.324 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:40.374 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:40.375 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:40.402 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:40.400+0000 7fb156dc1780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:40.402 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:40.401+0000 7fb156dc1780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:40.414 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:40.415 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:40.415 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:40.415 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:40.415 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:40.415 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:40.416 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:40.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:40.416 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:40.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:40.416 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:40.416 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:40.417 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:40.417 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.503347_583.1", 2023-10-18T20:13:40.417 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:40.417 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:40.417 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:40.418 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.503347Z", 2023-10-18T20:13:40.418 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:40.418 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:40.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:40.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:40.419 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:40.419 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:40.419 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:40.419 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:40.419 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:40.420 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:40.420 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:40.420 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:40.420 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:40.420 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:40.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:40.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:40.421 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:40.421 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:40.421 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:40.422 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:40.422 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:40.422 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:40.422 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:40.422 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:40.423 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:40.423 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:40.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:40.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:40.424 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:40.424 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:40.424 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:40.424 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:40.424 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:40.425 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:40.425 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:40.425 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:40.425 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:40.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:40.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:40.426 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:40.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:40.426 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:40.426 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:40.426 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:40.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:40.427 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:40Z" 2023-10-18T20:13:40.427 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:40.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.503347_583.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.503347Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:40Z" } 2023-10-18T20:13:40.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.503347_583.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.503347Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:40.429 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:40.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.503347_583.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:40.429 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.503347_583.1 2023-10-18T20:13:45.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:45.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:45.432 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:45.484 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:45.484 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:45.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:45.511+0000 7f8a3a81d780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:45.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:45.512+0000 7f8a3a81d780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:45.525 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:45.525 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:45.526 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:45.526 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:45.526 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:45.526 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:45.526 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:45.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:45.527 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:45.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:45.527 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:45.527 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:45.528 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:45.528 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.510020_588.1", 2023-10-18T20:13:45.528 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:45.528 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:45.528 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:45.529 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.510020Z", 2023-10-18T20:13:45.529 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:45.529 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:45.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:45.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:45.530 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:45.530 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:45.530 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:45.530 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:45.530 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:45.531 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:45.531 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:45.531 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:45.531 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:45.531 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:45.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:45.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:45.532 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:45.532 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:45.532 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:45.533 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:45.533 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:45.533 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:45.533 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:45.533 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:45.534 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:45.534 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:45.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:45.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:45.534 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:45.535 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:45.535 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:45.535 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:45.535 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:45.535 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:45.536 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:45.536 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:45.536 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:45.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:45.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:45.537 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:45.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:45.537 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:45.537 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:45.537 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:45.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:45.538 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:45Z" 2023-10-18T20:13:45.538 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:45.539 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.510020_588.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.510020Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:45Z" } 2023-10-18T20:13:45.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.510020_588.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.510020Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:45.540 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:45.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.510020_588.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:45.540 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.510020_588.1 2023-10-18T20:13:50.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:50.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:50.543 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:50.594 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:50.594 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:50.621 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:50.620+0000 7f26508f6780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:50.622 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:50.621+0000 7f26508f6780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:50.634 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:50.634 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:50.635 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:50.635 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:50.635 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:50.635 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.504567_586.1", 2023-10-18T20:13:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.504567Z", 2023-10-18T20:13:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:50.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:50.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:50.639 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:50.639 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:50.639 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:50.640 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:50.640 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:50.640 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:50.640 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:50.641 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:50.641 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:50.641 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:50.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:50.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:50.642 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:50.642 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:50.642 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:50.642 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:50.643 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:50.643 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:50.643 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:50.643 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:50.643 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:50.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:50.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:50.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:50.644 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:50.644 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:50.645 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:50.645 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:50.645 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:50.645 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:50.646 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:50.646 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:50.646 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:50.646 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:50.646 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:50.647 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:50.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:50.647 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:50.647 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:50.647 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:50.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:50.648 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:50Z" 2023-10-18T20:13:50.648 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:50.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.504567_586.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.504567Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:50Z" } 2023-10-18T20:13:50.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.504567_586.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.504567Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:50.650 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:50.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.504567_586.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:50.651 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.504567_586.1 2023-10-18T20:13:55.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:55.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:13:55.654 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:13:55.712 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:13:55.713 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:13:55.747 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:55.746+0000 7f7e72e17780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:55.748 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:13:55.747+0000 7f7e72e17780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:13:55.761 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:13:55.761 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:13:55.761 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:13:55.761 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:13:55.762 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:13:55.762 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:13:55.762 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:55.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:55.762 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:13:55.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:55.763 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:13:55.763 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:55.763 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:55.763 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.724211_606.1", 2023-10-18T20:13:55.764 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:55.764 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:13:55.764 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:55.764 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.724211Z", 2023-10-18T20:13:55.764 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:55.765 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:55.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:55.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:55.765 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:13:55.765 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:55.766 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:55.766 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:13:55.766 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:55.766 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:55.767 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:55.767 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:13:55.767 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:55.767 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:55.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:55.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:55.768 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:13:55.768 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:55.768 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:55.768 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:55.769 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:55.769 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:55.769 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:55.769 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:55.769 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:55.770 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:55.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:55.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:13:55.770 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:13:55.770 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:13:55.771 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:13:55.771 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:13:55.771 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:13:55.771 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:13:55.771 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:13:55.772 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:13:55.772 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:13:55.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:55.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:13:55.772 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:13:55.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:55.773 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:13:55.773 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:13:55.773 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:13:55.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:13:55.774 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:13:55Z" 2023-10-18T20:13:55.774 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:13:55.775 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.724211_606.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.724211Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:13:55Z" } 2023-10-18T20:13:55.775 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.724211_606.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.724211Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:13:55.776 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:55.776 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.724211_606.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:13:55.777 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.724211_606.1 2023-10-18T20:14:00.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:00.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:00.780 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:00.834 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:00.834 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:00.920 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:00.918+0000 7fe753a26780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:00.921 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:00.919+0000 7fe753a26780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:00.933 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:14:00.934 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:14:00.934 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:14:00.934 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:14:00.934 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:14:00.935 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:14:00.935 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:00.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:00.935 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:14:00.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:00.935 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:14:00.936 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:00.936 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:00.936 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.724211_606.1", 2023-10-18T20:14:00.936 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:00.936 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:14:00.937 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:00.937 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.724211Z", 2023-10-18T20:14:00.937 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:00.937 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:00.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:00.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:00.938 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:14:00.938 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:00.938 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:00.939 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:14:00.939 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:00.939 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:00.939 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:00.939 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:14:00.939 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:00.940 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:00.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:00.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:00.940 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:14:00.940 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:00.941 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:00.941 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:00.941 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:00.941 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:00.941 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:00.942 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:00.942 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:00.942 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:00.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:00.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:00.943 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:14:00.943 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:00.943 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:00.943 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:00.944 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:00.944 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:00.944 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:00.944 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:00.944 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:00.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:00.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:00.945 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:14:00.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:00.945 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:14:00.946 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:14:00.946 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:14:00.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:00.946 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:14:00Z" 2023-10-18T20:14:00.946 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:14:00.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.724211_606.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.724211Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:14:00Z" } 2023-10-18T20:14:00.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.724211_606.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.724211Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:14:00.948 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:00.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.724211_606.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:00.949 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.724211_606.1 2023-10-18T20:14:05.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:05.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:05.951 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:06.009 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:06.009 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:06.040 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:06.038+0000 7f5adfb5b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:06.041 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:06.039+0000 7f5adfb5b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:06.054 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:14:06.054 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:14:06.055 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:14:06.055 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:14:06.055 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:14:06.055 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:14:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:14:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:06.056 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:14:06.057 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:06.057 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:06.057 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659975.497973_578.1", 2023-10-18T20:14:06.057 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:06.057 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:14:06.058 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:06.058 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:55.497973Z", 2023-10-18T20:14:06.058 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:06.058 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:06.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:14:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:06.059 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:14:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:14:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:06.060 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:14:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:06.061 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:06.062 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:06.062 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:06.062 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:06.062 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:06.062 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:06.063 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:14:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:06.064 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:06.065 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:14:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:14:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:14:06.066 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:14:06.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:06.067 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:14:06Z" 2023-10-18T20:14:06.067 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:14:06.068 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659975.497973_578.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:55.497973Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:14:06Z" } 2023-10-18T20:14:06.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659975.497973_578.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:55.497973Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:14:06.069 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:06.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659975.497973_578.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:06.069 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697659979.175406_651.1 target=1_1697659975.497973_578.1 2023-10-18T20:14:11.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.072 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:11.123 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:11.124 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:11.152 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:11.150+0000 7f210cd5a780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:11.153 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:11.151+0000 7f210cd5a780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:11.166 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:14:11.166 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:14:11.166 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:14:11.166 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:14:11.167 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:14:11.167 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:14:11.167 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:11.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.167 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:14:11.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.168 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:14:11.168 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:11.169 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:11.169 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659979.175406_651.1", 2023-10-18T20:14:11.169 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:11.169 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:14:11.169 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:11.170 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:59.175406Z", 2023-10-18T20:14:11.170 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:11.170 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.171 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:14:11.171 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:11.171 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:11.171 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:14:11.171 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:11.172 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:11.172 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:11.172 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:14:11.172 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:11.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.173 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:14:11.173 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:11.173 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:14:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:14:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:14:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:14:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:14:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:14:11Z" 2023-10-18T20:14:11.179 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:14:11.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659979.175406_651.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:59.175406Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:14:11Z" } 2023-10-18T20:14:11.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659979.175406_651.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:59.175406Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:14:11.181 DEBUG:rgw_multi.tests:log_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:11.182 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:11.182 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:14:11.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bofrnf-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:14:11.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bofrnf-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:14:11.183 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bofrnf-44 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:14:11.234 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:11.234 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:11.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bofrnf-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:14:11.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bofrnf-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:14:11.297 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bofrnf-45 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:14:11.350 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:11.350 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:11.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.415 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:11.461 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:11.462 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:11.502 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:14:11.503 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:9", 2023-10-18T20:14:11.503 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000078", 2023-10-18T20:14:11.503 INFO:teuthology.orchestra.run.smithi082.stdout: "meta.history", 2023-10-18T20:14:11.503 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:9", 2023-10-18T20:14:11.503 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:4", 2023-10-18T20:14:11.504 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000070", 2023-10-18T20:14:11.504 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000104", 2023-10-18T20:14:11.504 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:2", 2023-10-18T20:14:11.504 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000026", 2023-10-18T20:14:11.504 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:9", 2023-10-18T20:14:11.505 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:2", 2023-10-18T20:14:11.505 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.8", 2023-10-18T20:14:11.505 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:3", 2023-10-18T20:14:11.505 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000028", 2023-10-18T20:14:11.505 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:9", 2023-10-18T20:14:11.506 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:3", 2023-10-18T20:14:11.506 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:9", 2023-10-18T20:14:11.506 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:2", 2023-10-18T20:14:11.506 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:7", 2023-10-18T20:14:11.506 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:7", 2023-10-18T20:14:11.507 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000040", 2023-10-18T20:14:11.507 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.6", 2023-10-18T20:14:11.507 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:6", 2023-10-18T20:14:11.507 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000015", 2023-10-18T20:14:11.507 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000069", 2023-10-18T20:14:11.508 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000095", 2023-10-18T20:14:11.508 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:8", 2023-10-18T20:14:11.508 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000003", 2023-10-18T20:14:11.508 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000047", 2023-10-18T20:14:11.508 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:4", 2023-10-18T20:14:11.509 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.419020ed-d3b2-4ce0-b36c-97abb284c459.3", 2023-10-18T20:14:11.509 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000035", 2023-10-18T20:14:11.509 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:8", 2023-10-18T20:14:11.509 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000037", 2023-10-18T20:14:11.509 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T20:14:11.510 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:10", 2023-10-18T20:14:11.510 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:0", 2023-10-18T20:14:11.510 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:7", 2023-10-18T20:14:11.510 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:3", 2023-10-18T20:14:11.510 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:6", 2023-10-18T20:14:11.511 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000024", 2023-10-18T20:14:11.511 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:7", 2023-10-18T20:14:11.511 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000000", 2023-10-18T20:14:11.511 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000031", 2023-10-18T20:14:11.511 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:2", 2023-10-18T20:14:11.512 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:2", 2023-10-18T20:14:11.512 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:7", 2023-10-18T20:14:11.512 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.3", 2023-10-18T20:14:11.512 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:7", 2023-10-18T20:14:11.512 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.419020ed-d3b2-4ce0-b36c-97abb284c459.0", 2023-10-18T20:14:11.513 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.419020ed-d3b2-4ce0-b36c-97abb284c459.2.retry", 2023-10-18T20:14:11.513 INFO:teuthology.orchestra.run.smithi082.stdout: "meta.log.67bf1060-aa43-46d9-8036-efd54ef55d7d.0", 2023-10-18T20:14:11.513 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:7", 2023-10-18T20:14:11.513 INFO:teuthology.orchestra.run.smithi082.stdout: "data_loggenerations_metadata", 2023-10-18T20:14:11.514 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:9", 2023-10-18T20:14:11.514 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000076", 2023-10-18T20:14:11.514 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000113", 2023-10-18T20:14:11.514 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000005", 2023-10-18T20:14:11.514 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000011", 2023-10-18T20:14:11.515 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000058", 2023-10-18T20:14:11.515 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000052", 2023-10-18T20:14:11.515 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:3", 2023-10-18T20:14:11.515 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:3", 2023-10-18T20:14:11.515 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:1", 2023-10-18T20:14:11.516 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:8", 2023-10-18T20:14:11.516 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:5", 2023-10-18T20:14:11.516 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:7", 2023-10-18T20:14:11.516 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.419020ed-d3b2-4ce0-b36c-97abb284c459.3.retry", 2023-10-18T20:14:11.516 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:0", 2023-10-18T20:14:11.516 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000088", 2023-10-18T20:14:11.517 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000080", 2023-10-18T20:14:11.517 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13", 2023-10-18T20:14:11.517 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000090", 2023-10-18T20:14:11.517 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000110", 2023-10-18T20:14:11.518 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:5", 2023-10-18T20:14:11.518 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000096", 2023-10-18T20:14:11.518 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:5", 2023-10-18T20:14:11.518 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000087", 2023-10-18T20:14:11.518 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000008", 2023-10-18T20:14:11.519 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:9", 2023-10-18T20:14:11.519 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000006", 2023-10-18T20:14:11.519 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000029", 2023-10-18T20:14:11.519 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:2", 2023-10-18T20:14:11.519 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000089", 2023-10-18T20:14:11.520 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:9", 2023-10-18T20:14:11.520 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000022", 2023-10-18T20:14:11.520 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9", 2023-10-18T20:14:11.520 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:8", 2023-10-18T20:14:11.520 INFO:teuthology.orchestra.run.smithi082.stdout: "meta.log.67bf1060-aa43-46d9-8036-efd54ef55d7d.1", 2023-10-18T20:14:11.521 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog.trim", 2023-10-18T20:14:11.521 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:6", 2023-10-18T20:14:11.521 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000016", 2023-10-18T20:14:11.521 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:10", 2023-10-18T20:14:11.522 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:7", 2023-10-18T20:14:11.522 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000041", 2023-10-18T20:14:11.522 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10", 2023-10-18T20:14:11.522 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:4", 2023-10-18T20:14:11.522 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:9", 2023-10-18T20:14:11.523 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000018", 2023-10-18T20:14:11.523 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000092", 2023-10-18T20:14:11.523 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:6", 2023-10-18T20:14:11.523 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:5", 2023-10-18T20:14:11.523 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:1", 2023-10-18T20:14:11.524 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:3", 2023-10-18T20:14:11.524 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:3", 2023-10-18T20:14:11.524 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.4", 2023-10-18T20:14:11.524 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:4", 2023-10-18T20:14:11.525 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:0", 2023-10-18T20:14:11.525 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:8", 2023-10-18T20:14:11.525 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:1", 2023-10-18T20:14:11.525 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:0", 2023-10-18T20:14:11.525 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000014", 2023-10-18T20:14:11.526 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:6", 2023-10-18T20:14:11.526 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7", 2023-10-18T20:14:11.526 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:5", 2023-10-18T20:14:11.526 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:1", 2023-10-18T20:14:11.527 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000112", 2023-10-18T20:14:11.527 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000007", 2023-10-18T20:14:11.527 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000021", 2023-10-18T20:14:11.527 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:2", 2023-10-18T20:14:11.527 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000064", 2023-10-18T20:14:11.528 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000071", 2023-10-18T20:14:11.528 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000074", 2023-10-18T20:14:11.528 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:6", 2023-10-18T20:14:11.528 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:5", 2023-10-18T20:14:11.529 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000081", 2023-10-18T20:14:11.529 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000009", 2023-10-18T20:14:11.529 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:1", 2023-10-18T20:14:11.529 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33", 2023-10-18T20:14:11.529 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:4", 2023-10-18T20:14:11.530 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000121", 2023-10-18T20:14:11.530 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.1", 2023-10-18T20:14:11.530 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:4", 2023-10-18T20:14:11.530 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000125", 2023-10-18T20:14:11.530 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000082", 2023-10-18T20:14:11.530 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:7", 2023-10-18T20:14:11.531 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:8", 2023-10-18T20:14:11.531 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000105", 2023-10-18T20:14:11.531 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000059", 2023-10-18T20:14:11.531 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:8", 2023-10-18T20:14:11.531 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:4", 2023-10-18T20:14:11.532 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000077", 2023-10-18T20:14:11.532 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000032", 2023-10-18T20:14:11.532 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:4", 2023-10-18T20:14:11.532 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:1", 2023-10-18T20:14:11.532 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:0", 2023-10-18T20:14:11.533 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:7", 2023-10-18T20:14:11.533 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:7", 2023-10-18T20:14:11.533 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000053", 2023-10-18T20:14:11.533 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:5", 2023-10-18T20:14:11.533 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:0", 2023-10-18T20:14:11.534 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000091", 2023-10-18T20:14:11.534 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12", 2023-10-18T20:14:11.534 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:8", 2023-10-18T20:14:11.534 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000065", 2023-10-18T20:14:11.534 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15", 2023-10-18T20:14:11.534 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000083", 2023-10-18T20:14:11.535 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000010", 2023-10-18T20:14:11.535 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:6", 2023-10-18T20:14:11.535 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:3", 2023-10-18T20:14:11.535 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.7", 2023-10-18T20:14:11.535 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.3", 2023-10-18T20:14:11.536 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:2", 2023-10-18T20:14:11.536 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:3", 2023-10-18T20:14:11.536 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:1", 2023-10-18T20:14:11.536 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:3", 2023-10-18T20:14:11.536 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:8", 2023-10-18T20:14:11.536 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:0", 2023-10-18T20:14:11.537 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:2", 2023-10-18T20:14:11.537 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:2", 2023-10-18T20:14:11.537 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000045", 2023-10-18T20:14:11.537 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8", 2023-10-18T20:14:11.537 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000002", 2023-10-18T20:14:11.538 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000116", 2023-10-18T20:14:11.538 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000034", 2023-10-18T20:14:11.538 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000101", 2023-10-18T20:14:11.538 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:0", 2023-10-18T20:14:11.538 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:6", 2023-10-18T20:14:11.539 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:2", 2023-10-18T20:14:11.539 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000079", 2023-10-18T20:14:11.539 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000049", 2023-10-18T20:14:11.539 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:2", 2023-10-18T20:14:11.539 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000117", 2023-10-18T20:14:11.539 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000044", 2023-10-18T20:14:11.540 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:10", 2023-10-18T20:14:11.540 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:5", 2023-10-18T20:14:11.540 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000066", 2023-10-18T20:14:11.540 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:1", 2023-10-18T20:14:11.540 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000068", 2023-10-18T20:14:11.541 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000085", 2023-10-18T20:14:11.541 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:3", 2023-10-18T20:14:11.541 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000073", 2023-10-18T20:14:11.541 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:9", 2023-10-18T20:14:11.541 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000038", 2023-10-18T20:14:11.542 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:0", 2023-10-18T20:14:11.542 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1", 2023-10-18T20:14:11.542 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.419020ed-d3b2-4ce0-b36c-97abb284c459.1", 2023-10-18T20:14:11.542 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000118", 2023-10-18T20:14:11.542 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:2", 2023-10-18T20:14:11.543 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:0", 2023-10-18T20:14:11.543 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000036", 2023-10-18T20:14:11.543 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32", 2023-10-18T20:14:11.543 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000103", 2023-10-18T20:14:11.543 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:10", 2023-10-18T20:14:11.544 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:2", 2023-10-18T20:14:11.544 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:10", 2023-10-18T20:14:11.544 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:4", 2023-10-18T20:14:11.544 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:9", 2023-10-18T20:14:11.544 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:5", 2023-10-18T20:14:11.544 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000119", 2023-10-18T20:14:11.545 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000098", 2023-10-18T20:14:11.545 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:0", 2023-10-18T20:14:11.545 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000027", 2023-10-18T20:14:11.545 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:10", 2023-10-18T20:14:11.545 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:8", 2023-10-18T20:14:11.546 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:7", 2023-10-18T20:14:11.546 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000019", 2023-10-18T20:14:11.546 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000039", 2023-10-18T20:14:11.546 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:6", 2023-10-18T20:14:11.546 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:8", 2023-10-18T20:14:11.546 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:3", 2023-10-18T20:14:11.547 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000100", 2023-10-18T20:14:11.547 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:7", 2023-10-18T20:14:11.547 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:3", 2023-10-18T20:14:11.547 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:1", 2023-10-18T20:14:11.548 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000093", 2023-10-18T20:14:11.548 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:0", 2023-10-18T20:14:11.548 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000004", 2023-10-18T20:14:11.548 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000063", 2023-10-18T20:14:11.548 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:7", 2023-10-18T20:14:11.549 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000122", 2023-10-18T20:14:11.549 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000057", 2023-10-18T20:14:11.549 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:6", 2023-10-18T20:14:11.549 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:2", 2023-10-18T20:14:11.549 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:3", 2023-10-18T20:14:11.550 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000054", 2023-10-18T20:14:11.550 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000114", 2023-10-18T20:14:11.550 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.2", 2023-10-18T20:14:11.550 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.0", 2023-10-18T20:14:11.550 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:6", 2023-10-18T20:14:11.550 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000012", 2023-10-18T20:14:11.551 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:10", 2023-10-18T20:14:11.551 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:1", 2023-10-18T20:14:11.551 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:5", 2023-10-18T20:14:11.551 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000084", 2023-10-18T20:14:11.551 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000043", 2023-10-18T20:14:11.552 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:10", 2023-10-18T20:14:11.552 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:9", 2023-10-18T20:14:11.552 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.419020ed-d3b2-4ce0-b36c-97abb284c459.1.retry", 2023-10-18T20:14:11.552 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:10", 2023-10-18T20:14:11.552 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:8", 2023-10-18T20:14:11.553 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000111", 2023-10-18T20:14:11.553 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000048", 2023-10-18T20:14:11.553 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000020", 2023-10-18T20:14:11.553 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:9", 2023-10-18T20:14:11.553 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:4", 2023-10-18T20:14:11.554 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000099", 2023-10-18T20:14:11.554 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000056", 2023-10-18T20:14:11.554 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:8", 2023-10-18T20:14:11.554 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:0", 2023-10-18T20:14:11.554 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:6", 2023-10-18T20:14:11.555 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000072", 2023-10-18T20:14:11.555 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:3", 2023-10-18T20:14:11.555 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000062", 2023-10-18T20:14:11.555 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11", 2023-10-18T20:14:11.555 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:0", 2023-10-18T20:14:11.555 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000115", 2023-10-18T20:14:11.556 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.1", 2023-10-18T20:14:11.556 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000033", 2023-10-18T20:14:11.556 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:6", 2023-10-18T20:14:11.556 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:1", 2023-10-18T20:14:11.556 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18", 2023-10-18T20:14:11.557 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:4", 2023-10-18T20:14:11.557 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:1", 2023-10-18T20:14:11.557 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:10", 2023-10-18T20:14:11.557 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.0.Bqm+4B+8RG0.0", 2023-10-18T20:14:11.557 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:10", 2023-10-18T20:14:11.558 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.1.U6Sy7SiVSX2.0", 2023-10-18T20:14:11.558 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000025", 2023-10-18T20:14:11.558 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:0", 2023-10-18T20:14:11.558 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000123", 2023-10-18T20:14:11.558 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:7", 2023-10-18T20:14:11.559 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:3", 2023-10-18T20:14:11.559 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:6", 2023-10-18T20:14:11.559 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:2", 2023-10-18T20:14:11.559 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:1", 2023-10-18T20:14:11.559 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000108", 2023-10-18T20:14:11.560 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000094", 2023-10-18T20:14:11.560 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14", 2023-10-18T20:14:11.560 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:1", 2023-10-18T20:14:11.560 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000060", 2023-10-18T20:14:11.560 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000109", 2023-10-18T20:14:11.561 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:7", 2023-10-18T20:14:11.561 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000013", 2023-10-18T20:14:11.561 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:10", 2023-10-18T20:14:11.561 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:10", 2023-10-18T20:14:11.561 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:1", 2023-10-18T20:14:11.562 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000051", 2023-10-18T20:14:11.562 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:9", 2023-10-18T20:14:11.562 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:3", 2023-10-18T20:14:11.562 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000106", 2023-10-18T20:14:11.562 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:8", 2023-10-18T20:14:11.563 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:0", 2023-10-18T20:14:11.563 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.2.4vuNzwmR2pV.0", 2023-10-18T20:14:11.563 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000017", 2023-10-18T20:14:11.563 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:5", 2023-10-18T20:14:11.563 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:4", 2023-10-18T20:14:11.564 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000023", 2023-10-18T20:14:11.564 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:10", 2023-10-18T20:14:11.564 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:5", 2023-10-18T20:14:11.564 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:10", 2023-10-18T20:14:11.564 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000107", 2023-10-18T20:14:11.565 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:5", 2023-10-18T20:14:11.565 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000046", 2023-10-18T20:14:11.565 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:8", 2023-10-18T20:14:11.565 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34", 2023-10-18T20:14:11.565 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16", 2023-10-18T20:14:11.565 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:1", 2023-10-18T20:14:11.566 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.3.xia0S/gdG1Q.0", 2023-10-18T20:14:11.566 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.2", 2023-10-18T20:14:11.566 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000067", 2023-10-18T20:14:11.566 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:8", 2023-10-18T20:14:11.566 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:4", 2023-10-18T20:14:11.567 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.419020ed-d3b2-4ce0-b36c-97abb284c459.2", 2023-10-18T20:14:11.567 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.9", 2023-10-18T20:14:11.567 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000086", 2023-10-18T20:14:11.567 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.5", 2023-10-18T20:14:11.568 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:9", 2023-10-18T20:14:11.568 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31", 2023-10-18T20:14:11.568 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:4", 2023-10-18T20:14:11.568 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:4", 2023-10-18T20:14:11.568 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17", 2023-10-18T20:14:11.569 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:6", 2023-10-18T20:14:11.569 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:5", 2023-10-18T20:14:11.569 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:9", 2023-10-18T20:14:11.569 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000050", 2023-10-18T20:14:11.569 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000102", 2023-10-18T20:14:11.570 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:2", 2023-10-18T20:14:11.570 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:5", 2023-10-18T20:14:11.570 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:10", 2023-10-18T20:14:11.570 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:6", 2023-10-18T20:14:11.570 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:5", 2023-10-18T20:14:11.571 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000124", 2023-10-18T20:14:11.571 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:4", 2023-10-18T20:14:11.571 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:1", 2023-10-18T20:14:11.571 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000001", 2023-10-18T20:14:11.571 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000126", 2023-10-18T20:14:11.572 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000055", 2023-10-18T20:14:11.572 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000061", 2023-10-18T20:14:11.572 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:5", 2023-10-18T20:14:11.572 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:9", 2023-10-18T20:14:11.572 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:8", 2023-10-18T20:14:11.572 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:10", 2023-10-18T20:14:11.573 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000097", 2023-10-18T20:14:11.573 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000042", 2023-10-18T20:14:11.573 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:6", 2023-10-18T20:14:11.573 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000075", 2023-10-18T20:14:11.573 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000120", 2023-10-18T20:14:11.574 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000030", 2023-10-18T20:14:11.574 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:4", 2023-10-18T20:14:11.574 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:0" 2023-10-18T20:14:11.574 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:14:11.574 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T20:14:11.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.577 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:11.624 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:11.624 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:11.666 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:14:11.666 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000078", 2023-10-18T20:14:11.667 INFO:teuthology.orchestra.run.smithi082.stdout: "meta.history", 2023-10-18T20:14:11.667 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:8", 2023-10-18T20:14:11.667 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000070", 2023-10-18T20:14:11.667 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000104", 2023-10-18T20:14:11.668 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:1", 2023-10-18T20:14:11.668 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000026", 2023-10-18T20:14:11.668 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:2", 2023-10-18T20:14:11.668 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12", 2023-10-18T20:14:11.668 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:7", 2023-10-18T20:14:11.669 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.8", 2023-10-18T20:14:11.669 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000028", 2023-10-18T20:14:11.669 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:1", 2023-10-18T20:14:11.669 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:2", 2023-10-18T20:14:11.669 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:5", 2023-10-18T20:14:11.670 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:10", 2023-10-18T20:14:11.670 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000040", 2023-10-18T20:14:11.670 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:6", 2023-10-18T20:14:11.670 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.6", 2023-10-18T20:14:11.671 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15", 2023-10-18T20:14:11.671 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000015", 2023-10-18T20:14:11.671 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:6", 2023-10-18T20:14:11.671 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.403a921b-fcfc-4ab2-b434-bf29d35a51b6.0.retry", 2023-10-18T20:14:11.672 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000069", 2023-10-18T20:14:11.672 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000095", 2023-10-18T20:14:11.672 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:9", 2023-10-18T20:14:11.672 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000003", 2023-10-18T20:14:11.673 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000047", 2023-10-18T20:14:11.673 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:2", 2023-10-18T20:14:11.673 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:10", 2023-10-18T20:14:11.673 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000035", 2023-10-18T20:14:11.673 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000037", 2023-10-18T20:14:11.674 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:8", 2023-10-18T20:14:11.674 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.0.ynr5jsveusi.0", 2023-10-18T20:14:11.674 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:9", 2023-10-18T20:14:11.674 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:10", 2023-10-18T20:14:11.674 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000024", 2023-10-18T20:14:11.675 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.1.7d8+0Dw6nO8.0", 2023-10-18T20:14:11.675 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33", 2023-10-18T20:14:11.675 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:5", 2023-10-18T20:14:11.675 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000000", 2023-10-18T20:14:11.675 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000031", 2023-10-18T20:14:11.675 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:9", 2023-10-18T20:14:11.676 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:3", 2023-10-18T20:14:11.676 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.3", 2023-10-18T20:14:11.676 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:1", 2023-10-18T20:14:11.676 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:7", 2023-10-18T20:14:11.676 INFO:teuthology.orchestra.run.smithi082.stdout: "meta.log.67bf1060-aa43-46d9-8036-efd54ef55d7d.0", 2023-10-18T20:14:11.677 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:4", 2023-10-18T20:14:11.677 INFO:teuthology.orchestra.run.smithi082.stdout: "data_loggenerations_metadata", 2023-10-18T20:14:11.677 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:8", 2023-10-18T20:14:11.677 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:8", 2023-10-18T20:14:11.677 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:8", 2023-10-18T20:14:11.677 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000076", 2023-10-18T20:14:11.678 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:2", 2023-10-18T20:14:11.678 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000113", 2023-10-18T20:14:11.678 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:10", 2023-10-18T20:14:11.678 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000005", 2023-10-18T20:14:11.679 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000011", 2023-10-18T20:14:11.679 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000058", 2023-10-18T20:14:11.679 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:4", 2023-10-18T20:14:11.679 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000052", 2023-10-18T20:14:11.679 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1", 2023-10-18T20:14:11.679 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:8", 2023-10-18T20:14:11.680 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000088", 2023-10-18T20:14:11.680 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000080", 2023-10-18T20:14:11.680 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000090", 2023-10-18T20:14:11.680 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:7", 2023-10-18T20:14:11.680 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:4", 2023-10-18T20:14:11.681 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000110", 2023-10-18T20:14:11.681 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000096", 2023-10-18T20:14:11.681 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:3", 2023-10-18T20:14:11.681 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:7", 2023-10-18T20:14:11.681 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:2", 2023-10-18T20:14:11.681 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9", 2023-10-18T20:14:11.682 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000087", 2023-10-18T20:14:11.682 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:3", 2023-10-18T20:14:11.682 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000008", 2023-10-18T20:14:11.682 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:1", 2023-10-18T20:14:11.683 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000006", 2023-10-18T20:14:11.683 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:4", 2023-10-18T20:14:11.683 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000029", 2023-10-18T20:14:11.683 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11", 2023-10-18T20:14:11.683 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000089", 2023-10-18T20:14:11.684 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000022", 2023-10-18T20:14:11.684 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:7", 2023-10-18T20:14:11.684 INFO:teuthology.orchestra.run.smithi082.stdout: "meta.log.67bf1060-aa43-46d9-8036-efd54ef55d7d.1", 2023-10-18T20:14:11.684 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog.trim", 2023-10-18T20:14:11.684 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:6", 2023-10-18T20:14:11.684 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000016", 2023-10-18T20:14:11.685 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:0", 2023-10-18T20:14:11.685 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:8", 2023-10-18T20:14:11.685 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000041", 2023-10-18T20:14:11.685 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:10", 2023-10-18T20:14:11.685 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:10", 2023-10-18T20:14:11.686 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:2", 2023-10-18T20:14:11.686 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000018", 2023-10-18T20:14:11.686 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:2", 2023-10-18T20:14:11.686 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000092", 2023-10-18T20:14:11.686 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:4", 2023-10-18T20:14:11.687 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:4", 2023-10-18T20:14:11.687 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.4", 2023-10-18T20:14:11.687 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17", 2023-10-18T20:14:11.687 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:8", 2023-10-18T20:14:11.687 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000014", 2023-10-18T20:14:11.688 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:10", 2023-10-18T20:14:11.688 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000112", 2023-10-18T20:14:11.688 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000007", 2023-10-18T20:14:11.688 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000021", 2023-10-18T20:14:11.688 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:5", 2023-10-18T20:14:11.689 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:8", 2023-10-18T20:14:11.689 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:6", 2023-10-18T20:14:11.689 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000064", 2023-10-18T20:14:11.689 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:5", 2023-10-18T20:14:11.689 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:1", 2023-10-18T20:14:11.690 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000071", 2023-10-18T20:14:11.690 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000074", 2023-10-18T20:14:11.690 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:3", 2023-10-18T20:14:11.690 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:1", 2023-10-18T20:14:11.691 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:3", 2023-10-18T20:14:11.691 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:0", 2023-10-18T20:14:11.691 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:1", 2023-10-18T20:14:11.691 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:1", 2023-10-18T20:14:11.691 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:10", 2023-10-18T20:14:11.692 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000081", 2023-10-18T20:14:11.692 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000009", 2023-10-18T20:14:11.692 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000121", 2023-10-18T20:14:11.692 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.1", 2023-10-18T20:14:11.692 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:6", 2023-10-18T20:14:11.692 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:9", 2023-10-18T20:14:11.693 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:0", 2023-10-18T20:14:11.693 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000125", 2023-10-18T20:14:11.693 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:0", 2023-10-18T20:14:11.693 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000082", 2023-10-18T20:14:11.693 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:1", 2023-10-18T20:14:11.694 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.403a921b-fcfc-4ab2-b434-bf29d35a51b6.0", 2023-10-18T20:14:11.694 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000105", 2023-10-18T20:14:11.694 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000059", 2023-10-18T20:14:11.694 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:3", 2023-10-18T20:14:11.695 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:10", 2023-10-18T20:14:11.695 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:2", 2023-10-18T20:14:11.695 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000077", 2023-10-18T20:14:11.695 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000032", 2023-10-18T20:14:11.695 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:4", 2023-10-18T20:14:11.696 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:8", 2023-10-18T20:14:11.696 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:3", 2023-10-18T20:14:11.696 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:7", 2023-10-18T20:14:11.696 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000053", 2023-10-18T20:14:11.696 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000091", 2023-10-18T20:14:11.697 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000065", 2023-10-18T20:14:11.697 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:3", 2023-10-18T20:14:11.697 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000083", 2023-10-18T20:14:11.697 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000010", 2023-10-18T20:14:11.698 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:0", 2023-10-18T20:14:11.698 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.403a921b-fcfc-4ab2-b434-bf29d35a51b6.3", 2023-10-18T20:14:11.698 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13", 2023-10-18T20:14:11.698 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.7", 2023-10-18T20:14:11.698 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.3", 2023-10-18T20:14:11.699 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:1", 2023-10-18T20:14:11.699 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:9", 2023-10-18T20:14:11.699 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:2", 2023-10-18T20:14:11.699 INFO:teuthology.orchestra.run.smithi082.stdout: "mdlog.sync-status.shard.2", 2023-10-18T20:14:11.699 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:3", 2023-10-18T20:14:11.699 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32", 2023-10-18T20:14:11.700 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:1", 2023-10-18T20:14:11.700 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000045", 2023-10-18T20:14:11.700 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000002", 2023-10-18T20:14:11.700 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.12", 2023-10-18T20:14:11.700 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:3", 2023-10-18T20:14:11.701 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000116", 2023-10-18T20:14:11.701 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:3", 2023-10-18T20:14:11.701 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000034", 2023-10-18T20:14:11.701 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:1", 2023-10-18T20:14:11.701 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:2", 2023-10-18T20:14:11.702 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000101", 2023-10-18T20:14:11.702 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.13", 2023-10-18T20:14:11.702 INFO:teuthology.orchestra.run.smithi082.stdout: "mdlog.sync-status.shard.1", 2023-10-18T20:14:11.702 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:1", 2023-10-18T20:14:11.702 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000079", 2023-10-18T20:14:11.703 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000049", 2023-10-18T20:14:11.703 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000117", 2023-10-18T20:14:11.703 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000044", 2023-10-18T20:14:11.703 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:5", 2023-10-18T20:14:11.703 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:7", 2023-10-18T20:14:11.704 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7", 2023-10-18T20:14:11.704 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:4", 2023-10-18T20:14:11.704 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000066", 2023-10-18T20:14:11.704 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:10", 2023-10-18T20:14:11.704 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:6", 2023-10-18T20:14:11.704 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000068", 2023-10-18T20:14:11.705 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:5", 2023-10-18T20:14:11.705 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000085", 2023-10-18T20:14:11.705 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:2", 2023-10-18T20:14:11.705 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:2", 2023-10-18T20:14:11.705 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34", 2023-10-18T20:14:11.706 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:4", 2023-10-18T20:14:11.706 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000073", 2023-10-18T20:14:11.706 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:0", 2023-10-18T20:14:11.706 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000038", 2023-10-18T20:14:11.707 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:7", 2023-10-18T20:14:11.707 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.10", 2023-10-18T20:14:11.707 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000118", 2023-10-18T20:14:11.707 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:8", 2023-10-18T20:14:11.707 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000036", 2023-10-18T20:14:11.708 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:0", 2023-10-18T20:14:11.708 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000103", 2023-10-18T20:14:11.708 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:6", 2023-10-18T20:14:11.708 INFO:teuthology.orchestra.run.smithi082.stdout: "mdlog.sync-status.shard.3", 2023-10-18T20:14:11.708 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000119", 2023-10-18T20:14:11.709 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000098", 2023-10-18T20:14:11.709 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18", 2023-10-18T20:14:11.709 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:3", 2023-10-18T20:14:11.709 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:7", 2023-10-18T20:14:11.709 INFO:teuthology.orchestra.run.smithi082.stdout: "mdlog.sync-status.shard.0", 2023-10-18T20:14:11.710 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10", 2023-10-18T20:14:11.710 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000027", 2023-10-18T20:14:11.710 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:6", 2023-10-18T20:14:11.710 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:10", 2023-10-18T20:14:11.710 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:10", 2023-10-18T20:14:11.711 INFO:teuthology.orchestra.run.smithi082.stdout: "mdlog.sync-status", 2023-10-18T20:14:11.711 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000019", 2023-10-18T20:14:11.711 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000039", 2023-10-18T20:14:11.711 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:9", 2023-10-18T20:14:11.711 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:5", 2023-10-18T20:14:11.712 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000100", 2023-10-18T20:14:11.712 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:5", 2023-10-18T20:14:11.712 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.403a921b-fcfc-4ab2-b434-bf29d35a51b6.1", 2023-10-18T20:14:11.712 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000093", 2023-10-18T20:14:11.712 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:5", 2023-10-18T20:14:11.713 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:8", 2023-10-18T20:14:11.713 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:9", 2023-10-18T20:14:11.713 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000004", 2023-10-18T20:14:11.713 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000063", 2023-10-18T20:14:11.713 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000122", 2023-10-18T20:14:11.713 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000057", 2023-10-18T20:14:11.714 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000054", 2023-10-18T20:14:11.714 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000114", 2023-10-18T20:14:11.714 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.2", 2023-10-18T20:14:11.714 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.0", 2023-10-18T20:14:11.714 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000012", 2023-10-18T20:14:11.715 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T20:14:11.715 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:9", 2023-10-18T20:14:11.715 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.403a921b-fcfc-4ab2-b434-bf29d35a51b6.2.retry", 2023-10-18T20:14:11.715 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:6", 2023-10-18T20:14:11.715 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000084", 2023-10-18T20:14:11.715 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000043", 2023-10-18T20:14:11.716 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:5", 2023-10-18T20:14:11.716 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.11", 2023-10-18T20:14:11.716 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:0", 2023-10-18T20:14:11.716 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:2", 2023-10-18T20:14:11.716 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:6", 2023-10-18T20:14:11.717 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000111", 2023-10-18T20:14:11.717 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000048", 2023-10-18T20:14:11.717 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14", 2023-10-18T20:14:11.717 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:0", 2023-10-18T20:14:11.717 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000020", 2023-10-18T20:14:11.717 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:3", 2023-10-18T20:14:11.718 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:10", 2023-10-18T20:14:11.718 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.2.+FYmcF6BNvm.0", 2023-10-18T20:14:11.718 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:5", 2023-10-18T20:14:11.718 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:8", 2023-10-18T20:14:11.718 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:9", 2023-10-18T20:14:11.719 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000099", 2023-10-18T20:14:11.719 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000056", 2023-10-18T20:14:11.719 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:5", 2023-10-18T20:14:11.719 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:4", 2023-10-18T20:14:11.719 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:4", 2023-10-18T20:14:11.720 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000072", 2023-10-18T20:14:11.720 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000062", 2023-10-18T20:14:11.720 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000115", 2023-10-18T20:14:11.720 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:4", 2023-10-18T20:14:11.720 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.1", 2023-10-18T20:14:11.721 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000033", 2023-10-18T20:14:11.721 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:4", 2023-10-18T20:14:11.721 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:5", 2023-10-18T20:14:11.721 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:3", 2023-10-18T20:14:11.721 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:9", 2023-10-18T20:14:11.721 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:4", 2023-10-18T20:14:11.722 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:6", 2023-10-18T20:14:11.722 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:5", 2023-10-18T20:14:11.722 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:10", 2023-10-18T20:14:11.722 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:7", 2023-10-18T20:14:11.722 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:0", 2023-10-18T20:14:11.723 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:7", 2023-10-18T20:14:11.723 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:0", 2023-10-18T20:14:11.723 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000025", 2023-10-18T20:14:11.723 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000123", 2023-10-18T20:14:11.723 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000108", 2023-10-18T20:14:11.724 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:9", 2023-10-18T20:14:11.724 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000094", 2023-10-18T20:14:11.724 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:9", 2023-10-18T20:14:11.724 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000060", 2023-10-18T20:14:11.724 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000109", 2023-10-18T20:14:11.724 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:2", 2023-10-18T20:14:11.725 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000013", 2023-10-18T20:14:11.725 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:4", 2023-10-18T20:14:11.725 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:0", 2023-10-18T20:14:11.725 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:6", 2023-10-18T20:14:11.725 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:9", 2023-10-18T20:14:11.726 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:2", 2023-10-18T20:14:11.726 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:8", 2023-10-18T20:14:11.726 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:0", 2023-10-18T20:14:11.726 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:7", 2023-10-18T20:14:11.726 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.403a921b-fcfc-4ab2-b434-bf29d35a51b6.2", 2023-10-18T20:14:11.727 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000051", 2023-10-18T20:14:11.727 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000106", 2023-10-18T20:14:11.727 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.3.ZCfJzSUinzz.0", 2023-10-18T20:14:11.727 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:5", 2023-10-18T20:14:11.727 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000017", 2023-10-18T20:14:11.728 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:6", 2023-10-18T20:14:11.728 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:6", 2023-10-18T20:14:11.728 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-12:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.12:1", 2023-10-18T20:14:11.728 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:2", 2023-10-18T20:14:11.728 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000023", 2023-10-18T20:14:11.729 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:3", 2023-10-18T20:14:11.729 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000107", 2023-10-18T20:14:11.729 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000046", 2023-10-18T20:14:11.729 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31", 2023-10-18T20:14:11.729 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-9:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.9:10", 2023-10-18T20:14:11.730 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:9", 2023-10-18T20:14:11.730 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:8", 2023-10-18T20:14:11.730 INFO:teuthology.orchestra.run.smithi082.stdout: "data_log.2", 2023-10-18T20:14:11.730 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.403a921b-fcfc-4ab2-b434-bf29d35a51b6.3.retry", 2023-10-18T20:14:11.730 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000067", 2023-10-18T20:14:11.730 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:1", 2023-10-18T20:14:11.731 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:9", 2023-10-18T20:14:11.731 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:8", 2023-10-18T20:14:11.731 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:0", 2023-10-18T20:14:11.731 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:9", 2023-10-18T20:14:11.731 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.9", 2023-10-18T20:14:11.732 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000086", 2023-10-18T20:14:11.732 INFO:teuthology.orchestra.run.smithi082.stdout: "sync.error-log.5", 2023-10-18T20:14:11.732 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-13:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.13:9", 2023-10-18T20:14:11.732 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-34:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.34:7", 2023-10-18T20:14:11.732 INFO:teuthology.orchestra.run.smithi082.stdout: "datalog.sync-status.shard.403a921b-fcfc-4ab2-b434-bf29d35a51b6.1.retry", 2023-10-18T20:14:11.732 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:0", 2023-10-18T20:14:11.733 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:6", 2023-10-18T20:14:11.733 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:10", 2023-10-18T20:14:11.733 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8:2", 2023-10-18T20:14:11.733 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:1", 2023-10-18T20:14:11.733 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:4", 2023-10-18T20:14:11.734 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-15:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.15:7", 2023-10-18T20:14:11.734 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:1", 2023-10-18T20:14:11.734 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000050", 2023-10-18T20:14:11.734 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000102", 2023-10-18T20:14:11.735 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-8:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.8", 2023-10-18T20:14:11.735 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-35:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5415.1:0", 2023-10-18T20:14:11.735 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-31:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.31:6", 2023-10-18T20:14:11.735 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000124", 2023-10-18T20:14:11.735 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000001", 2023-10-18T20:14:11.735 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16", 2023-10-18T20:14:11.736 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000126", 2023-10-18T20:14:11.736 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000055", 2023-10-18T20:14:11.736 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000061", 2023-10-18T20:14:11.736 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:8", 2023-10-18T20:14:11.736 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000097", 2023-10-18T20:14:11.737 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:5", 2023-10-18T20:14:11.737 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:3", 2023-10-18T20:14:11.737 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-10:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.10:7", 2023-10-18T20:14:11.737 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-11:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.11:3", 2023-10-18T20:14:11.737 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000042", 2023-10-18T20:14:11.738 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-18:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.18:7", 2023-10-18T20:14:11.738 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-16:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.16:6", 2023-10-18T20:14:11.738 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-7:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.7:4", 2023-10-18T20:14:11.738 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-17:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.17:5", 2023-10-18T20:14:11.738 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000075", 2023-10-18T20:14:11.738 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-32:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.32:0", 2023-10-18T20:14:11.739 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000120", 2023-10-18T20:14:11.739 INFO:teuthology.orchestra.run.smithi082.stdout: "obj_delete_at_hint.0000000030", 2023-10-18T20:14:11.739 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-33:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.33:7", 2023-10-18T20:14:11.739 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-14:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.14:10" 2023-10-18T20:14:11.739 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:14:11.740 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-10-18T20:14:11.741 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-18T20:14:11.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.742 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:11.790 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:11.790 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:11.833 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:14:11.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.834 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003470", 2023-10-18T20:14:11.834 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:14:11.395196Z" 2023-10-18T20:14:11.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.835 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002058", 2023-10-18T20:14:11.835 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:14:11.396141Z" 2023-10-18T20:14:11.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.836 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001213", 2023-10-18T20:14:11.836 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:14:11.393317Z" 2023-10-18T20:14:11.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.836 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-10-18T20:14:11.837 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:14:11.394232Z" 2023-10-18T20:14:11.837 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.837 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:14:11.838 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000003470', 1: '00000000000000000000:00000000000000002058', 2: '00000000000000000000:00000000000000001213', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:14:11.838 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:14:11.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:11.839 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:11.890 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:11.890 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:11.931 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:14:11.931 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:14:11.931 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:14:11.931 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:14:11.932 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:14:11.932 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 2477610155421060198 2023-10-18T20:14:11.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.932 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:14:11.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.933 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:14:11.933 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:11.933 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:11.933 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-10-18T20:14:11.934 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:11.934 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:11.934 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:11.934 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:05:46.060902Z" 2023-10-18T20:14:11.935 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.935 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:14:11.935 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:11.936 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:11.936 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-18T20:14:11.936 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:11.936 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:11.936 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:11.937 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:16.711504Z" 2023-10-18T20:14:11.937 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.937 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:14:11.938 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:11.938 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:11.938 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:14:11.938 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:11.939 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:11.939 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:11.939 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:50:16.613109Z" 2023-10-18T20:14:11.939 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:11.940 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:14:11.940 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:11.940 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:11.940 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003465", 2023-10-18T20:14:11.941 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:11.941 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:11.941 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:11.941 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:17.768198Z" 2023-10-18T20:14:11.941 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.942 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:11.942 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:14:11.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.942 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:14:11.943 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:14:11.943 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:14:11.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:11.943 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:14:11Z" 2023-10-18T20:14:11.943 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:14:11.944 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2477610155421060198 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:05:46.060902Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:16.711504Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:50:16.613109Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:17.768198Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:14:11Z" } 2023-10-18T20:14:11.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:05:46.060902Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:16.711504Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:50:16.613109Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:17.768198Z'}}] 2023-10-18T20:14:11.945 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003470', 1: '00000000000000000000:00000000000000002058', 2: '00000000000000000000:00000000000000001213', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:14:11.946 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002625', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000003465'} 2023-10-18T20:14:11.946 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003470 target=00000000000000000000:00000000000000002625, shard=1 master=00000000000000000000:00000000000000002058 target=00000000000000000000:00000000000000001210, shard=2 master=00000000000000000000:00000000000000001213 target=00000000000000000000:00000000000000000651, shard=3 master=00000000000000000000:00000000000000004311 target=00000000000000000000:00000000000000003465 2023-10-18T20:14:16.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:16.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:16.949 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:17.008 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:17.008 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:17.055 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:14:17.055 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:14:17.055 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:14:17.055 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:14:17.056 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:14:17.056 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 2477610155421060198 2023-10-18T20:14:17.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.056 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:14:17.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.057 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:14:17.057 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.057 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:17.057 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003470", 2023-10-18T20:14:17.058 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.058 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.058 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.059 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:14:11.395196Z" 2023-10-18T20:14:17.059 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.059 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:14:17.060 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.060 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:17.060 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002058", 2023-10-18T20:14:17.060 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.060 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.061 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.061 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:14:11.396141Z" 2023-10-18T20:14:17.061 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.062 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:14:17.062 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.062 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:17.063 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001213", 2023-10-18T20:14:17.063 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.063 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.063 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.063 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:14:11.393317Z" 2023-10-18T20:14:17.064 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.064 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:14:17.065 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.065 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:17.065 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-10-18T20:14:17.065 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.065 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.066 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.066 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:14:11.394232Z" 2023-10-18T20:14:17.066 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.066 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.066 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:14:17.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.067 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:14:17.067 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:14:17.067 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:14:17.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.068 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:14:17Z" 2023-10-18T20:14:17.068 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:14:17.069 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2477610155421060198 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003470", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:14:11.395196Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:14:11.396141Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001213", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:14:11.393317Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:14:11.394232Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:14:17Z" } 2023-10-18T20:14:17.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003470', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:14:11.395196Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:14:11.396141Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001213', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:14:11.393317Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:14:11.394232Z'}}] 2023-10-18T20:14:17.070 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003470', 1: '00000000000000000000:00000000000000002058', 2: '00000000000000000000:00000000000000001213', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:14:17.071 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003470', 1: '00000000000000000000:00000000000000002058', 2: '00000000000000000000:00000000000000001213', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:14:17.071 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:14:17.071 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-18T20:14:17.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:17.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:17.073 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:17.122 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:17.123 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:17.168 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:14:17.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.169 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001781", 2023-10-18T20:14:17.169 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:05:46.302485Z" 2023-10-18T20:14:17.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.170 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-18T20:14:17.170 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:16.697848Z" 2023-10-18T20:14:17.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.171 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:14:17.171 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:49:56.644123Z" 2023-10-18T20:14:17.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.172 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-10-18T20:14:17.172 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:02:17.857801Z" 2023-10-18T20:14:17.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.172 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:14:17.173 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001781', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:14:17.174 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:14:17.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:17.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:17.175 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:17.226 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:17.226 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:17.267 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:14:17.267 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:14:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:14:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:14:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:14:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:14:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:14:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:14:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001781", 2023-10-18T20:14:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:05:46.302485Z" 2023-10-18T20:14:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:14:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-18T20:14:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:16.697848Z" 2023-10-18T20:14:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:14:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000000651", 2023-10-18T20:14:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:49:56.644123Z" 2023-10-18T20:14:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:14:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:14:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-10-18T20:14:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:02:17.857801Z" 2023-10-18T20:14:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:14:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:14:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:14:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:14:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:14:17Z" 2023-10-18T20:14:17.280 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:14:17.281 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001781", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:05:46.302485Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:16.697848Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000651", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:49:56.644123Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:02:17.857801Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:14:17Z" } 2023-10-18T20:14:17.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001781', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:05:46.302485Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:16.697848Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000651', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:49:56.644123Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:02:17.857801Z'}}] 2023-10-18T20:14:17.282 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001781', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:14:17.283 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001781', 1: '00000000000000000000:00000000000000001210', 2: '00000000000000000000:00000000000000000651', 3: '00000000000000000000:00000000000000004311'} 2023-10-18T20:14:17.283 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:14:17.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2023-10-18T20:14:17.285 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-46 2023-10-18T20:14:17.285 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:14:17.286 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:14:17.286 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.287 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:14:17.287 DEBUG:boto:Data: 2023-10-18T20:14:17.287 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:14:17.288 DEBUG:boto:Port: 8000 2023-10-18T20:14:17.289 DEBUG:boto:Params: {} 2023-10-18T20:14:17.289 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:14:17.290 DEBUG:boto:Token: None 2023-10-18T20:14:17.290 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-46/ 2023-10-18T20:14:17.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SV4zx+8zN5ZUORpKgUcYPL9q9BU= 2023-10-18T20:14:17.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SV4zx+8zN5ZUORpKgUcYPL9q9BU=', 'Content-Length': '0'} 2023-10-18T20:14:17.307 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000064b5da262b33eee-0065303c99-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.308 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-47 2023-10-18T20:14:17.309 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:14:17.310 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:14:17.310 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.311 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:14:17.311 DEBUG:boto:Data: 2023-10-18T20:14:17.312 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:14:17.313 DEBUG:boto:Port: 8001 2023-10-18T20:14:17.313 DEBUG:boto:Params: {} 2023-10-18T20:14:17.314 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:14:17.314 DEBUG:boto:Token: None 2023-10-18T20:14:17.315 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-47/ 2023-10-18T20:14:17.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iLZBBkJL2O+Tz2dPUwnPyXbGBns= 2023-10-18T20:14:17.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iLZBBkJL2O+Tz2dPUwnPyXbGBns=', 'Content-Length': '0'} 2023-10-18T20:14:17.341 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000023afe913551ec064-0065303c99-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.342 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:14:17.343 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:14:17.343 DEBUG:boto:Method: HEAD 2023-10-18T20:14:17.344 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:14:17.344 DEBUG:boto:Data: 2023-10-18T20:14:17.344 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.345 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:14:17.345 DEBUG:boto:Port: 8000 2023-10-18T20:14:17.346 DEBUG:boto:Params: {} 2023-10-18T20:14:17.346 DEBUG:boto:Token: None 2023-10-18T20:14:17.347 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-46/ 2023-10-18T20:14:17.347 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWFUvK73B+1/2SgjlrQFzGsycoA= 2023-10-18T20:14:17.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWFUvK73B+1/2SgjlrQFzGsycoA=', 'Content-Length': '0'} 2023-10-18T20:14:17.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099ff0161016d728a-0065303c99-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.351 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:14:17.351 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:14:17.352 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.352 DEBUG:boto:Path: /bofrnf-46/obj1 2023-10-18T20:14:17.352 DEBUG:boto:Data: 2023-10-18T20:14:17.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:14:17.353 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:14:17.354 DEBUG:boto:Port: 8000 2023-10-18T20:14:17.354 DEBUG:boto:Params: {} 2023-10-18T20:14:17.355 DEBUG:boto:Token: None 2023-10-18T20:14:17.355 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-46/obj1 2023-10-18T20:14:17.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I9UTdVvIbQy2plsjodGvpXbCdCY= 2023-10-18T20:14:17.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I9UTdVvIbQy2plsjodGvpXbCdCY='} 2023-10-18T20:14:17.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0095bbc0ca869b0-0065303c99-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.362 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:14:17.363 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:14:17.363 DEBUG:boto:Method: HEAD 2023-10-18T20:14:17.364 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:14:17.364 DEBUG:boto:Data: 2023-10-18T20:14:17.364 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.365 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:14:17.365 DEBUG:boto:Port: 8000 2023-10-18T20:14:17.365 DEBUG:boto:Params: {} 2023-10-18T20:14:17.366 DEBUG:boto:Token: None 2023-10-18T20:14:17.366 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-46/ 2023-10-18T20:14:17.367 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWFUvK73B+1/2SgjlrQFzGsycoA= 2023-10-18T20:14:17.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWFUvK73B+1/2SgjlrQFzGsycoA=', 'Content-Length': '0'} 2023-10-18T20:14:17.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c20f3ac2637b999e-0065303c99-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.371 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:14:17.372 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:14:17.372 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.373 DEBUG:boto:Path: /bofrnf-46/obj2 2023-10-18T20:14:17.373 DEBUG:boto:Data: 2023-10-18T20:14:17.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:14:17.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:14:17.375 DEBUG:boto:Port: 8000 2023-10-18T20:14:17.375 DEBUG:boto:Params: {} 2023-10-18T20:14:17.377 DEBUG:boto:Token: None 2023-10-18T20:14:17.377 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-46/obj2 2023-10-18T20:14:17.378 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XMjw+zoh17xMaXPI67Cux75iQ5Y= 2023-10-18T20:14:17.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XMjw+zoh17xMaXPI67Cux75iQ5Y='} 2023-10-18T20:14:17.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c3bb3f6d65b21a9-0065303c99-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.384 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:14:17.384 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:14:17.385 DEBUG:boto:Method: HEAD 2023-10-18T20:14:17.385 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:14:17.385 DEBUG:boto:Data: 2023-10-18T20:14:17.386 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:14:17.386 DEBUG:boto:Port: 8000 2023-10-18T20:14:17.387 DEBUG:boto:Params: {} 2023-10-18T20:14:17.387 DEBUG:boto:Token: None 2023-10-18T20:14:17.387 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-46/ 2023-10-18T20:14:17.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWFUvK73B+1/2SgjlrQFzGsycoA= 2023-10-18T20:14:17.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWFUvK73B+1/2SgjlrQFzGsycoA=', 'Content-Length': '0'} 2023-10-18T20:14:17.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026fc091d3d8524aa-0065303c99-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.391 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:14:17.392 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:14:17.392 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.393 DEBUG:boto:Path: /bofrnf-46/obj3 2023-10-18T20:14:17.393 DEBUG:boto:Data: 2023-10-18T20:14:17.393 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:14:17.394 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:14:17.394 DEBUG:boto:Port: 8000 2023-10-18T20:14:17.395 DEBUG:boto:Params: {} 2023-10-18T20:14:17.395 DEBUG:boto:Token: None 2023-10-18T20:14:17.396 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-46/obj3 2023-10-18T20:14:17.396 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5z+yFuhAIwqfXmGPX1Q4ZRwpmHY= 2023-10-18T20:14:17.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5z+yFuhAIwqfXmGPX1Q4ZRwpmHY='} 2023-10-18T20:14:17.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce6cc1f5d2b45756-0065303c99-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.443 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:14:17.443 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:14:17.444 DEBUG:boto:Method: HEAD 2023-10-18T20:14:17.444 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:14:17.445 DEBUG:boto:Data: 2023-10-18T20:14:17.445 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:14:17.446 DEBUG:boto:Port: 8000 2023-10-18T20:14:17.446 DEBUG:boto:Params: {} 2023-10-18T20:14:17.447 DEBUG:boto:Token: None 2023-10-18T20:14:17.447 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-46/ 2023-10-18T20:14:17.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWFUvK73B+1/2SgjlrQFzGsycoA= 2023-10-18T20:14:17.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWFUvK73B+1/2SgjlrQFzGsycoA=', 'Content-Length': '0'} 2023-10-18T20:14:17.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a739139f8b8d0352-0065303c99-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.451 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:14:17.451 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:14:17.452 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.452 DEBUG:boto:Path: /bofrnf-46/obj4 2023-10-18T20:14:17.452 DEBUG:boto:Data: 2023-10-18T20:14:17.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:14:17.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:14:17.454 DEBUG:boto:Port: 8000 2023-10-18T20:14:17.454 DEBUG:boto:Params: {} 2023-10-18T20:14:17.455 DEBUG:boto:Token: None 2023-10-18T20:14:17.455 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-46/obj4 2023-10-18T20:14:17.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Glk7Sv4ui/NBWNsvRVJmLMjXIBs= 2023-10-18T20:14:17.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Glk7Sv4ui/NBWNsvRVJmLMjXIBs='} 2023-10-18T20:14:17.461 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fb750e85965b512-0065303c99-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.461 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:14:17.462 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:14:17.462 DEBUG:boto:Method: HEAD 2023-10-18T20:14:17.462 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:14:17.463 DEBUG:boto:Data: 2023-10-18T20:14:17.463 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.464 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:14:17.464 DEBUG:boto:Port: 8001 2023-10-18T20:14:17.465 DEBUG:boto:Params: {} 2023-10-18T20:14:17.465 DEBUG:boto:Token: None 2023-10-18T20:14:17.465 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-47/ 2023-10-18T20:14:17.466 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YAMYEbNN893AfgQOA43CLaL130U= 2023-10-18T20:14:17.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YAMYEbNN893AfgQOA43CLaL130U=', 'Content-Length': '0'} 2023-10-18T20:14:17.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011f5ddc0818bef37-0065303c99-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.469 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:14:17.469 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:14:17.470 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.470 DEBUG:boto:Path: /bofrnf-47/obj1 2023-10-18T20:14:17.471 DEBUG:boto:Data: 2023-10-18T20:14:17.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:14:17.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:14:17.472 DEBUG:boto:Port: 8001 2023-10-18T20:14:17.472 DEBUG:boto:Params: {} 2023-10-18T20:14:17.473 DEBUG:boto:Token: None 2023-10-18T20:14:17.473 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-47/obj1 2023-10-18T20:14:17.473 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bU96UtwzD+ZNJ608Bi59cSJaVMY= 2023-10-18T20:14:17.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bU96UtwzD+ZNJ608Bi59cSJaVMY='} 2023-10-18T20:14:17.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005070cd6d79bbc342-0065303c99-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.481 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:14:17.481 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:14:17.481 DEBUG:boto:Method: HEAD 2023-10-18T20:14:17.482 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:14:17.482 DEBUG:boto:Data: 2023-10-18T20:14:17.482 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.483 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:14:17.483 DEBUG:boto:Port: 8001 2023-10-18T20:14:17.484 DEBUG:boto:Params: {} 2023-10-18T20:14:17.484 DEBUG:boto:Token: None 2023-10-18T20:14:17.485 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-47/ 2023-10-18T20:14:17.485 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YAMYEbNN893AfgQOA43CLaL130U= 2023-10-18T20:14:17.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YAMYEbNN893AfgQOA43CLaL130U=', 'Content-Length': '0'} 2023-10-18T20:14:17.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c4295089d82bc2e-0065303c99-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.488 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:14:17.489 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:14:17.489 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.490 DEBUG:boto:Path: /bofrnf-47/obj2 2023-10-18T20:14:17.490 DEBUG:boto:Data: 2023-10-18T20:14:17.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:14:17.492 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:14:17.492 DEBUG:boto:Port: 8001 2023-10-18T20:14:17.493 DEBUG:boto:Params: {} 2023-10-18T20:14:17.493 DEBUG:boto:Token: None 2023-10-18T20:14:17.494 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-47/obj2 2023-10-18T20:14:17.494 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mjd+AdprY8yWbEa+XaWdgDYttGQ= 2023-10-18T20:14:17.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mjd+AdprY8yWbEa+XaWdgDYttGQ='} 2023-10-18T20:14:17.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009495fd9e3425cf45-0065303c99-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.505 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:14:17.506 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:14:17.506 DEBUG:boto:Method: HEAD 2023-10-18T20:14:17.507 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:14:17.507 DEBUG:boto:Data: 2023-10-18T20:14:17.507 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.508 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:14:17.508 DEBUG:boto:Port: 8001 2023-10-18T20:14:17.508 DEBUG:boto:Params: {} 2023-10-18T20:14:17.509 DEBUG:boto:Token: None 2023-10-18T20:14:17.509 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-47/ 2023-10-18T20:14:17.510 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YAMYEbNN893AfgQOA43CLaL130U= 2023-10-18T20:14:17.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YAMYEbNN893AfgQOA43CLaL130U=', 'Content-Length': '0'} 2023-10-18T20:14:17.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072ccedd80cf4066f-0065303c99-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.513 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:14:17.514 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:14:17.514 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.515 DEBUG:boto:Path: /bofrnf-47/obj3 2023-10-18T20:14:17.515 DEBUG:boto:Data: 2023-10-18T20:14:17.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:14:17.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:14:17.516 DEBUG:boto:Port: 8001 2023-10-18T20:14:17.517 DEBUG:boto:Params: {} 2023-10-18T20:14:17.517 DEBUG:boto:Token: None 2023-10-18T20:14:17.518 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-47/obj3 2023-10-18T20:14:17.518 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UXvcySBk/IALpJG/16aKt9ZtU0A= 2023-10-18T20:14:17.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UXvcySBk/IALpJG/16aKt9ZtU0A='} 2023-10-18T20:14:17.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b5f34ab017907be-0065303c99-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.524 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:14:17.524 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:14:17.525 DEBUG:boto:Method: HEAD 2023-10-18T20:14:17.525 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:14:17.526 DEBUG:boto:Data: 2023-10-18T20:14:17.526 DEBUG:boto:Headers: {} 2023-10-18T20:14:17.527 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:14:17.527 DEBUG:boto:Port: 8001 2023-10-18T20:14:17.527 DEBUG:boto:Params: {} 2023-10-18T20:14:17.528 DEBUG:boto:Token: None 2023-10-18T20:14:17.528 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-47/ 2023-10-18T20:14:17.528 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YAMYEbNN893AfgQOA43CLaL130U= 2023-10-18T20:14:17.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YAMYEbNN893AfgQOA43CLaL130U=', 'Content-Length': '0'} 2023-10-18T20:14:17.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b35c9dd53d8ca562-0065303c99-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.531 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:14:17.532 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:14:17.532 DEBUG:boto:Method: PUT 2023-10-18T20:14:17.533 DEBUG:boto:Path: /bofrnf-47/obj4 2023-10-18T20:14:17.533 DEBUG:boto:Data: 2023-10-18T20:14:17.534 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:14:17.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:14:17.535 DEBUG:boto:Port: 8001 2023-10-18T20:14:17.535 DEBUG:boto:Params: {} 2023-10-18T20:14:17.535 DEBUG:boto:Token: None 2023-10-18T20:14:17.536 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:14:17 GMT /bofrnf-47/obj4 2023-10-18T20:14:17.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QLWTMsjeWBTRMeE6rA6T+yBvX+g= 2023-10-18T20:14:17.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:14:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QLWTMsjeWBTRMeE6rA6T+yBvX+g='} 2023-10-18T20:14:17.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b89886ab69248f4-0065303c99-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:14:17.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:17.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:17.580 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:17.628 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:17.628 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:17.656 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:14:17.668 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:14:17.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.668 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660057.331818_676.1", 2023-10-18T20:14:17.669 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:14:17.331818Z" 2023-10-18T20:14:17.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.669 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:14:17.669 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:14:17.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.670 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:17.671 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:14:17.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.671 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:17.671 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:14:17.672 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.672 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:14:17.673 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660057.331818_676.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:17.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:14:17.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:17.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:17.675 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:17.724 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:17.724 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:17.751 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:17.750+0000 7f44075ea780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:17.752 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:17.751+0000 7f44075ea780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:17.764 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:14:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:14:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:14:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:14:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:14:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:14:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:14:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:14:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697659979.175406_651.1", 2023-10-18T20:14:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:14:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:12:59.175406Z", 2023-10-18T20:14:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:14:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:14:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:14:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:14:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:14:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:14:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:14:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:14:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:14:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:14:17Z" 2023-10-18T20:14:17.777 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:14:17.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697659979.175406_651.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:12:59.175406Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:14:17Z" } 2023-10-18T20:14:17.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697659979.175406_651.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:12:59.175406Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:14:17.779 DEBUG:rgw_multi.tests:log_status={0: '1_1697660057.331818_676.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:17.779 DEBUG:rgw_multi.tests:sync_status={0: '1_1697659979.175406_651.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:17.780 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697660057.331818_676.1 target=1_1697659979.175406_651.1 2023-10-18T20:14:22.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:22.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:14:22.782 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:14:22.839 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:22.839 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:22.870 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:22.869+0000 7f5c6be14780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:22.871 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:22.870+0000 7f5c6be14780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:14:22.883 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:14:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:14:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:14:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:14:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:14:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:14:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:14:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:14:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660057.331818_676.1", 2023-10-18T20:14:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:14:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:14:17.331818Z", 2023-10-18T20:14:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:14:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:14:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:14:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:14:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:14:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:14:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:14:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:14:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:14:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:14:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:14:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:14:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:14:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:14:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:14:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:14:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:14:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:14:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:14:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:14:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:14:22Z" 2023-10-18T20:14:22.897 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:14:22.898 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660057.331818_676.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:14:17.331818Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:14:22Z" } 2023-10-18T20:14:22.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660057.331818_676.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:14:17.331818Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:14:22.899 DEBUG:rgw_multi.tests:log_status={0: '1_1697660057.331818_676.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:22.899 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660057.331818_676.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:14:22.899 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:14:22.900 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-46 2023-10-18T20:14:22.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:14:22.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:14:22.901 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:14:22.958 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:14:22.958 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:14:22.990 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:22.989+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:14:27.991 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:27.990+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:14:32.992 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:32.990+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:14:35.720 INFO:tasks.rgw.c1.client.0.smithi082.stdout:2023-10-18T20:14:35.718+0000 7f838d952700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-18T20:14:37.992 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:37.991+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:14:42.993 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:42.991+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:14:47.994 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:47.992+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:14:52.994 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:52.992+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:14:57.995 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:14:57.993+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:15:02.996 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:02.993+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:15:07.996 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:07.995+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:15:12.997 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:12.996+0000 7fa64c9c4780 1 waiting to reach incremental sync.. 2023-10-18T20:15:17.999 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:17.997+0000 7fa64c9c4780 1 bucket sync caught up with source: 2023-10-18T20:15:17.999 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.17.6, , 00000000001.26.6, , 00000000001.20.6, , , 00000000001.20.6, , , ] 2023-10-18T20:15:18.000 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.17.6, , 00000000001.26.6, , 00000000001.20.6, , , 00000000001.20.6, , , ] 2023-10-18T20:15:18.000 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:17.997+0000 7fa64c9c4780 0 bucket checkpoint complete 2023-10-18T20:15:18.015 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-46 2023-10-18T20:15:18.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:15:18.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:15:18.017 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:15:18.079 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:15:18.080 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:15:18.117 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:18.115+0000 7f0883de1780 1 waiting for incremental sync to catch up: 2023-10-18T20:15:18.117 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.411.6, , , , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:18.117 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.411.6, , 00000000001.909.6, , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:23.118 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:23.116+0000 7f0883de1780 1 waiting for incremental sync to catch up: 2023-10-18T20:15:23.119 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.411.6, , , , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:23.119 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.411.6, , 00000000001.909.6, , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:28.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:28.118+0000 7f0883de1780 1 waiting for incremental sync to catch up: 2023-10-18T20:15:28.120 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.411.6, , , , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:28.120 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.411.6, , 00000000001.909.6, , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:33.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:33.119+0000 7f0883de1780 1 waiting for incremental sync to catch up: 2023-10-18T20:15:33.121 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.411.6, , , , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:33.121 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.411.6, , 00000000001.909.6, , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:38.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:38.120+0000 7f0883de1780 1 bucket sync caught up with source: 2023-10-18T20:15:38.122 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.411.6, , 00000000001.909.6, , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:38.122 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.411.6, , 00000000001.909.6, , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:15:38.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:38.120+0000 7f0883de1780 0 bucket checkpoint complete 2023-10-18T20:15:38.137 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-46 zones={a2, a1} 2023-10-18T20:15:38.138 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.139 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.139 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.139 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:15:38.140 DEBUG:boto:Data: 2023-10-18T20:15:38.140 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.140 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.141 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.141 DEBUG:boto:Params: {} 2023-10-18T20:15:38.142 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:15:38.143 DEBUG:boto:Token: None 2023-10-18T20:15:38.143 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9ctRCoAyZBF2Xhl6oxNMP2pu0Vc= 2023-10-18T20:15:38.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9ctRCoAyZBF2Xhl6oxNMP2pu0Vc=', 'Content-Length': '0'} 2023-10-18T20:15:38.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013dab16225fcc257-0065303cea-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.149 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.149 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.150 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.150 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:15:38.151 DEBUG:boto:Data: 2023-10-18T20:15:38.151 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.152 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.152 DEBUG:boto:Params: {} 2023-10-18T20:15:38.153 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:15:38.153 DEBUG:boto:Token: None 2023-10-18T20:15:38.154 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.154 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9ctRCoAyZBF2Xhl6oxNMP2pu0Vc= 2023-10-18T20:15:38.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9ctRCoAyZBF2Xhl6oxNMP2pu0Vc=', 'Content-Length': '0'} 2023-10-18T20:15:38.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006964eec31d5c81b7-0065303cea-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:15:38.159 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.160 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.160 DEBUG:boto:path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.161 DEBUG:boto:auth_path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.161 DEBUG:boto:Method: GET 2023-10-18T20:15:38.161 DEBUG:boto:Path: /bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.162 DEBUG:boto:Data: 2023-10-18T20:15:38.162 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.163 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.163 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.164 DEBUG:boto:Params: {} 2023-10-18T20:15:38.164 DEBUG:boto:Token: None 2023-10-18T20:15:38.165 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/?versions 2023-10-18T20:15:38.165 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/lKG0ku0UK+nCG15r9TjWLOzF60= 2023-10-18T20:15:38.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/lKG0ku0UK+nCG15r9TjWLOzF60=', 'Content-Length': '0'} 2023-10-18T20:15:38.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bd8bf4396878448-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.169 DEBUG:boto:b'bofrnf-46999falseobj1nulltrue2023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:15:38.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:15:38.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:15:38.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:15:38.171 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:15:38.172 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.172 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.172 DEBUG:boto:path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.173 DEBUG:boto:auth_path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.173 DEBUG:boto:Method: GET 2023-10-18T20:15:38.174 DEBUG:boto:Path: /bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.174 DEBUG:boto:Data: 2023-10-18T20:15:38.174 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.175 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.175 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.176 DEBUG:boto:Params: {} 2023-10-18T20:15:38.176 DEBUG:boto:Token: None 2023-10-18T20:15:38.176 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/?versions 2023-10-18T20:15:38.177 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/lKG0ku0UK+nCG15r9TjWLOzF60= 2023-10-18T20:15:38.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/lKG0ku0UK+nCG15r9TjWLOzF60=', 'Content-Length': '0'} 2023-10-18T20:15:38.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008337ab1f6aaaa618-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.180 DEBUG:boto:b'bofrnf-46999falseobj1nulltrue2023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:15:38.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:15:38.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:15:38.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:15:38.183 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.183 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.183 DEBUG:boto:path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.184 DEBUG:boto:auth_path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.184 DEBUG:boto:Method: GET 2023-10-18T20:15:38.185 DEBUG:boto:Path: /bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.185 DEBUG:boto:Data: 2023-10-18T20:15:38.186 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.186 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.186 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.187 DEBUG:boto:Params: {} 2023-10-18T20:15:38.187 DEBUG:boto:Token: None 2023-10-18T20:15:38.188 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/?versions 2023-10-18T20:15:38.188 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/lKG0ku0UK+nCG15r9TjWLOzF60= 2023-10-18T20:15:38.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/lKG0ku0UK+nCG15r9TjWLOzF60=', 'Content-Length': '0'} 2023-10-18T20:15:38.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d53e0c6264496731-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.192 DEBUG:boto:b'bofrnf-46999falseobj1nulltrue2023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.192 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.193 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.193 DEBUG:boto:path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.194 DEBUG:boto:auth_path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.194 DEBUG:boto:Method: GET 2023-10-18T20:15:38.195 DEBUG:boto:Path: /bofrnf-46/?versions&max-keys=999 2023-10-18T20:15:38.195 DEBUG:boto:Data: 2023-10-18T20:15:38.196 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.196 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.196 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.197 DEBUG:boto:Params: {} 2023-10-18T20:15:38.197 DEBUG:boto:Token: None 2023-10-18T20:15:38.198 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/?versions 2023-10-18T20:15:38.198 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/lKG0ku0UK+nCG15r9TjWLOzF60= 2023-10-18T20:15:38.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/lKG0ku0UK+nCG15r9TjWLOzF60=', 'Content-Length': '0'} 2023-10-18T20:15:38.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b645c705e50661e3-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.202 DEBUG:boto:b'bofrnf-46999falseobj1nulltrue2023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:15:38.203 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.203 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.204 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.204 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.205 DEBUG:boto:Method: GET 2023-10-18T20:15:38.205 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.205 DEBUG:boto:Data: 2023-10-18T20:15:38.206 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.206 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.207 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.207 DEBUG:boto:Params: {} 2023-10-18T20:15:38.208 DEBUG:boto:Token: None 2023-10-18T20:15:38.208 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.209 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hONiHN579IsN+o00s6HZmYM12k0= 2023-10-18T20:15:38.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hONiHN579IsN+o00s6HZmYM12k0=', 'Content-Length': '0'} 2023-10-18T20:15:38.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001601acde37f22c0a-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.212 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.212 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.213 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.213 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.214 DEBUG:boto:Method: GET 2023-10-18T20:15:38.214 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.215 DEBUG:boto:Data: 2023-10-18T20:15:38.215 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.216 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.216 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.216 DEBUG:boto:Params: {} 2023-10-18T20:15:38.217 DEBUG:boto:Token: None 2023-10-18T20:15:38.217 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.218 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hONiHN579IsN+o00s6HZmYM12k0= 2023-10-18T20:15:38.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hONiHN579IsN+o00s6HZmYM12k0=', 'Content-Length': '0'} 2023-10-18T20:15:38.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b62710c0a4f9845-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.222 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.222 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.223 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.223 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.224 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.224 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.225 DEBUG:boto:Data: 2023-10-18T20:15:38.225 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.226 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.227 DEBUG:boto:Params: {} 2023-10-18T20:15:38.227 DEBUG:boto:Token: None 2023-10-18T20:15:38.228 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.228 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o5wFkow2JdLA4ZiO7MSf3IHPpGA= 2023-10-18T20:15:38.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o5wFkow2JdLA4ZiO7MSf3IHPpGA=', 'Content-Length': '0'} 2023-10-18T20:15:38.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd7fc14b26da51e4-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.231 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.232 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.232 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.232 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.233 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.233 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.234 DEBUG:boto:Data: 2023-10-18T20:15:38.234 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.235 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.236 DEBUG:boto:Params: {} 2023-10-18T20:15:38.236 DEBUG:boto:Token: None 2023-10-18T20:15:38.237 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o5wFkow2JdLA4ZiO7MSf3IHPpGA= 2023-10-18T20:15:38.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o5wFkow2JdLA4ZiO7MSf3IHPpGA=', 'Content-Length': '0'} 2023-10-18T20:15:38.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e038d257e4134beb-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:15:38.241 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.241 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.242 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.242 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.243 DEBUG:boto:Method: GET 2023-10-18T20:15:38.243 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.244 DEBUG:boto:Data: 2023-10-18T20:15:38.244 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.245 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.245 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.245 DEBUG:boto:Params: {} 2023-10-18T20:15:38.246 DEBUG:boto:Token: None 2023-10-18T20:15:38.246 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.247 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hONiHN579IsN+o00s6HZmYM12k0= 2023-10-18T20:15:38.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hONiHN579IsN+o00s6HZmYM12k0=', 'Content-Length': '0'} 2023-10-18T20:15:38.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096f3e45df05e838d-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.290 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.291 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.291 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.292 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.292 DEBUG:boto:Method: GET 2023-10-18T20:15:38.293 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.293 DEBUG:boto:Data: 2023-10-18T20:15:38.294 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.295 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.295 DEBUG:boto:Params: {} 2023-10-18T20:15:38.296 DEBUG:boto:Token: None 2023-10-18T20:15:38.296 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:15:38.297 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hONiHN579IsN+o00s6HZmYM12k0= 2023-10-18T20:15:38.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hONiHN579IsN+o00s6HZmYM12k0=', 'Content-Length': '0'} 2023-10-18T20:15:38.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dce8758ecf65d02-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.301 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.301 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.302 DEBUG:boto:path=/bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.302 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.303 DEBUG:boto:Method: GET 2023-10-18T20:15:38.303 DEBUG:boto:Path: /bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.303 DEBUG:boto:Data: 2023-10-18T20:15:38.304 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.304 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.305 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.305 DEBUG:boto:Params: {} 2023-10-18T20:15:38.305 DEBUG:boto:Token: None 2023-10-18T20:15:38.306 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.306 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5b0a4aa698d046-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.310 DEBUG:boto:b'bofrnf-46obj11000falseobj12023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.310 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.311 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.311 DEBUG:boto:path=/bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.312 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.312 DEBUG:boto:Method: GET 2023-10-18T20:15:38.313 DEBUG:boto:Path: /bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.313 DEBUG:boto:Data: 2023-10-18T20:15:38.313 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.314 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.315 DEBUG:boto:Params: {} 2023-10-18T20:15:38.315 DEBUG:boto:Token: None 2023-10-18T20:15:38.316 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.316 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2b16526ee0fc8e-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.320 DEBUG:boto:b'bofrnf-46obj11000falseobj12023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.320 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.321 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.321 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.322 DEBUG:boto:Path: /bofrnf-46/obj1 2023-10-18T20:15:38.322 DEBUG:boto:Data: 2023-10-18T20:15:38.323 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.324 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.324 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.325 DEBUG:boto:Params: {} 2023-10-18T20:15:38.326 DEBUG:boto:Token: None 2023-10-18T20:15:38.327 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1 2023-10-18T20:15:38.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:34wXNG3+BEXBbnTLqr4XUNAT/H0= 2023-10-18T20:15:38.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:34wXNG3+BEXBbnTLqr4XUNAT/H0=', 'Content-Length': '0'} 2023-10-18T20:15:38.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4baf8aeeb513cc0-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.332 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.332 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.333 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.333 DEBUG:boto:Path: /bofrnf-46/obj1 2023-10-18T20:15:38.334 DEBUG:boto:Data: 2023-10-18T20:15:38.334 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.335 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.335 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.337 DEBUG:boto:Params: {} 2023-10-18T20:15:38.338 DEBUG:boto:Token: None 2023-10-18T20:15:38.339 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1 2023-10-18T20:15:38.339 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:34wXNG3+BEXBbnTLqr4XUNAT/H0= 2023-10-18T20:15:38.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:34wXNG3+BEXBbnTLqr4XUNAT/H0=', 'Content-Length': '0'} 2023-10-18T20:15:38.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008da15255fdc62bad-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:15:38.343 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.343 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.344 DEBUG:boto:Method: GET 2023-10-18T20:15:38.344 DEBUG:boto:Path: /bofrnf-46/obj1 2023-10-18T20:15:38.345 DEBUG:boto:Data: 2023-10-18T20:15:38.345 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.346 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.346 DEBUG:boto:Params: {} 2023-10-18T20:15:38.347 DEBUG:boto:Token: None 2023-10-18T20:15:38.347 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1 2023-10-18T20:15:38.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QkX+fxBmWc3v4aap0w+nt1Bn6Fg= 2023-10-18T20:15:38.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QkX+fxBmWc3v4aap0w+nt1Bn6Fg=', 'Content-Length': '0'} 2023-10-18T20:15:38.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8890cc3f929cdb9-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.351 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:15:38.351 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:15:38.352 DEBUG:boto:Method: GET 2023-10-18T20:15:38.352 DEBUG:boto:Path: /bofrnf-46/obj1 2023-10-18T20:15:38.353 DEBUG:boto:Data: 2023-10-18T20:15:38.353 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.354 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.354 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.354 DEBUG:boto:Params: {} 2023-10-18T20:15:38.355 DEBUG:boto:Token: None 2023-10-18T20:15:38.355 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj1 2023-10-18T20:15:38.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QkX+fxBmWc3v4aap0w+nt1Bn6Fg= 2023-10-18T20:15:38.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QkX+fxBmWc3v4aap0w+nt1Bn6Fg=', 'Content-Length': '0'} 2023-10-18T20:15:38.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee13febe8d26e4e8-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.402 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.403 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.403 DEBUG:boto:path=/bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.404 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.404 DEBUG:boto:Method: GET 2023-10-18T20:15:38.404 DEBUG:boto:Path: /bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.405 DEBUG:boto:Data: 2023-10-18T20:15:38.405 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.406 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.406 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.407 DEBUG:boto:Params: {} 2023-10-18T20:15:38.407 DEBUG:boto:Token: None 2023-10-18T20:15:38.408 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.408 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006160595f73971e95-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.412 DEBUG:boto:b'bofrnf-46obj11000falseobj12023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.412 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.413 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.413 DEBUG:boto:path=/bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.413 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.414 DEBUG:boto:Method: GET 2023-10-18T20:15:38.414 DEBUG:boto:Path: /bofrnf-46/?prefix=obj1 2023-10-18T20:15:38.415 DEBUG:boto:Data: 2023-10-18T20:15:38.415 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.416 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.416 DEBUG:boto:Params: {} 2023-10-18T20:15:38.417 DEBUG:boto:Token: None 2023-10-18T20:15:38.417 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eace66427e0159b5-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.421 DEBUG:boto:b'bofrnf-46obj11000falseobj12023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:15:38.422 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.422 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.423 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.423 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.424 DEBUG:boto:Method: GET 2023-10-18T20:15:38.424 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.424 DEBUG:boto:Data: 2023-10-18T20:15:38.425 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.425 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.426 DEBUG:boto:Params: {} 2023-10-18T20:15:38.426 DEBUG:boto:Token: None 2023-10-18T20:15:38.427 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.427 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EJgO20thTWO1Bw696Z6c535gKKI= 2023-10-18T20:15:38.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EJgO20thTWO1Bw696Z6c535gKKI=', 'Content-Length': '0'} 2023-10-18T20:15:38.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019e83e5ed50d00af-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.470 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.471 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.471 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.472 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.472 DEBUG:boto:Method: GET 2023-10-18T20:15:38.473 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.473 DEBUG:boto:Data: 2023-10-18T20:15:38.473 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.474 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.475 DEBUG:boto:Params: {} 2023-10-18T20:15:38.475 DEBUG:boto:Token: None 2023-10-18T20:15:38.476 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EJgO20thTWO1Bw696Z6c535gKKI= 2023-10-18T20:15:38.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EJgO20thTWO1Bw696Z6c535gKKI=', 'Content-Length': '0'} 2023-10-18T20:15:38.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fed642874e178ab-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.479 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.480 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.480 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.480 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.481 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.481 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.482 DEBUG:boto:Data: 2023-10-18T20:15:38.482 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.483 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.483 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.483 DEBUG:boto:Params: {} 2023-10-18T20:15:38.484 DEBUG:boto:Token: None 2023-10-18T20:15:38.484 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.485 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6MR/SqLQVQQpUa/FafCOquucM1Y= 2023-10-18T20:15:38.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6MR/SqLQVQQpUa/FafCOquucM1Y=', 'Content-Length': '0'} 2023-10-18T20:15:38.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091640648b6234b21-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.488 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.488 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.488 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.489 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.489 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.490 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.490 DEBUG:boto:Data: 2023-10-18T20:15:38.490 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.491 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.491 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.492 DEBUG:boto:Params: {} 2023-10-18T20:15:38.492 DEBUG:boto:Token: None 2023-10-18T20:15:38.493 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.493 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6MR/SqLQVQQpUa/FafCOquucM1Y= 2023-10-18T20:15:38.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6MR/SqLQVQQpUa/FafCOquucM1Y=', 'Content-Length': '0'} 2023-10-18T20:15:38.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b014c61d8789474-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:15:38.496 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.497 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.497 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.497 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.498 DEBUG:boto:Method: GET 2023-10-18T20:15:38.498 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.499 DEBUG:boto:Data: 2023-10-18T20:15:38.499 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.500 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.500 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.501 DEBUG:boto:Params: {} 2023-10-18T20:15:38.501 DEBUG:boto:Token: None 2023-10-18T20:15:38.502 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.502 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EJgO20thTWO1Bw696Z6c535gKKI= 2023-10-18T20:15:38.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EJgO20thTWO1Bw696Z6c535gKKI=', 'Content-Length': '0'} 2023-10-18T20:15:38.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000908b100f716cf2c9-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.505 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.506 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.506 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.507 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.507 DEBUG:boto:Method: GET 2023-10-18T20:15:38.508 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.508 DEBUG:boto:Data: 2023-10-18T20:15:38.508 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.509 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.509 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.510 DEBUG:boto:Params: {} 2023-10-18T20:15:38.510 DEBUG:boto:Token: None 2023-10-18T20:15:38.511 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:15:38.511 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EJgO20thTWO1Bw696Z6c535gKKI= 2023-10-18T20:15:38.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EJgO20thTWO1Bw696Z6c535gKKI=', 'Content-Length': '0'} 2023-10-18T20:15:38.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037ac3cb93add3686-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.515 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.515 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.515 DEBUG:boto:path=/bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.516 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.516 DEBUG:boto:Method: GET 2023-10-18T20:15:38.517 DEBUG:boto:Path: /bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.517 DEBUG:boto:Data: 2023-10-18T20:15:38.518 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.518 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.519 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.519 DEBUG:boto:Params: {} 2023-10-18T20:15:38.519 DEBUG:boto:Token: None 2023-10-18T20:15:38.520 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed3aa5a595f54c5-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.524 DEBUG:boto:b'bofrnf-46obj21000falseobj22023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.524 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.525 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.525 DEBUG:boto:path=/bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.526 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.526 DEBUG:boto:Method: GET 2023-10-18T20:15:38.527 DEBUG:boto:Path: /bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.527 DEBUG:boto:Data: 2023-10-18T20:15:38.527 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.528 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.528 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.529 DEBUG:boto:Params: {} 2023-10-18T20:15:38.529 DEBUG:boto:Token: None 2023-10-18T20:15:38.529 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.530 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6c4b53d7d9eab88-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.533 DEBUG:boto:b'bofrnf-46obj21000falseobj22023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.534 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.535 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.535 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.535 DEBUG:boto:Path: /bofrnf-46/obj2 2023-10-18T20:15:38.536 DEBUG:boto:Data: 2023-10-18T20:15:38.536 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.536 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.537 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.537 DEBUG:boto:Params: {} 2023-10-18T20:15:38.538 DEBUG:boto:Token: None 2023-10-18T20:15:38.538 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2 2023-10-18T20:15:38.539 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d5t9MQIqTdw5rZZo38MICS2gHpI= 2023-10-18T20:15:38.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d5t9MQIqTdw5rZZo38MICS2gHpI=', 'Content-Length': '0'} 2023-10-18T20:15:38.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a3c094aee1cbf4c-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.542 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.542 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.543 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.543 DEBUG:boto:Path: /bofrnf-46/obj2 2023-10-18T20:15:38.544 DEBUG:boto:Data: 2023-10-18T20:15:38.544 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.544 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.545 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.545 DEBUG:boto:Params: {} 2023-10-18T20:15:38.546 DEBUG:boto:Token: None 2023-10-18T20:15:38.546 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2 2023-10-18T20:15:38.547 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d5t9MQIqTdw5rZZo38MICS2gHpI= 2023-10-18T20:15:38.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d5t9MQIqTdw5rZZo38MICS2gHpI=', 'Content-Length': '0'} 2023-10-18T20:15:38.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000927541b956894de4-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:15:38.550 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.550 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.551 DEBUG:boto:Method: GET 2023-10-18T20:15:38.551 DEBUG:boto:Path: /bofrnf-46/obj2 2023-10-18T20:15:38.552 DEBUG:boto:Data: 2023-10-18T20:15:38.552 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.553 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.553 DEBUG:boto:Params: {} 2023-10-18T20:15:38.554 DEBUG:boto:Token: None 2023-10-18T20:15:38.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2 2023-10-18T20:15:38.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fXsKEUT12oWvDMaNPRNuhOpVoIQ= 2023-10-18T20:15:38.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fXsKEUT12oWvDMaNPRNuhOpVoIQ=', 'Content-Length': '0'} 2023-10-18T20:15:38.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab8f07c7c8561354-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.598 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:15:38.599 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:15:38.599 DEBUG:boto:Method: GET 2023-10-18T20:15:38.600 DEBUG:boto:Path: /bofrnf-46/obj2 2023-10-18T20:15:38.600 DEBUG:boto:Data: 2023-10-18T20:15:38.601 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.601 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.602 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.602 DEBUG:boto:Params: {} 2023-10-18T20:15:38.602 DEBUG:boto:Token: None 2023-10-18T20:15:38.603 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj2 2023-10-18T20:15:38.603 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fXsKEUT12oWvDMaNPRNuhOpVoIQ= 2023-10-18T20:15:38.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fXsKEUT12oWvDMaNPRNuhOpVoIQ=', 'Content-Length': '0'} 2023-10-18T20:15:38.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000698fa07e82a6d496-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.607 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.607 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.608 DEBUG:boto:path=/bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.608 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.609 DEBUG:boto:Method: GET 2023-10-18T20:15:38.609 DEBUG:boto:Path: /bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.609 DEBUG:boto:Data: 2023-10-18T20:15:38.610 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.610 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.611 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.611 DEBUG:boto:Params: {} 2023-10-18T20:15:38.611 DEBUG:boto:Token: None 2023-10-18T20:15:38.612 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.612 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081e948a46d9d3c33-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.616 DEBUG:boto:b'bofrnf-46obj21000falseobj22023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.617 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.617 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.618 DEBUG:boto:path=/bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.618 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.618 DEBUG:boto:Method: GET 2023-10-18T20:15:38.619 DEBUG:boto:Path: /bofrnf-46/?prefix=obj2 2023-10-18T20:15:38.619 DEBUG:boto:Data: 2023-10-18T20:15:38.620 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.620 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.621 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.621 DEBUG:boto:Params: {} 2023-10-18T20:15:38.621 DEBUG:boto:Token: None 2023-10-18T20:15:38.622 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.622 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004873c5a12f99d473-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.666 DEBUG:boto:b'bofrnf-46obj21000falseobj22023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:15:38.668 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.668 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.669 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.669 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.669 DEBUG:boto:Method: GET 2023-10-18T20:15:38.670 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.670 DEBUG:boto:Data: 2023-10-18T20:15:38.671 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.671 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.672 DEBUG:boto:Params: {} 2023-10-18T20:15:38.672 DEBUG:boto:Token: None 2023-10-18T20:15:38.673 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZLZRAWuEgD7NKYCGifuq5Icvfrg= 2023-10-18T20:15:38.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZLZRAWuEgD7NKYCGifuq5Icvfrg=', 'Content-Length': '0'} 2023-10-18T20:15:38.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009df91da92432bd14-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.677 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.677 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.678 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.678 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.679 DEBUG:boto:Method: GET 2023-10-18T20:15:38.679 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.679 DEBUG:boto:Data: 2023-10-18T20:15:38.680 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.680 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.681 DEBUG:boto:Params: {} 2023-10-18T20:15:38.681 DEBUG:boto:Token: None 2023-10-18T20:15:38.682 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.682 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZLZRAWuEgD7NKYCGifuq5Icvfrg= 2023-10-18T20:15:38.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZLZRAWuEgD7NKYCGifuq5Icvfrg=', 'Content-Length': '0'} 2023-10-18T20:15:38.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054013041640ad432-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.685 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.686 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.686 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.686 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.687 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.687 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.687 DEBUG:boto:Data: 2023-10-18T20:15:38.688 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.688 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.689 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.689 DEBUG:boto:Params: {} 2023-10-18T20:15:38.689 DEBUG:boto:Token: None 2023-10-18T20:15:38.690 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.690 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C2RWPbU1J5CbUOjJVKFf9hKzX+I= 2023-10-18T20:15:38.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C2RWPbU1J5CbUOjJVKFf9hKzX+I=', 'Content-Length': '0'} 2023-10-18T20:15:38.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b210c67c0e6d97ae-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.693 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.693 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.694 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.694 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.695 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.695 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.695 DEBUG:boto:Data: 2023-10-18T20:15:38.696 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.696 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.696 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.697 DEBUG:boto:Params: {} 2023-10-18T20:15:38.697 DEBUG:boto:Token: None 2023-10-18T20:15:38.698 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.698 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C2RWPbU1J5CbUOjJVKFf9hKzX+I= 2023-10-18T20:15:38.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C2RWPbU1J5CbUOjJVKFf9hKzX+I=', 'Content-Length': '0'} 2023-10-18T20:15:38.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3fd19862b3c4c41-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:15:38.701 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.702 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.702 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.702 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.703 DEBUG:boto:Method: GET 2023-10-18T20:15:38.703 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.704 DEBUG:boto:Data: 2023-10-18T20:15:38.704 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.704 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.705 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.705 DEBUG:boto:Params: {} 2023-10-18T20:15:38.706 DEBUG:boto:Token: None 2023-10-18T20:15:38.706 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.706 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZLZRAWuEgD7NKYCGifuq5Icvfrg= 2023-10-18T20:15:38.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZLZRAWuEgD7NKYCGifuq5Icvfrg=', 'Content-Length': '0'} 2023-10-18T20:15:38.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d1d010261e743cb-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.709 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.710 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.710 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.711 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.711 DEBUG:boto:Method: GET 2023-10-18T20:15:38.711 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.712 DEBUG:boto:Data: 2023-10-18T20:15:38.712 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.713 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.713 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.713 DEBUG:boto:Params: {} 2023-10-18T20:15:38.714 DEBUG:boto:Token: None 2023-10-18T20:15:38.714 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:15:38.715 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZLZRAWuEgD7NKYCGifuq5Icvfrg= 2023-10-18T20:15:38.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZLZRAWuEgD7NKYCGifuq5Icvfrg=', 'Content-Length': '0'} 2023-10-18T20:15:38.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041ba2612c14bc2dc-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.718 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.719 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.719 DEBUG:boto:path=/bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.719 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.720 DEBUG:boto:Method: GET 2023-10-18T20:15:38.720 DEBUG:boto:Path: /bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.721 DEBUG:boto:Data: 2023-10-18T20:15:38.721 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.721 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.722 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.722 DEBUG:boto:Params: {} 2023-10-18T20:15:38.723 DEBUG:boto:Token: None 2023-10-18T20:15:38.723 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.724 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae229dd68cc9234f-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.727 DEBUG:boto:b'bofrnf-46obj31000falseobj32023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.728 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.728 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.729 DEBUG:boto:path=/bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.729 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.730 DEBUG:boto:Method: GET 2023-10-18T20:15:38.730 DEBUG:boto:Path: /bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.730 DEBUG:boto:Data: 2023-10-18T20:15:38.731 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.731 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.732 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.732 DEBUG:boto:Params: {} 2023-10-18T20:15:38.732 DEBUG:boto:Token: None 2023-10-18T20:15:38.733 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.733 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b09d9e55b59ca39-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.737 DEBUG:boto:b'bofrnf-46obj31000falseobj32023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.738 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.738 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.738 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.739 DEBUG:boto:Path: /bofrnf-46/obj3 2023-10-18T20:15:38.739 DEBUG:boto:Data: 2023-10-18T20:15:38.740 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.740 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.741 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.741 DEBUG:boto:Params: {} 2023-10-18T20:15:38.741 DEBUG:boto:Token: None 2023-10-18T20:15:38.742 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3 2023-10-18T20:15:38.742 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx46NSuQwUbY7P5k5anfqPkzmr8= 2023-10-18T20:15:38.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx46NSuQwUbY7P5k5anfqPkzmr8=', 'Content-Length': '0'} 2023-10-18T20:15:38.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec1b53a9bbb9afd0-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.745 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.746 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.746 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.746 DEBUG:boto:Path: /bofrnf-46/obj3 2023-10-18T20:15:38.747 DEBUG:boto:Data: 2023-10-18T20:15:38.747 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.748 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.748 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.748 DEBUG:boto:Params: {} 2023-10-18T20:15:38.749 DEBUG:boto:Token: None 2023-10-18T20:15:38.749 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3 2023-10-18T20:15:38.750 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx46NSuQwUbY7P5k5anfqPkzmr8= 2023-10-18T20:15:38.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx46NSuQwUbY7P5k5anfqPkzmr8=', 'Content-Length': '0'} 2023-10-18T20:15:38.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e8edd8a30aaad00-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:15:38.753 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.754 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.754 DEBUG:boto:Method: GET 2023-10-18T20:15:38.754 DEBUG:boto:Path: /bofrnf-46/obj3 2023-10-18T20:15:38.755 DEBUG:boto:Data: 2023-10-18T20:15:38.755 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.756 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.756 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.756 DEBUG:boto:Params: {} 2023-10-18T20:15:38.757 DEBUG:boto:Token: None 2023-10-18T20:15:38.757 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3 2023-10-18T20:15:38.758 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3ito8uK42yroxrfYKn4LT3L6gCI= 2023-10-18T20:15:38.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3ito8uK42yroxrfYKn4LT3L6gCI=', 'Content-Length': '0'} 2023-10-18T20:15:38.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000914689f596fee205-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.761 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:15:38.761 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:15:38.762 DEBUG:boto:Method: GET 2023-10-18T20:15:38.762 DEBUG:boto:Path: /bofrnf-46/obj3 2023-10-18T20:15:38.763 DEBUG:boto:Data: 2023-10-18T20:15:38.763 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.764 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.764 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.764 DEBUG:boto:Params: {} 2023-10-18T20:15:38.765 DEBUG:boto:Token: None 2023-10-18T20:15:38.765 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj3 2023-10-18T20:15:38.766 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3ito8uK42yroxrfYKn4LT3L6gCI= 2023-10-18T20:15:38.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3ito8uK42yroxrfYKn4LT3L6gCI=', 'Content-Length': '0'} 2023-10-18T20:15:38.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ec644219a2ee21c-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.769 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.770 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.770 DEBUG:boto:path=/bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.771 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.771 DEBUG:boto:Method: GET 2023-10-18T20:15:38.771 DEBUG:boto:Path: /bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.772 DEBUG:boto:Data: 2023-10-18T20:15:38.772 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.772 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.773 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.773 DEBUG:boto:Params: {} 2023-10-18T20:15:38.774 DEBUG:boto:Token: None 2023-10-18T20:15:38.774 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.775 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6434e8280a430f4-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.778 DEBUG:boto:b'bofrnf-46obj31000falseobj32023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.779 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.779 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.779 DEBUG:boto:path=/bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.780 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.780 DEBUG:boto:Method: GET 2023-10-18T20:15:38.781 DEBUG:boto:Path: /bofrnf-46/?prefix=obj3 2023-10-18T20:15:38.781 DEBUG:boto:Data: 2023-10-18T20:15:38.781 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.782 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.782 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.782 DEBUG:boto:Params: {} 2023-10-18T20:15:38.783 DEBUG:boto:Token: None 2023-10-18T20:15:38.783 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.784 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bc17fa96a44fe64-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.787 DEBUG:boto:b'bofrnf-46obj31000falseobj32023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:15:38.788 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.789 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.789 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.790 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.790 DEBUG:boto:Method: GET 2023-10-18T20:15:38.791 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.791 DEBUG:boto:Data: 2023-10-18T20:15:38.791 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.792 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.792 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.793 DEBUG:boto:Params: {} 2023-10-18T20:15:38.793 DEBUG:boto:Token: None 2023-10-18T20:15:38.794 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.794 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b1923BE6kTxBCikF7h7V0pH+JTA= 2023-10-18T20:15:38.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b1923BE6kTxBCikF7h7V0pH+JTA=', 'Content-Length': '0'} 2023-10-18T20:15:38.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000680e71a4f855fbac-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.798 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.798 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.799 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.799 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.800 DEBUG:boto:Method: GET 2023-10-18T20:15:38.800 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.801 DEBUG:boto:Data: 2023-10-18T20:15:38.801 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.801 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.802 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.802 DEBUG:boto:Params: {} 2023-10-18T20:15:38.803 DEBUG:boto:Token: None 2023-10-18T20:15:38.803 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.804 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b1923BE6kTxBCikF7h7V0pH+JTA= 2023-10-18T20:15:38.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b1923BE6kTxBCikF7h7V0pH+JTA=', 'Content-Length': '0'} 2023-10-18T20:15:38.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d2dea202e16178a-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.807 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.808 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.808 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.808 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.809 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.809 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.810 DEBUG:boto:Data: 2023-10-18T20:15:38.810 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.811 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.811 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.812 DEBUG:boto:Params: {} 2023-10-18T20:15:38.812 DEBUG:boto:Token: None 2023-10-18T20:15:38.813 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.813 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6eNY8fW7eztgLIl2X4YChjqSKA= 2023-10-18T20:15:38.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6eNY8fW7eztgLIl2X4YChjqSKA=', 'Content-Length': '0'} 2023-10-18T20:15:38.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000888c2a5f32353410-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.816 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.817 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.817 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.818 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.818 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.819 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.819 DEBUG:boto:Data: 2023-10-18T20:15:38.819 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.820 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.820 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.821 DEBUG:boto:Params: {} 2023-10-18T20:15:38.821 DEBUG:boto:Token: None 2023-10-18T20:15:38.822 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.822 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6eNY8fW7eztgLIl2X4YChjqSKA= 2023-10-18T20:15:38.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6eNY8fW7eztgLIl2X4YChjqSKA=', 'Content-Length': '0'} 2023-10-18T20:15:38.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f401c2756f2c8cf-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:15:38.826 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.826 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.826 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.827 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.827 DEBUG:boto:Method: GET 2023-10-18T20:15:38.828 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.828 DEBUG:boto:Data: 2023-10-18T20:15:38.829 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.829 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.829 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.830 DEBUG:boto:Params: {} 2023-10-18T20:15:38.830 DEBUG:boto:Token: None 2023-10-18T20:15:38.831 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.831 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b1923BE6kTxBCikF7h7V0pH+JTA= 2023-10-18T20:15:38.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b1923BE6kTxBCikF7h7V0pH+JTA=', 'Content-Length': '0'} 2023-10-18T20:15:38.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000776edb24c0310a72-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.834 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.835 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.835 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.836 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.836 DEBUG:boto:Method: GET 2023-10-18T20:15:38.836 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.837 DEBUG:boto:Data: 2023-10-18T20:15:38.837 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.838 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.839 DEBUG:boto:Params: {} 2023-10-18T20:15:38.839 DEBUG:boto:Token: None 2023-10-18T20:15:38.839 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:15:38.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b1923BE6kTxBCikF7h7V0pH+JTA= 2023-10-18T20:15:38.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b1923BE6kTxBCikF7h7V0pH+JTA=', 'Content-Length': '0'} 2023-10-18T20:15:38.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9e8cb2efd47282e-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.886 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.887 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.887 DEBUG:boto:path=/bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.888 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.888 DEBUG:boto:Method: GET 2023-10-18T20:15:38.888 DEBUG:boto:Path: /bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.889 DEBUG:boto:Data: 2023-10-18T20:15:38.889 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.890 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.890 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.890 DEBUG:boto:Params: {} 2023-10-18T20:15:38.891 DEBUG:boto:Token: None 2023-10-18T20:15:38.891 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.892 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000600e5d901e096665-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.895 DEBUG:boto:b'bofrnf-46obj41000falseobj42023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.896 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.896 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.896 DEBUG:boto:path=/bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.897 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.897 DEBUG:boto:Method: GET 2023-10-18T20:15:38.898 DEBUG:boto:Path: /bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.898 DEBUG:boto:Data: 2023-10-18T20:15:38.898 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.899 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.899 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.900 DEBUG:boto:Params: {} 2023-10-18T20:15:38.900 DEBUG:boto:Token: None 2023-10-18T20:15:38.900 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.901 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86d46fadaec11fb-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.905 DEBUG:boto:b'bofrnf-46obj41000falseobj42023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.905 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.906 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.906 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.907 DEBUG:boto:Path: /bofrnf-46/obj4 2023-10-18T20:15:38.907 DEBUG:boto:Data: 2023-10-18T20:15:38.907 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.908 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.908 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.908 DEBUG:boto:Params: {} 2023-10-18T20:15:38.909 DEBUG:boto:Token: None 2023-10-18T20:15:38.909 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4 2023-10-18T20:15:38.910 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:19AsgeKTGLUnCYAH6VGBtIbPsHA= 2023-10-18T20:15:38.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:19AsgeKTGLUnCYAH6VGBtIbPsHA=', 'Content-Length': '0'} 2023-10-18T20:15:38.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079185b65f7d161f0-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.913 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.913 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.913 DEBUG:boto:Method: HEAD 2023-10-18T20:15:38.914 DEBUG:boto:Path: /bofrnf-46/obj4 2023-10-18T20:15:38.914 DEBUG:boto:Data: 2023-10-18T20:15:38.914 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.915 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.916 DEBUG:boto:Params: {} 2023-10-18T20:15:38.916 DEBUG:boto:Token: None 2023-10-18T20:15:38.917 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4 2023-10-18T20:15:38.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:19AsgeKTGLUnCYAH6VGBtIbPsHA= 2023-10-18T20:15:38.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:19AsgeKTGLUnCYAH6VGBtIbPsHA=', 'Content-Length': '0'} 2023-10-18T20:15:38.919 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002980eeaf79697b19-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:15:38.920 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.921 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.921 DEBUG:boto:Method: GET 2023-10-18T20:15:38.922 DEBUG:boto:Path: /bofrnf-46/obj4 2023-10-18T20:15:38.922 DEBUG:boto:Data: 2023-10-18T20:15:38.922 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.923 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.924 DEBUG:boto:Params: {} 2023-10-18T20:15:38.924 DEBUG:boto:Token: None 2023-10-18T20:15:38.925 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4 2023-10-18T20:15:38.925 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NE3eZuFIqCpIvHm3HDRjGeMpAdo= 2023-10-18T20:15:38.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NE3eZuFIqCpIvHm3HDRjGeMpAdo=', 'Content-Length': '0'} 2023-10-18T20:15:38.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043f8d8307c909bd6-0065303cea-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.974 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:15:38.975 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:15:38.975 DEBUG:boto:Method: GET 2023-10-18T20:15:38.975 DEBUG:boto:Path: /bofrnf-46/obj4 2023-10-18T20:15:38.976 DEBUG:boto:Data: 2023-10-18T20:15:38.976 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.977 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.977 DEBUG:boto:Params: {} 2023-10-18T20:15:38.978 DEBUG:boto:Token: None 2023-10-18T20:15:38.978 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/obj4 2023-10-18T20:15:38.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NE3eZuFIqCpIvHm3HDRjGeMpAdo= 2023-10-18T20:15:38.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NE3eZuFIqCpIvHm3HDRjGeMpAdo=', 'Content-Length': '0'} 2023-10-18T20:15:38.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000caaad2747f15d38c-0065303cea-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.983 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.983 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.984 DEBUG:boto:path=/bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.984 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.984 DEBUG:boto:Method: GET 2023-10-18T20:15:38.985 DEBUG:boto:Path: /bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.985 DEBUG:boto:Data: 2023-10-18T20:15:38.986 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.986 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:15:38.987 DEBUG:boto:Port: 8001 2023-10-18T20:15:38.987 DEBUG:boto:Params: {} 2023-10-18T20:15:38.987 DEBUG:boto:Token: None 2023-10-18T20:15:38.988 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.988 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:38.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f13d614a24b0fc9-0065303cea-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:38.992 DEBUG:boto:b'bofrnf-46obj41000falseobj42023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:38.992 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:15:38.993 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:15:38.993 DEBUG:boto:path=/bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.994 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.994 DEBUG:boto:Method: GET 2023-10-18T20:15:38.994 DEBUG:boto:Path: /bofrnf-46/?prefix=obj4 2023-10-18T20:15:38.995 DEBUG:boto:Data: 2023-10-18T20:15:38.995 DEBUG:boto:Headers: {} 2023-10-18T20:15:38.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:15:38.996 DEBUG:boto:Port: 8000 2023-10-18T20:15:38.996 DEBUG:boto:Params: {} 2023-10-18T20:15:38.997 DEBUG:boto:Token: None 2023-10-18T20:15:38.997 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:15:38 GMT /bofrnf-46/ 2023-10-18T20:15:38.998 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo= 2023-10-18T20:15:38.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:15:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ixgTvWamLnd52HRtkmp800oqmeo=', 'Content-Length': '0'} 2023-10-18T20:15:39.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b78997c27677151-0065303cea-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:15:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:15:39.001 DEBUG:boto:b'bofrnf-46obj41000falseobj42023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:15:39.002 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-46 zones={a2, a1} 2023-10-18T20:15:39.002 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-47 2023-10-18T20:15:39.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:15:39.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:15:39.004 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:15:39.067 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:15:39.068 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:15:39.104 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:39.103+0000 7f71ad69d780 1 bucket sync caught up with source: 2023-10-18T20:15:39.105 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.36.6, , 00000000001.23.6, , 00000000001.24.6, , , 00000000001.27.6, , , ] 2023-10-18T20:15:39.105 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.36.6, , 00000000001.23.6, , 00000000001.24.6, , , 00000000001.27.6, , , ] 2023-10-18T20:15:39.105 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:39.103+0000 7f71ad69d780 0 bucket checkpoint complete 2023-10-18T20:15:39.117 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-47 2023-10-18T20:15:39.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:15:39.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:15:39.119 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:15:39.176 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:15:39.176 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:15:39.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:39.208+0000 7f6d58054780 1 waiting to reach incremental sync.. 2023-10-18T20:15:44.210 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:44.208+0000 7f6d58054780 1 waiting to reach incremental sync.. 2023-10-18T20:15:49.211 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:49.209+0000 7f6d58054780 1 waiting to reach incremental sync.. 2023-10-18T20:15:54.211 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:54.209+0000 7f6d58054780 1 waiting to reach incremental sync.. 2023-10-18T20:15:59.213 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:15:59.210+0000 7f6d58054780 1 waiting to reach incremental sync.. 2023-10-18T20:16:04.212 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:04.211+0000 7f6d58054780 1 waiting to reach incremental sync.. 2023-10-18T20:16:09.213 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:09.212+0000 7f6d58054780 1 waiting to reach incremental sync.. 2023-10-18T20:16:14.214 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:14.212+0000 7f6d58054780 1 waiting to reach incremental sync.. 2023-10-18T20:16:19.215 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:19.214+0000 7f6d58054780 1 bucket sync caught up with source: 2023-10-18T20:16:19.216 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.574.6, , 00000000001.154.6, , 00000000001.678.6, , , 00000000001.918.6, , , ] 2023-10-18T20:16:19.216 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.574.6, , 00000000001.154.6, , 00000000001.678.6, , , 00000000001.918.6, , , ] 2023-10-18T20:16:19.216 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:19.214+0000 7f6d58054780 0 bucket checkpoint complete 2023-10-18T20:16:19.232 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-47 zones={a2, a1} 2023-10-18T20:16:19.233 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.234 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.234 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.235 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:16:19.235 DEBUG:boto:Data: 2023-10-18T20:16:19.235 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.236 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.236 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.237 DEBUG:boto:Params: {} 2023-10-18T20:16:19.237 DEBUG:boto:Token: None 2023-10-18T20:16:19.238 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.238 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZtbI7hMPINJDRW3sZh99YRgKW7M= 2023-10-18T20:16:19.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZtbI7hMPINJDRW3sZh99YRgKW7M=', 'Content-Length': '0'} 2023-10-18T20:16:19.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000063083eb1d87aaed-0065303d13-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.242 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.242 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.243 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.243 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:16:19.243 DEBUG:boto:Data: 2023-10-18T20:16:19.244 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.244 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.245 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.245 DEBUG:boto:Params: {} 2023-10-18T20:16:19.245 DEBUG:boto:Token: None 2023-10-18T20:16:19.246 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.246 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZtbI7hMPINJDRW3sZh99YRgKW7M= 2023-10-18T20:16:19.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZtbI7hMPINJDRW3sZh99YRgKW7M=', 'Content-Length': '0'} 2023-10-18T20:16:19.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007083179d1afbe608-0065303d13-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.249 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:16:19.250 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.250 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.251 DEBUG:boto:path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.251 DEBUG:boto:auth_path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.252 DEBUG:boto:Method: GET 2023-10-18T20:16:19.252 DEBUG:boto:Path: /bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.253 DEBUG:boto:Data: 2023-10-18T20:16:19.253 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.254 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.254 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.255 DEBUG:boto:Params: {} 2023-10-18T20:16:19.255 DEBUG:boto:Token: None 2023-10-18T20:16:19.256 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/?versions 2023-10-18T20:16:19.256 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R3A/cw5ewuvAWGwvHe24ytrQnOI= 2023-10-18T20:16:19.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3A/cw5ewuvAWGwvHe24ytrQnOI=', 'Content-Length': '0'} 2023-10-18T20:16:19.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008c5211268e8517e-0065303d13-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.260 DEBUG:boto:b'bofrnf-47999falseobj1nulltrue2023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:16:19.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:16:19.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:16:19.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:16:19.263 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:16:19.263 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.264 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.264 DEBUG:boto:path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.265 DEBUG:boto:auth_path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.265 DEBUG:boto:Method: GET 2023-10-18T20:16:19.265 DEBUG:boto:Path: /bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.266 DEBUG:boto:Data: 2023-10-18T20:16:19.266 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.267 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.267 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.268 DEBUG:boto:Params: {} 2023-10-18T20:16:19.268 DEBUG:boto:Token: None 2023-10-18T20:16:19.268 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/?versions 2023-10-18T20:16:19.269 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R3A/cw5ewuvAWGwvHe24ytrQnOI= 2023-10-18T20:16:19.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3A/cw5ewuvAWGwvHe24ytrQnOI=', 'Content-Length': '0'} 2023-10-18T20:16:19.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2bc7345799cbf2-0065303d13-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.272 DEBUG:boto:b'bofrnf-47999falseobj1nulltrue2023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:16:19.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:16:19.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:16:19.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:16:19.275 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.275 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.276 DEBUG:boto:path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.276 DEBUG:boto:auth_path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.276 DEBUG:boto:Method: GET 2023-10-18T20:16:19.277 DEBUG:boto:Path: /bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.277 DEBUG:boto:Data: 2023-10-18T20:16:19.277 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.278 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.278 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.279 DEBUG:boto:Params: {} 2023-10-18T20:16:19.279 DEBUG:boto:Token: None 2023-10-18T20:16:19.280 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/?versions 2023-10-18T20:16:19.280 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R3A/cw5ewuvAWGwvHe24ytrQnOI= 2023-10-18T20:16:19.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3A/cw5ewuvAWGwvHe24ytrQnOI=', 'Content-Length': '0'} 2023-10-18T20:16:19.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc39b26e2eff91d3-0065303d13-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.284 DEBUG:boto:b'bofrnf-47999falseobj1nulltrue2023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.285 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.286 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.287 DEBUG:boto:path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.287 DEBUG:boto:auth_path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.288 DEBUG:boto:Method: GET 2023-10-18T20:16:19.289 DEBUG:boto:Path: /bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:19.289 DEBUG:boto:Data: 2023-10-18T20:16:19.290 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.291 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.291 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.292 DEBUG:boto:Params: {} 2023-10-18T20:16:19.293 DEBUG:boto:Token: None 2023-10-18T20:16:19.294 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/?versions 2023-10-18T20:16:19.294 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R3A/cw5ewuvAWGwvHe24ytrQnOI= 2023-10-18T20:16:19.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3A/cw5ewuvAWGwvHe24ytrQnOI=', 'Content-Length': '0'} 2023-10-18T20:16:19.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f87a5d28713572c-0065303d13-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.299 DEBUG:boto:b'bofrnf-47999falseobj1nulltrue2023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:16:19.301 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.301 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.302 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.303 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.303 DEBUG:boto:Method: GET 2023-10-18T20:16:19.303 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.304 DEBUG:boto:Data: 2023-10-18T20:16:19.304 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.305 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.305 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.305 DEBUG:boto:Params: {} 2023-10-18T20:16:19.306 DEBUG:boto:Token: None 2023-10-18T20:16:19.306 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.307 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NWbG35zl/twkgsMwFUjysxB0mQc= 2023-10-18T20:16:19.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NWbG35zl/twkgsMwFUjysxB0mQc=', 'Content-Length': '0'} 2023-10-18T20:16:19.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004043e1394fe08dfd-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.350 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.351 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.351 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.352 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.352 DEBUG:boto:Method: GET 2023-10-18T20:16:19.353 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.353 DEBUG:boto:Data: 2023-10-18T20:16:19.353 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.354 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.354 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.355 DEBUG:boto:Params: {} 2023-10-18T20:16:19.355 DEBUG:boto:Token: None 2023-10-18T20:16:19.356 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NWbG35zl/twkgsMwFUjysxB0mQc= 2023-10-18T20:16:19.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NWbG35zl/twkgsMwFUjysxB0mQc=', 'Content-Length': '0'} 2023-10-18T20:16:19.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ead2e5464c4cdb3c-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.359 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.359 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.360 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.360 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.361 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.361 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.361 DEBUG:boto:Data: 2023-10-18T20:16:19.362 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.362 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.363 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.363 DEBUG:boto:Params: {} 2023-10-18T20:16:19.363 DEBUG:boto:Token: None 2023-10-18T20:16:19.364 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P9BY5O24AKBeDATi3ggnmQ6cND4= 2023-10-18T20:16:19.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P9BY5O24AKBeDATi3ggnmQ6cND4=', 'Content-Length': '0'} 2023-10-18T20:16:19.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f09925ec6ab0c4ca-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.367 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.367 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.368 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.368 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.368 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.369 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.369 DEBUG:boto:Data: 2023-10-18T20:16:19.369 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.370 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.371 DEBUG:boto:Params: {} 2023-10-18T20:16:19.371 DEBUG:boto:Token: None 2023-10-18T20:16:19.372 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.372 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P9BY5O24AKBeDATi3ggnmQ6cND4= 2023-10-18T20:16:19.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P9BY5O24AKBeDATi3ggnmQ6cND4=', 'Content-Length': '0'} 2023-10-18T20:16:19.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000610756e0cd95aa29-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:16:19.376 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.377 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.377 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.378 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.378 DEBUG:boto:Method: GET 2023-10-18T20:16:19.379 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.380 DEBUG:boto:Data: 2023-10-18T20:16:19.380 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.381 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.381 DEBUG:boto:Params: {} 2023-10-18T20:16:19.382 DEBUG:boto:Token: None 2023-10-18T20:16:19.383 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NWbG35zl/twkgsMwFUjysxB0mQc= 2023-10-18T20:16:19.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NWbG35zl/twkgsMwFUjysxB0mQc=', 'Content-Length': '0'} 2023-10-18T20:16:19.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1f1546d0df985ee-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.386 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.387 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.387 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.388 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.388 DEBUG:boto:Method: GET 2023-10-18T20:16:19.388 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.389 DEBUG:boto:Data: 2023-10-18T20:16:19.389 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.390 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.391 DEBUG:boto:Params: {} 2023-10-18T20:16:19.391 DEBUG:boto:Token: None 2023-10-18T20:16:19.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:19.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NWbG35zl/twkgsMwFUjysxB0mQc= 2023-10-18T20:16:19.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NWbG35zl/twkgsMwFUjysxB0mQc=', 'Content-Length': '0'} 2023-10-18T20:16:19.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3c020dea65e10f8-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.442 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.443 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.443 DEBUG:boto:path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.444 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.444 DEBUG:boto:Method: GET 2023-10-18T20:16:19.444 DEBUG:boto:Path: /bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.445 DEBUG:boto:Data: 2023-10-18T20:16:19.445 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.446 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.446 DEBUG:boto:Params: {} 2023-10-18T20:16:19.447 DEBUG:boto:Token: None 2023-10-18T20:16:19.447 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60ec3120e1950b9-0065303d13-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.451 DEBUG:boto:b'bofrnf-47obj11000falseobj12023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.452 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.452 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.453 DEBUG:boto:path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.453 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.454 DEBUG:boto:Method: GET 2023-10-18T20:16:19.454 DEBUG:boto:Path: /bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.454 DEBUG:boto:Data: 2023-10-18T20:16:19.455 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.455 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.456 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.456 DEBUG:boto:Params: {} 2023-10-18T20:16:19.457 DEBUG:boto:Token: None 2023-10-18T20:16:19.457 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.457 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040c0700bb0f78ed6-0065303d13-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.461 DEBUG:boto:b'bofrnf-47obj11000falseobj12023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.462 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.462 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.463 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.463 DEBUG:boto:Path: /bofrnf-47/obj1 2023-10-18T20:16:19.463 DEBUG:boto:Data: 2023-10-18T20:16:19.464 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.464 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.464 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.465 DEBUG:boto:Params: {} 2023-10-18T20:16:19.465 DEBUG:boto:Token: None 2023-10-18T20:16:19.466 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1 2023-10-18T20:16:19.466 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:am9uEmyVXtosCuisXWcpVs+kQcI= 2023-10-18T20:16:19.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:am9uEmyVXtosCuisXWcpVs+kQcI=', 'Content-Length': '0'} 2023-10-18T20:16:19.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7952e2607c02631-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.469 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.470 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.470 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.470 DEBUG:boto:Path: /bofrnf-47/obj1 2023-10-18T20:16:19.471 DEBUG:boto:Data: 2023-10-18T20:16:19.471 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.472 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.472 DEBUG:boto:Params: {} 2023-10-18T20:16:19.473 DEBUG:boto:Token: None 2023-10-18T20:16:19.473 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1 2023-10-18T20:16:19.473 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:am9uEmyVXtosCuisXWcpVs+kQcI= 2023-10-18T20:16:19.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:am9uEmyVXtosCuisXWcpVs+kQcI=', 'Content-Length': '0'} 2023-10-18T20:16:19.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008853ec9bda584ed3-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:16:19.476 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.477 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.477 DEBUG:boto:Method: GET 2023-10-18T20:16:19.478 DEBUG:boto:Path: /bofrnf-47/obj1 2023-10-18T20:16:19.478 DEBUG:boto:Data: 2023-10-18T20:16:19.478 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.479 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.479 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.479 DEBUG:boto:Params: {} 2023-10-18T20:16:19.480 DEBUG:boto:Token: None 2023-10-18T20:16:19.480 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1 2023-10-18T20:16:19.481 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ws3f0/NXEkey9jbHwu8Z71/93Ng= 2023-10-18T20:16:19.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ws3f0/NXEkey9jbHwu8Z71/93Ng=', 'Content-Length': '0'} 2023-10-18T20:16:19.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037766cd9a1ef7f8e-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.526 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:19.527 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:19.527 DEBUG:boto:Method: GET 2023-10-18T20:16:19.528 DEBUG:boto:Path: /bofrnf-47/obj1 2023-10-18T20:16:19.528 DEBUG:boto:Data: 2023-10-18T20:16:19.529 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.529 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.529 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.530 DEBUG:boto:Params: {} 2023-10-18T20:16:19.530 DEBUG:boto:Token: None 2023-10-18T20:16:19.531 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj1 2023-10-18T20:16:19.531 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ws3f0/NXEkey9jbHwu8Z71/93Ng= 2023-10-18T20:16:19.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ws3f0/NXEkey9jbHwu8Z71/93Ng=', 'Content-Length': '0'} 2023-10-18T20:16:19.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039959a42332f5d16-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.535 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.535 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.536 DEBUG:boto:path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.536 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.536 DEBUG:boto:Method: GET 2023-10-18T20:16:19.537 DEBUG:boto:Path: /bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.537 DEBUG:boto:Data: 2023-10-18T20:16:19.537 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.538 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.538 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.539 DEBUG:boto:Params: {} 2023-10-18T20:16:19.539 DEBUG:boto:Token: None 2023-10-18T20:16:19.539 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.540 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002be90da9f35ca17a-0065303d13-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.543 DEBUG:boto:b'bofrnf-47obj11000falseobj12023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.544 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.544 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.545 DEBUG:boto:path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.545 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.546 DEBUG:boto:Method: GET 2023-10-18T20:16:19.546 DEBUG:boto:Path: /bofrnf-47/?prefix=obj1 2023-10-18T20:16:19.546 DEBUG:boto:Data: 2023-10-18T20:16:19.547 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.547 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.548 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.548 DEBUG:boto:Params: {} 2023-10-18T20:16:19.549 DEBUG:boto:Token: None 2023-10-18T20:16:19.549 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.549 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ab7b0ba58ad5cf-0065303d13-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.553 DEBUG:boto:b'bofrnf-47obj11000falseobj12023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:16:19.554 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.554 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.555 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.555 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.555 DEBUG:boto:Method: GET 2023-10-18T20:16:19.556 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.556 DEBUG:boto:Data: 2023-10-18T20:16:19.557 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.557 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.557 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.558 DEBUG:boto:Params: {} 2023-10-18T20:16:19.558 DEBUG:boto:Token: None 2023-10-18T20:16:19.559 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.559 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UWlJ2YzD3fLNojqHud7r6cNSAsc= 2023-10-18T20:16:19.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UWlJ2YzD3fLNojqHud7r6cNSAsc=', 'Content-Length': '0'} 2023-10-18T20:16:19.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ddd8ecc021e0b40-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.562 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.563 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.563 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.564 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.564 DEBUG:boto:Method: GET 2023-10-18T20:16:19.564 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.565 DEBUG:boto:Data: 2023-10-18T20:16:19.565 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.566 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.566 DEBUG:boto:Params: {} 2023-10-18T20:16:19.567 DEBUG:boto:Token: None 2023-10-18T20:16:19.567 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UWlJ2YzD3fLNojqHud7r6cNSAsc= 2023-10-18T20:16:19.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UWlJ2YzD3fLNojqHud7r6cNSAsc=', 'Content-Length': '0'} 2023-10-18T20:16:19.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000681bfc636a3f924b-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.571 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.571 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.572 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.572 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.572 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.573 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.573 DEBUG:boto:Data: 2023-10-18T20:16:19.573 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.574 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.574 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.575 DEBUG:boto:Params: {} 2023-10-18T20:16:19.575 DEBUG:boto:Token: None 2023-10-18T20:16:19.575 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/q3JGu8/IxSmFq2NIlCC5x3X45U= 2023-10-18T20:16:19.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/q3JGu8/IxSmFq2NIlCC5x3X45U=', 'Content-Length': '0'} 2023-10-18T20:16:19.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc244fbbf750151c-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.579 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.579 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.579 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.580 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.580 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.581 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.581 DEBUG:boto:Data: 2023-10-18T20:16:19.581 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.582 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.582 DEBUG:boto:Params: {} 2023-10-18T20:16:19.583 DEBUG:boto:Token: None 2023-10-18T20:16:19.583 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.584 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/q3JGu8/IxSmFq2NIlCC5x3X45U= 2023-10-18T20:16:19.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/q3JGu8/IxSmFq2NIlCC5x3X45U=', 'Content-Length': '0'} 2023-10-18T20:16:19.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ce16326cee44f9b-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:16:19.587 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.588 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.588 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.588 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.589 DEBUG:boto:Method: GET 2023-10-18T20:16:19.589 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.590 DEBUG:boto:Data: 2023-10-18T20:16:19.590 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.590 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.591 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.591 DEBUG:boto:Params: {} 2023-10-18T20:16:19.592 DEBUG:boto:Token: None 2023-10-18T20:16:19.592 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.593 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UWlJ2YzD3fLNojqHud7r6cNSAsc= 2023-10-18T20:16:19.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UWlJ2YzD3fLNojqHud7r6cNSAsc=', 'Content-Length': '0'} 2023-10-18T20:16:19.595 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e51fb0be729e4fb9-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.596 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.596 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.597 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.597 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.598 DEBUG:boto:Method: GET 2023-10-18T20:16:19.598 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.598 DEBUG:boto:Data: 2023-10-18T20:16:19.599 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.599 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.600 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.600 DEBUG:boto:Params: {} 2023-10-18T20:16:19.601 DEBUG:boto:Token: None 2023-10-18T20:16:19.601 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:19.602 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UWlJ2YzD3fLNojqHud7r6cNSAsc= 2023-10-18T20:16:19.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UWlJ2YzD3fLNojqHud7r6cNSAsc=', 'Content-Length': '0'} 2023-10-18T20:16:19.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000261c19b9e54a0681-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.646 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.647 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.647 DEBUG:boto:path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.648 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.648 DEBUG:boto:Method: GET 2023-10-18T20:16:19.648 DEBUG:boto:Path: /bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.649 DEBUG:boto:Data: 2023-10-18T20:16:19.649 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.650 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.650 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.650 DEBUG:boto:Params: {} 2023-10-18T20:16:19.651 DEBUG:boto:Token: None 2023-10-18T20:16:19.651 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc715e19bc1771f-0065303d13-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.655 DEBUG:boto:b'bofrnf-47obj21000falseobj22023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.656 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.656 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.657 DEBUG:boto:path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.657 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.658 DEBUG:boto:Method: GET 2023-10-18T20:16:19.658 DEBUG:boto:Path: /bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.658 DEBUG:boto:Data: 2023-10-18T20:16:19.659 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.660 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.660 DEBUG:boto:Params: {} 2023-10-18T20:16:19.661 DEBUG:boto:Token: None 2023-10-18T20:16:19.661 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b86544b91f74dcb-0065303d13-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.665 DEBUG:boto:b'bofrnf-47obj21000falseobj22023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.666 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.667 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.667 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.667 DEBUG:boto:Path: /bofrnf-47/obj2 2023-10-18T20:16:19.668 DEBUG:boto:Data: 2023-10-18T20:16:19.668 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.669 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.669 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.669 DEBUG:boto:Params: {} 2023-10-18T20:16:19.670 DEBUG:boto:Token: None 2023-10-18T20:16:19.670 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2 2023-10-18T20:16:19.671 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3k4yvR0GdM3/uQgMiS0N1nqkrGU= 2023-10-18T20:16:19.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3k4yvR0GdM3/uQgMiS0N1nqkrGU=', 'Content-Length': '0'} 2023-10-18T20:16:19.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b27255ae16ebace-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.674 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.674 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.675 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.675 DEBUG:boto:Path: /bofrnf-47/obj2 2023-10-18T20:16:19.676 DEBUG:boto:Data: 2023-10-18T20:16:19.676 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.676 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.677 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.677 DEBUG:boto:Params: {} 2023-10-18T20:16:19.677 DEBUG:boto:Token: None 2023-10-18T20:16:19.678 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2 2023-10-18T20:16:19.678 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3k4yvR0GdM3/uQgMiS0N1nqkrGU= 2023-10-18T20:16:19.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3k4yvR0GdM3/uQgMiS0N1nqkrGU=', 'Content-Length': '0'} 2023-10-18T20:16:19.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ca5df9e89631b20-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:16:19.682 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.682 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.683 DEBUG:boto:Method: GET 2023-10-18T20:16:19.683 DEBUG:boto:Path: /bofrnf-47/obj2 2023-10-18T20:16:19.684 DEBUG:boto:Data: 2023-10-18T20:16:19.684 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.684 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.685 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.685 DEBUG:boto:Params: {} 2023-10-18T20:16:19.686 DEBUG:boto:Token: None 2023-10-18T20:16:19.686 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2 2023-10-18T20:16:19.687 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q/GXueQhKjYjCE/k8fBs12t4DgU= 2023-10-18T20:16:19.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q/GXueQhKjYjCE/k8fBs12t4DgU=', 'Content-Length': '0'} 2023-10-18T20:16:19.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab9e19b782df12c4-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.730 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:19.731 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:19.732 DEBUG:boto:Method: GET 2023-10-18T20:16:19.732 DEBUG:boto:Path: /bofrnf-47/obj2 2023-10-18T20:16:19.733 DEBUG:boto:Data: 2023-10-18T20:16:19.733 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.734 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.734 DEBUG:boto:Params: {} 2023-10-18T20:16:19.735 DEBUG:boto:Token: None 2023-10-18T20:16:19.736 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj2 2023-10-18T20:16:19.736 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q/GXueQhKjYjCE/k8fBs12t4DgU= 2023-10-18T20:16:19.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q/GXueQhKjYjCE/k8fBs12t4DgU=', 'Content-Length': '0'} 2023-10-18T20:16:19.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f9b10647c3899ce-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.740 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.740 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.741 DEBUG:boto:path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.741 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.742 DEBUG:boto:Method: GET 2023-10-18T20:16:19.742 DEBUG:boto:Path: /bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.743 DEBUG:boto:Data: 2023-10-18T20:16:19.743 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.743 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.744 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.744 DEBUG:boto:Params: {} 2023-10-18T20:16:19.745 DEBUG:boto:Token: None 2023-10-18T20:16:19.745 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.746 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1df4a1fcd7ce2f4-0065303d13-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.750 DEBUG:boto:b'bofrnf-47obj21000falseobj22023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.750 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.751 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.751 DEBUG:boto:path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.752 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.752 DEBUG:boto:Method: GET 2023-10-18T20:16:19.752 DEBUG:boto:Path: /bofrnf-47/?prefix=obj2 2023-10-18T20:16:19.753 DEBUG:boto:Data: 2023-10-18T20:16:19.753 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.754 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.755 DEBUG:boto:Params: {} 2023-10-18T20:16:19.755 DEBUG:boto:Token: None 2023-10-18T20:16:19.755 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a244ad498c896761-0065303d13-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.759 DEBUG:boto:b'bofrnf-47obj21000falseobj22023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:16:19.760 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.761 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.761 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.761 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.762 DEBUG:boto:Method: GET 2023-10-18T20:16:19.762 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.763 DEBUG:boto:Data: 2023-10-18T20:16:19.763 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.764 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.764 DEBUG:boto:Params: {} 2023-10-18T20:16:19.765 DEBUG:boto:Token: None 2023-10-18T20:16:19.765 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.766 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LkepNj8UIwA61s/VCzcjc3NH8a8= 2023-10-18T20:16:19.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LkepNj8UIwA61s/VCzcjc3NH8a8=', 'Content-Length': '0'} 2023-10-18T20:16:19.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056938aa08697934b-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.769 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.769 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.770 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.770 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.771 DEBUG:boto:Method: GET 2023-10-18T20:16:19.771 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.771 DEBUG:boto:Data: 2023-10-18T20:16:19.772 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.772 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.773 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.773 DEBUG:boto:Params: {} 2023-10-18T20:16:19.774 DEBUG:boto:Token: None 2023-10-18T20:16:19.774 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.775 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LkepNj8UIwA61s/VCzcjc3NH8a8= 2023-10-18T20:16:19.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LkepNj8UIwA61s/VCzcjc3NH8a8=', 'Content-Length': '0'} 2023-10-18T20:16:19.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f74eddf5d864daeb-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.818 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.819 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.819 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.820 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.820 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.820 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.821 DEBUG:boto:Data: 2023-10-18T20:16:19.821 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.822 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.822 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.823 DEBUG:boto:Params: {} 2023-10-18T20:16:19.823 DEBUG:boto:Token: None 2023-10-18T20:16:19.824 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UB1zHDSTamjgq4iXSdGng+FFgYU= 2023-10-18T20:16:19.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UB1zHDSTamjgq4iXSdGng+FFgYU=', 'Content-Length': '0'} 2023-10-18T20:16:19.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e5d2e5915a8aa2c-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.828 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.828 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.829 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.829 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.830 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.831 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.832 DEBUG:boto:Data: 2023-10-18T20:16:19.832 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.833 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.833 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.834 DEBUG:boto:Params: {} 2023-10-18T20:16:19.835 DEBUG:boto:Token: None 2023-10-18T20:16:19.836 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.836 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UB1zHDSTamjgq4iXSdGng+FFgYU= 2023-10-18T20:16:19.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UB1zHDSTamjgq4iXSdGng+FFgYU=', 'Content-Length': '0'} 2023-10-18T20:16:19.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008af33f53a3ba62b4-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:16:19.840 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.840 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.841 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.841 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.842 DEBUG:boto:Method: GET 2023-10-18T20:16:19.842 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.842 DEBUG:boto:Data: 2023-10-18T20:16:19.843 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.844 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.844 DEBUG:boto:Params: {} 2023-10-18T20:16:19.844 DEBUG:boto:Token: None 2023-10-18T20:16:19.845 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.845 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LkepNj8UIwA61s/VCzcjc3NH8a8= 2023-10-18T20:16:19.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LkepNj8UIwA61s/VCzcjc3NH8a8=', 'Content-Length': '0'} 2023-10-18T20:16:19.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1d6e92b0173bebf-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.849 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.849 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.850 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.850 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.851 DEBUG:boto:Method: GET 2023-10-18T20:16:19.851 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.851 DEBUG:boto:Data: 2023-10-18T20:16:19.852 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.852 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.852 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.853 DEBUG:boto:Params: {} 2023-10-18T20:16:19.853 DEBUG:boto:Token: None 2023-10-18T20:16:19.854 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:19.854 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LkepNj8UIwA61s/VCzcjc3NH8a8= 2023-10-18T20:16:19.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LkepNj8UIwA61s/VCzcjc3NH8a8=', 'Content-Length': '0'} 2023-10-18T20:16:19.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d340bb65b09e8504-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.858 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.858 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.859 DEBUG:boto:path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.859 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.859 DEBUG:boto:Method: GET 2023-10-18T20:16:19.860 DEBUG:boto:Path: /bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.860 DEBUG:boto:Data: 2023-10-18T20:16:19.861 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.861 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.862 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.862 DEBUG:boto:Params: {} 2023-10-18T20:16:19.862 DEBUG:boto:Token: None 2023-10-18T20:16:19.863 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.863 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009487c821eaaba3b9-0065303d13-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.867 DEBUG:boto:b'bofrnf-47obj31000falseobj32023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.867 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.868 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.868 DEBUG:boto:path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.869 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.869 DEBUG:boto:Method: GET 2023-10-18T20:16:19.870 DEBUG:boto:Path: /bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.870 DEBUG:boto:Data: 2023-10-18T20:16:19.870 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.871 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.871 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.872 DEBUG:boto:Params: {} 2023-10-18T20:16:19.872 DEBUG:boto:Token: None 2023-10-18T20:16:19.872 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.873 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006312451b7ca6ba50-0065303d13-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.876 DEBUG:boto:b'bofrnf-47obj31000falseobj32023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.877 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.877 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.878 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.878 DEBUG:boto:Path: /bofrnf-47/obj3 2023-10-18T20:16:19.879 DEBUG:boto:Data: 2023-10-18T20:16:19.879 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.880 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.880 DEBUG:boto:Params: {} 2023-10-18T20:16:19.881 DEBUG:boto:Token: None 2023-10-18T20:16:19.881 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3 2023-10-18T20:16:19.882 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vdySYa2sPYCjOLg7DHhzLs2vhSY= 2023-10-18T20:16:19.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vdySYa2sPYCjOLg7DHhzLs2vhSY=', 'Content-Length': '0'} 2023-10-18T20:16:19.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecc59bff0ff659b6-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.886 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.887 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.887 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.888 DEBUG:boto:Path: /bofrnf-47/obj3 2023-10-18T20:16:19.889 DEBUG:boto:Data: 2023-10-18T20:16:19.890 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.890 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.891 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.891 DEBUG:boto:Params: {} 2023-10-18T20:16:19.891 DEBUG:boto:Token: None 2023-10-18T20:16:19.892 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3 2023-10-18T20:16:19.892 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vdySYa2sPYCjOLg7DHhzLs2vhSY= 2023-10-18T20:16:19.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vdySYa2sPYCjOLg7DHhzLs2vhSY=', 'Content-Length': '0'} 2023-10-18T20:16:19.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077d3102f244cbf62-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:16:19.896 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.896 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.896 DEBUG:boto:Method: GET 2023-10-18T20:16:19.897 DEBUG:boto:Path: /bofrnf-47/obj3 2023-10-18T20:16:19.897 DEBUG:boto:Data: 2023-10-18T20:16:19.897 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.898 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.898 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.899 DEBUG:boto:Params: {} 2023-10-18T20:16:19.899 DEBUG:boto:Token: None 2023-10-18T20:16:19.900 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3 2023-10-18T20:16:19.900 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AvG4QcIKpJoIwNDPvpjsoHTwGKc= 2023-10-18T20:16:19.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AvG4QcIKpJoIwNDPvpjsoHTwGKc=', 'Content-Length': '0'} 2023-10-18T20:16:19.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2f8cc978c9f6f0f-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.942 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:19.943 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:19.943 DEBUG:boto:Method: GET 2023-10-18T20:16:19.944 DEBUG:boto:Path: /bofrnf-47/obj3 2023-10-18T20:16:19.944 DEBUG:boto:Data: 2023-10-18T20:16:19.944 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.945 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.945 DEBUG:boto:Params: {} 2023-10-18T20:16:19.946 DEBUG:boto:Token: None 2023-10-18T20:16:19.946 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj3 2023-10-18T20:16:19.947 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AvG4QcIKpJoIwNDPvpjsoHTwGKc= 2023-10-18T20:16:19.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AvG4QcIKpJoIwNDPvpjsoHTwGKc=', 'Content-Length': '0'} 2023-10-18T20:16:19.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000740e29954b614500-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.950 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.950 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.951 DEBUG:boto:path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.951 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.952 DEBUG:boto:Method: GET 2023-10-18T20:16:19.952 DEBUG:boto:Path: /bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.952 DEBUG:boto:Data: 2023-10-18T20:16:19.953 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.953 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.953 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.954 DEBUG:boto:Params: {} 2023-10-18T20:16:19.955 DEBUG:boto:Token: None 2023-10-18T20:16:19.955 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.955 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dff1cd4df7f7b8be-0065303d13-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.959 DEBUG:boto:b'bofrnf-47obj31000falseobj32023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.960 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:19.960 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:19.960 DEBUG:boto:path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.961 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.961 DEBUG:boto:Method: GET 2023-10-18T20:16:19.962 DEBUG:boto:Path: /bofrnf-47/?prefix=obj3 2023-10-18T20:16:19.962 DEBUG:boto:Data: 2023-10-18T20:16:19.963 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.963 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.963 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.964 DEBUG:boto:Params: {} 2023-10-18T20:16:19.964 DEBUG:boto:Token: None 2023-10-18T20:16:19.965 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/ 2023-10-18T20:16:19.965 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No= 2023-10-18T20:16:19.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orBZ9nQtsxnxOqQ/Wr3U2LF8/No=', 'Content-Length': '0'} 2023-10-18T20:16:19.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076e830022fb0ea6f-0065303d13-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.968 DEBUG:boto:b'bofrnf-47obj31000falseobj32023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:19.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:16:19.969 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:19.970 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:19.970 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.971 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.971 DEBUG:boto:Method: GET 2023-10-18T20:16:19.971 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.972 DEBUG:boto:Data: 2023-10-18T20:16:19.972 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.973 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.973 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.973 DEBUG:boto:Params: {} 2023-10-18T20:16:19.974 DEBUG:boto:Token: None 2023-10-18T20:16:19.974 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.975 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xo2kkixLIBfRYB2L/RgPzBF0Cl4= 2023-10-18T20:16:19.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xo2kkixLIBfRYB2L/RgPzBF0Cl4=', 'Content-Length': '0'} 2023-10-18T20:16:19.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002121957c43344f1f-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.978 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:19.978 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:19.979 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.979 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.980 DEBUG:boto:Method: GET 2023-10-18T20:16:19.980 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.980 DEBUG:boto:Data: 2023-10-18T20:16:19.981 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.981 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.982 DEBUG:boto:Params: {} 2023-10-18T20:16:19.982 DEBUG:boto:Token: None 2023-10-18T20:16:19.983 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xo2kkixLIBfRYB2L/RgPzBF0Cl4= 2023-10-18T20:16:19.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xo2kkixLIBfRYB2L/RgPzBF0Cl4=', 'Content-Length': '0'} 2023-10-18T20:16:19.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008313ee2f16a73894-0065303d13-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.986 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:19.987 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:19.987 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.987 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.988 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.988 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.988 DEBUG:boto:Data: 2023-10-18T20:16:19.989 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.989 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:19.990 DEBUG:boto:Port: 8001 2023-10-18T20:16:19.990 DEBUG:boto:Params: {} 2023-10-18T20:16:19.990 DEBUG:boto:Token: None 2023-10-18T20:16:19.991 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.991 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yLFjrnvf40C7m7gymy1qKzugPCE= 2023-10-18T20:16:19.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yLFjrnvf40C7m7gymy1qKzugPCE=', 'Content-Length': '0'} 2023-10-18T20:16:19.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007772001ddd05256f-0065303d13-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:19.994 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:19.995 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:19.995 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.995 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.996 DEBUG:boto:Method: HEAD 2023-10-18T20:16:19.996 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.997 DEBUG:boto:Data: 2023-10-18T20:16:19.997 DEBUG:boto:Headers: {} 2023-10-18T20:16:19.997 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:19.998 DEBUG:boto:Port: 8000 2023-10-18T20:16:19.998 DEBUG:boto:Params: {} 2023-10-18T20:16:19.999 DEBUG:boto:Token: None 2023-10-18T20:16:19.999 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:19 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:19.999 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yLFjrnvf40C7m7gymy1qKzugPCE= 2023-10-18T20:16:20.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yLFjrnvf40C7m7gymy1qKzugPCE=', 'Content-Length': '0'} 2023-10-18T20:16:20.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0bf76a231108706-0065303d14-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:16:20.003 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:20.003 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:20.004 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:20.004 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:20.004 DEBUG:boto:Method: GET 2023-10-18T20:16:20.005 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:20.005 DEBUG:boto:Data: 2023-10-18T20:16:20.006 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.006 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.006 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.007 DEBUG:boto:Params: {} 2023-10-18T20:16:20.007 DEBUG:boto:Token: None 2023-10-18T20:16:20.008 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:20.008 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rEf7hiC2SK0E3OeI/YUNACyLh2s= 2023-10-18T20:16:20.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rEf7hiC2SK0E3OeI/YUNACyLh2s=', 'Content-Length': '0'} 2023-10-18T20:16:20.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001344e82654b7240b-0065303d14-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.011 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:20.011 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:20.012 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:20.012 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:20.013 DEBUG:boto:Method: GET 2023-10-18T20:16:20.013 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:20.013 DEBUG:boto:Data: 2023-10-18T20:16:20.014 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.015 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.015 DEBUG:boto:Params: {} 2023-10-18T20:16:20.016 DEBUG:boto:Token: None 2023-10-18T20:16:20.016 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:20.017 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rEf7hiC2SK0E3OeI/YUNACyLh2s= 2023-10-18T20:16:20.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rEf7hiC2SK0E3OeI/YUNACyLh2s=', 'Content-Length': '0'} 2023-10-18T20:16:20.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c716082bd294db1-0065303d14-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.020 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:20.020 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:20.021 DEBUG:boto:path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.021 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.022 DEBUG:boto:Method: GET 2023-10-18T20:16:20.022 DEBUG:boto:Path: /bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.022 DEBUG:boto:Data: 2023-10-18T20:16:20.023 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.023 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.024 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.024 DEBUG:boto:Params: {} 2023-10-18T20:16:20.025 DEBUG:boto:Token: None 2023-10-18T20:16:20.025 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/ 2023-10-18T20:16:20.025 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SrE6JlW7aA2mAXGx5aMeCCEqOsI= 2023-10-18T20:16:20.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SrE6JlW7aA2mAXGx5aMeCCEqOsI=', 'Content-Length': '0'} 2023-10-18T20:16:20.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f14a59e4cb8bf51-0065303d14-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.070 DEBUG:boto:b'bofrnf-47obj41000falseobj42023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:20.071 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:20.071 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:20.072 DEBUG:boto:path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.072 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.073 DEBUG:boto:Method: GET 2023-10-18T20:16:20.073 DEBUG:boto:Path: /bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.074 DEBUG:boto:Data: 2023-10-18T20:16:20.074 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.074 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.075 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.075 DEBUG:boto:Params: {} 2023-10-18T20:16:20.076 DEBUG:boto:Token: None 2023-10-18T20:16:20.076 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/ 2023-10-18T20:16:20.077 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SrE6JlW7aA2mAXGx5aMeCCEqOsI= 2023-10-18T20:16:20.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SrE6JlW7aA2mAXGx5aMeCCEqOsI=', 'Content-Length': '0'} 2023-10-18T20:16:20.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000428ff7bce9645cdb-0065303d14-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.080 DEBUG:boto:b'bofrnf-47obj41000falseobj42023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:20.081 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:20.082 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:20.082 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.082 DEBUG:boto:Path: /bofrnf-47/obj4 2023-10-18T20:16:20.083 DEBUG:boto:Data: 2023-10-18T20:16:20.083 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.084 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.084 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.085 DEBUG:boto:Params: {} 2023-10-18T20:16:20.086 DEBUG:boto:Token: None 2023-10-18T20:16:20.087 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj4 2023-10-18T20:16:20.087 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vzEC55cMAYInstf7dRYgUC9wdI4= 2023-10-18T20:16:20.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vzEC55cMAYInstf7dRYgUC9wdI4=', 'Content-Length': '0'} 2023-10-18T20:16:20.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052439ae0dfcd4304-0065303d14-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.091 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:20.092 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:20.092 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.093 DEBUG:boto:Path: /bofrnf-47/obj4 2023-10-18T20:16:20.093 DEBUG:boto:Data: 2023-10-18T20:16:20.094 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.094 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.094 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.095 DEBUG:boto:Params: {} 2023-10-18T20:16:20.095 DEBUG:boto:Token: None 2023-10-18T20:16:20.096 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj4 2023-10-18T20:16:20.096 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vzEC55cMAYInstf7dRYgUC9wdI4= 2023-10-18T20:16:20.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vzEC55cMAYInstf7dRYgUC9wdI4=', 'Content-Length': '0'} 2023-10-18T20:16:20.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ece30752a66e62f-0065303d14-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:16:20.099 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:20.100 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:20.100 DEBUG:boto:Method: GET 2023-10-18T20:16:20.100 DEBUG:boto:Path: /bofrnf-47/obj4 2023-10-18T20:16:20.101 DEBUG:boto:Data: 2023-10-18T20:16:20.101 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.101 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.102 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.102 DEBUG:boto:Params: {} 2023-10-18T20:16:20.103 DEBUG:boto:Token: None 2023-10-18T20:16:20.103 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj4 2023-10-18T20:16:20.104 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NKoTCxZtKwblXCsvJCFw878bpaA= 2023-10-18T20:16:20.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NKoTCxZtKwblXCsvJCFw878bpaA=', 'Content-Length': '0'} 2023-10-18T20:16:20.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d09f67065fcf81dc-0065303d14-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.107 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:20.107 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:20.108 DEBUG:boto:Method: GET 2023-10-18T20:16:20.108 DEBUG:boto:Path: /bofrnf-47/obj4 2023-10-18T20:16:20.109 DEBUG:boto:Data: 2023-10-18T20:16:20.109 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.110 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.110 DEBUG:boto:Params: {} 2023-10-18T20:16:20.111 DEBUG:boto:Token: None 2023-10-18T20:16:20.111 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj4 2023-10-18T20:16:20.112 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NKoTCxZtKwblXCsvJCFw878bpaA= 2023-10-18T20:16:20.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NKoTCxZtKwblXCsvJCFw878bpaA=', 'Content-Length': '0'} 2023-10-18T20:16:20.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087ef8669dd072df8-0065303d14-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.115 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:20.115 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:20.116 DEBUG:boto:path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.116 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.117 DEBUG:boto:Method: GET 2023-10-18T20:16:20.117 DEBUG:boto:Path: /bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.118 DEBUG:boto:Data: 2023-10-18T20:16:20.118 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.118 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.119 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.119 DEBUG:boto:Params: {} 2023-10-18T20:16:20.120 DEBUG:boto:Token: None 2023-10-18T20:16:20.120 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/ 2023-10-18T20:16:20.121 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SrE6JlW7aA2mAXGx5aMeCCEqOsI= 2023-10-18T20:16:20.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SrE6JlW7aA2mAXGx5aMeCCEqOsI=', 'Content-Length': '0'} 2023-10-18T20:16:20.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d6cd826744fb79-0065303d14-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.166 DEBUG:boto:b'bofrnf-47obj41000falseobj42023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:20.167 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:20.168 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:20.168 DEBUG:boto:path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.168 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.169 DEBUG:boto:Method: GET 2023-10-18T20:16:20.169 DEBUG:boto:Path: /bofrnf-47/?prefix=obj4 2023-10-18T20:16:20.169 DEBUG:boto:Data: 2023-10-18T20:16:20.170 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.170 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.171 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.171 DEBUG:boto:Params: {} 2023-10-18T20:16:20.172 DEBUG:boto:Token: None 2023-10-18T20:16:20.172 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/ 2023-10-18T20:16:20.172 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SrE6JlW7aA2mAXGx5aMeCCEqOsI= 2023-10-18T20:16:20.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SrE6JlW7aA2mAXGx5aMeCCEqOsI=', 'Content-Length': '0'} 2023-10-18T20:16:20.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e4e801d4e5785b-0065303d14-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.176 DEBUG:boto:b'bofrnf-47obj41000falseobj42023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:20.176 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-47 zones={a2, a1} 2023-10-18T20:16:20.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bofrnf-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:16:20.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bofrnf-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:16:20.178 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bofrnf-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:16:20.236 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:20.236 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:20.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bofrnf-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:16:20.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bofrnf-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:16:20.306 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bofrnf-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:16:20.357 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:20.357 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:20.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bofrnf-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:16:20.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bofrnf-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:16:20.421 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bofrnf-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:16:20.472 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:20.473 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:20.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bofrnf-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:16:20.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bofrnf-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:16:20.536 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bofrnf-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:16:20.586 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:20.586 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:20.649 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:20.649 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:20.650 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.650 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:16:20.651 DEBUG:boto:Data: 2023-10-18T20:16:20.651 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.652 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.652 DEBUG:boto:Params: {} 2023-10-18T20:16:20.653 DEBUG:boto:Token: None 2023-10-18T20:16:20.653 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-46/ 2023-10-18T20:16:20.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5jXdR/4albIfGkAKOqpjRPNYRGw= 2023-10-18T20:16:20.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5jXdR/4albIfGkAKOqpjRPNYRGw=', 'Content-Length': '0'} 2023-10-18T20:16:20.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000341889921d7e32ef-0065303d14-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.657 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:20.657 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:20.658 DEBUG:boto:Method: PUT 2023-10-18T20:16:20.658 DEBUG:boto:Path: /bofrnf-46/obj5 2023-10-18T20:16:20.658 DEBUG:boto:Data: 2023-10-18T20:16:20.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:16:20.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.660 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.660 DEBUG:boto:Params: {} 2023-10-18T20:16:20.660 DEBUG:boto:Token: None 2023-10-18T20:16:20.661 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-46/obj5 2023-10-18T20:16:20.661 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rJ7XTNRCEFHBmsHk99bpQtIxC+Q= 2023-10-18T20:16:20.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rJ7XTNRCEFHBmsHk99bpQtIxC+Q='} 2023-10-18T20:16:20.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000969d1af1ee3605fc-0065303d14-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.667 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:20.667 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:20.668 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.668 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:16:20.668 DEBUG:boto:Data: 2023-10-18T20:16:20.669 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.669 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.670 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.670 DEBUG:boto:Params: {} 2023-10-18T20:16:20.671 DEBUG:boto:Token: None 2023-10-18T20:16:20.671 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-46/ 2023-10-18T20:16:20.671 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5jXdR/4albIfGkAKOqpjRPNYRGw= 2023-10-18T20:16:20.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5jXdR/4albIfGkAKOqpjRPNYRGw=', 'Content-Length': '0'} 2023-10-18T20:16:20.674 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1d0402cf70d5e2e-0065303d14-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.675 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:20.675 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:20.675 DEBUG:boto:Method: PUT 2023-10-18T20:16:20.676 DEBUG:boto:Path: /bofrnf-46/obj6 2023-10-18T20:16:20.676 DEBUG:boto:Data: 2023-10-18T20:16:20.677 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:16:20.677 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.677 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.678 DEBUG:boto:Params: {} 2023-10-18T20:16:20.678 DEBUG:boto:Token: None 2023-10-18T20:16:20.679 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-46/obj6 2023-10-18T20:16:20.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ok9i6usY5XkYYFMEwAQpewX9vrM= 2023-10-18T20:16:20.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ok9i6usY5XkYYFMEwAQpewX9vrM='} 2023-10-18T20:16:20.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054a2e85853fdd07e-0065303d14-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.723 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:20.723 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:20.724 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.724 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:16:20.725 DEBUG:boto:Data: 2023-10-18T20:16:20.725 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.726 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.726 DEBUG:boto:Params: {} 2023-10-18T20:16:20.727 DEBUG:boto:Token: None 2023-10-18T20:16:20.727 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-46/ 2023-10-18T20:16:20.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5jXdR/4albIfGkAKOqpjRPNYRGw= 2023-10-18T20:16:20.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5jXdR/4albIfGkAKOqpjRPNYRGw=', 'Content-Length': '0'} 2023-10-18T20:16:20.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000bc48598f72f8c6-0065303d14-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.731 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:20.731 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:20.732 DEBUG:boto:Method: PUT 2023-10-18T20:16:20.732 DEBUG:boto:Path: /bofrnf-46/obj7 2023-10-18T20:16:20.733 DEBUG:boto:Data: 2023-10-18T20:16:20.733 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:16:20.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.734 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.734 DEBUG:boto:Params: {} 2023-10-18T20:16:20.735 DEBUG:boto:Token: None 2023-10-18T20:16:20.735 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-46/obj7 2023-10-18T20:16:20.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AUKKURtHty0oyQNO/VujC7Fq3cE= 2023-10-18T20:16:20.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AUKKURtHty0oyQNO/VujC7Fq3cE='} 2023-10-18T20:16:20.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aeed43d1fa4cd152-0065303d14-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.744 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:20.744 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:20.744 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.745 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:16:20.745 DEBUG:boto:Data: 2023-10-18T20:16:20.745 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.746 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.746 DEBUG:boto:Params: {} 2023-10-18T20:16:20.747 DEBUG:boto:Token: None 2023-10-18T20:16:20.747 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-46/ 2023-10-18T20:16:20.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5jXdR/4albIfGkAKOqpjRPNYRGw= 2023-10-18T20:16:20.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5jXdR/4albIfGkAKOqpjRPNYRGw=', 'Content-Length': '0'} 2023-10-18T20:16:20.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ced0d700a688e78-0065303d14-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.751 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:20.751 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:20.752 DEBUG:boto:Method: PUT 2023-10-18T20:16:20.752 DEBUG:boto:Path: /bofrnf-46/obj8 2023-10-18T20:16:20.752 DEBUG:boto:Data: 2023-10-18T20:16:20.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:16:20.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:20.753 DEBUG:boto:Port: 8000 2023-10-18T20:16:20.754 DEBUG:boto:Params: {} 2023-10-18T20:16:20.754 DEBUG:boto:Token: None 2023-10-18T20:16:20.755 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-46/obj8 2023-10-18T20:16:20.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OXmT8p6PXg91YxPyqueXsqFp66c= 2023-10-18T20:16:20.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OXmT8p6PXg91YxPyqueXsqFp66c='} 2023-10-18T20:16:20.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3df49d661573342-0065303d14-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.803 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:20.803 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:20.804 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.804 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:16:20.805 DEBUG:boto:Data: 2023-10-18T20:16:20.805 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.806 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.806 DEBUG:boto:Params: {} 2023-10-18T20:16:20.806 DEBUG:boto:Token: None 2023-10-18T20:16:20.807 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/ 2023-10-18T20:16:20.807 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jv4sNmRjA+vGimjX1U+ZqFxQ5N4= 2023-10-18T20:16:20.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jv4sNmRjA+vGimjX1U+ZqFxQ5N4=', 'Content-Length': '0'} 2023-10-18T20:16:20.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b9af81867c0f09b-0065303d14-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.811 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:20.811 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:20.811 DEBUG:boto:Method: PUT 2023-10-18T20:16:20.812 DEBUG:boto:Path: /bofrnf-47/obj5 2023-10-18T20:16:20.812 DEBUG:boto:Data: 2023-10-18T20:16:20.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:16:20.813 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.813 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.814 DEBUG:boto:Params: {} 2023-10-18T20:16:20.814 DEBUG:boto:Token: None 2023-10-18T20:16:20.814 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj5 2023-10-18T20:16:20.815 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yLYDGP9UDIWO3n5DhEuhzNTptEI= 2023-10-18T20:16:20.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yLYDGP9UDIWO3n5DhEuhzNTptEI='} 2023-10-18T20:16:20.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053dff5a7d9735731-0065303d14-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.821 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:20.821 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:20.821 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.822 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:16:20.822 DEBUG:boto:Data: 2023-10-18T20:16:20.822 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.823 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.823 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.823 DEBUG:boto:Params: {} 2023-10-18T20:16:20.824 DEBUG:boto:Token: None 2023-10-18T20:16:20.824 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/ 2023-10-18T20:16:20.825 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jv4sNmRjA+vGimjX1U+ZqFxQ5N4= 2023-10-18T20:16:20.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jv4sNmRjA+vGimjX1U+ZqFxQ5N4=', 'Content-Length': '0'} 2023-10-18T20:16:20.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bd4f3f0246303a1-0065303d14-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.828 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:20.828 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:20.828 DEBUG:boto:Method: PUT 2023-10-18T20:16:20.829 DEBUG:boto:Path: /bofrnf-47/obj6 2023-10-18T20:16:20.829 DEBUG:boto:Data: 2023-10-18T20:16:20.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:16:20.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.830 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.831 DEBUG:boto:Params: {} 2023-10-18T20:16:20.831 DEBUG:boto:Token: None 2023-10-18T20:16:20.831 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj6 2023-10-18T20:16:20.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PGWf0N+hRTaamzjQ1gz6CWKY330= 2023-10-18T20:16:20.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PGWf0N+hRTaamzjQ1gz6CWKY330='} 2023-10-18T20:16:20.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea354b39f8ea2f23-0065303d14-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.837 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:20.838 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:20.838 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.839 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:16:20.839 DEBUG:boto:Data: 2023-10-18T20:16:20.839 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.840 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.840 DEBUG:boto:Params: {} 2023-10-18T20:16:20.841 DEBUG:boto:Token: None 2023-10-18T20:16:20.841 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/ 2023-10-18T20:16:20.841 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jv4sNmRjA+vGimjX1U+ZqFxQ5N4= 2023-10-18T20:16:20.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jv4sNmRjA+vGimjX1U+ZqFxQ5N4=', 'Content-Length': '0'} 2023-10-18T20:16:20.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084f44adcf0ef0ecd-0065303d14-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.844 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:20.845 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:20.845 DEBUG:boto:Method: PUT 2023-10-18T20:16:20.846 DEBUG:boto:Path: /bofrnf-47/obj7 2023-10-18T20:16:20.846 DEBUG:boto:Data: 2023-10-18T20:16:20.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:16:20.847 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.847 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.848 DEBUG:boto:Params: {} 2023-10-18T20:16:20.848 DEBUG:boto:Token: None 2023-10-18T20:16:20.848 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj7 2023-10-18T20:16:20.849 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QI8clVaX+TTsSNFyEkUtyY/evH8= 2023-10-18T20:16:20.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QI8clVaX+TTsSNFyEkUtyY/evH8='} 2023-10-18T20:16:20.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a67c68fdc9b2c9bc-0065303d14-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.895 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:20.895 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:20.896 DEBUG:boto:Method: HEAD 2023-10-18T20:16:20.896 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:16:20.897 DEBUG:boto:Data: 2023-10-18T20:16:20.897 DEBUG:boto:Headers: {} 2023-10-18T20:16:20.897 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.898 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.898 DEBUG:boto:Params: {} 2023-10-18T20:16:20.898 DEBUG:boto:Token: None 2023-10-18T20:16:20.899 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/ 2023-10-18T20:16:20.900 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jv4sNmRjA+vGimjX1U+ZqFxQ5N4= 2023-10-18T20:16:20.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jv4sNmRjA+vGimjX1U+ZqFxQ5N4=', 'Content-Length': '0'} 2023-10-18T20:16:20.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000869b09c93ed504b6-0065303d14-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.903 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:20.903 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:20.904 DEBUG:boto:Method: PUT 2023-10-18T20:16:20.904 DEBUG:boto:Path: /bofrnf-47/obj8 2023-10-18T20:16:20.904 DEBUG:boto:Data: 2023-10-18T20:16:20.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:16:20.905 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:20.905 DEBUG:boto:Port: 8001 2023-10-18T20:16:20.906 DEBUG:boto:Params: {} 2023-10-18T20:16:20.906 DEBUG:boto:Token: None 2023-10-18T20:16:20.907 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:16:20 GMT /bofrnf-47/obj8 2023-10-18T20:16:20.907 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8BS1iHyzUIFUYca8YmRBpJWNY24= 2023-10-18T20:16:20.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:16:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8BS1iHyzUIFUYca8YmRBpJWNY24='} 2023-10-18T20:16:20.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a7d02b72089198d-0065303d14-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:20.913 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-46 2023-10-18T20:16:20.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:16:20.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:16:20.914 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:16:20.970 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:20.971 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:21.004 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:21.003+0000 7f25829eb780 1 waiting for incremental sync to catch up: 2023-10-18T20:16:21.005 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.17.6, , 00000000001.26.6, , 00000000001.20.6, , , 00000000001.20.6, , , ] 2023-10-18T20:16:21.005 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.21.6, 00000000002.26.1, 00000000004.30.6, 00000000002.29.1, 00000000004.24.6, 00000000002.19.1, 00000000002.35.1, 00000000004.24.6, 00000000002.20.1, 00000000002.38.1, 00000000002.22.1] 2023-10-18T20:16:26.006 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:26.004+0000 7f25829eb780 1 waiting for incremental sync to catch up: 2023-10-18T20:16:26.006 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.17.6, , 00000000001.26.6, , 00000000001.20.6, , , 00000000001.20.6, , , ] 2023-10-18T20:16:26.007 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.21.6, 00000000002.26.1, 00000000004.30.6, 00000000002.29.1, 00000000004.24.6, 00000000002.19.1, 00000000002.35.1, 00000000004.24.6, 00000000002.20.1, 00000000002.38.1, 00000000002.22.1] 2023-10-18T20:16:31.007 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:31.006+0000 7f25829eb780 1 waiting for incremental sync to catch up: 2023-10-18T20:16:31.008 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.17.6, , 00000000001.26.6, , 00000000001.20.6, , , 00000000001.20.6, , , ] 2023-10-18T20:16:31.008 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.21.6, 00000000002.26.1, 00000000004.30.6, 00000000002.29.1, 00000000004.24.6, 00000000002.19.1, 00000000002.35.1, 00000000004.24.6, 00000000002.20.1, 00000000002.38.1, 00000000002.22.1] 2023-10-18T20:16:36.008 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:36.006+0000 7f25829eb780 1 bucket sync caught up with source: 2023-10-18T20:16:36.009 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.21.6, 00000000002.26.1, 00000000004.30.6, 00000000002.29.1, 00000000004.24.6, 00000000002.19.1, 00000000002.35.1, 00000000004.24.6, 00000000002.20.1, 00000000002.38.1, 00000000002.22.1] 2023-10-18T20:16:36.009 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.21.6, 00000000002.26.1, 00000000004.30.6, 00000000002.29.1, 00000000004.24.6, 00000000002.19.1, 00000000002.35.1, 00000000004.24.6, 00000000002.20.1, 00000000002.38.1, 00000000002.22.1] 2023-10-18T20:16:36.009 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:36.006+0000 7f25829eb780 0 bucket checkpoint complete 2023-10-18T20:16:36.021 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-46 2023-10-18T20:16:36.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:16:36.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:16:36.023 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:16:36.079 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:36.080 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:36.117 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:36.114+0000 7f3a568c0780 1 waiting for incremental sync to catch up: 2023-10-18T20:16:36.117 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.411.6, , 00000000001.909.6, , 00000000001.282.6, , , 00000000001.282.6, , , ] 2023-10-18T20:16:36.117 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.415.6, , 00000000002.913.6, , 00000000002.286.6, , , 00000000002.286.6, , , ] 2023-10-18T20:16:41.118 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:41.116+0000 7f3a568c0780 1 waiting for incremental sync to catch up: 2023-10-18T20:16:41.118 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000002.415.6, , 00000000001.909.6, , 00000000002.286.6, , , 00000000001.282.6, , , ] 2023-10-18T20:16:41.119 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.415.6, , 00000000002.913.6, , 00000000002.286.6, , , 00000000002.286.6, , , ] 2023-10-18T20:16:46.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:46.117+0000 7f3a568c0780 1 waiting for incremental sync to catch up: 2023-10-18T20:16:46.120 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000002.415.6, , 00000000001.909.6, , 00000000002.286.6, , , 00000000001.282.6, , , ] 2023-10-18T20:16:46.120 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.415.6, , 00000000002.913.6, , 00000000002.286.6, , , 00000000002.286.6, , , ] 2023-10-18T20:16:51.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:51.119+0000 7f3a568c0780 1 waiting for incremental sync to catch up: 2023-10-18T20:16:51.121 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000002.415.6, , 00000000001.909.6, , 00000000002.286.6, , , 00000000001.282.6, , , ] 2023-10-18T20:16:51.121 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.415.6, , 00000000002.913.6, , 00000000002.286.6, , , 00000000002.286.6, , , ] 2023-10-18T20:16:56.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:56.120+0000 7f3a568c0780 1 bucket sync caught up with source: 2023-10-18T20:16:56.122 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000002.415.6, , 00000000002.913.6, , 00000000002.286.6, , , 00000000002.286.6, , , ] 2023-10-18T20:16:56.122 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.415.6, , 00000000002.913.6, , 00000000002.286.6, , , 00000000002.286.6, , , ] 2023-10-18T20:16:56.123 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:56.120+0000 7f3a568c0780 0 bucket checkpoint complete 2023-10-18T20:16:56.138 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-46 zones={a2, a1} 2023-10-18T20:16:56.139 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.139 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.140 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.140 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:16:56.141 DEBUG:boto:Data: 2023-10-18T20:16:56.141 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.142 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.142 DEBUG:boto:Params: {} 2023-10-18T20:16:56.143 DEBUG:boto:Token: None 2023-10-18T20:16:56.143 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z6DfzX/AiMYwrVpkmD1w/YSNvsE= 2023-10-18T20:16:56.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z6DfzX/AiMYwrVpkmD1w/YSNvsE=', 'Content-Length': '0'} 2023-10-18T20:16:56.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afdfd337dfcb77c7-0065303d38-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.148 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.148 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.149 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.149 DEBUG:boto:Path: /bofrnf-46/ 2023-10-18T20:16:56.150 DEBUG:boto:Data: 2023-10-18T20:16:56.151 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.152 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.152 DEBUG:boto:Params: {} 2023-10-18T20:16:56.153 DEBUG:boto:Token: None 2023-10-18T20:16:56.153 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.154 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z6DfzX/AiMYwrVpkmD1w/YSNvsE= 2023-10-18T20:16:56.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z6DfzX/AiMYwrVpkmD1w/YSNvsE=', 'Content-Length': '0'} 2023-10-18T20:16:56.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8919ecfde60293a-0065303d38-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.157 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:16:56.158 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.158 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.159 DEBUG:boto:path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.159 DEBUG:boto:auth_path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.160 DEBUG:boto:Method: GET 2023-10-18T20:16:56.160 DEBUG:boto:Path: /bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.160 DEBUG:boto:Data: 2023-10-18T20:16:56.161 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.161 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.162 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.162 DEBUG:boto:Params: {} 2023-10-18T20:16:56.163 DEBUG:boto:Token: None 2023-10-18T20:16:56.163 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/?versions 2023-10-18T20:16:56.164 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YlMzzUw8aH3OLKzeNrDD4yGvnBE= 2023-10-18T20:16:56.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YlMzzUw8aH3OLKzeNrDD4yGvnBE=', 'Content-Length': '0'} 2023-10-18T20:16:56.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047d989829640d784-0065303d38-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.167 DEBUG:boto:b'bofrnf-46999falseobj1nulltrue2023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:16:20.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:16:20.681Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:16:20.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:16:20.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:16:56.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:16:56.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:16:56.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:16:56.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-18T20:16:56.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-18T20:16:56.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-18T20:16:56.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-18T20:16:56.171 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:16:56.172 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.172 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.173 DEBUG:boto:path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.173 DEBUG:boto:auth_path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.173 DEBUG:boto:Method: GET 2023-10-18T20:16:56.174 DEBUG:boto:Path: /bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.174 DEBUG:boto:Data: 2023-10-18T20:16:56.175 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.175 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.176 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.176 DEBUG:boto:Params: {} 2023-10-18T20:16:56.176 DEBUG:boto:Token: None 2023-10-18T20:16:56.177 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/?versions 2023-10-18T20:16:56.177 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YlMzzUw8aH3OLKzeNrDD4yGvnBE= 2023-10-18T20:16:56.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YlMzzUw8aH3OLKzeNrDD4yGvnBE=', 'Content-Length': '0'} 2023-10-18T20:16:56.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007ef1e212789e253-0065303d38-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.181 DEBUG:boto:b'bofrnf-46999falseobj1nulltrue2023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:16:20.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:16:20.681Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:16:20.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:16:20.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:16:56.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:16:56.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:16:56.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:16:56.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-18T20:16:56.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-18T20:16:56.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-18T20:16:56.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-18T20:16:56.186 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.187 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.187 DEBUG:boto:path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.188 DEBUG:boto:auth_path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.188 DEBUG:boto:Method: GET 2023-10-18T20:16:56.189 DEBUG:boto:Path: /bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.189 DEBUG:boto:Data: 2023-10-18T20:16:56.189 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.190 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.191 DEBUG:boto:Params: {} 2023-10-18T20:16:56.191 DEBUG:boto:Token: None 2023-10-18T20:16:56.192 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/?versions 2023-10-18T20:16:56.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YlMzzUw8aH3OLKzeNrDD4yGvnBE= 2023-10-18T20:16:56.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YlMzzUw8aH3OLKzeNrDD4yGvnBE=', 'Content-Length': '0'} 2023-10-18T20:16:56.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af09e63aa9af3d24-0065303d38-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.196 DEBUG:boto:b'bofrnf-46999falseobj1nulltrue2023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:16:20.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:16:20.681Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:16:20.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:16:20.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.197 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.197 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.197 DEBUG:boto:path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.198 DEBUG:boto:auth_path=/bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.198 DEBUG:boto:Method: GET 2023-10-18T20:16:56.199 DEBUG:boto:Path: /bofrnf-46/?versions&max-keys=999 2023-10-18T20:16:56.199 DEBUG:boto:Data: 2023-10-18T20:16:56.199 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.200 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.200 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.201 DEBUG:boto:Params: {} 2023-10-18T20:16:56.201 DEBUG:boto:Token: None 2023-10-18T20:16:56.202 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/?versions 2023-10-18T20:16:56.202 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YlMzzUw8aH3OLKzeNrDD4yGvnBE= 2023-10-18T20:16:56.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YlMzzUw8aH3OLKzeNrDD4yGvnBE=', 'Content-Length': '0'} 2023-10-18T20:16:56.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd391576087ef91a-0065303d38-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.206 DEBUG:boto:b'bofrnf-46999falseobj1nulltrue2023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:16:20.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:16:20.681Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:16:20.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:16:20.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:16:56.207 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.208 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.208 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.208 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.209 DEBUG:boto:Method: GET 2023-10-18T20:16:56.209 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.210 DEBUG:boto:Data: 2023-10-18T20:16:56.210 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.211 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.211 DEBUG:boto:Params: {} 2023-10-18T20:16:56.212 DEBUG:boto:Token: None 2023-10-18T20:16:56.212 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JGusK0Ori/KGhAciZ2J58kyMdg4= 2023-10-18T20:16:56.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JGusK0Ori/KGhAciZ2J58kyMdg4=', 'Content-Length': '0'} 2023-10-18T20:16:56.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5874614b1f9c876-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.258 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.259 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.259 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.260 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.260 DEBUG:boto:Method: GET 2023-10-18T20:16:56.261 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.261 DEBUG:boto:Data: 2023-10-18T20:16:56.262 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.262 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.262 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.263 DEBUG:boto:Params: {} 2023-10-18T20:16:56.263 DEBUG:boto:Token: None 2023-10-18T20:16:56.264 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.264 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JGusK0Ori/KGhAciZ2J58kyMdg4= 2023-10-18T20:16:56.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JGusK0Ori/KGhAciZ2J58kyMdg4=', 'Content-Length': '0'} 2023-10-18T20:16:56.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074088810e819c72d-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.268 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.268 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.269 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.269 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.269 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.270 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.270 DEBUG:boto:Data: 2023-10-18T20:16:56.271 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.271 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.271 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.272 DEBUG:boto:Params: {} 2023-10-18T20:16:56.272 DEBUG:boto:Token: None 2023-10-18T20:16:56.273 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.273 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e7sK3Rj3jDnvdFZq/1IhCEOPb/I= 2023-10-18T20:16:56.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e7sK3Rj3jDnvdFZq/1IhCEOPb/I=', 'Content-Length': '0'} 2023-10-18T20:16:56.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077c1a8d1375fc424-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.276 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.277 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.277 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.278 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.278 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.278 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.279 DEBUG:boto:Data: 2023-10-18T20:16:56.279 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.280 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.280 DEBUG:boto:Params: {} 2023-10-18T20:16:56.281 DEBUG:boto:Token: None 2023-10-18T20:16:56.281 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.282 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e7sK3Rj3jDnvdFZq/1IhCEOPb/I= 2023-10-18T20:16:56.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e7sK3Rj3jDnvdFZq/1IhCEOPb/I=', 'Content-Length': '0'} 2023-10-18T20:16:56.284 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8e238a834666ba2-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:16:56.285 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.285 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.286 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.286 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.287 DEBUG:boto:Method: GET 2023-10-18T20:16:56.287 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.288 DEBUG:boto:Data: 2023-10-18T20:16:56.288 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.289 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.289 DEBUG:boto:Params: {} 2023-10-18T20:16:56.290 DEBUG:boto:Token: None 2023-10-18T20:16:56.290 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JGusK0Ori/KGhAciZ2J58kyMdg4= 2023-10-18T20:16:56.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JGusK0Ori/KGhAciZ2J58kyMdg4=', 'Content-Length': '0'} 2023-10-18T20:16:56.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a0d35a30a45af8d-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.294 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.294 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.295 DEBUG:boto:path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.295 DEBUG:boto:auth_path=/bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.296 DEBUG:boto:Method: GET 2023-10-18T20:16:56.296 DEBUG:boto:Path: /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.296 DEBUG:boto:Data: 2023-10-18T20:16:56.297 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.297 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.298 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.298 DEBUG:boto:Params: {} 2023-10-18T20:16:56.299 DEBUG:boto:Token: None 2023-10-18T20:16:56.299 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1?versionId=null 2023-10-18T20:16:56.300 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JGusK0Ori/KGhAciZ2J58kyMdg4= 2023-10-18T20:16:56.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JGusK0Ori/KGhAciZ2J58kyMdg4=', 'Content-Length': '0'} 2023-10-18T20:16:56.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086d0e005db05e7ea-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.303 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.303 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.304 DEBUG:boto:path=/bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.304 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.304 DEBUG:boto:Method: GET 2023-10-18T20:16:56.305 DEBUG:boto:Path: /bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.305 DEBUG:boto:Data: 2023-10-18T20:16:56.306 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.306 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.307 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.307 DEBUG:boto:Params: {} 2023-10-18T20:16:56.307 DEBUG:boto:Token: None 2023-10-18T20:16:56.308 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.308 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c73b5d2077d3e9be-0065303d38-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.312 DEBUG:boto:b'bofrnf-46obj11000falseobj12023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.313 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.313 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.314 DEBUG:boto:path=/bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.314 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.315 DEBUG:boto:Method: GET 2023-10-18T20:16:56.315 DEBUG:boto:Path: /bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.315 DEBUG:boto:Data: 2023-10-18T20:16:56.316 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.316 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.317 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.317 DEBUG:boto:Params: {} 2023-10-18T20:16:56.317 DEBUG:boto:Token: None 2023-10-18T20:16:56.318 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.318 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1af358d250ff8ea-0065303d38-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.322 DEBUG:boto:b'bofrnf-46obj11000falseobj12023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.322 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.323 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.323 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.324 DEBUG:boto:Path: /bofrnf-46/obj1 2023-10-18T20:16:56.324 DEBUG:boto:Data: 2023-10-18T20:16:56.324 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.325 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.325 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.325 DEBUG:boto:Params: {} 2023-10-18T20:16:56.326 DEBUG:boto:Token: None 2023-10-18T20:16:56.326 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1 2023-10-18T20:16:56.327 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2vAASdCjEWlC0Rf6yMwkMa6mNDA= 2023-10-18T20:16:56.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2vAASdCjEWlC0Rf6yMwkMa6mNDA=', 'Content-Length': '0'} 2023-10-18T20:16:56.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099c07a19f02102c4-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.330 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.330 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.331 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.331 DEBUG:boto:Path: /bofrnf-46/obj1 2023-10-18T20:16:56.331 DEBUG:boto:Data: 2023-10-18T20:16:56.332 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.332 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.333 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.333 DEBUG:boto:Params: {} 2023-10-18T20:16:56.333 DEBUG:boto:Token: None 2023-10-18T20:16:56.334 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1 2023-10-18T20:16:56.334 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2vAASdCjEWlC0Rf6yMwkMa6mNDA= 2023-10-18T20:16:56.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2vAASdCjEWlC0Rf6yMwkMa6mNDA=', 'Content-Length': '0'} 2023-10-18T20:16:56.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006555c4392e3593a9-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:16:56.338 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.338 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.339 DEBUG:boto:Method: GET 2023-10-18T20:16:56.339 DEBUG:boto:Path: /bofrnf-46/obj1 2023-10-18T20:16:56.340 DEBUG:boto:Data: 2023-10-18T20:16:56.340 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.341 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.341 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.341 DEBUG:boto:Params: {} 2023-10-18T20:16:56.342 DEBUG:boto:Token: None 2023-10-18T20:16:56.342 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1 2023-10-18T20:16:56.343 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9ciRo+vJ0+i+EgR76NSGhxGBuRE= 2023-10-18T20:16:56.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9ciRo+vJ0+i+EgR76NSGhxGBuRE=', 'Content-Length': '0'} 2023-10-18T20:16:56.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000817cf84fe132c5d7-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.346 DEBUG:boto:path=/bofrnf-46/obj1 2023-10-18T20:16:56.346 DEBUG:boto:auth_path=/bofrnf-46/obj1 2023-10-18T20:16:56.347 DEBUG:boto:Method: GET 2023-10-18T20:16:56.347 DEBUG:boto:Path: /bofrnf-46/obj1 2023-10-18T20:16:56.348 DEBUG:boto:Data: 2023-10-18T20:16:56.348 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.348 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.349 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.349 DEBUG:boto:Params: {} 2023-10-18T20:16:56.350 DEBUG:boto:Token: None 2023-10-18T20:16:56.350 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj1 2023-10-18T20:16:56.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9ciRo+vJ0+i+EgR76NSGhxGBuRE= 2023-10-18T20:16:56.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9ciRo+vJ0+i+EgR76NSGhxGBuRE=', 'Content-Length': '0'} 2023-10-18T20:16:56.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e267e99e9e9c717-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.354 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.354 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.355 DEBUG:boto:path=/bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.355 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.355 DEBUG:boto:Method: GET 2023-10-18T20:16:56.356 DEBUG:boto:Path: /bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.356 DEBUG:boto:Data: 2023-10-18T20:16:56.357 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.357 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.357 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.358 DEBUG:boto:Params: {} 2023-10-18T20:16:56.358 DEBUG:boto:Token: None 2023-10-18T20:16:56.359 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.359 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d16bf6adc873bf99-0065303d38-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.363 DEBUG:boto:b'bofrnf-46obj11000falseobj12023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.363 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.364 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.364 DEBUG:boto:path=/bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.365 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.365 DEBUG:boto:Method: GET 2023-10-18T20:16:56.366 DEBUG:boto:Path: /bofrnf-46/?prefix=obj1 2023-10-18T20:16:56.366 DEBUG:boto:Data: 2023-10-18T20:16:56.366 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.367 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.368 DEBUG:boto:Params: {} 2023-10-18T20:16:56.368 DEBUG:boto:Token: None 2023-10-18T20:16:56.369 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067ef558d03f7949c-0065303d38-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.373 DEBUG:boto:b'bofrnf-46obj11000falseobj12023-10-18T20:14:17.358Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:16:56.374 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.374 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.375 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.375 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.375 DEBUG:boto:Method: GET 2023-10-18T20:16:56.376 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.376 DEBUG:boto:Data: 2023-10-18T20:16:56.377 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.378 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.378 DEBUG:boto:Params: {} 2023-10-18T20:16:56.378 DEBUG:boto:Token: None 2023-10-18T20:16:56.379 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.379 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iKyOIQbofRNQ+lTxQVQFNomyIc0= 2023-10-18T20:16:56.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iKyOIQbofRNQ+lTxQVQFNomyIc0=', 'Content-Length': '0'} 2023-10-18T20:16:56.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000326085c7745c7923-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.422 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.423 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.423 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.424 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.424 DEBUG:boto:Method: GET 2023-10-18T20:16:56.425 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.425 DEBUG:boto:Data: 2023-10-18T20:16:56.426 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.426 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.427 DEBUG:boto:Params: {} 2023-10-18T20:16:56.427 DEBUG:boto:Token: None 2023-10-18T20:16:56.428 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iKyOIQbofRNQ+lTxQVQFNomyIc0= 2023-10-18T20:16:56.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iKyOIQbofRNQ+lTxQVQFNomyIc0=', 'Content-Length': '0'} 2023-10-18T20:16:56.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009133994e605fdcac-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.431 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.432 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.432 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.433 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.433 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.433 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.434 DEBUG:boto:Data: 2023-10-18T20:16:56.434 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.435 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.435 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.435 DEBUG:boto:Params: {} 2023-10-18T20:16:56.436 DEBUG:boto:Token: None 2023-10-18T20:16:56.436 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.437 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDEPT9YS3vkdA3w/3m5I/VFAVkw= 2023-10-18T20:16:56.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDEPT9YS3vkdA3w/3m5I/VFAVkw=', 'Content-Length': '0'} 2023-10-18T20:16:56.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba1eb18d81bd8780-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.440 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.440 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.441 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.441 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.441 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.442 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.442 DEBUG:boto:Data: 2023-10-18T20:16:56.443 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.443 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.443 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.444 DEBUG:boto:Params: {} 2023-10-18T20:16:56.444 DEBUG:boto:Token: None 2023-10-18T20:16:56.445 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.445 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDEPT9YS3vkdA3w/3m5I/VFAVkw= 2023-10-18T20:16:56.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDEPT9YS3vkdA3w/3m5I/VFAVkw=', 'Content-Length': '0'} 2023-10-18T20:16:56.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000300b0cb80ab489c3-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:16:56.449 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.449 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.449 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.450 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.450 DEBUG:boto:Method: GET 2023-10-18T20:16:56.451 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.451 DEBUG:boto:Data: 2023-10-18T20:16:56.452 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.452 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.453 DEBUG:boto:Params: {} 2023-10-18T20:16:56.453 DEBUG:boto:Token: None 2023-10-18T20:16:56.454 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.454 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iKyOIQbofRNQ+lTxQVQFNomyIc0= 2023-10-18T20:16:56.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iKyOIQbofRNQ+lTxQVQFNomyIc0=', 'Content-Length': '0'} 2023-10-18T20:16:56.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a380080c47232c5d-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.457 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.458 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.458 DEBUG:boto:path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.459 DEBUG:boto:auth_path=/bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.459 DEBUG:boto:Method: GET 2023-10-18T20:16:56.460 DEBUG:boto:Path: /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.460 DEBUG:boto:Data: 2023-10-18T20:16:56.460 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.461 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.461 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.462 DEBUG:boto:Params: {} 2023-10-18T20:16:56.462 DEBUG:boto:Token: None 2023-10-18T20:16:56.463 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2?versionId=null 2023-10-18T20:16:56.463 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iKyOIQbofRNQ+lTxQVQFNomyIc0= 2023-10-18T20:16:56.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iKyOIQbofRNQ+lTxQVQFNomyIc0=', 'Content-Length': '0'} 2023-10-18T20:16:56.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d66ea2a419385bd8-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.510 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.511 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.511 DEBUG:boto:path=/bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.512 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.512 DEBUG:boto:Method: GET 2023-10-18T20:16:56.513 DEBUG:boto:Path: /bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.513 DEBUG:boto:Data: 2023-10-18T20:16:56.513 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.514 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.514 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.515 DEBUG:boto:Params: {} 2023-10-18T20:16:56.515 DEBUG:boto:Token: None 2023-10-18T20:16:56.516 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071406699311fafa2-0065303d38-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.520 DEBUG:boto:b'bofrnf-46obj21000falseobj22023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.521 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.521 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.522 DEBUG:boto:path=/bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.522 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.522 DEBUG:boto:Method: GET 2023-10-18T20:16:56.523 DEBUG:boto:Path: /bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.523 DEBUG:boto:Data: 2023-10-18T20:16:56.524 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.524 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.524 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.525 DEBUG:boto:Params: {} 2023-10-18T20:16:56.525 DEBUG:boto:Token: None 2023-10-18T20:16:56.526 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.526 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032b02f55cf276826-0065303d38-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.530 DEBUG:boto:b'bofrnf-46obj21000falseobj22023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.531 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.531 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.531 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.532 DEBUG:boto:Path: /bofrnf-46/obj2 2023-10-18T20:16:56.532 DEBUG:boto:Data: 2023-10-18T20:16:56.533 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.533 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.533 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.534 DEBUG:boto:Params: {} 2023-10-18T20:16:56.534 DEBUG:boto:Token: None 2023-10-18T20:16:56.535 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2 2023-10-18T20:16:56.535 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F7N2XACfdtdogMWybATaxWF+urc= 2023-10-18T20:16:56.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F7N2XACfdtdogMWybATaxWF+urc=', 'Content-Length': '0'} 2023-10-18T20:16:56.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000136b521f880c312c-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.538 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.539 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.539 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.539 DEBUG:boto:Path: /bofrnf-46/obj2 2023-10-18T20:16:56.540 DEBUG:boto:Data: 2023-10-18T20:16:56.540 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.541 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.541 DEBUG:boto:Params: {} 2023-10-18T20:16:56.542 DEBUG:boto:Token: None 2023-10-18T20:16:56.542 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2 2023-10-18T20:16:56.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F7N2XACfdtdogMWybATaxWF+urc= 2023-10-18T20:16:56.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F7N2XACfdtdogMWybATaxWF+urc=', 'Content-Length': '0'} 2023-10-18T20:16:56.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005788d15950aeb0c1-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:16:56.546 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.546 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.547 DEBUG:boto:Method: GET 2023-10-18T20:16:56.547 DEBUG:boto:Path: /bofrnf-46/obj2 2023-10-18T20:16:56.548 DEBUG:boto:Data: 2023-10-18T20:16:56.548 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.548 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.549 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.549 DEBUG:boto:Params: {} 2023-10-18T20:16:56.550 DEBUG:boto:Token: None 2023-10-18T20:16:56.550 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2 2023-10-18T20:16:56.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rH+qDW4U9lxn5mZE60GJtOAJNYw= 2023-10-18T20:16:56.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rH+qDW4U9lxn5mZE60GJtOAJNYw=', 'Content-Length': '0'} 2023-10-18T20:16:56.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007863d7a76438d0a6-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.597 DEBUG:boto:path=/bofrnf-46/obj2 2023-10-18T20:16:56.598 DEBUG:boto:auth_path=/bofrnf-46/obj2 2023-10-18T20:16:56.598 DEBUG:boto:Method: GET 2023-10-18T20:16:56.599 DEBUG:boto:Path: /bofrnf-46/obj2 2023-10-18T20:16:56.599 DEBUG:boto:Data: 2023-10-18T20:16:56.599 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.600 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.600 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.601 DEBUG:boto:Params: {} 2023-10-18T20:16:56.601 DEBUG:boto:Token: None 2023-10-18T20:16:56.602 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj2 2023-10-18T20:16:56.602 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rH+qDW4U9lxn5mZE60GJtOAJNYw= 2023-10-18T20:16:56.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rH+qDW4U9lxn5mZE60GJtOAJNYw=', 'Content-Length': '0'} 2023-10-18T20:16:56.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000643583d9d9f21866-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.605 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.606 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.606 DEBUG:boto:path=/bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.607 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.607 DEBUG:boto:Method: GET 2023-10-18T20:16:56.607 DEBUG:boto:Path: /bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.608 DEBUG:boto:Data: 2023-10-18T20:16:56.608 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.609 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.609 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.609 DEBUG:boto:Params: {} 2023-10-18T20:16:56.610 DEBUG:boto:Token: None 2023-10-18T20:16:56.610 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.611 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074aa8c21ff03a7bd-0065303d38-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.614 DEBUG:boto:b'bofrnf-46obj21000falseobj22023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.615 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.615 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.616 DEBUG:boto:path=/bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.616 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.617 DEBUG:boto:Method: GET 2023-10-18T20:16:56.617 DEBUG:boto:Path: /bofrnf-46/?prefix=obj2 2023-10-18T20:16:56.617 DEBUG:boto:Data: 2023-10-18T20:16:56.618 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.618 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.619 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.619 DEBUG:boto:Params: {} 2023-10-18T20:16:56.620 DEBUG:boto:Token: None 2023-10-18T20:16:56.620 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.621 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039396c7fed1f9afe-0065303d38-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.624 DEBUG:boto:b'bofrnf-46obj21000falseobj22023-10-18T20:14:17.380Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:16:56.625 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.626 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.626 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.626 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.627 DEBUG:boto:Method: GET 2023-10-18T20:16:56.627 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.628 DEBUG:boto:Data: 2023-10-18T20:16:56.628 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.629 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.629 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.629 DEBUG:boto:Params: {} 2023-10-18T20:16:56.630 DEBUG:boto:Token: None 2023-10-18T20:16:56.630 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.631 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:btL8qIRY6niit9M2sZEsv8MXlDk= 2023-10-18T20:16:56.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:btL8qIRY6niit9M2sZEsv8MXlDk=', 'Content-Length': '0'} 2023-10-18T20:16:56.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec4a68a061102467-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.634 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.634 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.635 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.635 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.636 DEBUG:boto:Method: GET 2023-10-18T20:16:56.636 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.636 DEBUG:boto:Data: 2023-10-18T20:16:56.637 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.637 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.638 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.638 DEBUG:boto:Params: {} 2023-10-18T20:16:56.639 DEBUG:boto:Token: None 2023-10-18T20:16:56.639 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.640 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:btL8qIRY6niit9M2sZEsv8MXlDk= 2023-10-18T20:16:56.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:btL8qIRY6niit9M2sZEsv8MXlDk=', 'Content-Length': '0'} 2023-10-18T20:16:56.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dff270274b7c181e-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.643 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.643 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.643 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.644 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.644 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.645 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.645 DEBUG:boto:Data: 2023-10-18T20:16:56.645 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.646 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.646 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.647 DEBUG:boto:Params: {} 2023-10-18T20:16:56.647 DEBUG:boto:Token: None 2023-10-18T20:16:56.647 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.648 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rKJ09sVfLmLkhBiZDOcPEFu7TH0= 2023-10-18T20:16:56.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rKJ09sVfLmLkhBiZDOcPEFu7TH0=', 'Content-Length': '0'} 2023-10-18T20:16:56.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2c7078906660d2b-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.651 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.651 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.652 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.652 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.652 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.653 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.653 DEBUG:boto:Data: 2023-10-18T20:16:56.654 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.654 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.655 DEBUG:boto:Params: {} 2023-10-18T20:16:56.655 DEBUG:boto:Token: None 2023-10-18T20:16:56.656 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rKJ09sVfLmLkhBiZDOcPEFu7TH0= 2023-10-18T20:16:56.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rKJ09sVfLmLkhBiZDOcPEFu7TH0=', 'Content-Length': '0'} 2023-10-18T20:16:56.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e53ec00f608851c-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:16:56.659 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.660 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.660 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.661 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.661 DEBUG:boto:Method: GET 2023-10-18T20:16:56.661 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.662 DEBUG:boto:Data: 2023-10-18T20:16:56.662 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.663 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.663 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.664 DEBUG:boto:Params: {} 2023-10-18T20:16:56.664 DEBUG:boto:Token: None 2023-10-18T20:16:56.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:btL8qIRY6niit9M2sZEsv8MXlDk= 2023-10-18T20:16:56.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:btL8qIRY6niit9M2sZEsv8MXlDk=', 'Content-Length': '0'} 2023-10-18T20:16:56.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004eb21502603bb18a-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.668 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.669 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.669 DEBUG:boto:path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.670 DEBUG:boto:auth_path=/bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.670 DEBUG:boto:Method: GET 2023-10-18T20:16:56.670 DEBUG:boto:Path: /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.671 DEBUG:boto:Data: 2023-10-18T20:16:56.671 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.672 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.672 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.673 DEBUG:boto:Params: {} 2023-10-18T20:16:56.673 DEBUG:boto:Token: None 2023-10-18T20:16:56.673 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3?versionId=null 2023-10-18T20:16:56.674 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:btL8qIRY6niit9M2sZEsv8MXlDk= 2023-10-18T20:16:56.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:btL8qIRY6niit9M2sZEsv8MXlDk=', 'Content-Length': '0'} 2023-10-18T20:16:56.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da1b2791f11c9669-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.722 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.723 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.723 DEBUG:boto:path=/bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.724 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.724 DEBUG:boto:Method: GET 2023-10-18T20:16:56.725 DEBUG:boto:Path: /bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.725 DEBUG:boto:Data: 2023-10-18T20:16:56.725 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.726 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.726 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.727 DEBUG:boto:Params: {} 2023-10-18T20:16:56.727 DEBUG:boto:Token: None 2023-10-18T20:16:56.728 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c72dbedffbebba0-0065303d38-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.732 DEBUG:boto:b'bofrnf-46obj31000falseobj32023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.732 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.733 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.733 DEBUG:boto:path=/bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.734 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.734 DEBUG:boto:Method: GET 2023-10-18T20:16:56.735 DEBUG:boto:Path: /bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.735 DEBUG:boto:Data: 2023-10-18T20:16:56.735 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.736 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.736 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.737 DEBUG:boto:Params: {} 2023-10-18T20:16:56.737 DEBUG:boto:Token: None 2023-10-18T20:16:56.738 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.738 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dccec48fa260851-0065303d38-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.742 DEBUG:boto:b'bofrnf-46obj31000falseobj32023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.742 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.743 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.743 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.744 DEBUG:boto:Path: /bofrnf-46/obj3 2023-10-18T20:16:56.744 DEBUG:boto:Data: 2023-10-18T20:16:56.744 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.745 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.745 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.745 DEBUG:boto:Params: {} 2023-10-18T20:16:56.746 DEBUG:boto:Token: None 2023-10-18T20:16:56.746 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3 2023-10-18T20:16:56.747 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5YabTl4/12wEKdlpKh/5E5uewLM= 2023-10-18T20:16:56.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5YabTl4/12wEKdlpKh/5E5uewLM=', 'Content-Length': '0'} 2023-10-18T20:16:56.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000965235df8559e75-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.750 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.750 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.751 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.751 DEBUG:boto:Path: /bofrnf-46/obj3 2023-10-18T20:16:56.752 DEBUG:boto:Data: 2023-10-18T20:16:56.752 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.752 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.753 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.753 DEBUG:boto:Params: {} 2023-10-18T20:16:56.753 DEBUG:boto:Token: None 2023-10-18T20:16:56.754 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3 2023-10-18T20:16:56.754 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5YabTl4/12wEKdlpKh/5E5uewLM= 2023-10-18T20:16:56.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5YabTl4/12wEKdlpKh/5E5uewLM=', 'Content-Length': '0'} 2023-10-18T20:16:56.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000933aa8349309df3c-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:16:56.758 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.758 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.759 DEBUG:boto:Method: GET 2023-10-18T20:16:56.759 DEBUG:boto:Path: /bofrnf-46/obj3 2023-10-18T20:16:56.760 DEBUG:boto:Data: 2023-10-18T20:16:56.760 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.760 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.761 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.761 DEBUG:boto:Params: {} 2023-10-18T20:16:56.762 DEBUG:boto:Token: None 2023-10-18T20:16:56.762 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3 2023-10-18T20:16:56.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AvRcIh8o5P7O4tbO5Js/VScMOls= 2023-10-18T20:16:56.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AvRcIh8o5P7O4tbO5Js/VScMOls=', 'Content-Length': '0'} 2023-10-18T20:16:56.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032389bdbfc1f5867-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.806 DEBUG:boto:path=/bofrnf-46/obj3 2023-10-18T20:16:56.807 DEBUG:boto:auth_path=/bofrnf-46/obj3 2023-10-18T20:16:56.808 DEBUG:boto:Method: GET 2023-10-18T20:16:56.808 DEBUG:boto:Path: /bofrnf-46/obj3 2023-10-18T20:16:56.808 DEBUG:boto:Data: 2023-10-18T20:16:56.809 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.810 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.810 DEBUG:boto:Params: {} 2023-10-18T20:16:56.811 DEBUG:boto:Token: None 2023-10-18T20:16:56.811 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj3 2023-10-18T20:16:56.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AvRcIh8o5P7O4tbO5Js/VScMOls= 2023-10-18T20:16:56.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AvRcIh8o5P7O4tbO5Js/VScMOls=', 'Content-Length': '0'} 2023-10-18T20:16:56.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058c1dea1dcf8a277-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.815 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.815 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.816 DEBUG:boto:path=/bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.816 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.817 DEBUG:boto:Method: GET 2023-10-18T20:16:56.817 DEBUG:boto:Path: /bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.817 DEBUG:boto:Data: 2023-10-18T20:16:56.818 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.818 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.819 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.819 DEBUG:boto:Params: {} 2023-10-18T20:16:56.820 DEBUG:boto:Token: None 2023-10-18T20:16:56.820 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50dbeb97e9568bc-0065303d38-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.824 DEBUG:boto:b'bofrnf-46obj31000falseobj32023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.825 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.825 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.826 DEBUG:boto:path=/bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.826 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.827 DEBUG:boto:Method: GET 2023-10-18T20:16:56.827 DEBUG:boto:Path: /bofrnf-46/?prefix=obj3 2023-10-18T20:16:56.827 DEBUG:boto:Data: 2023-10-18T20:16:56.828 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.828 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.829 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.829 DEBUG:boto:Params: {} 2023-10-18T20:16:56.829 DEBUG:boto:Token: None 2023-10-18T20:16:56.830 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.830 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c703ce05f989fc5c-0065303d38-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.834 DEBUG:boto:b'bofrnf-46obj31000falseobj32023-10-18T20:14:17.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:16:56.835 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:56.835 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:56.836 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.836 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.837 DEBUG:boto:Method: GET 2023-10-18T20:16:56.837 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.838 DEBUG:boto:Data: 2023-10-18T20:16:56.838 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.839 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.839 DEBUG:boto:Params: {} 2023-10-18T20:16:56.840 DEBUG:boto:Token: None 2023-10-18T20:16:56.840 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.841 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YT/RJKBzIg4o6PbvC9EAS2BiFOc= 2023-10-18T20:16:56.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YT/RJKBzIg4o6PbvC9EAS2BiFOc=', 'Content-Length': '0'} 2023-10-18T20:16:56.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b6814340e6ecd91-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.844 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:56.844 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:56.845 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.845 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.846 DEBUG:boto:Method: GET 2023-10-18T20:16:56.846 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.846 DEBUG:boto:Data: 2023-10-18T20:16:56.847 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.847 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.848 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.848 DEBUG:boto:Params: {} 2023-10-18T20:16:56.849 DEBUG:boto:Token: None 2023-10-18T20:16:56.849 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.850 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YT/RJKBzIg4o6PbvC9EAS2BiFOc= 2023-10-18T20:16:56.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YT/RJKBzIg4o6PbvC9EAS2BiFOc=', 'Content-Length': '0'} 2023-10-18T20:16:56.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b2a2bceae9c56d2-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.853 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:56.853 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:56.853 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.854 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.854 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.855 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.855 DEBUG:boto:Data: 2023-10-18T20:16:56.855 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.856 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.856 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.857 DEBUG:boto:Params: {} 2023-10-18T20:16:56.857 DEBUG:boto:Token: None 2023-10-18T20:16:56.858 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.858 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4pHrqyURYMUx4sKqFbV/dg2Mz9s= 2023-10-18T20:16:56.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4pHrqyURYMUx4sKqFbV/dg2Mz9s=', 'Content-Length': '0'} 2023-10-18T20:16:56.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004aa558cd303de56b-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.861 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:56.862 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:56.862 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.862 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.863 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.863 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.864 DEBUG:boto:Data: 2023-10-18T20:16:56.864 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.864 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.865 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.865 DEBUG:boto:Params: {} 2023-10-18T20:16:56.866 DEBUG:boto:Token: None 2023-10-18T20:16:56.866 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4pHrqyURYMUx4sKqFbV/dg2Mz9s= 2023-10-18T20:16:56.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4pHrqyURYMUx4sKqFbV/dg2Mz9s=', 'Content-Length': '0'} 2023-10-18T20:16:56.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031c6c5f81ac3d80a-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:16:56.870 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:56.870 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:56.871 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.871 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.872 DEBUG:boto:Method: GET 2023-10-18T20:16:56.872 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.872 DEBUG:boto:Data: 2023-10-18T20:16:56.873 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.874 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.874 DEBUG:boto:Params: {} 2023-10-18T20:16:56.875 DEBUG:boto:Token: None 2023-10-18T20:16:56.875 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.875 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YT/RJKBzIg4o6PbvC9EAS2BiFOc= 2023-10-18T20:16:56.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YT/RJKBzIg4o6PbvC9EAS2BiFOc=', 'Content-Length': '0'} 2023-10-18T20:16:56.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f1826a0d8c1fc45-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.922 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:56.923 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:56.923 DEBUG:boto:path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.924 DEBUG:boto:auth_path=/bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.924 DEBUG:boto:Method: GET 2023-10-18T20:16:56.925 DEBUG:boto:Path: /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.925 DEBUG:boto:Data: 2023-10-18T20:16:56.925 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.926 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.926 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.927 DEBUG:boto:Params: {} 2023-10-18T20:16:56.927 DEBUG:boto:Token: None 2023-10-18T20:16:56.928 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj4?versionId=null 2023-10-18T20:16:56.928 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YT/RJKBzIg4o6PbvC9EAS2BiFOc= 2023-10-18T20:16:56.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YT/RJKBzIg4o6PbvC9EAS2BiFOc=', 'Content-Length': '0'} 2023-10-18T20:16:56.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008faea06ae4937eda-0065303d38-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.931 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.932 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.932 DEBUG:boto:path=/bofrnf-46/?prefix=obj4 2023-10-18T20:16:56.932 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj4 2023-10-18T20:16:56.933 DEBUG:boto:Method: GET 2023-10-18T20:16:56.933 DEBUG:boto:Path: /bofrnf-46/?prefix=obj4 2023-10-18T20:16:56.934 DEBUG:boto:Data: 2023-10-18T20:16:56.934 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.935 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.935 DEBUG:boto:Params: {} 2023-10-18T20:16:56.936 DEBUG:boto:Token: None 2023-10-18T20:16:56.936 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.937 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed0e8daa78b6f814-0065303d38-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.940 DEBUG:boto:b'bofrnf-46obj41000falseobj42023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.941 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:56.941 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:56.942 DEBUG:boto:path=/bofrnf-46/?prefix=obj4 2023-10-18T20:16:56.942 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj4 2023-10-18T20:16:56.943 DEBUG:boto:Method: GET 2023-10-18T20:16:56.943 DEBUG:boto:Path: /bofrnf-46/?prefix=obj4 2023-10-18T20:16:56.944 DEBUG:boto:Data: 2023-10-18T20:16:56.944 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.944 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:56.945 DEBUG:boto:Port: 8000 2023-10-18T20:16:56.945 DEBUG:boto:Params: {} 2023-10-18T20:16:56.946 DEBUG:boto:Token: None 2023-10-18T20:16:56.946 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/ 2023-10-18T20:16:56.947 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg= 2023-10-18T20:16:56.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vioYAAkKA0C8rH2cqZqPw9Ma3Cg=', 'Content-Length': '0'} 2023-10-18T20:16:56.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f5d8ecd511fc7e8-0065303d38-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.990 DEBUG:boto:b'bofrnf-46obj41000falseobj42023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:56.991 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:56.991 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:56.992 DEBUG:boto:Method: HEAD 2023-10-18T20:16:56.992 DEBUG:boto:Path: /bofrnf-46/obj4 2023-10-18T20:16:56.993 DEBUG:boto:Data: 2023-10-18T20:16:56.993 DEBUG:boto:Headers: {} 2023-10-18T20:16:56.993 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:56.994 DEBUG:boto:Port: 8001 2023-10-18T20:16:56.994 DEBUG:boto:Params: {} 2023-10-18T20:16:56.995 DEBUG:boto:Token: None 2023-10-18T20:16:56.995 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:56 GMT /bofrnf-46/obj4 2023-10-18T20:16:56.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hNlmYp6IqDL7zwG0HKWaAmS9LR0= 2023-10-18T20:16:56.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hNlmYp6IqDL7zwG0HKWaAmS9LR0=', 'Content-Length': '0'} 2023-10-18T20:16:56.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c8a87090325f8b5-0065303d38-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:56.998 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:56.999 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:56.999 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.000 DEBUG:boto:Path: /bofrnf-46/obj4 2023-10-18T20:16:57.000 DEBUG:boto:Data: 2023-10-18T20:16:57.001 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.001 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.002 DEBUG:boto:Params: {} 2023-10-18T20:16:57.002 DEBUG:boto:Token: None 2023-10-18T20:16:57.003 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj4 2023-10-18T20:16:57.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zF09jTek12eUkdyyuq+Rg4OFTaY= 2023-10-18T20:16:57.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zF09jTek12eUkdyyuq+Rg4OFTaY=', 'Content-Length': '0'} 2023-10-18T20:16:57.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d03eeaea6474d78-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:16:57.008 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:57.009 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:57.009 DEBUG:boto:Method: GET 2023-10-18T20:16:57.010 DEBUG:boto:Path: /bofrnf-46/obj4 2023-10-18T20:16:57.010 DEBUG:boto:Data: 2023-10-18T20:16:57.010 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.011 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.011 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.012 DEBUG:boto:Params: {} 2023-10-18T20:16:57.012 DEBUG:boto:Token: None 2023-10-18T20:16:57.013 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj4 2023-10-18T20:16:57.013 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DrNHWcG5Wlpghc483DroKEfNzvk= 2023-10-18T20:16:57.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DrNHWcG5Wlpghc483DroKEfNzvk=', 'Content-Length': '0'} 2023-10-18T20:16:57.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053d1432fcfe05872-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.016 DEBUG:boto:path=/bofrnf-46/obj4 2023-10-18T20:16:57.017 DEBUG:boto:auth_path=/bofrnf-46/obj4 2023-10-18T20:16:57.017 DEBUG:boto:Method: GET 2023-10-18T20:16:57.018 DEBUG:boto:Path: /bofrnf-46/obj4 2023-10-18T20:16:57.018 DEBUG:boto:Data: 2023-10-18T20:16:57.018 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.019 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.019 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.020 DEBUG:boto:Params: {} 2023-10-18T20:16:57.020 DEBUG:boto:Token: None 2023-10-18T20:16:57.021 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj4 2023-10-18T20:16:57.021 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DrNHWcG5Wlpghc483DroKEfNzvk= 2023-10-18T20:16:57.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DrNHWcG5Wlpghc483DroKEfNzvk=', 'Content-Length': '0'} 2023-10-18T20:16:57.024 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068a2010ba55ee364-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.024 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.025 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.025 DEBUG:boto:path=/bofrnf-46/?prefix=obj4 2023-10-18T20:16:57.026 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj4 2023-10-18T20:16:57.026 DEBUG:boto:Method: GET 2023-10-18T20:16:57.026 DEBUG:boto:Path: /bofrnf-46/?prefix=obj4 2023-10-18T20:16:57.027 DEBUG:boto:Data: 2023-10-18T20:16:57.027 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.028 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.028 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.029 DEBUG:boto:Params: {} 2023-10-18T20:16:57.029 DEBUG:boto:Token: None 2023-10-18T20:16:57.029 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057bc9fd96253e1f6-0065303d39-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.033 DEBUG:boto:b'bofrnf-46obj41000falseobj42023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.034 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.034 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.035 DEBUG:boto:path=/bofrnf-46/?prefix=obj4 2023-10-18T20:16:57.035 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj4 2023-10-18T20:16:57.036 DEBUG:boto:Method: GET 2023-10-18T20:16:57.036 DEBUG:boto:Path: /bofrnf-46/?prefix=obj4 2023-10-18T20:16:57.036 DEBUG:boto:Data: 2023-10-18T20:16:57.037 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.037 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.038 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.038 DEBUG:boto:Params: {} 2023-10-18T20:16:57.038 DEBUG:boto:Token: None 2023-10-18T20:16:57.039 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f809dad73dd9595a-0065303d39-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.043 DEBUG:boto:b'bofrnf-46obj41000falseobj42023-10-18T20:14:17.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:16:57.044 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.044 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.045 DEBUG:boto:path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.045 DEBUG:boto:auth_path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.045 DEBUG:boto:Method: GET 2023-10-18T20:16:57.046 DEBUG:boto:Path: /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.046 DEBUG:boto:Data: 2023-10-18T20:16:57.047 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.047 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.047 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.048 DEBUG:boto:Params: {} 2023-10-18T20:16:57.048 DEBUG:boto:Token: None 2023-10-18T20:16:57.049 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/IVVpCpc1Vjmzy5pPkVZcs6/RG8= 2023-10-18T20:16:57.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/IVVpCpc1Vjmzy5pPkVZcs6/RG8=', 'Content-Length': '0'} 2023-10-18T20:16:57.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060a1d86240845db0-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.052 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.053 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.053 DEBUG:boto:path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.054 DEBUG:boto:auth_path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.054 DEBUG:boto:Method: GET 2023-10-18T20:16:57.055 DEBUG:boto:Path: /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.055 DEBUG:boto:Data: 2023-10-18T20:16:57.055 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.056 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.057 DEBUG:boto:Params: {} 2023-10-18T20:16:57.057 DEBUG:boto:Token: None 2023-10-18T20:16:57.058 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.058 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/IVVpCpc1Vjmzy5pPkVZcs6/RG8= 2023-10-18T20:16:57.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/IVVpCpc1Vjmzy5pPkVZcs6/RG8=', 'Content-Length': '0'} 2023-10-18T20:16:57.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f9566a4d4b6df36-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.102 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.103 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.103 DEBUG:boto:path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.104 DEBUG:boto:auth_path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.104 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.104 DEBUG:boto:Path: /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.105 DEBUG:boto:Data: 2023-10-18T20:16:57.105 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.105 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.106 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.106 DEBUG:boto:Params: {} 2023-10-18T20:16:57.107 DEBUG:boto:Token: None 2023-10-18T20:16:57.107 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/xGZW5S1HXucXCmvFXyg5wym3XE= 2023-10-18T20:16:57.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/xGZW5S1HXucXCmvFXyg5wym3XE=', 'Content-Length': '0'} 2023-10-18T20:16:57.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077bf17be66651ba3-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.111 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.111 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.111 DEBUG:boto:path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.112 DEBUG:boto:auth_path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.112 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.112 DEBUG:boto:Path: /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.113 DEBUG:boto:Data: 2023-10-18T20:16:57.113 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.114 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.114 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.114 DEBUG:boto:Params: {} 2023-10-18T20:16:57.115 DEBUG:boto:Token: None 2023-10-18T20:16:57.115 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.116 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/xGZW5S1HXucXCmvFXyg5wym3XE= 2023-10-18T20:16:57.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/xGZW5S1HXucXCmvFXyg5wym3XE=', 'Content-Length': '0'} 2023-10-18T20:16:57.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b190e040f6a4d91d-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:16:57.119 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.119 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.120 DEBUG:boto:path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.120 DEBUG:boto:auth_path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.121 DEBUG:boto:Method: GET 2023-10-18T20:16:57.121 DEBUG:boto:Path: /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.122 DEBUG:boto:Data: 2023-10-18T20:16:57.122 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.123 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.123 DEBUG:boto:Params: {} 2023-10-18T20:16:57.124 DEBUG:boto:Token: None 2023-10-18T20:16:57.124 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.125 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/IVVpCpc1Vjmzy5pPkVZcs6/RG8= 2023-10-18T20:16:57.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/IVVpCpc1Vjmzy5pPkVZcs6/RG8=', 'Content-Length': '0'} 2023-10-18T20:16:57.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035632c01ae3078a3-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.128 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.128 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.129 DEBUG:boto:path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.129 DEBUG:boto:auth_path=/bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.130 DEBUG:boto:Method: GET 2023-10-18T20:16:57.130 DEBUG:boto:Path: /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.130 DEBUG:boto:Data: 2023-10-18T20:16:57.131 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.131 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.132 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.132 DEBUG:boto:Params: {} 2023-10-18T20:16:57.133 DEBUG:boto:Token: None 2023-10-18T20:16:57.133 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5?versionId=null 2023-10-18T20:16:57.134 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/IVVpCpc1Vjmzy5pPkVZcs6/RG8= 2023-10-18T20:16:57.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/IVVpCpc1Vjmzy5pPkVZcs6/RG8=', 'Content-Length': '0'} 2023-10-18T20:16:57.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a58f357023c07962-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.137 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.137 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.138 DEBUG:boto:path=/bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.138 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.139 DEBUG:boto:Method: GET 2023-10-18T20:16:57.139 DEBUG:boto:Path: /bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.139 DEBUG:boto:Data: 2023-10-18T20:16:57.140 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.140 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.141 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.141 DEBUG:boto:Params: {} 2023-10-18T20:16:57.141 DEBUG:boto:Token: None 2023-10-18T20:16:57.142 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.142 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8eb2ff20d8820f1-0065303d39-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.146 DEBUG:boto:b'bofrnf-46obj51000falseobj52023-10-18T20:16:20.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.146 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.147 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.147 DEBUG:boto:path=/bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.148 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.148 DEBUG:boto:Method: GET 2023-10-18T20:16:57.149 DEBUG:boto:Path: /bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.149 DEBUG:boto:Data: 2023-10-18T20:16:57.149 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.150 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.151 DEBUG:boto:Params: {} 2023-10-18T20:16:57.151 DEBUG:boto:Token: None 2023-10-18T20:16:57.152 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d3f43ecd40c4928-0065303d39-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.155 DEBUG:boto:b'bofrnf-46obj51000falseobj52023-10-18T20:16:20.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.156 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.156 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.157 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.157 DEBUG:boto:Path: /bofrnf-46/obj5 2023-10-18T20:16:57.158 DEBUG:boto:Data: 2023-10-18T20:16:57.158 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.159 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.159 DEBUG:boto:Params: {} 2023-10-18T20:16:57.160 DEBUG:boto:Token: None 2023-10-18T20:16:57.160 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5 2023-10-18T20:16:57.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qNbpkHTBn51hiPKn9tBKSQ/3Xjc= 2023-10-18T20:16:57.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qNbpkHTBn51hiPKn9tBKSQ/3Xjc=', 'Content-Length': '0'} 2023-10-18T20:16:57.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000247739058ca4f4ca-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.164 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.164 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.164 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.165 DEBUG:boto:Path: /bofrnf-46/obj5 2023-10-18T20:16:57.165 DEBUG:boto:Data: 2023-10-18T20:16:57.166 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.166 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.167 DEBUG:boto:Params: {} 2023-10-18T20:16:57.167 DEBUG:boto:Token: None 2023-10-18T20:16:57.168 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5 2023-10-18T20:16:57.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qNbpkHTBn51hiPKn9tBKSQ/3Xjc= 2023-10-18T20:16:57.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qNbpkHTBn51hiPKn9tBKSQ/3Xjc=', 'Content-Length': '0'} 2023-10-18T20:16:57.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066afe101aefc8081-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:16:57.171 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.172 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.172 DEBUG:boto:Method: GET 2023-10-18T20:16:57.172 DEBUG:boto:Path: /bofrnf-46/obj5 2023-10-18T20:16:57.173 DEBUG:boto:Data: 2023-10-18T20:16:57.173 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.174 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.174 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.175 DEBUG:boto:Params: {} 2023-10-18T20:16:57.175 DEBUG:boto:Token: None 2023-10-18T20:16:57.176 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5 2023-10-18T20:16:57.176 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P6sNjOBRpfVt5wDzRHZJCMX6OzQ= 2023-10-18T20:16:57.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P6sNjOBRpfVt5wDzRHZJCMX6OzQ=', 'Content-Length': '0'} 2023-10-18T20:16:57.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074c4e00f7f9272ff-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.226 DEBUG:boto:path=/bofrnf-46/obj5 2023-10-18T20:16:57.227 DEBUG:boto:auth_path=/bofrnf-46/obj5 2023-10-18T20:16:57.227 DEBUG:boto:Method: GET 2023-10-18T20:16:57.228 DEBUG:boto:Path: /bofrnf-46/obj5 2023-10-18T20:16:57.228 DEBUG:boto:Data: 2023-10-18T20:16:57.229 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.229 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.230 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.230 DEBUG:boto:Params: {} 2023-10-18T20:16:57.231 DEBUG:boto:Token: None 2023-10-18T20:16:57.231 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj5 2023-10-18T20:16:57.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P6sNjOBRpfVt5wDzRHZJCMX6OzQ= 2023-10-18T20:16:57.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P6sNjOBRpfVt5wDzRHZJCMX6OzQ=', 'Content-Length': '0'} 2023-10-18T20:16:57.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006529f3adb5d3e8a9-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.235 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.235 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.236 DEBUG:boto:path=/bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.236 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.236 DEBUG:boto:Method: GET 2023-10-18T20:16:57.237 DEBUG:boto:Path: /bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.237 DEBUG:boto:Data: 2023-10-18T20:16:57.238 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.238 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.239 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.239 DEBUG:boto:Params: {} 2023-10-18T20:16:57.239 DEBUG:boto:Token: None 2023-10-18T20:16:57.240 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df6c5f0609f7410b-0065303d39-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.244 DEBUG:boto:b'bofrnf-46obj51000falseobj52023-10-18T20:16:20.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.245 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.245 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.245 DEBUG:boto:path=/bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.246 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.246 DEBUG:boto:Method: GET 2023-10-18T20:16:57.247 DEBUG:boto:Path: /bofrnf-46/?prefix=obj5 2023-10-18T20:16:57.247 DEBUG:boto:Data: 2023-10-18T20:16:57.247 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.248 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.249 DEBUG:boto:Params: {} 2023-10-18T20:16:57.249 DEBUG:boto:Token: None 2023-10-18T20:16:57.250 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad753edba3113d0f-0065303d39-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.294 DEBUG:boto:b'bofrnf-46obj51000falseobj52023-10-18T20:16:20.663Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:16:57.296 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.296 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.297 DEBUG:boto:path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.297 DEBUG:boto:auth_path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.297 DEBUG:boto:Method: GET 2023-10-18T20:16:57.298 DEBUG:boto:Path: /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.298 DEBUG:boto:Data: 2023-10-18T20:16:57.299 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.299 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.300 DEBUG:boto:Params: {} 2023-10-18T20:16:57.300 DEBUG:boto:Token: None 2023-10-18T20:16:57.301 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:if/ACqsL7iL9YziDXDck/3oQjKI= 2023-10-18T20:16:57.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:if/ACqsL7iL9YziDXDck/3oQjKI=', 'Content-Length': '0'} 2023-10-18T20:16:57.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fcd86c8bd61e150-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.305 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.305 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.306 DEBUG:boto:path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.306 DEBUG:boto:auth_path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.307 DEBUG:boto:Method: GET 2023-10-18T20:16:57.307 DEBUG:boto:Path: /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.308 DEBUG:boto:Data: 2023-10-18T20:16:57.308 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.309 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.309 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.310 DEBUG:boto:Params: {} 2023-10-18T20:16:57.310 DEBUG:boto:Token: None 2023-10-18T20:16:57.311 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.311 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:if/ACqsL7iL9YziDXDck/3oQjKI= 2023-10-18T20:16:57.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:if/ACqsL7iL9YziDXDck/3oQjKI=', 'Content-Length': '0'} 2023-10-18T20:16:57.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2b7a59322d2d9ff-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.314 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.315 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.315 DEBUG:boto:path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.316 DEBUG:boto:auth_path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.316 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.317 DEBUG:boto:Path: /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.317 DEBUG:boto:Data: 2023-10-18T20:16:57.317 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.318 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.318 DEBUG:boto:Params: {} 2023-10-18T20:16:57.319 DEBUG:boto:Token: None 2023-10-18T20:16:57.319 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ACSrBs5FPrvh6GfZeGkJLRhcs5M= 2023-10-18T20:16:57.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ACSrBs5FPrvh6GfZeGkJLRhcs5M=', 'Content-Length': '0'} 2023-10-18T20:16:57.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096ffe18d966501d4-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.323 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.323 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.324 DEBUG:boto:path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.324 DEBUG:boto:auth_path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.324 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.325 DEBUG:boto:Path: /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.325 DEBUG:boto:Data: 2023-10-18T20:16:57.326 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.326 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.326 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.327 DEBUG:boto:Params: {} 2023-10-18T20:16:57.327 DEBUG:boto:Token: None 2023-10-18T20:16:57.328 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ACSrBs5FPrvh6GfZeGkJLRhcs5M= 2023-10-18T20:16:57.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ACSrBs5FPrvh6GfZeGkJLRhcs5M=', 'Content-Length': '0'} 2023-10-18T20:16:57.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000042ef1e6bb5b2611-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:16:57.331 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.332 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.332 DEBUG:boto:path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.333 DEBUG:boto:auth_path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.333 DEBUG:boto:Method: GET 2023-10-18T20:16:57.334 DEBUG:boto:Path: /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.334 DEBUG:boto:Data: 2023-10-18T20:16:57.334 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.335 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.335 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.336 DEBUG:boto:Params: {} 2023-10-18T20:16:57.336 DEBUG:boto:Token: None 2023-10-18T20:16:57.337 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:if/ACqsL7iL9YziDXDck/3oQjKI= 2023-10-18T20:16:57.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:if/ACqsL7iL9YziDXDck/3oQjKI=', 'Content-Length': '0'} 2023-10-18T20:16:57.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004f66b7a883b07a6-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.340 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.341 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.341 DEBUG:boto:path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.342 DEBUG:boto:auth_path=/bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.342 DEBUG:boto:Method: GET 2023-10-18T20:16:57.343 DEBUG:boto:Path: /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.343 DEBUG:boto:Data: 2023-10-18T20:16:57.344 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.344 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.344 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.345 DEBUG:boto:Params: {} 2023-10-18T20:16:57.345 DEBUG:boto:Token: None 2023-10-18T20:16:57.346 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6?versionId=null 2023-10-18T20:16:57.346 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:if/ACqsL7iL9YziDXDck/3oQjKI= 2023-10-18T20:16:57.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:if/ACqsL7iL9YziDXDck/3oQjKI=', 'Content-Length': '0'} 2023-10-18T20:16:57.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f0416570e9faa72-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.349 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.350 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.350 DEBUG:boto:path=/bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.351 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.351 DEBUG:boto:Method: GET 2023-10-18T20:16:57.352 DEBUG:boto:Path: /bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.352 DEBUG:boto:Data: 2023-10-18T20:16:57.352 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.353 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.353 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.354 DEBUG:boto:Params: {} 2023-10-18T20:16:57.354 DEBUG:boto:Token: None 2023-10-18T20:16:57.355 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.355 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2cb2753c3dc70ee-0065303d39-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.359 DEBUG:boto:b'bofrnf-46obj61000falseobj62023-10-18T20:16:20.681Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.359 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.360 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.360 DEBUG:boto:path=/bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.361 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.361 DEBUG:boto:Method: GET 2023-10-18T20:16:57.361 DEBUG:boto:Path: /bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.362 DEBUG:boto:Data: 2023-10-18T20:16:57.362 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.363 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.363 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.364 DEBUG:boto:Params: {} 2023-10-18T20:16:57.364 DEBUG:boto:Token: None 2023-10-18T20:16:57.365 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.365 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d42d27adf05eec6c-0065303d39-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.369 DEBUG:boto:b'bofrnf-46obj61000falseobj62023-10-18T20:16:20.681Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.370 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.370 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.371 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.371 DEBUG:boto:Path: /bofrnf-46/obj6 2023-10-18T20:16:57.372 DEBUG:boto:Data: 2023-10-18T20:16:57.372 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.372 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.373 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.373 DEBUG:boto:Params: {} 2023-10-18T20:16:57.374 DEBUG:boto:Token: None 2023-10-18T20:16:57.374 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6 2023-10-18T20:16:57.375 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6wNt5vo2jf5SrtpITYSZ6rdnD3Y= 2023-10-18T20:16:57.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6wNt5vo2jf5SrtpITYSZ6rdnD3Y=', 'Content-Length': '0'} 2023-10-18T20:16:57.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004baf477e124178a6-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.378 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.378 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.378 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.379 DEBUG:boto:Path: /bofrnf-46/obj6 2023-10-18T20:16:57.379 DEBUG:boto:Data: 2023-10-18T20:16:57.380 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.380 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.381 DEBUG:boto:Params: {} 2023-10-18T20:16:57.381 DEBUG:boto:Token: None 2023-10-18T20:16:57.382 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6 2023-10-18T20:16:57.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6wNt5vo2jf5SrtpITYSZ6rdnD3Y= 2023-10-18T20:16:57.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6wNt5vo2jf5SrtpITYSZ6rdnD3Y=', 'Content-Length': '0'} 2023-10-18T20:16:57.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe2d818c42762ef8-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:16:57.386 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.386 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.387 DEBUG:boto:Method: GET 2023-10-18T20:16:57.387 DEBUG:boto:Path: /bofrnf-46/obj6 2023-10-18T20:16:57.387 DEBUG:boto:Data: 2023-10-18T20:16:57.388 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.388 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.389 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.389 DEBUG:boto:Params: {} 2023-10-18T20:16:57.390 DEBUG:boto:Token: None 2023-10-18T20:16:57.390 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6 2023-10-18T20:16:57.391 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tcDiPQQo3ZUN388kobGqgynZtTs= 2023-10-18T20:16:57.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tcDiPQQo3ZUN388kobGqgynZtTs=', 'Content-Length': '0'} 2023-10-18T20:16:57.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016b29f3a771d4f57-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.435 DEBUG:boto:path=/bofrnf-46/obj6 2023-10-18T20:16:57.435 DEBUG:boto:auth_path=/bofrnf-46/obj6 2023-10-18T20:16:57.436 DEBUG:boto:Method: GET 2023-10-18T20:16:57.436 DEBUG:boto:Path: /bofrnf-46/obj6 2023-10-18T20:16:57.437 DEBUG:boto:Data: 2023-10-18T20:16:57.437 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.437 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.438 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.438 DEBUG:boto:Params: {} 2023-10-18T20:16:57.439 DEBUG:boto:Token: None 2023-10-18T20:16:57.439 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj6 2023-10-18T20:16:57.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tcDiPQQo3ZUN388kobGqgynZtTs= 2023-10-18T20:16:57.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tcDiPQQo3ZUN388kobGqgynZtTs=', 'Content-Length': '0'} 2023-10-18T20:16:57.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e36d5ab6330b1606-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.443 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.443 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.444 DEBUG:boto:path=/bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.444 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.445 DEBUG:boto:Method: GET 2023-10-18T20:16:57.445 DEBUG:boto:Path: /bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.445 DEBUG:boto:Data: 2023-10-18T20:16:57.446 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.447 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.447 DEBUG:boto:Params: {} 2023-10-18T20:16:57.448 DEBUG:boto:Token: None 2023-10-18T20:16:57.448 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9ba753c39e6f6c0-0065303d39-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.452 DEBUG:boto:b'bofrnf-46obj61000falseobj62023-10-18T20:16:20.681Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.453 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.453 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.454 DEBUG:boto:path=/bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.454 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.454 DEBUG:boto:Method: GET 2023-10-18T20:16:57.455 DEBUG:boto:Path: /bofrnf-46/?prefix=obj6 2023-10-18T20:16:57.455 DEBUG:boto:Data: 2023-10-18T20:16:57.456 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.456 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.456 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.457 DEBUG:boto:Params: {} 2023-10-18T20:16:57.457 DEBUG:boto:Token: None 2023-10-18T20:16:57.458 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.458 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d76f593240f4013-0065303d39-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.502 DEBUG:boto:b'bofrnf-46obj61000falseobj62023-10-18T20:16:20.681Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:16:57.504 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.504 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.505 DEBUG:boto:path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.505 DEBUG:boto:auth_path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.505 DEBUG:boto:Method: GET 2023-10-18T20:16:57.506 DEBUG:boto:Path: /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.506 DEBUG:boto:Data: 2023-10-18T20:16:57.507 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.508 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.508 DEBUG:boto:Params: {} 2023-10-18T20:16:57.509 DEBUG:boto:Token: None 2023-10-18T20:16:57.509 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2QoxexOusok5XpMYONnxu1pMKdk= 2023-10-18T20:16:57.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2QoxexOusok5XpMYONnxu1pMKdk=', 'Content-Length': '0'} 2023-10-18T20:16:57.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d451fdad548a5fa-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.513 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.514 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.514 DEBUG:boto:path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.515 DEBUG:boto:auth_path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.515 DEBUG:boto:Method: GET 2023-10-18T20:16:57.515 DEBUG:boto:Path: /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.516 DEBUG:boto:Data: 2023-10-18T20:16:57.516 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.517 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.517 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.517 DEBUG:boto:Params: {} 2023-10-18T20:16:57.518 DEBUG:boto:Token: None 2023-10-18T20:16:57.518 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.519 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2QoxexOusok5XpMYONnxu1pMKdk= 2023-10-18T20:16:57.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2QoxexOusok5XpMYONnxu1pMKdk=', 'Content-Length': '0'} 2023-10-18T20:16:57.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8f082c308726041-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.522 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.522 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.523 DEBUG:boto:path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.523 DEBUG:boto:auth_path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.524 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.524 DEBUG:boto:Path: /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.525 DEBUG:boto:Data: 2023-10-18T20:16:57.525 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.526 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.526 DEBUG:boto:Params: {} 2023-10-18T20:16:57.527 DEBUG:boto:Token: None 2023-10-18T20:16:57.527 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.528 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uO+XzckY2CuUdOGaiLbFNzJFIy8= 2023-10-18T20:16:57.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uO+XzckY2CuUdOGaiLbFNzJFIy8=', 'Content-Length': '0'} 2023-10-18T20:16:57.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b1042e7e4f1d589-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.531 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.531 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.532 DEBUG:boto:path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.532 DEBUG:boto:auth_path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.532 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.533 DEBUG:boto:Path: /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.533 DEBUG:boto:Data: 2023-10-18T20:16:57.534 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.534 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.535 DEBUG:boto:Params: {} 2023-10-18T20:16:57.535 DEBUG:boto:Token: None 2023-10-18T20:16:57.536 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uO+XzckY2CuUdOGaiLbFNzJFIy8= 2023-10-18T20:16:57.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uO+XzckY2CuUdOGaiLbFNzJFIy8=', 'Content-Length': '0'} 2023-10-18T20:16:57.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ad6b8fda2bcd732-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:16:57.539 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.540 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.540 DEBUG:boto:path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.541 DEBUG:boto:auth_path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.541 DEBUG:boto:Method: GET 2023-10-18T20:16:57.541 DEBUG:boto:Path: /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.542 DEBUG:boto:Data: 2023-10-18T20:16:57.542 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.543 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.543 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.544 DEBUG:boto:Params: {} 2023-10-18T20:16:57.544 DEBUG:boto:Token: None 2023-10-18T20:16:57.544 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2QoxexOusok5XpMYONnxu1pMKdk= 2023-10-18T20:16:57.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2QoxexOusok5XpMYONnxu1pMKdk=', 'Content-Length': '0'} 2023-10-18T20:16:57.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016f48508c2c268cc-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.548 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.548 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.549 DEBUG:boto:path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.549 DEBUG:boto:auth_path=/bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.550 DEBUG:boto:Method: GET 2023-10-18T20:16:57.550 DEBUG:boto:Path: /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.551 DEBUG:boto:Data: 2023-10-18T20:16:57.551 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.551 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.552 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.552 DEBUG:boto:Params: {} 2023-10-18T20:16:57.553 DEBUG:boto:Token: None 2023-10-18T20:16:57.553 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7?versionId=null 2023-10-18T20:16:57.554 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2QoxexOusok5XpMYONnxu1pMKdk= 2023-10-18T20:16:57.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2QoxexOusok5XpMYONnxu1pMKdk=', 'Content-Length': '0'} 2023-10-18T20:16:57.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000004128da6abd9eec-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.557 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.558 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.558 DEBUG:boto:path=/bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.558 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.559 DEBUG:boto:Method: GET 2023-10-18T20:16:57.559 DEBUG:boto:Path: /bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.560 DEBUG:boto:Data: 2023-10-18T20:16:57.560 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.560 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.561 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.561 DEBUG:boto:Params: {} 2023-10-18T20:16:57.562 DEBUG:boto:Token: None 2023-10-18T20:16:57.562 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.563 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cc64eb22b9a3164-0065303d39-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.566 DEBUG:boto:b'bofrnf-46obj71000falseobj72023-10-18T20:16:20.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.567 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.567 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.568 DEBUG:boto:path=/bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.568 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.569 DEBUG:boto:Method: GET 2023-10-18T20:16:57.569 DEBUG:boto:Path: /bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.569 DEBUG:boto:Data: 2023-10-18T20:16:57.570 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.570 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.571 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.571 DEBUG:boto:Params: {} 2023-10-18T20:16:57.572 DEBUG:boto:Token: None 2023-10-18T20:16:57.572 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.573 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ed32dacd97cff4-0065303d39-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.576 DEBUG:boto:b'bofrnf-46obj71000falseobj72023-10-18T20:16:20.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.577 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.577 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.578 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.578 DEBUG:boto:Path: /bofrnf-46/obj7 2023-10-18T20:16:57.579 DEBUG:boto:Data: 2023-10-18T20:16:57.579 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.579 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.580 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.580 DEBUG:boto:Params: {} 2023-10-18T20:16:57.580 DEBUG:boto:Token: None 2023-10-18T20:16:57.581 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7 2023-10-18T20:16:57.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Ko7c1oUpvxMc0FHgJ9Tf4lHL8s= 2023-10-18T20:16:57.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Ko7c1oUpvxMc0FHgJ9Tf4lHL8s=', 'Content-Length': '0'} 2023-10-18T20:16:57.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aef14f6a8d6734eb-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.584 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.585 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.585 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.585 DEBUG:boto:Path: /bofrnf-46/obj7 2023-10-18T20:16:57.586 DEBUG:boto:Data: 2023-10-18T20:16:57.586 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.587 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.587 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.587 DEBUG:boto:Params: {} 2023-10-18T20:16:57.588 DEBUG:boto:Token: None 2023-10-18T20:16:57.588 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7 2023-10-18T20:16:57.589 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Ko7c1oUpvxMc0FHgJ9Tf4lHL8s= 2023-10-18T20:16:57.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Ko7c1oUpvxMc0FHgJ9Tf4lHL8s=', 'Content-Length': '0'} 2023-10-18T20:16:57.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009544dfe2647c03a0-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:16:57.592 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.592 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.593 DEBUG:boto:Method: GET 2023-10-18T20:16:57.593 DEBUG:boto:Path: /bofrnf-46/obj7 2023-10-18T20:16:57.594 DEBUG:boto:Data: 2023-10-18T20:16:57.594 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.595 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.595 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.595 DEBUG:boto:Params: {} 2023-10-18T20:16:57.596 DEBUG:boto:Token: None 2023-10-18T20:16:57.596 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7 2023-10-18T20:16:57.597 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e5uVfMHgvtw0Esx4bRzzeIttROs= 2023-10-18T20:16:57.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e5uVfMHgvtw0Esx4bRzzeIttROs=', 'Content-Length': '0'} 2023-10-18T20:16:57.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004325932ea32ea269-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.600 DEBUG:boto:path=/bofrnf-46/obj7 2023-10-18T20:16:57.601 DEBUG:boto:auth_path=/bofrnf-46/obj7 2023-10-18T20:16:57.601 DEBUG:boto:Method: GET 2023-10-18T20:16:57.601 DEBUG:boto:Path: /bofrnf-46/obj7 2023-10-18T20:16:57.602 DEBUG:boto:Data: 2023-10-18T20:16:57.602 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.603 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.603 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.604 DEBUG:boto:Params: {} 2023-10-18T20:16:57.604 DEBUG:boto:Token: None 2023-10-18T20:16:57.605 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj7 2023-10-18T20:16:57.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e5uVfMHgvtw0Esx4bRzzeIttROs= 2023-10-18T20:16:57.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e5uVfMHgvtw0Esx4bRzzeIttROs=', 'Content-Length': '0'} 2023-10-18T20:16:57.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b21bd7cbbb7c88da-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.650 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.651 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.651 DEBUG:boto:path=/bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.652 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.652 DEBUG:boto:Method: GET 2023-10-18T20:16:57.652 DEBUG:boto:Path: /bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.653 DEBUG:boto:Data: 2023-10-18T20:16:57.653 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.654 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.655 DEBUG:boto:Params: {} 2023-10-18T20:16:57.655 DEBUG:boto:Token: None 2023-10-18T20:16:57.656 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009027a9bff756b0e8-0065303d39-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.660 DEBUG:boto:b'bofrnf-46obj71000falseobj72023-10-18T20:16:20.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.660 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.661 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.661 DEBUG:boto:path=/bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.662 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.662 DEBUG:boto:Method: GET 2023-10-18T20:16:57.663 DEBUG:boto:Path: /bofrnf-46/?prefix=obj7 2023-10-18T20:16:57.663 DEBUG:boto:Data: 2023-10-18T20:16:57.663 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.664 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.664 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.665 DEBUG:boto:Params: {} 2023-10-18T20:16:57.665 DEBUG:boto:Token: None 2023-10-18T20:16:57.666 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.666 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d8ca6b8e3fb3936-0065303d39-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.670 DEBUG:boto:b'bofrnf-46obj71000falseobj72023-10-18T20:16:20.737Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:16:57.671 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.671 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.672 DEBUG:boto:path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.672 DEBUG:boto:auth_path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.673 DEBUG:boto:Method: GET 2023-10-18T20:16:57.673 DEBUG:boto:Path: /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.674 DEBUG:boto:Data: 2023-10-18T20:16:57.674 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.674 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.675 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.675 DEBUG:boto:Params: {} 2023-10-18T20:16:57.676 DEBUG:boto:Token: None 2023-10-18T20:16:57.676 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.677 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4FE7hWaeRaPvS0l5CQ2/L6u1Aqk= 2023-10-18T20:16:57.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4FE7hWaeRaPvS0l5CQ2/L6u1Aqk=', 'Content-Length': '0'} 2023-10-18T20:16:57.680 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0e93039b718c3e5-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.726 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.727 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.727 DEBUG:boto:path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.776 DEBUG:boto:auth_path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.777 DEBUG:boto:Method: GET 2023-10-18T20:16:57.777 DEBUG:boto:Path: /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.778 DEBUG:boto:Data: 2023-10-18T20:16:57.778 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.779 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.780 DEBUG:boto:Params: {} 2023-10-18T20:16:57.780 DEBUG:boto:Token: None 2023-10-18T20:16:57.781 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.781 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4FE7hWaeRaPvS0l5CQ2/L6u1Aqk= 2023-10-18T20:16:57.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4FE7hWaeRaPvS0l5CQ2/L6u1Aqk=', 'Content-Length': '0'} 2023-10-18T20:16:57.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018636948eb9d24c4-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.784 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.785 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.785 DEBUG:boto:path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.786 DEBUG:boto:auth_path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.786 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.786 DEBUG:boto:Path: /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.787 DEBUG:boto:Data: 2023-10-18T20:16:57.787 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.788 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.788 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.788 DEBUG:boto:Params: {} 2023-10-18T20:16:57.789 DEBUG:boto:Token: None 2023-10-18T20:16:57.789 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.790 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KyhmRfaW1fsIvPqk0XQFE3WoSUI= 2023-10-18T20:16:57.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KyhmRfaW1fsIvPqk0XQFE3WoSUI=', 'Content-Length': '0'} 2023-10-18T20:16:57.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd0df87d4c34dbe3-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.793 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.793 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.794 DEBUG:boto:path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.794 DEBUG:boto:auth_path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.794 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.795 DEBUG:boto:Path: /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.795 DEBUG:boto:Data: 2023-10-18T20:16:57.796 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.796 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.796 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.797 DEBUG:boto:Params: {} 2023-10-18T20:16:57.797 DEBUG:boto:Token: None 2023-10-18T20:16:57.798 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.798 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KyhmRfaW1fsIvPqk0XQFE3WoSUI= 2023-10-18T20:16:57.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KyhmRfaW1fsIvPqk0XQFE3WoSUI=', 'Content-Length': '0'} 2023-10-18T20:16:57.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d465288f290250af-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:16:57.802 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.802 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.803 DEBUG:boto:path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.803 DEBUG:boto:auth_path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.803 DEBUG:boto:Method: GET 2023-10-18T20:16:57.804 DEBUG:boto:Path: /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.804 DEBUG:boto:Data: 2023-10-18T20:16:57.805 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.806 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.806 DEBUG:boto:Params: {} 2023-10-18T20:16:57.806 DEBUG:boto:Token: None 2023-10-18T20:16:57.807 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.807 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4FE7hWaeRaPvS0l5CQ2/L6u1Aqk= 2023-10-18T20:16:57.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4FE7hWaeRaPvS0l5CQ2/L6u1Aqk=', 'Content-Length': '0'} 2023-10-18T20:16:57.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b142c61e68cc0d1-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.810 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.811 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.811 DEBUG:boto:path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.812 DEBUG:boto:auth_path=/bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.812 DEBUG:boto:Method: GET 2023-10-18T20:16:57.813 DEBUG:boto:Path: /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.813 DEBUG:boto:Data: 2023-10-18T20:16:57.813 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.814 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.815 DEBUG:boto:Params: {} 2023-10-18T20:16:57.815 DEBUG:boto:Token: None 2023-10-18T20:16:57.816 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8?versionId=null 2023-10-18T20:16:57.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4FE7hWaeRaPvS0l5CQ2/L6u1Aqk= 2023-10-18T20:16:57.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4FE7hWaeRaPvS0l5CQ2/L6u1Aqk=', 'Content-Length': '0'} 2023-10-18T20:16:57.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b14844e73a28e2ce-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.822 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.822 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.823 DEBUG:boto:path=/bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.823 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.824 DEBUG:boto:Method: GET 2023-10-18T20:16:57.824 DEBUG:boto:Path: /bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.825 DEBUG:boto:Data: 2023-10-18T20:16:57.825 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.825 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.826 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.826 DEBUG:boto:Params: {} 2023-10-18T20:16:57.827 DEBUG:boto:Token: None 2023-10-18T20:16:57.827 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.828 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5dcf7a0c49a1156-0065303d39-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.831 DEBUG:boto:b'bofrnf-46obj81000falseobj82023-10-18T20:16:20.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.832 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.832 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.833 DEBUG:boto:path=/bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.833 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.834 DEBUG:boto:Method: GET 2023-10-18T20:16:57.834 DEBUG:boto:Path: /bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.834 DEBUG:boto:Data: 2023-10-18T20:16:57.835 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.835 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.836 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.836 DEBUG:boto:Params: {} 2023-10-18T20:16:57.837 DEBUG:boto:Token: None 2023-10-18T20:16:57.837 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.837 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c8ed185768198af-0065303d39-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.841 DEBUG:boto:b'bofrnf-46obj81000falseobj82023-10-18T20:16:20.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.842 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.842 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.843 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.843 DEBUG:boto:Path: /bofrnf-46/obj8 2023-10-18T20:16:57.843 DEBUG:boto:Data: 2023-10-18T20:16:57.844 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.844 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.845 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.845 DEBUG:boto:Params: {} 2023-10-18T20:16:57.845 DEBUG:boto:Token: None 2023-10-18T20:16:57.846 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8 2023-10-18T20:16:57.846 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:11KEDvfFpNM0Dfnq544WziYDlOc= 2023-10-18T20:16:57.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:11KEDvfFpNM0Dfnq544WziYDlOc=', 'Content-Length': '0'} 2023-10-18T20:16:57.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b48398cf855800b-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.849 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.850 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.850 DEBUG:boto:Method: HEAD 2023-10-18T20:16:57.851 DEBUG:boto:Path: /bofrnf-46/obj8 2023-10-18T20:16:57.851 DEBUG:boto:Data: 2023-10-18T20:16:57.851 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.852 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.852 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.853 DEBUG:boto:Params: {} 2023-10-18T20:16:57.853 DEBUG:boto:Token: None 2023-10-18T20:16:57.853 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8 2023-10-18T20:16:57.854 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:11KEDvfFpNM0Dfnq544WziYDlOc= 2023-10-18T20:16:57.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:11KEDvfFpNM0Dfnq544WziYDlOc=', 'Content-Length': '0'} 2023-10-18T20:16:57.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b624da2002483545-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:16:57.857 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.858 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.858 DEBUG:boto:Method: GET 2023-10-18T20:16:57.859 DEBUG:boto:Path: /bofrnf-46/obj8 2023-10-18T20:16:57.859 DEBUG:boto:Data: 2023-10-18T20:16:57.859 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.860 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.860 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.861 DEBUG:boto:Params: {} 2023-10-18T20:16:57.861 DEBUG:boto:Token: None 2023-10-18T20:16:57.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8 2023-10-18T20:16:57.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/7xL7nAkwM4Vybl5TalmaAMI1os= 2023-10-18T20:16:57.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/7xL7nAkwM4Vybl5TalmaAMI1os=', 'Content-Length': '0'} 2023-10-18T20:16:57.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-46:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab21a995b1854270-0065303d39-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.865 DEBUG:boto:path=/bofrnf-46/obj8 2023-10-18T20:16:57.866 DEBUG:boto:auth_path=/bofrnf-46/obj8 2023-10-18T20:16:57.866 DEBUG:boto:Method: GET 2023-10-18T20:16:57.867 DEBUG:boto:Path: /bofrnf-46/obj8 2023-10-18T20:16:57.867 DEBUG:boto:Data: 2023-10-18T20:16:57.868 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.868 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.869 DEBUG:boto:Params: {} 2023-10-18T20:16:57.869 DEBUG:boto:Token: None 2023-10-18T20:16:57.870 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/obj8 2023-10-18T20:16:57.870 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/7xL7nAkwM4Vybl5TalmaAMI1os= 2023-10-18T20:16:57.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/7xL7nAkwM4Vybl5TalmaAMI1os=', 'Content-Length': '0'} 2023-10-18T20:16:57.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022a433e58a0617ab-0065303d39-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.873 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.874 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.874 DEBUG:boto:path=/bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.875 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.875 DEBUG:boto:Method: GET 2023-10-18T20:16:57.875 DEBUG:boto:Path: /bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.876 DEBUG:boto:Data: 2023-10-18T20:16:57.876 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:57.877 DEBUG:boto:Port: 8001 2023-10-18T20:16:57.877 DEBUG:boto:Params: {} 2023-10-18T20:16:57.878 DEBUG:boto:Token: None 2023-10-18T20:16:57.878 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.879 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e6838b5ff581a2-0065303d39-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.882 DEBUG:boto:b'bofrnf-46obj81000falseobj82023-10-18T20:16:20.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.883 DEBUG:boto:path=/bofrnf-46/ 2023-10-18T20:16:57.884 DEBUG:boto:auth_path=/bofrnf-46/ 2023-10-18T20:16:57.884 DEBUG:boto:path=/bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.884 DEBUG:boto:auth_path=/bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.885 DEBUG:boto:Method: GET 2023-10-18T20:16:57.885 DEBUG:boto:Path: /bofrnf-46/?prefix=obj8 2023-10-18T20:16:57.885 DEBUG:boto:Data: 2023-10-18T20:16:57.886 DEBUG:boto:Headers: {} 2023-10-18T20:16:57.886 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:57.887 DEBUG:boto:Port: 8000 2023-10-18T20:16:57.887 DEBUG:boto:Params: {} 2023-10-18T20:16:57.888 DEBUG:boto:Token: None 2023-10-18T20:16:57.888 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:57 GMT /bofrnf-46/ 2023-10-18T20:16:57.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck= 2023-10-18T20:16:57.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:57 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5LML4r2f4of6Jb9RvUj0gEBi3Ck=', 'Content-Length': '0'} 2023-10-18T20:16:57.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021276fe243a7a2a1-0065303d39-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:57.892 DEBUG:boto:b'bofrnf-46obj81000falseobj82023-10-18T20:16:20.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:57.893 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-46 zones={a2, a1} 2023-10-18T20:16:57.893 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-47 2023-10-18T20:16:57.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:16:57.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:16:57.895 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:16:57.961 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:57.961 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:57.998 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:57.996+0000 7f3e7e110780 1 bucket sync caught up with source: 2023-10-18T20:16:57.998 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.44.6, 00000000002.42.1, 00000000004.27.6, 00000000002.17.1, 00000000004.28.6, 00000000002.31.1, 00000000002.23.1, 00000000004.33.6, 00000000002.13.1, 00000000002.12.1, 00000000002.22.1] 2023-10-18T20:16:57.998 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.44.6, 00000000002.42.1, 00000000004.27.6, 00000000002.17.1, 00000000004.28.6, 00000000002.31.1, 00000000002.23.1, 00000000004.33.6, 00000000002.13.1, 00000000002.12.1, 00000000002.22.1] 2023-10-18T20:16:57.998 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:57.996+0000 7f3e7e110780 0 bucket checkpoint complete 2023-10-18T20:16:58.011 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-47 2023-10-18T20:16:58.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:16:58.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:16:58.013 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:16:58.071 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:58.071 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:58.105 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:58.104+0000 7f5143a9f780 1 bucket sync caught up with source: 2023-10-18T20:16:58.106 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000002.591.6, , 00000000002.167.6, , 00000000002.686.6, , , 00000000002.934.6, , , ] 2023-10-18T20:16:58.106 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.591.6, , 00000000002.167.6, , 00000000002.686.6, , , 00000000002.934.6, , , ] 2023-10-18T20:16:58.106 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:16:58.104+0000 7f5143a9f780 0 bucket checkpoint complete 2023-10-18T20:16:58.119 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-47 zones={a2, a1} 2023-10-18T20:16:58.120 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.120 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.121 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.121 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:16:58.121 DEBUG:boto:Data: 2023-10-18T20:16:58.122 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.123 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.123 DEBUG:boto:Params: {} 2023-10-18T20:16:58.124 DEBUG:boto:Token: None 2023-10-18T20:16:58.124 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.124 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sEwS90JSXZbuCzNS7fmR6SG4IvI= 2023-10-18T20:16:58.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sEwS90JSXZbuCzNS7fmR6SG4IvI=', 'Content-Length': '0'} 2023-10-18T20:16:58.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae31401c0e0d9bf5-0065303d3a-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.128 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.128 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.129 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.129 DEBUG:boto:Path: /bofrnf-47/ 2023-10-18T20:16:58.129 DEBUG:boto:Data: 2023-10-18T20:16:58.130 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.130 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.131 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.131 DEBUG:boto:Params: {} 2023-10-18T20:16:58.132 DEBUG:boto:Token: None 2023-10-18T20:16:58.132 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sEwS90JSXZbuCzNS7fmR6SG4IvI= 2023-10-18T20:16:58.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sEwS90JSXZbuCzNS7fmR6SG4IvI=', 'Content-Length': '0'} 2023-10-18T20:16:58.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d09358a9b0f9c3ed-0065303d3a-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.136 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:16:58.136 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.137 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.137 DEBUG:boto:path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.137 DEBUG:boto:auth_path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.138 DEBUG:boto:Method: GET 2023-10-18T20:16:58.138 DEBUG:boto:Path: /bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.139 DEBUG:boto:Data: 2023-10-18T20:16:58.139 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.140 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.140 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.140 DEBUG:boto:Params: {} 2023-10-18T20:16:58.141 DEBUG:boto:Token: None 2023-10-18T20:16:58.141 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/?versions 2023-10-18T20:16:58.142 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:StXrWVaB5piceUjvgvpZ0HU0E5U= 2023-10-18T20:16:58.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:StXrWVaB5piceUjvgvpZ0HU0E5U=', 'Content-Length': '0'} 2023-10-18T20:16:58.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042cfed0eb66a3cd1-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.145 DEBUG:boto:b'bofrnf-47999falseobj1nulltrue2023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:16:20.816Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:16:20.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:16:20.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:16:20.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:16:58.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:16:58.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:16:58.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:16:58.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-18T20:16:58.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-18T20:16:58.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-18T20:16:58.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-18T20:16:58.149 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:16:58.149 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.150 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.150 DEBUG:boto:path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.151 DEBUG:boto:auth_path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.151 DEBUG:boto:Method: GET 2023-10-18T20:16:58.151 DEBUG:boto:Path: /bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.152 DEBUG:boto:Data: 2023-10-18T20:16:58.152 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.153 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.153 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.153 DEBUG:boto:Params: {} 2023-10-18T20:16:58.154 DEBUG:boto:Token: None 2023-10-18T20:16:58.154 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/?versions 2023-10-18T20:16:58.155 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:StXrWVaB5piceUjvgvpZ0HU0E5U= 2023-10-18T20:16:58.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:StXrWVaB5piceUjvgvpZ0HU0E5U=', 'Content-Length': '0'} 2023-10-18T20:16:58.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8569a60da18008-0065303d3a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.158 DEBUG:boto:b'bofrnf-47999falseobj1nulltrue2023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:16:20.816Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:16:20.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:16:20.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:16:20.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:16:58.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:16:58.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:16:58.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:16:58.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-18T20:16:58.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-18T20:16:58.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-18T20:16:58.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-18T20:16:58.162 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.162 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.162 DEBUG:boto:path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.163 DEBUG:boto:auth_path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.163 DEBUG:boto:Method: GET 2023-10-18T20:16:58.164 DEBUG:boto:Path: /bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.164 DEBUG:boto:Data: 2023-10-18T20:16:58.164 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.165 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.165 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.165 DEBUG:boto:Params: {} 2023-10-18T20:16:58.166 DEBUG:boto:Token: None 2023-10-18T20:16:58.166 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/?versions 2023-10-18T20:16:58.167 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:StXrWVaB5piceUjvgvpZ0HU0E5U= 2023-10-18T20:16:58.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:StXrWVaB5piceUjvgvpZ0HU0E5U=', 'Content-Length': '0'} 2023-10-18T20:16:58.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071aff5b6679b271c-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.170 DEBUG:boto:b'bofrnf-47999falseobj1nulltrue2023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:16:20.816Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:16:20.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:16:20.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:16:20.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.171 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.171 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.172 DEBUG:boto:path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.172 DEBUG:boto:auth_path=/bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.172 DEBUG:boto:Method: GET 2023-10-18T20:16:58.173 DEBUG:boto:Path: /bofrnf-47/?versions&max-keys=999 2023-10-18T20:16:58.173 DEBUG:boto:Data: 2023-10-18T20:16:58.174 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.174 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.174 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.175 DEBUG:boto:Params: {} 2023-10-18T20:16:58.175 DEBUG:boto:Token: None 2023-10-18T20:16:58.176 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/?versions 2023-10-18T20:16:58.176 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:StXrWVaB5piceUjvgvpZ0HU0E5U= 2023-10-18T20:16:58.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:StXrWVaB5piceUjvgvpZ0HU0E5U=', 'Content-Length': '0'} 2023-10-18T20:16:58.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000557ad2ef8f72e601-0065303d3a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.179 DEBUG:boto:b'bofrnf-47999falseobj1nulltrue2023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:16:20.816Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:16:20.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:16:20.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:16:20.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:16:58.180 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.181 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.181 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.182 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.182 DEBUG:boto:Method: GET 2023-10-18T20:16:58.183 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.183 DEBUG:boto:Data: 2023-10-18T20:16:58.183 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.184 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.184 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.185 DEBUG:boto:Params: {} 2023-10-18T20:16:58.185 DEBUG:boto:Token: None 2023-10-18T20:16:58.185 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.186 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W+9MneRIP8c6fZRZHhZQjYPRGNc= 2023-10-18T20:16:58.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W+9MneRIP8c6fZRZHhZQjYPRGNc=', 'Content-Length': '0'} 2023-10-18T20:16:58.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e2450722baaedb3-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.189 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.189 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.190 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.190 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.191 DEBUG:boto:Method: GET 2023-10-18T20:16:58.191 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.191 DEBUG:boto:Data: 2023-10-18T20:16:58.192 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.192 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.193 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.193 DEBUG:boto:Params: {} 2023-10-18T20:16:58.194 DEBUG:boto:Token: None 2023-10-18T20:16:58.194 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.195 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W+9MneRIP8c6fZRZHhZQjYPRGNc= 2023-10-18T20:16:58.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W+9MneRIP8c6fZRZHhZQjYPRGNc=', 'Content-Length': '0'} 2023-10-18T20:16:58.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcf7ab111c24dab4-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.238 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.239 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.239 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.240 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.240 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.241 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.241 DEBUG:boto:Data: 2023-10-18T20:16:58.241 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.242 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.243 DEBUG:boto:Params: {} 2023-10-18T20:16:58.243 DEBUG:boto:Token: None 2023-10-18T20:16:58.244 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eg+PAx1ofQOiVZuTpSTkhdR1PIw= 2023-10-18T20:16:58.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eg+PAx1ofQOiVZuTpSTkhdR1PIw=', 'Content-Length': '0'} 2023-10-18T20:16:58.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043133528b7c2629a-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.247 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.247 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.247 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.248 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.248 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.249 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.249 DEBUG:boto:Data: 2023-10-18T20:16:58.249 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.250 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.250 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.250 DEBUG:boto:Params: {} 2023-10-18T20:16:58.251 DEBUG:boto:Token: None 2023-10-18T20:16:58.251 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eg+PAx1ofQOiVZuTpSTkhdR1PIw= 2023-10-18T20:16:58.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eg+PAx1ofQOiVZuTpSTkhdR1PIw=', 'Content-Length': '0'} 2023-10-18T20:16:58.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008736c51c1cb4a454-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:16:58.255 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.255 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.256 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.256 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.256 DEBUG:boto:Method: GET 2023-10-18T20:16:58.257 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.257 DEBUG:boto:Data: 2023-10-18T20:16:58.258 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.258 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.258 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.259 DEBUG:boto:Params: {} 2023-10-18T20:16:58.259 DEBUG:boto:Token: None 2023-10-18T20:16:58.260 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W+9MneRIP8c6fZRZHhZQjYPRGNc= 2023-10-18T20:16:58.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W+9MneRIP8c6fZRZHhZQjYPRGNc=', 'Content-Length': '0'} 2023-10-18T20:16:58.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5abfdbc86929c1e-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.306 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.307 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.307 DEBUG:boto:path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.308 DEBUG:boto:auth_path=/bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.308 DEBUG:boto:Method: GET 2023-10-18T20:16:58.308 DEBUG:boto:Path: /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.309 DEBUG:boto:Data: 2023-10-18T20:16:58.309 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.310 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.311 DEBUG:boto:Params: {} 2023-10-18T20:16:58.311 DEBUG:boto:Token: None 2023-10-18T20:16:58.311 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1?versionId=null 2023-10-18T20:16:58.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W+9MneRIP8c6fZRZHhZQjYPRGNc= 2023-10-18T20:16:58.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W+9MneRIP8c6fZRZHhZQjYPRGNc=', 'Content-Length': '0'} 2023-10-18T20:16:58.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4e8129b0c9e7f26-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.315 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.315 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.316 DEBUG:boto:path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.316 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.317 DEBUG:boto:Method: GET 2023-10-18T20:16:58.317 DEBUG:boto:Path: /bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.317 DEBUG:boto:Data: 2023-10-18T20:16:58.318 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.319 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.319 DEBUG:boto:Params: {} 2023-10-18T20:16:58.319 DEBUG:boto:Token: None 2023-10-18T20:16:58.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007812ed3be4e00ac0-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.323 DEBUG:boto:b'bofrnf-47obj11000falseobj12023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.324 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.324 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.325 DEBUG:boto:path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.325 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.325 DEBUG:boto:Method: GET 2023-10-18T20:16:58.326 DEBUG:boto:Path: /bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.326 DEBUG:boto:Data: 2023-10-18T20:16:58.327 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.327 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.328 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.328 DEBUG:boto:Params: {} 2023-10-18T20:16:58.328 DEBUG:boto:Token: None 2023-10-18T20:16:58.329 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.329 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec681ba81c42bb30-0065303d3a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.332 DEBUG:boto:b'bofrnf-47obj11000falseobj12023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.333 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.333 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.334 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.334 DEBUG:boto:Path: /bofrnf-47/obj1 2023-10-18T20:16:58.335 DEBUG:boto:Data: 2023-10-18T20:16:58.335 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.336 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.336 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.336 DEBUG:boto:Params: {} 2023-10-18T20:16:58.337 DEBUG:boto:Token: None 2023-10-18T20:16:58.337 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1 2023-10-18T20:16:58.338 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EcjruaphpPp/Z9TGOiOT8cm5uN8= 2023-10-18T20:16:58.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EcjruaphpPp/Z9TGOiOT8cm5uN8=', 'Content-Length': '0'} 2023-10-18T20:16:58.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000110751f72b0f6983-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.340 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.341 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.341 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.342 DEBUG:boto:Path: /bofrnf-47/obj1 2023-10-18T20:16:58.342 DEBUG:boto:Data: 2023-10-18T20:16:58.343 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.343 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.343 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.344 DEBUG:boto:Params: {} 2023-10-18T20:16:58.344 DEBUG:boto:Token: None 2023-10-18T20:16:58.345 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1 2023-10-18T20:16:58.345 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EcjruaphpPp/Z9TGOiOT8cm5uN8= 2023-10-18T20:16:58.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EcjruaphpPp/Z9TGOiOT8cm5uN8=', 'Content-Length': '0'} 2023-10-18T20:16:58.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d95a8320a54739fa-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:16:58.349 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.349 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.349 DEBUG:boto:Method: GET 2023-10-18T20:16:58.350 DEBUG:boto:Path: /bofrnf-47/obj1 2023-10-18T20:16:58.350 DEBUG:boto:Data: 2023-10-18T20:16:58.351 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.351 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.352 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.352 DEBUG:boto:Params: {} 2023-10-18T20:16:58.352 DEBUG:boto:Token: None 2023-10-18T20:16:58.353 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1 2023-10-18T20:16:58.353 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gnc5ELlIOYuNW9FDiXD9Z+FOFCs= 2023-10-18T20:16:58.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gnc5ELlIOYuNW9FDiXD9Z+FOFCs=', 'Content-Length': '0'} 2023-10-18T20:16:58.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc2ed33edba51b6d-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.356 DEBUG:boto:path=/bofrnf-47/obj1 2023-10-18T20:16:58.357 DEBUG:boto:auth_path=/bofrnf-47/obj1 2023-10-18T20:16:58.357 DEBUG:boto:Method: GET 2023-10-18T20:16:58.358 DEBUG:boto:Path: /bofrnf-47/obj1 2023-10-18T20:16:58.358 DEBUG:boto:Data: 2023-10-18T20:16:58.359 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.359 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.359 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.360 DEBUG:boto:Params: {} 2023-10-18T20:16:58.360 DEBUG:boto:Token: None 2023-10-18T20:16:58.361 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj1 2023-10-18T20:16:58.361 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gnc5ELlIOYuNW9FDiXD9Z+FOFCs= 2023-10-18T20:16:58.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gnc5ELlIOYuNW9FDiXD9Z+FOFCs=', 'Content-Length': '0'} 2023-10-18T20:16:58.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6ab69188457b9bc-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.364 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.364 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.365 DEBUG:boto:path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.365 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.365 DEBUG:boto:Method: GET 2023-10-18T20:16:58.366 DEBUG:boto:Path: /bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.366 DEBUG:boto:Data: 2023-10-18T20:16:58.367 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.368 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.368 DEBUG:boto:Params: {} 2023-10-18T20:16:58.368 DEBUG:boto:Token: None 2023-10-18T20:16:58.369 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e53be98eebd96d9b-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.372 DEBUG:boto:b'bofrnf-47obj11000falseobj12023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.373 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.374 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.374 DEBUG:boto:path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.374 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.375 DEBUG:boto:Method: GET 2023-10-18T20:16:58.375 DEBUG:boto:Path: /bofrnf-47/?prefix=obj1 2023-10-18T20:16:58.376 DEBUG:boto:Data: 2023-10-18T20:16:58.376 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.376 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.377 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.377 DEBUG:boto:Params: {} 2023-10-18T20:16:58.378 DEBUG:boto:Token: None 2023-10-18T20:16:58.378 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.379 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f692a5f85155f263-0065303d3a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.382 DEBUG:boto:b'bofrnf-47obj11000falseobj12023-10-18T20:14:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:16:58.383 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.383 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.384 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.384 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.385 DEBUG:boto:Method: GET 2023-10-18T20:16:58.385 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.385 DEBUG:boto:Data: 2023-10-18T20:16:58.386 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.387 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.387 DEBUG:boto:Params: {} 2023-10-18T20:16:58.387 DEBUG:boto:Token: None 2023-10-18T20:16:58.388 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vbfdOcdSnfqIOzsT9ytLAu8+R2M= 2023-10-18T20:16:58.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vbfdOcdSnfqIOzsT9ytLAu8+R2M=', 'Content-Length': '0'} 2023-10-18T20:16:58.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd3d2fc343244c6f-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.391 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.391 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.392 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.392 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.392 DEBUG:boto:Method: GET 2023-10-18T20:16:58.393 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.393 DEBUG:boto:Data: 2023-10-18T20:16:58.394 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.394 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.394 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.395 DEBUG:boto:Params: {} 2023-10-18T20:16:58.395 DEBUG:boto:Token: None 2023-10-18T20:16:58.396 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.396 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vbfdOcdSnfqIOzsT9ytLAu8+R2M= 2023-10-18T20:16:58.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vbfdOcdSnfqIOzsT9ytLAu8+R2M=', 'Content-Length': '0'} 2023-10-18T20:16:58.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfcdaaf7eff3c9bd-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.399 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.399 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.400 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.400 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.400 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.401 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.401 DEBUG:boto:Data: 2023-10-18T20:16:58.401 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.402 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.402 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.402 DEBUG:boto:Params: {} 2023-10-18T20:16:58.403 DEBUG:boto:Token: None 2023-10-18T20:16:58.403 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q2YOlbGNvACGY5k3z98xuQWEVqM= 2023-10-18T20:16:58.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q2YOlbGNvACGY5k3z98xuQWEVqM=', 'Content-Length': '0'} 2023-10-18T20:16:58.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f67f87088316e98-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.406 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.407 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.407 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.407 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.408 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.408 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.409 DEBUG:boto:Data: 2023-10-18T20:16:58.409 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.410 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.410 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.410 DEBUG:boto:Params: {} 2023-10-18T20:16:58.411 DEBUG:boto:Token: None 2023-10-18T20:16:58.411 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.412 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q2YOlbGNvACGY5k3z98xuQWEVqM= 2023-10-18T20:16:58.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q2YOlbGNvACGY5k3z98xuQWEVqM=', 'Content-Length': '0'} 2023-10-18T20:16:58.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091faeb959917fafe-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:16:58.415 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.415 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.416 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.416 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.416 DEBUG:boto:Method: GET 2023-10-18T20:16:58.417 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.417 DEBUG:boto:Data: 2023-10-18T20:16:58.418 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.418 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.419 DEBUG:boto:Params: {} 2023-10-18T20:16:58.419 DEBUG:boto:Token: None 2023-10-18T20:16:58.420 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vbfdOcdSnfqIOzsT9ytLAu8+R2M= 2023-10-18T20:16:58.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vbfdOcdSnfqIOzsT9ytLAu8+R2M=', 'Content-Length': '0'} 2023-10-18T20:16:58.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed9ac61de2226487-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.466 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.467 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.467 DEBUG:boto:path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.467 DEBUG:boto:auth_path=/bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.468 DEBUG:boto:Method: GET 2023-10-18T20:16:58.468 DEBUG:boto:Path: /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.469 DEBUG:boto:Data: 2023-10-18T20:16:58.469 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.469 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.470 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.470 DEBUG:boto:Params: {} 2023-10-18T20:16:58.471 DEBUG:boto:Token: None 2023-10-18T20:16:58.471 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2?versionId=null 2023-10-18T20:16:58.472 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vbfdOcdSnfqIOzsT9ytLAu8+R2M= 2023-10-18T20:16:58.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vbfdOcdSnfqIOzsT9ytLAu8+R2M=', 'Content-Length': '0'} 2023-10-18T20:16:58.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f2d73840541fc7f-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.474 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.475 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.475 DEBUG:boto:path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.476 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.476 DEBUG:boto:Method: GET 2023-10-18T20:16:58.476 DEBUG:boto:Path: /bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.477 DEBUG:boto:Data: 2023-10-18T20:16:58.477 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.478 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.478 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.478 DEBUG:boto:Params: {} 2023-10-18T20:16:58.479 DEBUG:boto:Token: None 2023-10-18T20:16:58.479 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.480 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ae37f2e2fff6a0-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.483 DEBUG:boto:b'bofrnf-47obj21000falseobj22023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.484 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.484 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.485 DEBUG:boto:path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.485 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.485 DEBUG:boto:Method: GET 2023-10-18T20:16:58.486 DEBUG:boto:Path: /bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.486 DEBUG:boto:Data: 2023-10-18T20:16:58.487 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.487 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.487 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.488 DEBUG:boto:Params: {} 2023-10-18T20:16:58.488 DEBUG:boto:Token: None 2023-10-18T20:16:58.489 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.489 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdcd6cf97c27fc7f-0065303d3a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.493 DEBUG:boto:b'bofrnf-47obj21000falseobj22023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.493 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.494 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.494 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.495 DEBUG:boto:Path: /bofrnf-47/obj2 2023-10-18T20:16:58.495 DEBUG:boto:Data: 2023-10-18T20:16:58.495 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.496 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.496 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.497 DEBUG:boto:Params: {} 2023-10-18T20:16:58.497 DEBUG:boto:Token: None 2023-10-18T20:16:58.497 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2 2023-10-18T20:16:58.498 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ipHmjZA40IZ1LUVpERY/+0kjdEE= 2023-10-18T20:16:58.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ipHmjZA40IZ1LUVpERY/+0kjdEE=', 'Content-Length': '0'} 2023-10-18T20:16:58.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012a23ed7690ec58e-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.501 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.501 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.501 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.502 DEBUG:boto:Path: /bofrnf-47/obj2 2023-10-18T20:16:58.502 DEBUG:boto:Data: 2023-10-18T20:16:58.503 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.503 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.504 DEBUG:boto:Params: {} 2023-10-18T20:16:58.504 DEBUG:boto:Token: None 2023-10-18T20:16:58.505 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2 2023-10-18T20:16:58.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ipHmjZA40IZ1LUVpERY/+0kjdEE= 2023-10-18T20:16:58.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ipHmjZA40IZ1LUVpERY/+0kjdEE=', 'Content-Length': '0'} 2023-10-18T20:16:58.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a55515c9d4c11f85-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:16:58.508 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.509 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.509 DEBUG:boto:Method: GET 2023-10-18T20:16:58.510 DEBUG:boto:Path: /bofrnf-47/obj2 2023-10-18T20:16:58.510 DEBUG:boto:Data: 2023-10-18T20:16:58.511 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.511 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.511 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.512 DEBUG:boto:Params: {} 2023-10-18T20:16:58.512 DEBUG:boto:Token: None 2023-10-18T20:16:58.513 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2 2023-10-18T20:16:58.513 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ITBXrfWkORGNaHtcjtrAO6vUPHk= 2023-10-18T20:16:58.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ITBXrfWkORGNaHtcjtrAO6vUPHk=', 'Content-Length': '0'} 2023-10-18T20:16:58.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c312d44b861758e8-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.516 DEBUG:boto:path=/bofrnf-47/obj2 2023-10-18T20:16:58.516 DEBUG:boto:auth_path=/bofrnf-47/obj2 2023-10-18T20:16:58.517 DEBUG:boto:Method: GET 2023-10-18T20:16:58.517 DEBUG:boto:Path: /bofrnf-47/obj2 2023-10-18T20:16:58.518 DEBUG:boto:Data: 2023-10-18T20:16:58.518 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.518 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.519 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.519 DEBUG:boto:Params: {} 2023-10-18T20:16:58.520 DEBUG:boto:Token: None 2023-10-18T20:16:58.520 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj2 2023-10-18T20:16:58.521 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ITBXrfWkORGNaHtcjtrAO6vUPHk= 2023-10-18T20:16:58.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ITBXrfWkORGNaHtcjtrAO6vUPHk=', 'Content-Length': '0'} 2023-10-18T20:16:58.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f9672ccde3de15f-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.523 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.524 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.524 DEBUG:boto:path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.525 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.525 DEBUG:boto:Method: GET 2023-10-18T20:16:58.525 DEBUG:boto:Path: /bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.526 DEBUG:boto:Data: 2023-10-18T20:16:58.526 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.527 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.527 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.527 DEBUG:boto:Params: {} 2023-10-18T20:16:58.528 DEBUG:boto:Token: None 2023-10-18T20:16:58.528 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.529 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000344348fe89ce0df8-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.532 DEBUG:boto:b'bofrnf-47obj21000falseobj22023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.533 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.533 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.533 DEBUG:boto:path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.534 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.534 DEBUG:boto:Method: GET 2023-10-18T20:16:58.535 DEBUG:boto:Path: /bofrnf-47/?prefix=obj2 2023-10-18T20:16:58.535 DEBUG:boto:Data: 2023-10-18T20:16:58.535 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.536 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.536 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.537 DEBUG:boto:Params: {} 2023-10-18T20:16:58.537 DEBUG:boto:Token: None 2023-10-18T20:16:58.537 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.538 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000367d59eee5420746-0065303d3a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.541 DEBUG:boto:b'bofrnf-47obj21000falseobj22023-10-18T20:14:17.496Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:16:58.542 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.543 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.543 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.543 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.544 DEBUG:boto:Method: GET 2023-10-18T20:16:58.544 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.544 DEBUG:boto:Data: 2023-10-18T20:16:58.545 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.545 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.546 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.546 DEBUG:boto:Params: {} 2023-10-18T20:16:58.547 DEBUG:boto:Token: None 2023-10-18T20:16:58.547 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.547 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pgbf+NpE5qabe+c4pACVrf6aRh8= 2023-10-18T20:16:58.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pgbf+NpE5qabe+c4pACVrf6aRh8=', 'Content-Length': '0'} 2023-10-18T20:16:58.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009622e95dc384f4b7-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.590 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.591 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.591 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.592 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.592 DEBUG:boto:Method: GET 2023-10-18T20:16:58.592 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.593 DEBUG:boto:Data: 2023-10-18T20:16:58.593 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.594 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.594 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.595 DEBUG:boto:Params: {} 2023-10-18T20:16:58.595 DEBUG:boto:Token: None 2023-10-18T20:16:58.596 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.596 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pgbf+NpE5qabe+c4pACVrf6aRh8= 2023-10-18T20:16:58.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pgbf+NpE5qabe+c4pACVrf6aRh8=', 'Content-Length': '0'} 2023-10-18T20:16:58.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8b0784ed0965088-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.599 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.600 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.600 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.600 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.601 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.601 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.601 DEBUG:boto:Data: 2023-10-18T20:16:58.602 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.603 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.603 DEBUG:boto:Params: {} 2023-10-18T20:16:58.603 DEBUG:boto:Token: None 2023-10-18T20:16:58.604 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2jkLkBxAySPddTMi3LJT4vHYLyU= 2023-10-18T20:16:58.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2jkLkBxAySPddTMi3LJT4vHYLyU=', 'Content-Length': '0'} 2023-10-18T20:16:58.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7c8602349211087-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.607 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.607 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.608 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.608 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.608 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.609 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.609 DEBUG:boto:Data: 2023-10-18T20:16:58.609 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.610 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.610 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.611 DEBUG:boto:Params: {} 2023-10-18T20:16:58.611 DEBUG:boto:Token: None 2023-10-18T20:16:58.611 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.612 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2jkLkBxAySPddTMi3LJT4vHYLyU= 2023-10-18T20:16:58.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2jkLkBxAySPddTMi3LJT4vHYLyU=', 'Content-Length': '0'} 2023-10-18T20:16:58.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cb05678873a0e00-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:16:58.615 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.616 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.616 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.617 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.617 DEBUG:boto:Method: GET 2023-10-18T20:16:58.617 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.618 DEBUG:boto:Data: 2023-10-18T20:16:58.618 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.619 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.619 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.620 DEBUG:boto:Params: {} 2023-10-18T20:16:58.620 DEBUG:boto:Token: None 2023-10-18T20:16:58.620 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.621 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pgbf+NpE5qabe+c4pACVrf6aRh8= 2023-10-18T20:16:58.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pgbf+NpE5qabe+c4pACVrf6aRh8=', 'Content-Length': '0'} 2023-10-18T20:16:58.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5aec0ca912eb54b-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.623 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.624 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.624 DEBUG:boto:path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.625 DEBUG:boto:auth_path=/bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.625 DEBUG:boto:Method: GET 2023-10-18T20:16:58.626 DEBUG:boto:Path: /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.626 DEBUG:boto:Data: 2023-10-18T20:16:58.626 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.627 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.627 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.628 DEBUG:boto:Params: {} 2023-10-18T20:16:58.628 DEBUG:boto:Token: None 2023-10-18T20:16:58.629 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3?versionId=null 2023-10-18T20:16:58.629 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pgbf+NpE5qabe+c4pACVrf6aRh8= 2023-10-18T20:16:58.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pgbf+NpE5qabe+c4pACVrf6aRh8=', 'Content-Length': '0'} 2023-10-18T20:16:58.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bdb67db5392bcc4-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.678 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.679 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.679 DEBUG:boto:path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.680 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.680 DEBUG:boto:Method: GET 2023-10-18T20:16:58.681 DEBUG:boto:Path: /bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.681 DEBUG:boto:Data: 2023-10-18T20:16:58.682 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.682 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.682 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.683 DEBUG:boto:Params: {} 2023-10-18T20:16:58.683 DEBUG:boto:Token: None 2023-10-18T20:16:58.684 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.684 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001afe019e4840f0a7-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.688 DEBUG:boto:b'bofrnf-47obj31000falseobj32023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.688 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.689 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.689 DEBUG:boto:path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.690 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.690 DEBUG:boto:Method: GET 2023-10-18T20:16:58.690 DEBUG:boto:Path: /bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.691 DEBUG:boto:Data: 2023-10-18T20:16:58.691 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.692 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.692 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.692 DEBUG:boto:Params: {} 2023-10-18T20:16:58.693 DEBUG:boto:Token: None 2023-10-18T20:16:58.693 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.694 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be30aafc619372c3-0065303d3a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.697 DEBUG:boto:b'bofrnf-47obj31000falseobj32023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.698 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.698 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.699 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.699 DEBUG:boto:Path: /bofrnf-47/obj3 2023-10-18T20:16:58.699 DEBUG:boto:Data: 2023-10-18T20:16:58.700 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.700 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.701 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.701 DEBUG:boto:Params: {} 2023-10-18T20:16:58.702 DEBUG:boto:Token: None 2023-10-18T20:16:58.702 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3 2023-10-18T20:16:58.703 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XHGp/suL+0SA33Ng4vIKXaTDtes= 2023-10-18T20:16:58.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XHGp/suL+0SA33Ng4vIKXaTDtes=', 'Content-Length': '0'} 2023-10-18T20:16:58.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bb32eda72c721c9-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.705 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.706 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.706 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.707 DEBUG:boto:Path: /bofrnf-47/obj3 2023-10-18T20:16:58.707 DEBUG:boto:Data: 2023-10-18T20:16:58.707 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.708 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.708 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.708 DEBUG:boto:Params: {} 2023-10-18T20:16:58.709 DEBUG:boto:Token: None 2023-10-18T20:16:58.709 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3 2023-10-18T20:16:58.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XHGp/suL+0SA33Ng4vIKXaTDtes= 2023-10-18T20:16:58.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XHGp/suL+0SA33Ng4vIKXaTDtes=', 'Content-Length': '0'} 2023-10-18T20:16:58.712 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b8c4e1ba2890121-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:16:58.713 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.713 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.714 DEBUG:boto:Method: GET 2023-10-18T20:16:58.714 DEBUG:boto:Path: /bofrnf-47/obj3 2023-10-18T20:16:58.715 DEBUG:boto:Data: 2023-10-18T20:16:58.715 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.716 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.716 DEBUG:boto:Params: {} 2023-10-18T20:16:58.716 DEBUG:boto:Token: None 2023-10-18T20:16:58.717 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3 2023-10-18T20:16:58.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:N+31n3qozLyxsvy2gWqV0nVaxfI= 2023-10-18T20:16:58.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:N+31n3qozLyxsvy2gWqV0nVaxfI=', 'Content-Length': '0'} 2023-10-18T20:16:58.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dacd3697c24500dc-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.766 DEBUG:boto:path=/bofrnf-47/obj3 2023-10-18T20:16:58.767 DEBUG:boto:auth_path=/bofrnf-47/obj3 2023-10-18T20:16:58.767 DEBUG:boto:Method: GET 2023-10-18T20:16:58.768 DEBUG:boto:Path: /bofrnf-47/obj3 2023-10-18T20:16:58.768 DEBUG:boto:Data: 2023-10-18T20:16:58.769 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.769 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.769 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.770 DEBUG:boto:Params: {} 2023-10-18T20:16:58.770 DEBUG:boto:Token: None 2023-10-18T20:16:58.771 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj3 2023-10-18T20:16:58.771 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:N+31n3qozLyxsvy2gWqV0nVaxfI= 2023-10-18T20:16:58.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:N+31n3qozLyxsvy2gWqV0nVaxfI=', 'Content-Length': '0'} 2023-10-18T20:16:58.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000458bf93f3b029724-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.774 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.775 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.775 DEBUG:boto:path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.776 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.776 DEBUG:boto:Method: GET 2023-10-18T20:16:58.777 DEBUG:boto:Path: /bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.777 DEBUG:boto:Data: 2023-10-18T20:16:58.777 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.778 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.778 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.779 DEBUG:boto:Params: {} 2023-10-18T20:16:58.779 DEBUG:boto:Token: None 2023-10-18T20:16:58.779 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d240fc0d07c6eae9-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.783 DEBUG:boto:b'bofrnf-47obj31000falseobj32023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.784 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.784 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.784 DEBUG:boto:path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.785 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.785 DEBUG:boto:Method: GET 2023-10-18T20:16:58.786 DEBUG:boto:Path: /bofrnf-47/?prefix=obj3 2023-10-18T20:16:58.786 DEBUG:boto:Data: 2023-10-18T20:16:58.786 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.787 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.788 DEBUG:boto:Params: {} 2023-10-18T20:16:58.788 DEBUG:boto:Token: None 2023-10-18T20:16:58.789 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ff68a6f861bc739-0065303d3a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.792 DEBUG:boto:b'bofrnf-47obj31000falseobj32023-10-18T20:14:17.519Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:16:58.793 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.794 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.794 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.795 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.795 DEBUG:boto:Method: GET 2023-10-18T20:16:58.795 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.796 DEBUG:boto:Data: 2023-10-18T20:16:58.796 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.797 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.797 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.798 DEBUG:boto:Params: {} 2023-10-18T20:16:58.798 DEBUG:boto:Token: None 2023-10-18T20:16:58.799 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ralPrcEwC2ILNxY2PNMzLqPTFa8= 2023-10-18T20:16:58.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ralPrcEwC2ILNxY2PNMzLqPTFa8=', 'Content-Length': '0'} 2023-10-18T20:16:58.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000856a409aefc85232-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.802 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.803 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.803 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.803 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.804 DEBUG:boto:Method: GET 2023-10-18T20:16:58.804 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.805 DEBUG:boto:Data: 2023-10-18T20:16:58.805 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.806 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.806 DEBUG:boto:Params: {} 2023-10-18T20:16:58.807 DEBUG:boto:Token: None 2023-10-18T20:16:58.807 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.808 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ralPrcEwC2ILNxY2PNMzLqPTFa8= 2023-10-18T20:16:58.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ralPrcEwC2ILNxY2PNMzLqPTFa8=', 'Content-Length': '0'} 2023-10-18T20:16:58.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000276b7dc0666c86e0-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.811 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.811 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.812 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.812 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.813 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.813 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.813 DEBUG:boto:Data: 2023-10-18T20:16:58.814 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.815 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.815 DEBUG:boto:Params: {} 2023-10-18T20:16:58.815 DEBUG:boto:Token: None 2023-10-18T20:16:58.816 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b2gOW3XhaxYES6GU4tiHACYzU7I= 2023-10-18T20:16:58.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b2gOW3XhaxYES6GU4tiHACYzU7I=', 'Content-Length': '0'} 2023-10-18T20:16:58.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000818036b18b0ef42f-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.819 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.819 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.820 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.820 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.821 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.821 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.821 DEBUG:boto:Data: 2023-10-18T20:16:58.822 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.822 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.822 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.823 DEBUG:boto:Params: {} 2023-10-18T20:16:58.823 DEBUG:boto:Token: None 2023-10-18T20:16:58.824 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b2gOW3XhaxYES6GU4tiHACYzU7I= 2023-10-18T20:16:58.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b2gOW3XhaxYES6GU4tiHACYzU7I=', 'Content-Length': '0'} 2023-10-18T20:16:58.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b406cf4f8618d7b-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:16:58.828 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.828 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.828 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.829 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.829 DEBUG:boto:Method: GET 2023-10-18T20:16:58.830 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.830 DEBUG:boto:Data: 2023-10-18T20:16:58.831 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.831 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.831 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.832 DEBUG:boto:Params: {} 2023-10-18T20:16:58.832 DEBUG:boto:Token: None 2023-10-18T20:16:58.833 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.833 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ralPrcEwC2ILNxY2PNMzLqPTFa8= 2023-10-18T20:16:58.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ralPrcEwC2ILNxY2PNMzLqPTFa8=', 'Content-Length': '0'} 2023-10-18T20:16:58.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022b972d4ef5e95d6-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.878 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.879 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.880 DEBUG:boto:path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.880 DEBUG:boto:auth_path=/bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.880 DEBUG:boto:Method: GET 2023-10-18T20:16:58.881 DEBUG:boto:Path: /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.881 DEBUG:boto:Data: 2023-10-18T20:16:58.882 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.882 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.883 DEBUG:boto:Params: {} 2023-10-18T20:16:58.883 DEBUG:boto:Token: None 2023-10-18T20:16:58.884 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4?versionId=null 2023-10-18T20:16:58.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ralPrcEwC2ILNxY2PNMzLqPTFa8= 2023-10-18T20:16:58.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ralPrcEwC2ILNxY2PNMzLqPTFa8=', 'Content-Length': '0'} 2023-10-18T20:16:58.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ded8246c052181ef-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.887 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.888 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.888 DEBUG:boto:path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:58.889 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:58.889 DEBUG:boto:Method: GET 2023-10-18T20:16:58.889 DEBUG:boto:Path: /bofrnf-47/?prefix=obj4 2023-10-18T20:16:58.890 DEBUG:boto:Data: 2023-10-18T20:16:58.890 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.891 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.891 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.892 DEBUG:boto:Params: {} 2023-10-18T20:16:58.892 DEBUG:boto:Token: None 2023-10-18T20:16:58.893 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.893 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000014dabec1f6db934-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.896 DEBUG:boto:b'bofrnf-47obj41000falseobj42023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.897 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.898 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.898 DEBUG:boto:path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:58.899 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:58.899 DEBUG:boto:Method: GET 2023-10-18T20:16:58.899 DEBUG:boto:Path: /bofrnf-47/?prefix=obj4 2023-10-18T20:16:58.900 DEBUG:boto:Data: 2023-10-18T20:16:58.900 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.901 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.901 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.902 DEBUG:boto:Params: {} 2023-10-18T20:16:58.902 DEBUG:boto:Token: None 2023-10-18T20:16:58.903 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.903 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000274971307027351d-0065303d3a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.946 DEBUG:boto:b'bofrnf-47obj41000falseobj42023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:58.947 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.948 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.948 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.949 DEBUG:boto:Path: /bofrnf-47/obj4 2023-10-18T20:16:58.949 DEBUG:boto:Data: 2023-10-18T20:16:58.949 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.950 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.950 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.951 DEBUG:boto:Params: {} 2023-10-18T20:16:58.951 DEBUG:boto:Token: None 2023-10-18T20:16:58.952 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4 2023-10-18T20:16:58.952 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uIo1c1+cnL1dUXqhkaKBWqceKkU= 2023-10-18T20:16:58.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uIo1c1+cnL1dUXqhkaKBWqceKkU=', 'Content-Length': '0'} 2023-10-18T20:16:58.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005233f3c2d8c29a48-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.955 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.956 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.956 DEBUG:boto:Method: HEAD 2023-10-18T20:16:58.956 DEBUG:boto:Path: /bofrnf-47/obj4 2023-10-18T20:16:58.957 DEBUG:boto:Data: 2023-10-18T20:16:58.957 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.958 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.958 DEBUG:boto:Params: {} 2023-10-18T20:16:58.959 DEBUG:boto:Token: None 2023-10-18T20:16:58.959 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4 2023-10-18T20:16:58.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uIo1c1+cnL1dUXqhkaKBWqceKkU= 2023-10-18T20:16:58.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uIo1c1+cnL1dUXqhkaKBWqceKkU=', 'Content-Length': '0'} 2023-10-18T20:16:58.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042d945c301823e88-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:16:58.963 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.963 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.964 DEBUG:boto:Method: GET 2023-10-18T20:16:58.964 DEBUG:boto:Path: /bofrnf-47/obj4 2023-10-18T20:16:58.965 DEBUG:boto:Data: 2023-10-18T20:16:58.965 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.966 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.966 DEBUG:boto:Params: {} 2023-10-18T20:16:58.967 DEBUG:boto:Token: None 2023-10-18T20:16:58.967 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4 2023-10-18T20:16:58.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d8cKtNf9ZeIZOC2CBtuHYBl1NZ4= 2023-10-18T20:16:58.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d8cKtNf9ZeIZOC2CBtuHYBl1NZ4=', 'Content-Length': '0'} 2023-10-18T20:16:58.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014aa1ca705d6141e-0065303d3a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.971 DEBUG:boto:path=/bofrnf-47/obj4 2023-10-18T20:16:58.971 DEBUG:boto:auth_path=/bofrnf-47/obj4 2023-10-18T20:16:58.972 DEBUG:boto:Method: GET 2023-10-18T20:16:58.972 DEBUG:boto:Path: /bofrnf-47/obj4 2023-10-18T20:16:58.972 DEBUG:boto:Data: 2023-10-18T20:16:58.973 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.973 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:58.974 DEBUG:boto:Port: 8000 2023-10-18T20:16:58.974 DEBUG:boto:Params: {} 2023-10-18T20:16:58.975 DEBUG:boto:Token: None 2023-10-18T20:16:58.975 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/obj4 2023-10-18T20:16:58.975 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d8cKtNf9ZeIZOC2CBtuHYBl1NZ4= 2023-10-18T20:16:58.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d8cKtNf9ZeIZOC2CBtuHYBl1NZ4=', 'Content-Length': '0'} 2023-10-18T20:16:58.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:14:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8d01db3cb5e2abb-0065303d3a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:58.978 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:58.979 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:58.979 DEBUG:boto:path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:58.979 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:58.980 DEBUG:boto:Method: GET 2023-10-18T20:16:58.980 DEBUG:boto:Path: /bofrnf-47/?prefix=obj4 2023-10-18T20:16:58.981 DEBUG:boto:Data: 2023-10-18T20:16:58.981 DEBUG:boto:Headers: {} 2023-10-18T20:16:58.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:58.982 DEBUG:boto:Port: 8001 2023-10-18T20:16:58.982 DEBUG:boto:Params: {} 2023-10-18T20:16:58.983 DEBUG:boto:Token: None 2023-10-18T20:16:58.983 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:58 GMT /bofrnf-47/ 2023-10-18T20:16:58.984 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g= 2023-10-18T20:16:58.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I8BfvgNdlszRm/TwkxGrY1Kzd+g=', 'Content-Length': '0'} 2023-10-18T20:16:58.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d191c17a2a26c2b7-0065303d3a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.030 DEBUG:boto:b'bofrnf-47obj41000falseobj42023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.031 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.032 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.032 DEBUG:boto:path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:59.032 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj4 2023-10-18T20:16:59.033 DEBUG:boto:Method: GET 2023-10-18T20:16:59.033 DEBUG:boto:Path: /bofrnf-47/?prefix=obj4 2023-10-18T20:16:59.034 DEBUG:boto:Data: 2023-10-18T20:16:59.034 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.034 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.035 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.035 DEBUG:boto:Params: {} 2023-10-18T20:16:59.036 DEBUG:boto:Token: None 2023-10-18T20:16:59.036 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.037 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087f4170db45ce965-0065303d3b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.040 DEBUG:boto:b'bofrnf-47obj41000falseobj42023-10-18T20:14:17.538Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:16:59.041 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.041 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.041 DEBUG:boto:path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.042 DEBUG:boto:auth_path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.042 DEBUG:boto:Method: GET 2023-10-18T20:16:59.043 DEBUG:boto:Path: /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.043 DEBUG:boto:Data: 2023-10-18T20:16:59.044 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.044 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.044 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.045 DEBUG:boto:Params: {} 2023-10-18T20:16:59.045 DEBUG:boto:Token: None 2023-10-18T20:16:59.046 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.046 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MSZ8BKecu2z/mZNKTrVtOqInomw= 2023-10-18T20:16:59.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MSZ8BKecu2z/mZNKTrVtOqInomw=', 'Content-Length': '0'} 2023-10-18T20:16:59.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9a272db49be518d-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.049 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.050 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.050 DEBUG:boto:path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.051 DEBUG:boto:auth_path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.051 DEBUG:boto:Method: GET 2023-10-18T20:16:59.051 DEBUG:boto:Path: /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.052 DEBUG:boto:Data: 2023-10-18T20:16:59.052 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.053 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.053 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.053 DEBUG:boto:Params: {} 2023-10-18T20:16:59.054 DEBUG:boto:Token: None 2023-10-18T20:16:59.054 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.055 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MSZ8BKecu2z/mZNKTrVtOqInomw= 2023-10-18T20:16:59.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MSZ8BKecu2z/mZNKTrVtOqInomw=', 'Content-Length': '0'} 2023-10-18T20:16:59.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005395c7bcce82cff7-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.058 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.058 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.059 DEBUG:boto:path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.059 DEBUG:boto:auth_path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.060 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.060 DEBUG:boto:Path: /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.060 DEBUG:boto:Data: 2023-10-18T20:16:59.061 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.061 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.061 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.062 DEBUG:boto:Params: {} 2023-10-18T20:16:59.062 DEBUG:boto:Token: None 2023-10-18T20:16:59.063 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.063 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zOIwtHoPXYU4DEzo7ymF4SIZQBM= 2023-10-18T20:16:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zOIwtHoPXYU4DEzo7ymF4SIZQBM=', 'Content-Length': '0'} 2023-10-18T20:16:59.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007539c63c73260379-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.066 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.066 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.067 DEBUG:boto:path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.067 DEBUG:boto:auth_path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.067 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.068 DEBUG:boto:Path: /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.068 DEBUG:boto:Data: 2023-10-18T20:16:59.068 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.069 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.069 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.070 DEBUG:boto:Params: {} 2023-10-18T20:16:59.070 DEBUG:boto:Token: None 2023-10-18T20:16:59.070 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.071 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zOIwtHoPXYU4DEzo7ymF4SIZQBM= 2023-10-18T20:16:59.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zOIwtHoPXYU4DEzo7ymF4SIZQBM=', 'Content-Length': '0'} 2023-10-18T20:16:59.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe81e7948b120744-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:16:59.074 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.074 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.075 DEBUG:boto:path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.075 DEBUG:boto:auth_path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.075 DEBUG:boto:Method: GET 2023-10-18T20:16:59.076 DEBUG:boto:Path: /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.076 DEBUG:boto:Data: 2023-10-18T20:16:59.077 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.077 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.078 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.078 DEBUG:boto:Params: {} 2023-10-18T20:16:59.078 DEBUG:boto:Token: None 2023-10-18T20:16:59.079 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.079 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MSZ8BKecu2z/mZNKTrVtOqInomw= 2023-10-18T20:16:59.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MSZ8BKecu2z/mZNKTrVtOqInomw=', 'Content-Length': '0'} 2023-10-18T20:16:59.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019f4c33240a3a05f-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.082 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.083 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.083 DEBUG:boto:path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.083 DEBUG:boto:auth_path=/bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.084 DEBUG:boto:Method: GET 2023-10-18T20:16:59.084 DEBUG:boto:Path: /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.085 DEBUG:boto:Data: 2023-10-18T20:16:59.085 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.086 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.086 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.086 DEBUG:boto:Params: {} 2023-10-18T20:16:59.087 DEBUG:boto:Token: None 2023-10-18T20:16:59.087 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5?versionId=null 2023-10-18T20:16:59.088 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MSZ8BKecu2z/mZNKTrVtOqInomw= 2023-10-18T20:16:59.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MSZ8BKecu2z/mZNKTrVtOqInomw=', 'Content-Length': '0'} 2023-10-18T20:16:59.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000040df9c9d2dc968d-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.091 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.091 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.092 DEBUG:boto:path=/bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.092 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.093 DEBUG:boto:Method: GET 2023-10-18T20:16:59.093 DEBUG:boto:Path: /bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.093 DEBUG:boto:Data: 2023-10-18T20:16:59.094 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.094 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.095 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.095 DEBUG:boto:Params: {} 2023-10-18T20:16:59.096 DEBUG:boto:Token: None 2023-10-18T20:16:59.096 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.097 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005fe3babd40649de-0065303d3b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.099 DEBUG:boto:b'bofrnf-47obj51000falseobj52023-10-18T20:16:20.816Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.100 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.101 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.101 DEBUG:boto:path=/bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.101 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.102 DEBUG:boto:Method: GET 2023-10-18T20:16:59.102 DEBUG:boto:Path: /bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.103 DEBUG:boto:Data: 2023-10-18T20:16:59.103 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.103 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.104 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.104 DEBUG:boto:Params: {} 2023-10-18T20:16:59.105 DEBUG:boto:Token: None 2023-10-18T20:16:59.105 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.105 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036b53ebb0f7e8e77-0065303d3b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.150 DEBUG:boto:b'bofrnf-47obj51000falseobj52023-10-18T20:16:20.816Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.151 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.151 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.152 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.152 DEBUG:boto:Path: /bofrnf-47/obj5 2023-10-18T20:16:59.152 DEBUG:boto:Data: 2023-10-18T20:16:59.153 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.153 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.154 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.154 DEBUG:boto:Params: {} 2023-10-18T20:16:59.155 DEBUG:boto:Token: None 2023-10-18T20:16:59.155 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5 2023-10-18T20:16:59.156 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iH0QXZUHWuWw0I1g+alMGmtaT8c= 2023-10-18T20:16:59.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iH0QXZUHWuWw0I1g+alMGmtaT8c=', 'Content-Length': '0'} 2023-10-18T20:16:59.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c95fec24cb944f18-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.158 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.159 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.159 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.160 DEBUG:boto:Path: /bofrnf-47/obj5 2023-10-18T20:16:59.160 DEBUG:boto:Data: 2023-10-18T20:16:59.160 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.161 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.161 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.162 DEBUG:boto:Params: {} 2023-10-18T20:16:59.162 DEBUG:boto:Token: None 2023-10-18T20:16:59.162 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5 2023-10-18T20:16:59.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iH0QXZUHWuWw0I1g+alMGmtaT8c= 2023-10-18T20:16:59.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iH0QXZUHWuWw0I1g+alMGmtaT8c=', 'Content-Length': '0'} 2023-10-18T20:16:59.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001dca381217260d0e-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:16:59.166 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.167 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.167 DEBUG:boto:Method: GET 2023-10-18T20:16:59.167 DEBUG:boto:Path: /bofrnf-47/obj5 2023-10-18T20:16:59.168 DEBUG:boto:Data: 2023-10-18T20:16:59.168 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.169 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.169 DEBUG:boto:Params: {} 2023-10-18T20:16:59.170 DEBUG:boto:Token: None 2023-10-18T20:16:59.170 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5 2023-10-18T20:16:59.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxVMjVcYjTQW94KMT2OAcikgSsM= 2023-10-18T20:16:59.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxVMjVcYjTQW94KMT2OAcikgSsM=', 'Content-Length': '0'} 2023-10-18T20:16:59.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba84b599d34d1906-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.174 DEBUG:boto:path=/bofrnf-47/obj5 2023-10-18T20:16:59.174 DEBUG:boto:auth_path=/bofrnf-47/obj5 2023-10-18T20:16:59.175 DEBUG:boto:Method: GET 2023-10-18T20:16:59.175 DEBUG:boto:Path: /bofrnf-47/obj5 2023-10-18T20:16:59.175 DEBUG:boto:Data: 2023-10-18T20:16:59.176 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.176 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.177 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.177 DEBUG:boto:Params: {} 2023-10-18T20:16:59.178 DEBUG:boto:Token: None 2023-10-18T20:16:59.178 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj5 2023-10-18T20:16:59.178 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxVMjVcYjTQW94KMT2OAcikgSsM= 2023-10-18T20:16:59.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxVMjVcYjTQW94KMT2OAcikgSsM=', 'Content-Length': '0'} 2023-10-18T20:16:59.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed5a8a1a70db40c2-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.181 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.182 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.182 DEBUG:boto:path=/bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.183 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.183 DEBUG:boto:Method: GET 2023-10-18T20:16:59.184 DEBUG:boto:Path: /bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.184 DEBUG:boto:Data: 2023-10-18T20:16:59.184 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.185 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.185 DEBUG:boto:Params: {} 2023-10-18T20:16:59.186 DEBUG:boto:Token: None 2023-10-18T20:16:59.186 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.187 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb7fb7472db7b0bf-0065303d3b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.190 DEBUG:boto:b'bofrnf-47obj51000falseobj52023-10-18T20:16:20.816Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.191 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.191 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.192 DEBUG:boto:path=/bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.192 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.192 DEBUG:boto:Method: GET 2023-10-18T20:16:59.193 DEBUG:boto:Path: /bofrnf-47/?prefix=obj5 2023-10-18T20:16:59.193 DEBUG:boto:Data: 2023-10-18T20:16:59.193 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.194 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.194 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.195 DEBUG:boto:Params: {} 2023-10-18T20:16:59.195 DEBUG:boto:Token: None 2023-10-18T20:16:59.196 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.196 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c541afc921f405ec-0065303d3b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.199 DEBUG:boto:b'bofrnf-47obj51000falseobj52023-10-18T20:16:20.816Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:16:59.201 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.201 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.201 DEBUG:boto:path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.202 DEBUG:boto:auth_path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.202 DEBUG:boto:Method: GET 2023-10-18T20:16:59.203 DEBUG:boto:Path: /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.203 DEBUG:boto:Data: 2023-10-18T20:16:59.203 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.204 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.204 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.205 DEBUG:boto:Params: {} 2023-10-18T20:16:59.205 DEBUG:boto:Token: None 2023-10-18T20:16:59.206 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.206 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jc5k2qUnRpvNm/aj2xWnw9fr79Y= 2023-10-18T20:16:59.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jc5k2qUnRpvNm/aj2xWnw9fr79Y=', 'Content-Length': '0'} 2023-10-18T20:16:59.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1e7a4dd7656cacb-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.209 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.209 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.210 DEBUG:boto:path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.210 DEBUG:boto:auth_path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.210 DEBUG:boto:Method: GET 2023-10-18T20:16:59.211 DEBUG:boto:Path: /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.211 DEBUG:boto:Data: 2023-10-18T20:16:59.212 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.212 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.212 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.213 DEBUG:boto:Params: {} 2023-10-18T20:16:59.213 DEBUG:boto:Token: None 2023-10-18T20:16:59.214 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.214 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jc5k2qUnRpvNm/aj2xWnw9fr79Y= 2023-10-18T20:16:59.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jc5k2qUnRpvNm/aj2xWnw9fr79Y=', 'Content-Length': '0'} 2023-10-18T20:16:59.217 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9dc5afe5c464b15-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.217 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.218 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.218 DEBUG:boto:path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.219 DEBUG:boto:auth_path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.219 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.219 DEBUG:boto:Path: /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.220 DEBUG:boto:Data: 2023-10-18T20:16:59.220 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.220 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.221 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.221 DEBUG:boto:Params: {} 2023-10-18T20:16:59.221 DEBUG:boto:Token: None 2023-10-18T20:16:59.222 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.222 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hhRJbuKFbD3Pj8DyXBVF1A0OBjI= 2023-10-18T20:16:59.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hhRJbuKFbD3Pj8DyXBVF1A0OBjI=', 'Content-Length': '0'} 2023-10-18T20:16:59.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000310ec79a3e6a77c7-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.225 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.225 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.226 DEBUG:boto:path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.226 DEBUG:boto:auth_path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.227 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.227 DEBUG:boto:Path: /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.228 DEBUG:boto:Data: 2023-10-18T20:16:59.228 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.228 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.229 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.229 DEBUG:boto:Params: {} 2023-10-18T20:16:59.229 DEBUG:boto:Token: None 2023-10-18T20:16:59.230 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.230 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hhRJbuKFbD3Pj8DyXBVF1A0OBjI= 2023-10-18T20:16:59.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hhRJbuKFbD3Pj8DyXBVF1A0OBjI=', 'Content-Length': '0'} 2023-10-18T20:16:59.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bed2b8b97d73230b-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:16:59.234 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.234 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.235 DEBUG:boto:path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.235 DEBUG:boto:auth_path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.235 DEBUG:boto:Method: GET 2023-10-18T20:16:59.236 DEBUG:boto:Path: /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.236 DEBUG:boto:Data: 2023-10-18T20:16:59.237 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.237 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.237 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.238 DEBUG:boto:Params: {} 2023-10-18T20:16:59.238 DEBUG:boto:Token: None 2023-10-18T20:16:59.239 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.239 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jc5k2qUnRpvNm/aj2xWnw9fr79Y= 2023-10-18T20:16:59.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jc5k2qUnRpvNm/aj2xWnw9fr79Y=', 'Content-Length': '0'} 2023-10-18T20:16:59.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e45e02e21eb0a0e-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.242 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.243 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.243 DEBUG:boto:path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.244 DEBUG:boto:auth_path=/bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.244 DEBUG:boto:Method: GET 2023-10-18T20:16:59.244 DEBUG:boto:Path: /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.245 DEBUG:boto:Data: 2023-10-18T20:16:59.245 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.245 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.246 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.246 DEBUG:boto:Params: {} 2023-10-18T20:16:59.247 DEBUG:boto:Token: None 2023-10-18T20:16:59.247 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6?versionId=null 2023-10-18T20:16:59.248 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jc5k2qUnRpvNm/aj2xWnw9fr79Y= 2023-10-18T20:16:59.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jc5k2qUnRpvNm/aj2xWnw9fr79Y=', 'Content-Length': '0'} 2023-10-18T20:16:59.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008155b592beb2c40d-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.251 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.251 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.252 DEBUG:boto:path=/bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.252 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.252 DEBUG:boto:Method: GET 2023-10-18T20:16:59.253 DEBUG:boto:Path: /bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.253 DEBUG:boto:Data: 2023-10-18T20:16:59.254 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.254 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.254 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.255 DEBUG:boto:Params: {} 2023-10-18T20:16:59.255 DEBUG:boto:Token: None 2023-10-18T20:16:59.256 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.256 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea3ae8eeda3d647b-0065303d3b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.259 DEBUG:boto:b'bofrnf-47obj61000falseobj62023-10-18T20:16:20.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.260 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.260 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.260 DEBUG:boto:path=/bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.261 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.261 DEBUG:boto:Method: GET 2023-10-18T20:16:59.262 DEBUG:boto:Path: /bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.262 DEBUG:boto:Data: 2023-10-18T20:16:59.263 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.263 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.263 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.264 DEBUG:boto:Params: {} 2023-10-18T20:16:59.264 DEBUG:boto:Token: None 2023-10-18T20:16:59.265 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.265 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e85c4213e07f1642-0065303d3b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.314 DEBUG:boto:b'bofrnf-47obj61000falseobj62023-10-18T20:16:20.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.315 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.316 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.316 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.316 DEBUG:boto:Path: /bofrnf-47/obj6 2023-10-18T20:16:59.317 DEBUG:boto:Data: 2023-10-18T20:16:59.317 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.318 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.318 DEBUG:boto:Params: {} 2023-10-18T20:16:59.319 DEBUG:boto:Token: None 2023-10-18T20:16:59.319 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6 2023-10-18T20:16:59.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b/Xmrmhd9QUcvX0f38qqqv8lrwM= 2023-10-18T20:16:59.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b/Xmrmhd9QUcvX0f38qqqv8lrwM=', 'Content-Length': '0'} 2023-10-18T20:16:59.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002842ac184156b387-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.323 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.323 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.324 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.324 DEBUG:boto:Path: /bofrnf-47/obj6 2023-10-18T20:16:59.324 DEBUG:boto:Data: 2023-10-18T20:16:59.325 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.325 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.326 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.326 DEBUG:boto:Params: {} 2023-10-18T20:16:59.327 DEBUG:boto:Token: None 2023-10-18T20:16:59.327 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6 2023-10-18T20:16:59.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b/Xmrmhd9QUcvX0f38qqqv8lrwM= 2023-10-18T20:16:59.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b/Xmrmhd9QUcvX0f38qqqv8lrwM=', 'Content-Length': '0'} 2023-10-18T20:16:59.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050e21ccf961bdc0a-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:16:59.331 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.331 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.332 DEBUG:boto:Method: GET 2023-10-18T20:16:59.332 DEBUG:boto:Path: /bofrnf-47/obj6 2023-10-18T20:16:59.332 DEBUG:boto:Data: 2023-10-18T20:16:59.333 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.334 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.334 DEBUG:boto:Params: {} 2023-10-18T20:16:59.334 DEBUG:boto:Token: None 2023-10-18T20:16:59.335 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6 2023-10-18T20:16:59.335 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2iqTqHzNzOEnN6yKVThYtzsZyik= 2023-10-18T20:16:59.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2iqTqHzNzOEnN6yKVThYtzsZyik=', 'Content-Length': '0'} 2023-10-18T20:16:59.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065e0e1ea069dbc62-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.338 DEBUG:boto:path=/bofrnf-47/obj6 2023-10-18T20:16:59.339 DEBUG:boto:auth_path=/bofrnf-47/obj6 2023-10-18T20:16:59.339 DEBUG:boto:Method: GET 2023-10-18T20:16:59.340 DEBUG:boto:Path: /bofrnf-47/obj6 2023-10-18T20:16:59.340 DEBUG:boto:Data: 2023-10-18T20:16:59.340 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.341 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.341 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.342 DEBUG:boto:Params: {} 2023-10-18T20:16:59.342 DEBUG:boto:Token: None 2023-10-18T20:16:59.343 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj6 2023-10-18T20:16:59.343 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2iqTqHzNzOEnN6yKVThYtzsZyik= 2023-10-18T20:16:59.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2iqTqHzNzOEnN6yKVThYtzsZyik=', 'Content-Length': '0'} 2023-10-18T20:16:59.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000161057f1bdba9a3c-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.346 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.347 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.347 DEBUG:boto:path=/bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.347 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.348 DEBUG:boto:Method: GET 2023-10-18T20:16:59.348 DEBUG:boto:Path: /bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.349 DEBUG:boto:Data: 2023-10-18T20:16:59.349 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.350 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.350 DEBUG:boto:Params: {} 2023-10-18T20:16:59.351 DEBUG:boto:Token: None 2023-10-18T20:16:59.351 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a971ca963595202-0065303d3b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.356 DEBUG:boto:b'bofrnf-47obj61000falseobj62023-10-18T20:16:20.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.357 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.357 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.358 DEBUG:boto:path=/bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.358 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.358 DEBUG:boto:Method: GET 2023-10-18T20:16:59.359 DEBUG:boto:Path: /bofrnf-47/?prefix=obj6 2023-10-18T20:16:59.359 DEBUG:boto:Data: 2023-10-18T20:16:59.360 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.360 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.361 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.361 DEBUG:boto:Params: {} 2023-10-18T20:16:59.361 DEBUG:boto:Token: None 2023-10-18T20:16:59.362 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.362 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005597e49277e0683d-0065303d3b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.366 DEBUG:boto:b'bofrnf-47obj61000falseobj62023-10-18T20:16:20.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:16:59.367 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.368 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.368 DEBUG:boto:path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.368 DEBUG:boto:auth_path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.369 DEBUG:boto:Method: GET 2023-10-18T20:16:59.369 DEBUG:boto:Path: /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.370 DEBUG:boto:Data: 2023-10-18T20:16:59.370 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.371 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.371 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.372 DEBUG:boto:Params: {} 2023-10-18T20:16:59.372 DEBUG:boto:Token: None 2023-10-18T20:16:59.372 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4HlrZexIXUZXHHXyaN4cPWGqKvQ= 2023-10-18T20:16:59.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4HlrZexIXUZXHHXyaN4cPWGqKvQ=', 'Content-Length': '0'} 2023-10-18T20:16:59.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036e37f1747cc7ee4-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.422 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.423 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.423 DEBUG:boto:path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.424 DEBUG:boto:auth_path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.424 DEBUG:boto:Method: GET 2023-10-18T20:16:59.425 DEBUG:boto:Path: /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.425 DEBUG:boto:Data: 2023-10-18T20:16:59.426 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.427 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.427 DEBUG:boto:Params: {} 2023-10-18T20:16:59.427 DEBUG:boto:Token: None 2023-10-18T20:16:59.428 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4HlrZexIXUZXHHXyaN4cPWGqKvQ= 2023-10-18T20:16:59.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4HlrZexIXUZXHHXyaN4cPWGqKvQ=', 'Content-Length': '0'} 2023-10-18T20:16:59.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006164de6244f61a78-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.432 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.432 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.433 DEBUG:boto:path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.433 DEBUG:boto:auth_path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.434 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.434 DEBUG:boto:Path: /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.435 DEBUG:boto:Data: 2023-10-18T20:16:59.435 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.435 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.436 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.436 DEBUG:boto:Params: {} 2023-10-18T20:16:59.437 DEBUG:boto:Token: None 2023-10-18T20:16:59.437 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.438 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:++iD+AqSlB5b0YDk2YkRfVWCiA0= 2023-10-18T20:16:59.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:++iD+AqSlB5b0YDk2YkRfVWCiA0=', 'Content-Length': '0'} 2023-10-18T20:16:59.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013b696ac21e1e796-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.440 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.441 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.441 DEBUG:boto:path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.442 DEBUG:boto:auth_path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.442 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.443 DEBUG:boto:Path: /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.443 DEBUG:boto:Data: 2023-10-18T20:16:59.443 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.444 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.444 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.445 DEBUG:boto:Params: {} 2023-10-18T20:16:59.445 DEBUG:boto:Token: None 2023-10-18T20:16:59.445 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.446 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:++iD+AqSlB5b0YDk2YkRfVWCiA0= 2023-10-18T20:16:59.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:++iD+AqSlB5b0YDk2YkRfVWCiA0=', 'Content-Length': '0'} 2023-10-18T20:16:59.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aecd527f9893d7d8-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:16:59.449 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.450 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.450 DEBUG:boto:path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.451 DEBUG:boto:auth_path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.451 DEBUG:boto:Method: GET 2023-10-18T20:16:59.452 DEBUG:boto:Path: /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.452 DEBUG:boto:Data: 2023-10-18T20:16:59.452 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.453 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.454 DEBUG:boto:Params: {} 2023-10-18T20:16:59.454 DEBUG:boto:Token: None 2023-10-18T20:16:59.455 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4HlrZexIXUZXHHXyaN4cPWGqKvQ= 2023-10-18T20:16:59.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4HlrZexIXUZXHHXyaN4cPWGqKvQ=', 'Content-Length': '0'} 2023-10-18T20:16:59.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f465b12b2f1245e6-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.458 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.459 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.459 DEBUG:boto:path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.459 DEBUG:boto:auth_path=/bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.460 DEBUG:boto:Method: GET 2023-10-18T20:16:59.460 DEBUG:boto:Path: /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.461 DEBUG:boto:Data: 2023-10-18T20:16:59.461 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.462 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.462 DEBUG:boto:Params: {} 2023-10-18T20:16:59.463 DEBUG:boto:Token: None 2023-10-18T20:16:59.463 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7?versionId=null 2023-10-18T20:16:59.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4HlrZexIXUZXHHXyaN4cPWGqKvQ= 2023-10-18T20:16:59.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4HlrZexIXUZXHHXyaN4cPWGqKvQ=', 'Content-Length': '0'} 2023-10-18T20:16:59.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040fbc1131c93d1be-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.467 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.468 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.468 DEBUG:boto:path=/bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.468 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.469 DEBUG:boto:Method: GET 2023-10-18T20:16:59.469 DEBUG:boto:Path: /bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.470 DEBUG:boto:Data: 2023-10-18T20:16:59.470 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.471 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.471 DEBUG:boto:Params: {} 2023-10-18T20:16:59.472 DEBUG:boto:Token: None 2023-10-18T20:16:59.472 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.473 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc24b19fc983587d-0065303d3b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.476 DEBUG:boto:b'bofrnf-47obj71000falseobj72023-10-18T20:16:20.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.477 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.478 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.478 DEBUG:boto:path=/bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.478 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.479 DEBUG:boto:Method: GET 2023-10-18T20:16:59.479 DEBUG:boto:Path: /bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.480 DEBUG:boto:Data: 2023-10-18T20:16:59.480 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.481 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.481 DEBUG:boto:Params: {} 2023-10-18T20:16:59.482 DEBUG:boto:Token: None 2023-10-18T20:16:59.482 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.483 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b0bf611b0611711-0065303d3b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.526 DEBUG:boto:b'bofrnf-47obj71000falseobj72023-10-18T20:16:20.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.527 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.528 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.528 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.529 DEBUG:boto:Path: /bofrnf-47/obj7 2023-10-18T20:16:59.529 DEBUG:boto:Data: 2023-10-18T20:16:59.529 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.530 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.531 DEBUG:boto:Params: {} 2023-10-18T20:16:59.531 DEBUG:boto:Token: None 2023-10-18T20:16:59.532 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7 2023-10-18T20:16:59.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j8yP96VfGfK4cdUpkyTqJldjCJk= 2023-10-18T20:16:59.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j8yP96VfGfK4cdUpkyTqJldjCJk=', 'Content-Length': '0'} 2023-10-18T20:16:59.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007990823b166b9afd-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.535 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.535 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.536 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.536 DEBUG:boto:Path: /bofrnf-47/obj7 2023-10-18T20:16:59.536 DEBUG:boto:Data: 2023-10-18T20:16:59.537 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.538 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.538 DEBUG:boto:Params: {} 2023-10-18T20:16:59.539 DEBUG:boto:Token: None 2023-10-18T20:16:59.539 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7 2023-10-18T20:16:59.539 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j8yP96VfGfK4cdUpkyTqJldjCJk= 2023-10-18T20:16:59.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j8yP96VfGfK4cdUpkyTqJldjCJk=', 'Content-Length': '0'} 2023-10-18T20:16:59.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8a58d0cf70b26ef-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:16:59.542 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.543 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.543 DEBUG:boto:Method: GET 2023-10-18T20:16:59.544 DEBUG:boto:Path: /bofrnf-47/obj7 2023-10-18T20:16:59.544 DEBUG:boto:Data: 2023-10-18T20:16:59.544 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.545 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.545 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.546 DEBUG:boto:Params: {} 2023-10-18T20:16:59.546 DEBUG:boto:Token: None 2023-10-18T20:16:59.546 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7 2023-10-18T20:16:59.547 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A9Up97mn6CRji38lW9fF2b4hLLc= 2023-10-18T20:16:59.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A9Up97mn6CRji38lW9fF2b4hLLc=', 'Content-Length': '0'} 2023-10-18T20:16:59.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a699afbecd5e00a4-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.550 DEBUG:boto:path=/bofrnf-47/obj7 2023-10-18T20:16:59.550 DEBUG:boto:auth_path=/bofrnf-47/obj7 2023-10-18T20:16:59.551 DEBUG:boto:Method: GET 2023-10-18T20:16:59.551 DEBUG:boto:Path: /bofrnf-47/obj7 2023-10-18T20:16:59.551 DEBUG:boto:Data: 2023-10-18T20:16:59.552 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.552 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.553 DEBUG:boto:Params: {} 2023-10-18T20:16:59.553 DEBUG:boto:Token: None 2023-10-18T20:16:59.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj7 2023-10-18T20:16:59.554 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A9Up97mn6CRji38lW9fF2b4hLLc= 2023-10-18T20:16:59.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A9Up97mn6CRji38lW9fF2b4hLLc=', 'Content-Length': '0'} 2023-10-18T20:16:59.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000218aef7c4ab6538b-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.557 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.558 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.558 DEBUG:boto:path=/bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.559 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.559 DEBUG:boto:Method: GET 2023-10-18T20:16:59.560 DEBUG:boto:Path: /bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.560 DEBUG:boto:Data: 2023-10-18T20:16:59.560 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.561 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.561 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.561 DEBUG:boto:Params: {} 2023-10-18T20:16:59.562 DEBUG:boto:Token: None 2023-10-18T20:16:59.562 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.563 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005d10742a5236856-0065303d3b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.566 DEBUG:boto:b'bofrnf-47obj71000falseobj72023-10-18T20:16:20.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.566 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.567 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.567 DEBUG:boto:path=/bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.568 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.568 DEBUG:boto:Method: GET 2023-10-18T20:16:59.568 DEBUG:boto:Path: /bofrnf-47/?prefix=obj7 2023-10-18T20:16:59.569 DEBUG:boto:Data: 2023-10-18T20:16:59.569 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.569 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.570 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.570 DEBUG:boto:Params: {} 2023-10-18T20:16:59.571 DEBUG:boto:Token: None 2023-10-18T20:16:59.571 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.572 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f496373191c4239-0065303d3b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.575 DEBUG:boto:b'bofrnf-47obj71000falseobj72023-10-18T20:16:20.850Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:16:59.576 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.576 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.577 DEBUG:boto:path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.577 DEBUG:boto:auth_path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.577 DEBUG:boto:Method: GET 2023-10-18T20:16:59.578 DEBUG:boto:Path: /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.578 DEBUG:boto:Data: 2023-10-18T20:16:59.579 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.579 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.579 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.580 DEBUG:boto:Params: {} 2023-10-18T20:16:59.580 DEBUG:boto:Token: None 2023-10-18T20:16:59.581 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzFt5CfeuKeMypSKwicRt2duAd4= 2023-10-18T20:16:59.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzFt5CfeuKeMypSKwicRt2duAd4=', 'Content-Length': '0'} 2023-10-18T20:16:59.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf1131b4a76b208f-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.584 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.584 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.585 DEBUG:boto:path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.585 DEBUG:boto:auth_path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.586 DEBUG:boto:Method: GET 2023-10-18T20:16:59.586 DEBUG:boto:Path: /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.586 DEBUG:boto:Data: 2023-10-18T20:16:59.587 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.587 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.588 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.588 DEBUG:boto:Params: {} 2023-10-18T20:16:59.588 DEBUG:boto:Token: None 2023-10-18T20:16:59.589 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.589 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzFt5CfeuKeMypSKwicRt2duAd4= 2023-10-18T20:16:59.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzFt5CfeuKeMypSKwicRt2duAd4=', 'Content-Length': '0'} 2023-10-18T20:16:59.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095d12c17d8096c27-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.592 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.593 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.593 DEBUG:boto:path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.593 DEBUG:boto:auth_path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.594 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.594 DEBUG:boto:Path: /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.595 DEBUG:boto:Data: 2023-10-18T20:16:59.595 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.595 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.596 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.596 DEBUG:boto:Params: {} 2023-10-18T20:16:59.597 DEBUG:boto:Token: None 2023-10-18T20:16:59.597 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.598 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2KVoFp850q/97NdsXIuQV6vBRmA= 2023-10-18T20:16:59.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2KVoFp850q/97NdsXIuQV6vBRmA=', 'Content-Length': '0'} 2023-10-18T20:16:59.600 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fceadb5d4a281717-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.600 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.601 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.601 DEBUG:boto:path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.602 DEBUG:boto:auth_path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.602 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.603 DEBUG:boto:Path: /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.603 DEBUG:boto:Data: 2023-10-18T20:16:59.603 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.604 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.604 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.605 DEBUG:boto:Params: {} 2023-10-18T20:16:59.605 DEBUG:boto:Token: None 2023-10-18T20:16:59.605 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.606 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2KVoFp850q/97NdsXIuQV6vBRmA= 2023-10-18T20:16:59.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2KVoFp850q/97NdsXIuQV6vBRmA=', 'Content-Length': '0'} 2023-10-18T20:16:59.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001488208f98032eb0-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:16:59.609 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.609 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.610 DEBUG:boto:path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.610 DEBUG:boto:auth_path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.611 DEBUG:boto:Method: GET 2023-10-18T20:16:59.611 DEBUG:boto:Path: /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.612 DEBUG:boto:Data: 2023-10-18T20:16:59.612 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.612 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.613 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.613 DEBUG:boto:Params: {} 2023-10-18T20:16:59.614 DEBUG:boto:Token: None 2023-10-18T20:16:59.614 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzFt5CfeuKeMypSKwicRt2duAd4= 2023-10-18T20:16:59.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzFt5CfeuKeMypSKwicRt2duAd4=', 'Content-Length': '0'} 2023-10-18T20:16:59.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003379874d915e7cff-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.618 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.618 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.618 DEBUG:boto:path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.619 DEBUG:boto:auth_path=/bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.619 DEBUG:boto:Method: GET 2023-10-18T20:16:59.620 DEBUG:boto:Path: /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.620 DEBUG:boto:Data: 2023-10-18T20:16:59.620 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.621 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.621 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.621 DEBUG:boto:Params: {} 2023-10-18T20:16:59.622 DEBUG:boto:Token: None 2023-10-18T20:16:59.622 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8?versionId=null 2023-10-18T20:16:59.623 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzFt5CfeuKeMypSKwicRt2duAd4= 2023-10-18T20:16:59.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzFt5CfeuKeMypSKwicRt2duAd4=', 'Content-Length': '0'} 2023-10-18T20:16:59.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5f2550f79963c63-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.626 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.626 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.627 DEBUG:boto:path=/bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.627 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.627 DEBUG:boto:Method: GET 2023-10-18T20:16:59.628 DEBUG:boto:Path: /bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.628 DEBUG:boto:Data: 2023-10-18T20:16:59.628 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.629 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.629 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.629 DEBUG:boto:Params: {} 2023-10-18T20:16:59.630 DEBUG:boto:Token: None 2023-10-18T20:16:59.630 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.631 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe2d80f3c227f963-0065303d3b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.634 DEBUG:boto:b'bofrnf-47obj81000falseobj82023-10-18T20:16:20.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.635 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.635 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.636 DEBUG:boto:path=/bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.636 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.636 DEBUG:boto:Method: GET 2023-10-18T20:16:59.637 DEBUG:boto:Path: /bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.637 DEBUG:boto:Data: 2023-10-18T20:16:59.637 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.638 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.638 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.639 DEBUG:boto:Params: {} 2023-10-18T20:16:59.639 DEBUG:boto:Token: None 2023-10-18T20:16:59.639 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.640 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c144e38a1e63657-0065303d3b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.643 DEBUG:boto:b'bofrnf-47obj81000falseobj82023-10-18T20:16:20.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.644 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.644 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.645 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.645 DEBUG:boto:Path: /bofrnf-47/obj8 2023-10-18T20:16:59.645 DEBUG:boto:Data: 2023-10-18T20:16:59.646 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.646 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.646 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.647 DEBUG:boto:Params: {} 2023-10-18T20:16:59.647 DEBUG:boto:Token: None 2023-10-18T20:16:59.648 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8 2023-10-18T20:16:59.648 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OlVrUxNwZGwqTuPuIfuUoaYBnpo= 2023-10-18T20:16:59.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlVrUxNwZGwqTuPuIfuUoaYBnpo=', 'Content-Length': '0'} 2023-10-18T20:16:59.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004562f397b1423e0f-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.651 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.651 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.652 DEBUG:boto:Method: HEAD 2023-10-18T20:16:59.652 DEBUG:boto:Path: /bofrnf-47/obj8 2023-10-18T20:16:59.652 DEBUG:boto:Data: 2023-10-18T20:16:59.653 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.653 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.654 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.654 DEBUG:boto:Params: {} 2023-10-18T20:16:59.654 DEBUG:boto:Token: None 2023-10-18T20:16:59.655 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8 2023-10-18T20:16:59.655 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OlVrUxNwZGwqTuPuIfuUoaYBnpo= 2023-10-18T20:16:59.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlVrUxNwZGwqTuPuIfuUoaYBnpo=', 'Content-Length': '0'} 2023-10-18T20:16:59.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea95363229e10335-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:16:59.658 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.658 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.661 DEBUG:boto:Method: GET 2023-10-18T20:16:59.661 DEBUG:boto:Path: /bofrnf-47/obj8 2023-10-18T20:16:59.662 DEBUG:boto:Data: 2023-10-18T20:16:59.662 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.663 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.663 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.663 DEBUG:boto:Params: {} 2023-10-18T20:16:59.664 DEBUG:boto:Token: None 2023-10-18T20:16:59.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8 2023-10-18T20:16:59.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTNlrj5AubKPQZHg9aF2fQhLKeU= 2023-10-18T20:16:59.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTNlrj5AubKPQZHg9aF2fQhLKeU=', 'Content-Length': '0'} 2023-10-18T20:16:59.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5a7394e34acd5c6-0065303d3b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.714 DEBUG:boto:path=/bofrnf-47/obj8 2023-10-18T20:16:59.715 DEBUG:boto:auth_path=/bofrnf-47/obj8 2023-10-18T20:16:59.715 DEBUG:boto:Method: GET 2023-10-18T20:16:59.716 DEBUG:boto:Path: /bofrnf-47/obj8 2023-10-18T20:16:59.716 DEBUG:boto:Data: 2023-10-18T20:16:59.717 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.717 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.717 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.718 DEBUG:boto:Params: {} 2023-10-18T20:16:59.718 DEBUG:boto:Token: None 2023-10-18T20:16:59.719 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/obj8 2023-10-18T20:16:59.719 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTNlrj5AubKPQZHg9aF2fQhLKeU= 2023-10-18T20:16:59.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTNlrj5AubKPQZHg9aF2fQhLKeU=', 'Content-Length': '0'} 2023-10-18T20:16:59.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:16:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-47:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e628af8f59a5a43-0065303d3b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.722 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.723 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.723 DEBUG:boto:path=/bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.723 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.724 DEBUG:boto:Method: GET 2023-10-18T20:16:59.724 DEBUG:boto:Path: /bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.725 DEBUG:boto:Data: 2023-10-18T20:16:59.725 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:16:59.726 DEBUG:boto:Port: 8001 2023-10-18T20:16:59.726 DEBUG:boto:Params: {} 2023-10-18T20:16:59.727 DEBUG:boto:Token: None 2023-10-18T20:16:59.727 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000161f2df8d98a0d51-0065303d3b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.731 DEBUG:boto:b'bofrnf-47obj81000falseobj82023-10-18T20:16:20.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.731 DEBUG:boto:path=/bofrnf-47/ 2023-10-18T20:16:59.732 DEBUG:boto:auth_path=/bofrnf-47/ 2023-10-18T20:16:59.732 DEBUG:boto:path=/bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.733 DEBUG:boto:auth_path=/bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.733 DEBUG:boto:Method: GET 2023-10-18T20:16:59.733 DEBUG:boto:Path: /bofrnf-47/?prefix=obj8 2023-10-18T20:16:59.734 DEBUG:boto:Data: 2023-10-18T20:16:59.734 DEBUG:boto:Headers: {} 2023-10-18T20:16:59.735 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:16:59.735 DEBUG:boto:Port: 8000 2023-10-18T20:16:59.735 DEBUG:boto:Params: {} 2023-10-18T20:16:59.736 DEBUG:boto:Token: None 2023-10-18T20:16:59.736 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:16:59 GMT /bofrnf-47/ 2023-10-18T20:16:59.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY= 2023-10-18T20:16:59.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iVTB7qyEABrVo/bvFx/jW34KxhY=', 'Content-Length': '0'} 2023-10-18T20:16:59.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc9c84705d074ccc-0065303d3b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:16:59.740 DEBUG:boto:b'bofrnf-47obj81000falseobj82023-10-18T20:16:20.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:16:59.741 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-47 zones={a2, a1} 2023-10-18T20:16:59.741 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-18T20:16:59.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:16:59.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:16:59.743 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:16:59.793 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:59.793 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:59.839 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:16:59.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:16:59.839 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005162", 2023-10-18T20:16:59.839 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:16:20.630232Z" 2023-10-18T20:16:59.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:16:59.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:16:59.840 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-10-18T20:16:59.840 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:16:35.771789Z" 2023-10-18T20:16:59.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:16:59.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:16:59.841 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-10-18T20:16:59.841 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:16:35.898756Z" 2023-10-18T20:16:59.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:16:59.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:16:59.842 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000007131", 2023-10-18T20:16:59.842 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:16:35.820660Z" 2023-10-18T20:16:59.842 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:16:59.842 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:16:59.843 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000005162', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000002905', 3: '00000000000000000000:00000000000000007131'} 2023-10-18T20:16:59.844 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:16:59.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:16:59.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:16:59.845 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:16:59.896 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:16:59.896 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:16:59.937 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:16:59.938 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:16:59.938 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:16:59.938 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:16:59.938 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:16:59.939 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 2477610155421060198 2023-10-18T20:16:59.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:16:59.939 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:16:59.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:16:59.939 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:16:59.939 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:16:59.940 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:16:59.940 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005162", 2023-10-18T20:16:59.940 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:16:59.940 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:16:59.940 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:16:59.941 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:16:20.630232Z" 2023-10-18T20:16:59.941 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:16:59.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:16:59.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:16:59.941 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:16:59.942 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:16:59.942 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:16:59.942 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-10-18T20:16:59.942 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:16:59.942 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:16:59.943 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:16:59.943 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:16:35.771789Z" 2023-10-18T20:16:59.943 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:16:59.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:16:59.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:16:59.944 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:16:59.944 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:16:59.944 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:16:59.944 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-10-18T20:16:59.944 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:16:59.945 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:16:59.945 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:16:59.945 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:16:35.898756Z" 2023-10-18T20:16:59.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:16:59.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:16:59.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:16:59.946 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:16:59.946 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:16:59.946 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:16:59.946 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000007131", 2023-10-18T20:16:59.947 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:16:59.947 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:16:59.947 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:16:59.947 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:16:35.820660Z" 2023-10-18T20:16:59.947 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:16:59.948 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:16:59.948 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:16:59.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:16:59.948 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:16:59.948 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:16:59.949 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:16:59.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:16:59.949 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:16:59Z" 2023-10-18T20:16:59.949 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:16:59.950 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2477610155421060198 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005162", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:16:20.630232Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:16:35.771789Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:16:35.898756Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007131", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:16:35.820660Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:16:59Z" } 2023-10-18T20:16:59.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005162', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:16:20.630232Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:16:35.771789Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:16:35.898756Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007131', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:16:35.820660Z'}}] 2023-10-18T20:16:59.951 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005162', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000002905', 3: '00000000000000000000:00000000000000007131'} 2023-10-18T20:16:59.952 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005162', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000002905', 3: '00000000000000000000:00000000000000007131'} 2023-10-18T20:16:59.952 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:16:59.952 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-18T20:16:59.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:16:59.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:16:59.953 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:17:00.002 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:00.002 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:00.048 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:17:00.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.048 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002626", 2023-10-18T20:17:00.049 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:14:17.548678Z" 2023-10-18T20:17:00.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.049 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002623", 2023-10-18T20:17:00.049 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:16:35.862932Z" 2023-10-18T20:17:00.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.050 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-18T20:17:00.050 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:16:35.934751Z" 2023-10-18T20:17:00.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.051 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000006285", 2023-10-18T20:17:00.051 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:16:35.862420Z" 2023-10-18T20:17:00.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.052 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:17:00.053 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002626', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000006285'} 2023-10-18T20:17:00.053 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:17:00.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:00.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:00.055 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:17:00.105 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:00.106 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:00.149 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:17:00.149 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:17:00.150 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:17:00.150 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:17:00.150 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:17:00.150 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:17:00.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.151 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:17:00.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.151 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:17:00.151 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:00.151 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:17:00.152 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002626", 2023-10-18T20:17:00.152 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:00.152 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:00.152 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:00.152 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:14:17.548678Z" 2023-10-18T20:17:00.153 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.153 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:17:00.153 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:00.154 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:17:00.154 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000002623", 2023-10-18T20:17:00.154 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:00.154 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:00.155 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:00.155 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:16:35.862932Z" 2023-10-18T20:17:00.155 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.156 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:17:00.156 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:00.156 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:17:00.156 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-18T20:17:00.156 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:00.157 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:00.157 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:00.157 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:16:35.934751Z" 2023-10-18T20:17:00.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.158 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:17:00.158 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:00.158 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:17:00.158 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000006285", 2023-10-18T20:17:00.159 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:00.159 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:00.159 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:00.159 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:16:35.862420Z" 2023-10-18T20:17:00.159 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.159 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.160 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:17:00.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.160 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:17:00.160 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:17:00.160 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:17:00.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.161 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:17:00Z" 2023-10-18T20:17:00.161 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:17:00.162 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002626", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:14:17.548678Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002623", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:16:35.862932Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:16:35.934751Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006285", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:16:35.862420Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:17:00Z" } 2023-10-18T20:17:00.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002626', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:14:17.548678Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002623', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:16:35.862932Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:16:35.934751Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006285', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:16:35.862420Z'}}] 2023-10-18T20:17:00.163 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002626', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000006285'} 2023-10-18T20:17:00.163 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002626', 1: '00000000000000000000:00000000000000002623', 2: '00000000000000000000:00000000000000001777', 3: '00000000000000000000:00000000000000006285'} 2023-10-18T20:17:00.164 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:17:00.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-10-18T20:17:00.165 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-48 2023-10-18T20:17:00.166 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:00.166 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:00.167 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.167 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:00.167 DEBUG:boto:Data: 2023-10-18T20:17:00.168 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.168 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:00.169 DEBUG:boto:Port: 8000 2023-10-18T20:17:00.169 DEBUG:boto:Params: {} 2023-10-18T20:17:00.169 DEBUG:boto:Token: None 2023-10-18T20:17:00.170 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-48/ 2023-10-18T20:17:00.170 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XROvwia6ggA0VqOl1QenBm4tx5c= 2023-10-18T20:17:00.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XROvwia6ggA0VqOl1QenBm4tx5c=', 'Content-Length': '0'} 2023-10-18T20:17:00.184 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005976aa5415779aa4-0065303d3c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.185 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-49 2023-10-18T20:17:00.185 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:00.185 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:00.186 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.186 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:00.187 DEBUG:boto:Data: 2023-10-18T20:17:00.187 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.187 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:00.188 DEBUG:boto:Port: 8001 2023-10-18T20:17:00.188 DEBUG:boto:Params: {} 2023-10-18T20:17:00.189 DEBUG:boto:Token: None 2023-10-18T20:17:00.189 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-49/ 2023-10-18T20:17:00.190 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PpJZKjJ/N699rhCfn7PkOK+QiH4= 2023-10-18T20:17:00.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PpJZKjJ/N699rhCfn7PkOK+QiH4=', 'Content-Length': '0'} 2023-10-18T20:17:00.213 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003785460a3f5e5ac8-0065303d3c-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.213 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:00.214 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:00.214 DEBUG:boto:Method: HEAD 2023-10-18T20:17:00.215 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:00.215 DEBUG:boto:Data: 2023-10-18T20:17:00.215 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.216 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:00.216 DEBUG:boto:Port: 8000 2023-10-18T20:17:00.217 DEBUG:boto:Params: {} 2023-10-18T20:17:00.217 DEBUG:boto:Token: None 2023-10-18T20:17:00.217 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-48/ 2023-10-18T20:17:00.218 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ND5oJ/MTtwT0AmwRZ0n6Urq21NI= 2023-10-18T20:17:00.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ND5oJ/MTtwT0AmwRZ0n6Urq21NI=', 'Content-Length': '0'} 2023-10-18T20:17:00.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000130cb7b183ad42ff-0065303d3c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.221 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:00.222 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:00.222 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.223 DEBUG:boto:Path: /bofrnf-48/obj1 2023-10-18T20:17:00.223 DEBUG:boto:Data: 2023-10-18T20:17:00.224 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:00.224 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:00.224 DEBUG:boto:Port: 8000 2023-10-18T20:17:00.225 DEBUG:boto:Params: {} 2023-10-18T20:17:00.225 DEBUG:boto:Token: None 2023-10-18T20:17:00.226 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-48/obj1 2023-10-18T20:17:00.226 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m6KRBg39WzzlQ4vrWvCUceIc9Iw= 2023-10-18T20:17:00.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m6KRBg39WzzlQ4vrWvCUceIc9Iw='} 2023-10-18T20:17:00.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006071fa45b40593c2-0065303d3c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.233 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:00.233 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:00.233 DEBUG:boto:Method: HEAD 2023-10-18T20:17:00.234 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:00.234 DEBUG:boto:Data: 2023-10-18T20:17:00.235 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:00.235 DEBUG:boto:Port: 8000 2023-10-18T20:17:00.236 DEBUG:boto:Params: {} 2023-10-18T20:17:00.236 DEBUG:boto:Token: None 2023-10-18T20:17:00.236 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-48/ 2023-10-18T20:17:00.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ND5oJ/MTtwT0AmwRZ0n6Urq21NI= 2023-10-18T20:17:00.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ND5oJ/MTtwT0AmwRZ0n6Urq21NI=', 'Content-Length': '0'} 2023-10-18T20:17:00.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d02ce955286971f-0065303d3c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.240 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:00.240 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:00.240 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.241 DEBUG:boto:Path: /bofrnf-48/obj2 2023-10-18T20:17:00.241 DEBUG:boto:Data: 2023-10-18T20:17:00.241 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:00.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:00.242 DEBUG:boto:Port: 8000 2023-10-18T20:17:00.243 DEBUG:boto:Params: {} 2023-10-18T20:17:00.243 DEBUG:boto:Token: None 2023-10-18T20:17:00.243 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-48/obj2 2023-10-18T20:17:00.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kDjQjjvbjguff3TkWzHylBs5nCs= 2023-10-18T20:17:00.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kDjQjjvbjguff3TkWzHylBs5nCs='} 2023-10-18T20:17:00.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b39011582087d0cf-0065303d3c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.291 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:00.292 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:00.292 DEBUG:boto:Method: HEAD 2023-10-18T20:17:00.292 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:00.293 DEBUG:boto:Data: 2023-10-18T20:17:00.293 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:00.294 DEBUG:boto:Port: 8000 2023-10-18T20:17:00.294 DEBUG:boto:Params: {} 2023-10-18T20:17:00.295 DEBUG:boto:Token: None 2023-10-18T20:17:00.295 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-48/ 2023-10-18T20:17:00.296 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ND5oJ/MTtwT0AmwRZ0n6Urq21NI= 2023-10-18T20:17:00.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ND5oJ/MTtwT0AmwRZ0n6Urq21NI=', 'Content-Length': '0'} 2023-10-18T20:17:00.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000933eba892998692d-0065303d3c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.299 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:00.299 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:00.300 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.300 DEBUG:boto:Path: /bofrnf-48/obj3 2023-10-18T20:17:00.301 DEBUG:boto:Data: 2023-10-18T20:17:00.301 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:00.301 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:00.302 DEBUG:boto:Port: 8000 2023-10-18T20:17:00.302 DEBUG:boto:Params: {} 2023-10-18T20:17:00.303 DEBUG:boto:Token: None 2023-10-18T20:17:00.303 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-48/obj3 2023-10-18T20:17:00.303 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+ZMqhZxZd1/ibMhnjQr8Fc6PA0s= 2023-10-18T20:17:00.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+ZMqhZxZd1/ibMhnjQr8Fc6PA0s='} 2023-10-18T20:17:00.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062cacac9546526ad-0065303d3c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.309 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:00.310 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:00.310 DEBUG:boto:Method: HEAD 2023-10-18T20:17:00.310 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:00.311 DEBUG:boto:Data: 2023-10-18T20:17:00.311 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.311 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:00.312 DEBUG:boto:Port: 8000 2023-10-18T20:17:00.312 DEBUG:boto:Params: {} 2023-10-18T20:17:00.312 DEBUG:boto:Token: None 2023-10-18T20:17:00.313 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-48/ 2023-10-18T20:17:00.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ND5oJ/MTtwT0AmwRZ0n6Urq21NI= 2023-10-18T20:17:00.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ND5oJ/MTtwT0AmwRZ0n6Urq21NI=', 'Content-Length': '0'} 2023-10-18T20:17:00.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000460bfa7f60a6b427-0065303d3c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.317 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:00.317 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:00.317 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.318 DEBUG:boto:Path: /bofrnf-48/obj4 2023-10-18T20:17:00.318 DEBUG:boto:Data: 2023-10-18T20:17:00.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:00.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:00.319 DEBUG:boto:Port: 8000 2023-10-18T20:17:00.320 DEBUG:boto:Params: {} 2023-10-18T20:17:00.320 DEBUG:boto:Token: None 2023-10-18T20:17:00.320 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-48/obj4 2023-10-18T20:17:00.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s/lueh8HTU4IzvIqzxthxUUwSk0= 2023-10-18T20:17:00.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s/lueh8HTU4IzvIqzxthxUUwSk0='} 2023-10-18T20:17:00.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000224a6e1dffddd524-0065303d3c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.367 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:00.368 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:00.368 DEBUG:boto:Method: HEAD 2023-10-18T20:17:00.368 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:00.369 DEBUG:boto:Data: 2023-10-18T20:17:00.369 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:00.370 DEBUG:boto:Port: 8001 2023-10-18T20:17:00.370 DEBUG:boto:Params: {} 2023-10-18T20:17:00.371 DEBUG:boto:Token: None 2023-10-18T20:17:00.371 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-49/ 2023-10-18T20:17:00.372 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nUXZpCvSChRScfmnJVpgNNaXVpE= 2023-10-18T20:17:00.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nUXZpCvSChRScfmnJVpgNNaXVpE=', 'Content-Length': '0'} 2023-10-18T20:17:00.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b713136cae772d7c-0065303d3c-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.375 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:00.375 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:00.376 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.376 DEBUG:boto:Path: /bofrnf-49/obj1 2023-10-18T20:17:00.377 DEBUG:boto:Data: 2023-10-18T20:17:00.377 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:00.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:00.378 DEBUG:boto:Port: 8001 2023-10-18T20:17:00.378 DEBUG:boto:Params: {} 2023-10-18T20:17:00.379 DEBUG:boto:Token: None 2023-10-18T20:17:00.379 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-49/obj1 2023-10-18T20:17:00.379 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UfCS1OllgIOibuIzJNU0ybGKhxU= 2023-10-18T20:17:00.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UfCS1OllgIOibuIzJNU0ybGKhxU='} 2023-10-18T20:17:00.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f46015424eaaf986-0065303d3c-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.386 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:00.387 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:00.387 DEBUG:boto:Method: HEAD 2023-10-18T20:17:00.387 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:00.388 DEBUG:boto:Data: 2023-10-18T20:17:00.388 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.388 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:00.389 DEBUG:boto:Port: 8001 2023-10-18T20:17:00.389 DEBUG:boto:Params: {} 2023-10-18T20:17:00.389 DEBUG:boto:Token: None 2023-10-18T20:17:00.390 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-49/ 2023-10-18T20:17:00.390 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nUXZpCvSChRScfmnJVpgNNaXVpE= 2023-10-18T20:17:00.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nUXZpCvSChRScfmnJVpgNNaXVpE=', 'Content-Length': '0'} 2023-10-18T20:17:00.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcc2aca6eea2d034-0065303d3c-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.393 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:00.394 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:00.394 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.394 DEBUG:boto:Path: /bofrnf-49/obj2 2023-10-18T20:17:00.395 DEBUG:boto:Data: 2023-10-18T20:17:00.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:00.396 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:00.396 DEBUG:boto:Port: 8001 2023-10-18T20:17:00.396 DEBUG:boto:Params: {} 2023-10-18T20:17:00.397 DEBUG:boto:Token: None 2023-10-18T20:17:00.397 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-49/obj2 2023-10-18T20:17:00.397 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F4chZBanDRUI5jYrgX9siQPD620= 2023-10-18T20:17:00.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F4chZBanDRUI5jYrgX9siQPD620='} 2023-10-18T20:17:00.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007bd219af4de2027b-0065303d3c-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.443 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:00.443 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:00.444 DEBUG:boto:Method: HEAD 2023-10-18T20:17:00.444 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:00.445 DEBUG:boto:Data: 2023-10-18T20:17:00.445 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:00.446 DEBUG:boto:Port: 8001 2023-10-18T20:17:00.446 DEBUG:boto:Params: {} 2023-10-18T20:17:00.447 DEBUG:boto:Token: None 2023-10-18T20:17:00.447 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-49/ 2023-10-18T20:17:00.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nUXZpCvSChRScfmnJVpgNNaXVpE= 2023-10-18T20:17:00.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nUXZpCvSChRScfmnJVpgNNaXVpE=', 'Content-Length': '0'} 2023-10-18T20:17:00.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca3658db43312035-0065303d3c-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.451 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:00.451 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:00.452 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.452 DEBUG:boto:Path: /bofrnf-49/obj3 2023-10-18T20:17:00.453 DEBUG:boto:Data: 2023-10-18T20:17:00.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:00.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:00.454 DEBUG:boto:Port: 8001 2023-10-18T20:17:00.454 DEBUG:boto:Params: {} 2023-10-18T20:17:00.455 DEBUG:boto:Token: None 2023-10-18T20:17:00.455 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-49/obj3 2023-10-18T20:17:00.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rC4yIkCa7ggdRDDIRT4ZZt60yPM= 2023-10-18T20:17:00.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rC4yIkCa7ggdRDDIRT4ZZt60yPM='} 2023-10-18T20:17:00.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a70282928f8d6e58-0065303d3c-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.462 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:00.463 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:00.463 DEBUG:boto:Method: HEAD 2023-10-18T20:17:00.463 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:00.464 DEBUG:boto:Data: 2023-10-18T20:17:00.464 DEBUG:boto:Headers: {} 2023-10-18T20:17:00.464 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:00.465 DEBUG:boto:Port: 8001 2023-10-18T20:17:00.465 DEBUG:boto:Params: {} 2023-10-18T20:17:00.466 DEBUG:boto:Token: None 2023-10-18T20:17:00.466 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-49/ 2023-10-18T20:17:00.466 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nUXZpCvSChRScfmnJVpgNNaXVpE= 2023-10-18T20:17:00.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nUXZpCvSChRScfmnJVpgNNaXVpE=', 'Content-Length': '0'} 2023-10-18T20:17:00.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8be529c8b4d658c-0065303d3c-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.469 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:00.470 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:00.470 DEBUG:boto:Method: PUT 2023-10-18T20:17:00.470 DEBUG:boto:Path: /bofrnf-49/obj4 2023-10-18T20:17:00.471 DEBUG:boto:Data: 2023-10-18T20:17:00.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:00.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:00.472 DEBUG:boto:Port: 8001 2023-10-18T20:17:00.472 DEBUG:boto:Params: {} 2023-10-18T20:17:00.472 DEBUG:boto:Token: None 2023-10-18T20:17:00.473 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:00 GMT /bofrnf-49/obj4 2023-10-18T20:17:00.473 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d8TtpSKdj7/S5GF1A9vbYkWqXc8= 2023-10-18T20:17:00.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d8TtpSKdj7/S5GF1A9vbYkWqXc8='} 2023-10-18T20:17:00.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fe36eb87967c35f-0065303d3c-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:00.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:00.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:00.520 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:17:00.568 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:00.568 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:00.596 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:17:00.608 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:17:00.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.609 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660220.203281_747.1", 2023-10-18T20:17:00.609 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:17:00.203281Z" 2023-10-18T20:17:00.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.609 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:17:00.610 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:17:00.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.610 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:00.610 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:17:00.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.611 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:00.611 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:17:00.611 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.612 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:17:00.613 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660220.203281_747.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:00.613 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:17:00.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:00.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:00.614 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:17:00.663 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:00.663 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:00.690 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:00.689+0000 7f5eac456780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:00.691 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:00.690+0000 7f5eac456780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:00.704 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:17:00.704 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:17:00.704 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:17:00.705 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:17:00.705 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:17:00.705 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:17:00.705 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:00.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.706 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:17:00.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.706 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:17:00.706 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:00.706 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:00.707 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660220.176741_733.1", 2023-10-18T20:17:00.707 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:00.707 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:17:00.707 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:00.707 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:17:00.176741Z", 2023-10-18T20:17:00.707 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:00.708 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.708 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:17:00.708 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:17:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:17:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:17:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:17:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:17:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:17:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:17:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:17:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:17:00Z" 2023-10-18T20:17:00.716 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:17:00.717 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660220.176741_733.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:17:00.176741Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:17:00Z" } 2023-10-18T20:17:00.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660220.176741_733.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:17:00.176741Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:17:00.718 DEBUG:rgw_multi.tests:log_status={0: '1_1697660220.203281_747.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:00.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660220.176741_733.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:00.719 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697660220.203281_747.1 target=1_1697660220.176741_733.1 2023-10-18T20:17:05.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:05.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:05.722 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:17:05.779 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:05.780 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:05.810 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:05.809+0000 7f33041a5780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:05.811 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:05.810+0000 7f33041a5780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:05.824 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:17:05.825 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:17:05.825 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:17:05.825 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:17:05.825 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:17:05.826 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:17:05.826 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:05.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:05.826 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:17:05.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:05.827 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:17:05.827 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:05.828 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:05.828 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660220.203281_747.1", 2023-10-18T20:17:05.828 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:05.829 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:17:05.829 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:05.830 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:17:00.203281Z", 2023-10-18T20:17:05.830 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:05.830 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:05.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:05.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:05.831 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:17:05.832 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:05.832 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:05.832 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:17:05.833 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:05.833 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:05.834 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:05.834 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:17:05.835 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:05.835 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:05.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:05.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:05.836 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:17:05.836 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:05.836 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:05.836 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:05.836 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:05.837 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:05.837 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:05.837 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:05.837 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:05.837 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:05.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:05.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:05.838 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:17:05.838 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:05.838 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:05.839 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:05.839 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:05.839 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:05.839 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:05.839 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:05.840 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:05.840 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:05.840 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:05.840 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:17:05.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:05.841 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:17:05.841 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:17:05.841 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:17:05.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:05.841 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:17:05Z" 2023-10-18T20:17:05.842 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:17:05.843 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660220.203281_747.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:17:00.203281Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:17:05Z" } 2023-10-18T20:17:05.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660220.203281_747.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:17:00.203281Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:17:05.844 DEBUG:rgw_multi.tests:log_status={0: '1_1697660220.203281_747.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:05.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660220.203281_747.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:05.844 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:17:05.845 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-48 2023-10-18T20:17:05.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:05.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:05.846 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:17:05.903 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:05.903 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:05.937 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:05.935+0000 7f8c22d21780 1 waiting to reach incremental sync.. 2023-10-18T20:17:10.938 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:10.936+0000 7f8c22d21780 1 waiting to reach incremental sync.. 2023-10-18T20:17:15.938 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:15.937+0000 7f8c22d21780 1 waiting to reach incremental sync.. 2023-10-18T20:17:20.940 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:20.938+0000 7f8c22d21780 1 bucket sync caught up with source: 2023-10-18T20:17:20.941 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.27.6, , 00000000001.23.6, , 00000000001.33.6, , , 00000000001.25.6, , , ] 2023-10-18T20:17:20.941 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.27.6, , 00000000001.23.6, , 00000000001.33.6, , , 00000000001.25.6, , , ] 2023-10-18T20:17:20.941 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:20.938+0000 7f8c22d21780 0 bucket checkpoint complete 2023-10-18T20:17:20.957 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-48 2023-10-18T20:17:20.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:20.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:20.959 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:17:21.020 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:21.021 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:21.056 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:21.055+0000 7f172b796780 1 waiting to reach incremental sync.. 2023-10-18T20:17:26.057 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:26.055+0000 7f172b796780 1 waiting to reach incremental sync.. 2023-10-18T20:17:31.058 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:31.055+0000 7f172b796780 1 waiting to reach incremental sync.. 2023-10-18T20:17:36.058 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:36.056+0000 7f172b796780 1 waiting to reach incremental sync.. 2023-10-18T20:17:41.060 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:41.059+0000 7f172b796780 1 bucket sync caught up with source: 2023-10-18T20:17:41.060 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.289.6, , 00000000001.664.6, , 00000000001.1138.6, , , 00000000001.804.6, , , ] 2023-10-18T20:17:41.061 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.289.6, , 00000000001.664.6, , 00000000001.1138.6, , , 00000000001.804.6, , , ] 2023-10-18T20:17:41.061 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:41.059+0000 7f172b796780 0 bucket checkpoint complete 2023-10-18T20:17:41.077 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-48 zones={a2, a1} 2023-10-18T20:17:41.077 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.078 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.078 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.078 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:41.079 DEBUG:boto:Data: 2023-10-18T20:17:41.079 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.080 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.080 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.081 DEBUG:boto:Params: {} 2023-10-18T20:17:41.081 DEBUG:boto:Token: None 2023-10-18T20:17:41.082 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.082 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ma03/0xkpAOyEVVL4R0Rj6qJgkE= 2023-10-18T20:17:41.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ma03/0xkpAOyEVVL4R0Rj6qJgkE=', 'Content-Length': '0'} 2023-10-18T20:17:41.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e0ffbafbe69df4d-0065303d65-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.086 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.086 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.087 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.087 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:41.088 DEBUG:boto:Data: 2023-10-18T20:17:41.088 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.088 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.089 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.089 DEBUG:boto:Params: {} 2023-10-18T20:17:41.089 DEBUG:boto:Token: None 2023-10-18T20:17:41.090 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.090 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ma03/0xkpAOyEVVL4R0Rj6qJgkE= 2023-10-18T20:17:41.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ma03/0xkpAOyEVVL4R0Rj6qJgkE=', 'Content-Length': '0'} 2023-10-18T20:17:41.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae22dedb2e213c1c-0065303d65-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.094 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:17:41.095 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.095 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.095 DEBUG:boto:path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.096 DEBUG:boto:auth_path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.096 DEBUG:boto:Method: GET 2023-10-18T20:17:41.097 DEBUG:boto:Path: /bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.097 DEBUG:boto:Data: 2023-10-18T20:17:41.098 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.098 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.099 DEBUG:boto:Params: {} 2023-10-18T20:17:41.099 DEBUG:boto:Token: None 2023-10-18T20:17:41.100 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/?versions 2023-10-18T20:17:41.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGkdMhzj3cRtOZwJIEwPKPbyyL0= 2023-10-18T20:17:41.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGkdMhzj3cRtOZwJIEwPKPbyyL0=', 'Content-Length': '0'} 2023-10-18T20:17:41.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019bb389a3e1858b-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.104 DEBUG:boto:b'bofrnf-48999falseobj1nulltrue2023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:17:41.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:17:41.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:17:41.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:17:41.106 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:17:41.106 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.107 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.107 DEBUG:boto:path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.107 DEBUG:boto:auth_path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.108 DEBUG:boto:Method: GET 2023-10-18T20:17:41.108 DEBUG:boto:Path: /bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.109 DEBUG:boto:Data: 2023-10-18T20:17:41.109 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.110 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.110 DEBUG:boto:Params: {} 2023-10-18T20:17:41.111 DEBUG:boto:Token: None 2023-10-18T20:17:41.111 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/?versions 2023-10-18T20:17:41.112 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGkdMhzj3cRtOZwJIEwPKPbyyL0= 2023-10-18T20:17:41.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGkdMhzj3cRtOZwJIEwPKPbyyL0=', 'Content-Length': '0'} 2023-10-18T20:17:41.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de8de20e6fb43876-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.115 DEBUG:boto:b'bofrnf-48999falseobj1nulltrue2023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:17:41.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:17:41.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:17:41.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:17:41.117 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.118 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.118 DEBUG:boto:path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.118 DEBUG:boto:auth_path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.119 DEBUG:boto:Method: GET 2023-10-18T20:17:41.119 DEBUG:boto:Path: /bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.120 DEBUG:boto:Data: 2023-10-18T20:17:41.120 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.120 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.121 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.121 DEBUG:boto:Params: {} 2023-10-18T20:17:41.121 DEBUG:boto:Token: None 2023-10-18T20:17:41.122 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/?versions 2023-10-18T20:17:41.122 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGkdMhzj3cRtOZwJIEwPKPbyyL0= 2023-10-18T20:17:41.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGkdMhzj3cRtOZwJIEwPKPbyyL0=', 'Content-Length': '0'} 2023-10-18T20:17:41.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deea644061117149-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.126 DEBUG:boto:b'bofrnf-48999falseobj1nulltrue2023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.126 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.127 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.127 DEBUG:boto:path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.128 DEBUG:boto:auth_path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.128 DEBUG:boto:Method: GET 2023-10-18T20:17:41.128 DEBUG:boto:Path: /bofrnf-48/?versions&max-keys=999 2023-10-18T20:17:41.129 DEBUG:boto:Data: 2023-10-18T20:17:41.129 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.130 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.130 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.131 DEBUG:boto:Params: {} 2023-10-18T20:17:41.131 DEBUG:boto:Token: None 2023-10-18T20:17:41.132 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/?versions 2023-10-18T20:17:41.132 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eGkdMhzj3cRtOZwJIEwPKPbyyL0= 2023-10-18T20:17:41.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eGkdMhzj3cRtOZwJIEwPKPbyyL0=', 'Content-Length': '0'} 2023-10-18T20:17:41.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b3d180a02ff996-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.135 DEBUG:boto:b'bofrnf-48999falseobj1nulltrue2023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:17:41.137 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.137 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.137 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.138 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.138 DEBUG:boto:Method: GET 2023-10-18T20:17:41.139 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.139 DEBUG:boto:Data: 2023-10-18T20:17:41.140 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.140 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.140 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.141 DEBUG:boto:Params: {} 2023-10-18T20:17:41.141 DEBUG:boto:Token: None 2023-10-18T20:17:41.142 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.142 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jq34ptn6CaOakFP1xALm5bbooTE= 2023-10-18T20:17:41.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jq34ptn6CaOakFP1xALm5bbooTE=', 'Content-Length': '0'} 2023-10-18T20:17:41.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000747465ff39d8fa0d-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.146 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.146 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.147 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.147 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.148 DEBUG:boto:Method: GET 2023-10-18T20:17:41.148 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.148 DEBUG:boto:Data: 2023-10-18T20:17:41.149 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.149 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.149 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.150 DEBUG:boto:Params: {} 2023-10-18T20:17:41.150 DEBUG:boto:Token: None 2023-10-18T20:17:41.151 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.151 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jq34ptn6CaOakFP1xALm5bbooTE= 2023-10-18T20:17:41.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jq34ptn6CaOakFP1xALm5bbooTE=', 'Content-Length': '0'} 2023-10-18T20:17:41.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000768ebd1cd00a433c-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.154 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.155 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.155 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.156 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.156 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.156 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.157 DEBUG:boto:Data: 2023-10-18T20:17:41.157 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.157 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.158 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.158 DEBUG:boto:Params: {} 2023-10-18T20:17:41.159 DEBUG:boto:Token: None 2023-10-18T20:17:41.159 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.160 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TsGTUIR0yR+u3abwyqq5Ik2Wuzw= 2023-10-18T20:17:41.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TsGTUIR0yR+u3abwyqq5Ik2Wuzw=', 'Content-Length': '0'} 2023-10-18T20:17:41.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f8ead3f81e5e476-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.163 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.163 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.163 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.164 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.164 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.165 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.165 DEBUG:boto:Data: 2023-10-18T20:17:41.165 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.166 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.167 DEBUG:boto:Params: {} 2023-10-18T20:17:41.167 DEBUG:boto:Token: None 2023-10-18T20:17:41.167 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TsGTUIR0yR+u3abwyqq5Ik2Wuzw= 2023-10-18T20:17:41.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TsGTUIR0yR+u3abwyqq5Ik2Wuzw=', 'Content-Length': '0'} 2023-10-18T20:17:41.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c765e55947d433be-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:17:41.171 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.171 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.172 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.172 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.173 DEBUG:boto:Method: GET 2023-10-18T20:17:41.173 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.174 DEBUG:boto:Data: 2023-10-18T20:17:41.174 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.175 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.175 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.175 DEBUG:boto:Params: {} 2023-10-18T20:17:41.176 DEBUG:boto:Token: None 2023-10-18T20:17:41.176 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.177 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jq34ptn6CaOakFP1xALm5bbooTE= 2023-10-18T20:17:41.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jq34ptn6CaOakFP1xALm5bbooTE=', 'Content-Length': '0'} 2023-10-18T20:17:41.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4b100642e5fff06-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.180 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.180 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.181 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.181 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.181 DEBUG:boto:Method: GET 2023-10-18T20:17:41.182 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.182 DEBUG:boto:Data: 2023-10-18T20:17:41.183 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.183 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.184 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.184 DEBUG:boto:Params: {} 2023-10-18T20:17:41.184 DEBUG:boto:Token: None 2023-10-18T20:17:41.185 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:17:41.185 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jq34ptn6CaOakFP1xALm5bbooTE= 2023-10-18T20:17:41.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jq34ptn6CaOakFP1xALm5bbooTE=', 'Content-Length': '0'} 2023-10-18T20:17:41.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f41d336981c3a587-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.188 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.189 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.189 DEBUG:boto:path=/bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.190 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.190 DEBUG:boto:Method: GET 2023-10-18T20:17:41.190 DEBUG:boto:Path: /bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.191 DEBUG:boto:Data: 2023-10-18T20:17:41.191 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.192 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.192 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.192 DEBUG:boto:Params: {} 2023-10-18T20:17:41.193 DEBUG:boto:Token: None 2023-10-18T20:17:41.193 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.194 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032a8d8870b9c6348-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.197 DEBUG:boto:b'bofrnf-48obj11000falseobj12023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.198 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.198 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.199 DEBUG:boto:path=/bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.199 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.199 DEBUG:boto:Method: GET 2023-10-18T20:17:41.200 DEBUG:boto:Path: /bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.200 DEBUG:boto:Data: 2023-10-18T20:17:41.200 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.201 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.201 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.202 DEBUG:boto:Params: {} 2023-10-18T20:17:41.202 DEBUG:boto:Token: None 2023-10-18T20:17:41.203 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.203 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aa0682547a9a84d-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.207 DEBUG:boto:b'bofrnf-48obj11000falseobj12023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.207 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.208 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.208 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.208 DEBUG:boto:Path: /bofrnf-48/obj1 2023-10-18T20:17:41.209 DEBUG:boto:Data: 2023-10-18T20:17:41.209 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.210 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.210 DEBUG:boto:Params: {} 2023-10-18T20:17:41.211 DEBUG:boto:Token: None 2023-10-18T20:17:41.211 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1 2023-10-18T20:17:41.212 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KmArG18dKkWnURAVqaUhdIFoVYc= 2023-10-18T20:17:41.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KmArG18dKkWnURAVqaUhdIFoVYc=', 'Content-Length': '0'} 2023-10-18T20:17:41.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005960f40772e7c2f1-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.215 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.215 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.215 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.216 DEBUG:boto:Path: /bofrnf-48/obj1 2023-10-18T20:17:41.216 DEBUG:boto:Data: 2023-10-18T20:17:41.217 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.217 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.217 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.218 DEBUG:boto:Params: {} 2023-10-18T20:17:41.218 DEBUG:boto:Token: None 2023-10-18T20:17:41.219 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1 2023-10-18T20:17:41.219 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KmArG18dKkWnURAVqaUhdIFoVYc= 2023-10-18T20:17:41.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KmArG18dKkWnURAVqaUhdIFoVYc=', 'Content-Length': '0'} 2023-10-18T20:17:41.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b03fe4bf24889e2f-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:17:41.222 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.223 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.223 DEBUG:boto:Method: GET 2023-10-18T20:17:41.224 DEBUG:boto:Path: /bofrnf-48/obj1 2023-10-18T20:17:41.224 DEBUG:boto:Data: 2023-10-18T20:17:41.224 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.225 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.225 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.226 DEBUG:boto:Params: {} 2023-10-18T20:17:41.226 DEBUG:boto:Token: None 2023-10-18T20:17:41.227 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1 2023-10-18T20:17:41.227 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RrGb0ojR3SVhYV3PS4YJ8Pu2eYM= 2023-10-18T20:17:41.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RrGb0ojR3SVhYV3PS4YJ8Pu2eYM=', 'Content-Length': '0'} 2023-10-18T20:17:41.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c9ea233b2dd471a-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.230 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:17:41.231 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:17:41.231 DEBUG:boto:Method: GET 2023-10-18T20:17:41.232 DEBUG:boto:Path: /bofrnf-48/obj1 2023-10-18T20:17:41.232 DEBUG:boto:Data: 2023-10-18T20:17:41.232 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.233 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.233 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.234 DEBUG:boto:Params: {} 2023-10-18T20:17:41.234 DEBUG:boto:Token: None 2023-10-18T20:17:41.235 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj1 2023-10-18T20:17:41.235 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RrGb0ojR3SVhYV3PS4YJ8Pu2eYM= 2023-10-18T20:17:41.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RrGb0ojR3SVhYV3PS4YJ8Pu2eYM=', 'Content-Length': '0'} 2023-10-18T20:17:41.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016a5e734f1454d33-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.278 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.279 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.279 DEBUG:boto:path=/bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.280 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.280 DEBUG:boto:Method: GET 2023-10-18T20:17:41.281 DEBUG:boto:Path: /bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.281 DEBUG:boto:Data: 2023-10-18T20:17:41.281 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.282 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.283 DEBUG:boto:Params: {} 2023-10-18T20:17:41.283 DEBUG:boto:Token: None 2023-10-18T20:17:41.284 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005baa2ad6931dcbc5-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.288 DEBUG:boto:b'bofrnf-48obj11000falseobj12023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.289 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.289 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.290 DEBUG:boto:path=/bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.290 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.290 DEBUG:boto:Method: GET 2023-10-18T20:17:41.291 DEBUG:boto:Path: /bofrnf-48/?prefix=obj1 2023-10-18T20:17:41.291 DEBUG:boto:Data: 2023-10-18T20:17:41.292 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.292 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.292 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.293 DEBUG:boto:Params: {} 2023-10-18T20:17:41.293 DEBUG:boto:Token: None 2023-10-18T20:17:41.294 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.294 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000339d6fa87ba98de-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.298 DEBUG:boto:b'bofrnf-48obj11000falseobj12023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:17:41.299 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.299 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.300 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.300 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.300 DEBUG:boto:Method: GET 2023-10-18T20:17:41.301 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.301 DEBUG:boto:Data: 2023-10-18T20:17:41.302 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.303 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.303 DEBUG:boto:Params: {} 2023-10-18T20:17:41.303 DEBUG:boto:Token: None 2023-10-18T20:17:41.304 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wItfD7WxwXl2rholwaLRNomhIF0= 2023-10-18T20:17:41.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wItfD7WxwXl2rholwaLRNomhIF0=', 'Content-Length': '0'} 2023-10-18T20:17:41.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acc142bfd50849e4-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.350 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.351 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.352 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.352 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.352 DEBUG:boto:Method: GET 2023-10-18T20:17:41.353 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.353 DEBUG:boto:Data: 2023-10-18T20:17:41.354 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.354 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.355 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.355 DEBUG:boto:Params: {} 2023-10-18T20:17:41.355 DEBUG:boto:Token: None 2023-10-18T20:17:41.356 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.357 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wItfD7WxwXl2rholwaLRNomhIF0= 2023-10-18T20:17:41.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wItfD7WxwXl2rholwaLRNomhIF0=', 'Content-Length': '0'} 2023-10-18T20:17:41.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adc546e42e4fe956-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.360 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.360 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.361 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.361 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.362 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.362 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.363 DEBUG:boto:Data: 2023-10-18T20:17:41.363 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.363 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.364 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.364 DEBUG:boto:Params: {} 2023-10-18T20:17:41.365 DEBUG:boto:Token: None 2023-10-18T20:17:41.365 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.365 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4Z/XjYh+Lge5zE9/tJh9gFP/JCI= 2023-10-18T20:17:41.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4Z/XjYh+Lge5zE9/tJh9gFP/JCI=', 'Content-Length': '0'} 2023-10-18T20:17:41.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a9232a45e2f2abb-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.368 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.369 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.369 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.370 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.370 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.370 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.371 DEBUG:boto:Data: 2023-10-18T20:17:41.371 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.372 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.372 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.372 DEBUG:boto:Params: {} 2023-10-18T20:17:41.373 DEBUG:boto:Token: None 2023-10-18T20:17:41.373 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.374 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4Z/XjYh+Lge5zE9/tJh9gFP/JCI= 2023-10-18T20:17:41.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4Z/XjYh+Lge5zE9/tJh9gFP/JCI=', 'Content-Length': '0'} 2023-10-18T20:17:41.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f1290155b19fe6b-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:17:41.377 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.378 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.378 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.378 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.379 DEBUG:boto:Method: GET 2023-10-18T20:17:41.379 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.380 DEBUG:boto:Data: 2023-10-18T20:17:41.380 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.381 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.381 DEBUG:boto:Params: {} 2023-10-18T20:17:41.382 DEBUG:boto:Token: None 2023-10-18T20:17:41.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wItfD7WxwXl2rholwaLRNomhIF0= 2023-10-18T20:17:41.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wItfD7WxwXl2rholwaLRNomhIF0=', 'Content-Length': '0'} 2023-10-18T20:17:41.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094bfdcb4ad68cf71-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.386 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.386 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.387 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.387 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.388 DEBUG:boto:Method: GET 2023-10-18T20:17:41.388 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.389 DEBUG:boto:Data: 2023-10-18T20:17:41.389 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.390 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.390 DEBUG:boto:Params: {} 2023-10-18T20:17:41.391 DEBUG:boto:Token: None 2023-10-18T20:17:41.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:17:41.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wItfD7WxwXl2rholwaLRNomhIF0= 2023-10-18T20:17:41.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wItfD7WxwXl2rholwaLRNomhIF0=', 'Content-Length': '0'} 2023-10-18T20:17:41.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022ce17cb05c068ad-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.395 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.395 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.396 DEBUG:boto:path=/bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.396 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.397 DEBUG:boto:Method: GET 2023-10-18T20:17:41.397 DEBUG:boto:Path: /bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.397 DEBUG:boto:Data: 2023-10-18T20:17:41.398 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.399 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.399 DEBUG:boto:Params: {} 2023-10-18T20:17:41.400 DEBUG:boto:Token: None 2023-10-18T20:17:41.400 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa975434bfd7f0d8-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.404 DEBUG:boto:b'bofrnf-48obj21000falseobj22023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.405 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.405 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.406 DEBUG:boto:path=/bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.406 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.407 DEBUG:boto:Method: GET 2023-10-18T20:17:41.407 DEBUG:boto:Path: /bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.408 DEBUG:boto:Data: 2023-10-18T20:17:41.408 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.409 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.409 DEBUG:boto:Params: {} 2023-10-18T20:17:41.410 DEBUG:boto:Token: None 2023-10-18T20:17:41.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ff006150ea7264-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.454 DEBUG:boto:b'bofrnf-48obj21000falseobj22023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.455 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.456 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.456 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.456 DEBUG:boto:Path: /bofrnf-48/obj2 2023-10-18T20:17:41.457 DEBUG:boto:Data: 2023-10-18T20:17:41.457 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.458 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.458 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.458 DEBUG:boto:Params: {} 2023-10-18T20:17:41.459 DEBUG:boto:Token: None 2023-10-18T20:17:41.459 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2 2023-10-18T20:17:41.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7QPgt2L++1cjv34M3mfYnCB6tic= 2023-10-18T20:17:41.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7QPgt2L++1cjv34M3mfYnCB6tic=', 'Content-Length': '0'} 2023-10-18T20:17:41.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b22e351fa68d3d8f-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.463 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.463 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.464 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.464 DEBUG:boto:Path: /bofrnf-48/obj2 2023-10-18T20:17:41.465 DEBUG:boto:Data: 2023-10-18T20:17:41.465 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.465 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.466 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.466 DEBUG:boto:Params: {} 2023-10-18T20:17:41.467 DEBUG:boto:Token: None 2023-10-18T20:17:41.467 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2 2023-10-18T20:17:41.467 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7QPgt2L++1cjv34M3mfYnCB6tic= 2023-10-18T20:17:41.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7QPgt2L++1cjv34M3mfYnCB6tic=', 'Content-Length': '0'} 2023-10-18T20:17:41.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ef5a20ce4ee42a2-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:17:41.471 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.471 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.472 DEBUG:boto:Method: GET 2023-10-18T20:17:41.472 DEBUG:boto:Path: /bofrnf-48/obj2 2023-10-18T20:17:41.473 DEBUG:boto:Data: 2023-10-18T20:17:41.473 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.473 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.474 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.474 DEBUG:boto:Params: {} 2023-10-18T20:17:41.475 DEBUG:boto:Token: None 2023-10-18T20:17:41.475 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2 2023-10-18T20:17:41.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JWw/IH/j0S54swHfmtjsbeSdgOc= 2023-10-18T20:17:41.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JWw/IH/j0S54swHfmtjsbeSdgOc=', 'Content-Length': '0'} 2023-10-18T20:17:41.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cefd0aa951b385b-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.479 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:17:41.479 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:17:41.480 DEBUG:boto:Method: GET 2023-10-18T20:17:41.480 DEBUG:boto:Path: /bofrnf-48/obj2 2023-10-18T20:17:41.481 DEBUG:boto:Data: 2023-10-18T20:17:41.481 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.482 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.482 DEBUG:boto:Params: {} 2023-10-18T20:17:41.483 DEBUG:boto:Token: None 2023-10-18T20:17:41.483 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj2 2023-10-18T20:17:41.484 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JWw/IH/j0S54swHfmtjsbeSdgOc= 2023-10-18T20:17:41.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JWw/IH/j0S54swHfmtjsbeSdgOc=', 'Content-Length': '0'} 2023-10-18T20:17:41.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f558cf5eddcd5c42-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.487 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.487 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.488 DEBUG:boto:path=/bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.488 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.488 DEBUG:boto:Method: GET 2023-10-18T20:17:41.489 DEBUG:boto:Path: /bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.489 DEBUG:boto:Data: 2023-10-18T20:17:41.490 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.490 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.490 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.491 DEBUG:boto:Params: {} 2023-10-18T20:17:41.491 DEBUG:boto:Token: None 2023-10-18T20:17:41.492 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.492 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af59db97ab5ac158-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.542 DEBUG:boto:b'bofrnf-48obj21000falseobj22023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.543 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.544 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.544 DEBUG:boto:path=/bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.545 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.545 DEBUG:boto:Method: GET 2023-10-18T20:17:41.545 DEBUG:boto:Path: /bofrnf-48/?prefix=obj2 2023-10-18T20:17:41.546 DEBUG:boto:Data: 2023-10-18T20:17:41.546 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.547 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.547 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.547 DEBUG:boto:Params: {} 2023-10-18T20:17:41.548 DEBUG:boto:Token: None 2023-10-18T20:17:41.548 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.549 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002ef243138a5cdd2-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.553 DEBUG:boto:b'bofrnf-48obj21000falseobj22023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:17:41.554 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.554 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.555 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.555 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.555 DEBUG:boto:Method: GET 2023-10-18T20:17:41.556 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.556 DEBUG:boto:Data: 2023-10-18T20:17:41.557 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.557 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.557 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.558 DEBUG:boto:Params: {} 2023-10-18T20:17:41.558 DEBUG:boto:Token: None 2023-10-18T20:17:41.559 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.559 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7356r1G40P7J86G0YzeangVWPGg= 2023-10-18T20:17:41.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7356r1G40P7J86G0YzeangVWPGg=', 'Content-Length': '0'} 2023-10-18T20:17:41.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007f0b7c6799ffcac-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.562 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.563 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.563 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.564 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.564 DEBUG:boto:Method: GET 2023-10-18T20:17:41.564 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.565 DEBUG:boto:Data: 2023-10-18T20:17:41.565 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.566 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.566 DEBUG:boto:Params: {} 2023-10-18T20:17:41.567 DEBUG:boto:Token: None 2023-10-18T20:17:41.567 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7356r1G40P7J86G0YzeangVWPGg= 2023-10-18T20:17:41.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7356r1G40P7J86G0YzeangVWPGg=', 'Content-Length': '0'} 2023-10-18T20:17:41.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003edc9d0d1024122e-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.571 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.573 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.573 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.574 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.574 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.575 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.575 DEBUG:boto:Data: 2023-10-18T20:17:41.575 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.576 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.576 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.576 DEBUG:boto:Params: {} 2023-10-18T20:17:41.577 DEBUG:boto:Token: None 2023-10-18T20:17:41.577 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.578 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B6VZrKVGt7L8hDuCwxbYEOYiSVI= 2023-10-18T20:17:41.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B6VZrKVGt7L8hDuCwxbYEOYiSVI=', 'Content-Length': '0'} 2023-10-18T20:17:41.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed5751f84a335f45-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.581 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.581 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.581 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.582 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.582 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.583 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.583 DEBUG:boto:Data: 2023-10-18T20:17:41.583 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.584 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.584 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.584 DEBUG:boto:Params: {} 2023-10-18T20:17:41.585 DEBUG:boto:Token: None 2023-10-18T20:17:41.585 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B6VZrKVGt7L8hDuCwxbYEOYiSVI= 2023-10-18T20:17:41.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B6VZrKVGt7L8hDuCwxbYEOYiSVI=', 'Content-Length': '0'} 2023-10-18T20:17:41.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ad2f3dc13c66b88-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:17:41.589 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.589 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.590 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.590 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.590 DEBUG:boto:Method: GET 2023-10-18T20:17:41.591 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.591 DEBUG:boto:Data: 2023-10-18T20:17:41.591 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.592 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.592 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.593 DEBUG:boto:Params: {} 2023-10-18T20:17:41.593 DEBUG:boto:Token: None 2023-10-18T20:17:41.593 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.594 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7356r1G40P7J86G0YzeangVWPGg= 2023-10-18T20:17:41.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7356r1G40P7J86G0YzeangVWPGg=', 'Content-Length': '0'} 2023-10-18T20:17:41.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be873166415e8bc0-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.597 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.597 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.598 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.598 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.599 DEBUG:boto:Method: GET 2023-10-18T20:17:41.599 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.599 DEBUG:boto:Data: 2023-10-18T20:17:41.600 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.600 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.601 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.601 DEBUG:boto:Params: {} 2023-10-18T20:17:41.601 DEBUG:boto:Token: None 2023-10-18T20:17:41.602 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:17:41.602 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7356r1G40P7J86G0YzeangVWPGg= 2023-10-18T20:17:41.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7356r1G40P7J86G0YzeangVWPGg=', 'Content-Length': '0'} 2023-10-18T20:17:41.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1c13ad442f9890a-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.605 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.606 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.606 DEBUG:boto:path=/bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.607 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.607 DEBUG:boto:Method: GET 2023-10-18T20:17:41.607 DEBUG:boto:Path: /bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.608 DEBUG:boto:Data: 2023-10-18T20:17:41.608 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.608 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.609 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.609 DEBUG:boto:Params: {} 2023-10-18T20:17:41.610 DEBUG:boto:Token: None 2023-10-18T20:17:41.610 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.611 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1b3166be2413737-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.614 DEBUG:boto:b'bofrnf-48obj31000falseobj32023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.615 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.615 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.616 DEBUG:boto:path=/bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.616 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.616 DEBUG:boto:Method: GET 2023-10-18T20:17:41.617 DEBUG:boto:Path: /bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.617 DEBUG:boto:Data: 2023-10-18T20:17:41.617 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.618 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.618 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.619 DEBUG:boto:Params: {} 2023-10-18T20:17:41.619 DEBUG:boto:Token: None 2023-10-18T20:17:41.619 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.620 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d82204ade257fa9-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.623 DEBUG:boto:b'bofrnf-48obj31000falseobj32023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.624 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.624 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.625 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.625 DEBUG:boto:Path: /bofrnf-48/obj3 2023-10-18T20:17:41.626 DEBUG:boto:Data: 2023-10-18T20:17:41.626 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.626 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.627 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.627 DEBUG:boto:Params: {} 2023-10-18T20:17:41.628 DEBUG:boto:Token: None 2023-10-18T20:17:41.628 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3 2023-10-18T20:17:41.628 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QKvCn1Exkz2S45LcYWqg+5M0CP0= 2023-10-18T20:17:41.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QKvCn1Exkz2S45LcYWqg+5M0CP0=', 'Content-Length': '0'} 2023-10-18T20:17:41.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000983e690e71cf3165-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.631 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.631 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.632 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.632 DEBUG:boto:Path: /bofrnf-48/obj3 2023-10-18T20:17:41.633 DEBUG:boto:Data: 2023-10-18T20:17:41.633 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.633 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.634 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.634 DEBUG:boto:Params: {} 2023-10-18T20:17:41.635 DEBUG:boto:Token: None 2023-10-18T20:17:41.635 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3 2023-10-18T20:17:41.635 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QKvCn1Exkz2S45LcYWqg+5M0CP0= 2023-10-18T20:17:41.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QKvCn1Exkz2S45LcYWqg+5M0CP0=', 'Content-Length': '0'} 2023-10-18T20:17:41.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bf4a4b94504f5a2-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:17:41.639 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.639 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.639 DEBUG:boto:Method: GET 2023-10-18T20:17:41.640 DEBUG:boto:Path: /bofrnf-48/obj3 2023-10-18T20:17:41.640 DEBUG:boto:Data: 2023-10-18T20:17:41.641 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.641 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.642 DEBUG:boto:Params: {} 2023-10-18T20:17:41.642 DEBUG:boto:Token: None 2023-10-18T20:17:41.643 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3 2023-10-18T20:17:41.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e+uplvKKywJnB8EafcJbAZS75Oo= 2023-10-18T20:17:41.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e+uplvKKywJnB8EafcJbAZS75Oo=', 'Content-Length': '0'} 2023-10-18T20:17:41.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c68ec0fa72075411-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.646 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:17:41.647 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:17:41.647 DEBUG:boto:Method: GET 2023-10-18T20:17:41.647 DEBUG:boto:Path: /bofrnf-48/obj3 2023-10-18T20:17:41.648 DEBUG:boto:Data: 2023-10-18T20:17:41.648 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.649 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.649 DEBUG:boto:Params: {} 2023-10-18T20:17:41.650 DEBUG:boto:Token: None 2023-10-18T20:17:41.650 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj3 2023-10-18T20:17:41.651 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e+uplvKKywJnB8EafcJbAZS75Oo= 2023-10-18T20:17:41.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e+uplvKKywJnB8EafcJbAZS75Oo=', 'Content-Length': '0'} 2023-10-18T20:17:41.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000518e98b78b5f65cb-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.654 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.654 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.655 DEBUG:boto:path=/bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.655 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.655 DEBUG:boto:Method: GET 2023-10-18T20:17:41.656 DEBUG:boto:Path: /bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.656 DEBUG:boto:Data: 2023-10-18T20:17:41.657 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.657 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.658 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.658 DEBUG:boto:Params: {} 2023-10-18T20:17:41.659 DEBUG:boto:Token: None 2023-10-18T20:17:41.659 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.659 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003223c5654c5ab691-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.663 DEBUG:boto:b'bofrnf-48obj31000falseobj32023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.664 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.664 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.665 DEBUG:boto:path=/bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.665 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.666 DEBUG:boto:Method: GET 2023-10-18T20:17:41.666 DEBUG:boto:Path: /bofrnf-48/?prefix=obj3 2023-10-18T20:17:41.666 DEBUG:boto:Data: 2023-10-18T20:17:41.667 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.667 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.668 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.668 DEBUG:boto:Params: {} 2023-10-18T20:17:41.668 DEBUG:boto:Token: None 2023-10-18T20:17:41.669 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.669 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df411380d5ce1bc7-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.673 DEBUG:boto:b'bofrnf-48obj31000falseobj32023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:17:41.674 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.675 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.675 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.675 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.676 DEBUG:boto:Method: GET 2023-10-18T20:17:41.676 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.676 DEBUG:boto:Data: 2023-10-18T20:17:41.677 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.677 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.677 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.678 DEBUG:boto:Params: {} 2023-10-18T20:17:41.678 DEBUG:boto:Token: None 2023-10-18T20:17:41.679 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tUyEQb1GjuerrbIUGoGPcP5tE4M= 2023-10-18T20:17:41.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tUyEQb1GjuerrbIUGoGPcP5tE4M=', 'Content-Length': '0'} 2023-10-18T20:17:41.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af5b5b63f02f57f7-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.722 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.723 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.723 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.724 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.724 DEBUG:boto:Method: GET 2023-10-18T20:17:41.724 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.725 DEBUG:boto:Data: 2023-10-18T20:17:41.725 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.726 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.726 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.726 DEBUG:boto:Params: {} 2023-10-18T20:17:41.727 DEBUG:boto:Token: None 2023-10-18T20:17:41.727 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tUyEQb1GjuerrbIUGoGPcP5tE4M= 2023-10-18T20:17:41.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tUyEQb1GjuerrbIUGoGPcP5tE4M=', 'Content-Length': '0'} 2023-10-18T20:17:41.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da2fa95d75753dd1-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.731 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.731 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.732 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.732 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.732 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.733 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.733 DEBUG:boto:Data: 2023-10-18T20:17:41.734 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.734 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.734 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.735 DEBUG:boto:Params: {} 2023-10-18T20:17:41.735 DEBUG:boto:Token: None 2023-10-18T20:17:41.736 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.736 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hHdRZ/+bfHZBxaRd56CKrj/KjLU= 2023-10-18T20:17:41.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hHdRZ/+bfHZBxaRd56CKrj/KjLU=', 'Content-Length': '0'} 2023-10-18T20:17:41.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b811b4b0c4979829-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.739 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.739 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.740 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.740 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.740 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.741 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.741 DEBUG:boto:Data: 2023-10-18T20:17:41.741 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.742 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.742 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.743 DEBUG:boto:Params: {} 2023-10-18T20:17:41.743 DEBUG:boto:Token: None 2023-10-18T20:17:41.743 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.744 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hHdRZ/+bfHZBxaRd56CKrj/KjLU= 2023-10-18T20:17:41.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hHdRZ/+bfHZBxaRd56CKrj/KjLU=', 'Content-Length': '0'} 2023-10-18T20:17:41.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000871d67ea9aaf5f6b-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:17:41.747 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.747 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.748 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.748 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.749 DEBUG:boto:Method: GET 2023-10-18T20:17:41.749 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.749 DEBUG:boto:Data: 2023-10-18T20:17:41.750 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.750 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.751 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.751 DEBUG:boto:Params: {} 2023-10-18T20:17:41.751 DEBUG:boto:Token: None 2023-10-18T20:17:41.752 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.752 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tUyEQb1GjuerrbIUGoGPcP5tE4M= 2023-10-18T20:17:41.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tUyEQb1GjuerrbIUGoGPcP5tE4M=', 'Content-Length': '0'} 2023-10-18T20:17:41.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5ccc05913a09084-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.756 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.756 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.757 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.757 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.757 DEBUG:boto:Method: GET 2023-10-18T20:17:41.758 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.758 DEBUG:boto:Data: 2023-10-18T20:17:41.758 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.759 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.759 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.760 DEBUG:boto:Params: {} 2023-10-18T20:17:41.760 DEBUG:boto:Token: None 2023-10-18T20:17:41.760 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:17:41.761 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tUyEQb1GjuerrbIUGoGPcP5tE4M= 2023-10-18T20:17:41.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tUyEQb1GjuerrbIUGoGPcP5tE4M=', 'Content-Length': '0'} 2023-10-18T20:17:41.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000022681c7538a9c60-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.806 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.807 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.807 DEBUG:boto:path=/bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.808 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.808 DEBUG:boto:Method: GET 2023-10-18T20:17:41.808 DEBUG:boto:Path: /bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.809 DEBUG:boto:Data: 2023-10-18T20:17:41.809 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.810 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.810 DEBUG:boto:Params: {} 2023-10-18T20:17:41.811 DEBUG:boto:Token: None 2023-10-18T20:17:41.811 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000957e9f61822b5b91-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.815 DEBUG:boto:b'bofrnf-48obj41000falseobj42023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.816 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.816 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.817 DEBUG:boto:path=/bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.817 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.817 DEBUG:boto:Method: GET 2023-10-18T20:17:41.818 DEBUG:boto:Path: /bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.818 DEBUG:boto:Data: 2023-10-18T20:17:41.819 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.819 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.820 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.820 DEBUG:boto:Params: {} 2023-10-18T20:17:41.820 DEBUG:boto:Token: None 2023-10-18T20:17:41.821 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cfb7931efb6f403-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.825 DEBUG:boto:b'bofrnf-48obj41000falseobj42023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.825 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.826 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.826 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.827 DEBUG:boto:Path: /bofrnf-48/obj4 2023-10-18T20:17:41.827 DEBUG:boto:Data: 2023-10-18T20:17:41.827 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.828 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.828 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.828 DEBUG:boto:Params: {} 2023-10-18T20:17:41.829 DEBUG:boto:Token: None 2023-10-18T20:17:41.829 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4 2023-10-18T20:17:41.830 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OWv9Iyw0ToOnozWqlWmepkkZXpo= 2023-10-18T20:17:41.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OWv9Iyw0ToOnozWqlWmepkkZXpo=', 'Content-Length': '0'} 2023-10-18T20:17:41.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f813c475d35bdc9-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.833 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.833 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.833 DEBUG:boto:Method: HEAD 2023-10-18T20:17:41.834 DEBUG:boto:Path: /bofrnf-48/obj4 2023-10-18T20:17:41.834 DEBUG:boto:Data: 2023-10-18T20:17:41.835 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.835 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.835 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.836 DEBUG:boto:Params: {} 2023-10-18T20:17:41.836 DEBUG:boto:Token: None 2023-10-18T20:17:41.837 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4 2023-10-18T20:17:41.837 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OWv9Iyw0ToOnozWqlWmepkkZXpo= 2023-10-18T20:17:41.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OWv9Iyw0ToOnozWqlWmepkkZXpo=', 'Content-Length': '0'} 2023-10-18T20:17:41.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c97b892474e6b135-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:17:41.840 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.841 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.841 DEBUG:boto:Method: GET 2023-10-18T20:17:41.842 DEBUG:boto:Path: /bofrnf-48/obj4 2023-10-18T20:17:41.842 DEBUG:boto:Data: 2023-10-18T20:17:41.842 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.843 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.844 DEBUG:boto:Params: {} 2023-10-18T20:17:41.844 DEBUG:boto:Token: None 2023-10-18T20:17:41.845 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4 2023-10-18T20:17:41.845 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Wzw4vVoKpZIIaZ759p0Zfs4EPqw= 2023-10-18T20:17:41.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Wzw4vVoKpZIIaZ759p0Zfs4EPqw=', 'Content-Length': '0'} 2023-10-18T20:17:41.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d89d1ef50450026-0065303d65-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.890 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:17:41.891 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:17:41.891 DEBUG:boto:Method: GET 2023-10-18T20:17:41.892 DEBUG:boto:Path: /bofrnf-48/obj4 2023-10-18T20:17:41.892 DEBUG:boto:Data: 2023-10-18T20:17:41.893 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.893 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.894 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.894 DEBUG:boto:Params: {} 2023-10-18T20:17:41.895 DEBUG:boto:Token: None 2023-10-18T20:17:41.895 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/obj4 2023-10-18T20:17:41.896 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Wzw4vVoKpZIIaZ759p0Zfs4EPqw= 2023-10-18T20:17:41.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Wzw4vVoKpZIIaZ759p0Zfs4EPqw=', 'Content-Length': '0'} 2023-10-18T20:17:41.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000381e72904d6dcc21-0065303d65-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.899 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.899 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.900 DEBUG:boto:path=/bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.900 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.901 DEBUG:boto:Method: GET 2023-10-18T20:17:41.901 DEBUG:boto:Path: /bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.901 DEBUG:boto:Data: 2023-10-18T20:17:41.902 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.902 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:41.903 DEBUG:boto:Port: 8001 2023-10-18T20:17:41.903 DEBUG:boto:Params: {} 2023-10-18T20:17:41.904 DEBUG:boto:Token: None 2023-10-18T20:17:41.904 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.905 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb853581dee6d1b5-0065303d65-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.908 DEBUG:boto:b'bofrnf-48obj41000falseobj42023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.909 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:41.909 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:41.910 DEBUG:boto:path=/bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.910 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.911 DEBUG:boto:Method: GET 2023-10-18T20:17:41.911 DEBUG:boto:Path: /bofrnf-48/?prefix=obj4 2023-10-18T20:17:41.911 DEBUG:boto:Data: 2023-10-18T20:17:41.912 DEBUG:boto:Headers: {} 2023-10-18T20:17:41.912 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:41.913 DEBUG:boto:Port: 8000 2023-10-18T20:17:41.913 DEBUG:boto:Params: {} 2023-10-18T20:17:41.914 DEBUG:boto:Token: None 2023-10-18T20:17:41.914 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:41 GMT /bofrnf-48/ 2023-10-18T20:17:41.914 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0= 2023-10-18T20:17:41.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6iRf1WAS/D86dCPqTDY+GwEAB/0=', 'Content-Length': '0'} 2023-10-18T20:17:41.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038ed3678d77267aa-0065303d65-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:41.918 DEBUG:boto:b'bofrnf-48obj41000falseobj42023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:41.919 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-48 zones={a2, a1} 2023-10-18T20:17:41.919 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-49 2023-10-18T20:17:41.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:41.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:41.921 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:17:41.985 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:41.986 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:42.022 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:42.021+0000 7f95bc341780 1 bucket sync caught up with source: 2023-10-18T20:17:42.022 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.33.6, , 00000000001.27.6, , 00000000001.26.6, , , 00000000001.28.6, , , ] 2023-10-18T20:17:42.023 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.33.6, , 00000000001.27.6, , 00000000001.26.6, , , 00000000001.28.6, , , ] 2023-10-18T20:17:42.023 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:42.021+0000 7f95bc341780 0 bucket checkpoint complete 2023-10-18T20:17:42.035 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-49 2023-10-18T20:17:42.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:42.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:42.037 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:17:42.094 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:42.094 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:42.129 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:42.127+0000 7ff7b29f1780 1 bucket sync caught up with source: 2023-10-18T20:17:42.129 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.681.6, , 00000000001.398.6, , 00000000001.906.6, , , 00000000001.402.6, , , ] 2023-10-18T20:17:42.129 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.681.6, , 00000000001.398.6, , 00000000001.906.6, , , 00000000001.402.6, , , ] 2023-10-18T20:17:42.130 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:42.127+0000 7ff7b29f1780 0 bucket checkpoint complete 2023-10-18T20:17:42.142 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-49 zones={a2, a1} 2023-10-18T20:17:42.142 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.143 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.143 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.143 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:42.144 DEBUG:boto:Data: 2023-10-18T20:17:42.144 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.145 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.145 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.145 DEBUG:boto:Params: {} 2023-10-18T20:17:42.146 DEBUG:boto:Token: None 2023-10-18T20:17:42.146 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.147 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:07Ek4ocPPzYCu1he+OUosXx8omc= 2023-10-18T20:17:42.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:07Ek4ocPPzYCu1he+OUosXx8omc=', 'Content-Length': '0'} 2023-10-18T20:17:42.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090a8f22244c00566-0065303d66-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.150 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.151 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.151 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.151 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:42.152 DEBUG:boto:Data: 2023-10-18T20:17:42.152 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.152 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.153 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.153 DEBUG:boto:Params: {} 2023-10-18T20:17:42.154 DEBUG:boto:Token: None 2023-10-18T20:17:42.154 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.155 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:07Ek4ocPPzYCu1he+OUosXx8omc= 2023-10-18T20:17:42.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:07Ek4ocPPzYCu1he+OUosXx8omc=', 'Content-Length': '0'} 2023-10-18T20:17:42.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9a88ab129b45b8f-0065303d66-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.157 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:17:42.158 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.158 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.159 DEBUG:boto:path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.159 DEBUG:boto:auth_path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.159 DEBUG:boto:Method: GET 2023-10-18T20:17:42.160 DEBUG:boto:Path: /bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.160 DEBUG:boto:Data: 2023-10-18T20:17:42.160 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.161 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.161 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.162 DEBUG:boto:Params: {} 2023-10-18T20:17:42.162 DEBUG:boto:Token: None 2023-10-18T20:17:42.162 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/?versions 2023-10-18T20:17:42.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3xlWhxccRE89AIfQ4k25Lj8kxkY= 2023-10-18T20:17:42.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3xlWhxccRE89AIfQ4k25Lj8kxkY=', 'Content-Length': '0'} 2023-10-18T20:17:42.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd34bb1b6d009bd6-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.166 DEBUG:boto:b'bofrnf-49999falseobj1nulltrue2023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:17:42.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:17:42.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:17:42.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:17:42.168 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:17:42.169 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.169 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.170 DEBUG:boto:path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.170 DEBUG:boto:auth_path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.170 DEBUG:boto:Method: GET 2023-10-18T20:17:42.171 DEBUG:boto:Path: /bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.171 DEBUG:boto:Data: 2023-10-18T20:17:42.172 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.172 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.172 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.173 DEBUG:boto:Params: {} 2023-10-18T20:17:42.173 DEBUG:boto:Token: None 2023-10-18T20:17:42.174 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/?versions 2023-10-18T20:17:42.174 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3xlWhxccRE89AIfQ4k25Lj8kxkY= 2023-10-18T20:17:42.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3xlWhxccRE89AIfQ4k25Lj8kxkY=', 'Content-Length': '0'} 2023-10-18T20:17:42.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001415e035800e8637-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.177 DEBUG:boto:b'bofrnf-49999falseobj1nulltrue2023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:17:42.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:17:42.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:17:42.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:17:42.179 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.180 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.180 DEBUG:boto:path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.181 DEBUG:boto:auth_path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.181 DEBUG:boto:Method: GET 2023-10-18T20:17:42.181 DEBUG:boto:Path: /bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.182 DEBUG:boto:Data: 2023-10-18T20:17:42.182 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.183 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.183 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.184 DEBUG:boto:Params: {} 2023-10-18T20:17:42.185 DEBUG:boto:Token: None 2023-10-18T20:17:42.185 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/?versions 2023-10-18T20:17:42.186 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3xlWhxccRE89AIfQ4k25Lj8kxkY= 2023-10-18T20:17:42.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3xlWhxccRE89AIfQ4k25Lj8kxkY=', 'Content-Length': '0'} 2023-10-18T20:17:42.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006e00bf122e00f5d-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.189 DEBUG:boto:b'bofrnf-49999falseobj1nulltrue2023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.190 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.190 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.191 DEBUG:boto:path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.191 DEBUG:boto:auth_path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.191 DEBUG:boto:Method: GET 2023-10-18T20:17:42.192 DEBUG:boto:Path: /bofrnf-49/?versions&max-keys=999 2023-10-18T20:17:42.192 DEBUG:boto:Data: 2023-10-18T20:17:42.193 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.194 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.194 DEBUG:boto:Params: {} 2023-10-18T20:17:42.194 DEBUG:boto:Token: None 2023-10-18T20:17:42.195 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/?versions 2023-10-18T20:17:42.195 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3xlWhxccRE89AIfQ4k25Lj8kxkY= 2023-10-18T20:17:42.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3xlWhxccRE89AIfQ4k25Lj8kxkY=', 'Content-Length': '0'} 2023-10-18T20:17:42.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090c34842c66f8c66-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.198 DEBUG:boto:b'bofrnf-49999falseobj1nulltrue2023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:17:42.200 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.200 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.200 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.201 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.201 DEBUG:boto:Method: GET 2023-10-18T20:17:42.202 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.202 DEBUG:boto:Data: 2023-10-18T20:17:42.202 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.203 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.203 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.204 DEBUG:boto:Params: {} 2023-10-18T20:17:42.204 DEBUG:boto:Token: None 2023-10-18T20:17:42.205 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.205 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:at+U0Yqj9S18qNCfBUiJywOrBIw= 2023-10-18T20:17:42.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:at+U0Yqj9S18qNCfBUiJywOrBIw=', 'Content-Length': '0'} 2023-10-18T20:17:42.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000283126fc4528e55d-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.208 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.209 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.209 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.209 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.210 DEBUG:boto:Method: GET 2023-10-18T20:17:42.210 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.211 DEBUG:boto:Data: 2023-10-18T20:17:42.211 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.212 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.212 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.213 DEBUG:boto:Params: {} 2023-10-18T20:17:42.213 DEBUG:boto:Token: None 2023-10-18T20:17:42.214 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.214 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:at+U0Yqj9S18qNCfBUiJywOrBIw= 2023-10-18T20:17:42.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:at+U0Yqj9S18qNCfBUiJywOrBIw=', 'Content-Length': '0'} 2023-10-18T20:17:42.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066b9e240d08419c1-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.217 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.217 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.218 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.218 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.219 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.219 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.219 DEBUG:boto:Data: 2023-10-18T20:17:42.220 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.220 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.221 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.221 DEBUG:boto:Params: {} 2023-10-18T20:17:42.221 DEBUG:boto:Token: None 2023-10-18T20:17:42.222 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.222 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sYD036T7xZLUllT0kzXEqSmH75I= 2023-10-18T20:17:42.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sYD036T7xZLUllT0kzXEqSmH75I=', 'Content-Length': '0'} 2023-10-18T20:17:42.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca1048f13e0bc86a-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.225 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.225 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.226 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.226 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.227 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.227 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.227 DEBUG:boto:Data: 2023-10-18T20:17:42.228 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.228 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.229 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.229 DEBUG:boto:Params: {} 2023-10-18T20:17:42.230 DEBUG:boto:Token: None 2023-10-18T20:17:42.230 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sYD036T7xZLUllT0kzXEqSmH75I= 2023-10-18T20:17:42.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sYD036T7xZLUllT0kzXEqSmH75I=', 'Content-Length': '0'} 2023-10-18T20:17:42.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ca02c791d101b2b-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:17:42.234 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.234 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.235 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.235 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.236 DEBUG:boto:Method: GET 2023-10-18T20:17:42.236 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.237 DEBUG:boto:Data: 2023-10-18T20:17:42.237 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.237 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.238 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.238 DEBUG:boto:Params: {} 2023-10-18T20:17:42.239 DEBUG:boto:Token: None 2023-10-18T20:17:42.239 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:at+U0Yqj9S18qNCfBUiJywOrBIw= 2023-10-18T20:17:42.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:at+U0Yqj9S18qNCfBUiJywOrBIw=', 'Content-Length': '0'} 2023-10-18T20:17:42.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000349689163baadd3e-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.242 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.243 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.243 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.244 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.244 DEBUG:boto:Method: GET 2023-10-18T20:17:42.245 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.245 DEBUG:boto:Data: 2023-10-18T20:17:42.245 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.246 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.246 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.247 DEBUG:boto:Params: {} 2023-10-18T20:17:42.247 DEBUG:boto:Token: None 2023-10-18T20:17:42.248 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:17:42.248 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:at+U0Yqj9S18qNCfBUiJywOrBIw= 2023-10-18T20:17:42.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:at+U0Yqj9S18qNCfBUiJywOrBIw=', 'Content-Length': '0'} 2023-10-18T20:17:42.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076ab1e525dfcbd50-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.251 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.251 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.252 DEBUG:boto:path=/bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.252 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.253 DEBUG:boto:Method: GET 2023-10-18T20:17:42.253 DEBUG:boto:Path: /bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.253 DEBUG:boto:Data: 2023-10-18T20:17:42.254 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.254 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.255 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.255 DEBUG:boto:Params: {} 2023-10-18T20:17:42.255 DEBUG:boto:Token: None 2023-10-18T20:17:42.256 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.256 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed7f5bbbd2c57e17-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.259 DEBUG:boto:b'bofrnf-49obj11000falseobj12023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.260 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.261 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.261 DEBUG:boto:path=/bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.261 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.262 DEBUG:boto:Method: GET 2023-10-18T20:17:42.262 DEBUG:boto:Path: /bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.263 DEBUG:boto:Data: 2023-10-18T20:17:42.263 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.264 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.264 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.265 DEBUG:boto:Params: {} 2023-10-18T20:17:42.265 DEBUG:boto:Token: None 2023-10-18T20:17:42.265 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.266 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fca0fee76920579-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.314 DEBUG:boto:b'bofrnf-49obj11000falseobj12023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.315 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.316 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.316 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.316 DEBUG:boto:Path: /bofrnf-49/obj1 2023-10-18T20:17:42.317 DEBUG:boto:Data: 2023-10-18T20:17:42.317 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.318 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.318 DEBUG:boto:Params: {} 2023-10-18T20:17:42.319 DEBUG:boto:Token: None 2023-10-18T20:17:42.319 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1 2023-10-18T20:17:42.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t7Gh3JCPjDcEhf+xRDtLmS1K/l0= 2023-10-18T20:17:42.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t7Gh3JCPjDcEhf+xRDtLmS1K/l0=', 'Content-Length': '0'} 2023-10-18T20:17:42.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c297d82a7385523a-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.322 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.323 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.323 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.324 DEBUG:boto:Path: /bofrnf-49/obj1 2023-10-18T20:17:42.324 DEBUG:boto:Data: 2023-10-18T20:17:42.324 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.325 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.325 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.326 DEBUG:boto:Params: {} 2023-10-18T20:17:42.326 DEBUG:boto:Token: None 2023-10-18T20:17:42.327 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1 2023-10-18T20:17:42.327 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t7Gh3JCPjDcEhf+xRDtLmS1K/l0= 2023-10-18T20:17:42.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t7Gh3JCPjDcEhf+xRDtLmS1K/l0=', 'Content-Length': '0'} 2023-10-18T20:17:42.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f9833fd049a21d6-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:17:42.330 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.330 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.331 DEBUG:boto:Method: GET 2023-10-18T20:17:42.331 DEBUG:boto:Path: /bofrnf-49/obj1 2023-10-18T20:17:42.332 DEBUG:boto:Data: 2023-10-18T20:17:42.332 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.332 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.333 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.333 DEBUG:boto:Params: {} 2023-10-18T20:17:42.334 DEBUG:boto:Token: None 2023-10-18T20:17:42.334 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1 2023-10-18T20:17:42.334 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ExhyCVWouRgMGBzbKpmKImH+SpI= 2023-10-18T20:17:42.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ExhyCVWouRgMGBzbKpmKImH+SpI=', 'Content-Length': '0'} 2023-10-18T20:17:42.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2a26e8d618973e3-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.337 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:17:42.338 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:17:42.338 DEBUG:boto:Method: GET 2023-10-18T20:17:42.339 DEBUG:boto:Path: /bofrnf-49/obj1 2023-10-18T20:17:42.339 DEBUG:boto:Data: 2023-10-18T20:17:42.339 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.340 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.340 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.341 DEBUG:boto:Params: {} 2023-10-18T20:17:42.341 DEBUG:boto:Token: None 2023-10-18T20:17:42.341 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj1 2023-10-18T20:17:42.342 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ExhyCVWouRgMGBzbKpmKImH+SpI= 2023-10-18T20:17:42.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ExhyCVWouRgMGBzbKpmKImH+SpI=', 'Content-Length': '0'} 2023-10-18T20:17:42.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000181abfc0a7db8b2a-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.345 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.345 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.345 DEBUG:boto:path=/bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.346 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.346 DEBUG:boto:Method: GET 2023-10-18T20:17:42.347 DEBUG:boto:Path: /bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.347 DEBUG:boto:Data: 2023-10-18T20:17:42.348 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.348 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.348 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.349 DEBUG:boto:Params: {} 2023-10-18T20:17:42.349 DEBUG:boto:Token: None 2023-10-18T20:17:42.350 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.350 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bdce9e0998b22dd-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.354 DEBUG:boto:b'bofrnf-49obj11000falseobj12023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.354 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.355 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.355 DEBUG:boto:path=/bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.355 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.356 DEBUG:boto:Method: GET 2023-10-18T20:17:42.356 DEBUG:boto:Path: /bofrnf-49/?prefix=obj1 2023-10-18T20:17:42.357 DEBUG:boto:Data: 2023-10-18T20:17:42.357 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.358 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.359 DEBUG:boto:Params: {} 2023-10-18T20:17:42.359 DEBUG:boto:Token: None 2023-10-18T20:17:42.359 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf73c79b369f6915-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.363 DEBUG:boto:b'bofrnf-49obj11000falseobj12023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:17:42.364 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.364 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.365 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.365 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.366 DEBUG:boto:Method: GET 2023-10-18T20:17:42.366 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.366 DEBUG:boto:Data: 2023-10-18T20:17:42.367 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.368 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.368 DEBUG:boto:Params: {} 2023-10-18T20:17:42.368 DEBUG:boto:Token: None 2023-10-18T20:17:42.369 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8eRV6Ex3xJEYI138LqRdfTxRAWc= 2023-10-18T20:17:42.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8eRV6Ex3xJEYI138LqRdfTxRAWc=', 'Content-Length': '0'} 2023-10-18T20:17:42.371 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056cbcad95fb74eff-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.372 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.372 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.373 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.373 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.374 DEBUG:boto:Method: GET 2023-10-18T20:17:42.374 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.374 DEBUG:boto:Data: 2023-10-18T20:17:42.375 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.375 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.376 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.376 DEBUG:boto:Params: {} 2023-10-18T20:17:42.376 DEBUG:boto:Token: None 2023-10-18T20:17:42.377 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.377 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8eRV6Ex3xJEYI138LqRdfTxRAWc= 2023-10-18T20:17:42.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8eRV6Ex3xJEYI138LqRdfTxRAWc=', 'Content-Length': '0'} 2023-10-18T20:17:42.380 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d0557acbf694ed6-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.380 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.381 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.381 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.381 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.382 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.382 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.383 DEBUG:boto:Data: 2023-10-18T20:17:42.383 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.383 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.384 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.384 DEBUG:boto:Params: {} 2023-10-18T20:17:42.384 DEBUG:boto:Token: None 2023-10-18T20:17:42.385 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.385 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NOhkJOLjY/sXRZ39syo9SS16gM4= 2023-10-18T20:17:42.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NOhkJOLjY/sXRZ39syo9SS16gM4=', 'Content-Length': '0'} 2023-10-18T20:17:42.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d32c8f7e58529f2a-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.388 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.388 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.388 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.389 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.389 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.389 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.390 DEBUG:boto:Data: 2023-10-18T20:17:42.390 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.391 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.391 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.392 DEBUG:boto:Params: {} 2023-10-18T20:17:42.392 DEBUG:boto:Token: None 2023-10-18T20:17:42.393 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.393 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NOhkJOLjY/sXRZ39syo9SS16gM4= 2023-10-18T20:17:42.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NOhkJOLjY/sXRZ39syo9SS16gM4=', 'Content-Length': '0'} 2023-10-18T20:17:42.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e313209fed3cc7a8-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:17:42.396 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.396 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.397 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.397 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.397 DEBUG:boto:Method: GET 2023-10-18T20:17:42.398 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.398 DEBUG:boto:Data: 2023-10-18T20:17:42.399 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.400 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.400 DEBUG:boto:Params: {} 2023-10-18T20:17:42.400 DEBUG:boto:Token: None 2023-10-18T20:17:42.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8eRV6Ex3xJEYI138LqRdfTxRAWc= 2023-10-18T20:17:42.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8eRV6Ex3xJEYI138LqRdfTxRAWc=', 'Content-Length': '0'} 2023-10-18T20:17:42.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d7d2578905c2d10-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.404 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.405 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.405 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.406 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.406 DEBUG:boto:Method: GET 2023-10-18T20:17:42.406 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.407 DEBUG:boto:Data: 2023-10-18T20:17:42.407 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.408 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.409 DEBUG:boto:Params: {} 2023-10-18T20:17:42.409 DEBUG:boto:Token: None 2023-10-18T20:17:42.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:17:42.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8eRV6Ex3xJEYI138LqRdfTxRAWc= 2023-10-18T20:17:42.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8eRV6Ex3xJEYI138LqRdfTxRAWc=', 'Content-Length': '0'} 2023-10-18T20:17:42.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4bd35981122131c-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.458 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.459 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.459 DEBUG:boto:path=/bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.460 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.460 DEBUG:boto:Method: GET 2023-10-18T20:17:42.461 DEBUG:boto:Path: /bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.461 DEBUG:boto:Data: 2023-10-18T20:17:42.461 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.462 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.463 DEBUG:boto:Params: {} 2023-10-18T20:17:42.463 DEBUG:boto:Token: None 2023-10-18T20:17:42.464 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c9d57c2297834e-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.468 DEBUG:boto:b'bofrnf-49obj21000falseobj22023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.469 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.470 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.471 DEBUG:boto:path=/bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.472 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.473 DEBUG:boto:Method: GET 2023-10-18T20:17:42.474 DEBUG:boto:Path: /bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.474 DEBUG:boto:Data: 2023-10-18T20:17:42.475 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.476 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.477 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.478 DEBUG:boto:Params: {} 2023-10-18T20:17:42.479 DEBUG:boto:Token: None 2023-10-18T20:17:42.479 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.480 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017fb99d9ae756144-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.485 DEBUG:boto:b'bofrnf-49obj21000falseobj22023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.486 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.487 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.488 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.489 DEBUG:boto:Path: /bofrnf-49/obj2 2023-10-18T20:17:42.489 DEBUG:boto:Data: 2023-10-18T20:17:42.490 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.491 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.492 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.492 DEBUG:boto:Params: {} 2023-10-18T20:17:42.493 DEBUG:boto:Token: None 2023-10-18T20:17:42.494 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2 2023-10-18T20:17:42.495 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAwZ9+G9cah7LVSrR8xewM9VYYY= 2023-10-18T20:17:42.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAwZ9+G9cah7LVSrR8xewM9VYYY=', 'Content-Length': '0'} 2023-10-18T20:17:42.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bd04998ffdedefe-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.499 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.500 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.500 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.501 DEBUG:boto:Path: /bofrnf-49/obj2 2023-10-18T20:17:42.502 DEBUG:boto:Data: 2023-10-18T20:17:42.503 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.504 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.504 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.505 DEBUG:boto:Params: {} 2023-10-18T20:17:42.506 DEBUG:boto:Token: None 2023-10-18T20:17:42.507 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2 2023-10-18T20:17:42.508 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAwZ9+G9cah7LVSrR8xewM9VYYY= 2023-10-18T20:17:42.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAwZ9+G9cah7LVSrR8xewM9VYYY=', 'Content-Length': '0'} 2023-10-18T20:17:42.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9807f47e0496e47-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:17:42.512 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.513 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.513 DEBUG:boto:Method: GET 2023-10-18T20:17:42.514 DEBUG:boto:Path: /bofrnf-49/obj2 2023-10-18T20:17:42.515 DEBUG:boto:Data: 2023-10-18T20:17:42.516 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.517 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.518 DEBUG:boto:Params: {} 2023-10-18T20:17:42.519 DEBUG:boto:Token: None 2023-10-18T20:17:42.519 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2 2023-10-18T20:17:42.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:McW0WQNQg0evnCUptaetUDFl53w= 2023-10-18T20:17:42.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:McW0WQNQg0evnCUptaetUDFl53w=', 'Content-Length': '0'} 2023-10-18T20:17:42.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064c879bbb83fd251-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.524 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:17:42.524 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:17:42.525 DEBUG:boto:Method: GET 2023-10-18T20:17:42.526 DEBUG:boto:Path: /bofrnf-49/obj2 2023-10-18T20:17:42.527 DEBUG:boto:Data: 2023-10-18T20:17:42.528 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.528 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.529 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.530 DEBUG:boto:Params: {} 2023-10-18T20:17:42.531 DEBUG:boto:Token: None 2023-10-18T20:17:42.532 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj2 2023-10-18T20:17:42.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:McW0WQNQg0evnCUptaetUDFl53w= 2023-10-18T20:17:42.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:McW0WQNQg0evnCUptaetUDFl53w=', 'Content-Length': '0'} 2023-10-18T20:17:42.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077e3421e72887b88-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.536 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.536 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.536 DEBUG:boto:path=/bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.537 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.537 DEBUG:boto:Method: GET 2023-10-18T20:17:42.538 DEBUG:boto:Path: /bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.538 DEBUG:boto:Data: 2023-10-18T20:17:42.538 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.539 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.539 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.540 DEBUG:boto:Params: {} 2023-10-18T20:17:42.540 DEBUG:boto:Token: None 2023-10-18T20:17:42.540 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.541 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000483cd15700b558d7-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.590 DEBUG:boto:b'bofrnf-49obj21000falseobj22023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.591 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.592 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.592 DEBUG:boto:path=/bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.593 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.593 DEBUG:boto:Method: GET 2023-10-18T20:17:42.593 DEBUG:boto:Path: /bofrnf-49/?prefix=obj2 2023-10-18T20:17:42.594 DEBUG:boto:Data: 2023-10-18T20:17:42.594 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.595 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.595 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.596 DEBUG:boto:Params: {} 2023-10-18T20:17:42.596 DEBUG:boto:Token: None 2023-10-18T20:17:42.597 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.597 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005758d75235ded58a-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.600 DEBUG:boto:b'bofrnf-49obj21000falseobj22023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:17:42.601 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.602 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.602 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.603 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.603 DEBUG:boto:Method: GET 2023-10-18T20:17:42.604 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.604 DEBUG:boto:Data: 2023-10-18T20:17:42.605 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.605 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.605 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.606 DEBUG:boto:Params: {} 2023-10-18T20:17:42.606 DEBUG:boto:Token: None 2023-10-18T20:17:42.607 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.607 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DpZmkIEYrTqRj+a80O5Fd9zdCU0= 2023-10-18T20:17:42.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DpZmkIEYrTqRj+a80O5Fd9zdCU0=', 'Content-Length': '0'} 2023-10-18T20:17:42.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac8582df3092af9c-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.610 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.610 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.611 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.611 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.612 DEBUG:boto:Method: GET 2023-10-18T20:17:42.612 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.612 DEBUG:boto:Data: 2023-10-18T20:17:42.613 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.613 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.614 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.614 DEBUG:boto:Params: {} 2023-10-18T20:17:42.614 DEBUG:boto:Token: None 2023-10-18T20:17:42.615 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DpZmkIEYrTqRj+a80O5Fd9zdCU0= 2023-10-18T20:17:42.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DpZmkIEYrTqRj+a80O5Fd9zdCU0=', 'Content-Length': '0'} 2023-10-18T20:17:42.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002854e10a97b21933-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.618 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.618 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.619 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.619 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.620 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.620 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.620 DEBUG:boto:Data: 2023-10-18T20:17:42.621 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.621 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.621 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.622 DEBUG:boto:Params: {} 2023-10-18T20:17:42.622 DEBUG:boto:Token: None 2023-10-18T20:17:42.623 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.623 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cr3T7Tts4fRXOC7+OFTcyVDIGLg= 2023-10-18T20:17:42.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cr3T7Tts4fRXOC7+OFTcyVDIGLg=', 'Content-Length': '0'} 2023-10-18T20:17:42.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000687793886311325b-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.626 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.626 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.627 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.627 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.627 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.628 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.628 DEBUG:boto:Data: 2023-10-18T20:17:42.629 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.629 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.629 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.630 DEBUG:boto:Params: {} 2023-10-18T20:17:42.630 DEBUG:boto:Token: None 2023-10-18T20:17:42.631 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.631 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cr3T7Tts4fRXOC7+OFTcyVDIGLg= 2023-10-18T20:17:42.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cr3T7Tts4fRXOC7+OFTcyVDIGLg=', 'Content-Length': '0'} 2023-10-18T20:17:42.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007360527da40fcf5d-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:17:42.635 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.635 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.635 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.636 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.636 DEBUG:boto:Method: GET 2023-10-18T20:17:42.637 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.637 DEBUG:boto:Data: 2023-10-18T20:17:42.637 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.638 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.638 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.639 DEBUG:boto:Params: {} 2023-10-18T20:17:42.639 DEBUG:boto:Token: None 2023-10-18T20:17:42.640 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.640 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DpZmkIEYrTqRj+a80O5Fd9zdCU0= 2023-10-18T20:17:42.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DpZmkIEYrTqRj+a80O5Fd9zdCU0=', 'Content-Length': '0'} 2023-10-18T20:17:42.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000347c1ee52fb117f9-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.643 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.643 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.644 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.644 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.645 DEBUG:boto:Method: GET 2023-10-18T20:17:42.645 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.645 DEBUG:boto:Data: 2023-10-18T20:17:42.646 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.646 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.647 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.647 DEBUG:boto:Params: {} 2023-10-18T20:17:42.648 DEBUG:boto:Token: None 2023-10-18T20:17:42.648 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:17:42.649 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DpZmkIEYrTqRj+a80O5Fd9zdCU0= 2023-10-18T20:17:42.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DpZmkIEYrTqRj+a80O5Fd9zdCU0=', 'Content-Length': '0'} 2023-10-18T20:17:42.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061e71afa8f9da893-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.652 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.652 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.653 DEBUG:boto:path=/bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.653 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.654 DEBUG:boto:Method: GET 2023-10-18T20:17:42.654 DEBUG:boto:Path: /bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.654 DEBUG:boto:Data: 2023-10-18T20:17:42.655 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.655 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.656 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.656 DEBUG:boto:Params: {} 2023-10-18T20:17:42.657 DEBUG:boto:Token: None 2023-10-18T20:17:42.657 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.658 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000939c40031fee6cd1-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.661 DEBUG:boto:b'bofrnf-49obj31000falseobj32023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.662 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.662 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.663 DEBUG:boto:path=/bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.663 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.663 DEBUG:boto:Method: GET 2023-10-18T20:17:42.664 DEBUG:boto:Path: /bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.664 DEBUG:boto:Data: 2023-10-18T20:17:42.665 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.665 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.666 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.666 DEBUG:boto:Params: {} 2023-10-18T20:17:42.667 DEBUG:boto:Token: None 2023-10-18T20:17:42.667 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029489a3495e61e51-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.718 DEBUG:boto:b'bofrnf-49obj31000falseobj32023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.719 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.719 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.720 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.720 DEBUG:boto:Path: /bofrnf-49/obj3 2023-10-18T20:17:42.721 DEBUG:boto:Data: 2023-10-18T20:17:42.721 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.721 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.722 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.722 DEBUG:boto:Params: {} 2023-10-18T20:17:42.723 DEBUG:boto:Token: None 2023-10-18T20:17:42.723 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3 2023-10-18T20:17:42.724 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GP/jN+CaeC5yfKAA6BH7WEeRmNI= 2023-10-18T20:17:42.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GP/jN+CaeC5yfKAA6BH7WEeRmNI=', 'Content-Length': '0'} 2023-10-18T20:17:42.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a936aecde55f57b-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.727 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.727 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.728 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.728 DEBUG:boto:Path: /bofrnf-49/obj3 2023-10-18T20:17:42.728 DEBUG:boto:Data: 2023-10-18T20:17:42.729 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.729 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.730 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.730 DEBUG:boto:Params: {} 2023-10-18T20:17:42.730 DEBUG:boto:Token: None 2023-10-18T20:17:42.731 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3 2023-10-18T20:17:42.731 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GP/jN+CaeC5yfKAA6BH7WEeRmNI= 2023-10-18T20:17:42.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GP/jN+CaeC5yfKAA6BH7WEeRmNI=', 'Content-Length': '0'} 2023-10-18T20:17:42.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cdf637fc87eb945-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:17:42.735 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.735 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.735 DEBUG:boto:Method: GET 2023-10-18T20:17:42.736 DEBUG:boto:Path: /bofrnf-49/obj3 2023-10-18T20:17:42.736 DEBUG:boto:Data: 2023-10-18T20:17:42.736 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.737 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.737 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.738 DEBUG:boto:Params: {} 2023-10-18T20:17:42.738 DEBUG:boto:Token: None 2023-10-18T20:17:42.739 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3 2023-10-18T20:17:42.739 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1+tg2JuYzTa8denDJRUs5Hbuuek= 2023-10-18T20:17:42.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1+tg2JuYzTa8denDJRUs5Hbuuek=', 'Content-Length': '0'} 2023-10-18T20:17:42.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b347e5359d15670c-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.742 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:17:42.743 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:17:42.743 DEBUG:boto:Method: GET 2023-10-18T20:17:42.743 DEBUG:boto:Path: /bofrnf-49/obj3 2023-10-18T20:17:42.744 DEBUG:boto:Data: 2023-10-18T20:17:42.744 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.745 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.745 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.745 DEBUG:boto:Params: {} 2023-10-18T20:17:42.746 DEBUG:boto:Token: None 2023-10-18T20:17:42.746 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj3 2023-10-18T20:17:42.747 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1+tg2JuYzTa8denDJRUs5Hbuuek= 2023-10-18T20:17:42.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1+tg2JuYzTa8denDJRUs5Hbuuek=', 'Content-Length': '0'} 2023-10-18T20:17:42.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ee591de68026dc0-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.750 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.750 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.750 DEBUG:boto:path=/bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.751 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.751 DEBUG:boto:Method: GET 2023-10-18T20:17:42.752 DEBUG:boto:Path: /bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.752 DEBUG:boto:Data: 2023-10-18T20:17:42.753 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.753 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.754 DEBUG:boto:Params: {} 2023-10-18T20:17:42.754 DEBUG:boto:Token: None 2023-10-18T20:17:42.755 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1371213a82e9b39-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.758 DEBUG:boto:b'bofrnf-49obj31000falseobj32023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.759 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.760 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.760 DEBUG:boto:path=/bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.760 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.761 DEBUG:boto:Method: GET 2023-10-18T20:17:42.761 DEBUG:boto:Path: /bofrnf-49/?prefix=obj3 2023-10-18T20:17:42.762 DEBUG:boto:Data: 2023-10-18T20:17:42.762 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.763 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.763 DEBUG:boto:Params: {} 2023-10-18T20:17:42.764 DEBUG:boto:Token: None 2023-10-18T20:17:42.764 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.765 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a00e61d1a05105b-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.768 DEBUG:boto:b'bofrnf-49obj31000falseobj32023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:17:42.768 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.769 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.769 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.770 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.770 DEBUG:boto:Method: GET 2023-10-18T20:17:42.770 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.771 DEBUG:boto:Data: 2023-10-18T20:17:42.771 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.772 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.772 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.772 DEBUG:boto:Params: {} 2023-10-18T20:17:42.773 DEBUG:boto:Token: None 2023-10-18T20:17:42.773 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rB3/vWx4D6k2Z3lK0J1M5pc10Qc= 2023-10-18T20:17:42.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rB3/vWx4D6k2Z3lK0J1M5pc10Qc=', 'Content-Length': '0'} 2023-10-18T20:17:42.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bd2634ab3a17976-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.818 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.819 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.819 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.820 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.820 DEBUG:boto:Method: GET 2023-10-18T20:17:42.821 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.821 DEBUG:boto:Data: 2023-10-18T20:17:42.821 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.822 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.823 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.823 DEBUG:boto:Params: {} 2023-10-18T20:17:42.824 DEBUG:boto:Token: None 2023-10-18T20:17:42.824 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.825 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rB3/vWx4D6k2Z3lK0J1M5pc10Qc= 2023-10-18T20:17:42.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rB3/vWx4D6k2Z3lK0J1M5pc10Qc=', 'Content-Length': '0'} 2023-10-18T20:17:42.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030eb175370e52e1c-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.828 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.829 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.829 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.829 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.830 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.830 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.831 DEBUG:boto:Data: 2023-10-18T20:17:42.831 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.831 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.832 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.832 DEBUG:boto:Params: {} 2023-10-18T20:17:42.832 DEBUG:boto:Token: None 2023-10-18T20:17:42.833 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.833 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mr/0+iuJOtmjml6Qno2HwHKM5Ls= 2023-10-18T20:17:42.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mr/0+iuJOtmjml6Qno2HwHKM5Ls=', 'Content-Length': '0'} 2023-10-18T20:17:42.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e2aced48036d71b-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.836 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.837 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.837 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.837 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.838 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.838 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.839 DEBUG:boto:Data: 2023-10-18T20:17:42.839 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.839 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.840 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.840 DEBUG:boto:Params: {} 2023-10-18T20:17:42.841 DEBUG:boto:Token: None 2023-10-18T20:17:42.841 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.842 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mr/0+iuJOtmjml6Qno2HwHKM5Ls= 2023-10-18T20:17:42.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mr/0+iuJOtmjml6Qno2HwHKM5Ls=', 'Content-Length': '0'} 2023-10-18T20:17:42.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fe8ebc509faaa62-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:17:42.845 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.845 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.846 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.846 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.846 DEBUG:boto:Method: GET 2023-10-18T20:17:42.847 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.847 DEBUG:boto:Data: 2023-10-18T20:17:42.848 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.848 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.849 DEBUG:boto:Params: {} 2023-10-18T20:17:42.849 DEBUG:boto:Token: None 2023-10-18T20:17:42.850 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.850 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rB3/vWx4D6k2Z3lK0J1M5pc10Qc= 2023-10-18T20:17:42.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rB3/vWx4D6k2Z3lK0J1M5pc10Qc=', 'Content-Length': '0'} 2023-10-18T20:17:42.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047acf8a13b3b91a7-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.853 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.854 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.854 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.855 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.855 DEBUG:boto:Method: GET 2023-10-18T20:17:42.855 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.856 DEBUG:boto:Data: 2023-10-18T20:17:42.856 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.856 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.857 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.857 DEBUG:boto:Params: {} 2023-10-18T20:17:42.858 DEBUG:boto:Token: None 2023-10-18T20:17:42.858 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:17:42.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rB3/vWx4D6k2Z3lK0J1M5pc10Qc= 2023-10-18T20:17:42.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rB3/vWx4D6k2Z3lK0J1M5pc10Qc=', 'Content-Length': '0'} 2023-10-18T20:17:42.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a02dcfa03e7f95bd-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.862 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.863 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.863 DEBUG:boto:path=/bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.864 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.865 DEBUG:boto:Method: GET 2023-10-18T20:17:42.865 DEBUG:boto:Path: /bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.866 DEBUG:boto:Data: 2023-10-18T20:17:42.866 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.867 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.867 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.868 DEBUG:boto:Params: {} 2023-10-18T20:17:42.868 DEBUG:boto:Token: None 2023-10-18T20:17:42.869 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.869 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d81949fee8304edc-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.872 DEBUG:boto:b'bofrnf-49obj41000falseobj42023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.873 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.873 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.874 DEBUG:boto:path=/bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.874 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.875 DEBUG:boto:Method: GET 2023-10-18T20:17:42.875 DEBUG:boto:Path: /bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.876 DEBUG:boto:Data: 2023-10-18T20:17:42.876 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.876 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.877 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.877 DEBUG:boto:Params: {} 2023-10-18T20:17:42.878 DEBUG:boto:Token: None 2023-10-18T20:17:42.878 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.878 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd52232f81c2edbc-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.882 DEBUG:boto:b'bofrnf-49obj41000falseobj42023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.882 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.883 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.883 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.883 DEBUG:boto:Path: /bofrnf-49/obj4 2023-10-18T20:17:42.884 DEBUG:boto:Data: 2023-10-18T20:17:42.885 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.885 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.886 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.886 DEBUG:boto:Params: {} 2023-10-18T20:17:42.887 DEBUG:boto:Token: None 2023-10-18T20:17:42.888 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4 2023-10-18T20:17:42.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UsBZev8YqYH20MtVKkxxiI/x3K0= 2023-10-18T20:17:42.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UsBZev8YqYH20MtVKkxxiI/x3K0=', 'Content-Length': '0'} 2023-10-18T20:17:42.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a817ffa9dec65d1a-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.892 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.893 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.894 DEBUG:boto:Method: HEAD 2023-10-18T20:17:42.894 DEBUG:boto:Path: /bofrnf-49/obj4 2023-10-18T20:17:42.894 DEBUG:boto:Data: 2023-10-18T20:17:42.895 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.895 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.895 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.896 DEBUG:boto:Params: {} 2023-10-18T20:17:42.896 DEBUG:boto:Token: None 2023-10-18T20:17:42.897 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4 2023-10-18T20:17:42.897 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UsBZev8YqYH20MtVKkxxiI/x3K0= 2023-10-18T20:17:42.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UsBZev8YqYH20MtVKkxxiI/x3K0=', 'Content-Length': '0'} 2023-10-18T20:17:42.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b883d440cbcd5ba-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:17:42.901 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.901 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.901 DEBUG:boto:Method: GET 2023-10-18T20:17:42.902 DEBUG:boto:Path: /bofrnf-49/obj4 2023-10-18T20:17:42.902 DEBUG:boto:Data: 2023-10-18T20:17:42.903 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.903 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.903 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.904 DEBUG:boto:Params: {} 2023-10-18T20:17:42.904 DEBUG:boto:Token: None 2023-10-18T20:17:42.905 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4 2023-10-18T20:17:42.905 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iEAYA7b+wNcVw936imgnXZ88ITo= 2023-10-18T20:17:42.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iEAYA7b+wNcVw936imgnXZ88ITo=', 'Content-Length': '0'} 2023-10-18T20:17:42.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c37fae92d941088-0065303d66-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.908 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:17:42.909 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:17:42.909 DEBUG:boto:Method: GET 2023-10-18T20:17:42.909 DEBUG:boto:Path: /bofrnf-49/obj4 2023-10-18T20:17:42.910 DEBUG:boto:Data: 2023-10-18T20:17:42.910 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.911 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.911 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.911 DEBUG:boto:Params: {} 2023-10-18T20:17:42.912 DEBUG:boto:Token: None 2023-10-18T20:17:42.912 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/obj4 2023-10-18T20:17:42.913 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iEAYA7b+wNcVw936imgnXZ88ITo= 2023-10-18T20:17:42.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iEAYA7b+wNcVw936imgnXZ88ITo=', 'Content-Length': '0'} 2023-10-18T20:17:42.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001daa208c21d5ae31-0065303d66-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.916 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.916 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.917 DEBUG:boto:path=/bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.917 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.917 DEBUG:boto:Method: GET 2023-10-18T20:17:42.918 DEBUG:boto:Path: /bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.918 DEBUG:boto:Data: 2023-10-18T20:17:42.919 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:42.919 DEBUG:boto:Port: 8001 2023-10-18T20:17:42.920 DEBUG:boto:Params: {} 2023-10-18T20:17:42.920 DEBUG:boto:Token: None 2023-10-18T20:17:42.920 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.921 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a9fa2e702830162-0065303d66-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.924 DEBUG:boto:b'bofrnf-49obj41000falseobj42023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.925 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:42.925 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:42.925 DEBUG:boto:path=/bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.926 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.926 DEBUG:boto:Method: GET 2023-10-18T20:17:42.927 DEBUG:boto:Path: /bofrnf-49/?prefix=obj4 2023-10-18T20:17:42.927 DEBUG:boto:Data: 2023-10-18T20:17:42.927 DEBUG:boto:Headers: {} 2023-10-18T20:17:42.928 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:42.928 DEBUG:boto:Port: 8000 2023-10-18T20:17:42.929 DEBUG:boto:Params: {} 2023-10-18T20:17:42.929 DEBUG:boto:Token: None 2023-10-18T20:17:42.930 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:17:42 GMT /bofrnf-49/ 2023-10-18T20:17:42.930 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg= 2023-10-18T20:17:42.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U8zb3ubbgE3NuVnoIgauEFWTAqg=', 'Content-Length': '0'} 2023-10-18T20:17:42.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010a466e5f02dc334-0065303d66-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:42.933 DEBUG:boto:b'bofrnf-49obj41000falseobj42023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:17:42.934 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-49 zones={a2, a1} 2023-10-18T20:17:42.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bofrnf-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:17:42.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bofrnf-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:17:42.935 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bofrnf-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:17:42.985 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:42.985 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:43.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bofrnf-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:17:43.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bofrnf-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:17:43.049 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bofrnf-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:17:43.099 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:43.099 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:43.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:43.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:43.161 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:17:43.209 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:43.210 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:17:43.249 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:17:43.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:43.250 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660263.147239_767.1", 2023-10-18T20:17:43.250 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:17:43.147239Z" 2023-10-18T20:17:43.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:43.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:43.251 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:17:43.251 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:17:43.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:43.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:43.252 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:43.252 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:17:43.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:43.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:43.252 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:43.253 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:17:43.253 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:43.253 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:17:43.254 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660263.147239_767.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:43.254 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:17:43.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:43.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:43.256 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:17:43.305 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:43.305 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:43.332 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:43.331+0000 7f58e9a61780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:43.333 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:43.332+0000 7f58e9a61780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:43.345 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:17:43.345 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:17:43.346 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:17:43.346 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:17:43.346 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:17:43.346 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:17:43.346 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:43.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:43.347 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:17:43.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:43.347 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:17:43.347 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:43.348 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:43.348 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660220.203281_747.1", 2023-10-18T20:17:43.348 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:43.348 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:17:43.348 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:43.349 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:17:00.203281Z", 2023-10-18T20:17:43.349 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:43.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:43.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:43.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:43.350 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:17:43.350 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:43.350 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:43.350 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:17:43.350 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:43.351 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:43.351 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:43.351 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:17:43.351 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:43.352 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:43.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:43.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:43.352 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:17:43.352 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:43.353 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:43.353 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:43.353 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:43.353 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:43.353 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:43.354 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:43.354 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:43.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:43.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:43.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:43.355 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:17:43.355 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:43.355 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:43.355 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:43.356 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:43.356 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:43.356 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:43.356 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:43.356 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:43.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:43.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:43.357 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:17:43.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:43.358 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:17:43.358 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:17:43.358 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:17:43.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:43.358 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:17:43Z" 2023-10-18T20:17:43.359 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:17:43.359 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660220.203281_747.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:17:00.203281Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:17:43Z" } 2023-10-18T20:17:43.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660220.203281_747.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:17:00.203281Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:17:43.360 DEBUG:rgw_multi.tests:log_status={0: '1_1697660263.147239_767.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:43.361 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660220.203281_747.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:43.361 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697660263.147239_767.1 target=1_1697660220.203281_747.1 2023-10-18T20:17:48.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:48.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:48.364 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:17:48.421 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:48.421 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:48.454 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:48.453+0000 7f2d1486c780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:48.455 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:48.454+0000 7f2d1486c780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:48.469 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:17:48.469 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:17:48.469 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:17:48.469 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:17:48.470 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:17:48.470 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:17:48.470 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:48.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:48.471 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:17:48.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:48.471 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:17:48.471 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:48.471 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:48.472 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660220.203281_747.1", 2023-10-18T20:17:48.472 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:48.472 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:17:48.472 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:48.473 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:17:00.203281Z", 2023-10-18T20:17:48.473 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:48.473 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:48.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:48.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:48.474 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:17:48.474 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:48.474 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:48.474 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:17:48.475 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:48.475 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:48.475 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:48.475 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:17:48.475 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:48.476 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:48.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:48.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:48.476 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:17:48.477 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:48.477 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:48.477 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:48.477 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:48.478 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:48.478 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:48.478 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:48.479 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:48.479 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:48.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:48.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:48.480 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:17:48.480 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:48.480 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:48.480 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:48.480 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:48.481 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:48.481 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:48.481 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:48.481 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:48.482 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:48.482 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:48.482 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:17:48.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:48.482 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:17:48.483 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:17:48.483 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:17:48.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:48.483 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:17:48Z" 2023-10-18T20:17:48.484 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:17:48.485 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660220.203281_747.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:17:00.203281Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:17:48Z" } 2023-10-18T20:17:48.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660220.203281_747.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:17:00.203281Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:17:48.486 DEBUG:rgw_multi.tests:log_status={0: '1_1697660263.147239_767.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:48.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660220.203281_747.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:48.487 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697660263.147239_767.1 target=1_1697660220.203281_747.1 2023-10-18T20:17:53.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:53.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:17:53.489 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:17:53.547 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:53.548 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:53.579 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:53.577+0000 7f8526e01780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:53.580 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:53.578+0000 7f8526e01780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:17:53.592 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:17:53.593 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:17:53.593 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:17:53.593 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:17:53.593 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:17:53.594 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:17:53.594 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:53.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:53.594 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:17:53.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:53.595 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:17:53.595 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:53.595 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:53.595 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660263.147239_767.1", 2023-10-18T20:17:53.596 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:53.596 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:17:53.596 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:53.596 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:17:43.147239Z", 2023-10-18T20:17:53.596 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:53.597 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:53.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:53.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:53.597 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:17:53.597 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:53.598 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:53.598 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:17:53.598 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:53.598 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:53.598 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:53.599 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:17:53.599 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:53.599 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:53.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:53.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:53.600 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:17:53.600 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:53.600 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:53.600 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:53.600 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:53.601 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:53.601 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:53.601 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:53.601 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:53.601 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:53.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:53.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:17:53.602 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:17:53.602 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:17:53.603 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:17:53.603 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:17:53.603 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:17:53.603 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:17:53.603 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:17:53.604 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:17:53.604 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:17:53.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:53.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:17:53.604 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:17:53.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:53.605 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:17:53.605 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:17:53.605 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:17:53.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:17:53.606 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:17:53Z" 2023-10-18T20:17:53.606 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:17:53.607 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660263.147239_767.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:17:43.147239Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:17:53Z" } 2023-10-18T20:17:53.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660263.147239_767.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:17:43.147239Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:17:53.608 DEBUG:rgw_multi.tests:log_status={0: '1_1697660263.147239_767.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:53.608 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660263.147239_767.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:17:53.609 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:17:53.609 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:53.609 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:53.610 DEBUG:boto:Method: HEAD 2023-10-18T20:17:53.610 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:53.611 DEBUG:boto:Data: 2023-10-18T20:17:53.611 DEBUG:boto:Headers: {} 2023-10-18T20:17:53.611 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:53.612 DEBUG:boto:Port: 8000 2023-10-18T20:17:53.612 DEBUG:boto:Params: {} 2023-10-18T20:17:53.613 DEBUG:boto:Token: None 2023-10-18T20:17:53.613 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-48/ 2023-10-18T20:17:53.614 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1YsZZWq2UlGOVF/elcmvyGH3Zmk= 2023-10-18T20:17:53.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1YsZZWq2UlGOVF/elcmvyGH3Zmk=', 'Content-Length': '0'} 2023-10-18T20:17:53.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004dc16cdf74a12d75-0065303d71-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.617 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:17:53.618 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:17:53.618 DEBUG:boto:Method: PUT 2023-10-18T20:17:53.618 DEBUG:boto:Path: /bofrnf-48/obj5 2023-10-18T20:17:53.619 DEBUG:boto:Data: 2023-10-18T20:17:53.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:53.620 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:53.620 DEBUG:boto:Port: 8000 2023-10-18T20:17:53.620 DEBUG:boto:Params: {} 2023-10-18T20:17:53.621 DEBUG:boto:Token: None 2023-10-18T20:17:53.621 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-48/obj5 2023-10-18T20:17:53.622 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Y136KZnTGF4MgNICYs1N3MxfyrA= 2023-10-18T20:17:53.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Y136KZnTGF4MgNICYs1N3MxfyrA='} 2023-10-18T20:17:53.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ef1871a09f3fe8d-0065303d71-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.628 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:53.628 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:53.628 DEBUG:boto:Method: HEAD 2023-10-18T20:17:53.629 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:53.629 DEBUG:boto:Data: 2023-10-18T20:17:53.630 DEBUG:boto:Headers: {} 2023-10-18T20:17:53.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:53.630 DEBUG:boto:Port: 8000 2023-10-18T20:17:53.631 DEBUG:boto:Params: {} 2023-10-18T20:17:53.631 DEBUG:boto:Token: None 2023-10-18T20:17:53.631 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-48/ 2023-10-18T20:17:53.632 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1YsZZWq2UlGOVF/elcmvyGH3Zmk= 2023-10-18T20:17:53.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1YsZZWq2UlGOVF/elcmvyGH3Zmk=', 'Content-Length': '0'} 2023-10-18T20:17:53.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2eac1a4d8f62d68-0065303d71-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.635 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:17:53.635 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:17:53.636 DEBUG:boto:Method: PUT 2023-10-18T20:17:53.636 DEBUG:boto:Path: /bofrnf-48/obj6 2023-10-18T20:17:53.637 DEBUG:boto:Data: 2023-10-18T20:17:53.637 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:53.637 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:53.638 DEBUG:boto:Port: 8000 2023-10-18T20:17:53.638 DEBUG:boto:Params: {} 2023-10-18T20:17:53.639 DEBUG:boto:Token: None 2023-10-18T20:17:53.639 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-48/obj6 2023-10-18T20:17:53.640 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OwgoA2ihO3ido+f1sztzN+mEhWM= 2023-10-18T20:17:53.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OwgoA2ihO3ido+f1sztzN+mEhWM='} 2023-10-18T20:17:53.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f21396537f99f914-0065303d71-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.687 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:53.688 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:53.689 DEBUG:boto:Method: HEAD 2023-10-18T20:17:53.689 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:53.690 DEBUG:boto:Data: 2023-10-18T20:17:53.691 DEBUG:boto:Headers: {} 2023-10-18T20:17:53.691 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:53.692 DEBUG:boto:Port: 8000 2023-10-18T20:17:53.693 DEBUG:boto:Params: {} 2023-10-18T20:17:53.693 DEBUG:boto:Token: None 2023-10-18T20:17:53.694 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-48/ 2023-10-18T20:17:53.695 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1YsZZWq2UlGOVF/elcmvyGH3Zmk= 2023-10-18T20:17:53.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1YsZZWq2UlGOVF/elcmvyGH3Zmk=', 'Content-Length': '0'} 2023-10-18T20:17:53.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000764a450932cd92d3-0065303d71-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.699 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:17:53.699 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:17:53.699 DEBUG:boto:Method: PUT 2023-10-18T20:17:53.700 DEBUG:boto:Path: /bofrnf-48/obj7 2023-10-18T20:17:53.700 DEBUG:boto:Data: 2023-10-18T20:17:53.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:53.701 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:53.701 DEBUG:boto:Port: 8000 2023-10-18T20:17:53.702 DEBUG:boto:Params: {} 2023-10-18T20:17:53.702 DEBUG:boto:Token: None 2023-10-18T20:17:53.703 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-48/obj7 2023-10-18T20:17:53.703 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WDIMmUg1sY5pQXbpOmcuuwO/zSI= 2023-10-18T20:17:53.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WDIMmUg1sY5pQXbpOmcuuwO/zSI='} 2023-10-18T20:17:53.709 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043049c983b785746-0065303d71-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.709 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:17:53.709 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:17:53.710 DEBUG:boto:Method: HEAD 2023-10-18T20:17:53.710 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:17:53.711 DEBUG:boto:Data: 2023-10-18T20:17:53.711 DEBUG:boto:Headers: {} 2023-10-18T20:17:53.711 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:53.712 DEBUG:boto:Port: 8000 2023-10-18T20:17:53.712 DEBUG:boto:Params: {} 2023-10-18T20:17:53.712 DEBUG:boto:Token: None 2023-10-18T20:17:53.713 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-48/ 2023-10-18T20:17:53.713 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1YsZZWq2UlGOVF/elcmvyGH3Zmk= 2023-10-18T20:17:53.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1YsZZWq2UlGOVF/elcmvyGH3Zmk=', 'Content-Length': '0'} 2023-10-18T20:17:53.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001c9a73234d92def-0065303d71-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.717 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:17:53.717 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:17:53.717 DEBUG:boto:Method: PUT 2023-10-18T20:17:53.718 DEBUG:boto:Path: /bofrnf-48/obj8 2023-10-18T20:17:53.718 DEBUG:boto:Data: 2023-10-18T20:17:53.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:53.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:17:53.720 DEBUG:boto:Port: 8000 2023-10-18T20:17:53.720 DEBUG:boto:Params: {} 2023-10-18T20:17:53.720 DEBUG:boto:Token: None 2023-10-18T20:17:53.721 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-48/obj8 2023-10-18T20:17:53.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XEi0HGDgqRplMfhemW5RkD+s680= 2023-10-18T20:17:53.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XEi0HGDgqRplMfhemW5RkD+s680='} 2023-10-18T20:17:53.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000805e73d57095882-0065303d71-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.727 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:53.728 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:53.728 DEBUG:boto:Method: HEAD 2023-10-18T20:17:53.729 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:53.729 DEBUG:boto:Data: 2023-10-18T20:17:53.729 DEBUG:boto:Headers: {} 2023-10-18T20:17:53.730 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:53.730 DEBUG:boto:Port: 8001 2023-10-18T20:17:53.731 DEBUG:boto:Params: {} 2023-10-18T20:17:53.731 DEBUG:boto:Token: None 2023-10-18T20:17:53.732 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-49/ 2023-10-18T20:17:53.732 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZbkE5ef7qa2U4ibdz5mYVAjrQ3s= 2023-10-18T20:17:53.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZbkE5ef7qa2U4ibdz5mYVAjrQ3s=', 'Content-Length': '0'} 2023-10-18T20:17:53.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001578cf83071c5c8a-0065303d71-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.735 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:17:53.736 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:17:53.736 DEBUG:boto:Method: PUT 2023-10-18T20:17:53.737 DEBUG:boto:Path: /bofrnf-49/obj5 2023-10-18T20:17:53.737 DEBUG:boto:Data: 2023-10-18T20:17:53.737 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:53.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:53.738 DEBUG:boto:Port: 8001 2023-10-18T20:17:53.739 DEBUG:boto:Params: {} 2023-10-18T20:17:53.739 DEBUG:boto:Token: None 2023-10-18T20:17:53.739 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-49/obj5 2023-10-18T20:17:53.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n9BMqvKQ6R7KzMsqyXl2zFU+/ak= 2023-10-18T20:17:53.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n9BMqvKQ6R7KzMsqyXl2zFU+/ak='} 2023-10-18T20:17:53.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bb51ab88f087dad-0065303d71-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.744 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:53.745 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:53.745 DEBUG:boto:Method: HEAD 2023-10-18T20:17:53.745 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:53.746 DEBUG:boto:Data: 2023-10-18T20:17:53.746 DEBUG:boto:Headers: {} 2023-10-18T20:17:53.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:53.747 DEBUG:boto:Port: 8001 2023-10-18T20:17:53.747 DEBUG:boto:Params: {} 2023-10-18T20:17:53.748 DEBUG:boto:Token: None 2023-10-18T20:17:53.748 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-49/ 2023-10-18T20:17:53.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZbkE5ef7qa2U4ibdz5mYVAjrQ3s= 2023-10-18T20:17:53.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZbkE5ef7qa2U4ibdz5mYVAjrQ3s=', 'Content-Length': '0'} 2023-10-18T20:17:53.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b695875acb7f79c7-0065303d71-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.752 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:17:53.752 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:17:53.752 DEBUG:boto:Method: PUT 2023-10-18T20:17:53.753 DEBUG:boto:Path: /bofrnf-49/obj6 2023-10-18T20:17:53.753 DEBUG:boto:Data: 2023-10-18T20:17:53.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:53.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:53.754 DEBUG:boto:Port: 8001 2023-10-18T20:17:53.755 DEBUG:boto:Params: {} 2023-10-18T20:17:53.755 DEBUG:boto:Token: None 2023-10-18T20:17:53.755 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-49/obj6 2023-10-18T20:17:53.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6vFLbjmOv0H2bEFJF5LZgJ/fjQo= 2023-10-18T20:17:53.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6vFLbjmOv0H2bEFJF5LZgJ/fjQo='} 2023-10-18T20:17:53.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ef2cc1cf0a3a1d7-0065303d71-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.803 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:53.803 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:53.804 DEBUG:boto:Method: HEAD 2023-10-18T20:17:53.804 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:53.805 DEBUG:boto:Data: 2023-10-18T20:17:53.805 DEBUG:boto:Headers: {} 2023-10-18T20:17:53.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:53.806 DEBUG:boto:Port: 8001 2023-10-18T20:17:53.806 DEBUG:boto:Params: {} 2023-10-18T20:17:53.807 DEBUG:boto:Token: None 2023-10-18T20:17:53.807 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-49/ 2023-10-18T20:17:53.808 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZbkE5ef7qa2U4ibdz5mYVAjrQ3s= 2023-10-18T20:17:53.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZbkE5ef7qa2U4ibdz5mYVAjrQ3s=', 'Content-Length': '0'} 2023-10-18T20:17:53.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079278dbdc2f8f770-0065303d71-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.811 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:17:53.811 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:17:53.812 DEBUG:boto:Method: PUT 2023-10-18T20:17:53.812 DEBUG:boto:Path: /bofrnf-49/obj7 2023-10-18T20:17:53.812 DEBUG:boto:Data: 2023-10-18T20:17:53.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:53.813 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:53.814 DEBUG:boto:Port: 8001 2023-10-18T20:17:53.814 DEBUG:boto:Params: {} 2023-10-18T20:17:53.814 DEBUG:boto:Token: None 2023-10-18T20:17:53.815 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-49/obj7 2023-10-18T20:17:53.815 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KIcGqkgZihhhK5MGxe3xWaNcv/o= 2023-10-18T20:17:53.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KIcGqkgZihhhK5MGxe3xWaNcv/o='} 2023-10-18T20:17:53.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d559bf8618f65a10-0065303d71-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.820 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:17:53.820 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:17:53.821 DEBUG:boto:Method: HEAD 2023-10-18T20:17:53.821 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:17:53.822 DEBUG:boto:Data: 2023-10-18T20:17:53.822 DEBUG:boto:Headers: {} 2023-10-18T20:17:53.823 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:53.823 DEBUG:boto:Port: 8001 2023-10-18T20:17:53.823 DEBUG:boto:Params: {} 2023-10-18T20:17:53.824 DEBUG:boto:Token: None 2023-10-18T20:17:53.824 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-49/ 2023-10-18T20:17:53.825 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZbkE5ef7qa2U4ibdz5mYVAjrQ3s= 2023-10-18T20:17:53.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZbkE5ef7qa2U4ibdz5mYVAjrQ3s=', 'Content-Length': '0'} 2023-10-18T20:17:53.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dece79588fcb13c2-0065303d71-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.828 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:17:53.828 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:17:53.829 DEBUG:boto:Method: PUT 2023-10-18T20:17:53.829 DEBUG:boto:Path: /bofrnf-49/obj8 2023-10-18T20:17:53.830 DEBUG:boto:Data: 2023-10-18T20:17:53.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T20:17:53.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:17:53.831 DEBUG:boto:Port: 8001 2023-10-18T20:17:53.831 DEBUG:boto:Params: {} 2023-10-18T20:17:53.831 DEBUG:boto:Token: None 2023-10-18T20:17:53.832 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 20:17:53 GMT /bofrnf-49/obj8 2023-10-18T20:17:53.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rl7m1sGxCJWmbv6J4Ab/JIUA0+Q= 2023-10-18T20:17:53.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:17:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rl7m1sGxCJWmbv6J4Ab/JIUA0+Q='} 2023-10-18T20:17:53.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6184b4b3190805c-0065303d71-1668-a2'), ('Date', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:17:53.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bofrnf-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:17:53.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bofrnf-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:17:53.880 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bofrnf-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:17:53.931 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:53.931 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:53.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bofrnf-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:17:53.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bofrnf-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:17:53.999 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bofrnf-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:17:54.049 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:54.050 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:54.117 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-48 2023-10-18T20:17:54.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:54.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:54.118 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:17:54.175 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:54.175 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:54.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:54.208+0000 7f1309cd9780 1 waiting for incremental sync to catch up: 2023-10-18T20:17:54.209 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.27.6, , 00000000001.23.6, , 00000000001.33.6, , , 00000000001.25.6, , , ] 2023-10-18T20:17:54.209 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.31.1, 00000000002.38.1, 00000000004.27.1, 00000000002.27.1, 00000000004.38.1, 00000000002.26.1, 00000000002.36.1, 00000000004.29.1, 00000000002.36.1, 00000000002.25.1, 00000000002.39.1] 2023-10-18T20:17:59.210 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:59.209+0000 7f1309cd9780 1 bucket sync caught up with source: 2023-10-18T20:17:59.211 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.31.1, 00000000002.38.1, 00000000004.27.1, 00000000002.27.1, 00000000004.38.1, 00000000002.26.1, 00000000002.36.1, 00000000004.29.1, 00000000002.36.1, 00000000002.25.1, 00000000002.39.1] 2023-10-18T20:17:59.211 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.31.1, 00000000002.38.1, 00000000004.27.1, 00000000002.27.1, 00000000004.38.1, 00000000002.26.1, 00000000002.36.1, 00000000004.29.1, 00000000002.36.1, 00000000002.25.1, 00000000002.39.1] 2023-10-18T20:17:59.211 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:59.209+0000 7f1309cd9780 0 bucket checkpoint complete 2023-10-18T20:17:59.227 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-48 2023-10-18T20:17:59.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:59.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:17:59.228 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:17:59.290 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:17:59.291 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:17:59.326 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:17:59.325+0000 7f2adff09780 1 waiting to reach incremental sync.. 2023-10-18T20:18:04.327 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:04.325+0000 7f2adff09780 1 waiting to reach incremental sync.. 2023-10-18T20:18:09.327 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:09.326+0000 7f2adff09780 1 waiting to reach incremental sync.. 2023-10-18T20:18:14.328 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:14.327+0000 7f2adff09780 1 waiting to reach incremental sync.. 2023-10-18T20:18:19.330 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:19.327+0000 7f2adff09780 1 bucket sync caught up with source: 2023-10-18T20:18:19.330 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.295.1, 00000000002.812.1, 00000000004.670.1, 00000000002.917.1, 00000000004.1147.1, 00000000002.203.1, 00000000002.693.1, 00000000004.810.1, 00000000002.464.1, 00000000002.542.1, 00000000002.1148.1] 2023-10-18T20:18:19.330 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.290.1, 00000000001.811.1, 00000000002.665.1, 00000000001.911.1, 00000000002.1139.1, 00000000001.197.1, 00000000001.685.1, 00000000002.805.1, 00000000001.458.1, 00000000001.541.1, 00000000001.1140.1] 2023-10-18T20:18:19.331 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:19.327+0000 7f2adff09780 0 bucket checkpoint complete 2023-10-18T20:18:19.346 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-48 zones={a2, a1} 2023-10-18T20:18:19.347 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.347 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.348 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.348 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:18:19.349 DEBUG:boto:Data: 2023-10-18T20:18:19.349 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.350 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.350 DEBUG:boto:Params: {} 2023-10-18T20:18:19.351 DEBUG:boto:Token: None 2023-10-18T20:18:19.351 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.352 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EBTta0rqbzQiTZzGFNWwUCjO9lU= 2023-10-18T20:18:19.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EBTta0rqbzQiTZzGFNWwUCjO9lU=', 'Content-Length': '0'} 2023-10-18T20:18:19.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e6a47250d20d021-0065303d8b-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.356 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.356 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.356 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.357 DEBUG:boto:Path: /bofrnf-48/ 2023-10-18T20:18:19.357 DEBUG:boto:Data: 2023-10-18T20:18:19.358 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.358 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.359 DEBUG:boto:Params: {} 2023-10-18T20:18:19.359 DEBUG:boto:Token: None 2023-10-18T20:18:19.360 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EBTta0rqbzQiTZzGFNWwUCjO9lU= 2023-10-18T20:18:19.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EBTta0rqbzQiTZzGFNWwUCjO9lU=', 'Content-Length': '0'} 2023-10-18T20:18:19.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cef6904dff210553-0065303d8b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.364 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:18:19.364 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.365 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.365 DEBUG:boto:path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.365 DEBUG:boto:auth_path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.366 DEBUG:boto:Method: GET 2023-10-18T20:18:19.366 DEBUG:boto:Path: /bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.366 DEBUG:boto:Data: 2023-10-18T20:18:19.367 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.368 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.368 DEBUG:boto:Params: {} 2023-10-18T20:18:19.368 DEBUG:boto:Token: None 2023-10-18T20:18:19.369 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/?versions 2023-10-18T20:18:19.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OlrScSu5uSTWvQmU7EknFssA4AY= 2023-10-18T20:18:19.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlrScSu5uSTWvQmU7EknFssA4AY=', 'Content-Length': '0'} 2023-10-18T20:18:19.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065855566933fd381-0065303d8b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.373 DEBUG:boto:b'bofrnf-48999falseobj1nulltrue2023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:17:53.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:17:53.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:17:53.705Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:17:53.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:18:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:18:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:18:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:18:19.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-18T20:18:19.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-18T20:18:19.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-18T20:18:19.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-18T20:18:19.377 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:18:19.377 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.378 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.378 DEBUG:boto:path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.379 DEBUG:boto:auth_path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.379 DEBUG:boto:Method: GET 2023-10-18T20:18:19.379 DEBUG:boto:Path: /bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.380 DEBUG:boto:Data: 2023-10-18T20:18:19.380 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.381 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.381 DEBUG:boto:Params: {} 2023-10-18T20:18:19.382 DEBUG:boto:Token: None 2023-10-18T20:18:19.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/?versions 2023-10-18T20:18:19.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OlrScSu5uSTWvQmU7EknFssA4AY= 2023-10-18T20:18:19.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlrScSu5uSTWvQmU7EknFssA4AY=', 'Content-Length': '0'} 2023-10-18T20:18:19.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91f68d4b5bc69b6-0065303d8b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.386 DEBUG:boto:b'bofrnf-48999falseobj1nulltrue2023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:17:53.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:17:53.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:17:53.705Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:17:53.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:18:19.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:18:19.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:18:19.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:18:19.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-18T20:18:19.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-18T20:18:19.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-18T20:18:19.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-18T20:18:19.390 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.391 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.391 DEBUG:boto:path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.392 DEBUG:boto:auth_path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.392 DEBUG:boto:Method: GET 2023-10-18T20:18:19.392 DEBUG:boto:Path: /bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.393 DEBUG:boto:Data: 2023-10-18T20:18:19.393 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.394 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.394 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.394 DEBUG:boto:Params: {} 2023-10-18T20:18:19.395 DEBUG:boto:Token: None 2023-10-18T20:18:19.395 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/?versions 2023-10-18T20:18:19.396 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OlrScSu5uSTWvQmU7EknFssA4AY= 2023-10-18T20:18:19.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlrScSu5uSTWvQmU7EknFssA4AY=', 'Content-Length': '0'} 2023-10-18T20:18:19.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9859e8680c171dd-0065303d8b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.399 DEBUG:boto:b'bofrnf-48999falseobj1nulltrue2023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:17:53.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:17:53.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:17:53.705Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:17:53.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.400 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.400 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.401 DEBUG:boto:path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.401 DEBUG:boto:auth_path=/bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.402 DEBUG:boto:Method: GET 2023-10-18T20:18:19.402 DEBUG:boto:Path: /bofrnf-48/?versions&max-keys=999 2023-10-18T20:18:19.402 DEBUG:boto:Data: 2023-10-18T20:18:19.403 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.403 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.403 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.404 DEBUG:boto:Params: {} 2023-10-18T20:18:19.404 DEBUG:boto:Token: None 2023-10-18T20:18:19.405 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/?versions 2023-10-18T20:18:19.405 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OlrScSu5uSTWvQmU7EknFssA4AY= 2023-10-18T20:18:19.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlrScSu5uSTWvQmU7EknFssA4AY=', 'Content-Length': '0'} 2023-10-18T20:18:19.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005332375abeda772-0065303d8b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.409 DEBUG:boto:b'bofrnf-48999falseobj1nulltrue2023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:17:53.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:17:53.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:17:53.705Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:17:53.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:18:19.410 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.411 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.411 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.411 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.412 DEBUG:boto:Method: GET 2023-10-18T20:18:19.412 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.412 DEBUG:boto:Data: 2023-10-18T20:18:19.413 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.413 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.414 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.414 DEBUG:boto:Params: {} 2023-10-18T20:18:19.415 DEBUG:boto:Token: None 2023-10-18T20:18:19.415 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.416 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CFxJbZy/+OmHLNQX38HJm0atXuI= 2023-10-18T20:18:19.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CFxJbZy/+OmHLNQX38HJm0atXuI=', 'Content-Length': '0'} 2023-10-18T20:18:19.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec2623bfc7551837-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.419 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.419 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.420 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.420 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.421 DEBUG:boto:Method: GET 2023-10-18T20:18:19.421 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.421 DEBUG:boto:Data: 2023-10-18T20:18:19.422 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.422 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.423 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.423 DEBUG:boto:Params: {} 2023-10-18T20:18:19.424 DEBUG:boto:Token: None 2023-10-18T20:18:19.424 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.425 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CFxJbZy/+OmHLNQX38HJm0atXuI= 2023-10-18T20:18:19.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CFxJbZy/+OmHLNQX38HJm0atXuI=', 'Content-Length': '0'} 2023-10-18T20:18:19.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d41119d30d2bda01-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.470 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.471 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.471 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.471 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.472 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.472 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.473 DEBUG:boto:Data: 2023-10-18T20:18:19.473 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.473 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.474 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.474 DEBUG:boto:Params: {} 2023-10-18T20:18:19.475 DEBUG:boto:Token: None 2023-10-18T20:18:19.475 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.475 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YTTBVopn0rb50d/2Rd1N2gMA5n0= 2023-10-18T20:18:19.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YTTBVopn0rb50d/2Rd1N2gMA5n0=', 'Content-Length': '0'} 2023-10-18T20:18:19.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7e1bb980a42c1ec-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.478 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.478 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.479 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.479 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.479 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.480 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.480 DEBUG:boto:Data: 2023-10-18T20:18:19.481 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.481 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.482 DEBUG:boto:Params: {} 2023-10-18T20:18:19.482 DEBUG:boto:Token: None 2023-10-18T20:18:19.482 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.483 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YTTBVopn0rb50d/2Rd1N2gMA5n0= 2023-10-18T20:18:19.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YTTBVopn0rb50d/2Rd1N2gMA5n0=', 'Content-Length': '0'} 2023-10-18T20:18:19.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ca3602e21736108-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:18:19.486 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.486 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.487 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.487 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.488 DEBUG:boto:Method: GET 2023-10-18T20:18:19.488 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.488 DEBUG:boto:Data: 2023-10-18T20:18:19.489 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.489 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.490 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.490 DEBUG:boto:Params: {} 2023-10-18T20:18:19.491 DEBUG:boto:Token: None 2023-10-18T20:18:19.491 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.491 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CFxJbZy/+OmHLNQX38HJm0atXuI= 2023-10-18T20:18:19.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CFxJbZy/+OmHLNQX38HJm0atXuI=', 'Content-Length': '0'} 2023-10-18T20:18:19.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a37de018ac13d10-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.495 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.495 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.495 DEBUG:boto:path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.496 DEBUG:boto:auth_path=/bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.496 DEBUG:boto:Method: GET 2023-10-18T20:18:19.497 DEBUG:boto:Path: /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.497 DEBUG:boto:Data: 2023-10-18T20:18:19.497 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.498 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.499 DEBUG:boto:Params: {} 2023-10-18T20:18:19.499 DEBUG:boto:Token: None 2023-10-18T20:18:19.500 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1?versionId=null 2023-10-18T20:18:19.500 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CFxJbZy/+OmHLNQX38HJm0atXuI= 2023-10-18T20:18:19.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CFxJbZy/+OmHLNQX38HJm0atXuI=', 'Content-Length': '0'} 2023-10-18T20:18:19.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9ca36376e35826d-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.503 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.504 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.504 DEBUG:boto:path=/bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.505 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.505 DEBUG:boto:Method: GET 2023-10-18T20:18:19.505 DEBUG:boto:Path: /bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.506 DEBUG:boto:Data: 2023-10-18T20:18:19.506 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.507 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.508 DEBUG:boto:Params: {} 2023-10-18T20:18:19.508 DEBUG:boto:Token: None 2023-10-18T20:18:19.508 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b960577de2a18fc2-0065303d8b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.512 DEBUG:boto:b'bofrnf-48obj11000falseobj12023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.513 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.514 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.514 DEBUG:boto:path=/bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.514 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.515 DEBUG:boto:Method: GET 2023-10-18T20:18:19.515 DEBUG:boto:Path: /bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.516 DEBUG:boto:Data: 2023-10-18T20:18:19.516 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.517 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.517 DEBUG:boto:Params: {} 2023-10-18T20:18:19.517 DEBUG:boto:Token: None 2023-10-18T20:18:19.518 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.518 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3ee7dd78d597323-0065303d8b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.522 DEBUG:boto:b'bofrnf-48obj11000falseobj12023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.522 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.523 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.523 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.523 DEBUG:boto:Path: /bofrnf-48/obj1 2023-10-18T20:18:19.524 DEBUG:boto:Data: 2023-10-18T20:18:19.524 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.525 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.525 DEBUG:boto:Params: {} 2023-10-18T20:18:19.526 DEBUG:boto:Token: None 2023-10-18T20:18:19.526 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1 2023-10-18T20:18:19.527 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Hnf84phLn3qZQGA2yrz5gvdtjk= 2023-10-18T20:18:19.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Hnf84phLn3qZQGA2yrz5gvdtjk=', 'Content-Length': '0'} 2023-10-18T20:18:19.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016702b1b627be11d-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.530 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.530 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.530 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.531 DEBUG:boto:Path: /bofrnf-48/obj1 2023-10-18T20:18:19.531 DEBUG:boto:Data: 2023-10-18T20:18:19.532 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.532 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.532 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.533 DEBUG:boto:Params: {} 2023-10-18T20:18:19.533 DEBUG:boto:Token: None 2023-10-18T20:18:19.534 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1 2023-10-18T20:18:19.534 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Hnf84phLn3qZQGA2yrz5gvdtjk= 2023-10-18T20:18:19.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Hnf84phLn3qZQGA2yrz5gvdtjk=', 'Content-Length': '0'} 2023-10-18T20:18:19.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e4be641f8c5abfc-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:18:19.538 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.538 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.538 DEBUG:boto:Method: GET 2023-10-18T20:18:19.539 DEBUG:boto:Path: /bofrnf-48/obj1 2023-10-18T20:18:19.539 DEBUG:boto:Data: 2023-10-18T20:18:19.540 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.540 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.541 DEBUG:boto:Params: {} 2023-10-18T20:18:19.541 DEBUG:boto:Token: None 2023-10-18T20:18:19.542 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1 2023-10-18T20:18:19.542 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p3RexVbQclhavzZck+jWC88JJzY= 2023-10-18T20:18:19.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p3RexVbQclhavzZck+jWC88JJzY=', 'Content-Length': '0'} 2023-10-18T20:18:19.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b881701791cb869-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.545 DEBUG:boto:path=/bofrnf-48/obj1 2023-10-18T20:18:19.546 DEBUG:boto:auth_path=/bofrnf-48/obj1 2023-10-18T20:18:19.546 DEBUG:boto:Method: GET 2023-10-18T20:18:19.547 DEBUG:boto:Path: /bofrnf-48/obj1 2023-10-18T20:18:19.547 DEBUG:boto:Data: 2023-10-18T20:18:19.547 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.548 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.548 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.549 DEBUG:boto:Params: {} 2023-10-18T20:18:19.549 DEBUG:boto:Token: None 2023-10-18T20:18:19.549 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj1 2023-10-18T20:18:19.550 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p3RexVbQclhavzZck+jWC88JJzY= 2023-10-18T20:18:19.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p3RexVbQclhavzZck+jWC88JJzY=', 'Content-Length': '0'} 2023-10-18T20:18:19.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d610a2f86437b165-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.553 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.553 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.554 DEBUG:boto:path=/bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.554 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.555 DEBUG:boto:Method: GET 2023-10-18T20:18:19.555 DEBUG:boto:Path: /bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.555 DEBUG:boto:Data: 2023-10-18T20:18:19.556 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.556 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.556 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.557 DEBUG:boto:Params: {} 2023-10-18T20:18:19.557 DEBUG:boto:Token: None 2023-10-18T20:18:19.558 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.558 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a8af658a866a56-0065303d8b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.562 DEBUG:boto:b'bofrnf-48obj11000falseobj12023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.562 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.563 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.563 DEBUG:boto:path=/bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.564 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.564 DEBUG:boto:Method: GET 2023-10-18T20:18:19.564 DEBUG:boto:Path: /bofrnf-48/?prefix=obj1 2023-10-18T20:18:19.565 DEBUG:boto:Data: 2023-10-18T20:18:19.565 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.566 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.566 DEBUG:boto:Params: {} 2023-10-18T20:18:19.567 DEBUG:boto:Token: None 2023-10-18T20:18:19.567 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b57d44c5907e7e7-0065303d8b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.571 DEBUG:boto:b'bofrnf-48obj11000falseobj12023-10-18T20:17:00.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:18:19.572 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.573 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.573 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.573 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.574 DEBUG:boto:Method: GET 2023-10-18T20:18:19.574 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.575 DEBUG:boto:Data: 2023-10-18T20:18:19.575 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.575 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.576 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.576 DEBUG:boto:Params: {} 2023-10-18T20:18:19.577 DEBUG:boto:Token: None 2023-10-18T20:18:19.577 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bq6t/cTC/tb8ZQWBEf+CLbwrgmg= 2023-10-18T20:18:19.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bq6t/cTC/tb8ZQWBEf+CLbwrgmg=', 'Content-Length': '0'} 2023-10-18T20:18:19.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9e164d631c4076c-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.580 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.581 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.581 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.582 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.582 DEBUG:boto:Method: GET 2023-10-18T20:18:19.582 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.583 DEBUG:boto:Data: 2023-10-18T20:18:19.583 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.584 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.584 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.584 DEBUG:boto:Params: {} 2023-10-18T20:18:19.585 DEBUG:boto:Token: None 2023-10-18T20:18:19.585 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bq6t/cTC/tb8ZQWBEf+CLbwrgmg= 2023-10-18T20:18:19.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bq6t/cTC/tb8ZQWBEf+CLbwrgmg=', 'Content-Length': '0'} 2023-10-18T20:18:19.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016bf8b6617b3d677-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.630 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.631 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.631 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.633 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.633 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.634 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.634 DEBUG:boto:Data: 2023-10-18T20:18:19.634 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.635 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.635 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.636 DEBUG:boto:Params: {} 2023-10-18T20:18:19.636 DEBUG:boto:Token: None 2023-10-18T20:18:19.637 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.637 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V+k+OBJ084UPDQij2iUnofCl7bI= 2023-10-18T20:18:19.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V+k+OBJ084UPDQij2iUnofCl7bI=', 'Content-Length': '0'} 2023-10-18T20:18:19.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3ba6dd8ba1c3f36-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.640 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.641 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.641 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.642 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.642 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.643 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.643 DEBUG:boto:Data: 2023-10-18T20:18:19.643 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.644 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.684 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.685 DEBUG:boto:Params: {} 2023-10-18T20:18:19.685 DEBUG:boto:Token: None 2023-10-18T20:18:19.686 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.686 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V+k+OBJ084UPDQij2iUnofCl7bI= 2023-10-18T20:18:19.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V+k+OBJ084UPDQij2iUnofCl7bI=', 'Content-Length': '0'} 2023-10-18T20:18:19.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e6bc99773533a55-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:18:19.690 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.691 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.691 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.692 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.692 DEBUG:boto:Method: GET 2023-10-18T20:18:19.692 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.693 DEBUG:boto:Data: 2023-10-18T20:18:19.693 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.694 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.694 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.695 DEBUG:boto:Params: {} 2023-10-18T20:18:19.695 DEBUG:boto:Token: None 2023-10-18T20:18:19.695 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.696 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bq6t/cTC/tb8ZQWBEf+CLbwrgmg= 2023-10-18T20:18:19.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bq6t/cTC/tb8ZQWBEf+CLbwrgmg=', 'Content-Length': '0'} 2023-10-18T20:18:19.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac6375acb9fa0760-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.700 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.700 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.700 DEBUG:boto:path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.701 DEBUG:boto:auth_path=/bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.701 DEBUG:boto:Method: GET 2023-10-18T20:18:19.702 DEBUG:boto:Path: /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.702 DEBUG:boto:Data: 2023-10-18T20:18:19.703 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.703 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.703 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.704 DEBUG:boto:Params: {} 2023-10-18T20:18:19.704 DEBUG:boto:Token: None 2023-10-18T20:18:19.705 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2?versionId=null 2023-10-18T20:18:19.705 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bq6t/cTC/tb8ZQWBEf+CLbwrgmg= 2023-10-18T20:18:19.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bq6t/cTC/tb8ZQWBEf+CLbwrgmg=', 'Content-Length': '0'} 2023-10-18T20:18:19.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f7b00418ab03b05-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.709 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.709 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.710 DEBUG:boto:path=/bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.710 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.711 DEBUG:boto:Method: GET 2023-10-18T20:18:19.711 DEBUG:boto:Path: /bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.712 DEBUG:boto:Data: 2023-10-18T20:18:19.712 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.712 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.713 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.713 DEBUG:boto:Params: {} 2023-10-18T20:18:19.714 DEBUG:boto:Token: None 2023-10-18T20:18:19.714 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.715 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000442662e3d7b44cc2-0065303d8b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.758 DEBUG:boto:b'bofrnf-48obj21000falseobj22023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.759 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.759 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.760 DEBUG:boto:path=/bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.760 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.761 DEBUG:boto:Method: GET 2023-10-18T20:18:19.761 DEBUG:boto:Path: /bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.761 DEBUG:boto:Data: 2023-10-18T20:18:19.762 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.762 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.763 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.763 DEBUG:boto:Params: {} 2023-10-18T20:18:19.764 DEBUG:boto:Token: None 2023-10-18T20:18:19.764 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.765 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011aa5f484a01f83a-0065303d8b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.769 DEBUG:boto:b'bofrnf-48obj21000falseobj22023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.769 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.770 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.770 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.771 DEBUG:boto:Path: /bofrnf-48/obj2 2023-10-18T20:18:19.771 DEBUG:boto:Data: 2023-10-18T20:18:19.771 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.772 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.772 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.773 DEBUG:boto:Params: {} 2023-10-18T20:18:19.773 DEBUG:boto:Token: None 2023-10-18T20:18:19.774 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2 2023-10-18T20:18:19.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5+IOtw0lxiLO6QbqvX3p3WDvSQs= 2023-10-18T20:18:19.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5+IOtw0lxiLO6QbqvX3p3WDvSQs=', 'Content-Length': '0'} 2023-10-18T20:18:19.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009ac184c5c2e772f-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.777 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.777 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.778 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.778 DEBUG:boto:Path: /bofrnf-48/obj2 2023-10-18T20:18:19.779 DEBUG:boto:Data: 2023-10-18T20:18:19.779 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.780 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.780 DEBUG:boto:Params: {} 2023-10-18T20:18:19.781 DEBUG:boto:Token: None 2023-10-18T20:18:19.781 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2 2023-10-18T20:18:19.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5+IOtw0lxiLO6QbqvX3p3WDvSQs= 2023-10-18T20:18:19.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5+IOtw0lxiLO6QbqvX3p3WDvSQs=', 'Content-Length': '0'} 2023-10-18T20:18:19.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012545b86f738de79-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:18:19.785 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.785 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.786 DEBUG:boto:Method: GET 2023-10-18T20:18:19.786 DEBUG:boto:Path: /bofrnf-48/obj2 2023-10-18T20:18:19.787 DEBUG:boto:Data: 2023-10-18T20:18:19.787 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.788 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.788 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.788 DEBUG:boto:Params: {} 2023-10-18T20:18:19.789 DEBUG:boto:Token: None 2023-10-18T20:18:19.789 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2 2023-10-18T20:18:19.790 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L9NB6buDm5A9s30vIY1t4lKA0Us= 2023-10-18T20:18:19.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L9NB6buDm5A9s30vIY1t4lKA0Us=', 'Content-Length': '0'} 2023-10-18T20:18:19.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e56548b8a3812c44-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.793 DEBUG:boto:path=/bofrnf-48/obj2 2023-10-18T20:18:19.793 DEBUG:boto:auth_path=/bofrnf-48/obj2 2023-10-18T20:18:19.794 DEBUG:boto:Method: GET 2023-10-18T20:18:19.794 DEBUG:boto:Path: /bofrnf-48/obj2 2023-10-18T20:18:19.795 DEBUG:boto:Data: 2023-10-18T20:18:19.795 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.795 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.796 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.796 DEBUG:boto:Params: {} 2023-10-18T20:18:19.797 DEBUG:boto:Token: None 2023-10-18T20:18:19.797 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj2 2023-10-18T20:18:19.797 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L9NB6buDm5A9s30vIY1t4lKA0Us= 2023-10-18T20:18:19.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L9NB6buDm5A9s30vIY1t4lKA0Us=', 'Content-Length': '0'} 2023-10-18T20:18:19.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c798f0347523a9e6-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.801 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.801 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.802 DEBUG:boto:path=/bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.802 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.802 DEBUG:boto:Method: GET 2023-10-18T20:18:19.803 DEBUG:boto:Path: /bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.803 DEBUG:boto:Data: 2023-10-18T20:18:19.804 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.804 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.805 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.805 DEBUG:boto:Params: {} 2023-10-18T20:18:19.806 DEBUG:boto:Token: None 2023-10-18T20:18:19.806 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.807 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c797c2385abebd-0065303d8b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.810 DEBUG:boto:b'bofrnf-48obj21000falseobj22023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.811 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.811 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.812 DEBUG:boto:path=/bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.812 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.813 DEBUG:boto:Method: GET 2023-10-18T20:18:19.813 DEBUG:boto:Path: /bofrnf-48/?prefix=obj2 2023-10-18T20:18:19.814 DEBUG:boto:Data: 2023-10-18T20:18:19.814 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.815 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.815 DEBUG:boto:Params: {} 2023-10-18T20:18:19.816 DEBUG:boto:Token: None 2023-10-18T20:18:19.816 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.817 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099fef1daf97763ce-0065303d8b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.820 DEBUG:boto:b'bofrnf-48obj21000falseobj22023-10-18T20:17:00.245Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:18:19.821 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.822 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.822 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.823 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.823 DEBUG:boto:Method: GET 2023-10-18T20:18:19.824 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.824 DEBUG:boto:Data: 2023-10-18T20:18:19.824 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.825 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.825 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.826 DEBUG:boto:Params: {} 2023-10-18T20:18:19.826 DEBUG:boto:Token: None 2023-10-18T20:18:19.827 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.827 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDnyZHTcKXxMM0RDRLtN8hxECqU= 2023-10-18T20:18:19.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDnyZHTcKXxMM0RDRLtN8hxECqU=', 'Content-Length': '0'} 2023-10-18T20:18:19.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a53123e8e5971156-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.830 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.831 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.831 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.831 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.832 DEBUG:boto:Method: GET 2023-10-18T20:18:19.832 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.833 DEBUG:boto:Data: 2023-10-18T20:18:19.833 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.833 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.834 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.834 DEBUG:boto:Params: {} 2023-10-18T20:18:19.835 DEBUG:boto:Token: None 2023-10-18T20:18:19.835 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.837 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDnyZHTcKXxMM0RDRLtN8hxECqU= 2023-10-18T20:18:19.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDnyZHTcKXxMM0RDRLtN8hxECqU=', 'Content-Length': '0'} 2023-10-18T20:18:19.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa96d61f9f30186d-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.840 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.840 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.841 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.841 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.841 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.842 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.842 DEBUG:boto:Data: 2023-10-18T20:18:19.843 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.843 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.844 DEBUG:boto:Params: {} 2023-10-18T20:18:19.844 DEBUG:boto:Token: None 2023-10-18T20:18:19.845 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.845 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CEoRw++s+xeV/XPIYjVnVhrmX7A= 2023-10-18T20:18:19.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CEoRw++s+xeV/XPIYjVnVhrmX7A=', 'Content-Length': '0'} 2023-10-18T20:18:19.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebedad01523f8b12-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.848 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.849 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.849 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.849 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.850 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.850 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.851 DEBUG:boto:Data: 2023-10-18T20:18:19.851 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.851 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.852 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.852 DEBUG:boto:Params: {} 2023-10-18T20:18:19.853 DEBUG:boto:Token: None 2023-10-18T20:18:19.853 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.853 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CEoRw++s+xeV/XPIYjVnVhrmX7A= 2023-10-18T20:18:19.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CEoRw++s+xeV/XPIYjVnVhrmX7A=', 'Content-Length': '0'} 2023-10-18T20:18:19.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f640ac724715125-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:18:19.856 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.857 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.857 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.858 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.858 DEBUG:boto:Method: GET 2023-10-18T20:18:19.859 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.859 DEBUG:boto:Data: 2023-10-18T20:18:19.859 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.860 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.860 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.861 DEBUG:boto:Params: {} 2023-10-18T20:18:19.861 DEBUG:boto:Token: None 2023-10-18T20:18:19.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDnyZHTcKXxMM0RDRLtN8hxECqU= 2023-10-18T20:18:19.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDnyZHTcKXxMM0RDRLtN8hxECqU=', 'Content-Length': '0'} 2023-10-18T20:18:19.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068be5db0b0fa1a33-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.865 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.866 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.866 DEBUG:boto:path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.867 DEBUG:boto:auth_path=/bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.867 DEBUG:boto:Method: GET 2023-10-18T20:18:19.868 DEBUG:boto:Path: /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.868 DEBUG:boto:Data: 2023-10-18T20:18:19.868 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.869 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.869 DEBUG:boto:Params: {} 2023-10-18T20:18:19.870 DEBUG:boto:Token: None 2023-10-18T20:18:19.871 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3?versionId=null 2023-10-18T20:18:19.871 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDnyZHTcKXxMM0RDRLtN8hxECqU= 2023-10-18T20:18:19.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDnyZHTcKXxMM0RDRLtN8hxECqU=', 'Content-Length': '0'} 2023-10-18T20:18:19.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000430148ba8697c0bc-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.914 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.915 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.915 DEBUG:boto:path=/bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.916 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.916 DEBUG:boto:Method: GET 2023-10-18T20:18:19.916 DEBUG:boto:Path: /bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.917 DEBUG:boto:Data: 2023-10-18T20:18:19.917 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.918 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.918 DEBUG:boto:Params: {} 2023-10-18T20:18:19.919 DEBUG:boto:Token: None 2023-10-18T20:18:19.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa7de957b6d244b-0065303d8b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.923 DEBUG:boto:b'bofrnf-48obj31000falseobj32023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.924 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.924 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.924 DEBUG:boto:path=/bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.925 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.925 DEBUG:boto:Method: GET 2023-10-18T20:18:19.926 DEBUG:boto:Path: /bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.926 DEBUG:boto:Data: 2023-10-18T20:18:19.927 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.927 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.927 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.928 DEBUG:boto:Params: {} 2023-10-18T20:18:19.928 DEBUG:boto:Token: None 2023-10-18T20:18:19.928 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2299d5acb8fd13e-0065303d8b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.933 DEBUG:boto:b'bofrnf-48obj31000falseobj32023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.933 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.934 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.934 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.935 DEBUG:boto:Path: /bofrnf-48/obj3 2023-10-18T20:18:19.935 DEBUG:boto:Data: 2023-10-18T20:18:19.935 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.936 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.936 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.936 DEBUG:boto:Params: {} 2023-10-18T20:18:19.937 DEBUG:boto:Token: None 2023-10-18T20:18:19.937 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3 2023-10-18T20:18:19.938 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GzMmSl/JSUy62CMpAhVfZ9iOk2o= 2023-10-18T20:18:19.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GzMmSl/JSUy62CMpAhVfZ9iOk2o=', 'Content-Length': '0'} 2023-10-18T20:18:19.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045cabe17ebe2891f-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.941 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.941 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.942 DEBUG:boto:Method: HEAD 2023-10-18T20:18:19.942 DEBUG:boto:Path: /bofrnf-48/obj3 2023-10-18T20:18:19.943 DEBUG:boto:Data: 2023-10-18T20:18:19.943 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.944 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.944 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.945 DEBUG:boto:Params: {} 2023-10-18T20:18:19.946 DEBUG:boto:Token: None 2023-10-18T20:18:19.947 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3 2023-10-18T20:18:19.947 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GzMmSl/JSUy62CMpAhVfZ9iOk2o= 2023-10-18T20:18:19.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GzMmSl/JSUy62CMpAhVfZ9iOk2o=', 'Content-Length': '0'} 2023-10-18T20:18:19.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000130866bfcd0248ac-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:18:19.952 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.953 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.954 DEBUG:boto:Method: GET 2023-10-18T20:18:19.955 DEBUG:boto:Path: /bofrnf-48/obj3 2023-10-18T20:18:19.955 DEBUG:boto:Data: 2023-10-18T20:18:19.956 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.957 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.958 DEBUG:boto:Params: {} 2023-10-18T20:18:19.959 DEBUG:boto:Token: None 2023-10-18T20:18:19.960 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3 2023-10-18T20:18:19.961 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tUga3hrDrAiMnMIkty931Lf0ozU= 2023-10-18T20:18:19.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tUga3hrDrAiMnMIkty931Lf0ozU=', 'Content-Length': '0'} 2023-10-18T20:18:19.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007195690049efe8e-0065303d8b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.965 DEBUG:boto:path=/bofrnf-48/obj3 2023-10-18T20:18:19.966 DEBUG:boto:auth_path=/bofrnf-48/obj3 2023-10-18T20:18:19.966 DEBUG:boto:Method: GET 2023-10-18T20:18:19.967 DEBUG:boto:Path: /bofrnf-48/obj3 2023-10-18T20:18:19.968 DEBUG:boto:Data: 2023-10-18T20:18:19.969 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.970 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.971 DEBUG:boto:Params: {} 2023-10-18T20:18:19.972 DEBUG:boto:Token: None 2023-10-18T20:18:19.973 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/obj3 2023-10-18T20:18:19.974 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tUga3hrDrAiMnMIkty931Lf0ozU= 2023-10-18T20:18:19.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tUga3hrDrAiMnMIkty931Lf0ozU=', 'Content-Length': '0'} 2023-10-18T20:18:19.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dd91ac51974dbcc-0065303d8b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.977 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.978 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.978 DEBUG:boto:path=/bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.979 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.979 DEBUG:boto:Method: GET 2023-10-18T20:18:19.980 DEBUG:boto:Path: /bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.980 DEBUG:boto:Data: 2023-10-18T20:18:19.980 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:19.981 DEBUG:boto:Port: 8001 2023-10-18T20:18:19.982 DEBUG:boto:Params: {} 2023-10-18T20:18:19.982 DEBUG:boto:Token: None 2023-10-18T20:18:19.983 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d6c85fde9f21207-0065303d8b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.987 DEBUG:boto:b'bofrnf-48obj31000falseobj32023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.988 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:19.988 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:19.988 DEBUG:boto:path=/bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.989 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.989 DEBUG:boto:Method: GET 2023-10-18T20:18:19.990 DEBUG:boto:Path: /bofrnf-48/?prefix=obj3 2023-10-18T20:18:19.990 DEBUG:boto:Data: 2023-10-18T20:18:19.991 DEBUG:boto:Headers: {} 2023-10-18T20:18:19.991 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:19.991 DEBUG:boto:Port: 8000 2023-10-18T20:18:19.992 DEBUG:boto:Params: {} 2023-10-18T20:18:19.992 DEBUG:boto:Token: None 2023-10-18T20:18:19.993 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:19 GMT /bofrnf-48/ 2023-10-18T20:18:19.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ= 2023-10-18T20:18:19.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1mmEzR8iPu840bRnSgUU4qCwqpQ=', 'Content-Length': '0'} 2023-10-18T20:18:19.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045dee9a3c3ff8206-0065303d8b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:19.997 DEBUG:boto:b'bofrnf-48obj31000falseobj32023-10-18T20:17:00.305Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:19.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:18:19.998 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:19.998 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:19.999 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:19.999 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.000 DEBUG:boto:Method: GET 2023-10-18T20:18:20.000 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.001 DEBUG:boto:Data: 2023-10-18T20:18:20.001 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.002 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.002 DEBUG:boto:Params: {} 2023-10-18T20:18:20.003 DEBUG:boto:Token: None 2023-10-18T20:18:20.003 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T4j6jIbaogtUR1z9VkPlM17XXKw= 2023-10-18T20:18:20.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T4j6jIbaogtUR1z9VkPlM17XXKw=', 'Content-Length': '0'} 2023-10-18T20:18:20.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e2496846f9bbd8e-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.050 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:20.051 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:20.052 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.052 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.053 DEBUG:boto:Method: GET 2023-10-18T20:18:20.053 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.053 DEBUG:boto:Data: 2023-10-18T20:18:20.054 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.054 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.055 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.055 DEBUG:boto:Params: {} 2023-10-18T20:18:20.056 DEBUG:boto:Token: None 2023-10-18T20:18:20.056 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.057 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T4j6jIbaogtUR1z9VkPlM17XXKw= 2023-10-18T20:18:20.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T4j6jIbaogtUR1z9VkPlM17XXKw=', 'Content-Length': '0'} 2023-10-18T20:18:20.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a319325f0a15780-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.060 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:20.061 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:20.061 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.062 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.062 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.062 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.063 DEBUG:boto:Data: 2023-10-18T20:18:20.063 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.064 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.064 DEBUG:boto:Params: {} 2023-10-18T20:18:20.065 DEBUG:boto:Token: None 2023-10-18T20:18:20.065 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.066 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ai67HFsp9AzFNg2o/1IhXWs2Nl4= 2023-10-18T20:18:20.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ai67HFsp9AzFNg2o/1IhXWs2Nl4=', 'Content-Length': '0'} 2023-10-18T20:18:20.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d73658dd853275c4-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.069 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:20.069 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:20.070 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.070 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.070 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.071 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.071 DEBUG:boto:Data: 2023-10-18T20:18:20.072 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.072 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.073 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.073 DEBUG:boto:Params: {} 2023-10-18T20:18:20.073 DEBUG:boto:Token: None 2023-10-18T20:18:20.074 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.074 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ai67HFsp9AzFNg2o/1IhXWs2Nl4= 2023-10-18T20:18:20.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ai67HFsp9AzFNg2o/1IhXWs2Nl4=', 'Content-Length': '0'} 2023-10-18T20:18:20.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017da478517b2158f-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:18:20.078 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:20.078 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:20.079 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.079 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.080 DEBUG:boto:Method: GET 2023-10-18T20:18:20.080 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.080 DEBUG:boto:Data: 2023-10-18T20:18:20.081 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.081 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.082 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.082 DEBUG:boto:Params: {} 2023-10-18T20:18:20.083 DEBUG:boto:Token: None 2023-10-18T20:18:20.083 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.084 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T4j6jIbaogtUR1z9VkPlM17XXKw= 2023-10-18T20:18:20.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T4j6jIbaogtUR1z9VkPlM17XXKw=', 'Content-Length': '0'} 2023-10-18T20:18:20.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba0fe8fbecd518e2-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.087 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:20.087 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:20.088 DEBUG:boto:path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.088 DEBUG:boto:auth_path=/bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.088 DEBUG:boto:Method: GET 2023-10-18T20:18:20.089 DEBUG:boto:Path: /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.089 DEBUG:boto:Data: 2023-10-18T20:18:20.090 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.090 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.090 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.091 DEBUG:boto:Params: {} 2023-10-18T20:18:20.091 DEBUG:boto:Token: None 2023-10-18T20:18:20.092 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4?versionId=null 2023-10-18T20:18:20.093 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T4j6jIbaogtUR1z9VkPlM17XXKw= 2023-10-18T20:18:20.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T4j6jIbaogtUR1z9VkPlM17XXKw=', 'Content-Length': '0'} 2023-10-18T20:18:20.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000346f0df77c40691f-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.138 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.139 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.139 DEBUG:boto:path=/bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.140 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.140 DEBUG:boto:Method: GET 2023-10-18T20:18:20.140 DEBUG:boto:Path: /bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.141 DEBUG:boto:Data: 2023-10-18T20:18:20.141 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.142 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.143 DEBUG:boto:Params: {} 2023-10-18T20:18:20.143 DEBUG:boto:Token: None 2023-10-18T20:18:20.144 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063e14046a3025692-0065303d8c-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.148 DEBUG:boto:b'bofrnf-48obj41000falseobj42023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.149 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.149 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.149 DEBUG:boto:path=/bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.150 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.150 DEBUG:boto:Method: GET 2023-10-18T20:18:20.151 DEBUG:boto:Path: /bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.151 DEBUG:boto:Data: 2023-10-18T20:18:20.152 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.152 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.152 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.153 DEBUG:boto:Params: {} 2023-10-18T20:18:20.153 DEBUG:boto:Token: None 2023-10-18T20:18:20.154 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.154 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098398e0ff457a84f-0065303d8c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.158 DEBUG:boto:b'bofrnf-48obj41000falseobj42023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.159 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:20.159 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:20.159 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.160 DEBUG:boto:Path: /bofrnf-48/obj4 2023-10-18T20:18:20.160 DEBUG:boto:Data: 2023-10-18T20:18:20.161 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.161 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.161 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.162 DEBUG:boto:Params: {} 2023-10-18T20:18:20.162 DEBUG:boto:Token: None 2023-10-18T20:18:20.163 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4 2023-10-18T20:18:20.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M0BDMnYluS1exs5p+hCr0wCHBrM= 2023-10-18T20:18:20.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M0BDMnYluS1exs5p+hCr0wCHBrM=', 'Content-Length': '0'} 2023-10-18T20:18:20.166 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000262947fea7c6b581-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.166 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:20.167 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:20.167 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.168 DEBUG:boto:Path: /bofrnf-48/obj4 2023-10-18T20:18:20.168 DEBUG:boto:Data: 2023-10-18T20:18:20.168 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.169 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.169 DEBUG:boto:Params: {} 2023-10-18T20:18:20.170 DEBUG:boto:Token: None 2023-10-18T20:18:20.170 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4 2023-10-18T20:18:20.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M0BDMnYluS1exs5p+hCr0wCHBrM= 2023-10-18T20:18:20.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M0BDMnYluS1exs5p+hCr0wCHBrM=', 'Content-Length': '0'} 2023-10-18T20:18:20.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008eec346faf1e94df-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:18:20.174 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:20.175 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:20.175 DEBUG:boto:Method: GET 2023-10-18T20:18:20.176 DEBUG:boto:Path: /bofrnf-48/obj4 2023-10-18T20:18:20.176 DEBUG:boto:Data: 2023-10-18T20:18:20.176 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.177 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.177 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.178 DEBUG:boto:Params: {} 2023-10-18T20:18:20.178 DEBUG:boto:Token: None 2023-10-18T20:18:20.179 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4 2023-10-18T20:18:20.179 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DWWBwoh2WfGfNZr+upVBFObnvJM= 2023-10-18T20:18:20.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DWWBwoh2WfGfNZr+upVBFObnvJM=', 'Content-Length': '0'} 2023-10-18T20:18:20.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1d917195f952b6d-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.183 DEBUG:boto:path=/bofrnf-48/obj4 2023-10-18T20:18:20.183 DEBUG:boto:auth_path=/bofrnf-48/obj4 2023-10-18T20:18:20.184 DEBUG:boto:Method: GET 2023-10-18T20:18:20.184 DEBUG:boto:Path: /bofrnf-48/obj4 2023-10-18T20:18:20.184 DEBUG:boto:Data: 2023-10-18T20:18:20.185 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.186 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.186 DEBUG:boto:Params: {} 2023-10-18T20:18:20.187 DEBUG:boto:Token: None 2023-10-18T20:18:20.187 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj4 2023-10-18T20:18:20.188 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DWWBwoh2WfGfNZr+upVBFObnvJM= 2023-10-18T20:18:20.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DWWBwoh2WfGfNZr+upVBFObnvJM=', 'Content-Length': '0'} 2023-10-18T20:18:20.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e14b2cde17b44b66-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.191 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.191 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.192 DEBUG:boto:path=/bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.192 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.193 DEBUG:boto:Method: GET 2023-10-18T20:18:20.193 DEBUG:boto:Path: /bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.194 DEBUG:boto:Data: 2023-10-18T20:18:20.194 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.195 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.195 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.195 DEBUG:boto:Params: {} 2023-10-18T20:18:20.196 DEBUG:boto:Token: None 2023-10-18T20:18:20.196 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.197 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036fe25ac72bee511-0065303d8c-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.200 DEBUG:boto:b'bofrnf-48obj41000falseobj42023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.201 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.201 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.202 DEBUG:boto:path=/bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.202 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.203 DEBUG:boto:Method: GET 2023-10-18T20:18:20.203 DEBUG:boto:Path: /bofrnf-48/?prefix=obj4 2023-10-18T20:18:20.204 DEBUG:boto:Data: 2023-10-18T20:18:20.204 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.204 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.205 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.205 DEBUG:boto:Params: {} 2023-10-18T20:18:20.206 DEBUG:boto:Token: None 2023-10-18T20:18:20.206 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.207 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043b10b732a0a7445-0065303d8c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.210 DEBUG:boto:b'bofrnf-48obj41000falseobj42023-10-18T20:17:00.322Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:18:20.211 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.212 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.212 DEBUG:boto:path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.213 DEBUG:boto:auth_path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.213 DEBUG:boto:Method: GET 2023-10-18T20:18:20.213 DEBUG:boto:Path: /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.214 DEBUG:boto:Data: 2023-10-18T20:18:20.214 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.215 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.215 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.215 DEBUG:boto:Params: {} 2023-10-18T20:18:20.216 DEBUG:boto:Token: None 2023-10-18T20:18:20.216 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.217 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YULahUcYbr2g5AXrAi8Yws16NAA= 2023-10-18T20:18:20.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YULahUcYbr2g5AXrAi8Yws16NAA=', 'Content-Length': '0'} 2023-10-18T20:18:20.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069d6e6048fd47f9b-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.262 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.263 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.263 DEBUG:boto:path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.264 DEBUG:boto:auth_path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.264 DEBUG:boto:Method: GET 2023-10-18T20:18:20.264 DEBUG:boto:Path: /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.265 DEBUG:boto:Data: 2023-10-18T20:18:20.265 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.266 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.266 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.266 DEBUG:boto:Params: {} 2023-10-18T20:18:20.267 DEBUG:boto:Token: None 2023-10-18T20:18:20.267 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.268 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YULahUcYbr2g5AXrAi8Yws16NAA= 2023-10-18T20:18:20.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YULahUcYbr2g5AXrAi8Yws16NAA=', 'Content-Length': '0'} 2023-10-18T20:18:20.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b26f3df080dcbd3c-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.271 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.271 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.272 DEBUG:boto:path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.272 DEBUG:boto:auth_path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.272 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.273 DEBUG:boto:Path: /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.273 DEBUG:boto:Data: 2023-10-18T20:18:20.274 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.274 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.274 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.275 DEBUG:boto:Params: {} 2023-10-18T20:18:20.275 DEBUG:boto:Token: None 2023-10-18T20:18:20.276 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4tgBsHcX5yEXOOgLex9sVqlf7hw= 2023-10-18T20:18:20.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4tgBsHcX5yEXOOgLex9sVqlf7hw=', 'Content-Length': '0'} 2023-10-18T20:18:20.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e16c2ca58424e8c-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.279 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.279 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.280 DEBUG:boto:path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.280 DEBUG:boto:auth_path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.280 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.281 DEBUG:boto:Path: /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.281 DEBUG:boto:Data: 2023-10-18T20:18:20.281 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.282 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.283 DEBUG:boto:Params: {} 2023-10-18T20:18:20.283 DEBUG:boto:Token: None 2023-10-18T20:18:20.283 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4tgBsHcX5yEXOOgLex9sVqlf7hw= 2023-10-18T20:18:20.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4tgBsHcX5yEXOOgLex9sVqlf7hw=', 'Content-Length': '0'} 2023-10-18T20:18:20.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d609f3eef6cab96-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:18:20.287 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.288 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.288 DEBUG:boto:path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.288 DEBUG:boto:auth_path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.289 DEBUG:boto:Method: GET 2023-10-18T20:18:20.289 DEBUG:boto:Path: /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.290 DEBUG:boto:Data: 2023-10-18T20:18:20.290 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.291 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.291 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.291 DEBUG:boto:Params: {} 2023-10-18T20:18:20.292 DEBUG:boto:Token: None 2023-10-18T20:18:20.292 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.293 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YULahUcYbr2g5AXrAi8Yws16NAA= 2023-10-18T20:18:20.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YULahUcYbr2g5AXrAi8Yws16NAA=', 'Content-Length': '0'} 2023-10-18T20:18:20.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034033c1c30b2ae63-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.296 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.296 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.297 DEBUG:boto:path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.297 DEBUG:boto:auth_path=/bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.298 DEBUG:boto:Method: GET 2023-10-18T20:18:20.298 DEBUG:boto:Path: /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.299 DEBUG:boto:Data: 2023-10-18T20:18:20.299 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.300 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.300 DEBUG:boto:Params: {} 2023-10-18T20:18:20.300 DEBUG:boto:Token: None 2023-10-18T20:18:20.301 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5?versionId=null 2023-10-18T20:18:20.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YULahUcYbr2g5AXrAi8Yws16NAA= 2023-10-18T20:18:20.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YULahUcYbr2g5AXrAi8Yws16NAA=', 'Content-Length': '0'} 2023-10-18T20:18:20.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000307cfce2f989f7a6-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.346 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.347 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.347 DEBUG:boto:path=/bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.348 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.348 DEBUG:boto:Method: GET 2023-10-18T20:18:20.348 DEBUG:boto:Path: /bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.349 DEBUG:boto:Data: 2023-10-18T20:18:20.349 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.350 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.350 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.350 DEBUG:boto:Params: {} 2023-10-18T20:18:20.351 DEBUG:boto:Token: None 2023-10-18T20:18:20.351 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.352 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004342f85eb67530c-0065303d8c-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.355 DEBUG:boto:b'bofrnf-48obj51000falseobj52023-10-18T20:17:53.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.356 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.357 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.357 DEBUG:boto:path=/bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.357 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.358 DEBUG:boto:Method: GET 2023-10-18T20:18:20.358 DEBUG:boto:Path: /bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.359 DEBUG:boto:Data: 2023-10-18T20:18:20.359 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.359 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.360 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.360 DEBUG:boto:Params: {} 2023-10-18T20:18:20.361 DEBUG:boto:Token: None 2023-10-18T20:18:20.361 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.362 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dedad602e6c8daf2-0065303d8c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.365 DEBUG:boto:b'bofrnf-48obj51000falseobj52023-10-18T20:17:53.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.366 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.366 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.367 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.367 DEBUG:boto:Path: /bofrnf-48/obj5 2023-10-18T20:18:20.367 DEBUG:boto:Data: 2023-10-18T20:18:20.368 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.368 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.369 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.369 DEBUG:boto:Params: {} 2023-10-18T20:18:20.369 DEBUG:boto:Token: None 2023-10-18T20:18:20.370 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5 2023-10-18T20:18:20.370 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kdngrcmuNiDr0NWqmF+JxuPDn98= 2023-10-18T20:18:20.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kdngrcmuNiDr0NWqmF+JxuPDn98=', 'Content-Length': '0'} 2023-10-18T20:18:20.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004743c63b4808cdbf-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.373 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.374 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.374 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.375 DEBUG:boto:Path: /bofrnf-48/obj5 2023-10-18T20:18:20.375 DEBUG:boto:Data: 2023-10-18T20:18:20.375 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.376 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.376 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.377 DEBUG:boto:Params: {} 2023-10-18T20:18:20.377 DEBUG:boto:Token: None 2023-10-18T20:18:20.377 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5 2023-10-18T20:18:20.378 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kdngrcmuNiDr0NWqmF+JxuPDn98= 2023-10-18T20:18:20.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kdngrcmuNiDr0NWqmF+JxuPDn98=', 'Content-Length': '0'} 2023-10-18T20:18:20.380 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f594e1732c6b13ed-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:18:20.381 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.381 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.382 DEBUG:boto:Method: GET 2023-10-18T20:18:20.382 DEBUG:boto:Path: /bofrnf-48/obj5 2023-10-18T20:18:20.383 DEBUG:boto:Data: 2023-10-18T20:18:20.383 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.384 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.384 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.384 DEBUG:boto:Params: {} 2023-10-18T20:18:20.385 DEBUG:boto:Token: None 2023-10-18T20:18:20.385 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5 2023-10-18T20:18:20.386 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gFn+9FCcvhXSIcSyVKToEa2OlNo= 2023-10-18T20:18:20.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gFn+9FCcvhXSIcSyVKToEa2OlNo=', 'Content-Length': '0'} 2023-10-18T20:18:20.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013eeab5ee4af7c8e-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.430 DEBUG:boto:path=/bofrnf-48/obj5 2023-10-18T20:18:20.431 DEBUG:boto:auth_path=/bofrnf-48/obj5 2023-10-18T20:18:20.431 DEBUG:boto:Method: GET 2023-10-18T20:18:20.432 DEBUG:boto:Path: /bofrnf-48/obj5 2023-10-18T20:18:20.432 DEBUG:boto:Data: 2023-10-18T20:18:20.433 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.433 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.434 DEBUG:boto:Params: {} 2023-10-18T20:18:20.434 DEBUG:boto:Token: None 2023-10-18T20:18:20.435 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj5 2023-10-18T20:18:20.435 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gFn+9FCcvhXSIcSyVKToEa2OlNo= 2023-10-18T20:18:20.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gFn+9FCcvhXSIcSyVKToEa2OlNo=', 'Content-Length': '0'} 2023-10-18T20:18:20.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a14fdf4680634a3-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.439 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.439 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.440 DEBUG:boto:path=/bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.440 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.440 DEBUG:boto:Method: GET 2023-10-18T20:18:20.441 DEBUG:boto:Path: /bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.441 DEBUG:boto:Data: 2023-10-18T20:18:20.442 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.443 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.443 DEBUG:boto:Params: {} 2023-10-18T20:18:20.443 DEBUG:boto:Token: None 2023-10-18T20:18:20.444 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.444 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f6f7f58846777c0-0065303d8c-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.448 DEBUG:boto:b'bofrnf-48obj51000falseobj52023-10-18T20:17:53.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.449 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.449 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.450 DEBUG:boto:path=/bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.450 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.451 DEBUG:boto:Method: GET 2023-10-18T20:18:20.451 DEBUG:boto:Path: /bofrnf-48/?prefix=obj5 2023-10-18T20:18:20.453 DEBUG:boto:Data: 2023-10-18T20:18:20.454 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.455 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.455 DEBUG:boto:Params: {} 2023-10-18T20:18:20.456 DEBUG:boto:Token: None 2023-10-18T20:18:20.456 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.457 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079fc6f9af016592c-0065303d8c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.460 DEBUG:boto:b'bofrnf-48obj51000falseobj52023-10-18T20:17:53.623Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:18:20.461 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.461 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.462 DEBUG:boto:path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.462 DEBUG:boto:auth_path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.463 DEBUG:boto:Method: GET 2023-10-18T20:18:20.463 DEBUG:boto:Path: /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.464 DEBUG:boto:Data: 2023-10-18T20:18:20.464 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.464 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.465 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.465 DEBUG:boto:Params: {} 2023-10-18T20:18:20.466 DEBUG:boto:Token: None 2023-10-18T20:18:20.466 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.467 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X/2p2FOMi0YWIf+wGcpkR4pl0dw= 2023-10-18T20:18:20.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X/2p2FOMi0YWIf+wGcpkR4pl0dw=', 'Content-Length': '0'} 2023-10-18T20:18:20.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bd32ef0fc133fa1-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.470 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.471 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.471 DEBUG:boto:path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.472 DEBUG:boto:auth_path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.472 DEBUG:boto:Method: GET 2023-10-18T20:18:20.473 DEBUG:boto:Path: /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.473 DEBUG:boto:Data: 2023-10-18T20:18:20.474 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.475 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.475 DEBUG:boto:Params: {} 2023-10-18T20:18:20.475 DEBUG:boto:Token: None 2023-10-18T20:18:20.476 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X/2p2FOMi0YWIf+wGcpkR4pl0dw= 2023-10-18T20:18:20.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X/2p2FOMi0YWIf+wGcpkR4pl0dw=', 'Content-Length': '0'} 2023-10-18T20:18:20.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0b84d5404dbac73-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.480 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.480 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.480 DEBUG:boto:path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.481 DEBUG:boto:auth_path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.481 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.482 DEBUG:boto:Path: /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.482 DEBUG:boto:Data: 2023-10-18T20:18:20.482 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.483 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.483 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.484 DEBUG:boto:Params: {} 2023-10-18T20:18:20.484 DEBUG:boto:Token: None 2023-10-18T20:18:20.484 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.485 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D2uxjWywpBDQkh/JrNev6Z8RMjs= 2023-10-18T20:18:20.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D2uxjWywpBDQkh/JrNev6Z8RMjs=', 'Content-Length': '0'} 2023-10-18T20:18:20.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e9530e88f276d50-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.488 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.488 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.489 DEBUG:boto:path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.489 DEBUG:boto:auth_path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.489 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.490 DEBUG:boto:Path: /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.490 DEBUG:boto:Data: 2023-10-18T20:18:20.491 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.491 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.492 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.492 DEBUG:boto:Params: {} 2023-10-18T20:18:20.492 DEBUG:boto:Token: None 2023-10-18T20:18:20.493 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.493 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D2uxjWywpBDQkh/JrNev6Z8RMjs= 2023-10-18T20:18:20.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D2uxjWywpBDQkh/JrNev6Z8RMjs=', 'Content-Length': '0'} 2023-10-18T20:18:20.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006eba706ce96e0054-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:18:20.497 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.497 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.498 DEBUG:boto:path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.498 DEBUG:boto:auth_path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.499 DEBUG:boto:Method: GET 2023-10-18T20:18:20.499 DEBUG:boto:Path: /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.499 DEBUG:boto:Data: 2023-10-18T20:18:20.500 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.500 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.501 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.501 DEBUG:boto:Params: {} 2023-10-18T20:18:20.501 DEBUG:boto:Token: None 2023-10-18T20:18:20.502 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.502 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X/2p2FOMi0YWIf+wGcpkR4pl0dw= 2023-10-18T20:18:20.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X/2p2FOMi0YWIf+wGcpkR4pl0dw=', 'Content-Length': '0'} 2023-10-18T20:18:20.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006873e76980efa19-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.505 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.506 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.506 DEBUG:boto:path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.507 DEBUG:boto:auth_path=/bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.507 DEBUG:boto:Method: GET 2023-10-18T20:18:20.507 DEBUG:boto:Path: /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.508 DEBUG:boto:Data: 2023-10-18T20:18:20.508 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.509 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.509 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.510 DEBUG:boto:Params: {} 2023-10-18T20:18:20.510 DEBUG:boto:Token: None 2023-10-18T20:18:20.511 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6?versionId=null 2023-10-18T20:18:20.511 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X/2p2FOMi0YWIf+wGcpkR4pl0dw= 2023-10-18T20:18:20.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X/2p2FOMi0YWIf+wGcpkR4pl0dw=', 'Content-Length': '0'} 2023-10-18T20:18:20.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fde3f0c5881c8026-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.554 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.555 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.555 DEBUG:boto:path=/bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.556 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.556 DEBUG:boto:Method: GET 2023-10-18T20:18:20.556 DEBUG:boto:Path: /bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.557 DEBUG:boto:Data: 2023-10-18T20:18:20.557 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.558 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.558 DEBUG:boto:Params: {} 2023-10-18T20:18:20.559 DEBUG:boto:Token: None 2023-10-18T20:18:20.559 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae4448e0d10af3fb-0065303d8c-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.563 DEBUG:boto:b'bofrnf-48obj61000falseobj62023-10-18T20:17:53.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.564 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.565 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.565 DEBUG:boto:path=/bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.565 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.566 DEBUG:boto:Method: GET 2023-10-18T20:18:20.566 DEBUG:boto:Path: /bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.567 DEBUG:boto:Data: 2023-10-18T20:18:20.567 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.567 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.568 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.568 DEBUG:boto:Params: {} 2023-10-18T20:18:20.569 DEBUG:boto:Token: None 2023-10-18T20:18:20.569 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.570 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6ba59468d90a0ae-0065303d8c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.573 DEBUG:boto:b'bofrnf-48obj61000falseobj62023-10-18T20:17:53.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.574 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.574 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.575 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.575 DEBUG:boto:Path: /bofrnf-48/obj6 2023-10-18T20:18:20.576 DEBUG:boto:Data: 2023-10-18T20:18:20.576 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.576 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.577 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.577 DEBUG:boto:Params: {} 2023-10-18T20:18:20.578 DEBUG:boto:Token: None 2023-10-18T20:18:20.578 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6 2023-10-18T20:18:20.578 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PgdJqiSlY5VDhqVfQ1l7XTGZHng= 2023-10-18T20:18:20.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PgdJqiSlY5VDhqVfQ1l7XTGZHng=', 'Content-Length': '0'} 2023-10-18T20:18:20.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cfbd1475ce2c69b-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.581 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.582 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.582 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.583 DEBUG:boto:Path: /bofrnf-48/obj6 2023-10-18T20:18:20.583 DEBUG:boto:Data: 2023-10-18T20:18:20.584 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.584 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.584 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.585 DEBUG:boto:Params: {} 2023-10-18T20:18:20.585 DEBUG:boto:Token: None 2023-10-18T20:18:20.585 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6 2023-10-18T20:18:20.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PgdJqiSlY5VDhqVfQ1l7XTGZHng= 2023-10-18T20:18:20.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PgdJqiSlY5VDhqVfQ1l7XTGZHng=', 'Content-Length': '0'} 2023-10-18T20:18:20.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000536a9aa9a9f6cda7-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:18:20.589 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.590 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.590 DEBUG:boto:Method: GET 2023-10-18T20:18:20.591 DEBUG:boto:Path: /bofrnf-48/obj6 2023-10-18T20:18:20.591 DEBUG:boto:Data: 2023-10-18T20:18:20.591 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.592 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.592 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.593 DEBUG:boto:Params: {} 2023-10-18T20:18:20.593 DEBUG:boto:Token: None 2023-10-18T20:18:20.594 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6 2023-10-18T20:18:20.594 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hw0D15iuo/Mm6PfikaEEgqloMdU= 2023-10-18T20:18:20.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hw0D15iuo/Mm6PfikaEEgqloMdU=', 'Content-Length': '0'} 2023-10-18T20:18:20.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088717e7284857c18-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.638 DEBUG:boto:path=/bofrnf-48/obj6 2023-10-18T20:18:20.639 DEBUG:boto:auth_path=/bofrnf-48/obj6 2023-10-18T20:18:20.639 DEBUG:boto:Method: GET 2023-10-18T20:18:20.640 DEBUG:boto:Path: /bofrnf-48/obj6 2023-10-18T20:18:20.640 DEBUG:boto:Data: 2023-10-18T20:18:20.640 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.641 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.642 DEBUG:boto:Params: {} 2023-10-18T20:18:20.642 DEBUG:boto:Token: None 2023-10-18T20:18:20.643 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj6 2023-10-18T20:18:20.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hw0D15iuo/Mm6PfikaEEgqloMdU= 2023-10-18T20:18:20.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hw0D15iuo/Mm6PfikaEEgqloMdU=', 'Content-Length': '0'} 2023-10-18T20:18:20.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e31b1ab48483d0c3-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.647 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.647 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.648 DEBUG:boto:path=/bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.648 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.648 DEBUG:boto:Method: GET 2023-10-18T20:18:20.649 DEBUG:boto:Path: /bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.649 DEBUG:boto:Data: 2023-10-18T20:18:20.650 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.650 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.650 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.651 DEBUG:boto:Params: {} 2023-10-18T20:18:20.651 DEBUG:boto:Token: None 2023-10-18T20:18:20.652 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d89b8d2db638c7f8-0065303d8c-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.657 DEBUG:boto:b'bofrnf-48obj61000falseobj62023-10-18T20:17:53.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.658 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.658 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.659 DEBUG:boto:path=/bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.659 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.659 DEBUG:boto:Method: GET 2023-10-18T20:18:20.660 DEBUG:boto:Path: /bofrnf-48/?prefix=obj6 2023-10-18T20:18:20.660 DEBUG:boto:Data: 2023-10-18T20:18:20.660 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.661 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.662 DEBUG:boto:Params: {} 2023-10-18T20:18:20.662 DEBUG:boto:Token: None 2023-10-18T20:18:20.663 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.663 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000848507d6e1795af9-0065303d8c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.667 DEBUG:boto:b'bofrnf-48obj61000falseobj62023-10-18T20:17:53.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:18:20.668 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.669 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.669 DEBUG:boto:path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.669 DEBUG:boto:auth_path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.670 DEBUG:boto:Method: GET 2023-10-18T20:18:20.670 DEBUG:boto:Path: /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.671 DEBUG:boto:Data: 2023-10-18T20:18:20.671 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.672 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.672 DEBUG:boto:Params: {} 2023-10-18T20:18:20.673 DEBUG:boto:Token: None 2023-10-18T20:18:20.673 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.674 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vFYCQk3YBWZvfvOmaM52GK0OUig= 2023-10-18T20:18:20.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vFYCQk3YBWZvfvOmaM52GK0OUig=', 'Content-Length': '0'} 2023-10-18T20:18:20.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd7a6641e7b92b37-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.677 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.678 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.678 DEBUG:boto:path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.679 DEBUG:boto:auth_path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.679 DEBUG:boto:Method: GET 2023-10-18T20:18:20.679 DEBUG:boto:Path: /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.680 DEBUG:boto:Data: 2023-10-18T20:18:20.680 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.681 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.682 DEBUG:boto:Params: {} 2023-10-18T20:18:20.683 DEBUG:boto:Token: None 2023-10-18T20:18:20.684 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.685 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vFYCQk3YBWZvfvOmaM52GK0OUig= 2023-10-18T20:18:20.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vFYCQk3YBWZvfvOmaM52GK0OUig=', 'Content-Length': '0'} 2023-10-18T20:18:20.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073226c4de560f6e3-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.730 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.731 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.731 DEBUG:boto:path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.731 DEBUG:boto:auth_path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.732 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.732 DEBUG:boto:Path: /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.733 DEBUG:boto:Data: 2023-10-18T20:18:20.733 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.734 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.734 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.734 DEBUG:boto:Params: {} 2023-10-18T20:18:20.735 DEBUG:boto:Token: None 2023-10-18T20:18:20.735 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.736 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A7KwzRE/NNBjLI6DzVQ6PbR9VfY= 2023-10-18T20:18:20.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A7KwzRE/NNBjLI6DzVQ6PbR9VfY=', 'Content-Length': '0'} 2023-10-18T20:18:20.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d46af82e23eeb1ab-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.739 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.739 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.740 DEBUG:boto:path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.740 DEBUG:boto:auth_path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.740 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.741 DEBUG:boto:Path: /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.741 DEBUG:boto:Data: 2023-10-18T20:18:20.742 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.742 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.742 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.743 DEBUG:boto:Params: {} 2023-10-18T20:18:20.743 DEBUG:boto:Token: None 2023-10-18T20:18:20.744 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.744 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A7KwzRE/NNBjLI6DzVQ6PbR9VfY= 2023-10-18T20:18:20.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A7KwzRE/NNBjLI6DzVQ6PbR9VfY=', 'Content-Length': '0'} 2023-10-18T20:18:20.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecaef022ef771807-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:18:20.748 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.748 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.749 DEBUG:boto:path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.749 DEBUG:boto:auth_path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.750 DEBUG:boto:Method: GET 2023-10-18T20:18:20.750 DEBUG:boto:Path: /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.751 DEBUG:boto:Data: 2023-10-18T20:18:20.751 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.752 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.752 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.753 DEBUG:boto:Params: {} 2023-10-18T20:18:20.753 DEBUG:boto:Token: None 2023-10-18T20:18:20.754 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.754 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vFYCQk3YBWZvfvOmaM52GK0OUig= 2023-10-18T20:18:20.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vFYCQk3YBWZvfvOmaM52GK0OUig=', 'Content-Length': '0'} 2023-10-18T20:18:20.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069678a5cf3bd4a3e-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.757 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.758 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.758 DEBUG:boto:path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.759 DEBUG:boto:auth_path=/bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.759 DEBUG:boto:Method: GET 2023-10-18T20:18:20.760 DEBUG:boto:Path: /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.760 DEBUG:boto:Data: 2023-10-18T20:18:20.760 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.761 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.762 DEBUG:boto:Params: {} 2023-10-18T20:18:20.762 DEBUG:boto:Token: None 2023-10-18T20:18:20.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7?versionId=null 2023-10-18T20:18:20.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vFYCQk3YBWZvfvOmaM52GK0OUig= 2023-10-18T20:18:20.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vFYCQk3YBWZvfvOmaM52GK0OUig=', 'Content-Length': '0'} 2023-10-18T20:18:20.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008277b60d78c5d9c5-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.766 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.767 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.767 DEBUG:boto:path=/bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.767 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.768 DEBUG:boto:Method: GET 2023-10-18T20:18:20.768 DEBUG:boto:Path: /bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.769 DEBUG:boto:Data: 2023-10-18T20:18:20.769 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.769 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.770 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.770 DEBUG:boto:Params: {} 2023-10-18T20:18:20.771 DEBUG:boto:Token: None 2023-10-18T20:18:20.771 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d386b7fd153ffcf-0065303d8c-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.775 DEBUG:boto:b'bofrnf-48obj71000falseobj72023-10-18T20:17:53.705Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.776 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.777 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.777 DEBUG:boto:path=/bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.777 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.778 DEBUG:boto:Method: GET 2023-10-18T20:18:20.778 DEBUG:boto:Path: /bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.779 DEBUG:boto:Data: 2023-10-18T20:18:20.779 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.780 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.780 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.780 DEBUG:boto:Params: {} 2023-10-18T20:18:20.781 DEBUG:boto:Token: None 2023-10-18T20:18:20.781 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001de2d5afb6eb352e-0065303d8c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.785 DEBUG:boto:b'bofrnf-48obj71000falseobj72023-10-18T20:17:53.705Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.786 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.786 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.787 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.787 DEBUG:boto:Path: /bofrnf-48/obj7 2023-10-18T20:18:20.788 DEBUG:boto:Data: 2023-10-18T20:18:20.788 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.788 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.789 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.789 DEBUG:boto:Params: {} 2023-10-18T20:18:20.790 DEBUG:boto:Token: None 2023-10-18T20:18:20.790 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7 2023-10-18T20:18:20.791 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4XdUhwgxqX9EEBXjUF67W7Ef0cM= 2023-10-18T20:18:20.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4XdUhwgxqX9EEBXjUF67W7Ef0cM=', 'Content-Length': '0'} 2023-10-18T20:18:20.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c097e6dd670f28eb-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.793 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.794 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.794 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.795 DEBUG:boto:Path: /bofrnf-48/obj7 2023-10-18T20:18:20.795 DEBUG:boto:Data: 2023-10-18T20:18:20.796 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.796 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.797 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.797 DEBUG:boto:Params: {} 2023-10-18T20:18:20.797 DEBUG:boto:Token: None 2023-10-18T20:18:20.798 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7 2023-10-18T20:18:20.798 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4XdUhwgxqX9EEBXjUF67W7Ef0cM= 2023-10-18T20:18:20.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4XdUhwgxqX9EEBXjUF67W7Ef0cM=', 'Content-Length': '0'} 2023-10-18T20:18:20.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015c7f7371f4ef2e3-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:18:20.801 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.802 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.802 DEBUG:boto:Method: GET 2023-10-18T20:18:20.803 DEBUG:boto:Path: /bofrnf-48/obj7 2023-10-18T20:18:20.803 DEBUG:boto:Data: 2023-10-18T20:18:20.804 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.804 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.804 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.805 DEBUG:boto:Params: {} 2023-10-18T20:18:20.805 DEBUG:boto:Token: None 2023-10-18T20:18:20.806 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7 2023-10-18T20:18:20.806 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AYVQxFshjAKplG3AfPM6Ng8v6Xc= 2023-10-18T20:18:20.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AYVQxFshjAKplG3AfPM6Ng8v6Xc=', 'Content-Length': '0'} 2023-10-18T20:18:20.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076ae4731fcd0ec6e-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.850 DEBUG:boto:path=/bofrnf-48/obj7 2023-10-18T20:18:20.851 DEBUG:boto:auth_path=/bofrnf-48/obj7 2023-10-18T20:18:20.851 DEBUG:boto:Method: GET 2023-10-18T20:18:20.852 DEBUG:boto:Path: /bofrnf-48/obj7 2023-10-18T20:18:20.852 DEBUG:boto:Data: 2023-10-18T20:18:20.853 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.853 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.854 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.854 DEBUG:boto:Params: {} 2023-10-18T20:18:20.855 DEBUG:boto:Token: None 2023-10-18T20:18:20.856 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj7 2023-10-18T20:18:20.856 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AYVQxFshjAKplG3AfPM6Ng8v6Xc= 2023-10-18T20:18:20.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AYVQxFshjAKplG3AfPM6Ng8v6Xc=', 'Content-Length': '0'} 2023-10-18T20:18:20.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a4a92e6a2bcc385-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.862 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.863 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.863 DEBUG:boto:path=/bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.864 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.864 DEBUG:boto:Method: GET 2023-10-18T20:18:20.864 DEBUG:boto:Path: /bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.865 DEBUG:boto:Data: 2023-10-18T20:18:20.865 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.866 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.866 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.867 DEBUG:boto:Params: {} 2023-10-18T20:18:20.867 DEBUG:boto:Token: None 2023-10-18T20:18:20.868 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.868 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bc66a604fcbe899-0065303d8c-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.872 DEBUG:boto:b'bofrnf-48obj71000falseobj72023-10-18T20:17:53.705Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.872 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.873 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.873 DEBUG:boto:path=/bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.874 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.874 DEBUG:boto:Method: GET 2023-10-18T20:18:20.874 DEBUG:boto:Path: /bofrnf-48/?prefix=obj7 2023-10-18T20:18:20.875 DEBUG:boto:Data: 2023-10-18T20:18:20.875 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.876 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.876 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.877 DEBUG:boto:Params: {} 2023-10-18T20:18:20.877 DEBUG:boto:Token: None 2023-10-18T20:18:20.877 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.878 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080b3d78c3b4b4206-0065303d8c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.922 DEBUG:boto:b'bofrnf-48obj71000falseobj72023-10-18T20:17:53.705Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:18:20.924 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:20.924 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:20.925 DEBUG:boto:path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.925 DEBUG:boto:auth_path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.925 DEBUG:boto:Method: GET 2023-10-18T20:18:20.926 DEBUG:boto:Path: /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.926 DEBUG:boto:Data: 2023-10-18T20:18:20.927 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.927 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.927 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.928 DEBUG:boto:Params: {} 2023-10-18T20:18:20.928 DEBUG:boto:Token: None 2023-10-18T20:18:20.929 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vRB2BtM7bfq9CjSxIrQZBW4LAuc= 2023-10-18T20:18:20.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vRB2BtM7bfq9CjSxIrQZBW4LAuc=', 'Content-Length': '0'} 2023-10-18T20:18:20.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070f576134261d83f-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.932 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:20.933 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:20.933 DEBUG:boto:path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.934 DEBUG:boto:auth_path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.934 DEBUG:boto:Method: GET 2023-10-18T20:18:20.935 DEBUG:boto:Path: /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.935 DEBUG:boto:Data: 2023-10-18T20:18:20.936 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.936 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.937 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.937 DEBUG:boto:Params: {} 2023-10-18T20:18:20.937 DEBUG:boto:Token: None 2023-10-18T20:18:20.938 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.938 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vRB2BtM7bfq9CjSxIrQZBW4LAuc= 2023-10-18T20:18:20.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vRB2BtM7bfq9CjSxIrQZBW4LAuc=', 'Content-Length': '0'} 2023-10-18T20:18:20.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1e2309caa44e663-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.941 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:20.942 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:20.942 DEBUG:boto:path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.943 DEBUG:boto:auth_path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.943 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.944 DEBUG:boto:Path: /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.944 DEBUG:boto:Data: 2023-10-18T20:18:20.944 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.945 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.945 DEBUG:boto:Params: {} 2023-10-18T20:18:20.946 DEBUG:boto:Token: None 2023-10-18T20:18:20.946 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.947 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4hqKHrKRQn+4DuKUlMXeMrT51MM= 2023-10-18T20:18:20.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4hqKHrKRQn+4DuKUlMXeMrT51MM=', 'Content-Length': '0'} 2023-10-18T20:18:20.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030b18b9bedc3e456-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.951 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:20.951 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:20.952 DEBUG:boto:path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.953 DEBUG:boto:auth_path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.953 DEBUG:boto:Method: HEAD 2023-10-18T20:18:20.954 DEBUG:boto:Path: /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.954 DEBUG:boto:Data: 2023-10-18T20:18:20.955 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.955 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.956 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.956 DEBUG:boto:Params: {} 2023-10-18T20:18:20.957 DEBUG:boto:Token: None 2023-10-18T20:18:20.957 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.958 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4hqKHrKRQn+4DuKUlMXeMrT51MM= 2023-10-18T20:18:20.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4hqKHrKRQn+4DuKUlMXeMrT51MM=', 'Content-Length': '0'} 2023-10-18T20:18:20.960 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d70e27226842793-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:18:20.962 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:20.962 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:20.963 DEBUG:boto:path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.963 DEBUG:boto:auth_path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.964 DEBUG:boto:Method: GET 2023-10-18T20:18:20.964 DEBUG:boto:Path: /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.965 DEBUG:boto:Data: 2023-10-18T20:18:20.965 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.965 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.966 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.966 DEBUG:boto:Params: {} 2023-10-18T20:18:20.967 DEBUG:boto:Token: None 2023-10-18T20:18:20.968 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vRB2BtM7bfq9CjSxIrQZBW4LAuc= 2023-10-18T20:18:20.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vRB2BtM7bfq9CjSxIrQZBW4LAuc=', 'Content-Length': '0'} 2023-10-18T20:18:20.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068e4bce55ec9bead-0065303d8c-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.971 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:20.972 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:20.972 DEBUG:boto:path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.973 DEBUG:boto:auth_path=/bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.973 DEBUG:boto:Method: GET 2023-10-18T20:18:20.974 DEBUG:boto:Path: /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.974 DEBUG:boto:Data: 2023-10-18T20:18:20.975 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.975 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.975 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.976 DEBUG:boto:Params: {} 2023-10-18T20:18:20.976 DEBUG:boto:Token: None 2023-10-18T20:18:20.977 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/obj8?versionId=null 2023-10-18T20:18:20.977 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vRB2BtM7bfq9CjSxIrQZBW4LAuc= 2023-10-18T20:18:20.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vRB2BtM7bfq9CjSxIrQZBW4LAuc=', 'Content-Length': '0'} 2023-10-18T20:18:20.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c433bff9323b6a4-0065303d8c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.981 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.981 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.981 DEBUG:boto:path=/bofrnf-48/?prefix=obj8 2023-10-18T20:18:20.982 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj8 2023-10-18T20:18:20.982 DEBUG:boto:Method: GET 2023-10-18T20:18:20.983 DEBUG:boto:Path: /bofrnf-48/?prefix=obj8 2023-10-18T20:18:20.983 DEBUG:boto:Data: 2023-10-18T20:18:20.984 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.984 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:20.985 DEBUG:boto:Port: 8001 2023-10-18T20:18:20.985 DEBUG:boto:Params: {} 2023-10-18T20:18:20.985 DEBUG:boto:Token: None 2023-10-18T20:18:20.986 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.987 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:20.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091136b195f6d6239-0065303d8c-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:20.990 DEBUG:boto:b'bofrnf-48obj81000falseobj82023-10-18T20:17:53.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:20.991 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:20.992 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:20.992 DEBUG:boto:path=/bofrnf-48/?prefix=obj8 2023-10-18T20:18:20.993 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj8 2023-10-18T20:18:20.993 DEBUG:boto:Method: GET 2023-10-18T20:18:20.993 DEBUG:boto:Path: /bofrnf-48/?prefix=obj8 2023-10-18T20:18:20.994 DEBUG:boto:Data: 2023-10-18T20:18:20.994 DEBUG:boto:Headers: {} 2023-10-18T20:18:20.995 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:20.995 DEBUG:boto:Port: 8000 2023-10-18T20:18:20.996 DEBUG:boto:Params: {} 2023-10-18T20:18:20.996 DEBUG:boto:Token: None 2023-10-18T20:18:20.997 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:20 GMT /bofrnf-48/ 2023-10-18T20:18:20.997 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10= 2023-10-18T20:18:20.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B445pqYuVOZqOqw/9X20D3bHQ10=', 'Content-Length': '0'} 2023-10-18T20:18:21.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7af6eafbe87180-0065303d8c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:21.001 DEBUG:boto:b'bofrnf-48obj81000falseobj82023-10-18T20:17:53.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:21.001 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:21.002 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:21.002 DEBUG:boto:Method: HEAD 2023-10-18T20:18:21.002 DEBUG:boto:Path: /bofrnf-48/obj8 2023-10-18T20:18:21.003 DEBUG:boto:Data: 2023-10-18T20:18:21.003 DEBUG:boto:Headers: {} 2023-10-18T20:18:21.004 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:21.004 DEBUG:boto:Port: 8001 2023-10-18T20:18:21.005 DEBUG:boto:Params: {} 2023-10-18T20:18:21.005 DEBUG:boto:Token: None 2023-10-18T20:18:21.005 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:21 GMT /bofrnf-48/obj8 2023-10-18T20:18:21.006 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MehiKIJr+ZO0KDBXies0myqxObI= 2023-10-18T20:18:21.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MehiKIJr+ZO0KDBXies0myqxObI=', 'Content-Length': '0'} 2023-10-18T20:18:21.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4891b97f8c41c05-0065303d8d-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:21.009 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:21.010 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:21.010 DEBUG:boto:Method: HEAD 2023-10-18T20:18:21.010 DEBUG:boto:Path: /bofrnf-48/obj8 2023-10-18T20:18:21.011 DEBUG:boto:Data: 2023-10-18T20:18:21.011 DEBUG:boto:Headers: {} 2023-10-18T20:18:21.012 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:21.012 DEBUG:boto:Port: 8000 2023-10-18T20:18:21.012 DEBUG:boto:Params: {} 2023-10-18T20:18:21.013 DEBUG:boto:Token: None 2023-10-18T20:18:21.013 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:21 GMT /bofrnf-48/obj8 2023-10-18T20:18:21.013 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MehiKIJr+ZO0KDBXies0myqxObI= 2023-10-18T20:18:21.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MehiKIJr+ZO0KDBXies0myqxObI=', 'Content-Length': '0'} 2023-10-18T20:18:21.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001241135493b5b23-0065303d8d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:21.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:18:21.017 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:21.017 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:21.018 DEBUG:boto:Method: GET 2023-10-18T20:18:21.018 DEBUG:boto:Path: /bofrnf-48/obj8 2023-10-18T20:18:21.019 DEBUG:boto:Data: 2023-10-18T20:18:21.019 DEBUG:boto:Headers: {} 2023-10-18T20:18:21.020 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:21.021 DEBUG:boto:Port: 8001 2023-10-18T20:18:21.021 DEBUG:boto:Params: {} 2023-10-18T20:18:21.022 DEBUG:boto:Token: None 2023-10-18T20:18:21.023 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:21 GMT /bofrnf-48/obj8 2023-10-18T20:18:21.023 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7/d/h9HCqZyF2vLHIv0OOzhmB8o= 2023-10-18T20:18:21.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7/d/h9HCqZyF2vLHIv0OOzhmB8o=', 'Content-Length': '0'} 2023-10-18T20:18:21.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-48:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.17'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002742f9f69f461052-0065303d8d-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:21.027 DEBUG:boto:path=/bofrnf-48/obj8 2023-10-18T20:18:21.027 DEBUG:boto:auth_path=/bofrnf-48/obj8 2023-10-18T20:18:21.028 DEBUG:boto:Method: GET 2023-10-18T20:18:21.028 DEBUG:boto:Path: /bofrnf-48/obj8 2023-10-18T20:18:21.029 DEBUG:boto:Data: 2023-10-18T20:18:21.029 DEBUG:boto:Headers: {} 2023-10-18T20:18:21.029 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:21.030 DEBUG:boto:Port: 8000 2023-10-18T20:18:21.030 DEBUG:boto:Params: {} 2023-10-18T20:18:21.031 DEBUG:boto:Token: None 2023-10-18T20:18:21.031 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:21 GMT /bofrnf-48/obj8 2023-10-18T20:18:21.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7/d/h9HCqZyF2vLHIv0OOzhmB8o= 2023-10-18T20:18:21.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7/d/h9HCqZyF2vLHIv0OOzhmB8o=', 'Content-Length': '0'} 2023-10-18T20:18:21.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6e8d7036558b080-0065303d8d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:21.034 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:21.035 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:21.035 DEBUG:boto:path=/bofrnf-48/?prefix=obj8 2023-10-18T20:18:21.036 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj8 2023-10-18T20:18:21.036 DEBUG:boto:Method: GET 2023-10-18T20:18:21.037 DEBUG:boto:Path: /bofrnf-48/?prefix=obj8 2023-10-18T20:18:21.038 DEBUG:boto:Data: 2023-10-18T20:18:21.038 DEBUG:boto:Headers: {} 2023-10-18T20:18:21.039 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:21.040 DEBUG:boto:Port: 8001 2023-10-18T20:18:21.041 DEBUG:boto:Params: {} 2023-10-18T20:18:21.041 DEBUG:boto:Token: None 2023-10-18T20:18:21.042 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:21 GMT /bofrnf-48/ 2023-10-18T20:18:21.043 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YrGbGQgn2UYI8pADGm20LqwwpXQ= 2023-10-18T20:18:21.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YrGbGQgn2UYI8pADGm20LqwwpXQ=', 'Content-Length': '0'} 2023-10-18T20:18:21.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021b2811e95ac942d-0065303d8d-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:21.090 DEBUG:boto:b'bofrnf-48obj81000falseobj82023-10-18T20:17:53.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:21.092 DEBUG:boto:path=/bofrnf-48/ 2023-10-18T20:18:21.092 DEBUG:boto:auth_path=/bofrnf-48/ 2023-10-18T20:18:21.092 DEBUG:boto:path=/bofrnf-48/?prefix=obj8 2023-10-18T20:18:21.093 DEBUG:boto:auth_path=/bofrnf-48/?prefix=obj8 2023-10-18T20:18:21.093 DEBUG:boto:Method: GET 2023-10-18T20:18:21.094 DEBUG:boto:Path: /bofrnf-48/?prefix=obj8 2023-10-18T20:18:21.094 DEBUG:boto:Data: 2023-10-18T20:18:21.094 DEBUG:boto:Headers: {} 2023-10-18T20:18:21.095 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:21.095 DEBUG:boto:Port: 8000 2023-10-18T20:18:21.096 DEBUG:boto:Params: {} 2023-10-18T20:18:21.096 DEBUG:boto:Token: None 2023-10-18T20:18:21.097 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:21 GMT /bofrnf-48/ 2023-10-18T20:18:21.097 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YrGbGQgn2UYI8pADGm20LqwwpXQ= 2023-10-18T20:18:21.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YrGbGQgn2UYI8pADGm20LqwwpXQ=', 'Content-Length': '0'} 2023-10-18T20:18:21.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8acdaf1360356de-0065303d8d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:21.101 DEBUG:boto:b'bofrnf-48obj81000falseobj82023-10-18T20:17:53.723Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:21.101 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-48 zones={a2, a1} 2023-10-18T20:18:21.102 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-49 2023-10-18T20:18:21.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:18:21.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:18:21.103 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:18:21.167 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:18:21.167 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:18:21.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:21.203+0000 7fd7cf3b7780 1 waiting for incremental sync to catch up: 2023-10-18T20:18:21.206 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000003.32.1, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:21.206 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000004.34.6, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:26.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:26.205+0000 7fd7cf3b7780 1 waiting for incremental sync to catch up: 2023-10-18T20:18:26.208 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000003.32.1, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:26.208 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000004.34.6, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:31.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:31.206+0000 7fd7cf3b7780 1 waiting for incremental sync to catch up: 2023-10-18T20:18:31.209 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000003.32.1, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:31.209 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000004.34.6, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:36.210 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:36.208+0000 7fd7cf3b7780 1 waiting for incremental sync to catch up: 2023-10-18T20:18:36.210 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000003.32.1, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:36.211 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000004.34.6, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:41.211 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:41.209+0000 7fd7cf3b7780 1 bucket sync caught up with source: 2023-10-18T20:18:41.211 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000004.34.6, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:41.212 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.39.6, 00000000002.27.1, 00000000004.36.6, 00000000002.36.1, 00000000004.32.6, 00000000002.15.1, 00000000002.37.1, 00000000004.34.6, 00000000002.30.1, 00000000002.43.1, 00000000002.28.1] 2023-10-18T20:18:41.212 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:41.209+0000 7fd7cf3b7780 0 bucket checkpoint complete 2023-10-18T20:18:41.227 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-49 2023-10-18T20:18:41.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:18:41.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:18:41.229 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:18:41.291 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:18:41.291 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:18:41.328 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:41.326+0000 7f3f3a178780 1 bucket sync caught up with source: 2023-10-18T20:18:41.328 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000004.692.1, 00000000002.199.1, 00000000004.415.1, 00000000002.939.1, 00000000004.919.1, 00000000002.164.1, 00000000002.460.1, 00000000004.416.1, 00000000002.920.1, 00000000002.667.1, 00000000002.913.1] 2023-10-18T20:18:41.328 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000004.692.1, 00000000002.199.1, 00000000004.415.1, 00000000002.939.1, 00000000004.919.1, 00000000002.164.1, 00000000002.460.1, 00000000004.416.1, 00000000002.920.1, 00000000002.667.1, 00000000002.913.1] 2023-10-18T20:18:41.329 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:41.326+0000 7f3f3a178780 0 bucket checkpoint complete 2023-10-18T20:18:41.341 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-49 zones={a2, a1} 2023-10-18T20:18:41.341 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.342 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.342 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.343 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:18:41.343 DEBUG:boto:Data: 2023-10-18T20:18:41.343 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.344 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.344 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.344 DEBUG:boto:Params: {} 2023-10-18T20:18:41.345 DEBUG:boto:Token: None 2023-10-18T20:18:41.345 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.346 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2EVcUGrEUuwKFnmlEgud6Ddbn7Q= 2023-10-18T20:18:41.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2EVcUGrEUuwKFnmlEgud6Ddbn7Q=', 'Content-Length': '0'} 2023-10-18T20:18:41.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070fe11ad2c7e5f26-0065303da1-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.349 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.350 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.350 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.350 DEBUG:boto:Path: /bofrnf-49/ 2023-10-18T20:18:41.351 DEBUG:boto:Data: 2023-10-18T20:18:41.351 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.352 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.352 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.352 DEBUG:boto:Params: {} 2023-10-18T20:18:41.353 DEBUG:boto:Token: None 2023-10-18T20:18:41.353 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.354 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2EVcUGrEUuwKFnmlEgud6Ddbn7Q= 2023-10-18T20:18:41.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2EVcUGrEUuwKFnmlEgud6Ddbn7Q=', 'Content-Length': '0'} 2023-10-18T20:18:41.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009be012ff3880a7c9-0065303da1-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.357 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:18:41.357 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.358 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.358 DEBUG:boto:path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.358 DEBUG:boto:auth_path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.359 DEBUG:boto:Method: GET 2023-10-18T20:18:41.359 DEBUG:boto:Path: /bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.360 DEBUG:boto:Data: 2023-10-18T20:18:41.360 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.361 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.361 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.361 DEBUG:boto:Params: {} 2023-10-18T20:18:41.362 DEBUG:boto:Token: None 2023-10-18T20:18:41.362 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/?versions 2023-10-18T20:18:41.363 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kIS+uKsUzqbfwGw4KTAEW9uJO/s= 2023-10-18T20:18:41.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kIS+uKsUzqbfwGw4KTAEW9uJO/s=', 'Content-Length': '0'} 2023-10-18T20:18:41.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e6ccb77f791998b-0065303da1-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.366 DEBUG:boto:b'bofrnf-49999falseobj1nulltrue2023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:17:53.741Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:17:53.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:17:53.817Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:17:53.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:18:41.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:18:41.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:18:41.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:18:41.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-18T20:18:41.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-18T20:18:41.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-18T20:18:41.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-18T20:18:41.370 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:18:41.370 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.371 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.371 DEBUG:boto:path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.371 DEBUG:boto:auth_path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.372 DEBUG:boto:Method: GET 2023-10-18T20:18:41.372 DEBUG:boto:Path: /bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.373 DEBUG:boto:Data: 2023-10-18T20:18:41.373 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.373 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.374 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.374 DEBUG:boto:Params: {} 2023-10-18T20:18:41.375 DEBUG:boto:Token: None 2023-10-18T20:18:41.375 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/?versions 2023-10-18T20:18:41.376 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kIS+uKsUzqbfwGw4KTAEW9uJO/s= 2023-10-18T20:18:41.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kIS+uKsUzqbfwGw4KTAEW9uJO/s=', 'Content-Length': '0'} 2023-10-18T20:18:41.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009233f77cdaace2a4-0065303da1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.379 DEBUG:boto:b'bofrnf-49999falseobj1nulltrue2023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:17:53.741Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:17:53.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:17:53.817Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:17:53.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-18T20:18:41.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-18T20:18:41.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-18T20:18:41.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-18T20:18:41.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-18T20:18:41.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-18T20:18:41.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-18T20:18:41.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-18T20:18:41.383 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.383 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.383 DEBUG:boto:path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.384 DEBUG:boto:auth_path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.384 DEBUG:boto:Method: GET 2023-10-18T20:18:41.385 DEBUG:boto:Path: /bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.385 DEBUG:boto:Data: 2023-10-18T20:18:41.385 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.386 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.387 DEBUG:boto:Params: {} 2023-10-18T20:18:41.387 DEBUG:boto:Token: None 2023-10-18T20:18:41.388 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/?versions 2023-10-18T20:18:41.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kIS+uKsUzqbfwGw4KTAEW9uJO/s= 2023-10-18T20:18:41.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kIS+uKsUzqbfwGw4KTAEW9uJO/s=', 'Content-Length': '0'} 2023-10-18T20:18:41.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c3d4ab3a6f2230f-0065303da1-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.392 DEBUG:boto:b'bofrnf-49999falseobj1nulltrue2023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:17:53.741Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:17:53.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:17:53.817Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:17:53.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.392 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.393 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.393 DEBUG:boto:path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.394 DEBUG:boto:auth_path=/bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.394 DEBUG:boto:Method: GET 2023-10-18T20:18:41.395 DEBUG:boto:Path: /bofrnf-49/?versions&max-keys=999 2023-10-18T20:18:41.395 DEBUG:boto:Data: 2023-10-18T20:18:41.395 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.396 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.396 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.397 DEBUG:boto:Params: {} 2023-10-18T20:18:41.397 DEBUG:boto:Token: None 2023-10-18T20:18:41.398 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/?versions 2023-10-18T20:18:41.398 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kIS+uKsUzqbfwGw4KTAEW9uJO/s= 2023-10-18T20:18:41.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kIS+uKsUzqbfwGw4KTAEW9uJO/s=', 'Content-Length': '0'} 2023-10-18T20:18:41.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e31c87b63fb517b3-0065303da1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.401 DEBUG:boto:b'bofrnf-49999falseobj1nulltrue2023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-18T20:17:53.741Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-18T20:17:53.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-18T20:17:53.817Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-18T20:17:53.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:18:41.403 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.403 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.404 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.404 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.404 DEBUG:boto:Method: GET 2023-10-18T20:18:41.405 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.405 DEBUG:boto:Data: 2023-10-18T20:18:41.406 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.406 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.406 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.407 DEBUG:boto:Params: {} 2023-10-18T20:18:41.407 DEBUG:boto:Token: None 2023-10-18T20:18:41.408 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.408 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DC2s9n9wVllsnmc1MHQWEd39gU0= 2023-10-18T20:18:41.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DC2s9n9wVllsnmc1MHQWEd39gU0=', 'Content-Length': '0'} 2023-10-18T20:18:41.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006410c08bff60fb5a-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.411 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.412 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.412 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.413 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.413 DEBUG:boto:Method: GET 2023-10-18T20:18:41.413 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.414 DEBUG:boto:Data: 2023-10-18T20:18:41.414 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.415 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.415 DEBUG:boto:Params: {} 2023-10-18T20:18:41.416 DEBUG:boto:Token: None 2023-10-18T20:18:41.416 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.417 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DC2s9n9wVllsnmc1MHQWEd39gU0= 2023-10-18T20:18:41.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DC2s9n9wVllsnmc1MHQWEd39gU0=', 'Content-Length': '0'} 2023-10-18T20:18:41.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096135ab3490477de-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.420 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.420 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.421 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.421 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.421 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.422 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.422 DEBUG:boto:Data: 2023-10-18T20:18:41.423 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.423 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.423 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.424 DEBUG:boto:Params: {} 2023-10-18T20:18:41.424 DEBUG:boto:Token: None 2023-10-18T20:18:41.425 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.425 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ImrBhP0pMTz9hK2/osi4ip3St9k= 2023-10-18T20:18:41.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ImrBhP0pMTz9hK2/osi4ip3St9k=', 'Content-Length': '0'} 2023-10-18T20:18:41.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a808f78729445b92-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.428 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.428 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.429 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.429 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.430 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.430 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.430 DEBUG:boto:Data: 2023-10-18T20:18:41.431 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.431 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.432 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.432 DEBUG:boto:Params: {} 2023-10-18T20:18:41.432 DEBUG:boto:Token: None 2023-10-18T20:18:41.433 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.433 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ImrBhP0pMTz9hK2/osi4ip3St9k= 2023-10-18T20:18:41.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ImrBhP0pMTz9hK2/osi4ip3St9k=', 'Content-Length': '0'} 2023-10-18T20:18:41.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f625af3e6fb958f-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:18:41.437 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.437 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.437 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.438 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.438 DEBUG:boto:Method: GET 2023-10-18T20:18:41.439 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.439 DEBUG:boto:Data: 2023-10-18T20:18:41.440 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.440 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.440 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.441 DEBUG:boto:Params: {} 2023-10-18T20:18:41.441 DEBUG:boto:Token: None 2023-10-18T20:18:41.442 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.442 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DC2s9n9wVllsnmc1MHQWEd39gU0= 2023-10-18T20:18:41.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DC2s9n9wVllsnmc1MHQWEd39gU0=', 'Content-Length': '0'} 2023-10-18T20:18:41.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b94d0fc8ff5d365c-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.445 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.446 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.446 DEBUG:boto:path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.447 DEBUG:boto:auth_path=/bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.447 DEBUG:boto:Method: GET 2023-10-18T20:18:41.447 DEBUG:boto:Path: /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.448 DEBUG:boto:Data: 2023-10-18T20:18:41.448 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.449 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.449 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.450 DEBUG:boto:Params: {} 2023-10-18T20:18:41.450 DEBUG:boto:Token: None 2023-10-18T20:18:41.451 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1?versionId=null 2023-10-18T20:18:41.451 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DC2s9n9wVllsnmc1MHQWEd39gU0= 2023-10-18T20:18:41.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DC2s9n9wVllsnmc1MHQWEd39gU0=', 'Content-Length': '0'} 2023-10-18T20:18:41.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000940138d468718e48-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.494 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.495 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.495 DEBUG:boto:path=/bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.496 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.496 DEBUG:boto:Method: GET 2023-10-18T20:18:41.496 DEBUG:boto:Path: /bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.497 DEBUG:boto:Data: 2023-10-18T20:18:41.497 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.498 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.499 DEBUG:boto:Params: {} 2023-10-18T20:18:41.499 DEBUG:boto:Token: None 2023-10-18T20:18:41.499 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.500 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cdf2bae2bdd5b9e-0065303da1-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.503 DEBUG:boto:b'bofrnf-49obj11000falseobj12023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.504 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.504 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.505 DEBUG:boto:path=/bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.505 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.505 DEBUG:boto:Method: GET 2023-10-18T20:18:41.506 DEBUG:boto:Path: /bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.506 DEBUG:boto:Data: 2023-10-18T20:18:41.507 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.507 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.508 DEBUG:boto:Params: {} 2023-10-18T20:18:41.508 DEBUG:boto:Token: None 2023-10-18T20:18:41.509 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d84340c56d9937e8-0065303da1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.512 DEBUG:boto:b'bofrnf-49obj11000falseobj12023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.513 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.513 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.514 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.514 DEBUG:boto:Path: /bofrnf-49/obj1 2023-10-18T20:18:41.514 DEBUG:boto:Data: 2023-10-18T20:18:41.515 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.515 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.516 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.516 DEBUG:boto:Params: {} 2023-10-18T20:18:41.516 DEBUG:boto:Token: None 2023-10-18T20:18:41.517 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1 2023-10-18T20:18:41.517 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bj6pFBc6Cmht9GReYUkHkn4WJ18= 2023-10-18T20:18:41.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bj6pFBc6Cmht9GReYUkHkn4WJ18=', 'Content-Length': '0'} 2023-10-18T20:18:41.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084b054a29649f21e-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.520 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.520 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.521 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.521 DEBUG:boto:Path: /bofrnf-49/obj1 2023-10-18T20:18:41.521 DEBUG:boto:Data: 2023-10-18T20:18:41.522 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.523 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.523 DEBUG:boto:Params: {} 2023-10-18T20:18:41.523 DEBUG:boto:Token: None 2023-10-18T20:18:41.524 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1 2023-10-18T20:18:41.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bj6pFBc6Cmht9GReYUkHkn4WJ18= 2023-10-18T20:18:41.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bj6pFBc6Cmht9GReYUkHkn4WJ18=', 'Content-Length': '0'} 2023-10-18T20:18:41.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005c3538f1e389be5-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-18T20:18:41.527 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.528 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.528 DEBUG:boto:Method: GET 2023-10-18T20:18:41.528 DEBUG:boto:Path: /bofrnf-49/obj1 2023-10-18T20:18:41.529 DEBUG:boto:Data: 2023-10-18T20:18:41.529 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.530 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.530 DEBUG:boto:Params: {} 2023-10-18T20:18:41.531 DEBUG:boto:Token: None 2023-10-18T20:18:41.531 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1 2023-10-18T20:18:41.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AjFHr+nNrxAgLhDvRxCZ6hn3hwA= 2023-10-18T20:18:41.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AjFHr+nNrxAgLhDvRxCZ6hn3hwA=', 'Content-Length': '0'} 2023-10-18T20:18:41.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2b2cf7dc7e08af9-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.574 DEBUG:boto:path=/bofrnf-49/obj1 2023-10-18T20:18:41.575 DEBUG:boto:auth_path=/bofrnf-49/obj1 2023-10-18T20:18:41.575 DEBUG:boto:Method: GET 2023-10-18T20:18:41.576 DEBUG:boto:Path: /bofrnf-49/obj1 2023-10-18T20:18:41.576 DEBUG:boto:Data: 2023-10-18T20:18:41.577 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.577 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.577 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.578 DEBUG:boto:Params: {} 2023-10-18T20:18:41.578 DEBUG:boto:Token: None 2023-10-18T20:18:41.579 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj1 2023-10-18T20:18:41.579 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AjFHr+nNrxAgLhDvRxCZ6hn3hwA= 2023-10-18T20:18:41.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AjFHr+nNrxAgLhDvRxCZ6hn3hwA=', 'Content-Length': '0'} 2023-10-18T20:18:41.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa936a545e09cd16-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.582 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.583 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.583 DEBUG:boto:path=/bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.584 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.584 DEBUG:boto:Method: GET 2023-10-18T20:18:41.584 DEBUG:boto:Path: /bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.585 DEBUG:boto:Data: 2023-10-18T20:18:41.585 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.586 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.586 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.587 DEBUG:boto:Params: {} 2023-10-18T20:18:41.587 DEBUG:boto:Token: None 2023-10-18T20:18:41.587 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.588 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000507e38465b49b10a-0065303da1-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.591 DEBUG:boto:b'bofrnf-49obj11000falseobj12023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.592 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.592 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.593 DEBUG:boto:path=/bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.593 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.594 DEBUG:boto:Method: GET 2023-10-18T20:18:41.594 DEBUG:boto:Path: /bofrnf-49/?prefix=obj1 2023-10-18T20:18:41.594 DEBUG:boto:Data: 2023-10-18T20:18:41.595 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.595 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.596 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.596 DEBUG:boto:Params: {} 2023-10-18T20:18:41.597 DEBUG:boto:Token: None 2023-10-18T20:18:41.597 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.597 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeac4038a2aa3a30-0065303da1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.601 DEBUG:boto:b'bofrnf-49obj11000falseobj12023-10-18T20:17:00.381Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:18:41.602 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.602 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.603 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.603 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.604 DEBUG:boto:Method: GET 2023-10-18T20:18:41.604 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.604 DEBUG:boto:Data: 2023-10-18T20:18:41.605 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.605 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.606 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.606 DEBUG:boto:Params: {} 2023-10-18T20:18:41.607 DEBUG:boto:Token: None 2023-10-18T20:18:41.607 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+Hg4+cIi4Qpp7wiFiU+gHfzpMX4= 2023-10-18T20:18:41.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+Hg4+cIi4Qpp7wiFiU+gHfzpMX4=', 'Content-Length': '0'} 2023-10-18T20:18:41.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000889615a0487577e4-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.610 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.611 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.611 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.612 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.612 DEBUG:boto:Method: GET 2023-10-18T20:18:41.612 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.613 DEBUG:boto:Data: 2023-10-18T20:18:41.613 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.614 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.614 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.615 DEBUG:boto:Params: {} 2023-10-18T20:18:41.615 DEBUG:boto:Token: None 2023-10-18T20:18:41.616 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.616 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+Hg4+cIi4Qpp7wiFiU+gHfzpMX4= 2023-10-18T20:18:41.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+Hg4+cIi4Qpp7wiFiU+gHfzpMX4=', 'Content-Length': '0'} 2023-10-18T20:18:41.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087338f1c772549f6-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.619 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.619 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.620 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.620 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.620 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.621 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.621 DEBUG:boto:Data: 2023-10-18T20:18:41.621 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.622 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.622 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.623 DEBUG:boto:Params: {} 2023-10-18T20:18:41.623 DEBUG:boto:Token: None 2023-10-18T20:18:41.624 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Hg9FTwXJgZLtV9dFTcFoneBKhl4= 2023-10-18T20:18:41.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Hg9FTwXJgZLtV9dFTcFoneBKhl4=', 'Content-Length': '0'} 2023-10-18T20:18:41.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ade709a77ebe8f5-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.627 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.627 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.628 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.628 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.628 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.629 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.629 DEBUG:boto:Data: 2023-10-18T20:18:41.629 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.630 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.631 DEBUG:boto:Params: {} 2023-10-18T20:18:41.631 DEBUG:boto:Token: None 2023-10-18T20:18:41.631 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.632 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Hg9FTwXJgZLtV9dFTcFoneBKhl4= 2023-10-18T20:18:41.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Hg9FTwXJgZLtV9dFTcFoneBKhl4=', 'Content-Length': '0'} 2023-10-18T20:18:41.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074bdf8c275372fb6-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:18:41.635 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.635 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.636 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.636 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.637 DEBUG:boto:Method: GET 2023-10-18T20:18:41.637 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.637 DEBUG:boto:Data: 2023-10-18T20:18:41.638 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.638 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.639 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.639 DEBUG:boto:Params: {} 2023-10-18T20:18:41.640 DEBUG:boto:Token: None 2023-10-18T20:18:41.640 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.640 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+Hg4+cIi4Qpp7wiFiU+gHfzpMX4= 2023-10-18T20:18:41.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+Hg4+cIi4Qpp7wiFiU+gHfzpMX4=', 'Content-Length': '0'} 2023-10-18T20:18:41.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ffdd1b881b26812-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.686 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.687 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.687 DEBUG:boto:path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.688 DEBUG:boto:auth_path=/bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.688 DEBUG:boto:Method: GET 2023-10-18T20:18:41.689 DEBUG:boto:Path: /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.689 DEBUG:boto:Data: 2023-10-18T20:18:41.689 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.690 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.691 DEBUG:boto:Params: {} 2023-10-18T20:18:41.691 DEBUG:boto:Token: None 2023-10-18T20:18:41.692 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2?versionId=null 2023-10-18T20:18:41.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+Hg4+cIi4Qpp7wiFiU+gHfzpMX4= 2023-10-18T20:18:41.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+Hg4+cIi4Qpp7wiFiU+gHfzpMX4=', 'Content-Length': '0'} 2023-10-18T20:18:41.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad400defdba89646-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.695 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.696 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.696 DEBUG:boto:path=/bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.696 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.697 DEBUG:boto:Method: GET 2023-10-18T20:18:41.697 DEBUG:boto:Path: /bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.698 DEBUG:boto:Data: 2023-10-18T20:18:41.698 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.699 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.699 DEBUG:boto:Params: {} 2023-10-18T20:18:41.700 DEBUG:boto:Token: None 2023-10-18T20:18:41.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd750165a38d6f9a-0065303da1-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.704 DEBUG:boto:b'bofrnf-49obj21000falseobj22023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.705 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.705 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.706 DEBUG:boto:path=/bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.706 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.706 DEBUG:boto:Method: GET 2023-10-18T20:18:41.707 DEBUG:boto:Path: /bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.707 DEBUG:boto:Data: 2023-10-18T20:18:41.708 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.708 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.709 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.709 DEBUG:boto:Params: {} 2023-10-18T20:18:41.709 DEBUG:boto:Token: None 2023-10-18T20:18:41.710 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306c7742806235b0-0065303da1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.754 DEBUG:boto:b'bofrnf-49obj21000falseobj22023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.755 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.756 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.756 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.756 DEBUG:boto:Path: /bofrnf-49/obj2 2023-10-18T20:18:41.757 DEBUG:boto:Data: 2023-10-18T20:18:41.757 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.758 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.758 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.759 DEBUG:boto:Params: {} 2023-10-18T20:18:41.759 DEBUG:boto:Token: None 2023-10-18T20:18:41.759 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2 2023-10-18T20:18:41.760 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SuIhLd1J5d1YKXJqnCF2+TKFs30= 2023-10-18T20:18:41.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SuIhLd1J5d1YKXJqnCF2+TKFs30=', 'Content-Length': '0'} 2023-10-18T20:18:41.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c60a250eefe8bef6-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.763 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.763 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.764 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.764 DEBUG:boto:Path: /bofrnf-49/obj2 2023-10-18T20:18:41.765 DEBUG:boto:Data: 2023-10-18T20:18:41.765 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.765 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.766 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.766 DEBUG:boto:Params: {} 2023-10-18T20:18:41.767 DEBUG:boto:Token: None 2023-10-18T20:18:41.767 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2 2023-10-18T20:18:41.768 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SuIhLd1J5d1YKXJqnCF2+TKFs30= 2023-10-18T20:18:41.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SuIhLd1J5d1YKXJqnCF2+TKFs30=', 'Content-Length': '0'} 2023-10-18T20:18:41.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005dd9b5c188f4442-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-18T20:18:41.771 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.772 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.772 DEBUG:boto:Method: GET 2023-10-18T20:18:41.772 DEBUG:boto:Path: /bofrnf-49/obj2 2023-10-18T20:18:41.773 DEBUG:boto:Data: 2023-10-18T20:18:41.773 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.774 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.774 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.775 DEBUG:boto:Params: {} 2023-10-18T20:18:41.775 DEBUG:boto:Token: None 2023-10-18T20:18:41.776 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2 2023-10-18T20:18:41.776 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o9qDcmY+9S5mfcb763TJfOPGzkE= 2023-10-18T20:18:41.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o9qDcmY+9S5mfcb763TJfOPGzkE=', 'Content-Length': '0'} 2023-10-18T20:18:41.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000174637a028c38f6b-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.779 DEBUG:boto:path=/bofrnf-49/obj2 2023-10-18T20:18:41.779 DEBUG:boto:auth_path=/bofrnf-49/obj2 2023-10-18T20:18:41.780 DEBUG:boto:Method: GET 2023-10-18T20:18:41.780 DEBUG:boto:Path: /bofrnf-49/obj2 2023-10-18T20:18:41.781 DEBUG:boto:Data: 2023-10-18T20:18:41.781 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.782 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.782 DEBUG:boto:Params: {} 2023-10-18T20:18:41.783 DEBUG:boto:Token: None 2023-10-18T20:18:41.783 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj2 2023-10-18T20:18:41.784 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o9qDcmY+9S5mfcb763TJfOPGzkE= 2023-10-18T20:18:41.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o9qDcmY+9S5mfcb763TJfOPGzkE=', 'Content-Length': '0'} 2023-10-18T20:18:41.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c2e44b152870e5d-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.787 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.787 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.788 DEBUG:boto:path=/bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.788 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.788 DEBUG:boto:Method: GET 2023-10-18T20:18:41.789 DEBUG:boto:Path: /bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.789 DEBUG:boto:Data: 2023-10-18T20:18:41.790 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.790 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.791 DEBUG:boto:Params: {} 2023-10-18T20:18:41.791 DEBUG:boto:Token: None 2023-10-18T20:18:41.792 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000561941b34fed374c-0065303da1-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.838 DEBUG:boto:b'bofrnf-49obj21000falseobj22023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.839 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.840 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.840 DEBUG:boto:path=/bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.841 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.841 DEBUG:boto:Method: GET 2023-10-18T20:18:41.841 DEBUG:boto:Path: /bofrnf-49/?prefix=obj2 2023-10-18T20:18:41.842 DEBUG:boto:Data: 2023-10-18T20:18:41.842 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.843 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.844 DEBUG:boto:Params: {} 2023-10-18T20:18:41.844 DEBUG:boto:Token: None 2023-10-18T20:18:41.845 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.845 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037ba4d45f165e4b1-0065303da1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.849 DEBUG:boto:b'bofrnf-49obj21000falseobj22023-10-18T20:17:00.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:18:41.850 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:41.850 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:41.851 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.851 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.851 DEBUG:boto:Method: GET 2023-10-18T20:18:41.852 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.852 DEBUG:boto:Data: 2023-10-18T20:18:41.853 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.853 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.853 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.854 DEBUG:boto:Params: {} 2023-10-18T20:18:41.854 DEBUG:boto:Token: None 2023-10-18T20:18:41.855 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.855 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Vhg+FlDitTYyf4VaYAHXv7mi+QM= 2023-10-18T20:18:41.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vhg+FlDitTYyf4VaYAHXv7mi+QM=', 'Content-Length': '0'} 2023-10-18T20:18:41.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007060359eb74156d3-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.858 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:41.859 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:41.859 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.860 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.860 DEBUG:boto:Method: GET 2023-10-18T20:18:41.860 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.861 DEBUG:boto:Data: 2023-10-18T20:18:41.861 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.862 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.862 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.863 DEBUG:boto:Params: {} 2023-10-18T20:18:41.863 DEBUG:boto:Token: None 2023-10-18T20:18:41.864 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.864 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Vhg+FlDitTYyf4VaYAHXv7mi+QM= 2023-10-18T20:18:41.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vhg+FlDitTYyf4VaYAHXv7mi+QM=', 'Content-Length': '0'} 2023-10-18T20:18:41.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a947f44ac0f830c-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.867 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:41.867 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:41.868 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.868 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.868 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.869 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.869 DEBUG:boto:Data: 2023-10-18T20:18:41.870 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.870 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.871 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.871 DEBUG:boto:Params: {} 2023-10-18T20:18:41.871 DEBUG:boto:Token: None 2023-10-18T20:18:41.872 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6D7csKoJfTsbg2ROBKP2zPQn+zw= 2023-10-18T20:18:41.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6D7csKoJfTsbg2ROBKP2zPQn+zw=', 'Content-Length': '0'} 2023-10-18T20:18:41.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7d261d50ae3b46f-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.875 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:41.876 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:41.876 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.876 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.877 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.877 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.877 DEBUG:boto:Data: 2023-10-18T20:18:41.878 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.878 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.879 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.879 DEBUG:boto:Params: {} 2023-10-18T20:18:41.880 DEBUG:boto:Token: None 2023-10-18T20:18:41.880 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6D7csKoJfTsbg2ROBKP2zPQn+zw= 2023-10-18T20:18:41.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6D7csKoJfTsbg2ROBKP2zPQn+zw=', 'Content-Length': '0'} 2023-10-18T20:18:41.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f0dc6c546b34990-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:18:41.884 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:41.884 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:41.885 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.885 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.886 DEBUG:boto:Method: GET 2023-10-18T20:18:41.886 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.886 DEBUG:boto:Data: 2023-10-18T20:18:41.887 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.887 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.888 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.888 DEBUG:boto:Params: {} 2023-10-18T20:18:41.889 DEBUG:boto:Token: None 2023-10-18T20:18:41.889 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.890 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Vhg+FlDitTYyf4VaYAHXv7mi+QM= 2023-10-18T20:18:41.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vhg+FlDitTYyf4VaYAHXv7mi+QM=', 'Content-Length': '0'} 2023-10-18T20:18:41.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000662e053d15fd6baf-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.892 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:41.893 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:41.893 DEBUG:boto:path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.894 DEBUG:boto:auth_path=/bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.894 DEBUG:boto:Method: GET 2023-10-18T20:18:41.895 DEBUG:boto:Path: /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.895 DEBUG:boto:Data: 2023-10-18T20:18:41.895 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.896 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.896 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.897 DEBUG:boto:Params: {} 2023-10-18T20:18:41.897 DEBUG:boto:Token: None 2023-10-18T20:18:41.898 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj3?versionId=null 2023-10-18T20:18:41.898 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Vhg+FlDitTYyf4VaYAHXv7mi+QM= 2023-10-18T20:18:41.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vhg+FlDitTYyf4VaYAHXv7mi+QM=', 'Content-Length': '0'} 2023-10-18T20:18:41.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001609c5955eeef5f8-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.942 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.943 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.943 DEBUG:boto:path=/bofrnf-49/?prefix=obj3 2023-10-18T20:18:41.944 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj3 2023-10-18T20:18:41.944 DEBUG:boto:Method: GET 2023-10-18T20:18:41.945 DEBUG:boto:Path: /bofrnf-49/?prefix=obj3 2023-10-18T20:18:41.945 DEBUG:boto:Data: 2023-10-18T20:18:41.945 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.946 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.946 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.947 DEBUG:boto:Params: {} 2023-10-18T20:18:41.947 DEBUG:boto:Token: None 2023-10-18T20:18:41.948 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c81f6019e763c99-0065303da1-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.951 DEBUG:boto:b'bofrnf-49obj31000falseobj32023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.952 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:41.952 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:41.953 DEBUG:boto:path=/bofrnf-49/?prefix=obj3 2023-10-18T20:18:41.953 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj3 2023-10-18T20:18:41.954 DEBUG:boto:Method: GET 2023-10-18T20:18:41.954 DEBUG:boto:Path: /bofrnf-49/?prefix=obj3 2023-10-18T20:18:41.955 DEBUG:boto:Data: 2023-10-18T20:18:41.955 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.955 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.956 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.956 DEBUG:boto:Params: {} 2023-10-18T20:18:41.957 DEBUG:boto:Token: None 2023-10-18T20:18:41.957 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/ 2023-10-18T20:18:41.958 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY= 2023-10-18T20:18:41.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DeVv4qb4mZjTbnn6EWjvbH2uYDY=', 'Content-Length': '0'} 2023-10-18T20:18:41.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e221ac0b98c957e9-0065303da1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.961 DEBUG:boto:b'bofrnf-49obj31000falseobj32023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:41.962 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:41.962 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:41.962 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.963 DEBUG:boto:Path: /bofrnf-49/obj3 2023-10-18T20:18:41.963 DEBUG:boto:Data: 2023-10-18T20:18:41.964 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.964 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.964 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.965 DEBUG:boto:Params: {} 2023-10-18T20:18:41.965 DEBUG:boto:Token: None 2023-10-18T20:18:41.966 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj3 2023-10-18T20:18:41.966 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bAzr7+EuGpDcTxT5CHZhzhw0bz4= 2023-10-18T20:18:41.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bAzr7+EuGpDcTxT5CHZhzhw0bz4=', 'Content-Length': '0'} 2023-10-18T20:18:41.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fee20627567fd57f-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.969 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:41.969 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:41.970 DEBUG:boto:Method: HEAD 2023-10-18T20:18:41.970 DEBUG:boto:Path: /bofrnf-49/obj3 2023-10-18T20:18:41.971 DEBUG:boto:Data: 2023-10-18T20:18:41.971 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.971 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:41.972 DEBUG:boto:Port: 8000 2023-10-18T20:18:41.972 DEBUG:boto:Params: {} 2023-10-18T20:18:41.973 DEBUG:boto:Token: None 2023-10-18T20:18:41.973 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj3 2023-10-18T20:18:41.974 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bAzr7+EuGpDcTxT5CHZhzhw0bz4= 2023-10-18T20:18:41.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bAzr7+EuGpDcTxT5CHZhzhw0bz4=', 'Content-Length': '0'} 2023-10-18T20:18:41.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab4277d4048adc23-0065303da1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:41.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-18T20:18:41.977 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:41.977 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:41.978 DEBUG:boto:Method: GET 2023-10-18T20:18:41.978 DEBUG:boto:Path: /bofrnf-49/obj3 2023-10-18T20:18:41.978 DEBUG:boto:Data: 2023-10-18T20:18:41.979 DEBUG:boto:Headers: {} 2023-10-18T20:18:41.979 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:41.980 DEBUG:boto:Port: 8001 2023-10-18T20:18:41.980 DEBUG:boto:Params: {} 2023-10-18T20:18:41.981 DEBUG:boto:Token: None 2023-10-18T20:18:41.981 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:41 GMT /bofrnf-49/obj3 2023-10-18T20:18:41.982 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IimsVL2OSCJGefivr7ynF0QLV5o= 2023-10-18T20:18:41.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IimsVL2OSCJGefivr7ynF0QLV5o=', 'Content-Length': '0'} 2023-10-18T20:18:41.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000277470967d463ac5-0065303da1-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.026 DEBUG:boto:path=/bofrnf-49/obj3 2023-10-18T20:18:42.027 DEBUG:boto:auth_path=/bofrnf-49/obj3 2023-10-18T20:18:42.027 DEBUG:boto:Method: GET 2023-10-18T20:18:42.028 DEBUG:boto:Path: /bofrnf-49/obj3 2023-10-18T20:18:42.028 DEBUG:boto:Data: 2023-10-18T20:18:42.029 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.029 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.030 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.030 DEBUG:boto:Params: {} 2023-10-18T20:18:42.031 DEBUG:boto:Token: None 2023-10-18T20:18:42.031 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj3 2023-10-18T20:18:42.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n9tSlxaCDOeNBfMrfutJUgUUOMM= 2023-10-18T20:18:42.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n9tSlxaCDOeNBfMrfutJUgUUOMM=', 'Content-Length': '0'} 2023-10-18T20:18:42.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009790fdf601e0b21e-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.035 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.035 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.035 DEBUG:boto:path=/bofrnf-49/?prefix=obj3 2023-10-18T20:18:42.036 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj3 2023-10-18T20:18:42.036 DEBUG:boto:Method: GET 2023-10-18T20:18:42.037 DEBUG:boto:Path: /bofrnf-49/?prefix=obj3 2023-10-18T20:18:42.037 DEBUG:boto:Data: 2023-10-18T20:18:42.037 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.038 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.038 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.039 DEBUG:boto:Params: {} 2023-10-18T20:18:42.039 DEBUG:boto:Token: None 2023-10-18T20:18:42.040 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a284b484e2beafb0-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.044 DEBUG:boto:b'bofrnf-49obj31000falseobj32023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.044 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.045 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.045 DEBUG:boto:path=/bofrnf-49/?prefix=obj3 2023-10-18T20:18:42.045 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj3 2023-10-18T20:18:42.046 DEBUG:boto:Method: GET 2023-10-18T20:18:42.046 DEBUG:boto:Path: /bofrnf-49/?prefix=obj3 2023-10-18T20:18:42.047 DEBUG:boto:Data: 2023-10-18T20:18:42.047 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.047 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.048 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.048 DEBUG:boto:Params: {} 2023-10-18T20:18:42.049 DEBUG:boto:Token: None 2023-10-18T20:18:42.049 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000276cd09a943407e5-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.053 DEBUG:boto:b'bofrnf-49obj31000falseobj32023-10-18T20:17:00.457Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:18:42.054 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.054 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.055 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.055 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.056 DEBUG:boto:Method: GET 2023-10-18T20:18:42.056 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.056 DEBUG:boto:Data: 2023-10-18T20:18:42.057 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.057 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.058 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.058 DEBUG:boto:Params: {} 2023-10-18T20:18:42.059 DEBUG:boto:Token: None 2023-10-18T20:18:42.059 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.060 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ayHOa1r2psiFMa4uc8ALUxPuW8k= 2023-10-18T20:18:42.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ayHOa1r2psiFMa4uc8ALUxPuW8k=', 'Content-Length': '0'} 2023-10-18T20:18:42.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e2c9fc68ea96520-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.063 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.063 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.064 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.064 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.064 DEBUG:boto:Method: GET 2023-10-18T20:18:42.065 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.065 DEBUG:boto:Data: 2023-10-18T20:18:42.066 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.067 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.067 DEBUG:boto:Params: {} 2023-10-18T20:18:42.068 DEBUG:boto:Token: None 2023-10-18T20:18:42.068 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.069 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ayHOa1r2psiFMa4uc8ALUxPuW8k= 2023-10-18T20:18:42.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ayHOa1r2psiFMa4uc8ALUxPuW8k=', 'Content-Length': '0'} 2023-10-18T20:18:42.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d282b53b3c893e1e-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.072 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.072 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.073 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.073 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.074 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.074 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.074 DEBUG:boto:Data: 2023-10-18T20:18:42.075 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.075 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.076 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.076 DEBUG:boto:Params: {} 2023-10-18T20:18:42.077 DEBUG:boto:Token: None 2023-10-18T20:18:42.077 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.077 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmEQ6A1V6qmXdftpoO66OCGuUKM= 2023-10-18T20:18:42.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmEQ6A1V6qmXdftpoO66OCGuUKM=', 'Content-Length': '0'} 2023-10-18T20:18:42.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fe9c9d19bdb918f-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.080 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.080 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.081 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.081 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.081 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.082 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.082 DEBUG:boto:Data: 2023-10-18T20:18:42.083 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.083 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.083 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.084 DEBUG:boto:Params: {} 2023-10-18T20:18:42.084 DEBUG:boto:Token: None 2023-10-18T20:18:42.085 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mmEQ6A1V6qmXdftpoO66OCGuUKM= 2023-10-18T20:18:42.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mmEQ6A1V6qmXdftpoO66OCGuUKM=', 'Content-Length': '0'} 2023-10-18T20:18:42.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000173da571437b40cc-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:18:42.088 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.089 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.089 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.089 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.090 DEBUG:boto:Method: GET 2023-10-18T20:18:42.090 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.091 DEBUG:boto:Data: 2023-10-18T20:18:42.091 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.092 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.092 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.092 DEBUG:boto:Params: {} 2023-10-18T20:18:42.093 DEBUG:boto:Token: None 2023-10-18T20:18:42.093 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.094 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ayHOa1r2psiFMa4uc8ALUxPuW8k= 2023-10-18T20:18:42.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ayHOa1r2psiFMa4uc8ALUxPuW8k=', 'Content-Length': '0'} 2023-10-18T20:18:42.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3a5dd13ce15e93e-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.097 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.098 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.098 DEBUG:boto:path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.098 DEBUG:boto:auth_path=/bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.099 DEBUG:boto:Method: GET 2023-10-18T20:18:42.099 DEBUG:boto:Path: /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.100 DEBUG:boto:Data: 2023-10-18T20:18:42.100 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.101 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.101 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.101 DEBUG:boto:Params: {} 2023-10-18T20:18:42.102 DEBUG:boto:Token: None 2023-10-18T20:18:42.102 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4?versionId=null 2023-10-18T20:18:42.103 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ayHOa1r2psiFMa4uc8ALUxPuW8k= 2023-10-18T20:18:42.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ayHOa1r2psiFMa4uc8ALUxPuW8k=', 'Content-Length': '0'} 2023-10-18T20:18:42.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b19eece7ab50d6a-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.106 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.106 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.107 DEBUG:boto:path=/bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.107 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.108 DEBUG:boto:Method: GET 2023-10-18T20:18:42.108 DEBUG:boto:Path: /bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.108 DEBUG:boto:Data: 2023-10-18T20:18:42.109 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.110 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.110 DEBUG:boto:Params: {} 2023-10-18T20:18:42.111 DEBUG:boto:Token: None 2023-10-18T20:18:42.111 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.111 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084b8267118e94357-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.115 DEBUG:boto:b'bofrnf-49obj41000falseobj42023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.115 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.116 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.116 DEBUG:boto:path=/bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.117 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.117 DEBUG:boto:Method: GET 2023-10-18T20:18:42.118 DEBUG:boto:Path: /bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.118 DEBUG:boto:Data: 2023-10-18T20:18:42.118 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.119 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.119 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.120 DEBUG:boto:Params: {} 2023-10-18T20:18:42.120 DEBUG:boto:Token: None 2023-10-18T20:18:42.121 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.121 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038dd2c501d9cbe65-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.124 DEBUG:boto:b'bofrnf-49obj41000falseobj42023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.125 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.125 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.126 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.126 DEBUG:boto:Path: /bofrnf-49/obj4 2023-10-18T20:18:42.129 DEBUG:boto:Data: 2023-10-18T20:18:42.129 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.130 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.130 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.131 DEBUG:boto:Params: {} 2023-10-18T20:18:42.131 DEBUG:boto:Token: None 2023-10-18T20:18:42.132 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4 2023-10-18T20:18:42.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nzn1agDMYT7ehuy2tLzkGGf2k5E= 2023-10-18T20:18:42.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nzn1agDMYT7ehuy2tLzkGGf2k5E=', 'Content-Length': '0'} 2023-10-18T20:18:42.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb6e9bc147b0de79-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.135 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.136 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.136 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.137 DEBUG:boto:Path: /bofrnf-49/obj4 2023-10-18T20:18:42.137 DEBUG:boto:Data: 2023-10-18T20:18:42.137 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.138 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.138 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.139 DEBUG:boto:Params: {} 2023-10-18T20:18:42.139 DEBUG:boto:Token: None 2023-10-18T20:18:42.139 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4 2023-10-18T20:18:42.140 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nzn1agDMYT7ehuy2tLzkGGf2k5E= 2023-10-18T20:18:42.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nzn1agDMYT7ehuy2tLzkGGf2k5E=', 'Content-Length': '0'} 2023-10-18T20:18:42.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e054bc743501325-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-18T20:18:42.143 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.143 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.144 DEBUG:boto:Method: GET 2023-10-18T20:18:42.144 DEBUG:boto:Path: /bofrnf-49/obj4 2023-10-18T20:18:42.145 DEBUG:boto:Data: 2023-10-18T20:18:42.145 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.146 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.146 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.146 DEBUG:boto:Params: {} 2023-10-18T20:18:42.147 DEBUG:boto:Token: None 2023-10-18T20:18:42.147 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4 2023-10-18T20:18:42.148 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:asV+AtX7llPF3hvj55V+wb4MZYE= 2023-10-18T20:18:42.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:asV+AtX7llPF3hvj55V+wb4MZYE=', 'Content-Length': '0'} 2023-10-18T20:18:42.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004221c1236398078b-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.194 DEBUG:boto:path=/bofrnf-49/obj4 2023-10-18T20:18:42.195 DEBUG:boto:auth_path=/bofrnf-49/obj4 2023-10-18T20:18:42.195 DEBUG:boto:Method: GET 2023-10-18T20:18:42.196 DEBUG:boto:Path: /bofrnf-49/obj4 2023-10-18T20:18:42.196 DEBUG:boto:Data: 2023-10-18T20:18:42.196 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.197 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.198 DEBUG:boto:Params: {} 2023-10-18T20:18:42.198 DEBUG:boto:Token: None 2023-10-18T20:18:42.199 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj4 2023-10-18T20:18:42.199 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:asV+AtX7llPF3hvj55V+wb4MZYE= 2023-10-18T20:18:42.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:asV+AtX7llPF3hvj55V+wb4MZYE=', 'Content-Length': '0'} 2023-10-18T20:18:42.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003355937c2135cdcb-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.202 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.202 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.203 DEBUG:boto:path=/bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.203 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.204 DEBUG:boto:Method: GET 2023-10-18T20:18:42.204 DEBUG:boto:Path: /bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.205 DEBUG:boto:Data: 2023-10-18T20:18:42.205 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.205 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.206 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.206 DEBUG:boto:Params: {} 2023-10-18T20:18:42.207 DEBUG:boto:Token: None 2023-10-18T20:18:42.207 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.207 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066611ab148bbfb6b-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.211 DEBUG:boto:b'bofrnf-49obj41000falseobj42023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.212 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.212 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.212 DEBUG:boto:path=/bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.213 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.213 DEBUG:boto:Method: GET 2023-10-18T20:18:42.214 DEBUG:boto:Path: /bofrnf-49/?prefix=obj4 2023-10-18T20:18:42.214 DEBUG:boto:Data: 2023-10-18T20:18:42.215 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.215 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.215 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.216 DEBUG:boto:Params: {} 2023-10-18T20:18:42.216 DEBUG:boto:Token: None 2023-10-18T20:18:42.217 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.217 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7c27719c12bd742-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.221 DEBUG:boto:b'bofrnf-49obj41000falseobj42023-10-18T20:17:00.475Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:18:42.222 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.222 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.223 DEBUG:boto:path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.223 DEBUG:boto:auth_path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.224 DEBUG:boto:Method: GET 2023-10-18T20:18:42.224 DEBUG:boto:Path: /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.224 DEBUG:boto:Data: 2023-10-18T20:18:42.225 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.225 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.226 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.226 DEBUG:boto:Params: {} 2023-10-18T20:18:42.226 DEBUG:boto:Token: None 2023-10-18T20:18:42.227 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.227 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+GJ/z5UM8Io6I7LTJ3JFMPzbRUw= 2023-10-18T20:18:42.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+GJ/z5UM8Io6I7LTJ3JFMPzbRUw=', 'Content-Length': '0'} 2023-10-18T20:18:42.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b557d8725f403f3-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.230 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.231 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.231 DEBUG:boto:path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.231 DEBUG:boto:auth_path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.232 DEBUG:boto:Method: GET 2023-10-18T20:18:42.232 DEBUG:boto:Path: /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.233 DEBUG:boto:Data: 2023-10-18T20:18:42.233 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.234 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.234 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.234 DEBUG:boto:Params: {} 2023-10-18T20:18:42.235 DEBUG:boto:Token: None 2023-10-18T20:18:42.235 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.236 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+GJ/z5UM8Io6I7LTJ3JFMPzbRUw= 2023-10-18T20:18:42.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+GJ/z5UM8Io6I7LTJ3JFMPzbRUw=', 'Content-Length': '0'} 2023-10-18T20:18:42.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab5d0ee7a7898261-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.239 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.239 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.240 DEBUG:boto:path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.240 DEBUG:boto:auth_path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.241 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.241 DEBUG:boto:Path: /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.241 DEBUG:boto:Data: 2023-10-18T20:18:42.242 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.242 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.243 DEBUG:boto:Params: {} 2023-10-18T20:18:42.243 DEBUG:boto:Token: None 2023-10-18T20:18:42.244 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YBhl4N6vt7HjUC85BwYIeFiQIsE= 2023-10-18T20:18:42.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YBhl4N6vt7HjUC85BwYIeFiQIsE=', 'Content-Length': '0'} 2023-10-18T20:18:42.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dec7eaf5b309421d-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.247 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.247 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.248 DEBUG:boto:path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.248 DEBUG:boto:auth_path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.248 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.249 DEBUG:boto:Path: /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.249 DEBUG:boto:Data: 2023-10-18T20:18:42.249 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.250 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.250 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.251 DEBUG:boto:Params: {} 2023-10-18T20:18:42.251 DEBUG:boto:Token: None 2023-10-18T20:18:42.251 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YBhl4N6vt7HjUC85BwYIeFiQIsE= 2023-10-18T20:18:42.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YBhl4N6vt7HjUC85BwYIeFiQIsE=', 'Content-Length': '0'} 2023-10-18T20:18:42.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eac1aec2786dcea8-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:18:42.255 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.255 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.256 DEBUG:boto:path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.256 DEBUG:boto:auth_path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.257 DEBUG:boto:Method: GET 2023-10-18T20:18:42.257 DEBUG:boto:Path: /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.257 DEBUG:boto:Data: 2023-10-18T20:18:42.258 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.258 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.259 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.259 DEBUG:boto:Params: {} 2023-10-18T20:18:42.259 DEBUG:boto:Token: None 2023-10-18T20:18:42.260 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+GJ/z5UM8Io6I7LTJ3JFMPzbRUw= 2023-10-18T20:18:42.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+GJ/z5UM8Io6I7LTJ3JFMPzbRUw=', 'Content-Length': '0'} 2023-10-18T20:18:42.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000753e9e70ef2f602b-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.306 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.307 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.307 DEBUG:boto:path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.308 DEBUG:boto:auth_path=/bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.308 DEBUG:boto:Method: GET 2023-10-18T20:18:42.308 DEBUG:boto:Path: /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.309 DEBUG:boto:Data: 2023-10-18T20:18:42.309 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.310 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.310 DEBUG:boto:Params: {} 2023-10-18T20:18:42.311 DEBUG:boto:Token: None 2023-10-18T20:18:42.311 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5?versionId=null 2023-10-18T20:18:42.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+GJ/z5UM8Io6I7LTJ3JFMPzbRUw= 2023-10-18T20:18:42.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+GJ/z5UM8Io6I7LTJ3JFMPzbRUw=', 'Content-Length': '0'} 2023-10-18T20:18:42.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb15988ac1505855-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.315 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.315 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.316 DEBUG:boto:path=/bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.316 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.316 DEBUG:boto:Method: GET 2023-10-18T20:18:42.317 DEBUG:boto:Path: /bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.317 DEBUG:boto:Data: 2023-10-18T20:18:42.318 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.319 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.319 DEBUG:boto:Params: {} 2023-10-18T20:18:42.320 DEBUG:boto:Token: None 2023-10-18T20:18:42.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066b0a8ea958284dc-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.324 DEBUG:boto:b'bofrnf-49obj51000falseobj52023-10-18T20:17:53.741Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.325 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.325 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.326 DEBUG:boto:path=/bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.326 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.327 DEBUG:boto:Method: GET 2023-10-18T20:18:42.327 DEBUG:boto:Path: /bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.327 DEBUG:boto:Data: 2023-10-18T20:18:42.328 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.329 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.329 DEBUG:boto:Params: {} 2023-10-18T20:18:42.329 DEBUG:boto:Token: None 2023-10-18T20:18:42.330 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc693974cee9921-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.334 DEBUG:boto:b'bofrnf-49obj51000falseobj52023-10-18T20:17:53.741Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.334 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.335 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.335 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.337 DEBUG:boto:Path: /bofrnf-49/obj5 2023-10-18T20:18:42.337 DEBUG:boto:Data: 2023-10-18T20:18:42.337 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.338 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.339 DEBUG:boto:Params: {} 2023-10-18T20:18:42.339 DEBUG:boto:Token: None 2023-10-18T20:18:42.340 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5 2023-10-18T20:18:42.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DHX9gDeXN3PaAi3rgv2S1zdZV3Y= 2023-10-18T20:18:42.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DHX9gDeXN3PaAi3rgv2S1zdZV3Y=', 'Content-Length': '0'} 2023-10-18T20:18:42.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a195d48ee1170f9f-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.343 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.343 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.344 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.344 DEBUG:boto:Path: /bofrnf-49/obj5 2023-10-18T20:18:42.344 DEBUG:boto:Data: 2023-10-18T20:18:42.345 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.345 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.346 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.346 DEBUG:boto:Params: {} 2023-10-18T20:18:42.346 DEBUG:boto:Token: None 2023-10-18T20:18:42.347 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5 2023-10-18T20:18:42.347 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DHX9gDeXN3PaAi3rgv2S1zdZV3Y= 2023-10-18T20:18:42.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DHX9gDeXN3PaAi3rgv2S1zdZV3Y=', 'Content-Length': '0'} 2023-10-18T20:18:42.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e33787751861c205-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-18T20:18:42.350 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.351 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.351 DEBUG:boto:Method: GET 2023-10-18T20:18:42.352 DEBUG:boto:Path: /bofrnf-49/obj5 2023-10-18T20:18:42.352 DEBUG:boto:Data: 2023-10-18T20:18:42.353 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.353 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.353 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.354 DEBUG:boto:Params: {} 2023-10-18T20:18:42.354 DEBUG:boto:Token: None 2023-10-18T20:18:42.355 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5 2023-10-18T20:18:42.355 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X9Q4cWWzQZXXdplrUpKZpbZPrvk= 2023-10-18T20:18:42.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X9Q4cWWzQZXXdplrUpKZpbZPrvk=', 'Content-Length': '0'} 2023-10-18T20:18:42.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a5799fdd11f8808-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.398 DEBUG:boto:path=/bofrnf-49/obj5 2023-10-18T20:18:42.399 DEBUG:boto:auth_path=/bofrnf-49/obj5 2023-10-18T20:18:42.399 DEBUG:boto:Method: GET 2023-10-18T20:18:42.400 DEBUG:boto:Path: /bofrnf-49/obj5 2023-10-18T20:18:42.400 DEBUG:boto:Data: 2023-10-18T20:18:42.401 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.402 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.402 DEBUG:boto:Params: {} 2023-10-18T20:18:42.402 DEBUG:boto:Token: None 2023-10-18T20:18:42.403 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj5 2023-10-18T20:18:42.403 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X9Q4cWWzQZXXdplrUpKZpbZPrvk= 2023-10-18T20:18:42.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X9Q4cWWzQZXXdplrUpKZpbZPrvk=', 'Content-Length': '0'} 2023-10-18T20:18:42.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078c7b00018caa421-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.406 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.407 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.407 DEBUG:boto:path=/bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.408 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.408 DEBUG:boto:Method: GET 2023-10-18T20:18:42.409 DEBUG:boto:Path: /bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.409 DEBUG:boto:Data: 2023-10-18T20:18:42.409 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.410 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.410 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.411 DEBUG:boto:Params: {} 2023-10-18T20:18:42.411 DEBUG:boto:Token: None 2023-10-18T20:18:42.412 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.412 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005006ab1d8b7ee7b5-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.415 DEBUG:boto:b'bofrnf-49obj51000falseobj52023-10-18T20:17:53.741Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.416 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.416 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.417 DEBUG:boto:path=/bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.417 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.418 DEBUG:boto:Method: GET 2023-10-18T20:18:42.418 DEBUG:boto:Path: /bofrnf-49/?prefix=obj5 2023-10-18T20:18:42.418 DEBUG:boto:Data: 2023-10-18T20:18:42.419 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.419 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.420 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.420 DEBUG:boto:Params: {} 2023-10-18T20:18:42.421 DEBUG:boto:Token: None 2023-10-18T20:18:42.421 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.421 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000011039879c213a68-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.425 DEBUG:boto:b'bofrnf-49obj51000falseobj52023-10-18T20:17:53.741Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:18:42.426 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.426 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.427 DEBUG:boto:path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.427 DEBUG:boto:auth_path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.427 DEBUG:boto:Method: GET 2023-10-18T20:18:42.428 DEBUG:boto:Path: /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.428 DEBUG:boto:Data: 2023-10-18T20:18:42.429 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.429 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.429 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.430 DEBUG:boto:Params: {} 2023-10-18T20:18:42.430 DEBUG:boto:Token: None 2023-10-18T20:18:42.431 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.431 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8T5gx38ZtiQFHlXG/I0RwxXC0cY= 2023-10-18T20:18:42.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8T5gx38ZtiQFHlXG/I0RwxXC0cY=', 'Content-Length': '0'} 2023-10-18T20:18:42.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a827639a435d522f-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.434 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.435 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.435 DEBUG:boto:path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.436 DEBUG:boto:auth_path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.436 DEBUG:boto:Method: GET 2023-10-18T20:18:42.437 DEBUG:boto:Path: /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.437 DEBUG:boto:Data: 2023-10-18T20:18:42.437 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.438 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.438 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.439 DEBUG:boto:Params: {} 2023-10-18T20:18:42.439 DEBUG:boto:Token: None 2023-10-18T20:18:42.440 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8T5gx38ZtiQFHlXG/I0RwxXC0cY= 2023-10-18T20:18:42.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8T5gx38ZtiQFHlXG/I0RwxXC0cY=', 'Content-Length': '0'} 2023-10-18T20:18:42.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000665b04f529da7f0d-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.443 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.443 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.444 DEBUG:boto:path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.444 DEBUG:boto:auth_path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.445 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.445 DEBUG:boto:Path: /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.445 DEBUG:boto:Data: 2023-10-18T20:18:42.446 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.447 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.447 DEBUG:boto:Params: {} 2023-10-18T20:18:42.448 DEBUG:boto:Token: None 2023-10-18T20:18:42.448 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.449 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rr6bHzhbta7XwRwRNsXbRP/8J58= 2023-10-18T20:18:42.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rr6bHzhbta7XwRwRNsXbRP/8J58=', 'Content-Length': '0'} 2023-10-18T20:18:42.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074bb377f0b85c7b0-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.451 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.452 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.452 DEBUG:boto:path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.453 DEBUG:boto:auth_path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.453 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.454 DEBUG:boto:Path: /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.454 DEBUG:boto:Data: 2023-10-18T20:18:42.454 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.455 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.455 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.456 DEBUG:boto:Params: {} 2023-10-18T20:18:42.456 DEBUG:boto:Token: None 2023-10-18T20:18:42.456 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.457 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rr6bHzhbta7XwRwRNsXbRP/8J58= 2023-10-18T20:18:42.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rr6bHzhbta7XwRwRNsXbRP/8J58=', 'Content-Length': '0'} 2023-10-18T20:18:42.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053e37264a349db31-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:18:42.460 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.461 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.461 DEBUG:boto:path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.461 DEBUG:boto:auth_path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.462 DEBUG:boto:Method: GET 2023-10-18T20:18:42.462 DEBUG:boto:Path: /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.463 DEBUG:boto:Data: 2023-10-18T20:18:42.463 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.464 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.464 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.464 DEBUG:boto:Params: {} 2023-10-18T20:18:42.465 DEBUG:boto:Token: None 2023-10-18T20:18:42.465 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.466 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8T5gx38ZtiQFHlXG/I0RwxXC0cY= 2023-10-18T20:18:42.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8T5gx38ZtiQFHlXG/I0RwxXC0cY=', 'Content-Length': '0'} 2023-10-18T20:18:42.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d95750e681b97de-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.510 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.511 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.511 DEBUG:boto:path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.511 DEBUG:boto:auth_path=/bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.512 DEBUG:boto:Method: GET 2023-10-18T20:18:42.512 DEBUG:boto:Path: /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.513 DEBUG:boto:Data: 2023-10-18T20:18:42.513 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.514 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.514 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.514 DEBUG:boto:Params: {} 2023-10-18T20:18:42.515 DEBUG:boto:Token: None 2023-10-18T20:18:42.515 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6?versionId=null 2023-10-18T20:18:42.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8T5gx38ZtiQFHlXG/I0RwxXC0cY= 2023-10-18T20:18:42.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8T5gx38ZtiQFHlXG/I0RwxXC0cY=', 'Content-Length': '0'} 2023-10-18T20:18:42.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e2ecc7f6c2e2507-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.519 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.519 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.520 DEBUG:boto:path=/bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.520 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.520 DEBUG:boto:Method: GET 2023-10-18T20:18:42.521 DEBUG:boto:Path: /bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.521 DEBUG:boto:Data: 2023-10-18T20:18:42.522 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.523 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.523 DEBUG:boto:Params: {} 2023-10-18T20:18:42.524 DEBUG:boto:Token: None 2023-10-18T20:18:42.524 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.525 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014b2785fcf3042a1-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.528 DEBUG:boto:b'bofrnf-49obj61000falseobj62023-10-18T20:17:53.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.528 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.529 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.529 DEBUG:boto:path=/bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.530 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.530 DEBUG:boto:Method: GET 2023-10-18T20:18:42.531 DEBUG:boto:Path: /bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.531 DEBUG:boto:Data: 2023-10-18T20:18:42.531 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.532 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.532 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.533 DEBUG:boto:Params: {} 2023-10-18T20:18:42.533 DEBUG:boto:Token: None 2023-10-18T20:18:42.533 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.534 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a132f9cbae924b8-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.582 DEBUG:boto:b'bofrnf-49obj61000falseobj62023-10-18T20:17:53.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.583 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.583 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.584 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.584 DEBUG:boto:Path: /bofrnf-49/obj6 2023-10-18T20:18:42.584 DEBUG:boto:Data: 2023-10-18T20:18:42.585 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.585 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.586 DEBUG:boto:Params: {} 2023-10-18T20:18:42.586 DEBUG:boto:Token: None 2023-10-18T20:18:42.587 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6 2023-10-18T20:18:42.587 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SdJ/fOyUQ+TmEeJe12StYF0P/5k= 2023-10-18T20:18:42.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SdJ/fOyUQ+TmEeJe12StYF0P/5k=', 'Content-Length': '0'} 2023-10-18T20:18:42.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f47cb0c17a81e959-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.590 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.590 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.591 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.591 DEBUG:boto:Path: /bofrnf-49/obj6 2023-10-18T20:18:42.592 DEBUG:boto:Data: 2023-10-18T20:18:42.592 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.593 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.593 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.593 DEBUG:boto:Params: {} 2023-10-18T20:18:42.594 DEBUG:boto:Token: None 2023-10-18T20:18:42.594 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6 2023-10-18T20:18:42.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SdJ/fOyUQ+TmEeJe12StYF0P/5k= 2023-10-18T20:18:42.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SdJ/fOyUQ+TmEeJe12StYF0P/5k=', 'Content-Length': '0'} 2023-10-18T20:18:42.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e0675055bc0fab8-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-18T20:18:42.598 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.599 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.599 DEBUG:boto:Method: GET 2023-10-18T20:18:42.599 DEBUG:boto:Path: /bofrnf-49/obj6 2023-10-18T20:18:42.600 DEBUG:boto:Data: 2023-10-18T20:18:42.600 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.601 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.601 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.602 DEBUG:boto:Params: {} 2023-10-18T20:18:42.602 DEBUG:boto:Token: None 2023-10-18T20:18:42.603 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6 2023-10-18T20:18:42.603 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8JrI6bIZWfo6MNxQPHEugjZk6Eg= 2023-10-18T20:18:42.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8JrI6bIZWfo6MNxQPHEugjZk6Eg=', 'Content-Length': '0'} 2023-10-18T20:18:42.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0aec64ac2011939-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.606 DEBUG:boto:path=/bofrnf-49/obj6 2023-10-18T20:18:42.606 DEBUG:boto:auth_path=/bofrnf-49/obj6 2023-10-18T20:18:42.607 DEBUG:boto:Method: GET 2023-10-18T20:18:42.607 DEBUG:boto:Path: /bofrnf-49/obj6 2023-10-18T20:18:42.608 DEBUG:boto:Data: 2023-10-18T20:18:42.608 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.609 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.609 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.609 DEBUG:boto:Params: {} 2023-10-18T20:18:42.610 DEBUG:boto:Token: None 2023-10-18T20:18:42.610 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj6 2023-10-18T20:18:42.611 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8JrI6bIZWfo6MNxQPHEugjZk6Eg= 2023-10-18T20:18:42.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8JrI6bIZWfo6MNxQPHEugjZk6Eg=', 'Content-Length': '0'} 2023-10-18T20:18:42.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068a815ccc3bfa634-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.614 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.614 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.615 DEBUG:boto:path=/bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.615 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.616 DEBUG:boto:Method: GET 2023-10-18T20:18:42.616 DEBUG:boto:Path: /bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.616 DEBUG:boto:Data: 2023-10-18T20:18:42.617 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.617 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.618 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.618 DEBUG:boto:Params: {} 2023-10-18T20:18:42.618 DEBUG:boto:Token: None 2023-10-18T20:18:42.619 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.619 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000371fa6792156c5d0-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.623 DEBUG:boto:b'bofrnf-49obj61000falseobj62023-10-18T20:17:53.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.623 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.624 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.624 DEBUG:boto:path=/bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.625 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.625 DEBUG:boto:Method: GET 2023-10-18T20:18:42.625 DEBUG:boto:Path: /bofrnf-49/?prefix=obj6 2023-10-18T20:18:42.626 DEBUG:boto:Data: 2023-10-18T20:18:42.626 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.627 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.627 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.627 DEBUG:boto:Params: {} 2023-10-18T20:18:42.628 DEBUG:boto:Token: None 2023-10-18T20:18:42.628 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.629 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643fb18712431518-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.632 DEBUG:boto:b'bofrnf-49obj61000falseobj62023-10-18T20:17:53.757Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:18:42.633 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.634 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.634 DEBUG:boto:path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.634 DEBUG:boto:auth_path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.635 DEBUG:boto:Method: GET 2023-10-18T20:18:42.635 DEBUG:boto:Path: /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.636 DEBUG:boto:Data: 2023-10-18T20:18:42.636 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.637 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.637 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.637 DEBUG:boto:Params: {} 2023-10-18T20:18:42.638 DEBUG:boto:Token: None 2023-10-18T20:18:42.638 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.639 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QHU/3BaRCubywxcoEPqcVSHOT4I= 2023-10-18T20:18:42.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QHU/3BaRCubywxcoEPqcVSHOT4I=', 'Content-Length': '0'} 2023-10-18T20:18:42.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac02501b67abf813-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.642 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.642 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.643 DEBUG:boto:path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.643 DEBUG:boto:auth_path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.644 DEBUG:boto:Method: GET 2023-10-18T20:18:42.644 DEBUG:boto:Path: /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.645 DEBUG:boto:Data: 2023-10-18T20:18:42.645 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.645 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.646 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.646 DEBUG:boto:Params: {} 2023-10-18T20:18:42.647 DEBUG:boto:Token: None 2023-10-18T20:18:42.647 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.648 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QHU/3BaRCubywxcoEPqcVSHOT4I= 2023-10-18T20:18:42.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QHU/3BaRCubywxcoEPqcVSHOT4I=', 'Content-Length': '0'} 2023-10-18T20:18:42.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e5ad0ce9f1c64c0-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.651 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.651 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.652 DEBUG:boto:path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.652 DEBUG:boto:auth_path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.652 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.653 DEBUG:boto:Path: /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.653 DEBUG:boto:Data: 2023-10-18T20:18:42.654 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.654 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.655 DEBUG:boto:Params: {} 2023-10-18T20:18:42.655 DEBUG:boto:Token: None 2023-10-18T20:18:42.656 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P7NfjUzTmlXzF6LucD3ML1xn1es= 2023-10-18T20:18:42.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P7NfjUzTmlXzF6LucD3ML1xn1es=', 'Content-Length': '0'} 2023-10-18T20:18:42.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c17a23396d69208-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.659 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.659 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.660 DEBUG:boto:path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.660 DEBUG:boto:auth_path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.660 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.661 DEBUG:boto:Path: /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.661 DEBUG:boto:Data: 2023-10-18T20:18:42.662 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.663 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.663 DEBUG:boto:Params: {} 2023-10-18T20:18:42.663 DEBUG:boto:Token: None 2023-10-18T20:18:42.664 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P7NfjUzTmlXzF6LucD3ML1xn1es= 2023-10-18T20:18:42.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P7NfjUzTmlXzF6LucD3ML1xn1es=', 'Content-Length': '0'} 2023-10-18T20:18:42.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004af5c515eee68dd8-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:18:42.668 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.668 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.669 DEBUG:boto:path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.669 DEBUG:boto:auth_path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.669 DEBUG:boto:Method: GET 2023-10-18T20:18:42.670 DEBUG:boto:Path: /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.670 DEBUG:boto:Data: 2023-10-18T20:18:42.671 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.671 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.672 DEBUG:boto:Params: {} 2023-10-18T20:18:42.672 DEBUG:boto:Token: None 2023-10-18T20:18:42.673 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QHU/3BaRCubywxcoEPqcVSHOT4I= 2023-10-18T20:18:42.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QHU/3BaRCubywxcoEPqcVSHOT4I=', 'Content-Length': '0'} 2023-10-18T20:18:42.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f68c0b8d6e688e6e-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.722 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.723 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.723 DEBUG:boto:path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.724 DEBUG:boto:auth_path=/bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.724 DEBUG:boto:Method: GET 2023-10-18T20:18:42.725 DEBUG:boto:Path: /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.725 DEBUG:boto:Data: 2023-10-18T20:18:42.725 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.726 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.726 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.727 DEBUG:boto:Params: {} 2023-10-18T20:18:42.727 DEBUG:boto:Token: None 2023-10-18T20:18:42.728 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7?versionId=null 2023-10-18T20:18:42.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QHU/3BaRCubywxcoEPqcVSHOT4I= 2023-10-18T20:18:42.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QHU/3BaRCubywxcoEPqcVSHOT4I=', 'Content-Length': '0'} 2023-10-18T20:18:42.731 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000505c05c5d18f5642-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.731 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.732 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.732 DEBUG:boto:path=/bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.733 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.733 DEBUG:boto:Method: GET 2023-10-18T20:18:42.733 DEBUG:boto:Path: /bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.734 DEBUG:boto:Data: 2023-10-18T20:18:42.734 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.735 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.735 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.736 DEBUG:boto:Params: {} 2023-10-18T20:18:42.736 DEBUG:boto:Token: None 2023-10-18T20:18:42.737 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000107ed305aaa522a8-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.740 DEBUG:boto:b'bofrnf-49obj71000falseobj72023-10-18T20:17:53.817Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.741 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.742 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.742 DEBUG:boto:path=/bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.742 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.743 DEBUG:boto:Method: GET 2023-10-18T20:18:42.743 DEBUG:boto:Path: /bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.744 DEBUG:boto:Data: 2023-10-18T20:18:42.744 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.745 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.745 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.745 DEBUG:boto:Params: {} 2023-10-18T20:18:42.746 DEBUG:boto:Token: None 2023-10-18T20:18:42.746 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.747 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e7ed722f8c1dc84-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.750 DEBUG:boto:b'bofrnf-49obj71000falseobj72023-10-18T20:17:53.817Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.751 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.751 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.751 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.752 DEBUG:boto:Path: /bofrnf-49/obj7 2023-10-18T20:18:42.752 DEBUG:boto:Data: 2023-10-18T20:18:42.753 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.753 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.754 DEBUG:boto:Params: {} 2023-10-18T20:18:42.754 DEBUG:boto:Token: None 2023-10-18T20:18:42.755 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7 2023-10-18T20:18:42.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:myXflRaEuTP3oDAA7eFlqOjurCs= 2023-10-18T20:18:42.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:myXflRaEuTP3oDAA7eFlqOjurCs=', 'Content-Length': '0'} 2023-10-18T20:18:42.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a68961e070cee468-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.758 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.758 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.759 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.759 DEBUG:boto:Path: /bofrnf-49/obj7 2023-10-18T20:18:42.759 DEBUG:boto:Data: 2023-10-18T20:18:42.760 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.760 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.761 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.761 DEBUG:boto:Params: {} 2023-10-18T20:18:42.761 DEBUG:boto:Token: None 2023-10-18T20:18:42.762 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7 2023-10-18T20:18:42.762 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:myXflRaEuTP3oDAA7eFlqOjurCs= 2023-10-18T20:18:42.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:myXflRaEuTP3oDAA7eFlqOjurCs=', 'Content-Length': '0'} 2023-10-18T20:18:42.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a64ded50f2f9c430-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-18T20:18:42.765 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.766 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.766 DEBUG:boto:Method: GET 2023-10-18T20:18:42.767 DEBUG:boto:Path: /bofrnf-49/obj7 2023-10-18T20:18:42.767 DEBUG:boto:Data: 2023-10-18T20:18:42.768 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.768 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.769 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.769 DEBUG:boto:Params: {} 2023-10-18T20:18:42.769 DEBUG:boto:Token: None 2023-10-18T20:18:42.770 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7 2023-10-18T20:18:42.770 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OmdvAa3q/9aFY/9gzWAmjvlWClU= 2023-10-18T20:18:42.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OmdvAa3q/9aFY/9gzWAmjvlWClU=', 'Content-Length': '0'} 2023-10-18T20:18:42.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba91295b36229382-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.773 DEBUG:boto:path=/bofrnf-49/obj7 2023-10-18T20:18:42.774 DEBUG:boto:auth_path=/bofrnf-49/obj7 2023-10-18T20:18:42.774 DEBUG:boto:Method: GET 2023-10-18T20:18:42.775 DEBUG:boto:Path: /bofrnf-49/obj7 2023-10-18T20:18:42.775 DEBUG:boto:Data: 2023-10-18T20:18:42.776 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.776 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.776 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.777 DEBUG:boto:Params: {} 2023-10-18T20:18:42.777 DEBUG:boto:Token: None 2023-10-18T20:18:42.778 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj7 2023-10-18T20:18:42.778 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OmdvAa3q/9aFY/9gzWAmjvlWClU= 2023-10-18T20:18:42.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OmdvAa3q/9aFY/9gzWAmjvlWClU=', 'Content-Length': '0'} 2023-10-18T20:18:42.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001da78a124e15c0ea-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.781 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.782 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.782 DEBUG:boto:path=/bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.782 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.783 DEBUG:boto:Method: GET 2023-10-18T20:18:42.783 DEBUG:boto:Path: /bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.784 DEBUG:boto:Data: 2023-10-18T20:18:42.784 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.784 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.785 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.785 DEBUG:boto:Params: {} 2023-10-18T20:18:42.786 DEBUG:boto:Token: None 2023-10-18T20:18:42.786 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.787 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2337b3cdaf5a134-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.790 DEBUG:boto:b'bofrnf-49obj71000falseobj72023-10-18T20:17:53.817Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.791 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.791 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.791 DEBUG:boto:path=/bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.792 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.792 DEBUG:boto:Method: GET 2023-10-18T20:18:42.793 DEBUG:boto:Path: /bofrnf-49/?prefix=obj7 2023-10-18T20:18:42.793 DEBUG:boto:Data: 2023-10-18T20:18:42.793 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.794 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.794 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.795 DEBUG:boto:Params: {} 2023-10-18T20:18:42.795 DEBUG:boto:Token: None 2023-10-18T20:18:42.796 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.796 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba34bb76cf2ee196-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.799 DEBUG:boto:b'bofrnf-49obj71000falseobj72023-10-18T20:17:53.817Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:18:42.800 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.801 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.801 DEBUG:boto:path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.802 DEBUG:boto:auth_path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.802 DEBUG:boto:Method: GET 2023-10-18T20:18:42.803 DEBUG:boto:Path: /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.803 DEBUG:boto:Data: 2023-10-18T20:18:42.803 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.804 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.804 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.805 DEBUG:boto:Params: {} 2023-10-18T20:18:42.805 DEBUG:boto:Token: None 2023-10-18T20:18:42.805 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.806 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MAXyJort1HhTuuKfs/TqVYlu4LI= 2023-10-18T20:18:42.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MAXyJort1HhTuuKfs/TqVYlu4LI=', 'Content-Length': '0'} 2023-10-18T20:18:42.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1ec769b8f0f32da-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.809 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.810 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.810 DEBUG:boto:path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.810 DEBUG:boto:auth_path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.811 DEBUG:boto:Method: GET 2023-10-18T20:18:42.811 DEBUG:boto:Path: /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.812 DEBUG:boto:Data: 2023-10-18T20:18:42.812 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.812 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.813 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.813 DEBUG:boto:Params: {} 2023-10-18T20:18:42.814 DEBUG:boto:Token: None 2023-10-18T20:18:42.814 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.815 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MAXyJort1HhTuuKfs/TqVYlu4LI= 2023-10-18T20:18:42.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MAXyJort1HhTuuKfs/TqVYlu4LI=', 'Content-Length': '0'} 2023-10-18T20:18:42.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b66bd25de7e72ec-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.818 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.818 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.818 DEBUG:boto:path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.819 DEBUG:boto:auth_path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.819 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.820 DEBUG:boto:Path: /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.820 DEBUG:boto:Data: 2023-10-18T20:18:42.820 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.821 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.821 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.822 DEBUG:boto:Params: {} 2023-10-18T20:18:42.822 DEBUG:boto:Token: None 2023-10-18T20:18:42.822 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.823 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3hwWLXAZwMlNdkzEzY53PCn4MTU= 2023-10-18T20:18:42.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3hwWLXAZwMlNdkzEzY53PCn4MTU=', 'Content-Length': '0'} 2023-10-18T20:18:42.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8afbc9850515836-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.826 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.826 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.826 DEBUG:boto:path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.827 DEBUG:boto:auth_path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.827 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.828 DEBUG:boto:Path: /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.828 DEBUG:boto:Data: 2023-10-18T20:18:42.828 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.829 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.829 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.830 DEBUG:boto:Params: {} 2023-10-18T20:18:42.830 DEBUG:boto:Token: None 2023-10-18T20:18:42.830 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.831 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3hwWLXAZwMlNdkzEzY53PCn4MTU= 2023-10-18T20:18:42.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3hwWLXAZwMlNdkzEzY53PCn4MTU=', 'Content-Length': '0'} 2023-10-18T20:18:42.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040ac5043dee306f1-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:18:42.834 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.834 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.835 DEBUG:boto:path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.835 DEBUG:boto:auth_path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.836 DEBUG:boto:Method: GET 2023-10-18T20:18:42.836 DEBUG:boto:Path: /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.836 DEBUG:boto:Data: 2023-10-18T20:18:42.837 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.837 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.838 DEBUG:boto:Params: {} 2023-10-18T20:18:42.838 DEBUG:boto:Token: None 2023-10-18T20:18:42.839 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.839 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MAXyJort1HhTuuKfs/TqVYlu4LI= 2023-10-18T20:18:42.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MAXyJort1HhTuuKfs/TqVYlu4LI=', 'Content-Length': '0'} 2023-10-18T20:18:42.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f4fd47306caf611-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.842 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.843 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.843 DEBUG:boto:path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.844 DEBUG:boto:auth_path=/bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.844 DEBUG:boto:Method: GET 2023-10-18T20:18:42.844 DEBUG:boto:Path: /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.845 DEBUG:boto:Data: 2023-10-18T20:18:42.845 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.846 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.846 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.846 DEBUG:boto:Params: {} 2023-10-18T20:18:42.847 DEBUG:boto:Token: None 2023-10-18T20:18:42.847 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8?versionId=null 2023-10-18T20:18:42.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MAXyJort1HhTuuKfs/TqVYlu4LI= 2023-10-18T20:18:42.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MAXyJort1HhTuuKfs/TqVYlu4LI=', 'Content-Length': '0'} 2023-10-18T20:18:42.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a411ce5909bfe635-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.851 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.851 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.851 DEBUG:boto:path=/bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.852 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.852 DEBUG:boto:Method: GET 2023-10-18T20:18:42.853 DEBUG:boto:Path: /bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.853 DEBUG:boto:Data: 2023-10-18T20:18:42.854 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.854 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.854 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.855 DEBUG:boto:Params: {} 2023-10-18T20:18:42.855 DEBUG:boto:Token: None 2023-10-18T20:18:42.856 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.856 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a3527fdc6fb1b9-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.859 DEBUG:boto:b'bofrnf-49obj81000falseobj82023-10-18T20:17:53.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.860 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.860 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.861 DEBUG:boto:path=/bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.861 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.862 DEBUG:boto:Method: GET 2023-10-18T20:18:42.862 DEBUG:boto:Path: /bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.862 DEBUG:boto:Data: 2023-10-18T20:18:42.863 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.863 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.864 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.864 DEBUG:boto:Params: {} 2023-10-18T20:18:42.865 DEBUG:boto:Token: None 2023-10-18T20:18:42.865 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.865 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d87fef8e0cae7fb-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.910 DEBUG:boto:b'bofrnf-49obj81000falseobj82023-10-18T20:17:53.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.911 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.911 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.912 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.912 DEBUG:boto:Path: /bofrnf-49/obj8 2023-10-18T20:18:42.912 DEBUG:boto:Data: 2023-10-18T20:18:42.913 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.913 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.914 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.914 DEBUG:boto:Params: {} 2023-10-18T20:18:42.914 DEBUG:boto:Token: None 2023-10-18T20:18:42.915 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8 2023-10-18T20:18:42.915 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IUIyRaMjRDwOAoBSyuytuK23igY= 2023-10-18T20:18:42.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IUIyRaMjRDwOAoBSyuytuK23igY=', 'Content-Length': '0'} 2023-10-18T20:18:42.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000120d61a590d63e81-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.918 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.918 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.919 DEBUG:boto:Method: HEAD 2023-10-18T20:18:42.919 DEBUG:boto:Path: /bofrnf-49/obj8 2023-10-18T20:18:42.919 DEBUG:boto:Data: 2023-10-18T20:18:42.920 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.920 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.921 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.921 DEBUG:boto:Params: {} 2023-10-18T20:18:42.921 DEBUG:boto:Token: None 2023-10-18T20:18:42.922 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8 2023-10-18T20:18:42.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IUIyRaMjRDwOAoBSyuytuK23igY= 2023-10-18T20:18:42.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IUIyRaMjRDwOAoBSyuytuK23igY=', 'Content-Length': '0'} 2023-10-18T20:18:42.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009df5498f2d8e1f8c-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-18T20:18:42.925 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.926 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.926 DEBUG:boto:Method: GET 2023-10-18T20:18:42.927 DEBUG:boto:Path: /bofrnf-49/obj8 2023-10-18T20:18:42.927 DEBUG:boto:Data: 2023-10-18T20:18:42.928 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.928 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.928 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.929 DEBUG:boto:Params: {} 2023-10-18T20:18:42.929 DEBUG:boto:Token: None 2023-10-18T20:18:42.930 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8 2023-10-18T20:18:42.930 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:05mDdwieMfQf3n6xnNWNmHwRUs4= 2023-10-18T20:18:42.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:05mDdwieMfQf3n6xnNWNmHwRUs4=', 'Content-Length': '0'} 2023-10-18T20:18:42.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0a3ed8cf1f59aba-0065303da2-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.933 DEBUG:boto:path=/bofrnf-49/obj8 2023-10-18T20:18:42.934 DEBUG:boto:auth_path=/bofrnf-49/obj8 2023-10-18T20:18:42.934 DEBUG:boto:Method: GET 2023-10-18T20:18:42.935 DEBUG:boto:Path: /bofrnf-49/obj8 2023-10-18T20:18:42.935 DEBUG:boto:Data: 2023-10-18T20:18:42.935 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.936 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.936 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.937 DEBUG:boto:Params: {} 2023-10-18T20:18:42.937 DEBUG:boto:Token: None 2023-10-18T20:18:42.938 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/obj8 2023-10-18T20:18:42.938 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:05mDdwieMfQf3n6xnNWNmHwRUs4= 2023-10-18T20:18:42.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:05mDdwieMfQf3n6xnNWNmHwRUs4=', 'Content-Length': '0'} 2023-10-18T20:18:42.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:17:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-49:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.18'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e1e1dd7fdfc85c1-0065303da2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.941 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.942 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.942 DEBUG:boto:path=/bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.942 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.943 DEBUG:boto:Method: GET 2023-10-18T20:18:42.943 DEBUG:boto:Path: /bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.944 DEBUG:boto:Data: 2023-10-18T20:18:42.944 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:42.945 DEBUG:boto:Port: 8001 2023-10-18T20:18:42.946 DEBUG:boto:Params: {} 2023-10-18T20:18:42.946 DEBUG:boto:Token: None 2023-10-18T20:18:42.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.947 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9d0dedc986e0d06-0065303da2-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.951 DEBUG:boto:b'bofrnf-49obj81000falseobj82023-10-18T20:17:53.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.952 DEBUG:boto:path=/bofrnf-49/ 2023-10-18T20:18:42.952 DEBUG:boto:auth_path=/bofrnf-49/ 2023-10-18T20:18:42.953 DEBUG:boto:path=/bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.953 DEBUG:boto:auth_path=/bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.954 DEBUG:boto:Method: GET 2023-10-18T20:18:42.954 DEBUG:boto:Path: /bofrnf-49/?prefix=obj8 2023-10-18T20:18:42.955 DEBUG:boto:Data: 2023-10-18T20:18:42.955 DEBUG:boto:Headers: {} 2023-10-18T20:18:42.956 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:42.956 DEBUG:boto:Port: 8000 2023-10-18T20:18:42.957 DEBUG:boto:Params: {} 2023-10-18T20:18:42.957 DEBUG:boto:Token: None 2023-10-18T20:18:42.958 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:42 GMT /bofrnf-49/ 2023-10-18T20:18:42.958 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc= 2023-10-18T20:18:42.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BiqV7gqN1leRTvvEFLnwluEEAc=', 'Content-Length': '0'} 2023-10-18T20:18:42.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d3dd88c79b1d3f0-0065303da2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:42.962 DEBUG:boto:b'bofrnf-49obj81000falseobj82023-10-18T20:17:53.834Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:18:42.962 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-49 zones={a2, a1} 2023-10-18T20:18:42.963 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-18T20:18:42.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:42.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:42.964 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:18:43.016 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:18:43.016 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:18:43.061 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:18:43.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.062 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000006854", 2023-10-18T20:18:43.062 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:17:54.098359Z" 2023-10-18T20:18:43.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.063 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000006579", 2023-10-18T20:18:43.063 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:18:17.061090Z" 2023-10-18T20:18:43.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.063 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-10-18T20:18:43.064 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:18:16.898488Z" 2023-10-18T20:18:43.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.064 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000009951", 2023-10-18T20:18:43.064 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:18:16.937870Z" 2023-10-18T20:18:43.065 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.065 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:18:43.066 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000006854', 1: '00000000000000000000:00000000000000006579', 2: '00000000000000000000:00000000000000004597', 3: '00000000000000000000:00000000000000009951'} 2023-10-18T20:18:43.066 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:18:43.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:43.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:43.068 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:18:43.119 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:18:43.119 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:18:43.161 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:18:43.162 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:18:43.162 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:18:43.162 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:18:43.162 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:18:43.163 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 2477610155421060198 2023-10-18T20:18:43.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.163 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:18:43.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.163 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:18:43.164 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:43.164 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:18:43.164 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000006854", 2023-10-18T20:18:43.164 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:43.164 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:43.165 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:43.165 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:17:54.098359Z" 2023-10-18T20:18:43.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.166 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:18:43.166 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:43.166 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:18:43.166 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000006579", 2023-10-18T20:18:43.166 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:43.167 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:43.167 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:43.167 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:18:17.061090Z" 2023-10-18T20:18:43.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.168 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:18:43.168 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:43.168 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:18:43.169 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-10-18T20:18:43.169 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:43.169 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:43.169 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:43.169 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:18:16.898488Z" 2023-10-18T20:18:43.170 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.170 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:18:43.170 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:43.171 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:18:43.171 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000009951", 2023-10-18T20:18:43.171 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:43.172 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:43.172 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:43.172 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:18:16.937870Z" 2023-10-18T20:18:43.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.173 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:18:43.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.173 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:18:43.173 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:18:43.173 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:18:43.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.174 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:18:43Z" 2023-10-18T20:18:43.174 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:18:43.175 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2477610155421060198 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006854", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:17:54.098359Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006579", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:18:17.061090Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:18:16.898488Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009951", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:18:16.937870Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:18:43Z" } 2023-10-18T20:18:43.175 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006854', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:17:54.098359Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006579', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:18:17.061090Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:18:16.898488Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009951', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:18:16.937870Z'}}] 2023-10-18T20:18:43.176 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006854', 1: '00000000000000000000:00000000000000006579', 2: '00000000000000000000:00000000000000004597', 3: '00000000000000000000:00000000000000009951'} 2023-10-18T20:18:43.176 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006854', 1: '00000000000000000000:00000000000000006579', 2: '00000000000000000000:00000000000000004597', 3: '00000000000000000000:00000000000000009951'} 2023-10-18T20:18:43.177 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:18:43.177 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-18T20:18:43.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:43.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:43.179 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:18:43.227 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:18:43.227 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:18:43.272 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:18:43.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.273 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004318", 2023-10-18T20:18:43.273 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:18:03.645130Z" 2023-10-18T20:18:43.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004600", 2023-10-18T20:18:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:18:03.646960Z" 2023-10-18T20:18:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003187", 2023-10-18T20:18:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:18:03.648505Z" 2023-10-18T20:18:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.275 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000008541", 2023-10-18T20:18:43.276 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:18:03.650064Z" 2023-10-18T20:18:43.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.276 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:18:43.277 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000004318', 1: '00000000000000000000:00000000000000004600', 2: '00000000000000000000:00000000000000003187', 3: '00000000000000000000:00000000000000008541'} 2023-10-18T20:18:43.278 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:18:43.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:43.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:43.279 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:18:43.330 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:18:43.330 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:18:43.377 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:18:43.378 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:18:43.378 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:18:43.378 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:18:43.378 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:18:43.378 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:18:43.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.379 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:18:43.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.379 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:18:43.380 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:43.380 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:18:43.380 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004318", 2023-10-18T20:18:43.380 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:43.380 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:43.381 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:43.381 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:18:03.645130Z" 2023-10-18T20:18:43.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.382 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:18:43.382 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:43.382 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:18:43.382 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000004600", 2023-10-18T20:18:43.382 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:43.383 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:43.383 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:43.383 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:18:03.646960Z" 2023-10-18T20:18:43.383 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.384 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:18:43.384 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:43.384 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:18:43.385 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000003187", 2023-10-18T20:18:43.385 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:43.385 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:43.385 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:43.385 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:18:03.648505Z" 2023-10-18T20:18:43.386 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:43.386 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:18:43.386 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:43.387 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:18:43.387 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000008541", 2023-10-18T20:18:43.387 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:43.387 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:43.387 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:43.388 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:18:03.650064Z" 2023-10-18T20:18:43.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:43.388 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:18:43.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.389 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:18:43.389 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:18:43.389 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:18:43.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:43.389 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:18:43Z" 2023-10-18T20:18:43.390 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:18:43.391 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004318", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:18:03.645130Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004600", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:18:03.646960Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:18:03.648505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008541", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:18:03.650064Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:18:43Z" } 2023-10-18T20:18:43.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004318', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:18:03.645130Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004600', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:18:03.646960Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:18:03.648505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008541', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:18:03.650064Z'}}] 2023-10-18T20:18:43.392 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004318', 1: '00000000000000000000:00000000000000004600', 2: '00000000000000000000:00000000000000003187', 3: '00000000000000000000:00000000000000008541'} 2023-10-18T20:18:43.392 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004318', 1: '00000000000000000000:00000000000000004600', 2: '00000000000000000000:00000000000000003187', 3: '00000000000000000000:00000000000000008541'} 2023-10-18T20:18:43.392 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:18:43.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-10-18T20:18:43.395 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-50 2023-10-18T20:18:43.395 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:18:43.395 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:18:43.396 DEBUG:boto:Method: PUT 2023-10-18T20:18:43.396 DEBUG:boto:Path: /bofrnf-50/ 2023-10-18T20:18:43.397 DEBUG:boto:Data: 2023-10-18T20:18:43.397 DEBUG:boto:Headers: {} 2023-10-18T20:18:43.397 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:43.398 DEBUG:boto:Port: 8000 2023-10-18T20:18:43.398 DEBUG:boto:Params: {} 2023-10-18T20:18:43.399 DEBUG:boto:Token: None 2023-10-18T20:18:43.399 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:18:43 GMT /bofrnf-50/ 2023-10-18T20:18:43.399 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q22TnaKOC4qr2U0wC52/r3yxqnA= 2023-10-18T20:18:43.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q22TnaKOC4qr2U0wC52/r3yxqnA=', 'Content-Length': '0'} 2023-10-18T20:18:43.413 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000037848d376094f4c9-0065303da3-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:43.414 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-51 2023-10-18T20:18:43.414 DEBUG:boto:path=/bofrnf-51/ 2023-10-18T20:18:43.415 DEBUG:boto:auth_path=/bofrnf-51/ 2023-10-18T20:18:43.415 DEBUG:boto:Method: PUT 2023-10-18T20:18:43.415 DEBUG:boto:Path: /bofrnf-51/ 2023-10-18T20:18:43.416 DEBUG:boto:Data: 2023-10-18T20:18:43.416 DEBUG:boto:Headers: {} 2023-10-18T20:18:43.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:18:43.417 DEBUG:boto:Port: 8001 2023-10-18T20:18:43.417 DEBUG:boto:Params: {} 2023-10-18T20:18:43.418 DEBUG:boto:Token: None 2023-10-18T20:18:43.418 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:18:43 GMT /bofrnf-51/ 2023-10-18T20:18:43.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ANMfEUdc0dH9VWGmombTdKZQD4Q= 2023-10-18T20:18:43.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ANMfEUdc0dH9VWGmombTdKZQD4Q=', 'Content-Length': '0'} 2023-10-18T20:18:43.443 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002db0b59ea2054d7a-0065303da3-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:43.443 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:18:43.444 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:18:43.444 DEBUG:boto:path=/bofrnf-50/MULTIPART?uploads 2023-10-18T20:18:43.444 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?uploads 2023-10-18T20:18:43.445 DEBUG:boto:Method: POST 2023-10-18T20:18:43.445 DEBUG:boto:Path: /bofrnf-50/MULTIPART?uploads 2023-10-18T20:18:43.446 DEBUG:boto:Data: 2023-10-18T20:18:43.446 DEBUG:boto:Headers: {} 2023-10-18T20:18:43.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:43.447 DEBUG:boto:Port: 8000 2023-10-18T20:18:43.447 DEBUG:boto:Params: {} 2023-10-18T20:18:43.447 DEBUG:boto:Token: None 2023-10-18T20:18:43.448 DEBUG:boto:StringToSign: POST Wed, 18 Oct 2023 20:18:43 GMT /bofrnf-50/MULTIPART?uploads 2023-10-18T20:18:43.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YfgjjXQYVZFBDglRCsY5ko5NZVM= 2023-10-18T20:18:43.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YfgjjXQYVZFBDglRCsY5ko5NZVM=', 'Content-Length': '0'} 2023-10-18T20:18:44.578 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe393ab8c59b7b61-0065303da3-1664-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:44.579 DEBUG:boto:b'bofrnf-50MULTIPART2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR' 2023-10-18T20:18:44.610 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.642 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.642 DEBUG:boto:path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=1 2023-10-18T20:18:44.643 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=1 2023-10-18T20:18:44.643 DEBUG:boto:Method: PUT 2023-10-18T20:18:44.644 DEBUG:boto:Path: /bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=1 2023-10-18T20:18:44.644 DEBUG:boto:Data: 2023-10-18T20:18:44.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-18T20:18:44.645 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:44.645 DEBUG:boto:Port: 8000 2023-10-18T20:18:44.645 DEBUG:boto:Params: {} 2023-10-18T20:18:44.646 DEBUG:boto:Token: None 2023-10-18T20:18:44.647 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Wed, 18 Oct 2023 20:18:44 GMT /bofrnf-50/MULTIPART?partNumber=1&uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.647 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:36sGotJ8etHesKbfIpk69LERQ9o= 2023-10-18T20:18:44.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:18:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:36sGotJ8etHesKbfIpk69LERQ9o='} 2023-10-18T20:18:44.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b2ece71ef2db923-0065303da4-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:44.711 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.712 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.712 DEBUG:boto:path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=2 2023-10-18T20:18:44.713 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=2 2023-10-18T20:18:44.713 DEBUG:boto:Method: PUT 2023-10-18T20:18:44.713 DEBUG:boto:Path: /bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=2 2023-10-18T20:18:44.714 DEBUG:boto:Data: 2023-10-18T20:18:44.714 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-18T20:18:44.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:44.715 DEBUG:boto:Port: 8000 2023-10-18T20:18:44.716 DEBUG:boto:Params: {} 2023-10-18T20:18:44.716 DEBUG:boto:Token: None 2023-10-18T20:18:44.717 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Wed, 18 Oct 2023 20:18:44 GMT /bofrnf-50/MULTIPART?partNumber=2&uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XjVsrSi+cz4Qg3MKu6kV6vWYIrM= 2023-10-18T20:18:44.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:18:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XjVsrSi+cz4Qg3MKu6kV6vWYIrM='} 2023-10-18T20:18:44.748 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e6dd60ea049fe9c-0065303da4-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:44.768 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.769 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.769 DEBUG:boto:path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=3 2023-10-18T20:18:44.770 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=3 2023-10-18T20:18:44.770 DEBUG:boto:Method: PUT 2023-10-18T20:18:44.771 DEBUG:boto:Path: /bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=3 2023-10-18T20:18:44.771 DEBUG:boto:Data: 2023-10-18T20:18:44.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-18T20:18:44.772 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:44.772 DEBUG:boto:Port: 8000 2023-10-18T20:18:44.773 DEBUG:boto:Params: {} 2023-10-18T20:18:44.773 DEBUG:boto:Token: None 2023-10-18T20:18:44.773 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Wed, 18 Oct 2023 20:18:44 GMT /bofrnf-50/MULTIPART?partNumber=3&uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nrqSpZqYLvbh6z1n1DNpiCcnCic= 2023-10-18T20:18:44.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:18:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nrqSpZqYLvbh6z1n1DNpiCcnCic='} 2023-10-18T20:18:44.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f87f6cc609829443-0065303da4-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:44.827 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.828 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.828 DEBUG:boto:path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=4 2023-10-18T20:18:44.828 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=4 2023-10-18T20:18:44.829 DEBUG:boto:Method: PUT 2023-10-18T20:18:44.829 DEBUG:boto:Path: /bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR&partNumber=4 2023-10-18T20:18:44.830 DEBUG:boto:Data: 2023-10-18T20:18:44.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-18T20:18:44.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:44.831 DEBUG:boto:Port: 8000 2023-10-18T20:18:44.831 DEBUG:boto:Params: {} 2023-10-18T20:18:44.832 DEBUG:boto:Token: None 2023-10-18T20:18:44.832 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Wed, 18 Oct 2023 20:18:44 GMT /bofrnf-50/MULTIPART?partNumber=4&uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.833 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yw/BOkgmUQE9coxqjhObC18B+xs= 2023-10-18T20:18:44.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:18:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yw/BOkgmUQE9coxqjhObC18B+xs='} 2023-10-18T20:18:44.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006288c21276d93e29-0065303da4-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:44.864 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.864 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.864 DEBUG:boto:path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.865 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.865 DEBUG:boto:Method: GET 2023-10-18T20:18:44.865 DEBUG:boto:Path: /bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.866 DEBUG:boto:Data: 2023-10-18T20:18:44.866 DEBUG:boto:Headers: {} 2023-10-18T20:18:44.867 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:44.867 DEBUG:boto:Port: 8000 2023-10-18T20:18:44.867 DEBUG:boto:Params: {} 2023-10-18T20:18:44.868 DEBUG:boto:Token: None 2023-10-18T20:18:44.868 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:18:44 GMT /bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.869 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G8HwgKvYEi/tOPy2TaBIXF+sKoU= 2023-10-18T20:18:44.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G8HwgKvYEi/tOPy2TaBIXF+sKoU=', 'Content-Length': '0'} 2023-10-18T20:18:44.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062bab11092944dc8-0065303da4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:44.915 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.915 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:18:44.915 DEBUG:boto:path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.916 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.916 DEBUG:boto:Method: POST 2023-10-18T20:18:44.916 DEBUG:boto:Path: /bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.917 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-10-18T20:18:44.917 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-10-18T20:18:44.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:18:44.918 DEBUG:boto:Port: 8000 2023-10-18T20:18:44.918 DEBUG:boto:Params: {} 2023-10-18T20:18:44.919 DEBUG:boto:Token: None 2023-10-18T20:18:44.919 DEBUG:boto:StringToSign: POST text/xml Wed, 18 Oct 2023 20:18:44 GMT /bofrnf-50/MULTIPART?uploadId=2~NwbAFk7P0xf6c4b33frb7UMvquDiiwR 2023-10-18T20:18:44.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jPTyDEV8NTflkbFXNq73CFTzY2c= 2023-10-18T20:18:44.920 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:18:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jPTyDEV8NTflkbFXNq73CFTzY2c=', 'Content-Length': '460'} 2023-10-18T20:18:44.928 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa5c771fafb3a231-0065303da4-1664-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:18:44.929 DEBUG:boto:http://smithi082.front.sepia.ceph.com:8000/bofrnf-50/MULTIPARTbofrnf-50MULTIPART 2023-10-18T20:18:44.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:44.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:44.931 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:18:44.979 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:18:44.979 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:18:45.008 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:18:45.020 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:18:45.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:45.021 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660323.432292_829.1", 2023-10-18T20:18:45.021 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:18:43.432292Z" 2023-10-18T20:18:45.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:45.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:45.022 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:18:45.022 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:18:45.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:45.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:45.023 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:18:45.023 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:18:45.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:45.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:45.024 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:18:45.024 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:18:45.024 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:45.024 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:18:45.025 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660323.432292_829.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:18:45.026 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:18:45.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:45.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:18:45.027 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:18:45.076 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:18:45.077 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:18:45.104 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:45.103+0000 7f944f38c780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:18:45.105 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:45.104+0000 7f944f38c780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:18:45.117 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:18:45.117 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:18:45.118 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:18:45.118 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:18:45.118 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:18:45.118 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:18:45.119 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:18:45.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:45.119 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:18:45.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:45.120 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:18:45.120 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:45.120 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:18:45.120 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660323.432292_829.1", 2023-10-18T20:18:45.120 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:45.121 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:18:45.121 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:45.121 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:18:43.432292Z", 2023-10-18T20:18:45.121 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:18:45.121 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:45.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:45.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:45.122 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:18:45.122 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:45.122 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:18:45.123 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:18:45.123 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:45.123 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:45.123 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:45.123 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:18:45.124 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:18:45.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:45.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:45.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:45.124 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:18:45.125 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:45.125 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:18:45.125 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:18:45.125 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:45.125 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:45.126 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:45.126 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:18:45.126 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:18:45.126 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:45.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:45.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:18:45.127 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:18:45.127 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:18:45.127 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:18:45.127 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:18:45.128 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:18:45.128 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:18:45.128 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:18:45.128 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:18:45.129 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:18:45.129 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:45.129 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:18:45.129 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:18:45.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:45.130 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:18:45.130 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:18:45.130 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:18:45.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:18:45.130 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:18:45Z" 2023-10-18T20:18:45.131 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:18:45.132 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660323.432292_829.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:18:43.432292Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:18:45Z" } 2023-10-18T20:18:45.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660323.432292_829.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:18:43.432292Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:18:45.133 DEBUG:rgw_multi.tests:log_status={0: '1_1697660323.432292_829.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:18:45.133 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660323.432292_829.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:18:45.133 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:18:45.134 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-50 2023-10-18T20:18:45.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:18:45.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:18:45.135 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:18:45.193 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:18:45.193 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:18:45.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:45.223+0000 7fe83bc05780 1 waiting to reach incremental sync.. 2023-10-18T20:18:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:50.224+0000 7fe83bc05780 1 waiting to reach incremental sync.. 2023-10-18T20:18:55.226 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:18:55.224+0000 7fe83bc05780 1 waiting to reach incremental sync.. 2023-10-18T20:19:00.228 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:00.226+0000 7fe83bc05780 1 bucket sync caught up with source: 2023-10-18T20:19:00.228 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000011.52.6, , , , , ] 2023-10-18T20:19:00.228 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000011.52.6, , , , , ] 2023-10-18T20:19:00.229 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:00.226+0000 7fe83bc05780 0 bucket checkpoint complete 2023-10-18T20:19:00.244 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-50 2023-10-18T20:19:00.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:19:00.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:19:00.246 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:19:00.310 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:19:00.310 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:19:00.347 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:00.345+0000 7fafe7363780 1 waiting to reach incremental sync.. 2023-10-18T20:19:05.347 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:05.346+0000 7fafe7363780 1 waiting to reach incremental sync.. 2023-10-18T20:19:10.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:10.345+0000 7fafe7363780 1 waiting to reach incremental sync.. 2023-10-18T20:19:15.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:15.346+0000 7fafe7363780 1 waiting to reach incremental sync.. 2023-10-18T20:19:20.351 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:20.348+0000 7fafe7363780 1 bucket sync caught up with source: 2023-10-18T20:19:20.351 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.471.6, , , , , ] 2023-10-18T20:19:20.352 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.471.6, , , , , ] 2023-10-18T20:19:20.352 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:20.348+0000 7fafe7363780 0 bucket checkpoint complete 2023-10-18T20:19:20.367 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-50 zones={a2, a1} 2023-10-18T20:19:20.367 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:20.368 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:20.368 DEBUG:boto:Method: HEAD 2023-10-18T20:19:20.369 DEBUG:boto:Path: /bofrnf-50/ 2023-10-18T20:19:20.369 DEBUG:boto:Data: 2023-10-18T20:19:20.369 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:20.370 DEBUG:boto:Port: 8001 2023-10-18T20:19:20.370 DEBUG:boto:Params: {} 2023-10-18T20:19:20.371 DEBUG:boto:Token: None 2023-10-18T20:19:20.372 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/ 2023-10-18T20:19:20.372 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:blO+QJCRqN02GXfbuQ+mTnIWcos= 2023-10-18T20:19:20.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:blO+QJCRqN02GXfbuQ+mTnIWcos=', 'Content-Length': '0'} 2023-10-18T20:19:20.376 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000461d22bc2d338cdb-0065303dc8-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.376 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:20.376 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:20.377 DEBUG:boto:Method: HEAD 2023-10-18T20:19:20.377 DEBUG:boto:Path: /bofrnf-50/ 2023-10-18T20:19:20.377 DEBUG:boto:Data: 2023-10-18T20:19:20.378 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.378 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:20.379 DEBUG:boto:Port: 8000 2023-10-18T20:19:20.379 DEBUG:boto:Params: {} 2023-10-18T20:19:20.379 DEBUG:boto:Token: None 2023-10-18T20:19:20.380 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/ 2023-10-18T20:19:20.380 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:blO+QJCRqN02GXfbuQ+mTnIWcos= 2023-10-18T20:19:20.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:blO+QJCRqN02GXfbuQ+mTnIWcos=', 'Content-Length': '0'} 2023-10-18T20:19:20.383 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007321c2ce0e7377e2-0065303dc8-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.384 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:19:20.384 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:20.385 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:20.385 DEBUG:boto:path=/bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.385 DEBUG:boto:auth_path=/bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.386 DEBUG:boto:Method: GET 2023-10-18T20:19:20.386 DEBUG:boto:Path: /bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.387 DEBUG:boto:Data: 2023-10-18T20:19:20.387 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.388 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:20.388 DEBUG:boto:Port: 8001 2023-10-18T20:19:20.388 DEBUG:boto:Params: {} 2023-10-18T20:19:20.389 DEBUG:boto:Token: None 2023-10-18T20:19:20.389 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/?versions 2023-10-18T20:19:20.390 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3N2IkLaBYzSZT4VCr7D9RZHpSzg= 2023-10-18T20:19:20.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3N2IkLaBYzSZT4VCr7D9RZHpSzg=', 'Content-Length': '0'} 2023-10-18T20:19:20.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d01ebe8664654c58-0065303dc8-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.394 DEBUG:boto:b'bofrnf-50999falseMULTIPARTnulltrue2023-10-18T20:18:44.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:19:20.395 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-10-18T20:19:20.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:19:20.395 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:20.396 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:20.396 DEBUG:boto:path=/bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.397 DEBUG:boto:auth_path=/bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.397 DEBUG:boto:Method: GET 2023-10-18T20:19:20.398 DEBUG:boto:Path: /bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.398 DEBUG:boto:Data: 2023-10-18T20:19:20.399 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:20.399 DEBUG:boto:Port: 8000 2023-10-18T20:19:20.400 DEBUG:boto:Params: {} 2023-10-18T20:19:20.400 DEBUG:boto:Token: None 2023-10-18T20:19:20.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/?versions 2023-10-18T20:19:20.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3N2IkLaBYzSZT4VCr7D9RZHpSzg= 2023-10-18T20:19:20.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3N2IkLaBYzSZT4VCr7D9RZHpSzg=', 'Content-Length': '0'} 2023-10-18T20:19:20.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f5cca6d0c6afb29-0065303dc8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.404 DEBUG:boto:b'bofrnf-50999falseMULTIPARTnulltrue2023-10-18T20:18:44.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:19:20.405 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-10-18T20:19:20.405 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:20.406 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:20.406 DEBUG:boto:path=/bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.407 DEBUG:boto:auth_path=/bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.407 DEBUG:boto:Method: GET 2023-10-18T20:19:20.407 DEBUG:boto:Path: /bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.408 DEBUG:boto:Data: 2023-10-18T20:19:20.408 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:20.409 DEBUG:boto:Port: 8001 2023-10-18T20:19:20.409 DEBUG:boto:Params: {} 2023-10-18T20:19:20.410 DEBUG:boto:Token: None 2023-10-18T20:19:20.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/?versions 2023-10-18T20:19:20.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3N2IkLaBYzSZT4VCr7D9RZHpSzg= 2023-10-18T20:19:20.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3N2IkLaBYzSZT4VCr7D9RZHpSzg=', 'Content-Length': '0'} 2023-10-18T20:19:20.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ac4c840af5cebf6-0065303dc8-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.414 DEBUG:boto:b'bofrnf-50999falseMULTIPARTnulltrue2023-10-18T20:18:44.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:19:20.415 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:20.415 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:20.415 DEBUG:boto:path=/bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.416 DEBUG:boto:auth_path=/bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.416 DEBUG:boto:Method: GET 2023-10-18T20:19:20.417 DEBUG:boto:Path: /bofrnf-50/?versions&max-keys=999 2023-10-18T20:19:20.417 DEBUG:boto:Data: 2023-10-18T20:19:20.417 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:20.418 DEBUG:boto:Port: 8000 2023-10-18T20:19:20.419 DEBUG:boto:Params: {} 2023-10-18T20:19:20.419 DEBUG:boto:Token: None 2023-10-18T20:19:20.419 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/?versions 2023-10-18T20:19:20.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3N2IkLaBYzSZT4VCr7D9RZHpSzg= 2023-10-18T20:19:20.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3N2IkLaBYzSZT4VCr7D9RZHpSzg=', 'Content-Length': '0'} 2023-10-18T20:19:20.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c195b2a50fb369c0-0065303dc8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.423 DEBUG:boto:b'bofrnf-50999falseMULTIPARTnulltrue2023-10-18T20:18:44.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:19:20.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-18T20:19:20.424 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.425 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.425 DEBUG:boto:path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.426 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.426 DEBUG:boto:Method: GET 2023-10-18T20:19:20.427 DEBUG:boto:Path: /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.427 DEBUG:boto:Data: 2023-10-18T20:19:20.428 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.428 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:20.428 DEBUG:boto:Port: 8001 2023-10-18T20:19:20.429 DEBUG:boto:Params: {} 2023-10-18T20:19:20.429 DEBUG:boto:Token: None 2023-10-18T20:19:20.430 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.430 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ykAJ9RHW8qWOwp+pYY4xPou2iOE= 2023-10-18T20:19:20.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ykAJ9RHW8qWOwp+pYY4xPou2iOE=', 'Content-Length': '0'} 2023-10-18T20:19:20.436 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-50:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.19'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000b89fd09625f21992-0065303dc8-1668-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.512 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.513 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.514 DEBUG:boto:path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.514 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.515 DEBUG:boto:Method: GET 2023-10-18T20:19:20.515 DEBUG:boto:Path: /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.515 DEBUG:boto:Data: 2023-10-18T20:19:20.516 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:20.517 DEBUG:boto:Port: 8000 2023-10-18T20:19:20.517 DEBUG:boto:Params: {} 2023-10-18T20:19:20.517 DEBUG:boto:Token: None 2023-10-18T20:19:20.518 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.519 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ykAJ9RHW8qWOwp+pYY4xPou2iOE= 2023-10-18T20:19:20.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ykAJ9RHW8qWOwp+pYY4xPou2iOE=', 'Content-Length': '0'} 2023-10-18T20:19:20.524 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000586c40f2f4888089-0065303dc8-1664-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.613 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.613 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.614 DEBUG:boto:path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.614 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.615 DEBUG:boto:Method: HEAD 2023-10-18T20:19:20.615 DEBUG:boto:Path: /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.615 DEBUG:boto:Data: 2023-10-18T20:19:20.616 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.616 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:20.617 DEBUG:boto:Port: 8001 2023-10-18T20:19:20.617 DEBUG:boto:Params: {} 2023-10-18T20:19:20.618 DEBUG:boto:Token: None 2023-10-18T20:19:20.618 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.619 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2kU4Laugc/H0kCPg7ZuRZts9o7Q= 2023-10-18T20:19:20.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2kU4Laugc/H0kCPg7ZuRZts9o7Q=', 'Content-Length': '0'} 2023-10-18T20:19:20.621 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-50:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.19'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000018eef246fb0cb8bf-0065303dc8-1668-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.622 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.622 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.623 DEBUG:boto:path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.623 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.624 DEBUG:boto:Method: HEAD 2023-10-18T20:19:20.624 DEBUG:boto:Path: /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.624 DEBUG:boto:Data: 2023-10-18T20:19:20.625 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:20.625 DEBUG:boto:Port: 8000 2023-10-18T20:19:20.626 DEBUG:boto:Params: {} 2023-10-18T20:19:20.626 DEBUG:boto:Token: None 2023-10-18T20:19:20.627 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.627 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2kU4Laugc/H0kCPg7ZuRZts9o7Q= 2023-10-18T20:19:20.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2kU4Laugc/H0kCPg7ZuRZts9o7Q=', 'Content-Length': '0'} 2023-10-18T20:19:20.630 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000009ee35731c35838f9-0065303dc8-1664-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-18T20:19:20.631 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.631 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.631 DEBUG:boto:path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.632 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.632 DEBUG:boto:Method: GET 2023-10-18T20:19:20.633 DEBUG:boto:Path: /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.633 DEBUG:boto:Data: 2023-10-18T20:19:20.634 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.634 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:20.634 DEBUG:boto:Port: 8001 2023-10-18T20:19:20.635 DEBUG:boto:Params: {} 2023-10-18T20:19:20.635 DEBUG:boto:Token: None 2023-10-18T20:19:20.636 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.636 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ykAJ9RHW8qWOwp+pYY4xPou2iOE= 2023-10-18T20:19:20.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ykAJ9RHW8qWOwp+pYY4xPou2iOE=', 'Content-Length': '0'} 2023-10-18T20:19:20.640 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-50:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.19'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000091a62e5223bfbaa4-0065303dc8-1668-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.719 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.720 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.720 DEBUG:boto:path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.721 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.721 DEBUG:boto:Method: GET 2023-10-18T20:19:20.721 DEBUG:boto:Path: /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.722 DEBUG:boto:Data: 2023-10-18T20:19:20.722 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:20.723 DEBUG:boto:Port: 8000 2023-10-18T20:19:20.724 DEBUG:boto:Params: {} 2023-10-18T20:19:20.724 DEBUG:boto:Token: None 2023-10-18T20:19:20.725 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART?versionId=null 2023-10-18T20:19:20.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ykAJ9RHW8qWOwp+pYY4xPou2iOE= 2023-10-18T20:19:20.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ykAJ9RHW8qWOwp+pYY4xPou2iOE=', 'Content-Length': '0'} 2023-10-18T20:19:20.730 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000fef2872a4d49a96e-0065303dc8-1664-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.816 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:20.817 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:20.817 DEBUG:boto:path=/bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:20.817 DEBUG:boto:auth_path=/bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:20.818 DEBUG:boto:Method: GET 2023-10-18T20:19:20.818 DEBUG:boto:Path: /bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:20.819 DEBUG:boto:Data: 2023-10-18T20:19:20.819 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.820 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:20.820 DEBUG:boto:Port: 8001 2023-10-18T20:19:20.820 DEBUG:boto:Params: {} 2023-10-18T20:19:20.821 DEBUG:boto:Token: None 2023-10-18T20:19:20.821 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/ 2023-10-18T20:19:20.822 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FymoyNLHQVNxRWbkaFprUk5LGoA= 2023-10-18T20:19:20.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FymoyNLHQVNxRWbkaFprUk5LGoA=', 'Content-Length': '0'} 2023-10-18T20:19:20.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c160e86a7cf50521-0065303dc8-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.826 DEBUG:boto:b'bofrnf-50MULTIPART1000falseMULTIPART2023-10-18T20:18:44.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:19:20.826 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:20.827 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:20.827 DEBUG:boto:path=/bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:20.828 DEBUG:boto:auth_path=/bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:20.828 DEBUG:boto:Method: GET 2023-10-18T20:19:20.828 DEBUG:boto:Path: /bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:20.829 DEBUG:boto:Data: 2023-10-18T20:19:20.829 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:20.830 DEBUG:boto:Port: 8000 2023-10-18T20:19:20.831 DEBUG:boto:Params: {} 2023-10-18T20:19:20.831 DEBUG:boto:Token: None 2023-10-18T20:19:20.831 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/ 2023-10-18T20:19:20.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FymoyNLHQVNxRWbkaFprUk5LGoA= 2023-10-18T20:19:20.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FymoyNLHQVNxRWbkaFprUk5LGoA=', 'Content-Length': '0'} 2023-10-18T20:19:20.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc76742457c57ec-0065303dc8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.835 DEBUG:boto:b'bofrnf-50MULTIPART1000falseMULTIPART2023-10-18T20:18:44.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:19:20.836 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.836 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.837 DEBUG:boto:Method: HEAD 2023-10-18T20:19:20.837 DEBUG:boto:Path: /bofrnf-50/MULTIPART 2023-10-18T20:19:20.837 DEBUG:boto:Data: 2023-10-18T20:19:20.838 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:20.839 DEBUG:boto:Port: 8001 2023-10-18T20:19:20.839 DEBUG:boto:Params: {} 2023-10-18T20:19:20.839 DEBUG:boto:Token: None 2023-10-18T20:19:20.840 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART 2023-10-18T20:19:20.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3JDpQal01UEeoO5aQYLTXo0JicI= 2023-10-18T20:19:20.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3JDpQal01UEeoO5aQYLTXo0JicI=', 'Content-Length': '0'} 2023-10-18T20:19:20.842 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-50:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.19'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000007d5ba27e20a57ce5-0065303dc8-1668-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.843 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.844 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.844 DEBUG:boto:Method: HEAD 2023-10-18T20:19:20.844 DEBUG:boto:Path: /bofrnf-50/MULTIPART 2023-10-18T20:19:20.845 DEBUG:boto:Data: 2023-10-18T20:19:20.845 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.845 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:20.846 DEBUG:boto:Port: 8000 2023-10-18T20:19:20.846 DEBUG:boto:Params: {} 2023-10-18T20:19:20.847 DEBUG:boto:Token: None 2023-10-18T20:19:20.847 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART 2023-10-18T20:19:20.847 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3JDpQal01UEeoO5aQYLTXo0JicI= 2023-10-18T20:19:20.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3JDpQal01UEeoO5aQYLTXo0JicI=', 'Content-Length': '0'} 2023-10-18T20:19:20.850 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000886c2570466d79d9-0065303dc8-1664-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-18T20:19:20.851 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.851 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.851 DEBUG:boto:Method: GET 2023-10-18T20:19:20.852 DEBUG:boto:Path: /bofrnf-50/MULTIPART 2023-10-18T20:19:20.852 DEBUG:boto:Data: 2023-10-18T20:19:20.853 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.853 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:20.853 DEBUG:boto:Port: 8001 2023-10-18T20:19:20.854 DEBUG:boto:Params: {} 2023-10-18T20:19:20.854 DEBUG:boto:Token: None 2023-10-18T20:19:20.855 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART 2023-10-18T20:19:20.855 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hvZMKLkdUHuxDfvHusWuIORdeb0= 2023-10-18T20:19:20.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hvZMKLkdUHuxDfvHusWuIORdeb0=', 'Content-Length': '0'} 2023-10-18T20:19:20.860 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-50:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.19'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000797d6af64bfed39d-0065303dc8-1668-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:20.952 DEBUG:boto:path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.953 DEBUG:boto:auth_path=/bofrnf-50/MULTIPART 2023-10-18T20:19:20.953 DEBUG:boto:Method: GET 2023-10-18T20:19:20.953 DEBUG:boto:Path: /bofrnf-50/MULTIPART 2023-10-18T20:19:20.954 DEBUG:boto:Data: 2023-10-18T20:19:20.954 DEBUG:boto:Headers: {} 2023-10-18T20:19:20.955 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:20.955 DEBUG:boto:Port: 8000 2023-10-18T20:19:20.955 DEBUG:boto:Params: {} 2023-10-18T20:19:20.956 DEBUG:boto:Token: None 2023-10-18T20:19:20.956 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:20 GMT /bofrnf-50/MULTIPART 2023-10-18T20:19:20.957 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hvZMKLkdUHuxDfvHusWuIORdeb0= 2023-10-18T20:19:20.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hvZMKLkdUHuxDfvHusWuIORdeb0=', 'Content-Length': '0'} 2023-10-18T20:19:20.962 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:18:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e73dfb508f889619-0065303dc8-1664-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:21.050 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:21.050 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:21.051 DEBUG:boto:path=/bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:21.051 DEBUG:boto:auth_path=/bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:21.052 DEBUG:boto:Method: GET 2023-10-18T20:19:21.052 DEBUG:boto:Path: /bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:21.053 DEBUG:boto:Data: 2023-10-18T20:19:21.053 DEBUG:boto:Headers: {} 2023-10-18T20:19:21.053 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:21.054 DEBUG:boto:Port: 8001 2023-10-18T20:19:21.054 DEBUG:boto:Params: {} 2023-10-18T20:19:21.055 DEBUG:boto:Token: None 2023-10-18T20:19:21.055 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:21 GMT /bofrnf-50/ 2023-10-18T20:19:21.056 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EUAGKMjCmusI4ctWLv+QlveyCM0= 2023-10-18T20:19:21.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EUAGKMjCmusI4ctWLv+QlveyCM0=', 'Content-Length': '0'} 2023-10-18T20:19:21.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f740cc72a15d8059-0065303dc9-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:21.059 DEBUG:boto:b'bofrnf-50MULTIPART1000falseMULTIPART2023-10-18T20:18:44.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:19:21.060 DEBUG:boto:path=/bofrnf-50/ 2023-10-18T20:19:21.060 DEBUG:boto:auth_path=/bofrnf-50/ 2023-10-18T20:19:21.061 DEBUG:boto:path=/bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:21.061 DEBUG:boto:auth_path=/bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:21.062 DEBUG:boto:Method: GET 2023-10-18T20:19:21.062 DEBUG:boto:Path: /bofrnf-50/?prefix=MULTIPART 2023-10-18T20:19:21.062 DEBUG:boto:Data: 2023-10-18T20:19:21.063 DEBUG:boto:Headers: {} 2023-10-18T20:19:21.063 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:21.064 DEBUG:boto:Port: 8000 2023-10-18T20:19:21.064 DEBUG:boto:Params: {} 2023-10-18T20:19:21.064 DEBUG:boto:Token: None 2023-10-18T20:19:21.065 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:21 GMT /bofrnf-50/ 2023-10-18T20:19:21.065 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EUAGKMjCmusI4ctWLv+QlveyCM0= 2023-10-18T20:19:21.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EUAGKMjCmusI4ctWLv+QlveyCM0=', 'Content-Length': '0'} 2023-10-18T20:19:21.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073ff1f0e94a524ea-0065303dc9-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:21.069 DEBUG:boto:b'bofrnf-50MULTIPART1000falseMULTIPART2023-10-18T20:18:44.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:19:21.070 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-50 zones={a2, a1} 2023-10-18T20:19:21.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-10-18T20:19:21.071 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-52 2023-10-18T20:19:21.071 DEBUG:boto:path=/bofrnf-52/ 2023-10-18T20:19:21.072 DEBUG:boto:auth_path=/bofrnf-52/ 2023-10-18T20:19:21.072 DEBUG:boto:Method: PUT 2023-10-18T20:19:21.072 DEBUG:boto:Path: /bofrnf-52/ 2023-10-18T20:19:21.073 DEBUG:boto:Data: 2023-10-18T20:19:21.073 DEBUG:boto:Headers: {} 2023-10-18T20:19:21.074 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:21.074 DEBUG:boto:Port: 8000 2023-10-18T20:19:21.074 DEBUG:boto:Params: {} 2023-10-18T20:19:21.075 DEBUG:boto:Token: None 2023-10-18T20:19:21.075 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:19:21 GMT /bofrnf-52/ 2023-10-18T20:19:21.076 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/zaVvZdL3Mash4k2ihz049dje1w= 2023-10-18T20:19:21.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/zaVvZdL3Mash4k2ihz049dje1w=', 'Content-Length': '0'} 2023-10-18T20:19:21.092 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e18d6e6e928c17b-0065303dc9-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:21.092 DEBUG:boto:path=/bofrnf-52/testobj-sse-c 2023-10-18T20:19:21.093 DEBUG:boto:auth_path=/bofrnf-52/testobj-sse-c 2023-10-18T20:19:21.093 DEBUG:boto:Method: PUT 2023-10-18T20:19:21.093 DEBUG:boto:Path: /bofrnf-52/testobj-sse-c 2023-10-18T20:19:21.094 DEBUG:boto:Data: 2023-10-18T20:19:21.094 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-18T20:19:21.095 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:21.095 DEBUG:boto:Port: 8000 2023-10-18T20:19:21.095 DEBUG:boto:Params: {} 2023-10-18T20:19:21.096 DEBUG:boto:Token: None 2023-10-18T20:19:21.096 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 18 Oct 2023 20:19:21 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /bofrnf-52/testobj-sse-c 2023-10-18T20:19:21.097 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ki/sVofPXZHfKoFq32qn21mVulw= 2023-10-18T20:19:21.097 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:19:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ki/sVofPXZHfKoFq32qn21mVulw='} 2023-10-18T20:19:21.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx000001c7da08b857829b2-0065303dc9-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:21.105 DEBUG:boto:path=/bofrnf-52/testobj-sse-kms 2023-10-18T20:19:21.105 DEBUG:boto:auth_path=/bofrnf-52/testobj-sse-kms 2023-10-18T20:19:21.106 DEBUG:boto:Method: PUT 2023-10-18T20:19:21.106 DEBUG:boto:Path: /bofrnf-52/testobj-sse-kms 2023-10-18T20:19:21.107 DEBUG:boto:Data: 2023-10-18T20:19:21.107 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-18T20:19:21.108 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:21.109 DEBUG:boto:Port: 8000 2023-10-18T20:19:21.109 DEBUG:boto:Params: {} 2023-10-18T20:19:21.110 DEBUG:boto:Token: None 2023-10-18T20:19:21.110 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 18 Oct 2023 20:19:21 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /bofrnf-52/testobj-sse-kms 2023-10-18T20:19:21.111 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+fxTz79ba0wb0hZ2Nw/j07ZDonQ= 2023-10-18T20:19:21.112 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:19:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+fxTz79ba0wb0hZ2Nw/j07ZDonQ='} 2023-10-18T20:19:21.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6Ym9mcm5mLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000efc6983cef5300b7-0065303dc9-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:21.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:19:21.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:19:21.156 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:19:21.208 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:19:21.209 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:19:21.240 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:19:21.253 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:19:21.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:21.253 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660361.089744_849.1", 2023-10-18T20:19:21.254 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:19:21.089744Z" 2023-10-18T20:19:21.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:21.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:21.254 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:19:21.255 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:19:21.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:21.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:21.255 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:19:21.255 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:19:21.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:21.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:21.256 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:19:21.256 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:19:21.256 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:21.256 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:19:21.257 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660361.089744_849.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:19:21.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:19:21.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:19:21.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:19:21.259 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:19:21.309 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:19:21.309 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:19:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:21.335+0000 7fc178ada780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:19:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:21.336+0000 7fc178ada780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:19:21.350 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:19:21.351 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:19:21.351 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:19:21.351 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:19:21.351 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:19:21.351 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:19:21.352 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:21.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:21.352 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:19:21.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:21.352 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:19:21.353 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:19:21.353 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:19:21.353 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660361.089744_849.1", 2023-10-18T20:19:21.353 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:19:21.353 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:19:21.354 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:19:21.354 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:19:21.089744Z", 2023-10-18T20:19:21.354 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:21.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:21.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:21.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:21.355 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:19:21.355 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:19:21.355 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:19:21.355 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:19:21.356 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:19:21.356 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:19:21.356 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:19:21.356 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:19:21.356 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:21.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:21.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:21.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:21.357 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:19:21.357 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:19:21.358 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:19:21.358 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:19:21.358 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:19:21.358 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:19:21.358 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:19:21.358 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:19:21.359 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:21.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:21.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:21.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:21.359 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:19:21.360 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:19:21.360 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:19:21.360 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:19:21.360 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:19:21.360 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:19:21.361 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:19:21.361 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:19:21.361 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:21.361 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:21.361 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:21.362 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:19:21.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:21.362 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:19:21.362 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:19:21.362 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:19:21.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:21.363 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:19:21Z" 2023-10-18T20:19:21.363 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:19:21.364 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660361.089744_849.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:19:21.089744Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:19:21Z" } 2023-10-18T20:19:21.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660361.089744_849.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:19:21.089744Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:19:21.365 DEBUG:rgw_multi.tests:log_status={0: '1_1697660361.089744_849.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:19:21.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660361.089744_849.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:19:21.365 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:19:21.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:19:21.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:19:21.367 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:19:21.423 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:19:21.423 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:19:21.456 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:21.454+0000 7fc035a0c780 1 waiting to reach incremental sync.. 2023-10-18T20:19:26.457 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:26.455+0000 7fc035a0c780 1 waiting to reach incremental sync.. 2023-10-18T20:19:31.457 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:31.456+0000 7fc035a0c780 1 waiting to reach incremental sync.. 2023-10-18T20:19:36.458 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:36.456+0000 7fc035a0c780 1 waiting to reach incremental sync.. 2023-10-18T20:19:41.460 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:41.458+0000 7fc035a0c780 1 bucket sync caught up with source: 2023-10-18T20:19:41.460 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.17.6, , , , 00000000001.36.6, , , , , ] 2023-10-18T20:19:41.460 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.17.6, , , , 00000000001.36.6, , , , , ] 2023-10-18T20:19:41.460 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:41.458+0000 7fc035a0c780 0 bucket checkpoint complete 2023-10-18T20:19:41.477 DEBUG:boto:path=/bofrnf-52/ 2023-10-18T20:19:41.477 DEBUG:boto:auth_path=/bofrnf-52/ 2023-10-18T20:19:41.478 DEBUG:boto:Method: HEAD 2023-10-18T20:19:41.478 DEBUG:boto:Path: /bofrnf-52/ 2023-10-18T20:19:41.478 DEBUG:boto:Data: 2023-10-18T20:19:41.479 DEBUG:boto:Headers: {} 2023-10-18T20:19:41.479 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:41.480 DEBUG:boto:Port: 8001 2023-10-18T20:19:41.480 DEBUG:boto:Params: {} 2023-10-18T20:19:41.480 DEBUG:boto:Token: None 2023-10-18T20:19:41.481 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-52/ 2023-10-18T20:19:41.481 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mgqdXmvcjhF/80L6FMu51QUL8pM= 2023-10-18T20:19:41.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mgqdXmvcjhF/80L6FMu51QUL8pM=', 'Content-Length': '0'} 2023-10-18T20:19:41.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080c9071b18c12e72-0065303ddd-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.485 DEBUG:boto:path=/bofrnf-52/testobj-sse-c 2023-10-18T20:19:41.485 DEBUG:boto:auth_path=/bofrnf-52/testobj-sse-c 2023-10-18T20:19:41.486 DEBUG:boto:Method: HEAD 2023-10-18T20:19:41.486 DEBUG:boto:Path: /bofrnf-52/testobj-sse-c 2023-10-18T20:19:41.486 DEBUG:boto:Data: 2023-10-18T20:19:41.487 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-10-18T20:19:41.487 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:41.487 DEBUG:boto:Port: 8001 2023-10-18T20:19:41.488 DEBUG:boto:Params: {} 2023-10-18T20:19:41.488 DEBUG:boto:Token: None 2023-10-18T20:19:41.489 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:41 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /bofrnf-52/testobj-sse-c 2023-10-18T20:19:41.489 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rfsQzFUgMGR+ieknDPhnPHCzDzc= 2023-10-18T20:19:41.490 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rfsQzFUgMGR+ieknDPhnPHCzDzc=', 'Content-Length': '0'} 2023-10-18T20:19:41.492 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-52:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.21'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000003c760f1d63dd3857-0065303ddd-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.492 DEBUG:boto:path=/bofrnf-52/testobj-sse-c 2023-10-18T20:19:41.493 DEBUG:boto:auth_path=/bofrnf-52/testobj-sse-c 2023-10-18T20:19:41.493 DEBUG:boto:Method: GET 2023-10-18T20:19:41.494 DEBUG:boto:Path: /bofrnf-52/testobj-sse-c 2023-10-18T20:19:41.494 DEBUG:boto:Data: 2023-10-18T20:19:41.495 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-10-18T20:19:41.495 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:41.495 DEBUG:boto:Port: 8001 2023-10-18T20:19:41.496 DEBUG:boto:Params: {} 2023-10-18T20:19:41.496 DEBUG:boto:Token: None 2023-10-18T20:19:41.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:41 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /bofrnf-52/testobj-sse-c 2023-10-18T20:19:41.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0IBlKgZ0G8yngLk7dPRjxgcRiUE= 2023-10-18T20:19:41.497 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0IBlKgZ0G8yngLk7dPRjxgcRiUE=', 'Content-Length': '0'} 2023-10-18T20:19:41.500 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-52:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.21'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000c0c8d275fdc28e58-0065303ddd-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.500 DEBUG:boto:path=/bofrnf-52/testobj-sse-kms 2023-10-18T20:19:41.501 DEBUG:boto:auth_path=/bofrnf-52/testobj-sse-kms 2023-10-18T20:19:41.501 DEBUG:boto:Method: HEAD 2023-10-18T20:19:41.502 DEBUG:boto:Path: /bofrnf-52/testobj-sse-kms 2023-10-18T20:19:41.502 DEBUG:boto:Data: 2023-10-18T20:19:41.502 DEBUG:boto:Headers: {} 2023-10-18T20:19:41.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:41.503 DEBUG:boto:Port: 8001 2023-10-18T20:19:41.504 DEBUG:boto:Params: {} 2023-10-18T20:19:41.504 DEBUG:boto:Token: None 2023-10-18T20:19:41.504 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-52/testobj-sse-kms 2023-10-18T20:19:41.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qioi9NOkGsxzbzgVljJxLKXW4Uc= 2023-10-18T20:19:41.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qioi9NOkGsxzbzgVljJxLKXW4Uc=', 'Content-Length': '0'} 2023-10-18T20:19:41.507 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-52:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.21'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000260da98afdb5cd82-0065303ddd-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.508 DEBUG:boto:path=/bofrnf-52/testobj-sse-kms 2023-10-18T20:19:41.508 DEBUG:boto:auth_path=/bofrnf-52/testobj-sse-kms 2023-10-18T20:19:41.508 DEBUG:boto:Method: GET 2023-10-18T20:19:41.509 DEBUG:boto:Path: /bofrnf-52/testobj-sse-kms 2023-10-18T20:19:41.509 DEBUG:boto:Data: 2023-10-18T20:19:41.509 DEBUG:boto:Headers: {} 2023-10-18T20:19:41.510 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:19:41.510 DEBUG:boto:Port: 8001 2023-10-18T20:19:41.511 DEBUG:boto:Params: {} 2023-10-18T20:19:41.511 DEBUG:boto:Token: None 2023-10-18T20:19:41.512 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-52/testobj-sse-kms 2023-10-18T20:19:41.512 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jGldsH2pk9ieEDS059z3UdNBwSE= 2023-10-18T20:19:41.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jGldsH2pk9ieEDS059z3UdNBwSE=', 'Content-Length': '0'} 2023-10-18T20:19:41.515 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-52:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.21'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000ffd558431fd9e691-0065303ddd-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.516 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2023-10-18T20:19:41.516 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-53 2023-10-18T20:19:41.517 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:19:41.517 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:19:41.518 DEBUG:boto:Method: PUT 2023-10-18T20:19:41.518 DEBUG:boto:Path: /bofrnf-53/ 2023-10-18T20:19:41.519 DEBUG:boto:Data: 2023-10-18T20:19:41.519 DEBUG:boto:Headers: {} 2023-10-18T20:19:41.519 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:41.520 DEBUG:boto:Port: 8000 2023-10-18T20:19:41.520 DEBUG:boto:Params: {} 2023-10-18T20:19:41.521 DEBUG:boto:Token: None 2023-10-18T20:19:41.521 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-53/ 2023-10-18T20:19:41.521 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xh/UY/MwyQ2AJ6EvbpLeiDqqvzg= 2023-10-18T20:19:41.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xh/UY/MwyQ2AJ6EvbpLeiDqqvzg=', 'Content-Length': '0'} 2023-10-18T20:19:41.540 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ba4bef3fb563b4c8-0065303ddd-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.540 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:19:41.541 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:19:41.541 DEBUG:boto:Method: HEAD 2023-10-18T20:19:41.541 DEBUG:boto:Path: /bofrnf-53/ 2023-10-18T20:19:41.542 DEBUG:boto:Data: 2023-10-18T20:19:41.542 DEBUG:boto:Headers: {} 2023-10-18T20:19:41.543 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:41.543 DEBUG:boto:Port: 8000 2023-10-18T20:19:41.543 DEBUG:boto:Params: {} 2023-10-18T20:19:41.544 DEBUG:boto:Token: None 2023-10-18T20:19:41.544 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-53/ 2023-10-18T20:19:41.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BGtOd46SufsVxbus8Lvyngqihtw= 2023-10-18T20:19:41.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BGtOd46SufsVxbus8Lvyngqihtw=', 'Content-Length': '0'} 2023-10-18T20:19:41.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ddb42715461a995-0065303ddd-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.548 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:19:41.549 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:19:41.549 DEBUG:boto:Method: PUT 2023-10-18T20:19:41.549 DEBUG:boto:Path: /bofrnf-53/a 2023-10-18T20:19:41.550 DEBUG:boto:Data: 2023-10-18T20:19:41.550 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:19:41.551 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:41.551 DEBUG:boto:Port: 8000 2023-10-18T20:19:41.551 DEBUG:boto:Params: {} 2023-10-18T20:19:41.552 DEBUG:boto:Token: None 2023-10-18T20:19:41.552 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-53/a 2023-10-18T20:19:41.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XMFbzrwXNy1IiW2HWQkSfpIfGmg= 2023-10-18T20:19:41.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XMFbzrwXNy1IiW2HWQkSfpIfGmg='} 2023-10-18T20:19:41.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000f5cf3e101ae48e-0065303ddd-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.560 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:19:41.561 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:19:41.561 DEBUG:boto:Method: HEAD 2023-10-18T20:19:41.561 DEBUG:boto:Path: /bofrnf-53/ 2023-10-18T20:19:41.562 DEBUG:boto:Data: 2023-10-18T20:19:41.562 DEBUG:boto:Headers: {} 2023-10-18T20:19:41.563 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:41.563 DEBUG:boto:Port: 8000 2023-10-18T20:19:41.564 DEBUG:boto:Params: {} 2023-10-18T20:19:41.564 DEBUG:boto:Token: None 2023-10-18T20:19:41.564 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-53/ 2023-10-18T20:19:41.565 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BGtOd46SufsVxbus8Lvyngqihtw= 2023-10-18T20:19:41.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BGtOd46SufsVxbus8Lvyngqihtw=', 'Content-Length': '0'} 2023-10-18T20:19:41.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0e5a86ba7947436-0065303ddd-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.568 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:19:41.569 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:19:41.569 DEBUG:boto:Method: PUT 2023-10-18T20:19:41.569 DEBUG:boto:Path: /bofrnf-53/b 2023-10-18T20:19:41.570 DEBUG:boto:Data: 2023-10-18T20:19:41.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:19:41.571 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:41.571 DEBUG:boto:Port: 8000 2023-10-18T20:19:41.571 DEBUG:boto:Params: {} 2023-10-18T20:19:41.572 DEBUG:boto:Token: None 2023-10-18T20:19:41.572 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-53/b 2023-10-18T20:19:41.573 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QD66FMeNWfYqk2f4myXZWZbNSMA= 2023-10-18T20:19:41.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QD66FMeNWfYqk2f4myXZWZbNSMA='} 2023-10-18T20:19:41.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007776adc8c33dd9e2-0065303ddd-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.619 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:19:41.619 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:19:41.620 DEBUG:boto:Method: HEAD 2023-10-18T20:19:41.620 DEBUG:boto:Path: /bofrnf-53/ 2023-10-18T20:19:41.621 DEBUG:boto:Data: 2023-10-18T20:19:41.621 DEBUG:boto:Headers: {} 2023-10-18T20:19:41.621 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:41.622 DEBUG:boto:Port: 8000 2023-10-18T20:19:41.622 DEBUG:boto:Params: {} 2023-10-18T20:19:41.623 DEBUG:boto:Token: None 2023-10-18T20:19:41.623 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-53/ 2023-10-18T20:19:41.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BGtOd46SufsVxbus8Lvyngqihtw= 2023-10-18T20:19:41.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BGtOd46SufsVxbus8Lvyngqihtw=', 'Content-Length': '0'} 2023-10-18T20:19:41.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f49be2c07a3892c-0065303ddd-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.627 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:19:41.628 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:19:41.628 DEBUG:boto:Method: PUT 2023-10-18T20:19:41.628 DEBUG:boto:Path: /bofrnf-53/c 2023-10-18T20:19:41.629 DEBUG:boto:Data: 2023-10-18T20:19:41.629 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:19:41.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:41.630 DEBUG:boto:Port: 8000 2023-10-18T20:19:41.630 DEBUG:boto:Params: {} 2023-10-18T20:19:41.633 DEBUG:boto:Token: None 2023-10-18T20:19:41.634 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-53/c 2023-10-18T20:19:41.634 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gCqQsc9COgEcM6rPw/KJ2Wyr4GE= 2023-10-18T20:19:41.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gCqQsc9COgEcM6rPw/KJ2Wyr4GE='} 2023-10-18T20:19:41.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055b79bb6bbc3066b-0065303ddd-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.640 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:19:41.641 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:19:41.641 DEBUG:boto:Method: HEAD 2023-10-18T20:19:41.642 DEBUG:boto:Path: /bofrnf-53/ 2023-10-18T20:19:41.642 DEBUG:boto:Data: 2023-10-18T20:19:41.643 DEBUG:boto:Headers: {} 2023-10-18T20:19:41.643 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:41.643 DEBUG:boto:Port: 8000 2023-10-18T20:19:41.644 DEBUG:boto:Params: {} 2023-10-18T20:19:41.644 DEBUG:boto:Token: None 2023-10-18T20:19:41.645 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-53/ 2023-10-18T20:19:41.645 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BGtOd46SufsVxbus8Lvyngqihtw= 2023-10-18T20:19:41.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BGtOd46SufsVxbus8Lvyngqihtw=', 'Content-Length': '0'} 2023-10-18T20:19:41.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaf96a970da7d7ce-0065303ddd-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.649 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:19:41.649 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:19:41.650 DEBUG:boto:Method: PUT 2023-10-18T20:19:41.650 DEBUG:boto:Path: /bofrnf-53/d 2023-10-18T20:19:41.650 DEBUG:boto:Data: 2023-10-18T20:19:41.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:19:41.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:19:41.652 DEBUG:boto:Port: 8000 2023-10-18T20:19:41.652 DEBUG:boto:Params: {} 2023-10-18T20:19:41.652 DEBUG:boto:Token: None 2023-10-18T20:19:41.653 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:19:41 GMT /bofrnf-53/d 2023-10-18T20:19:41.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hkOrdAJt2cSEWQ1Nlj9fqyNOO7I= 2023-10-18T20:19:41.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:19:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hkOrdAJt2cSEWQ1Nlj9fqyNOO7I='} 2023-10-18T20:19:41.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000535ae60d1c0082da-0065303ddd-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:19:41.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:19:41.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:19:41.704 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:19:41.755 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:19:41.755 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:19:41.785 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:19:41.797 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:19:41.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:41.798 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660381.537598_855.1", 2023-10-18T20:19:41.798 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:19:41.537598Z" 2023-10-18T20:19:41.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:41.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:41.798 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:19:41.799 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:19:41.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:41.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:41.799 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:19:41.800 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:19:41.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:41.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:41.800 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:19:41.800 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:19:41.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:41.801 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:19:41.802 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660381.537598_855.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:19:41.802 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:19:41.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:19:41.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:19:41.804 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:19:41.852 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:19:41.853 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:19:41.880 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:41.878+0000 7f7873bbc780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:19:41.881 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:41.880+0000 7f7873bbc780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:19:41.893 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:19:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:19:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:19:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:19:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:19:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:19:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:19:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:19:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:19:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:19:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660381.537598_855.1", 2023-10-18T20:19:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:19:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:19:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:19:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:19:41.537598Z", 2023-10-18T20:19:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:19:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:19:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:19:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:19:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:19:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:19:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:19:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:19:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:19:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:19:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:19:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:19:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:19:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:19:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:19:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:19:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:19:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:19:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:19:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:19:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:19:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:19:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:19:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:19:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:19:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:19:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:19:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:19:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:19:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:19:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:19:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:19:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:19:41Z" 2023-10-18T20:19:41.907 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:19:41.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660381.537598_855.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:19:41.537598Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:19:41Z" } 2023-10-18T20:19:41.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660381.537598_855.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:19:41.537598Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:19:41.909 DEBUG:rgw_multi.tests:log_status={0: '1_1697660381.537598_855.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:19:41.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660381.537598_855.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:19:41.910 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:19:41.910 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-53 2023-10-18T20:19:41.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:19:41.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:19:41.911 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-53 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:19:41.969 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:19:41.969 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:19:42.002 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:42.001+0000 7f216ce25780 1 waiting to reach incremental sync.. 2023-10-18T20:19:47.003 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:47.002+0000 7f216ce25780 1 waiting to reach incremental sync.. 2023-10-18T20:19:52.004 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:52.002+0000 7f216ce25780 1 waiting to reach incremental sync.. 2023-10-18T20:19:57.005 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:19:57.003+0000 7f216ce25780 1 waiting to reach incremental sync.. 2023-10-18T20:20:02.006 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:02.004+0000 7f216ce25780 1 bucket sync caught up with source: 2023-10-18T20:20:02.007 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.34.6, , , , 00000000001.45.6, , , 00000000001.31.6, , 00000000001.29.6] 2023-10-18T20:20:02.007 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.34.6, , , , 00000000001.45.6, , , 00000000001.31.6, , 00000000001.29.6] 2023-10-18T20:20:02.007 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:02.004+0000 7f216ce25780 0 bucket checkpoint complete 2023-10-18T20:20:02.023 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-53 2023-10-18T20:20:02.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:20:02.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:20:02.024 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-53 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:20:02.088 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:20:02.088 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:20:02.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:02.124+0000 7fdf0eaf2780 1 waiting to reach incremental sync.. 2023-10-18T20:20:07.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:07.124+0000 7fdf0eaf2780 1 waiting to reach incremental sync.. 2023-10-18T20:20:12.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:12.125+0000 7fdf0eaf2780 1 waiting to reach incremental sync.. 2023-10-18T20:20:17.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:17.126+0000 7fdf0eaf2780 1 waiting to reach incremental sync.. 2023-10-18T20:20:22.129 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:22.128+0000 7fdf0eaf2780 1 bucket sync caught up with source: 2023-10-18T20:20:22.130 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.299.6, , , , 00000000001.826.6, , , 00000000001.689.6, , 00000000001.549.6] 2023-10-18T20:20:22.130 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.299.6, , , , 00000000001.826.6, , , 00000000001.689.6, , 00000000001.549.6] 2023-10-18T20:20:22.130 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:22.128+0000 7fdf0eaf2780 0 bucket checkpoint complete 2023-10-18T20:20:22.146 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-53 zones={a2, a1} 2023-10-18T20:20:22.147 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.148 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.148 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.149 DEBUG:boto:Path: /bofrnf-53/ 2023-10-18T20:20:22.149 DEBUG:boto:Data: 2023-10-18T20:20:22.149 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.150 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.151 DEBUG:boto:Params: {} 2023-10-18T20:20:22.151 DEBUG:boto:Token: None 2023-10-18T20:20:22.152 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oQv4R5nhWt1GYw8E6k6/IkhUtOc= 2023-10-18T20:20:22.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oQv4R5nhWt1GYw8E6k6/IkhUtOc=', 'Content-Length': '0'} 2023-10-18T20:20:22.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7802ade60ffad5a-0065303e06-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.156 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.157 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.157 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.157 DEBUG:boto:Path: /bofrnf-53/ 2023-10-18T20:20:22.158 DEBUG:boto:Data: 2023-10-18T20:20:22.158 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.159 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.159 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.160 DEBUG:boto:Params: {} 2023-10-18T20:20:22.160 DEBUG:boto:Token: None 2023-10-18T20:20:22.160 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oQv4R5nhWt1GYw8E6k6/IkhUtOc= 2023-10-18T20:20:22.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oQv4R5nhWt1GYw8E6k6/IkhUtOc=', 'Content-Length': '0'} 2023-10-18T20:20:22.164 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090213bd82b3d33d1-0065303e06-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.165 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:20:22.165 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.166 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.166 DEBUG:boto:path=/bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.166 DEBUG:boto:auth_path=/bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.167 DEBUG:boto:Method: GET 2023-10-18T20:20:22.167 DEBUG:boto:Path: /bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.168 DEBUG:boto:Data: 2023-10-18T20:20:22.168 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.168 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.169 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.169 DEBUG:boto:Params: {} 2023-10-18T20:20:22.170 DEBUG:boto:Token: None 2023-10-18T20:20:22.170 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/?versions 2023-10-18T20:20:22.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WsTS+2lvxO9sBGc+su8wRfIWzBg= 2023-10-18T20:20:22.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WsTS+2lvxO9sBGc+su8wRfIWzBg=', 'Content-Length': '0'} 2023-10-18T20:20:22.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004672a7b008f4d7bb-0065303e06-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.174 DEBUG:boto:b'bofrnf-53999falseanulltrue2023-10-18T20:19:41.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:19:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:19:41.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:19:41.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.175 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:20:22.176 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:20:22.176 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:20:22.176 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:20:22.177 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:20:22.177 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.178 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.178 DEBUG:boto:path=/bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.179 DEBUG:boto:auth_path=/bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.179 DEBUG:boto:Method: GET 2023-10-18T20:20:22.179 DEBUG:boto:Path: /bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.180 DEBUG:boto:Data: 2023-10-18T20:20:22.180 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.181 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.181 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.181 DEBUG:boto:Params: {} 2023-10-18T20:20:22.182 DEBUG:boto:Token: None 2023-10-18T20:20:22.182 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/?versions 2023-10-18T20:20:22.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WsTS+2lvxO9sBGc+su8wRfIWzBg= 2023-10-18T20:20:22.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WsTS+2lvxO9sBGc+su8wRfIWzBg=', 'Content-Length': '0'} 2023-10-18T20:20:22.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000049c6d716138bc0a-0065303e06-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.186 DEBUG:boto:b'bofrnf-53999falseanulltrue2023-10-18T20:19:41.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:19:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:19:41.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:19:41.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.187 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:20:22.188 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:20:22.188 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:20:22.188 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:20:22.189 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.189 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.190 DEBUG:boto:path=/bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.190 DEBUG:boto:auth_path=/bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.190 DEBUG:boto:Method: GET 2023-10-18T20:20:22.191 DEBUG:boto:Path: /bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.191 DEBUG:boto:Data: 2023-10-18T20:20:22.192 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.192 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.193 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.193 DEBUG:boto:Params: {} 2023-10-18T20:20:22.193 DEBUG:boto:Token: None 2023-10-18T20:20:22.194 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/?versions 2023-10-18T20:20:22.194 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WsTS+2lvxO9sBGc+su8wRfIWzBg= 2023-10-18T20:20:22.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WsTS+2lvxO9sBGc+su8wRfIWzBg=', 'Content-Length': '0'} 2023-10-18T20:20:22.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083a209c1f7d76aa4-0065303e06-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.198 DEBUG:boto:b'bofrnf-53999falseanulltrue2023-10-18T20:19:41.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:19:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:19:41.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:19:41.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.199 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.199 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.200 DEBUG:boto:path=/bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.200 DEBUG:boto:auth_path=/bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.201 DEBUG:boto:Method: GET 2023-10-18T20:20:22.201 DEBUG:boto:Path: /bofrnf-53/?versions&max-keys=999 2023-10-18T20:20:22.201 DEBUG:boto:Data: 2023-10-18T20:20:22.202 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.202 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.203 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.203 DEBUG:boto:Params: {} 2023-10-18T20:20:22.203 DEBUG:boto:Token: None 2023-10-18T20:20:22.204 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/?versions 2023-10-18T20:20:22.204 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WsTS+2lvxO9sBGc+su8wRfIWzBg= 2023-10-18T20:20:22.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WsTS+2lvxO9sBGc+su8wRfIWzBg=', 'Content-Length': '0'} 2023-10-18T20:20:22.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053b5c2e2b545db14-0065303e06-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.208 DEBUG:boto:b'bofrnf-53999falseanulltrue2023-10-18T20:19:41.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:19:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:19:41.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:19:41.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:20:22.209 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.210 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.210 DEBUG:boto:path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.210 DEBUG:boto:auth_path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.211 DEBUG:boto:Method: GET 2023-10-18T20:20:22.211 DEBUG:boto:Path: /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.212 DEBUG:boto:Data: 2023-10-18T20:20:22.212 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.213 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.213 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.213 DEBUG:boto:Params: {} 2023-10-18T20:20:22.214 DEBUG:boto:Token: None 2023-10-18T20:20:22.214 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qYKlK7+tvcXS+CCOheS+uBX4Qwc= 2023-10-18T20:20:22.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qYKlK7+tvcXS+CCOheS+uBX4Qwc=', 'Content-Length': '0'} 2023-10-18T20:20:22.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009debddd6d8b592ad-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.258 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.259 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.259 DEBUG:boto:path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.259 DEBUG:boto:auth_path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.260 DEBUG:boto:Method: GET 2023-10-18T20:20:22.260 DEBUG:boto:Path: /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.261 DEBUG:boto:Data: 2023-10-18T20:20:22.261 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.262 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.262 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.262 DEBUG:boto:Params: {} 2023-10-18T20:20:22.263 DEBUG:boto:Token: None 2023-10-18T20:20:22.263 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.264 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qYKlK7+tvcXS+CCOheS+uBX4Qwc= 2023-10-18T20:20:22.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qYKlK7+tvcXS+CCOheS+uBX4Qwc=', 'Content-Length': '0'} 2023-10-18T20:20:22.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b2e8f53aeebc8a-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.267 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.267 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.268 DEBUG:boto:path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.268 DEBUG:boto:auth_path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.268 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.269 DEBUG:boto:Path: /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.269 DEBUG:boto:Data: 2023-10-18T20:20:22.269 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.270 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.270 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.271 DEBUG:boto:Params: {} 2023-10-18T20:20:22.271 DEBUG:boto:Token: None 2023-10-18T20:20:22.273 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.273 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AWDfS77c8QWQ7kStodlQF4Ujcso= 2023-10-18T20:20:22.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AWDfS77c8QWQ7kStodlQF4Ujcso=', 'Content-Length': '0'} 2023-10-18T20:20:22.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e90320fdf4cd4491-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.276 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.276 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.277 DEBUG:boto:path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.277 DEBUG:boto:auth_path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.278 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.278 DEBUG:boto:Path: /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.278 DEBUG:boto:Data: 2023-10-18T20:20:22.279 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.280 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.280 DEBUG:boto:Params: {} 2023-10-18T20:20:22.280 DEBUG:boto:Token: None 2023-10-18T20:20:22.281 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.281 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AWDfS77c8QWQ7kStodlQF4Ujcso= 2023-10-18T20:20:22.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AWDfS77c8QWQ7kStodlQF4Ujcso=', 'Content-Length': '0'} 2023-10-18T20:20:22.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f57f8a393aa2d35-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:20:22.285 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.285 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.286 DEBUG:boto:path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.286 DEBUG:boto:auth_path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.286 DEBUG:boto:Method: GET 2023-10-18T20:20:22.287 DEBUG:boto:Path: /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.287 DEBUG:boto:Data: 2023-10-18T20:20:22.288 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.289 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.289 DEBUG:boto:Params: {} 2023-10-18T20:20:22.290 DEBUG:boto:Token: None 2023-10-18T20:20:22.290 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qYKlK7+tvcXS+CCOheS+uBX4Qwc= 2023-10-18T20:20:22.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qYKlK7+tvcXS+CCOheS+uBX4Qwc=', 'Content-Length': '0'} 2023-10-18T20:20:22.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29bb3dcce0fdfdf-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.294 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.295 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.295 DEBUG:boto:path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.295 DEBUG:boto:auth_path=/bofrnf-53/a?versionId=null 2023-10-18T20:20:22.296 DEBUG:boto:Method: GET 2023-10-18T20:20:22.296 DEBUG:boto:Path: /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.297 DEBUG:boto:Data: 2023-10-18T20:20:22.297 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.298 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.298 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.299 DEBUG:boto:Params: {} 2023-10-18T20:20:22.299 DEBUG:boto:Token: None 2023-10-18T20:20:22.300 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a?versionId=null 2023-10-18T20:20:22.300 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qYKlK7+tvcXS+CCOheS+uBX4Qwc= 2023-10-18T20:20:22.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qYKlK7+tvcXS+CCOheS+uBX4Qwc=', 'Content-Length': '0'} 2023-10-18T20:20:22.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000546f027136582683-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.303 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.304 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.304 DEBUG:boto:path=/bofrnf-53/?prefix=a 2023-10-18T20:20:22.304 DEBUG:boto:auth_path=/bofrnf-53/?prefix=a 2023-10-18T20:20:22.305 DEBUG:boto:Method: GET 2023-10-18T20:20:22.305 DEBUG:boto:Path: /bofrnf-53/?prefix=a 2023-10-18T20:20:22.306 DEBUG:boto:Data: 2023-10-18T20:20:22.306 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.307 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.307 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.308 DEBUG:boto:Params: {} 2023-10-18T20:20:22.308 DEBUG:boto:Token: None 2023-10-18T20:20:22.309 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.309 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec734c295951052b-0065303e06-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.313 DEBUG:boto:b'bofrnf-53a1000falsea2023-10-18T20:19:41.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.313 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.314 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.314 DEBUG:boto:path=/bofrnf-53/?prefix=a 2023-10-18T20:20:22.315 DEBUG:boto:auth_path=/bofrnf-53/?prefix=a 2023-10-18T20:20:22.315 DEBUG:boto:Method: GET 2023-10-18T20:20:22.316 DEBUG:boto:Path: /bofrnf-53/?prefix=a 2023-10-18T20:20:22.316 DEBUG:boto:Data: 2023-10-18T20:20:22.316 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.317 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.317 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.318 DEBUG:boto:Params: {} 2023-10-18T20:20:22.318 DEBUG:boto:Token: None 2023-10-18T20:20:22.319 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.319 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001933678f16da77d5-0065303e06-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.323 DEBUG:boto:b'bofrnf-53a1000falsea2023-10-18T20:19:41.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.323 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.324 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.324 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.324 DEBUG:boto:Path: /bofrnf-53/a 2023-10-18T20:20:22.325 DEBUG:boto:Data: 2023-10-18T20:20:22.325 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.326 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.326 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.326 DEBUG:boto:Params: {} 2023-10-18T20:20:22.327 DEBUG:boto:Token: None 2023-10-18T20:20:22.327 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a 2023-10-18T20:20:22.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E49rnZh9KY6Ahkcxb+FJSiWjtg8= 2023-10-18T20:20:22.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E49rnZh9KY6Ahkcxb+FJSiWjtg8=', 'Content-Length': '0'} 2023-10-18T20:20:22.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c62311fdb85b4780-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.331 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.331 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.332 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.332 DEBUG:boto:Path: /bofrnf-53/a 2023-10-18T20:20:22.332 DEBUG:boto:Data: 2023-10-18T20:20:22.333 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.334 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.334 DEBUG:boto:Params: {} 2023-10-18T20:20:22.335 DEBUG:boto:Token: None 2023-10-18T20:20:22.335 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a 2023-10-18T20:20:22.335 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E49rnZh9KY6Ahkcxb+FJSiWjtg8= 2023-10-18T20:20:22.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E49rnZh9KY6Ahkcxb+FJSiWjtg8=', 'Content-Length': '0'} 2023-10-18T20:20:22.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008238e4c8c1870c92-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:20:22.339 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.339 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.340 DEBUG:boto:Method: GET 2023-10-18T20:20:22.340 DEBUG:boto:Path: /bofrnf-53/a 2023-10-18T20:20:22.341 DEBUG:boto:Data: 2023-10-18T20:20:22.341 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.341 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.342 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.342 DEBUG:boto:Params: {} 2023-10-18T20:20:22.343 DEBUG:boto:Token: None 2023-10-18T20:20:22.343 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a 2023-10-18T20:20:22.344 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYqx9fxpdRtWQcRFq7PN1yYX1LU= 2023-10-18T20:20:22.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYqx9fxpdRtWQcRFq7PN1yYX1LU=', 'Content-Length': '0'} 2023-10-18T20:20:22.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1661e9f29338dd3-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.347 DEBUG:boto:path=/bofrnf-53/a 2023-10-18T20:20:22.347 DEBUG:boto:auth_path=/bofrnf-53/a 2023-10-18T20:20:22.348 DEBUG:boto:Method: GET 2023-10-18T20:20:22.348 DEBUG:boto:Path: /bofrnf-53/a 2023-10-18T20:20:22.349 DEBUG:boto:Data: 2023-10-18T20:20:22.349 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.350 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.350 DEBUG:boto:Params: {} 2023-10-18T20:20:22.351 DEBUG:boto:Token: None 2023-10-18T20:20:22.351 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/a 2023-10-18T20:20:22.352 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYqx9fxpdRtWQcRFq7PN1yYX1LU= 2023-10-18T20:20:22.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYqx9fxpdRtWQcRFq7PN1yYX1LU=', 'Content-Length': '0'} 2023-10-18T20:20:22.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c6ef9c42217379-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.394 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.395 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.395 DEBUG:boto:path=/bofrnf-53/?prefix=a 2023-10-18T20:20:22.396 DEBUG:boto:auth_path=/bofrnf-53/?prefix=a 2023-10-18T20:20:22.396 DEBUG:boto:Method: GET 2023-10-18T20:20:22.396 DEBUG:boto:Path: /bofrnf-53/?prefix=a 2023-10-18T20:20:22.397 DEBUG:boto:Data: 2023-10-18T20:20:22.397 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.398 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.399 DEBUG:boto:Params: {} 2023-10-18T20:20:22.399 DEBUG:boto:Token: None 2023-10-18T20:20:22.399 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.400 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2b43cbac330b815-0065303e06-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.403 DEBUG:boto:b'bofrnf-53a1000falsea2023-10-18T20:19:41.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.404 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.404 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.405 DEBUG:boto:path=/bofrnf-53/?prefix=a 2023-10-18T20:20:22.405 DEBUG:boto:auth_path=/bofrnf-53/?prefix=a 2023-10-18T20:20:22.406 DEBUG:boto:Method: GET 2023-10-18T20:20:22.406 DEBUG:boto:Path: /bofrnf-53/?prefix=a 2023-10-18T20:20:22.407 DEBUG:boto:Data: 2023-10-18T20:20:22.407 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.408 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.408 DEBUG:boto:Params: {} 2023-10-18T20:20:22.409 DEBUG:boto:Token: None 2023-10-18T20:20:22.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038ff452c9a30821f-0065303e06-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.414 DEBUG:boto:b'bofrnf-53a1000falsea2023-10-18T20:19:41.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:20:22.415 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.415 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.416 DEBUG:boto:path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.416 DEBUG:boto:auth_path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.416 DEBUG:boto:Method: GET 2023-10-18T20:20:22.417 DEBUG:boto:Path: /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.417 DEBUG:boto:Data: 2023-10-18T20:20:22.418 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.419 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.419 DEBUG:boto:Params: {} 2023-10-18T20:20:22.419 DEBUG:boto:Token: None 2023-10-18T20:20:22.420 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lTZs0CICh+fJv+g9ebky1Z0Z+ns= 2023-10-18T20:20:22.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lTZs0CICh+fJv+g9ebky1Z0Z+ns=', 'Content-Length': '0'} 2023-10-18T20:20:22.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067eae30b3c15f503-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.470 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.471 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.471 DEBUG:boto:path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.471 DEBUG:boto:auth_path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.472 DEBUG:boto:Method: GET 2023-10-18T20:20:22.472 DEBUG:boto:Path: /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.473 DEBUG:boto:Data: 2023-10-18T20:20:22.473 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.474 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.475 DEBUG:boto:Params: {} 2023-10-18T20:20:22.475 DEBUG:boto:Token: None 2023-10-18T20:20:22.476 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lTZs0CICh+fJv+g9ebky1Z0Z+ns= 2023-10-18T20:20:22.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lTZs0CICh+fJv+g9ebky1Z0Z+ns=', 'Content-Length': '0'} 2023-10-18T20:20:22.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f9b15587efd5c2-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.479 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.480 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.480 DEBUG:boto:path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.480 DEBUG:boto:auth_path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.481 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.481 DEBUG:boto:Path: /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.482 DEBUG:boto:Data: 2023-10-18T20:20:22.482 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.482 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.483 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.483 DEBUG:boto:Params: {} 2023-10-18T20:20:22.484 DEBUG:boto:Token: None 2023-10-18T20:20:22.484 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.485 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yRl5B5fFPgCG2RQfJkYMjHNYt7I= 2023-10-18T20:20:22.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yRl5B5fFPgCG2RQfJkYMjHNYt7I=', 'Content-Length': '0'} 2023-10-18T20:20:22.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000749b5295302a1f91-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.488 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.488 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.489 DEBUG:boto:path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.489 DEBUG:boto:auth_path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.489 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.490 DEBUG:boto:Path: /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.490 DEBUG:boto:Data: 2023-10-18T20:20:22.491 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.491 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.492 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.492 DEBUG:boto:Params: {} 2023-10-18T20:20:22.492 DEBUG:boto:Token: None 2023-10-18T20:20:22.493 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.493 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yRl5B5fFPgCG2RQfJkYMjHNYt7I= 2023-10-18T20:20:22.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yRl5B5fFPgCG2RQfJkYMjHNYt7I=', 'Content-Length': '0'} 2023-10-18T20:20:22.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b17e6e62010ad194-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:20:22.497 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.497 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.497 DEBUG:boto:path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.498 DEBUG:boto:auth_path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.498 DEBUG:boto:Method: GET 2023-10-18T20:20:22.499 DEBUG:boto:Path: /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.499 DEBUG:boto:Data: 2023-10-18T20:20:22.500 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.500 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.500 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.501 DEBUG:boto:Params: {} 2023-10-18T20:20:22.501 DEBUG:boto:Token: None 2023-10-18T20:20:22.502 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.502 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lTZs0CICh+fJv+g9ebky1Z0Z+ns= 2023-10-18T20:20:22.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lTZs0CICh+fJv+g9ebky1Z0Z+ns=', 'Content-Length': '0'} 2023-10-18T20:20:22.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001913ef74734c308c-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.505 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.506 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.506 DEBUG:boto:path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.507 DEBUG:boto:auth_path=/bofrnf-53/b?versionId=null 2023-10-18T20:20:22.507 DEBUG:boto:Method: GET 2023-10-18T20:20:22.508 DEBUG:boto:Path: /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.508 DEBUG:boto:Data: 2023-10-18T20:20:22.509 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.509 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.509 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.510 DEBUG:boto:Params: {} 2023-10-18T20:20:22.510 DEBUG:boto:Token: None 2023-10-18T20:20:22.511 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b?versionId=null 2023-10-18T20:20:22.511 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lTZs0CICh+fJv+g9ebky1Z0Z+ns= 2023-10-18T20:20:22.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lTZs0CICh+fJv+g9ebky1Z0Z+ns=', 'Content-Length': '0'} 2023-10-18T20:20:22.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084cf0fd9d8f93523-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.514 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.515 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.515 DEBUG:boto:path=/bofrnf-53/?prefix=b 2023-10-18T20:20:22.515 DEBUG:boto:auth_path=/bofrnf-53/?prefix=b 2023-10-18T20:20:22.516 DEBUG:boto:Method: GET 2023-10-18T20:20:22.516 DEBUG:boto:Path: /bofrnf-53/?prefix=b 2023-10-18T20:20:22.517 DEBUG:boto:Data: 2023-10-18T20:20:22.517 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.518 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.518 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.518 DEBUG:boto:Params: {} 2023-10-18T20:20:22.519 DEBUG:boto:Token: None 2023-10-18T20:20:22.519 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bacf5a861b489d5-0065303e06-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.523 DEBUG:boto:b'bofrnf-53b1000falseb2023-10-18T20:19:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.524 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.524 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.525 DEBUG:boto:path=/bofrnf-53/?prefix=b 2023-10-18T20:20:22.525 DEBUG:boto:auth_path=/bofrnf-53/?prefix=b 2023-10-18T20:20:22.526 DEBUG:boto:Method: GET 2023-10-18T20:20:22.526 DEBUG:boto:Path: /bofrnf-53/?prefix=b 2023-10-18T20:20:22.527 DEBUG:boto:Data: 2023-10-18T20:20:22.527 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.527 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.528 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.528 DEBUG:boto:Params: {} 2023-10-18T20:20:22.529 DEBUG:boto:Token: None 2023-10-18T20:20:22.529 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.530 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004400a2affb73770-0065303e06-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.534 DEBUG:boto:b'bofrnf-53b1000falseb2023-10-18T20:19:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.534 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.535 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.535 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.535 DEBUG:boto:Path: /bofrnf-53/b 2023-10-18T20:20:22.536 DEBUG:boto:Data: 2023-10-18T20:20:22.536 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.537 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.537 DEBUG:boto:Params: {} 2023-10-18T20:20:22.538 DEBUG:boto:Token: None 2023-10-18T20:20:22.538 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b 2023-10-18T20:20:22.539 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HB+XXu4BltByf/XC2mEkKMywQ3s= 2023-10-18T20:20:22.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HB+XXu4BltByf/XC2mEkKMywQ3s=', 'Content-Length': '0'} 2023-10-18T20:20:22.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c1ea1744d86bbb-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.542 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.542 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.543 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.543 DEBUG:boto:Path: /bofrnf-53/b 2023-10-18T20:20:22.543 DEBUG:boto:Data: 2023-10-18T20:20:22.544 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.544 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.545 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.545 DEBUG:boto:Params: {} 2023-10-18T20:20:22.545 DEBUG:boto:Token: None 2023-10-18T20:20:22.546 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b 2023-10-18T20:20:22.546 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HB+XXu4BltByf/XC2mEkKMywQ3s= 2023-10-18T20:20:22.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HB+XXu4BltByf/XC2mEkKMywQ3s=', 'Content-Length': '0'} 2023-10-18T20:20:22.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb9b3f0bcc267d28-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:20:22.551 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.552 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.552 DEBUG:boto:Method: GET 2023-10-18T20:20:22.553 DEBUG:boto:Path: /bofrnf-53/b 2023-10-18T20:20:22.553 DEBUG:boto:Data: 2023-10-18T20:20:22.553 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.554 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.554 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.555 DEBUG:boto:Params: {} 2023-10-18T20:20:22.555 DEBUG:boto:Token: None 2023-10-18T20:20:22.556 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b 2023-10-18T20:20:22.556 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3wMw5m5v6JVDaS1pRDRYO9LzJ1E= 2023-10-18T20:20:22.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wMw5m5v6JVDaS1pRDRYO9LzJ1E=', 'Content-Length': '0'} 2023-10-18T20:20:22.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad9f1ffe9d7a9d8-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.602 DEBUG:boto:path=/bofrnf-53/b 2023-10-18T20:20:22.603 DEBUG:boto:auth_path=/bofrnf-53/b 2023-10-18T20:20:22.603 DEBUG:boto:Method: GET 2023-10-18T20:20:22.604 DEBUG:boto:Path: /bofrnf-53/b 2023-10-18T20:20:22.604 DEBUG:boto:Data: 2023-10-18T20:20:22.605 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.605 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.606 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.606 DEBUG:boto:Params: {} 2023-10-18T20:20:22.607 DEBUG:boto:Token: None 2023-10-18T20:20:22.607 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/b 2023-10-18T20:20:22.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3wMw5m5v6JVDaS1pRDRYO9LzJ1E= 2023-10-18T20:20:22.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wMw5m5v6JVDaS1pRDRYO9LzJ1E=', 'Content-Length': '0'} 2023-10-18T20:20:22.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000760385d9ea85004f-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.611 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.612 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.612 DEBUG:boto:path=/bofrnf-53/?prefix=b 2023-10-18T20:20:22.613 DEBUG:boto:auth_path=/bofrnf-53/?prefix=b 2023-10-18T20:20:22.613 DEBUG:boto:Method: GET 2023-10-18T20:20:22.613 DEBUG:boto:Path: /bofrnf-53/?prefix=b 2023-10-18T20:20:22.614 DEBUG:boto:Data: 2023-10-18T20:20:22.614 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.615 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.615 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.615 DEBUG:boto:Params: {} 2023-10-18T20:20:22.616 DEBUG:boto:Token: None 2023-10-18T20:20:22.616 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.617 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048eca4a22ca7ce39-0065303e06-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.621 DEBUG:boto:b'bofrnf-53b1000falseb2023-10-18T20:19:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.621 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.622 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.622 DEBUG:boto:path=/bofrnf-53/?prefix=b 2023-10-18T20:20:22.623 DEBUG:boto:auth_path=/bofrnf-53/?prefix=b 2023-10-18T20:20:22.623 DEBUG:boto:Method: GET 2023-10-18T20:20:22.623 DEBUG:boto:Path: /bofrnf-53/?prefix=b 2023-10-18T20:20:22.624 DEBUG:boto:Data: 2023-10-18T20:20:22.624 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.625 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.626 DEBUG:boto:Params: {} 2023-10-18T20:20:22.626 DEBUG:boto:Token: None 2023-10-18T20:20:22.627 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.627 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ccc9fcb68edaf4a-0065303e06-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.670 DEBUG:boto:b'bofrnf-53b1000falseb2023-10-18T20:19:41.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:20:22.671 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.672 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.672 DEBUG:boto:path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.673 DEBUG:boto:auth_path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.673 DEBUG:boto:Method: GET 2023-10-18T20:20:22.674 DEBUG:boto:Path: /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.674 DEBUG:boto:Data: 2023-10-18T20:20:22.674 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.675 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.675 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.676 DEBUG:boto:Params: {} 2023-10-18T20:20:22.676 DEBUG:boto:Token: None 2023-10-18T20:20:22.677 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.677 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AzE3Q5JJQTNGudOZaZqb9pLk/DI= 2023-10-18T20:20:22.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AzE3Q5JJQTNGudOZaZqb9pLk/DI=', 'Content-Length': '0'} 2023-10-18T20:20:22.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d85505bb0a85c771-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.680 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.681 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.681 DEBUG:boto:path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.682 DEBUG:boto:auth_path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.682 DEBUG:boto:Method: GET 2023-10-18T20:20:22.682 DEBUG:boto:Path: /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.683 DEBUG:boto:Data: 2023-10-18T20:20:22.683 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.684 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.684 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.684 DEBUG:boto:Params: {} 2023-10-18T20:20:22.685 DEBUG:boto:Token: None 2023-10-18T20:20:22.685 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.686 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AzE3Q5JJQTNGudOZaZqb9pLk/DI= 2023-10-18T20:20:22.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AzE3Q5JJQTNGudOZaZqb9pLk/DI=', 'Content-Length': '0'} 2023-10-18T20:20:22.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aaa4aa4d96d664a-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.689 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.690 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.690 DEBUG:boto:path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.690 DEBUG:boto:auth_path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.691 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.691 DEBUG:boto:Path: /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.692 DEBUG:boto:Data: 2023-10-18T20:20:22.692 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.692 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.693 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.693 DEBUG:boto:Params: {} 2023-10-18T20:20:22.694 DEBUG:boto:Token: None 2023-10-18T20:20:22.694 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.695 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z28r/bTk8Bq9SBY7NcT+roiaZ0w= 2023-10-18T20:20:22.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z28r/bTk8Bq9SBY7NcT+roiaZ0w=', 'Content-Length': '0'} 2023-10-18T20:20:22.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225b831d428c63eb-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.698 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.698 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.699 DEBUG:boto:path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.699 DEBUG:boto:auth_path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.699 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.700 DEBUG:boto:Path: /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.700 DEBUG:boto:Data: 2023-10-18T20:20:22.701 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.701 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.701 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.702 DEBUG:boto:Params: {} 2023-10-18T20:20:22.702 DEBUG:boto:Token: None 2023-10-18T20:20:22.703 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.703 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z28r/bTk8Bq9SBY7NcT+roiaZ0w= 2023-10-18T20:20:22.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z28r/bTk8Bq9SBY7NcT+roiaZ0w=', 'Content-Length': '0'} 2023-10-18T20:20:22.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9f76c733e06edc5-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:20:22.707 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.707 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.708 DEBUG:boto:path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.708 DEBUG:boto:auth_path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.708 DEBUG:boto:Method: GET 2023-10-18T20:20:22.709 DEBUG:boto:Path: /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.709 DEBUG:boto:Data: 2023-10-18T20:20:22.710 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.710 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.711 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.711 DEBUG:boto:Params: {} 2023-10-18T20:20:22.711 DEBUG:boto:Token: None 2023-10-18T20:20:22.712 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AzE3Q5JJQTNGudOZaZqb9pLk/DI= 2023-10-18T20:20:22.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AzE3Q5JJQTNGudOZaZqb9pLk/DI=', 'Content-Length': '0'} 2023-10-18T20:20:22.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0a7008e8592ddac-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.715 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.716 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.716 DEBUG:boto:path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.717 DEBUG:boto:auth_path=/bofrnf-53/c?versionId=null 2023-10-18T20:20:22.717 DEBUG:boto:Method: GET 2023-10-18T20:20:22.718 DEBUG:boto:Path: /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.718 DEBUG:boto:Data: 2023-10-18T20:20:22.719 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.719 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.720 DEBUG:boto:Params: {} 2023-10-18T20:20:22.720 DEBUG:boto:Token: None 2023-10-18T20:20:22.721 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c?versionId=null 2023-10-18T20:20:22.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AzE3Q5JJQTNGudOZaZqb9pLk/DI= 2023-10-18T20:20:22.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AzE3Q5JJQTNGudOZaZqb9pLk/DI=', 'Content-Length': '0'} 2023-10-18T20:20:22.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6e4c0f26cf63e0-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.725 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.725 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.725 DEBUG:boto:path=/bofrnf-53/?prefix=c 2023-10-18T20:20:22.726 DEBUG:boto:auth_path=/bofrnf-53/?prefix=c 2023-10-18T20:20:22.726 DEBUG:boto:Method: GET 2023-10-18T20:20:22.727 DEBUG:boto:Path: /bofrnf-53/?prefix=c 2023-10-18T20:20:22.727 DEBUG:boto:Data: 2023-10-18T20:20:22.728 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.728 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.729 DEBUG:boto:Params: {} 2023-10-18T20:20:22.729 DEBUG:boto:Token: None 2023-10-18T20:20:22.730 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.730 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002383592afa996f27-0065303e06-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.734 DEBUG:boto:b'bofrnf-53c1000falsec2023-10-18T20:19:41.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.735 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.735 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.735 DEBUG:boto:path=/bofrnf-53/?prefix=c 2023-10-18T20:20:22.736 DEBUG:boto:auth_path=/bofrnf-53/?prefix=c 2023-10-18T20:20:22.736 DEBUG:boto:Method: GET 2023-10-18T20:20:22.737 DEBUG:boto:Path: /bofrnf-53/?prefix=c 2023-10-18T20:20:22.737 DEBUG:boto:Data: 2023-10-18T20:20:22.738 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.738 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.739 DEBUG:boto:Params: {} 2023-10-18T20:20:22.739 DEBUG:boto:Token: None 2023-10-18T20:20:22.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d8380b77306794-0065303e06-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.744 DEBUG:boto:b'bofrnf-53c1000falsec2023-10-18T20:19:41.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.744 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.745 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.745 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.746 DEBUG:boto:Path: /bofrnf-53/c 2023-10-18T20:20:22.746 DEBUG:boto:Data: 2023-10-18T20:20:22.747 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.747 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.748 DEBUG:boto:Params: {} 2023-10-18T20:20:22.748 DEBUG:boto:Token: None 2023-10-18T20:20:22.749 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c 2023-10-18T20:20:22.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q7+B6D9P+FZ7wYmebodQqn0bv94= 2023-10-18T20:20:22.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q7+B6D9P+FZ7wYmebodQqn0bv94=', 'Content-Length': '0'} 2023-10-18T20:20:22.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e57bf182c636ab-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.755 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.755 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.756 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.756 DEBUG:boto:Path: /bofrnf-53/c 2023-10-18T20:20:22.757 DEBUG:boto:Data: 2023-10-18T20:20:22.757 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.758 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.758 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.758 DEBUG:boto:Params: {} 2023-10-18T20:20:22.759 DEBUG:boto:Token: None 2023-10-18T20:20:22.759 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c 2023-10-18T20:20:22.760 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q7+B6D9P+FZ7wYmebodQqn0bv94= 2023-10-18T20:20:22.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q7+B6D9P+FZ7wYmebodQqn0bv94=', 'Content-Length': '0'} 2023-10-18T20:20:22.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011b213873d784f78-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:20:22.763 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.764 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.764 DEBUG:boto:Method: GET 2023-10-18T20:20:22.765 DEBUG:boto:Path: /bofrnf-53/c 2023-10-18T20:20:22.765 DEBUG:boto:Data: 2023-10-18T20:20:22.766 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.766 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.766 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.767 DEBUG:boto:Params: {} 2023-10-18T20:20:22.767 DEBUG:boto:Token: None 2023-10-18T20:20:22.768 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c 2023-10-18T20:20:22.768 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JEKtbE8ENNOXjRlWHVEVJhk3nh8= 2023-10-18T20:20:22.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JEKtbE8ENNOXjRlWHVEVJhk3nh8=', 'Content-Length': '0'} 2023-10-18T20:20:22.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e60d62215aa6bb-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.771 DEBUG:boto:path=/bofrnf-53/c 2023-10-18T20:20:22.772 DEBUG:boto:auth_path=/bofrnf-53/c 2023-10-18T20:20:22.772 DEBUG:boto:Method: GET 2023-10-18T20:20:22.773 DEBUG:boto:Path: /bofrnf-53/c 2023-10-18T20:20:22.773 DEBUG:boto:Data: 2023-10-18T20:20:22.774 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.774 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.775 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.775 DEBUG:boto:Params: {} 2023-10-18T20:20:22.775 DEBUG:boto:Token: None 2023-10-18T20:20:22.776 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/c 2023-10-18T20:20:22.777 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JEKtbE8ENNOXjRlWHVEVJhk3nh8= 2023-10-18T20:20:22.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JEKtbE8ENNOXjRlWHVEVJhk3nh8=', 'Content-Length': '0'} 2023-10-18T20:20:22.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007073546b858c2bc-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.822 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.823 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.823 DEBUG:boto:path=/bofrnf-53/?prefix=c 2023-10-18T20:20:22.824 DEBUG:boto:auth_path=/bofrnf-53/?prefix=c 2023-10-18T20:20:22.824 DEBUG:boto:Method: GET 2023-10-18T20:20:22.824 DEBUG:boto:Path: /bofrnf-53/?prefix=c 2023-10-18T20:20:22.825 DEBUG:boto:Data: 2023-10-18T20:20:22.825 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.826 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.826 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.826 DEBUG:boto:Params: {} 2023-10-18T20:20:22.827 DEBUG:boto:Token: None 2023-10-18T20:20:22.827 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.828 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000654f3f08562b0078-0065303e06-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.831 DEBUG:boto:b'bofrnf-53c1000falsec2023-10-18T20:19:41.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.832 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.833 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.833 DEBUG:boto:path=/bofrnf-53/?prefix=c 2023-10-18T20:20:22.833 DEBUG:boto:auth_path=/bofrnf-53/?prefix=c 2023-10-18T20:20:22.834 DEBUG:boto:Method: GET 2023-10-18T20:20:22.834 DEBUG:boto:Path: /bofrnf-53/?prefix=c 2023-10-18T20:20:22.835 DEBUG:boto:Data: 2023-10-18T20:20:22.835 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.836 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.836 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.836 DEBUG:boto:Params: {} 2023-10-18T20:20:22.837 DEBUG:boto:Token: None 2023-10-18T20:20:22.837 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.838 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dd0ae6a8794a32b-0065303e06-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.841 DEBUG:boto:b'bofrnf-53c1000falsec2023-10-18T20:19:41.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:20:22.843 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:22.843 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:22.843 DEBUG:boto:path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.844 DEBUG:boto:auth_path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.844 DEBUG:boto:Method: GET 2023-10-18T20:20:22.845 DEBUG:boto:Path: /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.845 DEBUG:boto:Data: 2023-10-18T20:20:22.846 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.846 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.846 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.847 DEBUG:boto:Params: {} 2023-10-18T20:20:22.847 DEBUG:boto:Token: None 2023-10-18T20:20:22.848 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a+OB8hOoFPUFz8B8taVAr9tlDyw= 2023-10-18T20:20:22.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a+OB8hOoFPUFz8B8taVAr9tlDyw=', 'Content-Length': '0'} 2023-10-18T20:20:22.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09c8d9da9bc50f7-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.894 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:22.895 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:22.895 DEBUG:boto:path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.896 DEBUG:boto:auth_path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.896 DEBUG:boto:Method: GET 2023-10-18T20:20:22.896 DEBUG:boto:Path: /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.897 DEBUG:boto:Data: 2023-10-18T20:20:22.897 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.898 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.898 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.899 DEBUG:boto:Params: {} 2023-10-18T20:20:22.899 DEBUG:boto:Token: None 2023-10-18T20:20:22.899 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.900 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a+OB8hOoFPUFz8B8taVAr9tlDyw= 2023-10-18T20:20:22.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a+OB8hOoFPUFz8B8taVAr9tlDyw=', 'Content-Length': '0'} 2023-10-18T20:20:22.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037bbf831cc5810eb-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.903 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:22.904 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:22.904 DEBUG:boto:path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.904 DEBUG:boto:auth_path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.905 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.905 DEBUG:boto:Path: /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.906 DEBUG:boto:Data: 2023-10-18T20:20:22.906 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.906 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.907 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.907 DEBUG:boto:Params: {} 2023-10-18T20:20:22.908 DEBUG:boto:Token: None 2023-10-18T20:20:22.908 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.908 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sAWvMxDOEgZGNuNH9iAuMg9oa9Y= 2023-10-18T20:20:22.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sAWvMxDOEgZGNuNH9iAuMg9oa9Y=', 'Content-Length': '0'} 2023-10-18T20:20:22.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c508adca657fc0c-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.912 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:22.912 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:22.912 DEBUG:boto:path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.913 DEBUG:boto:auth_path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.913 DEBUG:boto:Method: HEAD 2023-10-18T20:20:22.914 DEBUG:boto:Path: /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.914 DEBUG:boto:Data: 2023-10-18T20:20:22.914 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.915 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.916 DEBUG:boto:Params: {} 2023-10-18T20:20:22.916 DEBUG:boto:Token: None 2023-10-18T20:20:22.916 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sAWvMxDOEgZGNuNH9iAuMg9oa9Y= 2023-10-18T20:20:22.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sAWvMxDOEgZGNuNH9iAuMg9oa9Y=', 'Content-Length': '0'} 2023-10-18T20:20:22.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2cc6b08a9defe22-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:20:22.920 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:22.921 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:22.921 DEBUG:boto:path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.922 DEBUG:boto:auth_path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.922 DEBUG:boto:Method: GET 2023-10-18T20:20:22.922 DEBUG:boto:Path: /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.923 DEBUG:boto:Data: 2023-10-18T20:20:22.923 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.924 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.924 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.925 DEBUG:boto:Params: {} 2023-10-18T20:20:22.925 DEBUG:boto:Token: None 2023-10-18T20:20:22.925 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a+OB8hOoFPUFz8B8taVAr9tlDyw= 2023-10-18T20:20:22.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a+OB8hOoFPUFz8B8taVAr9tlDyw=', 'Content-Length': '0'} 2023-10-18T20:20:22.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424f48a3bd17f45f-0065303e06-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.929 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:22.930 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:22.930 DEBUG:boto:path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.931 DEBUG:boto:auth_path=/bofrnf-53/d?versionId=null 2023-10-18T20:20:22.931 DEBUG:boto:Method: GET 2023-10-18T20:20:22.931 DEBUG:boto:Path: /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.932 DEBUG:boto:Data: 2023-10-18T20:20:22.932 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.933 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.933 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.933 DEBUG:boto:Params: {} 2023-10-18T20:20:22.934 DEBUG:boto:Token: None 2023-10-18T20:20:22.934 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/d?versionId=null 2023-10-18T20:20:22.935 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a+OB8hOoFPUFz8B8taVAr9tlDyw= 2023-10-18T20:20:22.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a+OB8hOoFPUFz8B8taVAr9tlDyw=', 'Content-Length': '0'} 2023-10-18T20:20:22.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2c6432f01e841f-0065303e06-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.978 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.979 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.979 DEBUG:boto:path=/bofrnf-53/?prefix=d 2023-10-18T20:20:22.980 DEBUG:boto:auth_path=/bofrnf-53/?prefix=d 2023-10-18T20:20:22.980 DEBUG:boto:Method: GET 2023-10-18T20:20:22.980 DEBUG:boto:Path: /bofrnf-53/?prefix=d 2023-10-18T20:20:22.981 DEBUG:boto:Data: 2023-10-18T20:20:22.981 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.982 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:22.982 DEBUG:boto:Port: 8001 2023-10-18T20:20:22.982 DEBUG:boto:Params: {} 2023-10-18T20:20:22.983 DEBUG:boto:Token: None 2023-10-18T20:20:22.983 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.984 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f029e6bcdda9ed-0065303e06-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.988 DEBUG:boto:b'bofrnf-53d1000falsed2023-10-18T20:19:41.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.988 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:22.989 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:22.989 DEBUG:boto:path=/bofrnf-53/?prefix=d 2023-10-18T20:20:22.990 DEBUG:boto:auth_path=/bofrnf-53/?prefix=d 2023-10-18T20:20:22.990 DEBUG:boto:Method: GET 2023-10-18T20:20:22.990 DEBUG:boto:Path: /bofrnf-53/?prefix=d 2023-10-18T20:20:22.991 DEBUG:boto:Data: 2023-10-18T20:20:22.991 DEBUG:boto:Headers: {} 2023-10-18T20:20:22.992 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:22.992 DEBUG:boto:Port: 8000 2023-10-18T20:20:22.993 DEBUG:boto:Params: {} 2023-10-18T20:20:22.993 DEBUG:boto:Token: None 2023-10-18T20:20:22.993 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:22 GMT /bofrnf-53/ 2023-10-18T20:20:22.994 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI= 2023-10-18T20:20:22.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h8k8YRZHLRwmoSgSfrH3xTjCvqI=', 'Content-Length': '0'} 2023-10-18T20:20:22.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b160dfe7d899416-0065303e06-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:22.998 DEBUG:boto:b'bofrnf-53d1000falsed2023-10-18T20:19:41.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:22.998 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:22.999 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:22.999 DEBUG:boto:Method: HEAD 2023-10-18T20:20:23.000 DEBUG:boto:Path: /bofrnf-53/d 2023-10-18T20:20:23.000 DEBUG:boto:Data: 2023-10-18T20:20:23.001 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:23.001 DEBUG:boto:Port: 8001 2023-10-18T20:20:23.002 DEBUG:boto:Params: {} 2023-10-18T20:20:23.002 DEBUG:boto:Token: None 2023-10-18T20:20:23.003 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-53/d 2023-10-18T20:20:23.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:facqoZj/1tAdcrRxL6C0/ky2u4c= 2023-10-18T20:20:23.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:facqoZj/1tAdcrRxL6C0/ky2u4c=', 'Content-Length': '0'} 2023-10-18T20:20:23.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009331eef719b9daec-0065303e07-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.006 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:23.006 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:23.007 DEBUG:boto:Method: HEAD 2023-10-18T20:20:23.007 DEBUG:boto:Path: /bofrnf-53/d 2023-10-18T20:20:23.008 DEBUG:boto:Data: 2023-10-18T20:20:23.008 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.008 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.009 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.009 DEBUG:boto:Params: {} 2023-10-18T20:20:23.010 DEBUG:boto:Token: None 2023-10-18T20:20:23.010 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-53/d 2023-10-18T20:20:23.011 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:facqoZj/1tAdcrRxL6C0/ky2u4c= 2023-10-18T20:20:23.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:facqoZj/1tAdcrRxL6C0/ky2u4c=', 'Content-Length': '0'} 2023-10-18T20:20:23.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f2dd5569771467-0065303e07-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:20:23.014 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:23.014 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:23.015 DEBUG:boto:Method: GET 2023-10-18T20:20:23.015 DEBUG:boto:Path: /bofrnf-53/d 2023-10-18T20:20:23.016 DEBUG:boto:Data: 2023-10-18T20:20:23.016 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.017 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:23.017 DEBUG:boto:Port: 8001 2023-10-18T20:20:23.017 DEBUG:boto:Params: {} 2023-10-18T20:20:23.018 DEBUG:boto:Token: None 2023-10-18T20:20:23.018 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-53/d 2023-10-18T20:20:23.019 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8I74vCwiLs+K1WiLcA/rHa63kAc= 2023-10-18T20:20:23.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8I74vCwiLs+K1WiLcA/rHa63kAc=', 'Content-Length': '0'} 2023-10-18T20:20:23.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b9b32c8c5da9b9a-0065303e07-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.062 DEBUG:boto:path=/bofrnf-53/d 2023-10-18T20:20:23.063 DEBUG:boto:auth_path=/bofrnf-53/d 2023-10-18T20:20:23.063 DEBUG:boto:Method: GET 2023-10-18T20:20:23.063 DEBUG:boto:Path: /bofrnf-53/d 2023-10-18T20:20:23.064 DEBUG:boto:Data: 2023-10-18T20:20:23.064 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.065 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.065 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.066 DEBUG:boto:Params: {} 2023-10-18T20:20:23.066 DEBUG:boto:Token: None 2023-10-18T20:20:23.067 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-53/d 2023-10-18T20:20:23.067 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8I74vCwiLs+K1WiLcA/rHa63kAc= 2023-10-18T20:20:23.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8I74vCwiLs+K1WiLcA/rHa63kAc=', 'Content-Length': '0'} 2023-10-18T20:20:23.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:19:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099801f88ef2ea198-0065303e07-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.070 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:23.070 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:23.071 DEBUG:boto:path=/bofrnf-53/?prefix=d 2023-10-18T20:20:23.071 DEBUG:boto:auth_path=/bofrnf-53/?prefix=d 2023-10-18T20:20:23.072 DEBUG:boto:Method: GET 2023-10-18T20:20:23.072 DEBUG:boto:Path: /bofrnf-53/?prefix=d 2023-10-18T20:20:23.072 DEBUG:boto:Data: 2023-10-18T20:20:23.073 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:23.073 DEBUG:boto:Port: 8001 2023-10-18T20:20:23.074 DEBUG:boto:Params: {} 2023-10-18T20:20:23.074 DEBUG:boto:Token: None 2023-10-18T20:20:23.075 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-53/ 2023-10-18T20:20:23.075 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o1z97WxSMaLv3AYXtgyZyIsXCBc= 2023-10-18T20:20:23.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o1z97WxSMaLv3AYXtgyZyIsXCBc=', 'Content-Length': '0'} 2023-10-18T20:20:23.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000274f38b0e6ecb59-0065303e07-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.079 DEBUG:boto:b'bofrnf-53d1000falsed2023-10-18T20:19:41.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:23.080 DEBUG:boto:path=/bofrnf-53/ 2023-10-18T20:20:23.080 DEBUG:boto:auth_path=/bofrnf-53/ 2023-10-18T20:20:23.081 DEBUG:boto:path=/bofrnf-53/?prefix=d 2023-10-18T20:20:23.081 DEBUG:boto:auth_path=/bofrnf-53/?prefix=d 2023-10-18T20:20:23.081 DEBUG:boto:Method: GET 2023-10-18T20:20:23.082 DEBUG:boto:Path: /bofrnf-53/?prefix=d 2023-10-18T20:20:23.082 DEBUG:boto:Data: 2023-10-18T20:20:23.083 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.083 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.084 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.084 DEBUG:boto:Params: {} 2023-10-18T20:20:23.084 DEBUG:boto:Token: None 2023-10-18T20:20:23.085 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-53/ 2023-10-18T20:20:23.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o1z97WxSMaLv3AYXtgyZyIsXCBc= 2023-10-18T20:20:23.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o1z97WxSMaLv3AYXtgyZyIsXCBc=', 'Content-Length': '0'} 2023-10-18T20:20:23.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000488657ae35b4b348-0065303e07-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.089 DEBUG:boto:b'bofrnf-53d1000falsed2023-10-18T20:19:41.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:23.090 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-53 zones={a2, a1} 2023-10-18T20:20:23.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:20:23.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:20:23.091 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:20:23.144 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:20:23.145 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:20:23.193 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-54 2023-10-18T20:20:23.194 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:23.194 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:23.195 DEBUG:boto:Method: PUT 2023-10-18T20:20:23.195 DEBUG:boto:Path: /bofrnf-54/ 2023-10-18T20:20:23.195 DEBUG:boto:Data: 2023-10-18T20:20:23.196 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.196 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.196 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.197 DEBUG:boto:Params: {} 2023-10-18T20:20:23.197 DEBUG:boto:Token: None 2023-10-18T20:20:23.198 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-54/ 2023-10-18T20:20:23.198 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uo4WEYi9ESIECa8AGxf+zVJb4CU= 2023-10-18T20:20:23.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uo4WEYi9ESIECa8AGxf+zVJb4CU=', 'Content-Length': '0'} 2023-10-18T20:20:23.214 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d224ae3583c32401-0065303e07-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.214 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:23.214 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:23.215 DEBUG:boto:Method: HEAD 2023-10-18T20:20:23.215 DEBUG:boto:Path: /bofrnf-54/ 2023-10-18T20:20:23.216 DEBUG:boto:Data: 2023-10-18T20:20:23.216 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.216 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.217 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.217 DEBUG:boto:Params: {} 2023-10-18T20:20:23.218 DEBUG:boto:Token: None 2023-10-18T20:20:23.218 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-54/ 2023-10-18T20:20:23.218 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DCPh4R+HgJ8S0ybWiqhkXtzB3YU= 2023-10-18T20:20:23.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DCPh4R+HgJ8S0ybWiqhkXtzB3YU=', 'Content-Length': '0'} 2023-10-18T20:20:23.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c521749f1a65a8f-0065303e07-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.222 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:23.222 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:23.223 DEBUG:boto:Method: PUT 2023-10-18T20:20:23.223 DEBUG:boto:Path: /bofrnf-54/a 2023-10-18T20:20:23.223 DEBUG:boto:Data: 2023-10-18T20:20:23.224 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:20:23.224 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.224 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.225 DEBUG:boto:Params: {} 2023-10-18T20:20:23.225 DEBUG:boto:Token: None 2023-10-18T20:20:23.226 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-54/a 2023-10-18T20:20:23.226 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4MYtpNmBb4kdzshbrv2n3XbzCII= 2023-10-18T20:20:23.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4MYtpNmBb4kdzshbrv2n3XbzCII='} 2023-10-18T20:20:23.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d35d0655b348a3a-0065303e07-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.233 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:23.233 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:23.234 DEBUG:boto:Method: HEAD 2023-10-18T20:20:23.234 DEBUG:boto:Path: /bofrnf-54/ 2023-10-18T20:20:23.235 DEBUG:boto:Data: 2023-10-18T20:20:23.235 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.236 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.236 DEBUG:boto:Params: {} 2023-10-18T20:20:23.236 DEBUG:boto:Token: None 2023-10-18T20:20:23.237 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-54/ 2023-10-18T20:20:23.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DCPh4R+HgJ8S0ybWiqhkXtzB3YU= 2023-10-18T20:20:23.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DCPh4R+HgJ8S0ybWiqhkXtzB3YU=', 'Content-Length': '0'} 2023-10-18T20:20:23.240 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf9bea9c4c234c2e-0065303e07-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.241 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:23.241 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:23.241 DEBUG:boto:Method: PUT 2023-10-18T20:20:23.242 DEBUG:boto:Path: /bofrnf-54/b 2023-10-18T20:20:23.242 DEBUG:boto:Data: 2023-10-18T20:20:23.242 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:20:23.243 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.243 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.244 DEBUG:boto:Params: {} 2023-10-18T20:20:23.244 DEBUG:boto:Token: None 2023-10-18T20:20:23.244 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-54/b 2023-10-18T20:20:23.245 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:snZl0C97l4ll9Bp0JhQdhZ+MfcI= 2023-10-18T20:20:23.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:snZl0C97l4ll9Bp0JhQdhZ+MfcI='} 2023-10-18T20:20:23.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebda2c6651cd0d30-0065303e07-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.291 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:23.291 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:23.292 DEBUG:boto:Method: HEAD 2023-10-18T20:20:23.292 DEBUG:boto:Path: /bofrnf-54/ 2023-10-18T20:20:23.292 DEBUG:boto:Data: 2023-10-18T20:20:23.293 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.293 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.293 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.294 DEBUG:boto:Params: {} 2023-10-18T20:20:23.294 DEBUG:boto:Token: None 2023-10-18T20:20:23.295 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-54/ 2023-10-18T20:20:23.295 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DCPh4R+HgJ8S0ybWiqhkXtzB3YU= 2023-10-18T20:20:23.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DCPh4R+HgJ8S0ybWiqhkXtzB3YU=', 'Content-Length': '0'} 2023-10-18T20:20:23.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007be80a400e537dd6-0065303e07-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.299 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:23.299 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:23.299 DEBUG:boto:Method: PUT 2023-10-18T20:20:23.300 DEBUG:boto:Path: /bofrnf-54/c 2023-10-18T20:20:23.300 DEBUG:boto:Data: 2023-10-18T20:20:23.300 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:20:23.301 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.301 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.302 DEBUG:boto:Params: {} 2023-10-18T20:20:23.302 DEBUG:boto:Token: None 2023-10-18T20:20:23.302 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-54/c 2023-10-18T20:20:23.303 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kGueSdWlnEqRYDNiLu0t4zRvnCY= 2023-10-18T20:20:23.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kGueSdWlnEqRYDNiLu0t4zRvnCY='} 2023-10-18T20:20:23.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004db76b9b53184c0-0065303e07-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.309 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:23.309 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:23.310 DEBUG:boto:Method: HEAD 2023-10-18T20:20:23.310 DEBUG:boto:Path: /bofrnf-54/ 2023-10-18T20:20:23.310 DEBUG:boto:Data: 2023-10-18T20:20:23.311 DEBUG:boto:Headers: {} 2023-10-18T20:20:23.311 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.311 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.312 DEBUG:boto:Params: {} 2023-10-18T20:20:23.312 DEBUG:boto:Token: None 2023-10-18T20:20:23.313 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-54/ 2023-10-18T20:20:23.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DCPh4R+HgJ8S0ybWiqhkXtzB3YU= 2023-10-18T20:20:23.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DCPh4R+HgJ8S0ybWiqhkXtzB3YU=', 'Content-Length': '0'} 2023-10-18T20:20:23.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d104992dc5d359b-0065303e07-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.317 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:23.317 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:23.318 DEBUG:boto:Method: PUT 2023-10-18T20:20:23.318 DEBUG:boto:Path: /bofrnf-54/d 2023-10-18T20:20:23.318 DEBUG:boto:Data: 2023-10-18T20:20:23.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:20:23.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:23.319 DEBUG:boto:Port: 8000 2023-10-18T20:20:23.320 DEBUG:boto:Params: {} 2023-10-18T20:20:23.320 DEBUG:boto:Token: None 2023-10-18T20:20:23.321 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:20:23 GMT /bofrnf-54/d 2023-10-18T20:20:23.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uOKj/MgxmVtZq1guDxdpXqrlbvQ= 2023-10-18T20:20:23.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:20:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uOKj/MgxmVtZq1guDxdpXqrlbvQ='} 2023-10-18T20:20:23.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe209d4aa5eeaf7c-0065303e07-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:23.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:20:23.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:20:23.368 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:20:23.416 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:20:23.417 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:20:23.446 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:20:23.458 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:20:23.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:23.459 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660423.211791_873.1", 2023-10-18T20:20:23.459 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:20:23.211791Z" 2023-10-18T20:20:23.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:23.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:23.460 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:20:23.460 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:20:23.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:23.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:23.461 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:20:23.461 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:20:23.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:23.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:23.461 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:20:23.462 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:20:23.462 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:23.462 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:20:23.463 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660423.211791_873.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:20:23.464 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:20:23.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:20:23.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:20:23.465 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:20:23.514 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:20:23.514 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:20:23.543 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:23.542+0000 7f9c1da22780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:20:23.544 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:23.543+0000 7f9c1da22780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:20:23.556 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:20:23.556 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:20:23.556 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:20:23.556 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:20:23.557 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:20:23.557 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:20:23.557 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:23.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:23.558 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:20:23.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:23.558 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:20:23.558 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:20:23.558 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:20:23.559 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660423.205682_867.1", 2023-10-18T20:20:23.559 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:20:23.559 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:20:23.559 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:20:23.559 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:20:23.205682Z", 2023-10-18T20:20:23.560 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:23.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:23.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:23.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:23.560 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:20:23.560 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:20:23.561 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:20:23.561 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:20:23.561 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:20:23.561 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:20:23.561 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:20:23.562 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:20:23.562 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:23.562 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:23.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:23.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:23.563 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:20:23.563 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:20:23.563 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:20:23.563 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:20:23.563 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:20:23.563 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:20:23.564 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:20:23.564 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:20:23.564 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:23.564 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:23.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:23.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:23.565 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:20:23.565 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:20:23.565 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:20:23.565 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:20:23.565 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:20:23.566 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:20:23.566 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:20:23.566 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:20:23.566 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:23.567 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:23.567 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:23.569 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:20:23.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:23.569 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:20:23.570 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:20:23.570 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:20:23.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:23.570 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:20:23Z" 2023-10-18T20:20:23.570 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:20:23.571 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660423.205682_867.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:20:23.205682Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:20:23Z" } 2023-10-18T20:20:23.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660423.205682_867.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:20:23.205682Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:20:23.572 DEBUG:rgw_multi.tests:log_status={0: '1_1697660423.211791_873.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:20:23.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660423.205682_867.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:20:23.573 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697660423.211791_873.1 target=1_1697660423.205682_867.1 2023-10-18T20:20:28.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:20:28.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:20:28.576 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:20:28.631 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:20:28.631 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:20:28.661 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:28.660+0000 7f147a4f9780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:20:28.662 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:28.661+0000 7f147a4f9780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:20:28.675 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:20:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:20:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:20:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:20:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:20:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:20:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:20:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:20:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:20:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:20:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660423.211791_873.1", 2023-10-18T20:20:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:20:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:20:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:20:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:20:23.211791Z", 2023-10-18T20:20:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:20:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:20:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:20:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:20:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:20:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:20:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:20:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:20:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:20:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:20:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:20:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:20:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:20:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:20:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:20:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:20:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:20:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:20:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:20:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:20:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:20:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:20:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:20:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:20:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:20:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:20:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:20:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:20:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:20:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:20:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:20:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:20:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:20:28Z" 2023-10-18T20:20:28.689 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:20:28.690 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660423.211791_873.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:20:23.211791Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:20:28Z" } 2023-10-18T20:20:28.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660423.211791_873.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:20:23.211791Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:20:28.691 DEBUG:rgw_multi.tests:log_status={0: '1_1697660423.211791_873.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:20:28.691 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660423.211791_873.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:20:28.692 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:20:28.692 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-54 2023-10-18T20:20:28.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:20:28.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:20:28.694 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:20:28.751 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:20:28.751 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:20:28.784 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:28.783+0000 7f8ef30cb780 1 waiting to reach incremental sync.. 2023-10-18T20:20:33.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:33.784+0000 7f8ef30cb780 1 waiting to reach incremental sync.. 2023-10-18T20:20:38.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:38.785+0000 7f8ef30cb780 1 bucket sync caught up with source: 2023-10-18T20:20:38.787 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.27.6, , , , 00000000001.43.6, , , 00000000001.42.6, , 00000000001.32.6] 2023-10-18T20:20:38.788 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.27.6, , , , 00000000001.43.6, , , 00000000001.42.6, , 00000000001.32.6] 2023-10-18T20:20:38.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:38.785+0000 7f8ef30cb780 0 bucket checkpoint complete 2023-10-18T20:20:38.803 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-54 2023-10-18T20:20:38.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:20:38.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:20:38.805 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-54 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:20:38.866 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:20:38.867 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:20:38.905 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:38.904+0000 7fa991784780 1 waiting to reach incremental sync.. 2023-10-18T20:20:43.906 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:43.905+0000 7fa991784780 1 waiting to reach incremental sync.. 2023-10-18T20:20:48.907 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:48.905+0000 7fa991784780 1 waiting to reach incremental sync.. 2023-10-18T20:20:53.907 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:53.905+0000 7fa991784780 1 waiting to reach incremental sync.. 2023-10-18T20:20:58.909 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:58.906+0000 7fa991784780 1 bucket sync caught up with source: 2023-10-18T20:20:58.909 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.290.6, , , , 00000000001.701.6, , , 00000000001.937.6, , 00000000001.215.6] 2023-10-18T20:20:58.910 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.290.6, , , , 00000000001.701.6, , , 00000000001.937.6, , 00000000001.215.6] 2023-10-18T20:20:58.910 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:20:58.906+0000 7fa991784780 0 bucket checkpoint complete 2023-10-18T20:20:58.926 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-54 zones={a2, a1} 2023-10-18T20:20:58.926 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:58.927 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:58.927 DEBUG:boto:Method: HEAD 2023-10-18T20:20:58.928 DEBUG:boto:Path: /bofrnf-54/ 2023-10-18T20:20:58.928 DEBUG:boto:Data: 2023-10-18T20:20:58.929 DEBUG:boto:Headers: {} 2023-10-18T20:20:58.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:58.929 DEBUG:boto:Port: 8001 2023-10-18T20:20:58.930 DEBUG:boto:Params: {} 2023-10-18T20:20:58.930 DEBUG:boto:Token: None 2023-10-18T20:20:58.931 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:58 GMT /bofrnf-54/ 2023-10-18T20:20:58.931 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+2742wYvI8/2b/rKtSOywKiJbjo= 2023-10-18T20:20:58.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+2742wYvI8/2b/rKtSOywKiJbjo=', 'Content-Length': '0'} 2023-10-18T20:20:58.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000762c8c187492ca0a-0065303e2a-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:20:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:58.935 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:58.935 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:58.936 DEBUG:boto:Method: HEAD 2023-10-18T20:20:58.936 DEBUG:boto:Path: /bofrnf-54/ 2023-10-18T20:20:58.937 DEBUG:boto:Data: 2023-10-18T20:20:58.937 DEBUG:boto:Headers: {} 2023-10-18T20:20:58.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:58.938 DEBUG:boto:Port: 8000 2023-10-18T20:20:58.938 DEBUG:boto:Params: {} 2023-10-18T20:20:58.939 DEBUG:boto:Token: None 2023-10-18T20:20:58.939 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:58 GMT /bofrnf-54/ 2023-10-18T20:20:58.940 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+2742wYvI8/2b/rKtSOywKiJbjo= 2023-10-18T20:20:58.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+2742wYvI8/2b/rKtSOywKiJbjo=', 'Content-Length': '0'} 2023-10-18T20:20:58.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002da1a6ef692781d5-0065303e2a-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:20:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:58.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:20:58.943 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:58.944 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:58.944 DEBUG:boto:path=/bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.945 DEBUG:boto:auth_path=/bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.945 DEBUG:boto:Method: GET 2023-10-18T20:20:58.946 DEBUG:boto:Path: /bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.946 DEBUG:boto:Data: 2023-10-18T20:20:58.946 DEBUG:boto:Headers: {} 2023-10-18T20:20:58.947 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:58.947 DEBUG:boto:Port: 8001 2023-10-18T20:20:58.948 DEBUG:boto:Params: {} 2023-10-18T20:20:58.948 DEBUG:boto:Token: None 2023-10-18T20:20:58.949 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:58 GMT /bofrnf-54/?versions 2023-10-18T20:20:58.949 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHvQs9WuKPNzKZGxGURLRBGJ2S8= 2023-10-18T20:20:58.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHvQs9WuKPNzKZGxGURLRBGJ2S8=', 'Content-Length': '0'} 2023-10-18T20:20:58.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e21bfc9dae5f5817-0065303e2a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:58.953 DEBUG:boto:b'bofrnf-54999falseanulltrue2023-10-18T20:20:23.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:20:23.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:20:23.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:20:23.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:58.954 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:20:58.955 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:20:58.955 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:20:58.955 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:20:58.956 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:20:58.956 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:58.956 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:58.957 DEBUG:boto:path=/bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.957 DEBUG:boto:auth_path=/bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.957 DEBUG:boto:Method: GET 2023-10-18T20:20:58.958 DEBUG:boto:Path: /bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.958 DEBUG:boto:Data: 2023-10-18T20:20:58.959 DEBUG:boto:Headers: {} 2023-10-18T20:20:58.959 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:58.959 DEBUG:boto:Port: 8000 2023-10-18T20:20:58.960 DEBUG:boto:Params: {} 2023-10-18T20:20:58.960 DEBUG:boto:Token: None 2023-10-18T20:20:58.961 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:58 GMT /bofrnf-54/?versions 2023-10-18T20:20:58.961 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHvQs9WuKPNzKZGxGURLRBGJ2S8= 2023-10-18T20:20:58.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHvQs9WuKPNzKZGxGURLRBGJ2S8=', 'Content-Length': '0'} 2023-10-18T20:20:58.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b03f37ef4c015b9-0065303e2a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:58.965 DEBUG:boto:b'bofrnf-54999falseanulltrue2023-10-18T20:20:23.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:20:23.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:20:23.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:20:23.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:58.965 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:20:58.966 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:20:58.966 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:20:58.967 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:20:58.967 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:58.967 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:58.968 DEBUG:boto:path=/bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.968 DEBUG:boto:auth_path=/bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.968 DEBUG:boto:Method: GET 2023-10-18T20:20:58.969 DEBUG:boto:Path: /bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.969 DEBUG:boto:Data: 2023-10-18T20:20:58.970 DEBUG:boto:Headers: {} 2023-10-18T20:20:58.970 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:58.970 DEBUG:boto:Port: 8001 2023-10-18T20:20:58.971 DEBUG:boto:Params: {} 2023-10-18T20:20:58.971 DEBUG:boto:Token: None 2023-10-18T20:20:58.972 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:58 GMT /bofrnf-54/?versions 2023-10-18T20:20:58.972 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHvQs9WuKPNzKZGxGURLRBGJ2S8= 2023-10-18T20:20:58.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHvQs9WuKPNzKZGxGURLRBGJ2S8=', 'Content-Length': '0'} 2023-10-18T20:20:58.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6af7ea97c8f1841-0065303e2a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:58.976 DEBUG:boto:b'bofrnf-54999falseanulltrue2023-10-18T20:20:23.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:20:23.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:20:23.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:20:23.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:58.976 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:58.977 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:58.977 DEBUG:boto:path=/bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.977 DEBUG:boto:auth_path=/bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.978 DEBUG:boto:Method: GET 2023-10-18T20:20:58.978 DEBUG:boto:Path: /bofrnf-54/?versions&max-keys=999 2023-10-18T20:20:58.979 DEBUG:boto:Data: 2023-10-18T20:20:58.979 DEBUG:boto:Headers: {} 2023-10-18T20:20:58.980 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:58.980 DEBUG:boto:Port: 8000 2023-10-18T20:20:58.980 DEBUG:boto:Params: {} 2023-10-18T20:20:58.981 DEBUG:boto:Token: None 2023-10-18T20:20:58.981 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:58 GMT /bofrnf-54/?versions 2023-10-18T20:20:58.982 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHvQs9WuKPNzKZGxGURLRBGJ2S8= 2023-10-18T20:20:58.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHvQs9WuKPNzKZGxGURLRBGJ2S8=', 'Content-Length': '0'} 2023-10-18T20:20:58.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9d917056f8bd79f-0065303e2a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:58.985 DEBUG:boto:b'bofrnf-54999falseanulltrue2023-10-18T20:20:23.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:20:23.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:20:23.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:20:23.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:58.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:20:58.986 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:58.987 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:58.987 DEBUG:boto:path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:58.987 DEBUG:boto:auth_path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:58.988 DEBUG:boto:Method: GET 2023-10-18T20:20:58.988 DEBUG:boto:Path: /bofrnf-54/a?versionId=null 2023-10-18T20:20:58.989 DEBUG:boto:Data: 2023-10-18T20:20:58.989 DEBUG:boto:Headers: {} 2023-10-18T20:20:58.989 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:58.990 DEBUG:boto:Port: 8001 2023-10-18T20:20:58.990 DEBUG:boto:Params: {} 2023-10-18T20:20:58.991 DEBUG:boto:Token: None 2023-10-18T20:20:58.991 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:58 GMT /bofrnf-54/a?versionId=null 2023-10-18T20:20:58.992 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YFKF/nN8L+fbhpw2wMrvlFMnULY= 2023-10-18T20:20:58.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YFKF/nN8L+fbhpw2wMrvlFMnULY=', 'Content-Length': '0'} 2023-10-18T20:20:58.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000309f263b7e9234de-0065303e2a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.038 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:59.039 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:59.039 DEBUG:boto:path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.040 DEBUG:boto:auth_path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.040 DEBUG:boto:Method: GET 2023-10-18T20:20:59.040 DEBUG:boto:Path: /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.041 DEBUG:boto:Data: 2023-10-18T20:20:59.041 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.041 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.042 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.042 DEBUG:boto:Params: {} 2023-10-18T20:20:59.043 DEBUG:boto:Token: None 2023-10-18T20:20:59.043 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNJ7raQQLtLbwZB9W5ZG2QmOg5o= 2023-10-18T20:20:59.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNJ7raQQLtLbwZB9W5ZG2QmOg5o=', 'Content-Length': '0'} 2023-10-18T20:20:59.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035527c0c1f52b975-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.047 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:59.047 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:59.047 DEBUG:boto:path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.048 DEBUG:boto:auth_path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.048 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.048 DEBUG:boto:Path: /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.049 DEBUG:boto:Data: 2023-10-18T20:20:59.049 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.050 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.050 DEBUG:boto:Params: {} 2023-10-18T20:20:59.051 DEBUG:boto:Token: None 2023-10-18T20:20:59.051 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tDZPctNapRFAKcjdMUm115ombnY= 2023-10-18T20:20:59.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tDZPctNapRFAKcjdMUm115ombnY=', 'Content-Length': '0'} 2023-10-18T20:20:59.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e5e5c3c3091663-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.055 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:59.055 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:59.055 DEBUG:boto:path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.056 DEBUG:boto:auth_path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.056 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.057 DEBUG:boto:Path: /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.057 DEBUG:boto:Data: 2023-10-18T20:20:59.057 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.058 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.058 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.058 DEBUG:boto:Params: {} 2023-10-18T20:20:59.059 DEBUG:boto:Token: None 2023-10-18T20:20:59.059 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.060 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tDZPctNapRFAKcjdMUm115ombnY= 2023-10-18T20:20:59.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tDZPctNapRFAKcjdMUm115ombnY=', 'Content-Length': '0'} 2023-10-18T20:20:59.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070bf71470e4fcf6a-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:20:59.063 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:59.063 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:59.064 DEBUG:boto:path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.064 DEBUG:boto:auth_path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.064 DEBUG:boto:Method: GET 2023-10-18T20:20:59.065 DEBUG:boto:Path: /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.065 DEBUG:boto:Data: 2023-10-18T20:20:59.065 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.066 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.067 DEBUG:boto:Params: {} 2023-10-18T20:20:59.067 DEBUG:boto:Token: None 2023-10-18T20:20:59.067 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.068 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNJ7raQQLtLbwZB9W5ZG2QmOg5o= 2023-10-18T20:20:59.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNJ7raQQLtLbwZB9W5ZG2QmOg5o=', 'Content-Length': '0'} 2023-10-18T20:20:59.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c01732ff65d0903b-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.071 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:59.071 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:59.072 DEBUG:boto:path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.072 DEBUG:boto:auth_path=/bofrnf-54/a?versionId=null 2023-10-18T20:20:59.073 DEBUG:boto:Method: GET 2023-10-18T20:20:59.073 DEBUG:boto:Path: /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.073 DEBUG:boto:Data: 2023-10-18T20:20:59.074 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.074 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.074 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.075 DEBUG:boto:Params: {} 2023-10-18T20:20:59.075 DEBUG:boto:Token: None 2023-10-18T20:20:59.076 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/a?versionId=null 2023-10-18T20:20:59.076 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNJ7raQQLtLbwZB9W5ZG2QmOg5o= 2023-10-18T20:20:59.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNJ7raQQLtLbwZB9W5ZG2QmOg5o=', 'Content-Length': '0'} 2023-10-18T20:20:59.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399e9fa2538b0adf-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.079 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.080 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.080 DEBUG:boto:path=/bofrnf-54/?prefix=a 2023-10-18T20:20:59.081 DEBUG:boto:auth_path=/bofrnf-54/?prefix=a 2023-10-18T20:20:59.081 DEBUG:boto:Method: GET 2023-10-18T20:20:59.081 DEBUG:boto:Path: /bofrnf-54/?prefix=a 2023-10-18T20:20:59.082 DEBUG:boto:Data: 2023-10-18T20:20:59.082 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.083 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.083 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.083 DEBUG:boto:Params: {} 2023-10-18T20:20:59.084 DEBUG:boto:Token: None 2023-10-18T20:20:59.084 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042fbdc01be63a83c-0065303e2b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.130 DEBUG:boto:b'bofrnf-54a1000falsea2023-10-18T20:20:23.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.131 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.131 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.132 DEBUG:boto:path=/bofrnf-54/?prefix=a 2023-10-18T20:20:59.132 DEBUG:boto:auth_path=/bofrnf-54/?prefix=a 2023-10-18T20:20:59.132 DEBUG:boto:Method: GET 2023-10-18T20:20:59.133 DEBUG:boto:Path: /bofrnf-54/?prefix=a 2023-10-18T20:20:59.133 DEBUG:boto:Data: 2023-10-18T20:20:59.134 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.134 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.134 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.135 DEBUG:boto:Params: {} 2023-10-18T20:20:59.135 DEBUG:boto:Token: None 2023-10-18T20:20:59.136 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.136 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044f1a76c9688382d-0065303e2b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.140 DEBUG:boto:b'bofrnf-54a1000falsea2023-10-18T20:20:23.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.140 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:59.140 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:59.141 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.141 DEBUG:boto:Path: /bofrnf-54/a 2023-10-18T20:20:59.142 DEBUG:boto:Data: 2023-10-18T20:20:59.142 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.143 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.143 DEBUG:boto:Params: {} 2023-10-18T20:20:59.145 DEBUG:boto:Token: None 2023-10-18T20:20:59.145 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/a 2023-10-18T20:20:59.146 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yIx9FwsgolGLPcTHZw2camQpzwo= 2023-10-18T20:20:59.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yIx9FwsgolGLPcTHZw2camQpzwo=', 'Content-Length': '0'} 2023-10-18T20:20:59.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5a04e462867cd6f-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.149 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:59.149 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:59.149 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.150 DEBUG:boto:Path: /bofrnf-54/a 2023-10-18T20:20:59.150 DEBUG:boto:Data: 2023-10-18T20:20:59.151 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.151 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.152 DEBUG:boto:Params: {} 2023-10-18T20:20:59.152 DEBUG:boto:Token: None 2023-10-18T20:20:59.153 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/a 2023-10-18T20:20:59.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yIx9FwsgolGLPcTHZw2camQpzwo= 2023-10-18T20:20:59.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yIx9FwsgolGLPcTHZw2camQpzwo=', 'Content-Length': '0'} 2023-10-18T20:20:59.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d19cf094e67d3191-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:20:59.156 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:59.157 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:59.157 DEBUG:boto:Method: GET 2023-10-18T20:20:59.157 DEBUG:boto:Path: /bofrnf-54/a 2023-10-18T20:20:59.158 DEBUG:boto:Data: 2023-10-18T20:20:59.158 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.159 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.159 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.159 DEBUG:boto:Params: {} 2023-10-18T20:20:59.160 DEBUG:boto:Token: None 2023-10-18T20:20:59.160 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/a 2023-10-18T20:20:59.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zG2dtcjCv9A4CDaytZstfKf88f0= 2023-10-18T20:20:59.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zG2dtcjCv9A4CDaytZstfKf88f0=', 'Content-Length': '0'} 2023-10-18T20:20:59.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fcec07c9e440dd2-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.164 DEBUG:boto:path=/bofrnf-54/a 2023-10-18T20:20:59.164 DEBUG:boto:auth_path=/bofrnf-54/a 2023-10-18T20:20:59.164 DEBUG:boto:Method: GET 2023-10-18T20:20:59.165 DEBUG:boto:Path: /bofrnf-54/a 2023-10-18T20:20:59.165 DEBUG:boto:Data: 2023-10-18T20:20:59.166 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.166 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.167 DEBUG:boto:Params: {} 2023-10-18T20:20:59.167 DEBUG:boto:Token: None 2023-10-18T20:20:59.168 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/a 2023-10-18T20:20:59.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zG2dtcjCv9A4CDaytZstfKf88f0= 2023-10-18T20:20:59.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zG2dtcjCv9A4CDaytZstfKf88f0=', 'Content-Length': '0'} 2023-10-18T20:20:59.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14f5d0e5d9214c1-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.214 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.215 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.215 DEBUG:boto:path=/bofrnf-54/?prefix=a 2023-10-18T20:20:59.216 DEBUG:boto:auth_path=/bofrnf-54/?prefix=a 2023-10-18T20:20:59.216 DEBUG:boto:Method: GET 2023-10-18T20:20:59.216 DEBUG:boto:Path: /bofrnf-54/?prefix=a 2023-10-18T20:20:59.217 DEBUG:boto:Data: 2023-10-18T20:20:59.217 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.218 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.218 DEBUG:boto:Params: {} 2023-10-18T20:20:59.219 DEBUG:boto:Token: None 2023-10-18T20:20:59.219 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbfd472712a2e800-0065303e2b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.223 DEBUG:boto:b'bofrnf-54a1000falsea2023-10-18T20:20:23.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.224 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.224 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.225 DEBUG:boto:path=/bofrnf-54/?prefix=a 2023-10-18T20:20:59.225 DEBUG:boto:auth_path=/bofrnf-54/?prefix=a 2023-10-18T20:20:59.225 DEBUG:boto:Method: GET 2023-10-18T20:20:59.226 DEBUG:boto:Path: /bofrnf-54/?prefix=a 2023-10-18T20:20:59.226 DEBUG:boto:Data: 2023-10-18T20:20:59.227 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.227 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.227 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.228 DEBUG:boto:Params: {} 2023-10-18T20:20:59.228 DEBUG:boto:Token: None 2023-10-18T20:20:59.228 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.229 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4ff0c615513461f-0065303e2b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.233 DEBUG:boto:b'bofrnf-54a1000falsea2023-10-18T20:20:23.228Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:20:59.234 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.234 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.235 DEBUG:boto:path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.235 DEBUG:boto:auth_path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.235 DEBUG:boto:Method: GET 2023-10-18T20:20:59.236 DEBUG:boto:Path: /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.236 DEBUG:boto:Data: 2023-10-18T20:20:59.237 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.237 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.237 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.238 DEBUG:boto:Params: {} 2023-10-18T20:20:59.238 DEBUG:boto:Token: None 2023-10-18T20:20:59.238 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.239 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G07c4xGQSzOgvNR4AxTLQPQR9x4= 2023-10-18T20:20:59.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G07c4xGQSzOgvNR4AxTLQPQR9x4=', 'Content-Length': '0'} 2023-10-18T20:20:59.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000981a57b1f27fd703-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.242 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.243 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.243 DEBUG:boto:path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.243 DEBUG:boto:auth_path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.244 DEBUG:boto:Method: GET 2023-10-18T20:20:59.244 DEBUG:boto:Path: /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.245 DEBUG:boto:Data: 2023-10-18T20:20:59.245 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.245 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.246 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.246 DEBUG:boto:Params: {} 2023-10-18T20:20:59.247 DEBUG:boto:Token: None 2023-10-18T20:20:59.247 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.247 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G07c4xGQSzOgvNR4AxTLQPQR9x4= 2023-10-18T20:20:59.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G07c4xGQSzOgvNR4AxTLQPQR9x4=', 'Content-Length': '0'} 2023-10-18T20:20:59.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036e99f81ec285d33-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.251 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.251 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.251 DEBUG:boto:path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.252 DEBUG:boto:auth_path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.252 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.253 DEBUG:boto:Path: /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.253 DEBUG:boto:Data: 2023-10-18T20:20:59.253 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.254 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.254 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.255 DEBUG:boto:Params: {} 2023-10-18T20:20:59.255 DEBUG:boto:Token: None 2023-10-18T20:20:59.255 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.256 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F5iUamVJo1bYw7ZMe2wDkA+GnU4= 2023-10-18T20:20:59.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F5iUamVJo1bYw7ZMe2wDkA+GnU4=', 'Content-Length': '0'} 2023-10-18T20:20:59.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bb55d78b1430295-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.259 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.259 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.260 DEBUG:boto:path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.260 DEBUG:boto:auth_path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.261 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.261 DEBUG:boto:Path: /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.261 DEBUG:boto:Data: 2023-10-18T20:20:59.262 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.262 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.262 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.263 DEBUG:boto:Params: {} 2023-10-18T20:20:59.263 DEBUG:boto:Token: None 2023-10-18T20:20:59.264 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.264 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F5iUamVJo1bYw7ZMe2wDkA+GnU4= 2023-10-18T20:20:59.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F5iUamVJo1bYw7ZMe2wDkA+GnU4=', 'Content-Length': '0'} 2023-10-18T20:20:59.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d17259ee0f2f51a-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:20:59.267 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.268 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.268 DEBUG:boto:path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.269 DEBUG:boto:auth_path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.269 DEBUG:boto:Method: GET 2023-10-18T20:20:59.269 DEBUG:boto:Path: /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.270 DEBUG:boto:Data: 2023-10-18T20:20:59.270 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.271 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.271 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.272 DEBUG:boto:Params: {} 2023-10-18T20:20:59.272 DEBUG:boto:Token: None 2023-10-18T20:20:59.272 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.273 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G07c4xGQSzOgvNR4AxTLQPQR9x4= 2023-10-18T20:20:59.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G07c4xGQSzOgvNR4AxTLQPQR9x4=', 'Content-Length': '0'} 2023-10-18T20:20:59.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b841344da09fd2bc-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.318 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.319 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.319 DEBUG:boto:path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.320 DEBUG:boto:auth_path=/bofrnf-54/b?versionId=null 2023-10-18T20:20:59.320 DEBUG:boto:Method: GET 2023-10-18T20:20:59.320 DEBUG:boto:Path: /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.321 DEBUG:boto:Data: 2023-10-18T20:20:59.321 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.321 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.322 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.322 DEBUG:boto:Params: {} 2023-10-18T20:20:59.323 DEBUG:boto:Token: None 2023-10-18T20:20:59.323 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b?versionId=null 2023-10-18T20:20:59.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G07c4xGQSzOgvNR4AxTLQPQR9x4= 2023-10-18T20:20:59.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G07c4xGQSzOgvNR4AxTLQPQR9x4=', 'Content-Length': '0'} 2023-10-18T20:20:59.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051fc3997446286fc-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.327 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.327 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.328 DEBUG:boto:path=/bofrnf-54/?prefix=b 2023-10-18T20:20:59.328 DEBUG:boto:auth_path=/bofrnf-54/?prefix=b 2023-10-18T20:20:59.328 DEBUG:boto:Method: GET 2023-10-18T20:20:59.329 DEBUG:boto:Path: /bofrnf-54/?prefix=b 2023-10-18T20:20:59.329 DEBUG:boto:Data: 2023-10-18T20:20:59.330 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.330 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.330 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.331 DEBUG:boto:Params: {} 2023-10-18T20:20:59.331 DEBUG:boto:Token: None 2023-10-18T20:20:59.332 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.332 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080c3cdbfeae4c779-0065303e2b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.336 DEBUG:boto:b'bofrnf-54b1000falseb2023-10-18T20:20:23.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.337 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.337 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.337 DEBUG:boto:path=/bofrnf-54/?prefix=b 2023-10-18T20:20:59.338 DEBUG:boto:auth_path=/bofrnf-54/?prefix=b 2023-10-18T20:20:59.338 DEBUG:boto:Method: GET 2023-10-18T20:20:59.338 DEBUG:boto:Path: /bofrnf-54/?prefix=b 2023-10-18T20:20:59.339 DEBUG:boto:Data: 2023-10-18T20:20:59.339 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.340 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.340 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.340 DEBUG:boto:Params: {} 2023-10-18T20:20:59.341 DEBUG:boto:Token: None 2023-10-18T20:20:59.341 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.342 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000879ec72e73d4b598-0065303e2b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.386 DEBUG:boto:b'bofrnf-54b1000falseb2023-10-18T20:20:23.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.387 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.387 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.388 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.388 DEBUG:boto:Path: /bofrnf-54/b 2023-10-18T20:20:59.389 DEBUG:boto:Data: 2023-10-18T20:20:59.389 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.390 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.390 DEBUG:boto:Params: {} 2023-10-18T20:20:59.391 DEBUG:boto:Token: None 2023-10-18T20:20:59.391 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b 2023-10-18T20:20:59.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pU318OylboVKOwRWWgGITWGIQRc= 2023-10-18T20:20:59.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pU318OylboVKOwRWWgGITWGIQRc=', 'Content-Length': '0'} 2023-10-18T20:20:59.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e03abc1adf3fa74b-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.395 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.395 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.396 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.396 DEBUG:boto:Path: /bofrnf-54/b 2023-10-18T20:20:59.396 DEBUG:boto:Data: 2023-10-18T20:20:59.397 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.397 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.398 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.398 DEBUG:boto:Params: {} 2023-10-18T20:20:59.398 DEBUG:boto:Token: None 2023-10-18T20:20:59.399 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b 2023-10-18T20:20:59.399 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pU318OylboVKOwRWWgGITWGIQRc= 2023-10-18T20:20:59.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pU318OylboVKOwRWWgGITWGIQRc=', 'Content-Length': '0'} 2023-10-18T20:20:59.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc51efebdd78f32b-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:20:59.403 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.403 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.404 DEBUG:boto:Method: GET 2023-10-18T20:20:59.404 DEBUG:boto:Path: /bofrnf-54/b 2023-10-18T20:20:59.404 DEBUG:boto:Data: 2023-10-18T20:20:59.405 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.405 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.406 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.406 DEBUG:boto:Params: {} 2023-10-18T20:20:59.407 DEBUG:boto:Token: None 2023-10-18T20:20:59.407 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b 2023-10-18T20:20:59.407 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:21rV2hvuCUOid2ovSXToiuj9PEI= 2023-10-18T20:20:59.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:21rV2hvuCUOid2ovSXToiuj9PEI=', 'Content-Length': '0'} 2023-10-18T20:20:59.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000481bbe6f0a7929a5-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.411 DEBUG:boto:path=/bofrnf-54/b 2023-10-18T20:20:59.411 DEBUG:boto:auth_path=/bofrnf-54/b 2023-10-18T20:20:59.412 DEBUG:boto:Method: GET 2023-10-18T20:20:59.412 DEBUG:boto:Path: /bofrnf-54/b 2023-10-18T20:20:59.412 DEBUG:boto:Data: 2023-10-18T20:20:59.413 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.413 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.414 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.414 DEBUG:boto:Params: {} 2023-10-18T20:20:59.415 DEBUG:boto:Token: None 2023-10-18T20:20:59.415 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/b 2023-10-18T20:20:59.416 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:21rV2hvuCUOid2ovSXToiuj9PEI= 2023-10-18T20:20:59.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:21rV2hvuCUOid2ovSXToiuj9PEI=', 'Content-Length': '0'} 2023-10-18T20:20:59.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094de91ac6ce3c59b-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.419 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.419 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.420 DEBUG:boto:path=/bofrnf-54/?prefix=b 2023-10-18T20:20:59.420 DEBUG:boto:auth_path=/bofrnf-54/?prefix=b 2023-10-18T20:20:59.421 DEBUG:boto:Method: GET 2023-10-18T20:20:59.421 DEBUG:boto:Path: /bofrnf-54/?prefix=b 2023-10-18T20:20:59.421 DEBUG:boto:Data: 2023-10-18T20:20:59.422 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.422 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.423 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.423 DEBUG:boto:Params: {} 2023-10-18T20:20:59.424 DEBUG:boto:Token: None 2023-10-18T20:20:59.424 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.424 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065d9bc745c61863e-0065303e2b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.470 DEBUG:boto:b'bofrnf-54b1000falseb2023-10-18T20:20:23.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.471 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.471 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.472 DEBUG:boto:path=/bofrnf-54/?prefix=b 2023-10-18T20:20:59.472 DEBUG:boto:auth_path=/bofrnf-54/?prefix=b 2023-10-18T20:20:59.473 DEBUG:boto:Method: GET 2023-10-18T20:20:59.473 DEBUG:boto:Path: /bofrnf-54/?prefix=b 2023-10-18T20:20:59.474 DEBUG:boto:Data: 2023-10-18T20:20:59.474 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.475 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.475 DEBUG:boto:Params: {} 2023-10-18T20:20:59.476 DEBUG:boto:Token: None 2023-10-18T20:20:59.476 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.477 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba586dd81dc8566a-0065303e2b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.481 DEBUG:boto:b'bofrnf-54b1000falseb2023-10-18T20:20:23.246Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:20:59.482 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.482 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.483 DEBUG:boto:path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.483 DEBUG:boto:auth_path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.483 DEBUG:boto:Method: GET 2023-10-18T20:20:59.484 DEBUG:boto:Path: /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.484 DEBUG:boto:Data: 2023-10-18T20:20:59.485 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.485 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.486 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.486 DEBUG:boto:Params: {} 2023-10-18T20:20:59.487 DEBUG:boto:Token: None 2023-10-18T20:20:59.487 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.488 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QbPW4U65704+d2ISt08Ny3p3WjI= 2023-10-18T20:20:59.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QbPW4U65704+d2ISt08Ny3p3WjI=', 'Content-Length': '0'} 2023-10-18T20:20:59.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019908191d6138613-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.491 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.491 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.492 DEBUG:boto:path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.492 DEBUG:boto:auth_path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.493 DEBUG:boto:Method: GET 2023-10-18T20:20:59.493 DEBUG:boto:Path: /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.494 DEBUG:boto:Data: 2023-10-18T20:20:59.494 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.495 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.495 DEBUG:boto:Params: {} 2023-10-18T20:20:59.496 DEBUG:boto:Token: None 2023-10-18T20:20:59.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QbPW4U65704+d2ISt08Ny3p3WjI= 2023-10-18T20:20:59.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QbPW4U65704+d2ISt08Ny3p3WjI=', 'Content-Length': '0'} 2023-10-18T20:20:59.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eee10447323f58af-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.500 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.500 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.501 DEBUG:boto:path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.501 DEBUG:boto:auth_path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.501 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.502 DEBUG:boto:Path: /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.502 DEBUG:boto:Data: 2023-10-18T20:20:59.503 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.503 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.504 DEBUG:boto:Params: {} 2023-10-18T20:20:59.504 DEBUG:boto:Token: None 2023-10-18T20:20:59.505 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vJhXH2dO+NgCXwqykpXSA00y7qg= 2023-10-18T20:20:59.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vJhXH2dO+NgCXwqykpXSA00y7qg=', 'Content-Length': '0'} 2023-10-18T20:20:59.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016979c6a9cde0f1b-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.508 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.508 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.509 DEBUG:boto:path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.509 DEBUG:boto:auth_path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.510 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.510 DEBUG:boto:Path: /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.510 DEBUG:boto:Data: 2023-10-18T20:20:59.511 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.511 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.512 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.512 DEBUG:boto:Params: {} 2023-10-18T20:20:59.512 DEBUG:boto:Token: None 2023-10-18T20:20:59.513 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.513 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vJhXH2dO+NgCXwqykpXSA00y7qg= 2023-10-18T20:20:59.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vJhXH2dO+NgCXwqykpXSA00y7qg=', 'Content-Length': '0'} 2023-10-18T20:20:59.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040226b02a6b67e9a-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:20:59.516 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.517 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.517 DEBUG:boto:path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.518 DEBUG:boto:auth_path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.518 DEBUG:boto:Method: GET 2023-10-18T20:20:59.518 DEBUG:boto:Path: /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.519 DEBUG:boto:Data: 2023-10-18T20:20:59.519 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.520 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.520 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.521 DEBUG:boto:Params: {} 2023-10-18T20:20:59.521 DEBUG:boto:Token: None 2023-10-18T20:20:59.521 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.522 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QbPW4U65704+d2ISt08Ny3p3WjI= 2023-10-18T20:20:59.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QbPW4U65704+d2ISt08Ny3p3WjI=', 'Content-Length': '0'} 2023-10-18T20:20:59.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f43c0dc55484f9c-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.525 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.525 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.526 DEBUG:boto:path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.526 DEBUG:boto:auth_path=/bofrnf-54/c?versionId=null 2023-10-18T20:20:59.527 DEBUG:boto:Method: GET 2023-10-18T20:20:59.527 DEBUG:boto:Path: /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.527 DEBUG:boto:Data: 2023-10-18T20:20:59.528 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.528 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.529 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.529 DEBUG:boto:Params: {} 2023-10-18T20:20:59.529 DEBUG:boto:Token: None 2023-10-18T20:20:59.530 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c?versionId=null 2023-10-18T20:20:59.530 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QbPW4U65704+d2ISt08Ny3p3WjI= 2023-10-18T20:20:59.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QbPW4U65704+d2ISt08Ny3p3WjI=', 'Content-Length': '0'} 2023-10-18T20:20:59.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055b0bac7b4abb44e-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.534 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.534 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.534 DEBUG:boto:path=/bofrnf-54/?prefix=c 2023-10-18T20:20:59.535 DEBUG:boto:auth_path=/bofrnf-54/?prefix=c 2023-10-18T20:20:59.535 DEBUG:boto:Method: GET 2023-10-18T20:20:59.536 DEBUG:boto:Path: /bofrnf-54/?prefix=c 2023-10-18T20:20:59.536 DEBUG:boto:Data: 2023-10-18T20:20:59.536 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.537 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.538 DEBUG:boto:Params: {} 2023-10-18T20:20:59.538 DEBUG:boto:Token: None 2023-10-18T20:20:59.539 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.539 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043d604a4203e8f95-0065303e2b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.543 DEBUG:boto:b'bofrnf-54c1000falsec2023-10-18T20:20:23.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.543 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.544 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.544 DEBUG:boto:path=/bofrnf-54/?prefix=c 2023-10-18T20:20:59.545 DEBUG:boto:auth_path=/bofrnf-54/?prefix=c 2023-10-18T20:20:59.545 DEBUG:boto:Method: GET 2023-10-18T20:20:59.545 DEBUG:boto:Path: /bofrnf-54/?prefix=c 2023-10-18T20:20:59.546 DEBUG:boto:Data: 2023-10-18T20:20:59.546 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.547 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.547 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.547 DEBUG:boto:Params: {} 2023-10-18T20:20:59.548 DEBUG:boto:Token: None 2023-10-18T20:20:59.548 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.549 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f60dfac9c763aaea-0065303e2b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.555 DEBUG:boto:b'bofrnf-54c1000falsec2023-10-18T20:20:23.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.556 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.556 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.556 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.557 DEBUG:boto:Path: /bofrnf-54/c 2023-10-18T20:20:59.557 DEBUG:boto:Data: 2023-10-18T20:20:59.557 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.558 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.559 DEBUG:boto:Params: {} 2023-10-18T20:20:59.559 DEBUG:boto:Token: None 2023-10-18T20:20:59.560 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c 2023-10-18T20:20:59.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W+gxrwe9YrGx2CjLI5wLUz9QRlc= 2023-10-18T20:20:59.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W+gxrwe9YrGx2CjLI5wLUz9QRlc=', 'Content-Length': '0'} 2023-10-18T20:20:59.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc73cc3dbb009902-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.563 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.563 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.564 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.564 DEBUG:boto:Path: /bofrnf-54/c 2023-10-18T20:20:59.564 DEBUG:boto:Data: 2023-10-18T20:20:59.565 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.565 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.566 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.566 DEBUG:boto:Params: {} 2023-10-18T20:20:59.567 DEBUG:boto:Token: None 2023-10-18T20:20:59.567 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c 2023-10-18T20:20:59.567 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W+gxrwe9YrGx2CjLI5wLUz9QRlc= 2023-10-18T20:20:59.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W+gxrwe9YrGx2CjLI5wLUz9QRlc=', 'Content-Length': '0'} 2023-10-18T20:20:59.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c400545ed3481859-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:20:59.571 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.571 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.572 DEBUG:boto:Method: GET 2023-10-18T20:20:59.572 DEBUG:boto:Path: /bofrnf-54/c 2023-10-18T20:20:59.572 DEBUG:boto:Data: 2023-10-18T20:20:59.573 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.573 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.574 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.574 DEBUG:boto:Params: {} 2023-10-18T20:20:59.575 DEBUG:boto:Token: None 2023-10-18T20:20:59.575 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c 2023-10-18T20:20:59.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DnWCMHoSYEDobGKRim5+Ced9P38= 2023-10-18T20:20:59.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DnWCMHoSYEDobGKRim5+Ced9P38=', 'Content-Length': '0'} 2023-10-18T20:20:59.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000929f657b7303ea23-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.622 DEBUG:boto:path=/bofrnf-54/c 2023-10-18T20:20:59.623 DEBUG:boto:auth_path=/bofrnf-54/c 2023-10-18T20:20:59.623 DEBUG:boto:Method: GET 2023-10-18T20:20:59.624 DEBUG:boto:Path: /bofrnf-54/c 2023-10-18T20:20:59.624 DEBUG:boto:Data: 2023-10-18T20:20:59.625 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.626 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.626 DEBUG:boto:Params: {} 2023-10-18T20:20:59.626 DEBUG:boto:Token: None 2023-10-18T20:20:59.627 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/c 2023-10-18T20:20:59.627 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DnWCMHoSYEDobGKRim5+Ced9P38= 2023-10-18T20:20:59.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DnWCMHoSYEDobGKRim5+Ced9P38=', 'Content-Length': '0'} 2023-10-18T20:20:59.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002797d75ca5194225-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.631 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.631 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.631 DEBUG:boto:path=/bofrnf-54/?prefix=c 2023-10-18T20:20:59.632 DEBUG:boto:auth_path=/bofrnf-54/?prefix=c 2023-10-18T20:20:59.632 DEBUG:boto:Method: GET 2023-10-18T20:20:59.633 DEBUG:boto:Path: /bofrnf-54/?prefix=c 2023-10-18T20:20:59.633 DEBUG:boto:Data: 2023-10-18T20:20:59.634 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.634 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.634 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.635 DEBUG:boto:Params: {} 2023-10-18T20:20:59.635 DEBUG:boto:Token: None 2023-10-18T20:20:59.636 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.636 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e398ce1afba72065-0065303e2b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.640 DEBUG:boto:b'bofrnf-54c1000falsec2023-10-18T20:20:23.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.641 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.641 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.642 DEBUG:boto:path=/bofrnf-54/?prefix=c 2023-10-18T20:20:59.642 DEBUG:boto:auth_path=/bofrnf-54/?prefix=c 2023-10-18T20:20:59.643 DEBUG:boto:Method: GET 2023-10-18T20:20:59.643 DEBUG:boto:Path: /bofrnf-54/?prefix=c 2023-10-18T20:20:59.643 DEBUG:boto:Data: 2023-10-18T20:20:59.644 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.644 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.645 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.645 DEBUG:boto:Params: {} 2023-10-18T20:20:59.645 DEBUG:boto:Token: None 2023-10-18T20:20:59.646 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.646 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035023498738df740-0065303e2b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.650 DEBUG:boto:b'bofrnf-54c1000falsec2023-10-18T20:20:23.305Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:20:59.651 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.651 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.652 DEBUG:boto:path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.652 DEBUG:boto:auth_path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.653 DEBUG:boto:Method: GET 2023-10-18T20:20:59.653 DEBUG:boto:Path: /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.654 DEBUG:boto:Data: 2023-10-18T20:20:59.654 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.655 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.655 DEBUG:boto:Params: {} 2023-10-18T20:20:59.656 DEBUG:boto:Token: None 2023-10-18T20:20:59.656 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.657 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4KNXi4iSa+3T7WlONjh56j8Qrl0= 2023-10-18T20:20:59.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4KNXi4iSa+3T7WlONjh56j8Qrl0=', 'Content-Length': '0'} 2023-10-18T20:20:59.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090dbe982e811e8ff-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.660 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.660 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.661 DEBUG:boto:path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.661 DEBUG:boto:auth_path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.661 DEBUG:boto:Method: GET 2023-10-18T20:20:59.662 DEBUG:boto:Path: /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.662 DEBUG:boto:Data: 2023-10-18T20:20:59.663 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.663 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.664 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.664 DEBUG:boto:Params: {} 2023-10-18T20:20:59.664 DEBUG:boto:Token: None 2023-10-18T20:20:59.665 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4KNXi4iSa+3T7WlONjh56j8Qrl0= 2023-10-18T20:20:59.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4KNXi4iSa+3T7WlONjh56j8Qrl0=', 'Content-Length': '0'} 2023-10-18T20:20:59.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd6fdbdf3fdc60bb-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.668 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.669 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.669 DEBUG:boto:path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.670 DEBUG:boto:auth_path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.670 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.670 DEBUG:boto:Path: /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.671 DEBUG:boto:Data: 2023-10-18T20:20:59.671 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.672 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.672 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.672 DEBUG:boto:Params: {} 2023-10-18T20:20:59.673 DEBUG:boto:Token: None 2023-10-18T20:20:59.673 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.674 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j/leY8NpLl+FDDR9F2xO9htvfvM= 2023-10-18T20:20:59.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j/leY8NpLl+FDDR9F2xO9htvfvM=', 'Content-Length': '0'} 2023-10-18T20:20:59.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3de8ce5609b822-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.677 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.677 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.677 DEBUG:boto:path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.678 DEBUG:boto:auth_path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.678 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.679 DEBUG:boto:Path: /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.679 DEBUG:boto:Data: 2023-10-18T20:20:59.679 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.680 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.681 DEBUG:boto:Params: {} 2023-10-18T20:20:59.681 DEBUG:boto:Token: None 2023-10-18T20:20:59.681 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.682 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j/leY8NpLl+FDDR9F2xO9htvfvM= 2023-10-18T20:20:59.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j/leY8NpLl+FDDR9F2xO9htvfvM=', 'Content-Length': '0'} 2023-10-18T20:20:59.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000401a8b001cce7659-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:20:59.685 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.686 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.686 DEBUG:boto:path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.687 DEBUG:boto:auth_path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.687 DEBUG:boto:Method: GET 2023-10-18T20:20:59.687 DEBUG:boto:Path: /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.688 DEBUG:boto:Data: 2023-10-18T20:20:59.688 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.689 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.690 DEBUG:boto:Params: {} 2023-10-18T20:20:59.690 DEBUG:boto:Token: None 2023-10-18T20:20:59.691 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.691 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4KNXi4iSa+3T7WlONjh56j8Qrl0= 2023-10-18T20:20:59.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4KNXi4iSa+3T7WlONjh56j8Qrl0=', 'Content-Length': '0'} 2023-10-18T20:20:59.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c32603105c916b48-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.734 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.735 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.735 DEBUG:boto:path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.736 DEBUG:boto:auth_path=/bofrnf-54/d?versionId=null 2023-10-18T20:20:59.736 DEBUG:boto:Method: GET 2023-10-18T20:20:59.737 DEBUG:boto:Path: /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.737 DEBUG:boto:Data: 2023-10-18T20:20:59.737 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.738 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.739 DEBUG:boto:Params: {} 2023-10-18T20:20:59.739 DEBUG:boto:Token: None 2023-10-18T20:20:59.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d?versionId=null 2023-10-18T20:20:59.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4KNXi4iSa+3T7WlONjh56j8Qrl0= 2023-10-18T20:20:59.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4KNXi4iSa+3T7WlONjh56j8Qrl0=', 'Content-Length': '0'} 2023-10-18T20:20:59.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cf957be780b666-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.743 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.744 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.744 DEBUG:boto:path=/bofrnf-54/?prefix=d 2023-10-18T20:20:59.744 DEBUG:boto:auth_path=/bofrnf-54/?prefix=d 2023-10-18T20:20:59.745 DEBUG:boto:Method: GET 2023-10-18T20:20:59.745 DEBUG:boto:Path: /bofrnf-54/?prefix=d 2023-10-18T20:20:59.746 DEBUG:boto:Data: 2023-10-18T20:20:59.746 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.747 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.747 DEBUG:boto:Params: {} 2023-10-18T20:20:59.748 DEBUG:boto:Token: None 2023-10-18T20:20:59.748 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fddf686b3507081e-0065303e2b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.752 DEBUG:boto:b'bofrnf-54d1000falsed2023-10-18T20:20:23.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.753 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.753 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.754 DEBUG:boto:path=/bofrnf-54/?prefix=d 2023-10-18T20:20:59.754 DEBUG:boto:auth_path=/bofrnf-54/?prefix=d 2023-10-18T20:20:59.755 DEBUG:boto:Method: GET 2023-10-18T20:20:59.758 DEBUG:boto:Path: /bofrnf-54/?prefix=d 2023-10-18T20:20:59.758 DEBUG:boto:Data: 2023-10-18T20:20:59.759 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.759 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.760 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.760 DEBUG:boto:Params: {} 2023-10-18T20:20:59.760 DEBUG:boto:Token: None 2023-10-18T20:20:59.761 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.761 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002de0b96ac1d7caa3-0065303e2b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.765 DEBUG:boto:b'bofrnf-54d1000falsed2023-10-18T20:20:23.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.766 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.766 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.767 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.767 DEBUG:boto:Path: /bofrnf-54/d 2023-10-18T20:20:59.768 DEBUG:boto:Data: 2023-10-18T20:20:59.768 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.768 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.769 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.769 DEBUG:boto:Params: {} 2023-10-18T20:20:59.770 DEBUG:boto:Token: None 2023-10-18T20:20:59.770 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d 2023-10-18T20:20:59.771 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g6zgSefLTw4z40lXw6YsmuUED/k= 2023-10-18T20:20:59.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g6zgSefLTw4z40lXw6YsmuUED/k=', 'Content-Length': '0'} 2023-10-18T20:20:59.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000333cdd51f4d9a78e-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.774 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.774 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.774 DEBUG:boto:Method: HEAD 2023-10-18T20:20:59.775 DEBUG:boto:Path: /bofrnf-54/d 2023-10-18T20:20:59.775 DEBUG:boto:Data: 2023-10-18T20:20:59.776 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.776 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.776 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.777 DEBUG:boto:Params: {} 2023-10-18T20:20:59.777 DEBUG:boto:Token: None 2023-10-18T20:20:59.777 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d 2023-10-18T20:20:59.778 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g6zgSefLTw4z40lXw6YsmuUED/k= 2023-10-18T20:20:59.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g6zgSefLTw4z40lXw6YsmuUED/k=', 'Content-Length': '0'} 2023-10-18T20:20:59.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001dc1e572d278db4-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:20:59.781 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.782 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.782 DEBUG:boto:Method: GET 2023-10-18T20:20:59.782 DEBUG:boto:Path: /bofrnf-54/d 2023-10-18T20:20:59.783 DEBUG:boto:Data: 2023-10-18T20:20:59.783 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.784 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.784 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.784 DEBUG:boto:Params: {} 2023-10-18T20:20:59.785 DEBUG:boto:Token: None 2023-10-18T20:20:59.785 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d 2023-10-18T20:20:59.786 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NOTMe4A+FLa1sa6Y9EHci12B5Yw= 2023-10-18T20:20:59.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NOTMe4A+FLa1sa6Y9EHci12B5Yw=', 'Content-Length': '0'} 2023-10-18T20:20:59.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-54:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085f87b3b60e52503-0065303e2b-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.789 DEBUG:boto:path=/bofrnf-54/d 2023-10-18T20:20:59.789 DEBUG:boto:auth_path=/bofrnf-54/d 2023-10-18T20:20:59.790 DEBUG:boto:Method: GET 2023-10-18T20:20:59.790 DEBUG:boto:Path: /bofrnf-54/d 2023-10-18T20:20:59.791 DEBUG:boto:Data: 2023-10-18T20:20:59.791 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.792 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.792 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.792 DEBUG:boto:Params: {} 2023-10-18T20:20:59.793 DEBUG:boto:Token: None 2023-10-18T20:20:59.793 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/d 2023-10-18T20:20:59.794 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NOTMe4A+FLa1sa6Y9EHci12B5Yw= 2023-10-18T20:20:59.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NOTMe4A+FLa1sa6Y9EHci12B5Yw=', 'Content-Length': '0'} 2023-10-18T20:20:59.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:20:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0e779751612bd77-0065303e2b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.797 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.797 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.798 DEBUG:boto:path=/bofrnf-54/?prefix=d 2023-10-18T20:20:59.798 DEBUG:boto:auth_path=/bofrnf-54/?prefix=d 2023-10-18T20:20:59.799 DEBUG:boto:Method: GET 2023-10-18T20:20:59.799 DEBUG:boto:Path: /bofrnf-54/?prefix=d 2023-10-18T20:20:59.800 DEBUG:boto:Data: 2023-10-18T20:20:59.800 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.800 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:20:59.801 DEBUG:boto:Port: 8001 2023-10-18T20:20:59.801 DEBUG:boto:Params: {} 2023-10-18T20:20:59.802 DEBUG:boto:Token: None 2023-10-18T20:20:59.802 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.803 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb68efb75f04058f-0065303e2b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.806 DEBUG:boto:b'bofrnf-54d1000falsed2023-10-18T20:20:23.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.807 DEBUG:boto:path=/bofrnf-54/ 2023-10-18T20:20:59.807 DEBUG:boto:auth_path=/bofrnf-54/ 2023-10-18T20:20:59.808 DEBUG:boto:path=/bofrnf-54/?prefix=d 2023-10-18T20:20:59.808 DEBUG:boto:auth_path=/bofrnf-54/?prefix=d 2023-10-18T20:20:59.809 DEBUG:boto:Method: GET 2023-10-18T20:20:59.809 DEBUG:boto:Path: /bofrnf-54/?prefix=d 2023-10-18T20:20:59.810 DEBUG:boto:Data: 2023-10-18T20:20:59.810 DEBUG:boto:Headers: {} 2023-10-18T20:20:59.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:20:59.811 DEBUG:boto:Port: 8000 2023-10-18T20:20:59.811 DEBUG:boto:Params: {} 2023-10-18T20:20:59.812 DEBUG:boto:Token: None 2023-10-18T20:20:59.812 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:20:59 GMT /bofrnf-54/ 2023-10-18T20:20:59.813 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k= 2023-10-18T20:20:59.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:20:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FDtG+w5PQf3+ZzBynpaPLFNvY8k=', 'Content-Length': '0'} 2023-10-18T20:20:59.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000489fa1292d9de1f0-0065303e2b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:20:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:20:59.816 DEBUG:boto:b'bofrnf-54d1000falsed2023-10-18T20:20:23.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:20:59.817 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-54 zones={a2, a1} 2023-10-18T20:20:59.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:20:59.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:20:59.819 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:20:59.872 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:20:59.872 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:20:59.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bofrnf-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:20:59.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bofrnf-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:20:59.927 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bofrnf-54 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:20:59.977 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:20:59.977 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:00.018 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:21:00.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bofrnf-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:00.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bofrnf-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:00.021 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bofrnf-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:21:00.068 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:00.068 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:00.108 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:21:00.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:00.109 INFO:teuthology.orchestra.run.smithi082.stdout: "op_id": "1#00000000001.34.6", 2023-10-18T20:21:00.109 INFO:teuthology.orchestra.run.smithi082.stdout: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.69189234759230606", 2023-10-18T20:21:00.109 INFO:teuthology.orchestra.run.smithi082.stdout: "op": "write", 2023-10-18T20:21:00.110 INFO:teuthology.orchestra.run.smithi082.stdout: "object": "a", 2023-10-18T20:21:00.110 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:21:00.110 INFO:teuthology.orchestra.run.smithi082.stdout: "state": "complete", 2023-10-18T20:21:00.110 INFO:teuthology.orchestra.run.smithi082.stdout: "index_ver": 1, 2023-10-18T20:21:00.111 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:19:41.555381279Z", 2023-10-18T20:21:00.111 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:21:00.111 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:21:00.111 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:21:00.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:00.112 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog_flags": 0, 2023-10-18T20:21:00.112 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned": false, 2023-10-18T20:21:00.112 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "", 2023-10-18T20:21:00.112 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "", 2023-10-18T20:21:00.112 INFO:teuthology.orchestra.run.smithi082.stdout: "zones_trace": [ 2023-10-18T20:21:00.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:00.113 INFO:teuthology.orchestra.run.smithi082.stdout: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22" 2023-10-18T20:21:00.113 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:00.113 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:00.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:00.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:00.114 INFO:teuthology.orchestra.run.smithi082.stdout: "op_id": "5#00000000001.45.6", 2023-10-18T20:21:00.114 INFO:teuthology.orchestra.run.smithi082.stdout: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.6006365997030736602", 2023-10-18T20:21:00.114 INFO:teuthology.orchestra.run.smithi082.stdout: "op": "write", 2023-10-18T20:21:00.114 INFO:teuthology.orchestra.run.smithi082.stdout: "object": "d", 2023-10-18T20:21:00.115 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:21:00.115 INFO:teuthology.orchestra.run.smithi082.stdout: "state": "complete", 2023-10-18T20:21:00.115 INFO:teuthology.orchestra.run.smithi082.stdout: "index_ver": 1, 2023-10-18T20:21:00.115 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:19:41.655394829Z", 2023-10-18T20:21:00.115 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:21:00.116 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:21:00.116 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:21:00.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:00.116 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog_flags": 0, 2023-10-18T20:21:00.116 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned": false, 2023-10-18T20:21:00.117 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "", 2023-10-18T20:21:00.117 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "", 2023-10-18T20:21:00.117 INFO:teuthology.orchestra.run.smithi082.stdout: "zones_trace": [ 2023-10-18T20:21:00.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:00.117 INFO:teuthology.orchestra.run.smithi082.stdout: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22" 2023-10-18T20:21:00.118 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:00.118 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:00.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:00.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:00.118 INFO:teuthology.orchestra.run.smithi082.stdout: "op_id": "8#00000000001.31.6", 2023-10-18T20:21:00.119 INFO:teuthology.orchestra.run.smithi082.stdout: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.6176576623098398315", 2023-10-18T20:21:00.119 INFO:teuthology.orchestra.run.smithi082.stdout: "op": "write", 2023-10-18T20:21:00.119 INFO:teuthology.orchestra.run.smithi082.stdout: "object": "c", 2023-10-18T20:21:00.119 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:21:00.119 INFO:teuthology.orchestra.run.smithi082.stdout: "state": "complete", 2023-10-18T20:21:00.120 INFO:teuthology.orchestra.run.smithi082.stdout: "index_ver": 1, 2023-10-18T20:21:00.120 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:19:41.636157683Z", 2023-10-18T20:21:00.120 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:21:00.120 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:21:00.120 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:21:00.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:00.121 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog_flags": 0, 2023-10-18T20:21:00.121 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned": false, 2023-10-18T20:21:00.121 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "", 2023-10-18T20:21:00.121 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "", 2023-10-18T20:21:00.122 INFO:teuthology.orchestra.run.smithi082.stdout: "zones_trace": [ 2023-10-18T20:21:00.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:00.122 INFO:teuthology.orchestra.run.smithi082.stdout: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22" 2023-10-18T20:21:00.122 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:00.122 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:00.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:00.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:00.123 INFO:teuthology.orchestra.run.smithi082.stdout: "op_id": "10#00000000001.29.6", 2023-10-18T20:21:00.123 INFO:teuthology.orchestra.run.smithi082.stdout: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.8608258815545956834", 2023-10-18T20:21:00.124 INFO:teuthology.orchestra.run.smithi082.stdout: "op": "write", 2023-10-18T20:21:00.124 INFO:teuthology.orchestra.run.smithi082.stdout: "object": "b", 2023-10-18T20:21:00.124 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:21:00.124 INFO:teuthology.orchestra.run.smithi082.stdout: "state": "complete", 2023-10-18T20:21:00.124 INFO:teuthology.orchestra.run.smithi082.stdout: "index_ver": 1, 2023-10-18T20:21:00.125 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:19:41.575308243Z", 2023-10-18T20:21:00.125 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:21:00.125 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:21:00.125 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:21:00.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:00.125 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog_flags": 0, 2023-10-18T20:21:00.126 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned": false, 2023-10-18T20:21:00.126 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "", 2023-10-18T20:21:00.126 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "", 2023-10-18T20:21:00.126 INFO:teuthology.orchestra.run.smithi082.stdout: "zones_trace": [ 2023-10-18T20:21:00.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:00.127 INFO:teuthology.orchestra.run.smithi082.stdout: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-53:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.22" 2023-10-18T20:21:00.127 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:00.127 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:00.127 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:00.127 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:21:00.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:21:00.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:21:00.129 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:21:00.176 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:00.176 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:00.218 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.215+0000 7efcce464700 -1 req 2571241888147741440 0.001999962s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-36:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.3,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-36:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.3,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.218 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.216+0000 7efdb6634700 -1 req 7697100527690261195 0.002999942s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-5:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.5,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-5:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.5,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.219 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.216+0000 7efcdac7d700 -1 req 7681260826343013713 0.002999942s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-24:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.24,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-24:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.24,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.223 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.221+0000 7efdac620700 -1 req 3969924962215699620 0.001999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-41:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.10,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-41:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.10,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.226 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.224+0000 7efdbae3d700 -1 req 6709923874176874845 0.001999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-27:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.27,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-27:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.27,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.269 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.266+0000 7efd94df1700 -1 req 849504133966407505 0.002999942s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-19:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.19,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-19:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.19,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.271 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.269+0000 7efd7c5c0700 -1 req 10158537958131324937 0.001999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-3:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.3,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-3:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.3,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.272 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.269+0000 7efd6f5a6700 -1 req 17802909294422993371 0.001999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-30:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.30,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-30:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.30,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.311 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.309+0000 7efd82dcd700 -1 req 5526694413815538947 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-40:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.9,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-40:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.9,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.315 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.313+0000 7efd7ddc3700 -1 req 842547643406064976 0.001999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-25:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.25,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-25:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.25,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.315 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.313+0000 7efd715aa700 -1 req 14021866128232149241 0.001999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-22:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.22,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-22:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.22,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.362 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.360+0000 7efd5957a700 -1 req 8588574399039958655 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-39:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.8,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-39:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.8,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.405 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.402+0000 7efd5cd81700 -1 req 3016279732967579894 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-37:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.6,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-37:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.6,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.406 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.404+0000 7efd4b55e700 -1 req 16485151074788034248 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-2:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.2,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-2:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.2,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.443 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.441+0000 7efd57576700 -1 req 13923796856404131681 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-29:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.29,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-29:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.29,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.447 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.445+0000 7efd5256c700 -1 req 9674785125467464547 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-23:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.23,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-23:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.23,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.487 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.484+0000 7efd24510700 -1 req 145735477503306859 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-28:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.28,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-28:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.28,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.494 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.492+0000 7efd3252c700 -1 req 2473842477546059970 0.001999962s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-20:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.20,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-20:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.20,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.497 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.494+0000 7efd25512700 -1 req 7102709242633730614 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-4:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.4,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-4:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.4,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.537 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.535+0000 7efd0cce1700 -1 req 7011021831047307778 0.001999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-43:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.12,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-43:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.12,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.537 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.535+0000 7efd0fce7700 -1 req 12144778764653887254 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-38:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.7,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-38:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.7,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.582 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.580+0000 7efd034ce700 -1 req 2860920620881164124 0.001999962s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-6:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.6,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-6:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.6,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.582 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.580+0000 7efcffcc7700 -1 req 11926375473174883292 0.001999962s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-21:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.21,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-21:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.21,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.585 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.583+0000 7efcecca1700 -1 req 18445032816498725289 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-51:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.20,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-51:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.20,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.674 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.671+0000 7efdbbe3f700 -1 req 8324155079888876390 0.001999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-1:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.1,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-1:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.1,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.720 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.718+0000 7efdaa61c700 -1 req 8554833125325408971 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-42:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.11,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-42:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.11,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.763 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:21:00.761+0000 7efd825cc700 -1 req 10462488542933650768 0.001999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-26:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.26,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-26:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.26,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:21:00.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bofrnf-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:00.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bofrnf-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:00.821 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bofrnf-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:21:00.868 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:00.868 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:00.908 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:21:00.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2023-10-18T20:21:00.912 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-55 2023-10-18T20:21:00.912 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:00.912 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:00.913 DEBUG:boto:Method: PUT 2023-10-18T20:21:00.913 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:00.914 DEBUG:boto:Data: 2023-10-18T20:21:00.914 DEBUG:boto:Headers: {} 2023-10-18T20:21:00.914 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:00.915 DEBUG:boto:Port: 8000 2023-10-18T20:21:00.915 DEBUG:boto:Params: {} 2023-10-18T20:21:00.916 DEBUG:boto:Token: None 2023-10-18T20:21:00.916 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:21:00 GMT /bofrnf-55/ 2023-10-18T20:21:00.916 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2uVPweGQnzfHFl3I8uWC/RUj2wE= 2023-10-18T20:21:00.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2uVPweGQnzfHFl3I8uWC/RUj2wE=', 'Content-Length': '0'} 2023-10-18T20:21:00.931 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000056104c21b7acc6b4-0065303e2c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:00.931 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:00.932 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:00.932 DEBUG:boto:Method: HEAD 2023-10-18T20:21:00.932 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:00.933 DEBUG:boto:Data: 2023-10-18T20:21:00.933 DEBUG:boto:Headers: {} 2023-10-18T20:21:00.934 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:00.934 DEBUG:boto:Port: 8000 2023-10-18T20:21:00.934 DEBUG:boto:Params: {} 2023-10-18T20:21:00.935 DEBUG:boto:Token: None 2023-10-18T20:21:00.935 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:00 GMT /bofrnf-55/ 2023-10-18T20:21:00.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:x377Kl0Gb5i1GiRmpoCoxu2taWQ= 2023-10-18T20:21:00.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x377Kl0Gb5i1GiRmpoCoxu2taWQ=', 'Content-Length': '0'} 2023-10-18T20:21:00.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058af3ac74dbc75bb-0065303e2c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:00.939 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:00.939 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:00.940 DEBUG:boto:Method: PUT 2023-10-18T20:21:00.940 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:21:00.941 DEBUG:boto:Data: 2023-10-18T20:21:00.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:21:00.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:00.942 DEBUG:boto:Port: 8000 2023-10-18T20:21:00.942 DEBUG:boto:Params: {} 2023-10-18T20:21:00.943 DEBUG:boto:Token: None 2023-10-18T20:21:00.943 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:21:00 GMT /bofrnf-55/a 2023-10-18T20:21:00.944 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uln18/VYbHswwKUJGTxbB8W1Q30= 2023-10-18T20:21:00.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uln18/VYbHswwKUJGTxbB8W1Q30='} 2023-10-18T20:21:00.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a016bd9be9b1def-0065303e2c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:00.987 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:00.987 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:00.988 DEBUG:boto:Method: HEAD 2023-10-18T20:21:00.988 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:00.988 DEBUG:boto:Data: 2023-10-18T20:21:00.989 DEBUG:boto:Headers: {} 2023-10-18T20:21:00.989 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:00.990 DEBUG:boto:Port: 8000 2023-10-18T20:21:00.990 DEBUG:boto:Params: {} 2023-10-18T20:21:00.991 DEBUG:boto:Token: None 2023-10-18T20:21:00.991 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:00 GMT /bofrnf-55/ 2023-10-18T20:21:00.991 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:x377Kl0Gb5i1GiRmpoCoxu2taWQ= 2023-10-18T20:21:00.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x377Kl0Gb5i1GiRmpoCoxu2taWQ=', 'Content-Length': '0'} 2023-10-18T20:21:00.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069ac03f711c23f08-0065303e2c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:00.995 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:00.995 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:00.995 DEBUG:boto:Method: PUT 2023-10-18T20:21:00.996 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:21:00.996 DEBUG:boto:Data: 2023-10-18T20:21:00.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:21:00.997 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:00.997 DEBUG:boto:Port: 8000 2023-10-18T20:21:00.998 DEBUG:boto:Params: {} 2023-10-18T20:21:00.998 DEBUG:boto:Token: None 2023-10-18T20:21:00.999 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:21:00 GMT /bofrnf-55/b 2023-10-18T20:21:00.999 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZnamhqBVicTE0gDzmaajR2YDFfc= 2023-10-18T20:21:01.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZnamhqBVicTE0gDzmaajR2YDFfc='} 2023-10-18T20:21:01.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000975b2a83e71e3cf3-0065303e2c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:01.005 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:01.005 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:01.005 DEBUG:boto:Method: HEAD 2023-10-18T20:21:01.006 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:01.006 DEBUG:boto:Data: 2023-10-18T20:21:01.007 DEBUG:boto:Headers: {} 2023-10-18T20:21:01.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:01.007 DEBUG:boto:Port: 8000 2023-10-18T20:21:01.008 DEBUG:boto:Params: {} 2023-10-18T20:21:01.008 DEBUG:boto:Token: None 2023-10-18T20:21:01.009 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:01 GMT /bofrnf-55/ 2023-10-18T20:21:01.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzZFnI8aZWQOCdxxA4sKxGYXa8o= 2023-10-18T20:21:01.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzZFnI8aZWQOCdxxA4sKxGYXa8o=', 'Content-Length': '0'} 2023-10-18T20:21:01.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086d70e9986a51ab9-0065303e2d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:01.012 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:01.013 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:01.013 DEBUG:boto:Method: PUT 2023-10-18T20:21:01.014 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:21:01.014 DEBUG:boto:Data: 2023-10-18T20:21:01.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:21:01.015 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:01.015 DEBUG:boto:Port: 8000 2023-10-18T20:21:01.016 DEBUG:boto:Params: {} 2023-10-18T20:21:01.016 DEBUG:boto:Token: None 2023-10-18T20:21:01.017 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:21:01 GMT /bofrnf-55/c 2023-10-18T20:21:01.017 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V+auCOajrI8u8tDxlNs3u3xGVhY= 2023-10-18T20:21:01.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V+auCOajrI8u8tDxlNs3u3xGVhY='} 2023-10-18T20:21:01.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da5351333ea643d2-0065303e2d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:01.023 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:01.023 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:01.024 DEBUG:boto:Method: HEAD 2023-10-18T20:21:01.024 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:01.025 DEBUG:boto:Data: 2023-10-18T20:21:01.025 DEBUG:boto:Headers: {} 2023-10-18T20:21:01.026 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:01.026 DEBUG:boto:Port: 8000 2023-10-18T20:21:01.026 DEBUG:boto:Params: {} 2023-10-18T20:21:01.027 DEBUG:boto:Token: None 2023-10-18T20:21:01.027 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:01 GMT /bofrnf-55/ 2023-10-18T20:21:01.028 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzZFnI8aZWQOCdxxA4sKxGYXa8o= 2023-10-18T20:21:01.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzZFnI8aZWQOCdxxA4sKxGYXa8o=', 'Content-Length': '0'} 2023-10-18T20:21:01.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f2be196f721e975-0065303e2d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:01.031 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:01.031 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:01.032 DEBUG:boto:Method: PUT 2023-10-18T20:21:01.032 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:21:01.033 DEBUG:boto:Data: 2023-10-18T20:21:01.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:21:01.033 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:01.034 DEBUG:boto:Port: 8000 2023-10-18T20:21:01.034 DEBUG:boto:Params: {} 2023-10-18T20:21:01.035 DEBUG:boto:Token: None 2023-10-18T20:21:01.035 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:21:01 GMT /bofrnf-55/d 2023-10-18T20:21:01.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nUiO0AGtFyBsrDanrKv59PKuf3s= 2023-10-18T20:21:01.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nUiO0AGtFyBsrDanrKv59PKuf3s='} 2023-10-18T20:21:01.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d03f1398795df2f-0065303e2d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:01.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:01.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:01.080 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:21:01.129 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:01.129 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:01.161 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:21:01.173 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:21:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660460.928951_896.1", 2023-10-18T20:21:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:21:00.928951Z" 2023-10-18T20:21:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:21:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:21:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:21:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:21:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:21:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:21:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:01.177 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:21:01.178 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660460.928951_896.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:21:01.179 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:21:01.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:01.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:01.180 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:21:01.229 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:01.229 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:01.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:01.254+0000 7f6b7ab59780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:21:01.257 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:01.255+0000 7f6b7ab59780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:21:01.269 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:21:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:21:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:21:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:21:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:21:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:21:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:21:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:21:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:21:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:21:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:21:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660460.928951_896.1", 2023-10-18T20:21:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:21:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:21:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:21:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:21:00.928951Z", 2023-10-18T20:21:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:21:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:21:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:21:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:21:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:21:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:21:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:21:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:21:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:21:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:21:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:21:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:21:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:21:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:21:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:21:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:21:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:21:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:21:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:21:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:21:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:21:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:21:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:21:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:21:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:21:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:21:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:21:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:21:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:21:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:21:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:21:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:21:01Z" 2023-10-18T20:21:01.284 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:21:01.284 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660460.928951_896.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:21:00.928951Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:21:01Z" } 2023-10-18T20:21:01.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660460.928951_896.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:21:00.928951Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:21:01.285 DEBUG:rgw_multi.tests:log_status={0: '1_1697660460.928951_896.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:21:01.286 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660460.928951_896.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:21:01.286 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:21:01.286 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-55 2023-10-18T20:21:01.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:21:01.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:21:01.288 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:21:01.344 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:01.344 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:01.381 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:01.379+0000 7fdfbc03a780 1 waiting to reach incremental sync.. 2023-10-18T20:21:06.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:06.380+0000 7fdfbc03a780 1 waiting to reach incremental sync.. 2023-10-18T20:21:11.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:11.381+0000 7fdfbc03a780 1 waiting to reach incremental sync.. 2023-10-18T20:21:16.383 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:16.382+0000 7fdfbc03a780 1 waiting to reach incremental sync.. 2023-10-18T20:21:21.385 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:21.383+0000 7fdfbc03a780 1 bucket sync caught up with source: 2023-10-18T20:21:21.385 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.22.6, , , , 00000000001.53.6, , , 00000000001.62.6, , 00000000001.24.6] 2023-10-18T20:21:21.385 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.22.6, , , , 00000000001.53.6, , , 00000000001.62.6, , 00000000001.24.6] 2023-10-18T20:21:21.385 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:21.383+0000 7fdfbc03a780 0 bucket checkpoint complete 2023-10-18T20:21:21.401 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-55 2023-10-18T20:21:21.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:21:21.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:21:21.402 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:21:21.463 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:21.463 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:21.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:21.498+0000 7f4d58d54780 1 waiting for incremental sync to catch up: 2023-10-18T20:21:21.500 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , ] 2023-10-18T20:21:21.500 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T20:21:26.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:26.500+0000 7f4d58d54780 1 waiting for incremental sync to catch up: 2023-10-18T20:21:26.501 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , ] 2023-10-18T20:21:26.502 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T20:21:31.502 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:31.501+0000 7f4d58d54780 1 waiting for incremental sync to catch up: 2023-10-18T20:21:31.503 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , ] 2023-10-18T20:21:31.503 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T20:21:36.504 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:36.502+0000 7f4d58d54780 1 waiting for incremental sync to catch up: 2023-10-18T20:21:36.504 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , ] 2023-10-18T20:21:36.504 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T20:21:41.505 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:41.503+0000 7f4d58d54780 1 bucket sync caught up with source: 2023-10-18T20:21:41.505 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T20:21:41.506 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T20:21:41.506 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:41.503+0000 7f4d58d54780 0 bucket checkpoint complete 2023-10-18T20:21:41.521 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-55 zones={a2, a1} 2023-10-18T20:21:41.522 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.522 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.522 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.523 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:41.523 DEBUG:boto:Data: 2023-10-18T20:21:41.524 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.524 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.524 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.525 DEBUG:boto:Params: {} 2023-10-18T20:21:41.525 DEBUG:boto:Token: None 2023-10-18T20:21:41.526 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.526 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EzCimMfrmwk8njidnEYeicj0yLo= 2023-10-18T20:21:41.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EzCimMfrmwk8njidnEYeicj0yLo=', 'Content-Length': '0'} 2023-10-18T20:21:41.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c35b62503bcea76-0065303e55-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.530 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.530 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.531 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.531 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:41.531 DEBUG:boto:Data: 2023-10-18T20:21:41.532 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.532 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.532 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.533 DEBUG:boto:Params: {} 2023-10-18T20:21:41.533 DEBUG:boto:Token: None 2023-10-18T20:21:41.534 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.534 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EzCimMfrmwk8njidnEYeicj0yLo= 2023-10-18T20:21:41.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EzCimMfrmwk8njidnEYeicj0yLo=', 'Content-Length': '0'} 2023-10-18T20:21:41.537 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc8fe1cba2267e2b-0065303e55-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.538 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:21:41.538 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.539 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.539 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.539 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.540 DEBUG:boto:Method: GET 2023-10-18T20:21:41.540 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.540 DEBUG:boto:Data: 2023-10-18T20:21:41.541 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.541 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.542 DEBUG:boto:Params: {} 2023-10-18T20:21:41.542 DEBUG:boto:Token: None 2023-10-18T20:21:41.543 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/?versions 2023-10-18T20:21:41.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts= 2023-10-18T20:21:41.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts=', 'Content-Length': '0'} 2023-10-18T20:21:41.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a32928af39ce04b-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.547 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.548 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:21:41.548 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:21:41.548 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:21:41.549 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:21:41.549 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:21:41.549 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.550 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.550 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.551 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.551 DEBUG:boto:Method: GET 2023-10-18T20:21:41.551 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.552 DEBUG:boto:Data: 2023-10-18T20:21:41.552 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.553 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.553 DEBUG:boto:Params: {} 2023-10-18T20:21:41.554 DEBUG:boto:Token: None 2023-10-18T20:21:41.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/?versions 2023-10-18T20:21:41.554 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts= 2023-10-18T20:21:41.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts=', 'Content-Length': '0'} 2023-10-18T20:21:41.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b2158e088204886-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.558 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.559 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:21:41.559 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:21:41.559 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:21:41.560 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:21:41.560 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.561 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.561 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.561 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.562 DEBUG:boto:Method: GET 2023-10-18T20:21:41.562 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.562 DEBUG:boto:Data: 2023-10-18T20:21:41.563 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.563 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.564 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.564 DEBUG:boto:Params: {} 2023-10-18T20:21:41.564 DEBUG:boto:Token: None 2023-10-18T20:21:41.565 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/?versions 2023-10-18T20:21:41.565 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts= 2023-10-18T20:21:41.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts=', 'Content-Length': '0'} 2023-10-18T20:21:41.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fcfd26a2ac0cb92-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.568 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.569 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.570 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.570 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.570 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.571 DEBUG:boto:Method: GET 2023-10-18T20:21:41.571 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:21:41.571 DEBUG:boto:Data: 2023-10-18T20:21:41.572 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.572 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.573 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.573 DEBUG:boto:Params: {} 2023-10-18T20:21:41.573 DEBUG:boto:Token: None 2023-10-18T20:21:41.574 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/?versions 2023-10-18T20:21:41.574 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts= 2023-10-18T20:21:41.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts=', 'Content-Length': '0'} 2023-10-18T20:21:41.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d9a28acf1892d7-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.578 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:21:41.579 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.579 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.580 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.580 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.580 DEBUG:boto:Method: GET 2023-10-18T20:21:41.581 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.581 DEBUG:boto:Data: 2023-10-18T20:21:41.582 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.582 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.583 DEBUG:boto:Params: {} 2023-10-18T20:21:41.583 DEBUG:boto:Token: None 2023-10-18T20:21:41.584 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.584 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY= 2023-10-18T20:21:41.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY=', 'Content-Length': '0'} 2023-10-18T20:21:41.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0280921e841cbb5-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.630 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.631 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.631 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.632 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.632 DEBUG:boto:Method: GET 2023-10-18T20:21:41.632 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.633 DEBUG:boto:Data: 2023-10-18T20:21:41.633 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.634 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.634 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.634 DEBUG:boto:Params: {} 2023-10-18T20:21:41.635 DEBUG:boto:Token: None 2023-10-18T20:21:41.635 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.636 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY= 2023-10-18T20:21:41.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY=', 'Content-Length': '0'} 2023-10-18T20:21:41.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005019f9547219a8af-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.639 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.639 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.639 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.640 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.640 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.640 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.641 DEBUG:boto:Data: 2023-10-18T20:21:41.641 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.642 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.642 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.642 DEBUG:boto:Params: {} 2023-10-18T20:21:41.643 DEBUG:boto:Token: None 2023-10-18T20:21:41.643 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.644 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:27sd5rlqNlya999wZlrtqmqG6XA= 2023-10-18T20:21:41.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:27sd5rlqNlya999wZlrtqmqG6XA=', 'Content-Length': '0'} 2023-10-18T20:21:41.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bd19b49ad0fdab4-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.646 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.647 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.647 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.648 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.648 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.648 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.649 DEBUG:boto:Data: 2023-10-18T20:21:41.649 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.650 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.650 DEBUG:boto:Params: {} 2023-10-18T20:21:41.650 DEBUG:boto:Token: None 2023-10-18T20:21:41.651 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.651 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:27sd5rlqNlya999wZlrtqmqG6XA= 2023-10-18T20:21:41.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:27sd5rlqNlya999wZlrtqmqG6XA=', 'Content-Length': '0'} 2023-10-18T20:21:41.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1566f945e18b60d-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:21:41.654 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.655 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.655 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.656 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.656 DEBUG:boto:Method: GET 2023-10-18T20:21:41.656 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.657 DEBUG:boto:Data: 2023-10-18T20:21:41.657 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.657 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.658 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.658 DEBUG:boto:Params: {} 2023-10-18T20:21:41.659 DEBUG:boto:Token: None 2023-10-18T20:21:41.659 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.659 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY= 2023-10-18T20:21:41.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY=', 'Content-Length': '0'} 2023-10-18T20:21:41.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008128dd1abef6c92f-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.662 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.663 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.663 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.663 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:21:41.664 DEBUG:boto:Method: GET 2023-10-18T20:21:41.664 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.665 DEBUG:boto:Data: 2023-10-18T20:21:41.665 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.665 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.666 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.666 DEBUG:boto:Params: {} 2023-10-18T20:21:41.667 DEBUG:boto:Token: None 2023-10-18T20:21:41.667 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:21:41.667 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY= 2023-10-18T20:21:41.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY=', 'Content-Length': '0'} 2023-10-18T20:21:41.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000129555e14a1d8c65-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.671 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.671 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.672 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:21:41.672 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:21:41.672 DEBUG:boto:Method: GET 2023-10-18T20:21:41.673 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:21:41.673 DEBUG:boto:Data: 2023-10-18T20:21:41.673 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.674 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.674 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.675 DEBUG:boto:Params: {} 2023-10-18T20:21:41.675 DEBUG:boto:Token: None 2023-10-18T20:21:41.675 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T20:21:41.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T20:21:41.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074dc656e0f9e5be8-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.679 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.680 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.680 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.680 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:21:41.681 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:21:41.681 DEBUG:boto:Method: GET 2023-10-18T20:21:41.682 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:21:41.682 DEBUG:boto:Data: 2023-10-18T20:21:41.682 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.683 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.683 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.684 DEBUG:boto:Params: {} 2023-10-18T20:21:41.684 DEBUG:boto:Token: None 2023-10-18T20:21:41.684 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.685 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T20:21:41.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T20:21:41.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f50170520ee83b26-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.688 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.689 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.689 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.689 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.690 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:21:41.690 DEBUG:boto:Data: 2023-10-18T20:21:41.691 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.691 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.691 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.692 DEBUG:boto:Params: {} 2023-10-18T20:21:41.692 DEBUG:boto:Token: None 2023-10-18T20:21:41.692 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a 2023-10-18T20:21:41.693 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CMAlwgVDuXBGLv0lqaY9oiuEDDU= 2023-10-18T20:21:41.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CMAlwgVDuXBGLv0lqaY9oiuEDDU=', 'Content-Length': '0'} 2023-10-18T20:21:41.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18a19b53d16b663-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.696 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.696 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.696 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.697 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:21:41.697 DEBUG:boto:Data: 2023-10-18T20:21:41.697 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.698 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.699 DEBUG:boto:Params: {} 2023-10-18T20:21:41.699 DEBUG:boto:Token: None 2023-10-18T20:21:41.699 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a 2023-10-18T20:21:41.700 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CMAlwgVDuXBGLv0lqaY9oiuEDDU= 2023-10-18T20:21:41.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CMAlwgVDuXBGLv0lqaY9oiuEDDU=', 'Content-Length': '0'} 2023-10-18T20:21:41.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f633762180734e2-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:21:41.703 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.704 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.704 DEBUG:boto:Method: GET 2023-10-18T20:21:41.704 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:21:41.705 DEBUG:boto:Data: 2023-10-18T20:21:41.705 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.705 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.706 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.706 DEBUG:boto:Params: {} 2023-10-18T20:21:41.707 DEBUG:boto:Token: None 2023-10-18T20:21:41.707 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a 2023-10-18T20:21:41.707 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:72ljtWOcoL0aZ9ikI5eW95ta+so= 2023-10-18T20:21:41.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:72ljtWOcoL0aZ9ikI5eW95ta+so=', 'Content-Length': '0'} 2023-10-18T20:21:41.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a544c997790d8301-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.750 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:21:41.751 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:21:41.751 DEBUG:boto:Method: GET 2023-10-18T20:21:41.752 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:21:41.752 DEBUG:boto:Data: 2023-10-18T20:21:41.752 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.753 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.754 DEBUG:boto:Params: {} 2023-10-18T20:21:41.754 DEBUG:boto:Token: None 2023-10-18T20:21:41.755 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/a 2023-10-18T20:21:41.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:72ljtWOcoL0aZ9ikI5eW95ta+so= 2023-10-18T20:21:41.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:72ljtWOcoL0aZ9ikI5eW95ta+so=', 'Content-Length': '0'} 2023-10-18T20:21:41.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000750b9676c35d792-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.758 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.759 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.759 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:21:41.759 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:21:41.760 DEBUG:boto:Method: GET 2023-10-18T20:21:41.760 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:21:41.761 DEBUG:boto:Data: 2023-10-18T20:21:41.761 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.762 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.762 DEBUG:boto:Params: {} 2023-10-18T20:21:41.762 DEBUG:boto:Token: None 2023-10-18T20:21:41.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T20:21:41.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T20:21:41.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc178e45eca0c6f3-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.767 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.767 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.768 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.768 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:21:41.768 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:21:41.769 DEBUG:boto:Method: GET 2023-10-18T20:21:41.769 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:21:41.769 DEBUG:boto:Data: 2023-10-18T20:21:41.770 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.771 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.771 DEBUG:boto:Params: {} 2023-10-18T20:21:41.772 DEBUG:boto:Token: None 2023-10-18T20:21:41.772 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T20:21:41.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T20:21:41.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000493a4de580b8468c-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.776 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:21:41.777 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.777 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.778 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.778 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.778 DEBUG:boto:Method: GET 2023-10-18T20:21:41.779 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.779 DEBUG:boto:Data: 2023-10-18T20:21:41.780 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.780 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.780 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.781 DEBUG:boto:Params: {} 2023-10-18T20:21:41.781 DEBUG:boto:Token: None 2023-10-18T20:21:41.781 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4= 2023-10-18T20:21:41.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4=', 'Content-Length': '0'} 2023-10-18T20:21:41.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad4ab715d34bed4d-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.785 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.786 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.786 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.786 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.787 DEBUG:boto:Method: GET 2023-10-18T20:21:41.787 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.788 DEBUG:boto:Data: 2023-10-18T20:21:41.788 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.788 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.789 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.789 DEBUG:boto:Params: {} 2023-10-18T20:21:41.790 DEBUG:boto:Token: None 2023-10-18T20:21:41.790 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.790 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4= 2023-10-18T20:21:41.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4=', 'Content-Length': '0'} 2023-10-18T20:21:41.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf79866089006e2-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.794 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.794 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.794 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.795 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.795 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.795 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.796 DEBUG:boto:Data: 2023-10-18T20:21:41.796 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.796 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.797 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.797 DEBUG:boto:Params: {} 2023-10-18T20:21:41.798 DEBUG:boto:Token: None 2023-10-18T20:21:41.798 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.798 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5VtbzXtJ+fpIpyfpwqzVr8RKDKU= 2023-10-18T20:21:41.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5VtbzXtJ+fpIpyfpwqzVr8RKDKU=', 'Content-Length': '0'} 2023-10-18T20:21:41.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd021948b5c2462-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.801 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.802 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.802 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.803 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.803 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.803 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.804 DEBUG:boto:Data: 2023-10-18T20:21:41.804 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.804 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.805 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.805 DEBUG:boto:Params: {} 2023-10-18T20:21:41.805 DEBUG:boto:Token: None 2023-10-18T20:21:41.806 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.806 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5VtbzXtJ+fpIpyfpwqzVr8RKDKU= 2023-10-18T20:21:41.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5VtbzXtJ+fpIpyfpwqzVr8RKDKU=', 'Content-Length': '0'} 2023-10-18T20:21:41.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e5f45c0083e1923-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:21:41.810 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.810 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.811 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.811 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.811 DEBUG:boto:Method: GET 2023-10-18T20:21:41.812 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.812 DEBUG:boto:Data: 2023-10-18T20:21:41.813 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.813 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.813 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.814 DEBUG:boto:Params: {} 2023-10-18T20:21:41.814 DEBUG:boto:Token: None 2023-10-18T20:21:41.815 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.815 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4= 2023-10-18T20:21:41.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4=', 'Content-Length': '0'} 2023-10-18T20:21:41.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d9d435668f7593-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.818 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.819 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.819 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.819 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:21:41.820 DEBUG:boto:Method: GET 2023-10-18T20:21:41.820 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.821 DEBUG:boto:Data: 2023-10-18T20:21:41.821 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.821 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.822 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.822 DEBUG:boto:Params: {} 2023-10-18T20:21:41.823 DEBUG:boto:Token: None 2023-10-18T20:21:41.823 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:21:41.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4= 2023-10-18T20:21:41.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4=', 'Content-Length': '0'} 2023-10-18T20:21:41.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea0a0092c549751-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.827 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.827 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.827 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:21:41.828 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:21:41.828 DEBUG:boto:Method: GET 2023-10-18T20:21:41.829 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:21:41.829 DEBUG:boto:Data: 2023-10-18T20:21:41.829 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.830 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.831 DEBUG:boto:Params: {} 2023-10-18T20:21:41.831 DEBUG:boto:Token: None 2023-10-18T20:21:41.831 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T20:21:41.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T20:21:41.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b18aa87746605b4-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.835 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.836 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.836 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.837 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:21:41.837 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:21:41.837 DEBUG:boto:Method: GET 2023-10-18T20:21:41.838 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:21:41.838 DEBUG:boto:Data: 2023-10-18T20:21:41.839 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.839 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.839 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.840 DEBUG:boto:Params: {} 2023-10-18T20:21:41.840 DEBUG:boto:Token: None 2023-10-18T20:21:41.841 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.841 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T20:21:41.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T20:21:41.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1faf967e02f13d0-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.890 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.891 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.892 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.892 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.892 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:21:41.893 DEBUG:boto:Data: 2023-10-18T20:21:41.893 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.893 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.894 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.894 DEBUG:boto:Params: {} 2023-10-18T20:21:41.895 DEBUG:boto:Token: None 2023-10-18T20:21:41.895 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b 2023-10-18T20:21:41.896 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KxPoZahOkpYAEAD4wmb+vEpGtX4= 2023-10-18T20:21:41.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KxPoZahOkpYAEAD4wmb+vEpGtX4=', 'Content-Length': '0'} 2023-10-18T20:21:41.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007978a015d6cb4d33-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.898 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.899 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.899 DEBUG:boto:Method: HEAD 2023-10-18T20:21:41.900 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:21:41.900 DEBUG:boto:Data: 2023-10-18T20:21:41.900 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.901 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.901 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.902 DEBUG:boto:Params: {} 2023-10-18T20:21:41.902 DEBUG:boto:Token: None 2023-10-18T20:21:41.902 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b 2023-10-18T20:21:41.903 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KxPoZahOkpYAEAD4wmb+vEpGtX4= 2023-10-18T20:21:41.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KxPoZahOkpYAEAD4wmb+vEpGtX4=', 'Content-Length': '0'} 2023-10-18T20:21:41.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb0a38942c009f8-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:21:41.907 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.907 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.907 DEBUG:boto:Method: GET 2023-10-18T20:21:41.908 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:21:41.908 DEBUG:boto:Data: 2023-10-18T20:21:41.909 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.909 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.909 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.910 DEBUG:boto:Params: {} 2023-10-18T20:21:41.910 DEBUG:boto:Token: None 2023-10-18T20:21:41.911 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b 2023-10-18T20:21:41.911 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1P3yaw8yvtthhxTp1kxo7HQ1nvs= 2023-10-18T20:21:41.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1P3yaw8yvtthhxTp1kxo7HQ1nvs=', 'Content-Length': '0'} 2023-10-18T20:21:41.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005379c903731241bb-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.914 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:21:41.915 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:21:41.915 DEBUG:boto:Method: GET 2023-10-18T20:21:41.915 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:21:41.916 DEBUG:boto:Data: 2023-10-18T20:21:41.916 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.916 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.917 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.917 DEBUG:boto:Params: {} 2023-10-18T20:21:41.918 DEBUG:boto:Token: None 2023-10-18T20:21:41.918 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/b 2023-10-18T20:21:41.919 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1P3yaw8yvtthhxTp1kxo7HQ1nvs= 2023-10-18T20:21:41.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1P3yaw8yvtthhxTp1kxo7HQ1nvs=', 'Content-Length': '0'} 2023-10-18T20:21:41.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4364a99dfe1248e-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.922 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.922 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.923 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:21:41.923 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:21:41.923 DEBUG:boto:Method: GET 2023-10-18T20:21:41.924 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:21:41.924 DEBUG:boto:Data: 2023-10-18T20:21:41.925 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.925 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.925 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.926 DEBUG:boto:Params: {} 2023-10-18T20:21:41.926 DEBUG:boto:Token: None 2023-10-18T20:21:41.927 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.927 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T20:21:41.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T20:21:41.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000638f2a568a0e2ede-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.931 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.931 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:41.932 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:41.932 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:21:41.932 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:21:41.933 DEBUG:boto:Method: GET 2023-10-18T20:21:41.933 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:21:41.934 DEBUG:boto:Data: 2023-10-18T20:21:41.934 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.935 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.935 DEBUG:boto:Params: {} 2023-10-18T20:21:41.936 DEBUG:boto:Token: None 2023-10-18T20:21:41.936 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/ 2023-10-18T20:21:41.937 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T20:21:41.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T20:21:41.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071bcaecb6dfe663b-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.940 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:41.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:21:41.941 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:41.942 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:41.942 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:41.943 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:41.943 DEBUG:boto:Method: GET 2023-10-18T20:21:41.944 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:21:41.944 DEBUG:boto:Data: 2023-10-18T20:21:41.944 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:41.945 DEBUG:boto:Port: 8001 2023-10-18T20:21:41.946 DEBUG:boto:Params: {} 2023-10-18T20:21:41.946 DEBUG:boto:Token: None 2023-10-18T20:21:41.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:21:41.947 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:41ovE1ZvU5vrcUdqnwnIkbnXp5c= 2023-10-18T20:21:41.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:41ovE1ZvU5vrcUdqnwnIkbnXp5c=', 'Content-Length': '0'} 2023-10-18T20:21:41.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c7889b9496baf6-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.990 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:41.991 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:41.991 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:41.992 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:41.992 DEBUG:boto:Method: GET 2023-10-18T20:21:41.992 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:21:41.993 DEBUG:boto:Data: 2023-10-18T20:21:41.993 DEBUG:boto:Headers: {} 2023-10-18T20:21:41.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:41.994 DEBUG:boto:Port: 8000 2023-10-18T20:21:41.994 DEBUG:boto:Params: {} 2023-10-18T20:21:41.995 DEBUG:boto:Token: None 2023-10-18T20:21:41.995 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:41 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:21:41.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:41ovE1ZvU5vrcUdqnwnIkbnXp5c= 2023-10-18T20:21:41.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:41ovE1ZvU5vrcUdqnwnIkbnXp5c=', 'Content-Length': '0'} 2023-10-18T20:21:41.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea14ee9be02a8f56-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:41.999 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:41.999 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:42.000 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:42.000 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:42.001 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.001 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:21:42.001 DEBUG:boto:Data: 2023-10-18T20:21:42.002 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.002 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.002 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.003 DEBUG:boto:Params: {} 2023-10-18T20:21:42.003 DEBUG:boto:Token: None 2023-10-18T20:21:42.004 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:21:42.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5FMFMzt189MfHXhS1GI5hC0E6mA= 2023-10-18T20:21:42.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5FMFMzt189MfHXhS1GI5hC0E6mA=', 'Content-Length': '0'} 2023-10-18T20:21:42.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f01a3afed3e1afb-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.007 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:42.007 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:42.008 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:42.008 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:42.009 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.009 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:21:42.009 DEBUG:boto:Data: 2023-10-18T20:21:42.010 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.010 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.011 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.011 DEBUG:boto:Params: {} 2023-10-18T20:21:42.012 DEBUG:boto:Token: None 2023-10-18T20:21:42.012 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:21:42.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5FMFMzt189MfHXhS1GI5hC0E6mA= 2023-10-18T20:21:42.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5FMFMzt189MfHXhS1GI5hC0E6mA=', 'Content-Length': '0'} 2023-10-18T20:21:42.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005670f21fc88dbc83-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:21:42.016 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:42.016 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:42.017 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:42.017 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:42.017 DEBUG:boto:Method: GET 2023-10-18T20:21:42.018 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:21:42.018 DEBUG:boto:Data: 2023-10-18T20:21:42.019 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.019 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.019 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.020 DEBUG:boto:Params: {} 2023-10-18T20:21:42.020 DEBUG:boto:Token: None 2023-10-18T20:21:42.021 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:21:42.021 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:57H4m8JGBgxh3fs0PZ9tFQHXbO0= 2023-10-18T20:21:42.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:57H4m8JGBgxh3fs0PZ9tFQHXbO0=', 'Content-Length': '0'} 2023-10-18T20:21:42.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009554d2b092579d3c-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.024 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:42.025 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:42.025 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:42.025 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:21:42.026 DEBUG:boto:Method: GET 2023-10-18T20:21:42.026 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:21:42.027 DEBUG:boto:Data: 2023-10-18T20:21:42.027 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.027 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.028 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.028 DEBUG:boto:Params: {} 2023-10-18T20:21:42.028 DEBUG:boto:Token: None 2023-10-18T20:21:42.029 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:21:42.029 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:57H4m8JGBgxh3fs0PZ9tFQHXbO0= 2023-10-18T20:21:42.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:57H4m8JGBgxh3fs0PZ9tFQHXbO0=', 'Content-Length': '0'} 2023-10-18T20:21:42.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3da0683ce699d2-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.074 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.075 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.075 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:21:42.076 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:21:42.076 DEBUG:boto:Method: GET 2023-10-18T20:21:42.076 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:21:42.077 DEBUG:boto:Data: 2023-10-18T20:21:42.077 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.078 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.078 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.079 DEBUG:boto:Params: {} 2023-10-18T20:21:42.079 DEBUG:boto:Token: None 2023-10-18T20:21:42.079 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.080 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T20:21:42.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T20:21:42.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d9262fe74fb463c-0065303e56-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.083 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:42.084 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.084 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.085 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:21:42.085 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:21:42.086 DEBUG:boto:Method: GET 2023-10-18T20:21:42.086 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:21:42.086 DEBUG:boto:Data: 2023-10-18T20:21:42.087 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.087 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.088 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.088 DEBUG:boto:Params: {} 2023-10-18T20:21:42.088 DEBUG:boto:Token: None 2023-10-18T20:21:42.089 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T20:21:42.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T20:21:42.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d053b65a8e866bd6-0065303e56-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.093 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:42.094 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:42.094 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:42.095 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.095 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:21:42.095 DEBUG:boto:Data: 2023-10-18T20:21:42.096 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.096 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.096 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.097 DEBUG:boto:Params: {} 2023-10-18T20:21:42.097 DEBUG:boto:Token: None 2023-10-18T20:21:42.098 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/c 2023-10-18T20:21:42.098 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dehh8R/IuGyB+l3H1IbvP2p3JXk= 2023-10-18T20:21:42.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dehh8R/IuGyB+l3H1IbvP2p3JXk=', 'Content-Length': '0'} 2023-10-18T20:21:42.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca36833d58edc82f-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.101 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:42.101 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:42.102 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.102 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:21:42.102 DEBUG:boto:Data: 2023-10-18T20:21:42.103 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.103 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.103 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.104 DEBUG:boto:Params: {} 2023-10-18T20:21:42.104 DEBUG:boto:Token: None 2023-10-18T20:21:42.105 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/c 2023-10-18T20:21:42.105 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dehh8R/IuGyB+l3H1IbvP2p3JXk= 2023-10-18T20:21:42.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dehh8R/IuGyB+l3H1IbvP2p3JXk=', 'Content-Length': '0'} 2023-10-18T20:21:42.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c55160e32e1d8580-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:21:42.108 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:42.109 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:42.109 DEBUG:boto:Method: GET 2023-10-18T20:21:42.110 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:21:42.110 DEBUG:boto:Data: 2023-10-18T20:21:42.110 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.111 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.111 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.112 DEBUG:boto:Params: {} 2023-10-18T20:21:42.112 DEBUG:boto:Token: None 2023-10-18T20:21:42.113 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/c 2023-10-18T20:21:42.113 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bxzv+VNCtBgbxxPKvuMjYItgnjg= 2023-10-18T20:21:42.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxzv+VNCtBgbxxPKvuMjYItgnjg=', 'Content-Length': '0'} 2023-10-18T20:21:42.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000918048bf345124ff-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.158 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:21:42.159 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:21:42.159 DEBUG:boto:Method: GET 2023-10-18T20:21:42.160 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:21:42.160 DEBUG:boto:Data: 2023-10-18T20:21:42.160 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.161 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.161 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.161 DEBUG:boto:Params: {} 2023-10-18T20:21:42.162 DEBUG:boto:Token: None 2023-10-18T20:21:42.162 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/c 2023-10-18T20:21:42.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bxzv+VNCtBgbxxPKvuMjYItgnjg= 2023-10-18T20:21:42.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxzv+VNCtBgbxxPKvuMjYItgnjg=', 'Content-Length': '0'} 2023-10-18T20:21:42.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a9dea0fa09239f-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.166 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.167 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.167 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:21:42.167 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:21:42.168 DEBUG:boto:Method: GET 2023-10-18T20:21:42.168 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:21:42.169 DEBUG:boto:Data: 2023-10-18T20:21:42.169 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.170 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.170 DEBUG:boto:Params: {} 2023-10-18T20:21:42.170 DEBUG:boto:Token: None 2023-10-18T20:21:42.171 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T20:21:42.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T20:21:42.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e5f03ac86590839-0065303e56-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.175 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:42.175 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.176 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.176 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:21:42.177 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:21:42.177 DEBUG:boto:Method: GET 2023-10-18T20:21:42.178 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:21:42.178 DEBUG:boto:Data: 2023-10-18T20:21:42.178 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.179 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.179 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.180 DEBUG:boto:Params: {} 2023-10-18T20:21:42.180 DEBUG:boto:Token: None 2023-10-18T20:21:42.180 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.181 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T20:21:42.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T20:21:42.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be8e8c1886e81ae1-0065303e56-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.184 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:42.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:21:42.185 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.186 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.186 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.187 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.187 DEBUG:boto:Method: GET 2023-10-18T20:21:42.187 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.188 DEBUG:boto:Data: 2023-10-18T20:21:42.188 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.189 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.189 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.189 DEBUG:boto:Params: {} 2023-10-18T20:21:42.190 DEBUG:boto:Token: None 2023-10-18T20:21:42.190 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.191 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM= 2023-10-18T20:21:42.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM=', 'Content-Length': '0'} 2023-10-18T20:21:42.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096e7039955a0fd33-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.194 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.194 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.195 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.195 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.195 DEBUG:boto:Method: GET 2023-10-18T20:21:42.196 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.196 DEBUG:boto:Data: 2023-10-18T20:21:42.196 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.197 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.197 DEBUG:boto:Params: {} 2023-10-18T20:21:42.198 DEBUG:boto:Token: None 2023-10-18T20:21:42.198 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.199 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM= 2023-10-18T20:21:42.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM=', 'Content-Length': '0'} 2023-10-18T20:21:42.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c0ba3cdf06d9d2-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.202 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.202 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.203 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.203 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.204 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.204 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.204 DEBUG:boto:Data: 2023-10-18T20:21:42.205 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.205 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.205 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.206 DEBUG:boto:Params: {} 2023-10-18T20:21:42.206 DEBUG:boto:Token: None 2023-10-18T20:21:42.206 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.207 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OlowGsMXdnKwod4Th7Des/4DaRU= 2023-10-18T20:21:42.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlowGsMXdnKwod4Th7Des/4DaRU=', 'Content-Length': '0'} 2023-10-18T20:21:42.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001332fe4ba4de79aa-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.210 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.210 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.211 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.211 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.212 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.212 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.212 DEBUG:boto:Data: 2023-10-18T20:21:42.213 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.213 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.213 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.214 DEBUG:boto:Params: {} 2023-10-18T20:21:42.214 DEBUG:boto:Token: None 2023-10-18T20:21:42.215 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OlowGsMXdnKwod4Th7Des/4DaRU= 2023-10-18T20:21:42.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlowGsMXdnKwod4Th7Des/4DaRU=', 'Content-Length': '0'} 2023-10-18T20:21:42.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31fbe1de578c189-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:21:42.219 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.219 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.219 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.220 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.220 DEBUG:boto:Method: GET 2023-10-18T20:21:42.221 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.221 DEBUG:boto:Data: 2023-10-18T20:21:42.221 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.222 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.222 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.222 DEBUG:boto:Params: {} 2023-10-18T20:21:42.223 DEBUG:boto:Token: None 2023-10-18T20:21:42.223 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.224 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM= 2023-10-18T20:21:42.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM=', 'Content-Length': '0'} 2023-10-18T20:21:42.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a84f5d8143ccc41e-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.227 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.227 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.228 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.228 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:21:42.228 DEBUG:boto:Method: GET 2023-10-18T20:21:42.229 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.229 DEBUG:boto:Data: 2023-10-18T20:21:42.229 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.230 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.231 DEBUG:boto:Params: {} 2023-10-18T20:21:42.231 DEBUG:boto:Token: None 2023-10-18T20:21:42.231 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:21:42.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM= 2023-10-18T20:21:42.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM=', 'Content-Length': '0'} 2023-10-18T20:21:42.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4f2acefdca7421c-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.235 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.236 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.236 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:21:42.236 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:21:42.237 DEBUG:boto:Method: GET 2023-10-18T20:21:42.237 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:21:42.237 DEBUG:boto:Data: 2023-10-18T20:21:42.238 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.238 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.239 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.239 DEBUG:boto:Params: {} 2023-10-18T20:21:42.239 DEBUG:boto:Token: None 2023-10-18T20:21:42.240 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T20:21:42.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T20:21:42.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec3769eaa346aa3-0065303e56-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.286 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:42.287 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.287 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.288 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:21:42.288 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:21:42.288 DEBUG:boto:Method: GET 2023-10-18T20:21:42.289 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:21:42.289 DEBUG:boto:Data: 2023-10-18T20:21:42.290 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.290 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.290 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.291 DEBUG:boto:Params: {} 2023-10-18T20:21:42.291 DEBUG:boto:Token: None 2023-10-18T20:21:42.292 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.292 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T20:21:42.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T20:21:42.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028326a75eb0dd4a3-0065303e56-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.296 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:42.296 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.297 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.297 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.298 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:21:42.298 DEBUG:boto:Data: 2023-10-18T20:21:42.298 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.299 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.300 DEBUG:boto:Params: {} 2023-10-18T20:21:42.300 DEBUG:boto:Token: None 2023-10-18T20:21:42.301 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d 2023-10-18T20:21:42.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/uIDX+rs7FfLM+UZguUTHSYXJDQ= 2023-10-18T20:21:42.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/uIDX+rs7FfLM+UZguUTHSYXJDQ=', 'Content-Length': '0'} 2023-10-18T20:21:42.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc3d92d60bf355d8-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.304 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.304 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.305 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.305 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:21:42.306 DEBUG:boto:Data: 2023-10-18T20:21:42.306 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.306 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.307 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.307 DEBUG:boto:Params: {} 2023-10-18T20:21:42.308 DEBUG:boto:Token: None 2023-10-18T20:21:42.308 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d 2023-10-18T20:21:42.308 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/uIDX+rs7FfLM+UZguUTHSYXJDQ= 2023-10-18T20:21:42.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/uIDX+rs7FfLM+UZguUTHSYXJDQ=', 'Content-Length': '0'} 2023-10-18T20:21:42.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f4aecac62e76e0-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:21:42.312 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.312 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.313 DEBUG:boto:Method: GET 2023-10-18T20:21:42.313 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:21:42.313 DEBUG:boto:Data: 2023-10-18T20:21:42.314 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.315 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.315 DEBUG:boto:Params: {} 2023-10-18T20:21:42.315 DEBUG:boto:Token: None 2023-10-18T20:21:42.316 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d 2023-10-18T20:21:42.316 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wbUwzp6z1qbVhDtEabXHt9v8Wpw= 2023-10-18T20:21:42.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wbUwzp6z1qbVhDtEabXHt9v8Wpw=', 'Content-Length': '0'} 2023-10-18T20:21:42.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a479ee35f1a9e9-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.319 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:21:42.320 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:21:42.320 DEBUG:boto:Method: GET 2023-10-18T20:21:42.321 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:21:42.321 DEBUG:boto:Data: 2023-10-18T20:21:42.321 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.322 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.322 DEBUG:boto:Params: {} 2023-10-18T20:21:42.323 DEBUG:boto:Token: None 2023-10-18T20:21:42.323 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/d 2023-10-18T20:21:42.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wbUwzp6z1qbVhDtEabXHt9v8Wpw= 2023-10-18T20:21:42.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wbUwzp6z1qbVhDtEabXHt9v8Wpw=', 'Content-Length': '0'} 2023-10-18T20:21:42.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000947af33f560632b3-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.366 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.367 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.367 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:21:42.368 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:21:42.368 DEBUG:boto:Method: GET 2023-10-18T20:21:42.368 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:21:42.369 DEBUG:boto:Data: 2023-10-18T20:21:42.369 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:21:42.370 DEBUG:boto:Port: 8001 2023-10-18T20:21:42.370 DEBUG:boto:Params: {} 2023-10-18T20:21:42.371 DEBUG:boto:Token: None 2023-10-18T20:21:42.371 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.372 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T20:21:42.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T20:21:42.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d96be79510cf446-0065303e56-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.375 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:42.376 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.376 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.377 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:21:42.377 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:21:42.377 DEBUG:boto:Method: GET 2023-10-18T20:21:42.378 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:21:42.378 DEBUG:boto:Data: 2023-10-18T20:21:42.379 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.379 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.379 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.380 DEBUG:boto:Params: {} 2023-10-18T20:21:42.380 DEBUG:boto:Token: None 2023-10-18T20:21:42.380 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.381 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T20:21:42.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T20:21:42.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009db5466ad971893d-0065303e56-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.385 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:21:42.385 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-55 zones={a2, a1} 2023-10-18T20:21:42.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:21:42.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:21:42.386 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:21:42.440 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:42.440 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:42.486 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:21:42.486 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.486 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding": "None", 2023-10-18T20:21:42.487 INFO:teuthology.orchestra.run.smithi082.stdout: "current_index": { 2023-10-18T20:21:42.487 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 0, 2023-10-18T20:21:42.487 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.487 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "Normal", 2023-10-18T20:21:42.487 INFO:teuthology.orchestra.run.smithi082.stdout: "normal": { 2023-10-18T20:21:42.488 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 11, 2023-10-18T20:21:42.488 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:21:42.488 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.488 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.488 INFO:teuthology.orchestra.run.smithi082.stdout: "logs": [ 2023-10-18T20:21:42.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.489 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 0, 2023-10-18T20:21:42.489 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.489 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:21:42.489 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:21:42.490 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 0, 2023-10-18T20:21:42.490 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.490 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 11, 2023-10-18T20:21:42.490 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:21:42.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.491 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.491 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.491 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.491 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:42.491 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.492 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:21:42.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bofrnf-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:42.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bofrnf-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:42.493 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bofrnf-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:21:42.543 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:42.543 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:42.584 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:21:42.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.585 INFO:teuthology.orchestra.run.smithi082.stdout: "op_id": "1#00000000001.22.6", 2023-10-18T20:21:42.585 INFO:teuthology.orchestra.run.smithi082.stdout: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.6485583521340136943", 2023-10-18T20:21:42.585 INFO:teuthology.orchestra.run.smithi082.stdout: "op": "write", 2023-10-18T20:21:42.585 INFO:teuthology.orchestra.run.smithi082.stdout: "object": "a", 2023-10-18T20:21:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:21:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "state": "complete", 2023-10-18T20:21:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "index_ver": 1, 2023-10-18T20:21:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:21:00.946035145Z", 2023-10-18T20:21:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:21:42.587 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:21:42.587 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:21:42.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.587 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog_flags": 0, 2023-10-18T20:21:42.587 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned": false, 2023-10-18T20:21:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "", 2023-10-18T20:21:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "", 2023-10-18T20:21:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: "zones_trace": [ 2023-10-18T20:21:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24" 2023-10-18T20:21:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: "op_id": "5#00000000001.53.6", 2023-10-18T20:21:42.590 INFO:teuthology.orchestra.run.smithi082.stdout: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.7855387404454453039", 2023-10-18T20:21:42.590 INFO:teuthology.orchestra.run.smithi082.stdout: "op": "write", 2023-10-18T20:21:42.590 INFO:teuthology.orchestra.run.smithi082.stdout: "object": "d", 2023-10-18T20:21:42.590 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:21:42.590 INFO:teuthology.orchestra.run.smithi082.stdout: "state": "complete", 2023-10-18T20:21:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "index_ver": 1, 2023-10-18T20:21:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:21:01.038076432Z", 2023-10-18T20:21:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:21:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:21:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:21:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog_flags": 0, 2023-10-18T20:21:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned": false, 2023-10-18T20:21:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "", 2023-10-18T20:21:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "", 2023-10-18T20:21:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: "zones_trace": [ 2023-10-18T20:21:42.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.593 INFO:teuthology.orchestra.run.smithi082.stdout: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24" 2023-10-18T20:21:42.593 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.593 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:42.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.594 INFO:teuthology.orchestra.run.smithi082.stdout: "op_id": "8#00000000001.62.6", 2023-10-18T20:21:42.594 INFO:teuthology.orchestra.run.smithi082.stdout: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.15732007203871540178", 2023-10-18T20:21:42.594 INFO:teuthology.orchestra.run.smithi082.stdout: "op": "write", 2023-10-18T20:21:42.594 INFO:teuthology.orchestra.run.smithi082.stdout: "object": "c", 2023-10-18T20:21:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:21:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: "state": "complete", 2023-10-18T20:21:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: "index_ver": 1, 2023-10-18T20:21:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:21:01.019275931Z", 2023-10-18T20:21:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:21:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:21:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:21:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog_flags": 0, 2023-10-18T20:21:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned": false, 2023-10-18T20:21:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "", 2023-10-18T20:21:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "", 2023-10-18T20:21:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: "zones_trace": [ 2023-10-18T20:21:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24" 2023-10-18T20:21:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: "op_id": "10#00000000001.24.6", 2023-10-18T20:21:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.10906357668614388979", 2023-10-18T20:21:42.599 INFO:teuthology.orchestra.run.smithi082.stdout: "op": "write", 2023-10-18T20:21:42.599 INFO:teuthology.orchestra.run.smithi082.stdout: "object": "b", 2023-10-18T20:21:42.599 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:21:42.599 INFO:teuthology.orchestra.run.smithi082.stdout: "state": "complete", 2023-10-18T20:21:42.599 INFO:teuthology.orchestra.run.smithi082.stdout: "index_ver": 1, 2023-10-18T20:21:42.600 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:21:01.001180380Z", 2023-10-18T20:21:42.600 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:21:42.600 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:21:42.600 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:21:42.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.600 INFO:teuthology.orchestra.run.smithi082.stdout: "bilog_flags": 0, 2023-10-18T20:21:42.601 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned": false, 2023-10-18T20:21:42.601 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "", 2023-10-18T20:21:42.601 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "", 2023-10-18T20:21:42.601 INFO:teuthology.orchestra.run.smithi082.stdout: "zones_trace": [ 2023-10-18T20:21:42.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.601 INFO:teuthology.orchestra.run.smithi082.stdout: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24" 2023-10-18T20:21:42.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.602 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:42.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.602 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:21:42.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'bofrnf-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:21:42.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'bofrnf-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:21:42.604 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket bofrnf-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:21:42.653 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:42.653 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:42.692 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:21:42.692 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-55 2023-10-18T20:21:42.695 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 4 2023-10-18T20:21:42.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:21:42.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:21:42.729 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:21:42.775 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:42.776 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:42.816 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:21:42.816 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding": "None", 2023-10-18T20:21:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: "current_index": { 2023-10-18T20:21:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 1, 2023-10-18T20:21:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "Normal", 2023-10-18T20:21:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: "normal": { 2023-10-18T20:21:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:21:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:21:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: "logs": [ 2023-10-18T20:21:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 0, 2023-10-18T20:21:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:21:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:21:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 0, 2023-10-18T20:21:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 11, 2023-10-18T20:21:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:21:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:21:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:21:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 1, 2023-10-18T20:21:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:21:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:21:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 1, 2023-10-18T20:21:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:21:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:21:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:21:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:21:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:21:42.824 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:21:42.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bofrnf-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:42.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bofrnf-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:21:42.826 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bofrnf-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:21:42.874 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:42.874 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:42.915 INFO:teuthology.orchestra.run.smithi082.stdout:[] 2023-10-18T20:21:42.916 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.917 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.917 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.917 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:42.918 DEBUG:boto:Data: 2023-10-18T20:21:42.918 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.919 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.919 DEBUG:boto:Params: {} 2023-10-18T20:21:42.919 DEBUG:boto:Token: None 2023-10-18T20:21:42.920 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI= 2023-10-18T20:21:42.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI=', 'Content-Length': '0'} 2023-10-18T20:21:42.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000754800ce3f3ba7bc-0065303e56-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.923 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:21:42.924 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:21:42.924 DEBUG:boto:Method: PUT 2023-10-18T20:21:42.924 DEBUG:boto:Path: /bofrnf-55/e 2023-10-18T20:21:42.925 DEBUG:boto:Data: 2023-10-18T20:21:42.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:21:42.925 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.926 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.926 DEBUG:boto:Params: {} 2023-10-18T20:21:42.926 DEBUG:boto:Token: None 2023-10-18T20:21:42.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/e 2023-10-18T20:21:42.927 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GyTRN6mm4QW8oa+rsTZYaUXbjcM= 2023-10-18T20:21:42.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GyTRN6mm4QW8oa+rsTZYaUXbjcM='} 2023-10-18T20:21:42.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090d8adb23f86ed37-0065303e56-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.933 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.933 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.933 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.934 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:42.934 DEBUG:boto:Data: 2023-10-18T20:21:42.935 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.935 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.936 DEBUG:boto:Params: {} 2023-10-18T20:21:42.936 DEBUG:boto:Token: None 2023-10-18T20:21:42.936 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.937 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI= 2023-10-18T20:21:42.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI=', 'Content-Length': '0'} 2023-10-18T20:21:42.939 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051eed752673acffc-0065303e56-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.939 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:21:42.940 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:21:42.940 DEBUG:boto:Method: PUT 2023-10-18T20:21:42.941 DEBUG:boto:Path: /bofrnf-55/f 2023-10-18T20:21:42.941 DEBUG:boto:Data: 2023-10-18T20:21:42.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:21:42.942 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.942 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.942 DEBUG:boto:Params: {} 2023-10-18T20:21:42.943 DEBUG:boto:Token: None 2023-10-18T20:21:42.943 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/f 2023-10-18T20:21:42.944 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IC+1wr+iUxmCHm2pB5srWC6JWlY= 2023-10-18T20:21:42.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IC+1wr+iUxmCHm2pB5srWC6JWlY='} 2023-10-18T20:21:42.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000946c0b97b12c85b9-0065303e56-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.987 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:42.987 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:42.987 DEBUG:boto:Method: HEAD 2023-10-18T20:21:42.988 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:42.988 DEBUG:boto:Data: 2023-10-18T20:21:42.988 DEBUG:boto:Headers: {} 2023-10-18T20:21:42.989 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.989 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.990 DEBUG:boto:Params: {} 2023-10-18T20:21:42.990 DEBUG:boto:Token: None 2023-10-18T20:21:42.990 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/ 2023-10-18T20:21:42.991 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI= 2023-10-18T20:21:42.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI=', 'Content-Length': '0'} 2023-10-18T20:21:42.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090557a50764eec5c-0065303e56-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:42.993 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:21:42.994 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:21:42.994 DEBUG:boto:Method: PUT 2023-10-18T20:21:42.995 DEBUG:boto:Path: /bofrnf-55/g 2023-10-18T20:21:42.995 DEBUG:boto:Data: 2023-10-18T20:21:42.995 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:21:42.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:42.996 DEBUG:boto:Port: 8000 2023-10-18T20:21:42.996 DEBUG:boto:Params: {} 2023-10-18T20:21:42.997 DEBUG:boto:Token: None 2023-10-18T20:21:42.997 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:21:42 GMT /bofrnf-55/g 2023-10-18T20:21:42.997 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:trWagEhN+Z4ye27tKucu+Tsyc2w= 2023-10-18T20:21:42.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:trWagEhN+Z4ye27tKucu+Tsyc2w='} 2023-10-18T20:21:43.003 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5da45b902c93e52-0065303e56-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:43.003 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:21:43.004 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:21:43.004 DEBUG:boto:Method: HEAD 2023-10-18T20:21:43.005 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:21:43.005 DEBUG:boto:Data: 2023-10-18T20:21:43.005 DEBUG:boto:Headers: {} 2023-10-18T20:21:43.006 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:43.006 DEBUG:boto:Port: 8000 2023-10-18T20:21:43.006 DEBUG:boto:Params: {} 2023-10-18T20:21:43.007 DEBUG:boto:Token: None 2023-10-18T20:21:43.007 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:21:43 GMT /bofrnf-55/ 2023-10-18T20:21:43.007 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Y0T0shiz+ICcGEeDEYb8kX+gGU= 2023-10-18T20:21:43.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Y0T0shiz+ICcGEeDEYb8kX+gGU=', 'Content-Length': '0'} 2023-10-18T20:21:43.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018f11b17168ef4e1-0065303e57-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:43.010 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:21:43.010 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:21:43.011 DEBUG:boto:Method: PUT 2023-10-18T20:21:43.011 DEBUG:boto:Path: /bofrnf-55/h 2023-10-18T20:21:43.011 DEBUG:boto:Data: 2023-10-18T20:21:43.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:21:43.012 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:21:43.012 DEBUG:boto:Port: 8000 2023-10-18T20:21:43.013 DEBUG:boto:Params: {} 2023-10-18T20:21:43.013 DEBUG:boto:Token: None 2023-10-18T20:21:43.013 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:21:43 GMT /bofrnf-55/h 2023-10-18T20:21:43.014 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:coFmLIDH4CbCEGkgeynAPaRhoTY= 2023-10-18T20:21:43.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:coFmLIDH4CbCEGkgeynAPaRhoTY='} 2023-10-18T20:21:43.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e92a86e8296f51a-0065303e57-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:21:43.018 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-55 2023-10-18T20:21:43.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:21:43.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:21:43.020 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:21:43.075 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:21:43.075 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:21:43.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:43.107+0000 7f5400246780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T20:21:48.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:48.107+0000 7f5400246780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T20:21:53.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:53.107+0000 7f5400246780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T20:21:58.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:21:58.108+0000 7f5400246780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T20:22:03.111 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:03.110+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:03.112 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:22:03.112 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:08.112 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:08.111+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:08.113 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:22:08.113 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:13.113 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:13.112+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:13.114 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:22:13.114 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:18.114 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:18.113+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:18.115 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:22:18.115 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:23.115 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:23.113+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:23.115 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:23.115 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:28.116 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:28.114+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:28.116 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:28.116 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:33.116 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:33.115+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:33.117 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:33.117 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:38.117 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:38.115+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:38.117 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:38.118 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:43.118 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:43.116+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:43.118 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:43.119 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:48.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:48.116+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:48.119 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:48.119 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:53.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:53.118+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:53.120 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:53.120 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:58.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:22:58.119+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:22:58.121 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:22:58.121 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:03.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:03.120+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:03.122 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:03.122 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:08.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:08.121+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:08.122 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:08.123 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:13.123 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:13.122+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:13.123 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:13.123 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:18.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:18.122+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:18.124 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:18.124 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:23.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:23.123+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:23.125 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:23.125 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:28.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:28.124+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:28.126 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:28.126 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:33.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:33.124+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:33.127 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:33.127 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:38.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:38.125+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:38.128 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:38.128 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:43.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:43.126+0000 7f5400246780 1 bucket sync caught up with source: 2023-10-18T20:23:43.128 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:43.128 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T20:23:43.129 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:43.126+0000 7f5400246780 0 bucket checkpoint complete 2023-10-18T20:23:43.145 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-55 2023-10-18T20:23:43.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:23:43.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:23:43.147 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:23:43.208 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:23:43.208 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:23:43.245 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:43.243+0000 7f353e08f780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:43.246 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000001.33.6] 2023-10-18T20:23:43.246 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T20:23:48.247 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:48.245+0000 7f353e08f780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:48.247 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000001.33.6] 2023-10-18T20:23:48.248 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T20:23:53.248 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:53.247+0000 7f353e08f780 1 waiting for incremental sync to catch up: 2023-10-18T20:23:53.249 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000001.33.6] 2023-10-18T20:23:53.250 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T20:23:58.250 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:58.248+0000 7f353e08f780 1 bucket sync caught up with source: 2023-10-18T20:23:58.250 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T20:23:58.250 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T20:23:58.251 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:23:58.248+0000 7f353e08f780 0 bucket checkpoint complete 2023-10-18T20:23:58.266 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-55 zones={a2, a1} 2023-10-18T20:23:58.266 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.267 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.267 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.268 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:23:58.268 DEBUG:boto:Data: 2023-10-18T20:23:58.268 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.269 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.270 DEBUG:boto:Params: {} 2023-10-18T20:23:58.270 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:23:58.271 DEBUG:boto:Token: None 2023-10-18T20:23:58.271 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.272 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f8J6oA462jMsqfFlKCxGlZ03jWg= 2023-10-18T20:23:58.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f8J6oA462jMsqfFlKCxGlZ03jWg=', 'Content-Length': '0'} 2023-10-18T20:23:58.277 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005eb0e81be5ead3d1-0065303ede-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.277 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.278 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.278 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.278 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:23:58.279 DEBUG:boto:Data: 2023-10-18T20:23:58.279 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.280 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.280 DEBUG:boto:Params: {} 2023-10-18T20:23:58.281 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:23:58.281 DEBUG:boto:Token: None 2023-10-18T20:23:58.281 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.282 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f8J6oA462jMsqfFlKCxGlZ03jWg= 2023-10-18T20:23:58.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f8J6oA462jMsqfFlKCxGlZ03jWg=', 'Content-Length': '0'} 2023-10-18T20:23:58.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060ec187054c20c08-0065303ede-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.286 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:23:58.286 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.287 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.287 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.287 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.288 DEBUG:boto:Method: GET 2023-10-18T20:23:58.288 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.289 DEBUG:boto:Data: 2023-10-18T20:23:58.289 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.289 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.290 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.290 DEBUG:boto:Params: {} 2023-10-18T20:23:58.291 DEBUG:boto:Token: None 2023-10-18T20:23:58.291 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/?versions 2023-10-18T20:23:58.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU= 2023-10-18T20:23:58.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU=', 'Content-Length': '0'} 2023-10-18T20:23:58.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004833dab8e7eb3522-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.295 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.296 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:23:58.296 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:23:58.296 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:23:58.297 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:23:58.297 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-18T20:23:58.297 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-18T20:23:58.298 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-18T20:23:58.298 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-18T20:23:58.299 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:23:58.299 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.299 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.300 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.300 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.300 DEBUG:boto:Method: GET 2023-10-18T20:23:58.301 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.301 DEBUG:boto:Data: 2023-10-18T20:23:58.302 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.302 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.303 DEBUG:boto:Params: {} 2023-10-18T20:23:58.303 DEBUG:boto:Token: None 2023-10-18T20:23:58.304 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/?versions 2023-10-18T20:23:58.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU= 2023-10-18T20:23:58.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU=', 'Content-Length': '0'} 2023-10-18T20:23:58.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000877859b63eb4671b-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.307 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.308 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:23:58.308 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:23:58.309 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:23:58.309 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:23:58.309 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-18T20:23:58.310 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-18T20:23:58.310 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-18T20:23:58.310 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-18T20:23:58.311 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.311 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.312 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.312 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.312 DEBUG:boto:Method: GET 2023-10-18T20:23:58.313 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.313 DEBUG:boto:Data: 2023-10-18T20:23:58.313 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.314 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.314 DEBUG:boto:Params: {} 2023-10-18T20:23:58.315 DEBUG:boto:Token: None 2023-10-18T20:23:58.315 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/?versions 2023-10-18T20:23:58.316 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU= 2023-10-18T20:23:58.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU=', 'Content-Length': '0'} 2023-10-18T20:23:58.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fca159179740267-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.319 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.320 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.321 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.321 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.321 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.322 DEBUG:boto:Method: GET 2023-10-18T20:23:58.322 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:23:58.322 DEBUG:boto:Data: 2023-10-18T20:23:58.323 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.323 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.324 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.324 DEBUG:boto:Params: {} 2023-10-18T20:23:58.324 DEBUG:boto:Token: None 2023-10-18T20:23:58.325 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/?versions 2023-10-18T20:23:58.325 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU= 2023-10-18T20:23:58.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU=', 'Content-Length': '0'} 2023-10-18T20:23:58.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003330c97c4335ab4b-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.328 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:23:58.330 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.330 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.331 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.331 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.331 DEBUG:boto:Method: GET 2023-10-18T20:23:58.332 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.332 DEBUG:boto:Data: 2023-10-18T20:23:58.333 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.333 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.334 DEBUG:boto:Params: {} 2023-10-18T20:23:58.334 DEBUG:boto:Token: None 2023-10-18T20:23:58.335 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.335 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM= 2023-10-18T20:23:58.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM=', 'Content-Length': '0'} 2023-10-18T20:23:58.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6bf22d9e23a641-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.338 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.339 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.339 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.339 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.340 DEBUG:boto:Method: GET 2023-10-18T20:23:58.340 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.341 DEBUG:boto:Data: 2023-10-18T20:23:58.341 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.341 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.342 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.342 DEBUG:boto:Params: {} 2023-10-18T20:23:58.343 DEBUG:boto:Token: None 2023-10-18T20:23:58.343 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.344 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM= 2023-10-18T20:23:58.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM=', 'Content-Length': '0'} 2023-10-18T20:23:58.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b6751e78ac8ea3d-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.347 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.347 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.348 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.348 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.348 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.349 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.349 DEBUG:boto:Data: 2023-10-18T20:23:58.349 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.350 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.350 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.351 DEBUG:boto:Params: {} 2023-10-18T20:23:58.351 DEBUG:boto:Token: None 2023-10-18T20:23:58.351 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.352 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mds7q/yUg+cQBVJSEMLvMtV1TWI= 2023-10-18T20:23:58.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mds7q/yUg+cQBVJSEMLvMtV1TWI=', 'Content-Length': '0'} 2023-10-18T20:23:58.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005689680253f2b795-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.355 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.355 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.355 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.356 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.356 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.356 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.357 DEBUG:boto:Data: 2023-10-18T20:23:58.357 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.358 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.358 DEBUG:boto:Params: {} 2023-10-18T20:23:58.359 DEBUG:boto:Token: None 2023-10-18T20:23:58.359 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mds7q/yUg+cQBVJSEMLvMtV1TWI= 2023-10-18T20:23:58.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mds7q/yUg+cQBVJSEMLvMtV1TWI=', 'Content-Length': '0'} 2023-10-18T20:23:58.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002441e7adb49b6923-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:23:58.363 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.363 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.364 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.364 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.365 DEBUG:boto:Method: GET 2023-10-18T20:23:58.365 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.365 DEBUG:boto:Data: 2023-10-18T20:23:58.366 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.366 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.366 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.367 DEBUG:boto:Params: {} 2023-10-18T20:23:58.367 DEBUG:boto:Token: None 2023-10-18T20:23:58.368 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.368 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM= 2023-10-18T20:23:58.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM=', 'Content-Length': '0'} 2023-10-18T20:23:58.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed3de3c63f75eeb7-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.414 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.415 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.415 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.416 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:23:58.416 DEBUG:boto:Method: GET 2023-10-18T20:23:58.416 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.417 DEBUG:boto:Data: 2023-10-18T20:23:58.417 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.418 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.418 DEBUG:boto:Params: {} 2023-10-18T20:23:58.419 DEBUG:boto:Token: None 2023-10-18T20:23:58.419 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:23:58.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM= 2023-10-18T20:23:58.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM=', 'Content-Length': '0'} 2023-10-18T20:23:58.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbdf432b0deb3953-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.423 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.423 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.424 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:23:58.424 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:23:58.424 DEBUG:boto:Method: GET 2023-10-18T20:23:58.425 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:23:58.425 DEBUG:boto:Data: 2023-10-18T20:23:58.425 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.426 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.427 DEBUG:boto:Params: {} 2023-10-18T20:23:58.427 DEBUG:boto:Token: None 2023-10-18T20:23:58.427 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a2cd8aae3f4f2a-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.431 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.432 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.432 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.433 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:23:58.433 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:23:58.434 DEBUG:boto:Method: GET 2023-10-18T20:23:58.434 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:23:58.434 DEBUG:boto:Data: 2023-10-18T20:23:58.435 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.435 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.436 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.436 DEBUG:boto:Params: {} 2023-10-18T20:23:58.436 DEBUG:boto:Token: None 2023-10-18T20:23:58.437 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.437 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef4fea989229a130-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.482 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.483 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.483 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.484 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.484 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:23:58.484 DEBUG:boto:Data: 2023-10-18T20:23:58.485 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.485 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.485 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.486 DEBUG:boto:Params: {} 2023-10-18T20:23:58.486 DEBUG:boto:Token: None 2023-10-18T20:23:58.487 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a 2023-10-18T20:23:58.487 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XYIJc8CVeP+7mipjKmcAuxo4vSk= 2023-10-18T20:23:58.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XYIJc8CVeP+7mipjKmcAuxo4vSk=', 'Content-Length': '0'} 2023-10-18T20:23:58.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebd81d05c3598f3-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.490 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.491 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.491 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.491 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:23:58.492 DEBUG:boto:Data: 2023-10-18T20:23:58.492 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.492 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.493 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.493 DEBUG:boto:Params: {} 2023-10-18T20:23:58.493 DEBUG:boto:Token: None 2023-10-18T20:23:58.494 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a 2023-10-18T20:23:58.494 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XYIJc8CVeP+7mipjKmcAuxo4vSk= 2023-10-18T20:23:58.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XYIJc8CVeP+7mipjKmcAuxo4vSk=', 'Content-Length': '0'} 2023-10-18T20:23:58.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b364a417db127c23-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:23:58.498 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.498 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.499 DEBUG:boto:Method: GET 2023-10-18T20:23:58.499 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:23:58.499 DEBUG:boto:Data: 2023-10-18T20:23:58.500 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.500 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.501 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.501 DEBUG:boto:Params: {} 2023-10-18T20:23:58.501 DEBUG:boto:Token: None 2023-10-18T20:23:58.502 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a 2023-10-18T20:23:58.502 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KeCUNxiv1G4GrF+n2z4FEAW035M= 2023-10-18T20:23:58.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KeCUNxiv1G4GrF+n2z4FEAW035M=', 'Content-Length': '0'} 2023-10-18T20:23:58.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a736e715ee21c711-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.505 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:23:58.505 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:23:58.506 DEBUG:boto:Method: GET 2023-10-18T20:23:58.506 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:23:58.507 DEBUG:boto:Data: 2023-10-18T20:23:58.507 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.508 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.508 DEBUG:boto:Params: {} 2023-10-18T20:23:58.509 DEBUG:boto:Token: None 2023-10-18T20:23:58.509 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/a 2023-10-18T20:23:58.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KeCUNxiv1G4GrF+n2z4FEAW035M= 2023-10-18T20:23:58.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KeCUNxiv1G4GrF+n2z4FEAW035M=', 'Content-Length': '0'} 2023-10-18T20:23:58.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea8a926f3500e3cc-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.513 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.513 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.514 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:23:58.514 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:23:58.514 DEBUG:boto:Method: GET 2023-10-18T20:23:58.515 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:23:58.515 DEBUG:boto:Data: 2023-10-18T20:23:58.515 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.516 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.517 DEBUG:boto:Params: {} 2023-10-18T20:23:58.517 DEBUG:boto:Token: None 2023-10-18T20:23:58.517 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.518 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008be1e3c086d9b22b-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.521 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.522 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.522 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.523 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:23:58.523 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:23:58.524 DEBUG:boto:Method: GET 2023-10-18T20:23:58.524 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:23:58.524 DEBUG:boto:Data: 2023-10-18T20:23:58.525 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.526 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.526 DEBUG:boto:Params: {} 2023-10-18T20:23:58.526 DEBUG:boto:Token: None 2023-10-18T20:23:58.527 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.527 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1f2c860acbb17b4-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.531 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:23:58.532 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.532 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.532 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.533 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.533 DEBUG:boto:Method: GET 2023-10-18T20:23:58.534 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.534 DEBUG:boto:Data: 2023-10-18T20:23:58.534 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.535 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.535 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.536 DEBUG:boto:Params: {} 2023-10-18T20:23:58.536 DEBUG:boto:Token: None 2023-10-18T20:23:58.536 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.537 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g= 2023-10-18T20:23:58.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g=', 'Content-Length': '0'} 2023-10-18T20:23:58.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de10cfa24b1fa0be-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.540 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.540 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.541 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.541 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.541 DEBUG:boto:Method: GET 2023-10-18T20:23:58.542 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.542 DEBUG:boto:Data: 2023-10-18T20:23:58.542 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.543 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.543 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.544 DEBUG:boto:Params: {} 2023-10-18T20:23:58.544 DEBUG:boto:Token: None 2023-10-18T20:23:58.544 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g= 2023-10-18T20:23:58.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g=', 'Content-Length': '0'} 2023-10-18T20:23:58.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b96991f0a0f57c-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.548 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.548 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.548 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.549 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.549 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.550 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.550 DEBUG:boto:Data: 2023-10-18T20:23:58.550 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.551 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.551 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.551 DEBUG:boto:Params: {} 2023-10-18T20:23:58.552 DEBUG:boto:Token: None 2023-10-18T20:23:58.552 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.552 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z2A1w71eJB8L1ovmdhHOGpIY6t8= 2023-10-18T20:23:58.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z2A1w71eJB8L1ovmdhHOGpIY6t8=', 'Content-Length': '0'} 2023-10-18T20:23:58.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7eee11c68726c7-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.555 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.556 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.556 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.556 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.557 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.557 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.557 DEBUG:boto:Data: 2023-10-18T20:23:58.558 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.559 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.559 DEBUG:boto:Params: {} 2023-10-18T20:23:58.559 DEBUG:boto:Token: None 2023-10-18T20:23:58.560 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z2A1w71eJB8L1ovmdhHOGpIY6t8= 2023-10-18T20:23:58.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z2A1w71eJB8L1ovmdhHOGpIY6t8=', 'Content-Length': '0'} 2023-10-18T20:23:58.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007559d99ee65a5b0-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:23:58.563 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.564 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.564 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.565 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.565 DEBUG:boto:Method: GET 2023-10-18T20:23:58.565 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.566 DEBUG:boto:Data: 2023-10-18T20:23:58.566 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.567 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.567 DEBUG:boto:Params: {} 2023-10-18T20:23:58.568 DEBUG:boto:Token: None 2023-10-18T20:23:58.568 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g= 2023-10-18T20:23:58.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g=', 'Content-Length': '0'} 2023-10-18T20:23:58.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849ecec599b9292a-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.614 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.615 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.615 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.615 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:23:58.616 DEBUG:boto:Method: GET 2023-10-18T20:23:58.616 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.617 DEBUG:boto:Data: 2023-10-18T20:23:58.617 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.617 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.618 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.618 DEBUG:boto:Params: {} 2023-10-18T20:23:58.619 DEBUG:boto:Token: None 2023-10-18T20:23:58.619 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:23:58.619 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g= 2023-10-18T20:23:58.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g=', 'Content-Length': '0'} 2023-10-18T20:23:58.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cee3574b23660161-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.622 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.623 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.623 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:23:58.624 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:23:58.624 DEBUG:boto:Method: GET 2023-10-18T20:23:58.624 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:23:58.625 DEBUG:boto:Data: 2023-10-18T20:23:58.625 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.626 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.626 DEBUG:boto:Params: {} 2023-10-18T20:23:58.627 DEBUG:boto:Token: None 2023-10-18T20:23:58.627 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.627 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a66d9255ad90c51d-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.631 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.632 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.632 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.632 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:23:58.633 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:23:58.633 DEBUG:boto:Method: GET 2023-10-18T20:23:58.634 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:23:58.634 DEBUG:boto:Data: 2023-10-18T20:23:58.634 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.635 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.635 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.635 DEBUG:boto:Params: {} 2023-10-18T20:23:58.636 DEBUG:boto:Token: None 2023-10-18T20:23:58.636 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.637 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7331a7cbb56ba4d-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.640 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.640 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.641 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.641 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.641 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:23:58.642 DEBUG:boto:Data: 2023-10-18T20:23:58.642 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.642 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.643 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.643 DEBUG:boto:Params: {} 2023-10-18T20:23:58.644 DEBUG:boto:Token: None 2023-10-18T20:23:58.644 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b 2023-10-18T20:23:58.644 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qd598u9h42m0FbFvGW7dDqUn30g= 2023-10-18T20:23:58.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qd598u9h42m0FbFvGW7dDqUn30g=', 'Content-Length': '0'} 2023-10-18T20:23:58.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c72f1632ac02081-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.647 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.648 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.648 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.648 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:23:58.649 DEBUG:boto:Data: 2023-10-18T20:23:58.649 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.650 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.650 DEBUG:boto:Params: {} 2023-10-18T20:23:58.651 DEBUG:boto:Token: None 2023-10-18T20:23:58.651 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b 2023-10-18T20:23:58.651 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qd598u9h42m0FbFvGW7dDqUn30g= 2023-10-18T20:23:58.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qd598u9h42m0FbFvGW7dDqUn30g=', 'Content-Length': '0'} 2023-10-18T20:23:58.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011fbeb689c48d9cd-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:23:58.654 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.655 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.655 DEBUG:boto:Method: GET 2023-10-18T20:23:58.656 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:23:58.656 DEBUG:boto:Data: 2023-10-18T20:23:58.656 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.657 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.657 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.657 DEBUG:boto:Params: {} 2023-10-18T20:23:58.658 DEBUG:boto:Token: None 2023-10-18T20:23:58.658 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b 2023-10-18T20:23:58.659 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8soovHvLKFAjoFsWC5uDD+YKud4= 2023-10-18T20:23:58.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8soovHvLKFAjoFsWC5uDD+YKud4=', 'Content-Length': '0'} 2023-10-18T20:23:58.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea8dcf9c5e42f5d-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.702 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:23:58.703 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:23:58.703 DEBUG:boto:Method: GET 2023-10-18T20:23:58.703 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:23:58.704 DEBUG:boto:Data: 2023-10-18T20:23:58.704 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.705 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.705 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.705 DEBUG:boto:Params: {} 2023-10-18T20:23:58.706 DEBUG:boto:Token: None 2023-10-18T20:23:58.706 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/b 2023-10-18T20:23:58.707 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8soovHvLKFAjoFsWC5uDD+YKud4= 2023-10-18T20:23:58.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8soovHvLKFAjoFsWC5uDD+YKud4=', 'Content-Length': '0'} 2023-10-18T20:23:58.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae018d3b97bb055f-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.710 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.710 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.711 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:23:58.711 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:23:58.711 DEBUG:boto:Method: GET 2023-10-18T20:23:58.712 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:23:58.712 DEBUG:boto:Data: 2023-10-18T20:23:58.712 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.713 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.713 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.714 DEBUG:boto:Params: {} 2023-10-18T20:23:58.714 DEBUG:boto:Token: None 2023-10-18T20:23:58.715 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.715 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b07f7ea042c1dab-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.719 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.719 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.720 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.720 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:23:58.720 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:23:58.721 DEBUG:boto:Method: GET 2023-10-18T20:23:58.721 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:23:58.721 DEBUG:boto:Data: 2023-10-18T20:23:58.722 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.722 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.723 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.723 DEBUG:boto:Params: {} 2023-10-18T20:23:58.723 DEBUG:boto:Token: None 2023-10-18T20:23:58.724 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.724 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000255deedb5e61c5ed-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.727 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:23:58.728 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.729 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.729 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.729 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.730 DEBUG:boto:Method: GET 2023-10-18T20:23:58.730 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.731 DEBUG:boto:Data: 2023-10-18T20:23:58.731 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.731 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.732 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.732 DEBUG:boto:Params: {} 2023-10-18T20:23:58.733 DEBUG:boto:Token: None 2023-10-18T20:23:58.733 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.733 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ= 2023-10-18T20:23:58.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ=', 'Content-Length': '0'} 2023-10-18T20:23:58.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ce3bec564ac3a9-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.737 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.737 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.738 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.738 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.738 DEBUG:boto:Method: GET 2023-10-18T20:23:58.739 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.739 DEBUG:boto:Data: 2023-10-18T20:23:58.740 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.740 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.740 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.741 DEBUG:boto:Params: {} 2023-10-18T20:23:58.741 DEBUG:boto:Token: None 2023-10-18T20:23:58.742 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.742 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ= 2023-10-18T20:23:58.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ=', 'Content-Length': '0'} 2023-10-18T20:23:58.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025305484fb3926e3-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.745 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.745 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.746 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.746 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.747 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.747 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.747 DEBUG:boto:Data: 2023-10-18T20:23:58.748 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.748 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.748 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.749 DEBUG:boto:Params: {} 2023-10-18T20:23:58.749 DEBUG:boto:Token: None 2023-10-18T20:23:58.750 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.750 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7Yrpln1bqgj64a8dGB85nT1seco= 2023-10-18T20:23:58.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7Yrpln1bqgj64a8dGB85nT1seco=', 'Content-Length': '0'} 2023-10-18T20:23:58.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc76c32a805cd20-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.753 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.753 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.754 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.754 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.755 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.755 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.755 DEBUG:boto:Data: 2023-10-18T20:23:58.756 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.756 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.756 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.757 DEBUG:boto:Params: {} 2023-10-18T20:23:58.757 DEBUG:boto:Token: None 2023-10-18T20:23:58.758 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.758 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7Yrpln1bqgj64a8dGB85nT1seco= 2023-10-18T20:23:58.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7Yrpln1bqgj64a8dGB85nT1seco=', 'Content-Length': '0'} 2023-10-18T20:23:58.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a10c7453ed39dfb-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:23:58.761 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.762 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.762 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.763 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.763 DEBUG:boto:Method: GET 2023-10-18T20:23:58.763 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.764 DEBUG:boto:Data: 2023-10-18T20:23:58.764 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.765 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.765 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.765 DEBUG:boto:Params: {} 2023-10-18T20:23:58.766 DEBUG:boto:Token: None 2023-10-18T20:23:58.766 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.767 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ= 2023-10-18T20:23:58.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ=', 'Content-Length': '0'} 2023-10-18T20:23:58.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d307facc171e2f-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.770 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.770 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.771 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.771 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:23:58.771 DEBUG:boto:Method: GET 2023-10-18T20:23:58.772 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.772 DEBUG:boto:Data: 2023-10-18T20:23:58.773 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.773 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.774 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.774 DEBUG:boto:Params: {} 2023-10-18T20:23:58.774 DEBUG:boto:Token: None 2023-10-18T20:23:58.775 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:23:58.775 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ= 2023-10-18T20:23:58.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ=', 'Content-Length': '0'} 2023-10-18T20:23:58.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000951b9ed7bc46d6ed-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.779 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.779 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.779 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:23:58.780 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:23:58.780 DEBUG:boto:Method: GET 2023-10-18T20:23:58.780 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:23:58.781 DEBUG:boto:Data: 2023-10-18T20:23:58.781 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.782 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.782 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.783 DEBUG:boto:Params: {} 2023-10-18T20:23:58.783 DEBUG:boto:Token: None 2023-10-18T20:23:58.783 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.784 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62dff7773096b39-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.788 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.788 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.789 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.789 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:23:58.790 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:23:58.790 DEBUG:boto:Method: GET 2023-10-18T20:23:58.790 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:23:58.791 DEBUG:boto:Data: 2023-10-18T20:23:58.791 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.792 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.792 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.792 DEBUG:boto:Params: {} 2023-10-18T20:23:58.793 DEBUG:boto:Token: None 2023-10-18T20:23:58.793 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.794 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000145554170ab26a27-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.796 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.797 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.797 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.798 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.798 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:23:58.799 DEBUG:boto:Data: 2023-10-18T20:23:58.799 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.800 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.800 DEBUG:boto:Params: {} 2023-10-18T20:23:58.800 DEBUG:boto:Token: None 2023-10-18T20:23:58.801 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c 2023-10-18T20:23:58.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ccGS0r8Ie0YPSFUBVynwu7MXtgo= 2023-10-18T20:23:58.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ccGS0r8Ie0YPSFUBVynwu7MXtgo=', 'Content-Length': '0'} 2023-10-18T20:23:58.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e13c25246267e192-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.804 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.805 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.805 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.805 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:23:58.806 DEBUG:boto:Data: 2023-10-18T20:23:58.806 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.807 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.807 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.807 DEBUG:boto:Params: {} 2023-10-18T20:23:58.808 DEBUG:boto:Token: None 2023-10-18T20:23:58.808 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c 2023-10-18T20:23:58.809 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ccGS0r8Ie0YPSFUBVynwu7MXtgo= 2023-10-18T20:23:58.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ccGS0r8Ie0YPSFUBVynwu7MXtgo=', 'Content-Length': '0'} 2023-10-18T20:23:58.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000163d6630590ab3fb-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:23:58.812 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.812 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.813 DEBUG:boto:Method: GET 2023-10-18T20:23:58.813 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:23:58.814 DEBUG:boto:Data: 2023-10-18T20:23:58.814 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.815 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.815 DEBUG:boto:Params: {} 2023-10-18T20:23:58.816 DEBUG:boto:Token: None 2023-10-18T20:23:58.816 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c 2023-10-18T20:23:58.817 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nXYQBHClFfnxdpyqIt6bcQzRuCQ= 2023-10-18T20:23:58.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nXYQBHClFfnxdpyqIt6bcQzRuCQ=', 'Content-Length': '0'} 2023-10-18T20:23:58.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef69a4fe4415599-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.862 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:23:58.863 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:23:58.863 DEBUG:boto:Method: GET 2023-10-18T20:23:58.863 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:23:58.864 DEBUG:boto:Data: 2023-10-18T20:23:58.864 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.865 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.865 DEBUG:boto:Params: {} 2023-10-18T20:23:58.866 DEBUG:boto:Token: None 2023-10-18T20:23:58.866 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/c 2023-10-18T20:23:58.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nXYQBHClFfnxdpyqIt6bcQzRuCQ= 2023-10-18T20:23:58.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nXYQBHClFfnxdpyqIt6bcQzRuCQ=', 'Content-Length': '0'} 2023-10-18T20:23:58.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e387901d23f23ae7-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.870 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.870 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.870 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:23:58.871 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:23:58.871 DEBUG:boto:Method: GET 2023-10-18T20:23:58.872 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:23:58.872 DEBUG:boto:Data: 2023-10-18T20:23:58.872 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.873 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.874 DEBUG:boto:Params: {} 2023-10-18T20:23:58.874 DEBUG:boto:Token: None 2023-10-18T20:23:58.875 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.875 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb9707a32293ff6f-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.879 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.879 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.880 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.880 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:23:58.880 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:23:58.881 DEBUG:boto:Method: GET 2023-10-18T20:23:58.881 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:23:58.881 DEBUG:boto:Data: 2023-10-18T20:23:58.882 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.883 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.883 DEBUG:boto:Params: {} 2023-10-18T20:23:58.883 DEBUG:boto:Token: None 2023-10-18T20:23:58.884 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6b0c19c8232d57-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.887 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:58.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:23:58.888 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:58.889 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:58.889 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.889 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.890 DEBUG:boto:Method: GET 2023-10-18T20:23:58.890 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.891 DEBUG:boto:Data: 2023-10-18T20:23:58.891 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.891 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.892 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.892 DEBUG:boto:Params: {} 2023-10-18T20:23:58.893 DEBUG:boto:Token: None 2023-10-18T20:23:58.893 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.894 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY= 2023-10-18T20:23:58.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY=', 'Content-Length': '0'} 2023-10-18T20:23:58.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051cb853f4a055d5a-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.897 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:58.897 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:58.897 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.898 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.898 DEBUG:boto:Method: GET 2023-10-18T20:23:58.899 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.899 DEBUG:boto:Data: 2023-10-18T20:23:58.900 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.900 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.901 DEBUG:boto:Params: {} 2023-10-18T20:23:58.901 DEBUG:boto:Token: None 2023-10-18T20:23:58.902 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.902 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY= 2023-10-18T20:23:58.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY=', 'Content-Length': '0'} 2023-10-18T20:23:58.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd8b9e175c451c9e-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.946 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:58.947 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:58.947 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.947 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.948 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.948 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.948 DEBUG:boto:Data: 2023-10-18T20:23:58.949 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.950 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.950 DEBUG:boto:Params: {} 2023-10-18T20:23:58.950 DEBUG:boto:Token: None 2023-10-18T20:23:58.951 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TUvRScOz5i5q8iRORQ2eJLNXnCc= 2023-10-18T20:23:58.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TUvRScOz5i5q8iRORQ2eJLNXnCc=', 'Content-Length': '0'} 2023-10-18T20:23:58.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697fcac8e3b72d8f-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.954 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:58.954 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:58.955 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.955 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.956 DEBUG:boto:Method: HEAD 2023-10-18T20:23:58.956 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.956 DEBUG:boto:Data: 2023-10-18T20:23:58.957 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.957 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.958 DEBUG:boto:Params: {} 2023-10-18T20:23:58.958 DEBUG:boto:Token: None 2023-10-18T20:23:58.959 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TUvRScOz5i5q8iRORQ2eJLNXnCc= 2023-10-18T20:23:58.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TUvRScOz5i5q8iRORQ2eJLNXnCc=', 'Content-Length': '0'} 2023-10-18T20:23:58.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004909b62f150a0819-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:23:58.962 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:58.963 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:58.963 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.964 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.964 DEBUG:boto:Method: GET 2023-10-18T20:23:58.964 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.965 DEBUG:boto:Data: 2023-10-18T20:23:58.965 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.966 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.966 DEBUG:boto:Params: {} 2023-10-18T20:23:58.967 DEBUG:boto:Token: None 2023-10-18T20:23:58.967 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY= 2023-10-18T20:23:58.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY=', 'Content-Length': '0'} 2023-10-18T20:23:58.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000306a6a5bf330878d-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.971 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:58.971 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:58.971 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.972 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:23:58.972 DEBUG:boto:Method: GET 2023-10-18T20:23:58.973 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.973 DEBUG:boto:Data: 2023-10-18T20:23:58.973 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:58.974 DEBUG:boto:Port: 8000 2023-10-18T20:23:58.975 DEBUG:boto:Params: {} 2023-10-18T20:23:58.975 DEBUG:boto:Token: None 2023-10-18T20:23:58.976 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:23:58.976 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY= 2023-10-18T20:23:58.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY=', 'Content-Length': '0'} 2023-10-18T20:23:58.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002608140d20991d10-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:58.979 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:58.979 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:58.980 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:23:58.980 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:23:58.980 DEBUG:boto:Method: GET 2023-10-18T20:23:58.981 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:23:58.981 DEBUG:boto:Data: 2023-10-18T20:23:58.981 DEBUG:boto:Headers: {} 2023-10-18T20:23:58.982 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:58.982 DEBUG:boto:Port: 8001 2023-10-18T20:23:58.983 DEBUG:boto:Params: {} 2023-10-18T20:23:58.983 DEBUG:boto:Token: None 2023-10-18T20:23:58.984 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:58 GMT /bofrnf-55/ 2023-10-18T20:23:58.984 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T20:23:58.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T20:23:58.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9722fbfa996c98-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.030 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.031 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.031 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.032 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:23:59.032 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:23:59.033 DEBUG:boto:Method: GET 2023-10-18T20:23:59.033 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:23:59.033 DEBUG:boto:Data: 2023-10-18T20:23:59.034 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.034 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.034 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.035 DEBUG:boto:Params: {} 2023-10-18T20:23:59.035 DEBUG:boto:Token: None 2023-10-18T20:23:59.036 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cac177959c254ae-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.039 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.040 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:59.040 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:59.041 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.041 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:23:59.041 DEBUG:boto:Data: 2023-10-18T20:23:59.042 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.043 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.043 DEBUG:boto:Params: {} 2023-10-18T20:23:59.043 DEBUG:boto:Token: None 2023-10-18T20:23:59.044 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/d 2023-10-18T20:23:59.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RizmJPOGJYJl5V8rD1Gnc1ktREs= 2023-10-18T20:23:59.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RizmJPOGJYJl5V8rD1Gnc1ktREs=', 'Content-Length': '0'} 2023-10-18T20:23:59.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047820ab1b7113bc0-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.047 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:59.047 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:59.048 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.048 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:23:59.048 DEBUG:boto:Data: 2023-10-18T20:23:59.049 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.049 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.050 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.050 DEBUG:boto:Params: {} 2023-10-18T20:23:59.050 DEBUG:boto:Token: None 2023-10-18T20:23:59.051 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/d 2023-10-18T20:23:59.051 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RizmJPOGJYJl5V8rD1Gnc1ktREs= 2023-10-18T20:23:59.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RizmJPOGJYJl5V8rD1Gnc1ktREs=', 'Content-Length': '0'} 2023-10-18T20:23:59.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2194d6aadf14847-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:23:59.054 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:59.055 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:59.055 DEBUG:boto:Method: GET 2023-10-18T20:23:59.056 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:23:59.056 DEBUG:boto:Data: 2023-10-18T20:23:59.056 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.057 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.057 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.058 DEBUG:boto:Params: {} 2023-10-18T20:23:59.058 DEBUG:boto:Token: None 2023-10-18T20:23:59.058 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/d 2023-10-18T20:23:59.059 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4WdJ3/FR3ND+1/C4eXv3zmjlBLs= 2023-10-18T20:23:59.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4WdJ3/FR3ND+1/C4eXv3zmjlBLs=', 'Content-Length': '0'} 2023-10-18T20:23:59.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a23d9316681379f6-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.062 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:23:59.062 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:23:59.063 DEBUG:boto:Method: GET 2023-10-18T20:23:59.063 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:23:59.064 DEBUG:boto:Data: 2023-10-18T20:23:59.064 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.065 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.065 DEBUG:boto:Params: {} 2023-10-18T20:23:59.066 DEBUG:boto:Token: None 2023-10-18T20:23:59.066 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/d 2023-10-18T20:23:59.067 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4WdJ3/FR3ND+1/C4eXv3zmjlBLs= 2023-10-18T20:23:59.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4WdJ3/FR3ND+1/C4eXv3zmjlBLs=', 'Content-Length': '0'} 2023-10-18T20:23:59.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e6683433f074e1-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.070 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.070 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.070 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:23:59.071 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:23:59.071 DEBUG:boto:Method: GET 2023-10-18T20:23:59.072 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:23:59.072 DEBUG:boto:Data: 2023-10-18T20:23:59.073 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.073 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.074 DEBUG:boto:Params: {} 2023-10-18T20:23:59.074 DEBUG:boto:Token: None 2023-10-18T20:23:59.075 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.075 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b7f36de4bda7824-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.079 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.079 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.080 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.080 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:23:59.080 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:23:59.081 DEBUG:boto:Method: GET 2023-10-18T20:23:59.081 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:23:59.081 DEBUG:boto:Data: 2023-10-18T20:23:59.082 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.083 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.083 DEBUG:boto:Params: {} 2023-10-18T20:23:59.084 DEBUG:boto:Token: None 2023-10-18T20:23:59.084 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.084 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb8c907da4950bfd-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.087 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:23:59.088 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.089 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.089 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.090 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.090 DEBUG:boto:Method: GET 2023-10-18T20:23:59.090 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.091 DEBUG:boto:Data: 2023-10-18T20:23:59.091 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.092 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.092 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.092 DEBUG:boto:Params: {} 2023-10-18T20:23:59.093 DEBUG:boto:Token: None 2023-10-18T20:23:59.093 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.094 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM= 2023-10-18T20:23:59.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM=', 'Content-Length': '0'} 2023-10-18T20:23:59.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6270d5aa322b8a-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.138 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.139 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.139 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.139 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.140 DEBUG:boto:Method: GET 2023-10-18T20:23:59.140 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.141 DEBUG:boto:Data: 2023-10-18T20:23:59.141 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.142 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.142 DEBUG:boto:Params: {} 2023-10-18T20:23:59.143 DEBUG:boto:Token: None 2023-10-18T20:23:59.143 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM= 2023-10-18T20:23:59.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM=', 'Content-Length': '0'} 2023-10-18T20:23:59.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d3530cea960d1a-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.147 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.147 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.148 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.148 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.148 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.149 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.149 DEBUG:boto:Data: 2023-10-18T20:23:59.149 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.150 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.151 DEBUG:boto:Params: {} 2023-10-18T20:23:59.151 DEBUG:boto:Token: None 2023-10-18T20:23:59.151 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:26t2Ce9Uj8E3AgFALSm2UbjBwV8= 2023-10-18T20:23:59.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:26t2Ce9Uj8E3AgFALSm2UbjBwV8=', 'Content-Length': '0'} 2023-10-18T20:23:59.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4df156bb0a6994-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.155 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.155 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.155 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.156 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.156 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.157 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.157 DEBUG:boto:Data: 2023-10-18T20:23:59.157 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.158 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.158 DEBUG:boto:Params: {} 2023-10-18T20:23:59.159 DEBUG:boto:Token: None 2023-10-18T20:23:59.159 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.160 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:26t2Ce9Uj8E3AgFALSm2UbjBwV8= 2023-10-18T20:23:59.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:26t2Ce9Uj8E3AgFALSm2UbjBwV8=', 'Content-Length': '0'} 2023-10-18T20:23:59.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035746ce7076269e6-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:23:59.163 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.163 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.164 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.164 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.164 DEBUG:boto:Method: GET 2023-10-18T20:23:59.165 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.165 DEBUG:boto:Data: 2023-10-18T20:23:59.166 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.166 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.167 DEBUG:boto:Params: {} 2023-10-18T20:23:59.167 DEBUG:boto:Token: None 2023-10-18T20:23:59.168 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM= 2023-10-18T20:23:59.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM=', 'Content-Length': '0'} 2023-10-18T20:23:59.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001709b1f21edc9f3b-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.171 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.171 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.172 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.172 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:23:59.172 DEBUG:boto:Method: GET 2023-10-18T20:23:59.173 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.173 DEBUG:boto:Data: 2023-10-18T20:23:59.174 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.174 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.174 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.175 DEBUG:boto:Params: {} 2023-10-18T20:23:59.175 DEBUG:boto:Token: None 2023-10-18T20:23:59.176 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:23:59.176 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM= 2023-10-18T20:23:59.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM=', 'Content-Length': '0'} 2023-10-18T20:23:59.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd90dec48de190b-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.222 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.223 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.223 DEBUG:boto:path=/bofrnf-55/?prefix=e 2023-10-18T20:23:59.224 DEBUG:boto:auth_path=/bofrnf-55/?prefix=e 2023-10-18T20:23:59.224 DEBUG:boto:Method: GET 2023-10-18T20:23:59.224 DEBUG:boto:Path: /bofrnf-55/?prefix=e 2023-10-18T20:23:59.225 DEBUG:boto:Data: 2023-10-18T20:23:59.225 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.225 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.226 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.226 DEBUG:boto:Params: {} 2023-10-18T20:23:59.227 DEBUG:boto:Token: None 2023-10-18T20:23:59.227 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.228 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014899c4e5ac3b0e8-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.231 DEBUG:boto:b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.232 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.232 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.233 DEBUG:boto:path=/bofrnf-55/?prefix=e 2023-10-18T20:23:59.233 DEBUG:boto:auth_path=/bofrnf-55/?prefix=e 2023-10-18T20:23:59.233 DEBUG:boto:Method: GET 2023-10-18T20:23:59.234 DEBUG:boto:Path: /bofrnf-55/?prefix=e 2023-10-18T20:23:59.234 DEBUG:boto:Data: 2023-10-18T20:23:59.235 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.235 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.236 DEBUG:boto:Params: {} 2023-10-18T20:23:59.236 DEBUG:boto:Token: None 2023-10-18T20:23:59.237 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c4f197508e48918-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.240 DEBUG:boto:b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.240 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.241 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.241 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.242 DEBUG:boto:Path: /bofrnf-55/e 2023-10-18T20:23:59.242 DEBUG:boto:Data: 2023-10-18T20:23:59.242 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.243 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.243 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.243 DEBUG:boto:Params: {} 2023-10-18T20:23:59.244 DEBUG:boto:Token: None 2023-10-18T20:23:59.244 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e 2023-10-18T20:23:59.245 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:l2gOwWUenThOHdxRbD8UorjpVDY= 2023-10-18T20:23:59.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:l2gOwWUenThOHdxRbD8UorjpVDY=', 'Content-Length': '0'} 2023-10-18T20:23:59.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004140a2def0083957-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.247 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.248 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.248 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.248 DEBUG:boto:Path: /bofrnf-55/e 2023-10-18T20:23:59.249 DEBUG:boto:Data: 2023-10-18T20:23:59.249 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.249 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.250 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.250 DEBUG:boto:Params: {} 2023-10-18T20:23:59.251 DEBUG:boto:Token: None 2023-10-18T20:23:59.251 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e 2023-10-18T20:23:59.251 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:l2gOwWUenThOHdxRbD8UorjpVDY= 2023-10-18T20:23:59.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:l2gOwWUenThOHdxRbD8UorjpVDY=', 'Content-Length': '0'} 2023-10-18T20:23:59.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae54690d162ac4c3-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:23:59.255 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.255 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.255 DEBUG:boto:Method: GET 2023-10-18T20:23:59.256 DEBUG:boto:Path: /bofrnf-55/e 2023-10-18T20:23:59.256 DEBUG:boto:Data: 2023-10-18T20:23:59.257 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.257 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.257 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.258 DEBUG:boto:Params: {} 2023-10-18T20:23:59.258 DEBUG:boto:Token: None 2023-10-18T20:23:59.259 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e 2023-10-18T20:23:59.259 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:noTU/pTZhNnxJfHH3puJQYiir7o= 2023-10-18T20:23:59.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:noTU/pTZhNnxJfHH3puJQYiir7o=', 'Content-Length': '0'} 2023-10-18T20:23:59.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c0906ce1917af8a-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.302 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:23:59.303 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:23:59.303 DEBUG:boto:Method: GET 2023-10-18T20:23:59.303 DEBUG:boto:Path: /bofrnf-55/e 2023-10-18T20:23:59.304 DEBUG:boto:Data: 2023-10-18T20:23:59.304 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.305 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.305 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.305 DEBUG:boto:Params: {} 2023-10-18T20:23:59.306 DEBUG:boto:Token: None 2023-10-18T20:23:59.306 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/e 2023-10-18T20:23:59.307 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:noTU/pTZhNnxJfHH3puJQYiir7o= 2023-10-18T20:23:59.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:noTU/pTZhNnxJfHH3puJQYiir7o=', 'Content-Length': '0'} 2023-10-18T20:23:59.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c176e77bc4d67321-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.310 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.310 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.311 DEBUG:boto:path=/bofrnf-55/?prefix=e 2023-10-18T20:23:59.311 DEBUG:boto:auth_path=/bofrnf-55/?prefix=e 2023-10-18T20:23:59.311 DEBUG:boto:Method: GET 2023-10-18T20:23:59.312 DEBUG:boto:Path: /bofrnf-55/?prefix=e 2023-10-18T20:23:59.312 DEBUG:boto:Data: 2023-10-18T20:23:59.313 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.313 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.313 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.314 DEBUG:boto:Params: {} 2023-10-18T20:23:59.314 DEBUG:boto:Token: None 2023-10-18T20:23:59.315 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d66545d9cfe896-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.319 DEBUG:boto:b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.319 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.320 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.320 DEBUG:boto:path=/bofrnf-55/?prefix=e 2023-10-18T20:23:59.320 DEBUG:boto:auth_path=/bofrnf-55/?prefix=e 2023-10-18T20:23:59.321 DEBUG:boto:Method: GET 2023-10-18T20:23:59.321 DEBUG:boto:Path: /bofrnf-55/?prefix=e 2023-10-18T20:23:59.321 DEBUG:boto:Data: 2023-10-18T20:23:59.322 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.323 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.323 DEBUG:boto:Params: {} 2023-10-18T20:23:59.323 DEBUG:boto:Token: None 2023-10-18T20:23:59.324 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2b0450cfa4cf4f2-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.327 DEBUG:boto:b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:23:59.328 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.329 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.329 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.329 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.330 DEBUG:boto:Method: GET 2023-10-18T20:23:59.330 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.331 DEBUG:boto:Data: 2023-10-18T20:23:59.331 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.331 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.332 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.332 DEBUG:boto:Params: {} 2023-10-18T20:23:59.333 DEBUG:boto:Token: None 2023-10-18T20:23:59.333 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.334 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU= 2023-10-18T20:23:59.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU=', 'Content-Length': '0'} 2023-10-18T20:23:59.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9f400b1fce70b90-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.337 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.337 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.338 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.338 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.339 DEBUG:boto:Method: GET 2023-10-18T20:23:59.339 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.339 DEBUG:boto:Data: 2023-10-18T20:23:59.340 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.340 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.340 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.341 DEBUG:boto:Params: {} 2023-10-18T20:23:59.341 DEBUG:boto:Token: None 2023-10-18T20:23:59.342 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.342 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU= 2023-10-18T20:23:59.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU=', 'Content-Length': '0'} 2023-10-18T20:23:59.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b02d459c17b7b4fe-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.345 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.345 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.346 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.346 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.347 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.347 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.347 DEBUG:boto:Data: 2023-10-18T20:23:59.348 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.348 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.348 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.349 DEBUG:boto:Params: {} 2023-10-18T20:23:59.349 DEBUG:boto:Token: None 2023-10-18T20:23:59.350 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.350 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jGynnztRkhaxIuzinxsiIyVrE1Y= 2023-10-18T20:23:59.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jGynnztRkhaxIuzinxsiIyVrE1Y=', 'Content-Length': '0'} 2023-10-18T20:23:59.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5a7ee4d917fe622-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.353 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.353 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.354 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.354 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.354 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.355 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.355 DEBUG:boto:Data: 2023-10-18T20:23:59.355 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.356 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.356 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.356 DEBUG:boto:Params: {} 2023-10-18T20:23:59.357 DEBUG:boto:Token: None 2023-10-18T20:23:59.357 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jGynnztRkhaxIuzinxsiIyVrE1Y= 2023-10-18T20:23:59.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jGynnztRkhaxIuzinxsiIyVrE1Y=', 'Content-Length': '0'} 2023-10-18T20:23:59.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa1246dba38ee23d-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:23:59.361 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.362 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.362 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.362 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.363 DEBUG:boto:Method: GET 2023-10-18T20:23:59.363 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.364 DEBUG:boto:Data: 2023-10-18T20:23:59.364 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.364 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.365 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.365 DEBUG:boto:Params: {} 2023-10-18T20:23:59.365 DEBUG:boto:Token: None 2023-10-18T20:23:59.366 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.366 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU= 2023-10-18T20:23:59.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU=', 'Content-Length': '0'} 2023-10-18T20:23:59.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ec5345dcecea4fa-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.369 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.370 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.370 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.370 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:23:59.371 DEBUG:boto:Method: GET 2023-10-18T20:23:59.371 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.371 DEBUG:boto:Data: 2023-10-18T20:23:59.372 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.372 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.373 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.373 DEBUG:boto:Params: {} 2023-10-18T20:23:59.373 DEBUG:boto:Token: None 2023-10-18T20:23:59.374 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:23:59.374 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU= 2023-10-18T20:23:59.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU=', 'Content-Length': '0'} 2023-10-18T20:23:59.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b791356fde548bdb-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.378 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.378 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.378 DEBUG:boto:path=/bofrnf-55/?prefix=f 2023-10-18T20:23:59.379 DEBUG:boto:auth_path=/bofrnf-55/?prefix=f 2023-10-18T20:23:59.379 DEBUG:boto:Method: GET 2023-10-18T20:23:59.380 DEBUG:boto:Path: /bofrnf-55/?prefix=f 2023-10-18T20:23:59.380 DEBUG:boto:Data: 2023-10-18T20:23:59.380 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.381 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.381 DEBUG:boto:Params: {} 2023-10-18T20:23:59.382 DEBUG:boto:Token: None 2023-10-18T20:23:59.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e860933b9745c231-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.386 DEBUG:boto:b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.387 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.387 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.387 DEBUG:boto:path=/bofrnf-55/?prefix=f 2023-10-18T20:23:59.388 DEBUG:boto:auth_path=/bofrnf-55/?prefix=f 2023-10-18T20:23:59.388 DEBUG:boto:Method: GET 2023-10-18T20:23:59.389 DEBUG:boto:Path: /bofrnf-55/?prefix=f 2023-10-18T20:23:59.389 DEBUG:boto:Data: 2023-10-18T20:23:59.389 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.390 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.391 DEBUG:boto:Params: {} 2023-10-18T20:23:59.391 DEBUG:boto:Token: None 2023-10-18T20:23:59.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049b4af5f43f9a0a8-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.395 DEBUG:boto:b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.395 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.396 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.396 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.396 DEBUG:boto:Path: /bofrnf-55/f 2023-10-18T20:23:59.397 DEBUG:boto:Data: 2023-10-18T20:23:59.397 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.397 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.398 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.398 DEBUG:boto:Params: {} 2023-10-18T20:23:59.399 DEBUG:boto:Token: None 2023-10-18T20:23:59.399 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f 2023-10-18T20:23:59.399 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/dX0LLV4/ypNDP67ZcxOu9s6d04= 2023-10-18T20:23:59.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/dX0LLV4/ypNDP67ZcxOu9s6d04=', 'Content-Length': '0'} 2023-10-18T20:23:59.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d181d83001352d4-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.402 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.403 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.403 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.403 DEBUG:boto:Path: /bofrnf-55/f 2023-10-18T20:23:59.404 DEBUG:boto:Data: 2023-10-18T20:23:59.404 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.404 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.405 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.405 DEBUG:boto:Params: {} 2023-10-18T20:23:59.406 DEBUG:boto:Token: None 2023-10-18T20:23:59.406 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f 2023-10-18T20:23:59.406 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/dX0LLV4/ypNDP67ZcxOu9s6d04= 2023-10-18T20:23:59.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/dX0LLV4/ypNDP67ZcxOu9s6d04=', 'Content-Length': '0'} 2023-10-18T20:23:59.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752c0ea56a3cacc5-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:23:59.410 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.410 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.410 DEBUG:boto:Method: GET 2023-10-18T20:23:59.411 DEBUG:boto:Path: /bofrnf-55/f 2023-10-18T20:23:59.411 DEBUG:boto:Data: 2023-10-18T20:23:59.412 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.412 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.412 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.413 DEBUG:boto:Params: {} 2023-10-18T20:23:59.413 DEBUG:boto:Token: None 2023-10-18T20:23:59.414 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f 2023-10-18T20:23:59.414 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h7Z5rAbpNeiYAD1AM36dUtAm8XM= 2023-10-18T20:23:59.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h7Z5rAbpNeiYAD1AM36dUtAm8XM=', 'Content-Length': '0'} 2023-10-18T20:23:59.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e778487d794954-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.417 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:23:59.417 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:23:59.418 DEBUG:boto:Method: GET 2023-10-18T20:23:59.418 DEBUG:boto:Path: /bofrnf-55/f 2023-10-18T20:23:59.419 DEBUG:boto:Data: 2023-10-18T20:23:59.419 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.419 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.420 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.420 DEBUG:boto:Params: {} 2023-10-18T20:23:59.420 DEBUG:boto:Token: None 2023-10-18T20:23:59.421 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/f 2023-10-18T20:23:59.421 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h7Z5rAbpNeiYAD1AM36dUtAm8XM= 2023-10-18T20:23:59.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h7Z5rAbpNeiYAD1AM36dUtAm8XM=', 'Content-Length': '0'} 2023-10-18T20:23:59.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c179b96d3715b773-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.466 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.467 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.467 DEBUG:boto:path=/bofrnf-55/?prefix=f 2023-10-18T20:23:59.468 DEBUG:boto:auth_path=/bofrnf-55/?prefix=f 2023-10-18T20:23:59.468 DEBUG:boto:Method: GET 2023-10-18T20:23:59.468 DEBUG:boto:Path: /bofrnf-55/?prefix=f 2023-10-18T20:23:59.469 DEBUG:boto:Data: 2023-10-18T20:23:59.469 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.469 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.470 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.470 DEBUG:boto:Params: {} 2023-10-18T20:23:59.471 DEBUG:boto:Token: None 2023-10-18T20:23:59.471 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.472 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089bda83c5debefd1-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.475 DEBUG:boto:b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.475 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.476 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.476 DEBUG:boto:path=/bofrnf-55/?prefix=f 2023-10-18T20:23:59.477 DEBUG:boto:auth_path=/bofrnf-55/?prefix=f 2023-10-18T20:23:59.477 DEBUG:boto:Method: GET 2023-10-18T20:23:59.477 DEBUG:boto:Path: /bofrnf-55/?prefix=f 2023-10-18T20:23:59.478 DEBUG:boto:Data: 2023-10-18T20:23:59.478 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.479 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.479 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.479 DEBUG:boto:Params: {} 2023-10-18T20:23:59.480 DEBUG:boto:Token: None 2023-10-18T20:23:59.480 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.481 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000165021f4f48a885b-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.484 DEBUG:boto:b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:23:59.485 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.485 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.485 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.486 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.486 DEBUG:boto:Method: GET 2023-10-18T20:23:59.487 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.487 DEBUG:boto:Data: 2023-10-18T20:23:59.487 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.488 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.488 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.488 DEBUG:boto:Params: {} 2023-10-18T20:23:59.489 DEBUG:boto:Token: None 2023-10-18T20:23:59.489 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.490 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4= 2023-10-18T20:23:59.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4=', 'Content-Length': '0'} 2023-10-18T20:23:59.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053fd9b340dbbd5c7-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.534 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.535 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.535 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.535 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.536 DEBUG:boto:Method: GET 2023-10-18T20:23:59.536 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.537 DEBUG:boto:Data: 2023-10-18T20:23:59.537 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.538 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.538 DEBUG:boto:Params: {} 2023-10-18T20:23:59.539 DEBUG:boto:Token: None 2023-10-18T20:23:59.539 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.540 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4= 2023-10-18T20:23:59.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4=', 'Content-Length': '0'} 2023-10-18T20:23:59.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080df66b79225e746-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.543 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.543 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.543 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.544 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.544 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.545 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.545 DEBUG:boto:Data: 2023-10-18T20:23:59.545 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.546 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.546 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.546 DEBUG:boto:Params: {} 2023-10-18T20:23:59.547 DEBUG:boto:Token: None 2023-10-18T20:23:59.547 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.548 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SHxbxTKPLzw3HPv60rBex12FfvQ= 2023-10-18T20:23:59.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SHxbxTKPLzw3HPv60rBex12FfvQ=', 'Content-Length': '0'} 2023-10-18T20:23:59.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004112121f0e80634d-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.550 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.551 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.551 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.552 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.552 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.552 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.553 DEBUG:boto:Data: 2023-10-18T20:23:59.553 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.554 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.554 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.554 DEBUG:boto:Params: {} 2023-10-18T20:23:59.555 DEBUG:boto:Token: None 2023-10-18T20:23:59.555 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.556 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SHxbxTKPLzw3HPv60rBex12FfvQ= 2023-10-18T20:23:59.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SHxbxTKPLzw3HPv60rBex12FfvQ=', 'Content-Length': '0'} 2023-10-18T20:23:59.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dee9962d776226ca-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:23:59.559 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.559 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.560 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.560 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.560 DEBUG:boto:Method: GET 2023-10-18T20:23:59.561 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.561 DEBUG:boto:Data: 2023-10-18T20:23:59.562 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.562 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.563 DEBUG:boto:Params: {} 2023-10-18T20:23:59.563 DEBUG:boto:Token: None 2023-10-18T20:23:59.564 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4= 2023-10-18T20:23:59.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4=', 'Content-Length': '0'} 2023-10-18T20:23:59.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483ca418e9818aeb-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.567 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.568 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.568 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.568 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:23:59.569 DEBUG:boto:Method: GET 2023-10-18T20:23:59.569 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.570 DEBUG:boto:Data: 2023-10-18T20:23:59.570 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.570 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.571 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.571 DEBUG:boto:Params: {} 2023-10-18T20:23:59.572 DEBUG:boto:Token: None 2023-10-18T20:23:59.572 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:23:59.572 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4= 2023-10-18T20:23:59.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4=', 'Content-Length': '0'} 2023-10-18T20:23:59.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6632788d5e6fed0-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.576 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.576 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.576 DEBUG:boto:path=/bofrnf-55/?prefix=g 2023-10-18T20:23:59.577 DEBUG:boto:auth_path=/bofrnf-55/?prefix=g 2023-10-18T20:23:59.577 DEBUG:boto:Method: GET 2023-10-18T20:23:59.578 DEBUG:boto:Path: /bofrnf-55/?prefix=g 2023-10-18T20:23:59.578 DEBUG:boto:Data: 2023-10-18T20:23:59.578 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.579 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.579 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.580 DEBUG:boto:Params: {} 2023-10-18T20:23:59.580 DEBUG:boto:Token: None 2023-10-18T20:23:59.580 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044a3f570bdfed105-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.585 DEBUG:boto:b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.585 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.586 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.586 DEBUG:boto:path=/bofrnf-55/?prefix=g 2023-10-18T20:23:59.587 DEBUG:boto:auth_path=/bofrnf-55/?prefix=g 2023-10-18T20:23:59.587 DEBUG:boto:Method: GET 2023-10-18T20:23:59.587 DEBUG:boto:Path: /bofrnf-55/?prefix=g 2023-10-18T20:23:59.588 DEBUG:boto:Data: 2023-10-18T20:23:59.588 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.588 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.589 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.589 DEBUG:boto:Params: {} 2023-10-18T20:23:59.590 DEBUG:boto:Token: None 2023-10-18T20:23:59.590 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.590 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005049e7de0704f91f-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.634 DEBUG:boto:b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.635 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.635 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.636 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.636 DEBUG:boto:Path: /bofrnf-55/g 2023-10-18T20:23:59.636 DEBUG:boto:Data: 2023-10-18T20:23:59.637 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.637 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.638 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.638 DEBUG:boto:Params: {} 2023-10-18T20:23:59.638 DEBUG:boto:Token: None 2023-10-18T20:23:59.639 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g 2023-10-18T20:23:59.639 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GiMOUh0TGs9ftUDj57xa9Ln3gTE= 2023-10-18T20:23:59.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GiMOUh0TGs9ftUDj57xa9Ln3gTE=', 'Content-Length': '0'} 2023-10-18T20:23:59.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c412d7d45264e0-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.642 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.643 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.643 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.644 DEBUG:boto:Path: /bofrnf-55/g 2023-10-18T20:23:59.644 DEBUG:boto:Data: 2023-10-18T20:23:59.644 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.645 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.645 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.645 DEBUG:boto:Params: {} 2023-10-18T20:23:59.646 DEBUG:boto:Token: None 2023-10-18T20:23:59.646 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g 2023-10-18T20:23:59.647 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GiMOUh0TGs9ftUDj57xa9Ln3gTE= 2023-10-18T20:23:59.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GiMOUh0TGs9ftUDj57xa9Ln3gTE=', 'Content-Length': '0'} 2023-10-18T20:23:59.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b82e4090e883c1f-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:23:59.650 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.651 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.651 DEBUG:boto:Method: GET 2023-10-18T20:23:59.652 DEBUG:boto:Path: /bofrnf-55/g 2023-10-18T20:23:59.652 DEBUG:boto:Data: 2023-10-18T20:23:59.652 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.653 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.653 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.654 DEBUG:boto:Params: {} 2023-10-18T20:23:59.654 DEBUG:boto:Token: None 2023-10-18T20:23:59.654 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g 2023-10-18T20:23:59.655 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GLPR4Av3J//GKbAnIE3HPsyHiGE= 2023-10-18T20:23:59.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GLPR4Av3J//GKbAnIE3HPsyHiGE=', 'Content-Length': '0'} 2023-10-18T20:23:59.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006696f78f6853af74-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.658 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:23:59.658 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:23:59.659 DEBUG:boto:Method: GET 2023-10-18T20:23:59.659 DEBUG:boto:Path: /bofrnf-55/g 2023-10-18T20:23:59.659 DEBUG:boto:Data: 2023-10-18T20:23:59.660 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.660 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.661 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.661 DEBUG:boto:Params: {} 2023-10-18T20:23:59.661 DEBUG:boto:Token: None 2023-10-18T20:23:59.662 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/g 2023-10-18T20:23:59.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GLPR4Av3J//GKbAnIE3HPsyHiGE= 2023-10-18T20:23:59.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GLPR4Av3J//GKbAnIE3HPsyHiGE=', 'Content-Length': '0'} 2023-10-18T20:23:59.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008979b3f284457427-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.666 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.666 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.666 DEBUG:boto:path=/bofrnf-55/?prefix=g 2023-10-18T20:23:59.667 DEBUG:boto:auth_path=/bofrnf-55/?prefix=g 2023-10-18T20:23:59.667 DEBUG:boto:Method: GET 2023-10-18T20:23:59.668 DEBUG:boto:Path: /bofrnf-55/?prefix=g 2023-10-18T20:23:59.668 DEBUG:boto:Data: 2023-10-18T20:23:59.668 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.669 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.669 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.669 DEBUG:boto:Params: {} 2023-10-18T20:23:59.670 DEBUG:boto:Token: None 2023-10-18T20:23:59.670 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.671 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a173eb2c7cdc7329-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.674 DEBUG:boto:b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.675 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.675 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.675 DEBUG:boto:path=/bofrnf-55/?prefix=g 2023-10-18T20:23:59.676 DEBUG:boto:auth_path=/bofrnf-55/?prefix=g 2023-10-18T20:23:59.676 DEBUG:boto:Method: GET 2023-10-18T20:23:59.677 DEBUG:boto:Path: /bofrnf-55/?prefix=g 2023-10-18T20:23:59.677 DEBUG:boto:Data: 2023-10-18T20:23:59.677 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.678 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.678 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.679 DEBUG:boto:Params: {} 2023-10-18T20:23:59.679 DEBUG:boto:Token: None 2023-10-18T20:23:59.679 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.680 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d741b8fe5aad225a-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.683 DEBUG:boto:b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:23:59.683 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.684 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.684 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.685 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.685 DEBUG:boto:Method: GET 2023-10-18T20:23:59.685 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.686 DEBUG:boto:Data: 2023-10-18T20:23:59.686 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.687 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.687 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.687 DEBUG:boto:Params: {} 2023-10-18T20:23:59.688 DEBUG:boto:Token: None 2023-10-18T20:23:59.688 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.689 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss= 2023-10-18T20:23:59.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss=', 'Content-Length': '0'} 2023-10-18T20:23:59.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041848d2cc48fde29-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.692 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.692 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.693 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.693 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.694 DEBUG:boto:Method: GET 2023-10-18T20:23:59.694 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.694 DEBUG:boto:Data: 2023-10-18T20:23:59.695 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.695 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.696 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.696 DEBUG:boto:Params: {} 2023-10-18T20:23:59.696 DEBUG:boto:Token: None 2023-10-18T20:23:59.697 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.697 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss= 2023-10-18T20:23:59.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss=', 'Content-Length': '0'} 2023-10-18T20:23:59.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010157719a226cc04-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.700 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.701 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.701 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.701 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.702 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.702 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.702 DEBUG:boto:Data: 2023-10-18T20:23:59.703 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.703 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.704 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.704 DEBUG:boto:Params: {} 2023-10-18T20:23:59.704 DEBUG:boto:Token: None 2023-10-18T20:23:59.705 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.705 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cDoJ+8PRRwIER7ntyIdd4PXKU/M= 2023-10-18T20:23:59.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cDoJ+8PRRwIER7ntyIdd4PXKU/M=', 'Content-Length': '0'} 2023-10-18T20:23:59.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002fe041bd987872a-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.708 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.708 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.708 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.709 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.709 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.710 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.710 DEBUG:boto:Data: 2023-10-18T20:23:59.710 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.711 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.711 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.711 DEBUG:boto:Params: {} 2023-10-18T20:23:59.712 DEBUG:boto:Token: None 2023-10-18T20:23:59.712 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.713 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cDoJ+8PRRwIER7ntyIdd4PXKU/M= 2023-10-18T20:23:59.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cDoJ+8PRRwIER7ntyIdd4PXKU/M=', 'Content-Length': '0'} 2023-10-18T20:23:59.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff85ebd16472031-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:23:59.716 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.716 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.716 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.717 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.717 DEBUG:boto:Method: GET 2023-10-18T20:23:59.717 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.718 DEBUG:boto:Data: 2023-10-18T20:23:59.718 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.719 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.719 DEBUG:boto:Params: {} 2023-10-18T20:23:59.720 DEBUG:boto:Token: None 2023-10-18T20:23:59.720 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss= 2023-10-18T20:23:59.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss=', 'Content-Length': '0'} 2023-10-18T20:23:59.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001702364cc86ebac-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.766 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.767 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.767 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.767 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:23:59.768 DEBUG:boto:Method: GET 2023-10-18T20:23:59.768 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.769 DEBUG:boto:Data: 2023-10-18T20:23:59.769 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.769 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.770 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.770 DEBUG:boto:Params: {} 2023-10-18T20:23:59.771 DEBUG:boto:Token: None 2023-10-18T20:23:59.771 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:23:59.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss= 2023-10-18T20:23:59.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss=', 'Content-Length': '0'} 2023-10-18T20:23:59.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007261dd9e24b3a73d-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.775 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.775 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.776 DEBUG:boto:path=/bofrnf-55/?prefix=h 2023-10-18T20:23:59.776 DEBUG:boto:auth_path=/bofrnf-55/?prefix=h 2023-10-18T20:23:59.776 DEBUG:boto:Method: GET 2023-10-18T20:23:59.777 DEBUG:boto:Path: /bofrnf-55/?prefix=h 2023-10-18T20:23:59.777 DEBUG:boto:Data: 2023-10-18T20:23:59.777 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.778 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.778 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.779 DEBUG:boto:Params: {} 2023-10-18T20:23:59.779 DEBUG:boto:Token: None 2023-10-18T20:23:59.780 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5019b80a50c7b7f-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.784 DEBUG:boto:b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.784 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.785 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.785 DEBUG:boto:path=/bofrnf-55/?prefix=h 2023-10-18T20:23:59.785 DEBUG:boto:auth_path=/bofrnf-55/?prefix=h 2023-10-18T20:23:59.786 DEBUG:boto:Method: GET 2023-10-18T20:23:59.786 DEBUG:boto:Path: /bofrnf-55/?prefix=h 2023-10-18T20:23:59.787 DEBUG:boto:Data: 2023-10-18T20:23:59.787 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.788 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.788 DEBUG:boto:Params: {} 2023-10-18T20:23:59.788 DEBUG:boto:Token: None 2023-10-18T20:23:59.789 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cecbb84c8635d038-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.834 DEBUG:boto:b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.835 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.835 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.836 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.836 DEBUG:boto:Path: /bofrnf-55/h 2023-10-18T20:23:59.837 DEBUG:boto:Data: 2023-10-18T20:23:59.837 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.838 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.838 DEBUG:boto:Params: {} 2023-10-18T20:23:59.838 DEBUG:boto:Token: None 2023-10-18T20:23:59.839 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h 2023-10-18T20:23:59.839 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:an234AdKI2DjD1h9YWgCQsNjsa4= 2023-10-18T20:23:59.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:an234AdKI2DjD1h9YWgCQsNjsa4=', 'Content-Length': '0'} 2023-10-18T20:23:59.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001019913fbcb7cd04-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.842 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.843 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.843 DEBUG:boto:Method: HEAD 2023-10-18T20:23:59.843 DEBUG:boto:Path: /bofrnf-55/h 2023-10-18T20:23:59.844 DEBUG:boto:Data: 2023-10-18T20:23:59.844 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.844 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.845 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.845 DEBUG:boto:Params: {} 2023-10-18T20:23:59.846 DEBUG:boto:Token: None 2023-10-18T20:23:59.846 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h 2023-10-18T20:23:59.846 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:an234AdKI2DjD1h9YWgCQsNjsa4= 2023-10-18T20:23:59.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:an234AdKI2DjD1h9YWgCQsNjsa4=', 'Content-Length': '0'} 2023-10-18T20:23:59.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e7652665a85a09-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:23:59.850 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.850 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.851 DEBUG:boto:Method: GET 2023-10-18T20:23:59.851 DEBUG:boto:Path: /bofrnf-55/h 2023-10-18T20:23:59.852 DEBUG:boto:Data: 2023-10-18T20:23:59.852 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.852 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.853 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.853 DEBUG:boto:Params: {} 2023-10-18T20:23:59.854 DEBUG:boto:Token: None 2023-10-18T20:23:59.854 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h 2023-10-18T20:23:59.855 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:51DIphLkMKb+nzztXQdLudW57Rw= 2023-10-18T20:23:59.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:51DIphLkMKb+nzztXQdLudW57Rw=', 'Content-Length': '0'} 2023-10-18T20:23:59.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000684cb9c6693ffa1b-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.858 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:23:59.858 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:23:59.859 DEBUG:boto:Method: GET 2023-10-18T20:23:59.859 DEBUG:boto:Path: /bofrnf-55/h 2023-10-18T20:23:59.860 DEBUG:boto:Data: 2023-10-18T20:23:59.860 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.860 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.861 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.861 DEBUG:boto:Params: {} 2023-10-18T20:23:59.862 DEBUG:boto:Token: None 2023-10-18T20:23:59.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/h 2023-10-18T20:23:59.863 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:51DIphLkMKb+nzztXQdLudW57Rw= 2023-10-18T20:23:59.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:51DIphLkMKb+nzztXQdLudW57Rw=', 'Content-Length': '0'} 2023-10-18T20:23:59.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde129f11b8f8444-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.866 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.866 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.866 DEBUG:boto:path=/bofrnf-55/?prefix=h 2023-10-18T20:23:59.867 DEBUG:boto:auth_path=/bofrnf-55/?prefix=h 2023-10-18T20:23:59.867 DEBUG:boto:Method: GET 2023-10-18T20:23:59.868 DEBUG:boto:Path: /bofrnf-55/?prefix=h 2023-10-18T20:23:59.868 DEBUG:boto:Data: 2023-10-18T20:23:59.868 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:23:59.869 DEBUG:boto:Port: 8001 2023-10-18T20:23:59.870 DEBUG:boto:Params: {} 2023-10-18T20:23:59.870 DEBUG:boto:Token: None 2023-10-18T20:23:59.870 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.871 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2c24862b736ac3b-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.874 DEBUG:boto:b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.875 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:23:59.875 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:23:59.876 DEBUG:boto:path=/bofrnf-55/?prefix=h 2023-10-18T20:23:59.876 DEBUG:boto:auth_path=/bofrnf-55/?prefix=h 2023-10-18T20:23:59.876 DEBUG:boto:Method: GET 2023-10-18T20:23:59.877 DEBUG:boto:Path: /bofrnf-55/?prefix=h 2023-10-18T20:23:59.877 DEBUG:boto:Data: 2023-10-18T20:23:59.878 DEBUG:boto:Headers: {} 2023-10-18T20:23:59.878 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:23:59.878 DEBUG:boto:Port: 8000 2023-10-18T20:23:59.879 DEBUG:boto:Params: {} 2023-10-18T20:23:59.879 DEBUG:boto:Token: None 2023-10-18T20:23:59.880 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:23:59 GMT /bofrnf-55/ 2023-10-18T20:23:59.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T20:23:59.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T20:23:59.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000255163e66fa67db4-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:23:59.883 DEBUG:boto:b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:23:59.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-55 zones={a2, a1} 2023-10-18T20:23:59.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'bofrnf-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:23:59.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'bofrnf-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:23:59.885 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket bofrnf-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:23:59.941 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:23:59.941 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:23:59.984 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:23:59.984 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-55 2023-10-18T20:23:59.985 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 8 2023-10-18T20:24:00.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:00.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:00.013 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:24:00.059 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:00.060 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:00.099 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:24:00.100 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:00.100 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding": "None", 2023-10-18T20:24:00.100 INFO:teuthology.orchestra.run.smithi082.stdout: "current_index": { 2023-10-18T20:24:00.100 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 2, 2023-10-18T20:24:00.100 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:00.101 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "Normal", 2023-10-18T20:24:00.101 INFO:teuthology.orchestra.run.smithi082.stdout: "normal": { 2023-10-18T20:24:00.101 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:24:00.101 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:00.101 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.102 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:00.102 INFO:teuthology.orchestra.run.smithi082.stdout: "logs": [ 2023-10-18T20:24:00.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:00.102 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 0, 2023-10-18T20:24:00.103 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:00.103 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:24:00.103 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:24:00.103 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 0, 2023-10-18T20:24:00.103 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:00.103 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 11, 2023-10-18T20:24:00.104 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:00.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:00.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:00.105 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 1, 2023-10-18T20:24:00.105 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:00.105 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:24:00.105 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:24:00.106 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 1, 2023-10-18T20:24:00.106 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:00.106 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:24:00.106 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:00.106 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.106 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:00.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:00.107 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 2, 2023-10-18T20:24:00.107 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:00.108 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:24:00.108 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:24:00.108 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 2, 2023-10-18T20:24:00.108 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:00.108 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:24:00.108 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:00.109 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.109 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.109 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.109 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.109 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:24:00.110 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:00.110 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:24:00.111 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-55 2023-10-18T20:24:00.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:00.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:00.112 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:24:00.168 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:00.168 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:00.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:00.199+0000 7f15ebab8780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:24:05.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:05.200+0000 7f15ebab8780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:24:10.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:10.201+0000 7f15ebab8780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:24:15.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:15.202+0000 7f15ebab8780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:24:20.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:20.202+0000 7f15ebab8780 1 bucket sync caught up with empty source 2023-10-18T20:24:20.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:20.202+0000 7f15ebab8780 0 bucket checkpoint complete 2023-10-18T20:24:20.220 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-55 2023-10-18T20:24:20.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:20.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:20.222 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:24:20.285 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:20.285 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:20.322 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:20.321+0000 7f549f84a780 1 bucket sync caught up with source: 2023-10-18T20:24:20.322 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T20:24:20.323 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T20:24:20.323 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:20.321+0000 7f549f84a780 0 bucket checkpoint complete 2023-10-18T20:24:20.336 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-55 zones={a2, a1} 2023-10-18T20:24:20.336 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.336 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.337 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.337 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:24:20.338 DEBUG:boto:Data: 2023-10-18T20:24:20.338 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.339 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.339 DEBUG:boto:Params: {} 2023-10-18T20:24:20.341 DEBUG:boto:Token: None 2023-10-18T20:24:20.341 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.342 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mhadj15V93uMESLg/uS93VOb3kE= 2023-10-18T20:24:20.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mhadj15V93uMESLg/uS93VOb3kE=', 'Content-Length': '0'} 2023-10-18T20:24:20.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ae5a31a48da1f96-0065303ef4-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.345 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.345 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.346 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.346 DEBUG:boto:Path: /bofrnf-55/ 2023-10-18T20:24:20.347 DEBUG:boto:Data: 2023-10-18T20:24:20.347 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.347 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.348 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.348 DEBUG:boto:Params: {} 2023-10-18T20:24:20.349 DEBUG:boto:Token: None 2023-10-18T20:24:20.349 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.349 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mhadj15V93uMESLg/uS93VOb3kE= 2023-10-18T20:24:20.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mhadj15V93uMESLg/uS93VOb3kE=', 'Content-Length': '0'} 2023-10-18T20:24:20.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006925ecdf4d2e498e-0065303ef4-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.352 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:24:20.352 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.353 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.353 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.354 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.354 DEBUG:boto:Method: GET 2023-10-18T20:24:20.354 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.355 DEBUG:boto:Data: 2023-10-18T20:24:20.355 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.356 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.356 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.356 DEBUG:boto:Params: {} 2023-10-18T20:24:20.357 DEBUG:boto:Token: None 2023-10-18T20:24:20.357 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/?versions 2023-10-18T20:24:20.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs= 2023-10-18T20:24:20.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs=', 'Content-Length': '0'} 2023-10-18T20:24:20.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa0e1ab17fe7dc4-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.361 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.362 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:24:20.362 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:24:20.363 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:24:20.363 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:24:20.363 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-18T20:24:20.364 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-18T20:24:20.364 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-18T20:24:20.364 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-18T20:24:20.365 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:24:20.365 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.365 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.366 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.366 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.367 DEBUG:boto:Method: GET 2023-10-18T20:24:20.367 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.367 DEBUG:boto:Data: 2023-10-18T20:24:20.368 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.368 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.368 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.369 DEBUG:boto:Params: {} 2023-10-18T20:24:20.369 DEBUG:boto:Token: None 2023-10-18T20:24:20.369 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/?versions 2023-10-18T20:24:20.370 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs= 2023-10-18T20:24:20.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs=', 'Content-Length': '0'} 2023-10-18T20:24:20.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf8344ec2839924-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.373 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.374 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:24:20.374 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:24:20.375 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:24:20.375 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:24:20.375 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-18T20:24:20.376 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-18T20:24:20.376 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-18T20:24:20.377 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-18T20:24:20.377 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.377 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.378 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.378 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.379 DEBUG:boto:Method: GET 2023-10-18T20:24:20.379 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.379 DEBUG:boto:Data: 2023-10-18T20:24:20.380 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.380 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.381 DEBUG:boto:Params: {} 2023-10-18T20:24:20.381 DEBUG:boto:Token: None 2023-10-18T20:24:20.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/?versions 2023-10-18T20:24:20.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs= 2023-10-18T20:24:20.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs=', 'Content-Length': '0'} 2023-10-18T20:24:20.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc5ce2d6faf16c24-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.385 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.386 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.387 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.387 DEBUG:boto:path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.387 DEBUG:boto:auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.388 DEBUG:boto:Method: GET 2023-10-18T20:24:20.388 DEBUG:boto:Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T20:24:20.389 DEBUG:boto:Data: 2023-10-18T20:24:20.389 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.390 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.390 DEBUG:boto:Params: {} 2023-10-18T20:24:20.390 DEBUG:boto:Token: None 2023-10-18T20:24:20.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/?versions 2023-10-18T20:24:20.391 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs= 2023-10-18T20:24:20.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs=', 'Content-Length': '0'} 2023-10-18T20:24:20.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c3b9687c74290ed-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.394 DEBUG:boto:b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:24:20.395 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.396 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.397 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.397 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.397 DEBUG:boto:Method: GET 2023-10-18T20:24:20.398 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.398 DEBUG:boto:Data: 2023-10-18T20:24:20.399 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.399 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.400 DEBUG:boto:Params: {} 2023-10-18T20:24:20.400 DEBUG:boto:Token: None 2023-10-18T20:24:20.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU= 2023-10-18T20:24:20.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU=', 'Content-Length': '0'} 2023-10-18T20:24:20.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f17c7fb2837efee-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.404 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.404 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.405 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.405 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.405 DEBUG:boto:Method: GET 2023-10-18T20:24:20.406 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.406 DEBUG:boto:Data: 2023-10-18T20:24:20.406 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.407 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.407 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.407 DEBUG:boto:Params: {} 2023-10-18T20:24:20.408 DEBUG:boto:Token: None 2023-10-18T20:24:20.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.409 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU= 2023-10-18T20:24:20.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU=', 'Content-Length': '0'} 2023-10-18T20:24:20.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa8a964836f904ee-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.411 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.412 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.412 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.413 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.413 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.413 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.414 DEBUG:boto:Data: 2023-10-18T20:24:20.414 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.414 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.415 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.415 DEBUG:boto:Params: {} 2023-10-18T20:24:20.416 DEBUG:boto:Token: None 2023-10-18T20:24:20.416 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.417 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y84ufVmK3/zQ8W2VjBHAvQ4W4LE= 2023-10-18T20:24:20.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y84ufVmK3/zQ8W2VjBHAvQ4W4LE=', 'Content-Length': '0'} 2023-10-18T20:24:20.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f9131a0cc61e5e-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.419 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.420 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.420 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.420 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.421 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.421 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.421 DEBUG:boto:Data: 2023-10-18T20:24:20.422 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.422 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.423 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.423 DEBUG:boto:Params: {} 2023-10-18T20:24:20.423 DEBUG:boto:Token: None 2023-10-18T20:24:20.424 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.424 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y84ufVmK3/zQ8W2VjBHAvQ4W4LE= 2023-10-18T20:24:20.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y84ufVmK3/zQ8W2VjBHAvQ4W4LE=', 'Content-Length': '0'} 2023-10-18T20:24:20.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b39ae72631aed3-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:24:20.427 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.428 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.428 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.428 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.429 DEBUG:boto:Method: GET 2023-10-18T20:24:20.429 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.430 DEBUG:boto:Data: 2023-10-18T20:24:20.430 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.430 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.431 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.431 DEBUG:boto:Params: {} 2023-10-18T20:24:20.432 DEBUG:boto:Token: None 2023-10-18T20:24:20.432 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.432 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU= 2023-10-18T20:24:20.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU=', 'Content-Length': '0'} 2023-10-18T20:24:20.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45030f7e84079f4-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.435 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.436 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.436 DEBUG:boto:path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.436 DEBUG:boto:auth_path=/bofrnf-55/a?versionId=null 2023-10-18T20:24:20.437 DEBUG:boto:Method: GET 2023-10-18T20:24:20.437 DEBUG:boto:Path: /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.437 DEBUG:boto:Data: 2023-10-18T20:24:20.438 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.438 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.439 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.439 DEBUG:boto:Params: {} 2023-10-18T20:24:20.439 DEBUG:boto:Token: None 2023-10-18T20:24:20.440 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a?versionId=null 2023-10-18T20:24:20.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU= 2023-10-18T20:24:20.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU=', 'Content-Length': '0'} 2023-10-18T20:24:20.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c7ac62a4ee5c91e-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.486 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.487 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.487 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:24:20.487 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:24:20.488 DEBUG:boto:Method: GET 2023-10-18T20:24:20.488 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:24:20.489 DEBUG:boto:Data: 2023-10-18T20:24:20.489 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.489 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.490 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.490 DEBUG:boto:Params: {} 2023-10-18T20:24:20.490 DEBUG:boto:Token: None 2023-10-18T20:24:20.491 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.491 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000593cb046d9c0aba7-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.495 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.495 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.496 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.496 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:24:20.496 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:24:20.497 DEBUG:boto:Method: GET 2023-10-18T20:24:20.497 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:24:20.498 DEBUG:boto:Data: 2023-10-18T20:24:20.498 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.499 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.499 DEBUG:boto:Params: {} 2023-10-18T20:24:20.500 DEBUG:boto:Token: None 2023-10-18T20:24:20.500 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.500 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee355deaaa7febc2-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.503 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.504 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.504 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.505 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.505 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:24:20.505 DEBUG:boto:Data: 2023-10-18T20:24:20.506 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.506 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.507 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.507 DEBUG:boto:Params: {} 2023-10-18T20:24:20.507 DEBUG:boto:Token: None 2023-10-18T20:24:20.508 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a 2023-10-18T20:24:20.508 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zjXwn04OLDZRQkDHIXzbHIOb6Rk= 2023-10-18T20:24:20.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zjXwn04OLDZRQkDHIXzbHIOb6Rk=', 'Content-Length': '0'} 2023-10-18T20:24:20.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000269a8c653e84ef16-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.511 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.512 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.512 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.512 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:24:20.513 DEBUG:boto:Data: 2023-10-18T20:24:20.513 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.513 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.514 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.514 DEBUG:boto:Params: {} 2023-10-18T20:24:20.515 DEBUG:boto:Token: None 2023-10-18T20:24:20.515 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a 2023-10-18T20:24:20.515 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zjXwn04OLDZRQkDHIXzbHIOb6Rk= 2023-10-18T20:24:20.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zjXwn04OLDZRQkDHIXzbHIOb6Rk=', 'Content-Length': '0'} 2023-10-18T20:24:20.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e4bce094c592e71-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:24:20.518 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.519 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.519 DEBUG:boto:Method: GET 2023-10-18T20:24:20.520 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:24:20.520 DEBUG:boto:Data: 2023-10-18T20:24:20.520 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.521 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.521 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.521 DEBUG:boto:Params: {} 2023-10-18T20:24:20.522 DEBUG:boto:Token: None 2023-10-18T20:24:20.522 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a 2023-10-18T20:24:20.523 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r4I+izeyPq9/cUpeLUXvAbmyVGM= 2023-10-18T20:24:20.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r4I+izeyPq9/cUpeLUXvAbmyVGM=', 'Content-Length': '0'} 2023-10-18T20:24:20.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6bd36e225cb943-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.566 DEBUG:boto:path=/bofrnf-55/a 2023-10-18T20:24:20.567 DEBUG:boto:auth_path=/bofrnf-55/a 2023-10-18T20:24:20.567 DEBUG:boto:Method: GET 2023-10-18T20:24:20.568 DEBUG:boto:Path: /bofrnf-55/a 2023-10-18T20:24:20.568 DEBUG:boto:Data: 2023-10-18T20:24:20.568 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.569 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.569 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.570 DEBUG:boto:Params: {} 2023-10-18T20:24:20.570 DEBUG:boto:Token: None 2023-10-18T20:24:20.570 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/a 2023-10-18T20:24:20.571 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r4I+izeyPq9/cUpeLUXvAbmyVGM= 2023-10-18T20:24:20.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r4I+izeyPq9/cUpeLUXvAbmyVGM=', 'Content-Length': '0'} 2023-10-18T20:24:20.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b17c6f03fd281f-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.574 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.574 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.575 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:24:20.575 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:24:20.576 DEBUG:boto:Method: GET 2023-10-18T20:24:20.576 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:24:20.576 DEBUG:boto:Data: 2023-10-18T20:24:20.577 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.577 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.577 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.578 DEBUG:boto:Params: {} 2023-10-18T20:24:20.578 DEBUG:boto:Token: None 2023-10-18T20:24:20.579 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.579 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4745b9f548832f5-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.582 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.583 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.583 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.584 DEBUG:boto:path=/bofrnf-55/?prefix=a 2023-10-18T20:24:20.584 DEBUG:boto:auth_path=/bofrnf-55/?prefix=a 2023-10-18T20:24:20.585 DEBUG:boto:Method: GET 2023-10-18T20:24:20.585 DEBUG:boto:Path: /bofrnf-55/?prefix=a 2023-10-18T20:24:20.585 DEBUG:boto:Data: 2023-10-18T20:24:20.586 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.586 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.586 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.587 DEBUG:boto:Params: {} 2023-10-18T20:24:20.587 DEBUG:boto:Token: None 2023-10-18T20:24:20.588 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.588 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008371df0ec411380f-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.591 DEBUG:boto:b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:24:20.592 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.592 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.592 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.593 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.593 DEBUG:boto:Method: GET 2023-10-18T20:24:20.594 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.594 DEBUG:boto:Data: 2023-10-18T20:24:20.594 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.595 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.595 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.595 DEBUG:boto:Params: {} 2023-10-18T20:24:20.596 DEBUG:boto:Token: None 2023-10-18T20:24:20.596 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.597 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8= 2023-10-18T20:24:20.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8=', 'Content-Length': '0'} 2023-10-18T20:24:20.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059fedec66fad5bc9-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.600 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.600 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.600 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.601 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.601 DEBUG:boto:Method: GET 2023-10-18T20:24:20.602 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.602 DEBUG:boto:Data: 2023-10-18T20:24:20.602 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.603 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.603 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.604 DEBUG:boto:Params: {} 2023-10-18T20:24:20.604 DEBUG:boto:Token: None 2023-10-18T20:24:20.604 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8= 2023-10-18T20:24:20.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8=', 'Content-Length': '0'} 2023-10-18T20:24:20.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02fd0f2a5c1e3a1-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.650 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.651 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.651 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.651 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.652 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.652 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.653 DEBUG:boto:Data: 2023-10-18T20:24:20.653 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.653 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.654 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.654 DEBUG:boto:Params: {} 2023-10-18T20:24:20.654 DEBUG:boto:Token: None 2023-10-18T20:24:20.655 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.655 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nisZSm52Fu1gW5LCoYV6MlbeqCk= 2023-10-18T20:24:20.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nisZSm52Fu1gW5LCoYV6MlbeqCk=', 'Content-Length': '0'} 2023-10-18T20:24:20.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33402a8ea690b73-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.658 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.658 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.659 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.659 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.659 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.660 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.660 DEBUG:boto:Data: 2023-10-18T20:24:20.661 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.661 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.662 DEBUG:boto:Params: {} 2023-10-18T20:24:20.662 DEBUG:boto:Token: None 2023-10-18T20:24:20.663 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.663 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nisZSm52Fu1gW5LCoYV6MlbeqCk= 2023-10-18T20:24:20.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nisZSm52Fu1gW5LCoYV6MlbeqCk=', 'Content-Length': '0'} 2023-10-18T20:24:20.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d00d7d670b73b9-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:24:20.666 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.667 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.667 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.667 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.668 DEBUG:boto:Method: GET 2023-10-18T20:24:20.668 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.669 DEBUG:boto:Data: 2023-10-18T20:24:20.669 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.669 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.670 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.670 DEBUG:boto:Params: {} 2023-10-18T20:24:20.671 DEBUG:boto:Token: None 2023-10-18T20:24:20.671 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.671 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8= 2023-10-18T20:24:20.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8=', 'Content-Length': '0'} 2023-10-18T20:24:20.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b991eb31cf8b5dfd-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.674 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.675 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.675 DEBUG:boto:path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.675 DEBUG:boto:auth_path=/bofrnf-55/b?versionId=null 2023-10-18T20:24:20.676 DEBUG:boto:Method: GET 2023-10-18T20:24:20.676 DEBUG:boto:Path: /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.677 DEBUG:boto:Data: 2023-10-18T20:24:20.677 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.677 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.678 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.678 DEBUG:boto:Params: {} 2023-10-18T20:24:20.679 DEBUG:boto:Token: None 2023-10-18T20:24:20.679 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b?versionId=null 2023-10-18T20:24:20.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8= 2023-10-18T20:24:20.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8=', 'Content-Length': '0'} 2023-10-18T20:24:20.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389ff9173ab5c4ac-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.682 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.683 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.683 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:24:20.684 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:24:20.684 DEBUG:boto:Method: GET 2023-10-18T20:24:20.684 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:24:20.685 DEBUG:boto:Data: 2023-10-18T20:24:20.685 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.686 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.686 DEBUG:boto:Params: {} 2023-10-18T20:24:20.687 DEBUG:boto:Token: None 2023-10-18T20:24:20.687 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d7a5f6e04352f45-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.691 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.691 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.692 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.692 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:24:20.692 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:24:20.693 DEBUG:boto:Method: GET 2023-10-18T20:24:20.693 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:24:20.694 DEBUG:boto:Data: 2023-10-18T20:24:20.694 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.694 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.695 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.695 DEBUG:boto:Params: {} 2023-10-18T20:24:20.696 DEBUG:boto:Token: None 2023-10-18T20:24:20.696 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.696 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000139a641fabdd655c-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.699 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.700 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.700 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.700 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.701 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:24:20.701 DEBUG:boto:Data: 2023-10-18T20:24:20.701 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.702 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.702 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.702 DEBUG:boto:Params: {} 2023-10-18T20:24:20.703 DEBUG:boto:Token: None 2023-10-18T20:24:20.703 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b 2023-10-18T20:24:20.704 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hE0PBU4SG9FRrvD2QO7JDVdINCg= 2023-10-18T20:24:20.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hE0PBU4SG9FRrvD2QO7JDVdINCg=', 'Content-Length': '0'} 2023-10-18T20:24:20.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005daa74ff4d25f707-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.707 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.707 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.707 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.708 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:24:20.708 DEBUG:boto:Data: 2023-10-18T20:24:20.708 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.709 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.709 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.709 DEBUG:boto:Params: {} 2023-10-18T20:24:20.710 DEBUG:boto:Token: None 2023-10-18T20:24:20.710 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b 2023-10-18T20:24:20.711 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hE0PBU4SG9FRrvD2QO7JDVdINCg= 2023-10-18T20:24:20.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hE0PBU4SG9FRrvD2QO7JDVdINCg=', 'Content-Length': '0'} 2023-10-18T20:24:20.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5eae9a46ba0482-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:24:20.714 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.714 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.715 DEBUG:boto:Method: GET 2023-10-18T20:24:20.715 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:24:20.715 DEBUG:boto:Data: 2023-10-18T20:24:20.716 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.716 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.717 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.717 DEBUG:boto:Params: {} 2023-10-18T20:24:20.717 DEBUG:boto:Token: None 2023-10-18T20:24:20.718 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b 2023-10-18T20:24:20.718 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Hk6HYZ0AwZF8aswzSychJ5INgU= 2023-10-18T20:24:20.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Hk6HYZ0AwZF8aswzSychJ5INgU=', 'Content-Length': '0'} 2023-10-18T20:24:20.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ed0406bf92a4479-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.721 DEBUG:boto:path=/bofrnf-55/b 2023-10-18T20:24:20.721 DEBUG:boto:auth_path=/bofrnf-55/b 2023-10-18T20:24:20.722 DEBUG:boto:Method: GET 2023-10-18T20:24:20.722 DEBUG:boto:Path: /bofrnf-55/b 2023-10-18T20:24:20.723 DEBUG:boto:Data: 2023-10-18T20:24:20.723 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.724 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.724 DEBUG:boto:Params: {} 2023-10-18T20:24:20.724 DEBUG:boto:Token: None 2023-10-18T20:24:20.725 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/b 2023-10-18T20:24:20.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Hk6HYZ0AwZF8aswzSychJ5INgU= 2023-10-18T20:24:20.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Hk6HYZ0AwZF8aswzSychJ5INgU=', 'Content-Length': '0'} 2023-10-18T20:24:20.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49b5a5d14286bea-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.728 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.729 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.729 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:24:20.729 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:24:20.730 DEBUG:boto:Method: GET 2023-10-18T20:24:20.730 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:24:20.730 DEBUG:boto:Data: 2023-10-18T20:24:20.731 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.731 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.732 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.732 DEBUG:boto:Params: {} 2023-10-18T20:24:20.732 DEBUG:boto:Token: None 2023-10-18T20:24:20.733 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.733 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006966e88320bbf339-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.737 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.737 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.738 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.738 DEBUG:boto:path=/bofrnf-55/?prefix=b 2023-10-18T20:24:20.738 DEBUG:boto:auth_path=/bofrnf-55/?prefix=b 2023-10-18T20:24:20.739 DEBUG:boto:Method: GET 2023-10-18T20:24:20.739 DEBUG:boto:Path: /bofrnf-55/?prefix=b 2023-10-18T20:24:20.739 DEBUG:boto:Data: 2023-10-18T20:24:20.740 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.740 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.741 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.741 DEBUG:boto:Params: {} 2023-10-18T20:24:20.741 DEBUG:boto:Token: None 2023-10-18T20:24:20.742 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.742 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013eba0e45571c5a2-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.745 DEBUG:boto:b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:24:20.746 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.747 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.747 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.748 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.748 DEBUG:boto:Method: GET 2023-10-18T20:24:20.748 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.749 DEBUG:boto:Data: 2023-10-18T20:24:20.749 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.749 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.750 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.750 DEBUG:boto:Params: {} 2023-10-18T20:24:20.751 DEBUG:boto:Token: None 2023-10-18T20:24:20.751 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.751 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA= 2023-10-18T20:24:20.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA=', 'Content-Length': '0'} 2023-10-18T20:24:20.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5078623edd49d68-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.754 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.755 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.755 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.755 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.756 DEBUG:boto:Method: GET 2023-10-18T20:24:20.756 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.757 DEBUG:boto:Data: 2023-10-18T20:24:20.757 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.757 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.758 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.758 DEBUG:boto:Params: {} 2023-10-18T20:24:20.759 DEBUG:boto:Token: None 2023-10-18T20:24:20.759 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.759 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA= 2023-10-18T20:24:20.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA=', 'Content-Length': '0'} 2023-10-18T20:24:20.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd058875b88a68d-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.763 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.763 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.763 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.764 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.764 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.764 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.765 DEBUG:boto:Data: 2023-10-18T20:24:20.765 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.766 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.766 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.766 DEBUG:boto:Params: {} 2023-10-18T20:24:20.767 DEBUG:boto:Token: None 2023-10-18T20:24:20.767 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.768 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FaZzi79yTbHI2xI5MU8kRb2ta6o= 2023-10-18T20:24:20.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FaZzi79yTbHI2xI5MU8kRb2ta6o=', 'Content-Length': '0'} 2023-10-18T20:24:20.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000259e01ae3a89302d-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.770 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.771 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.771 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.771 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.772 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.772 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.773 DEBUG:boto:Data: 2023-10-18T20:24:20.773 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.773 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.774 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.774 DEBUG:boto:Params: {} 2023-10-18T20:24:20.775 DEBUG:boto:Token: None 2023-10-18T20:24:20.775 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.775 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FaZzi79yTbHI2xI5MU8kRb2ta6o= 2023-10-18T20:24:20.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FaZzi79yTbHI2xI5MU8kRb2ta6o=', 'Content-Length': '0'} 2023-10-18T20:24:20.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c63ae4e7740161-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:24:20.779 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.779 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.780 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.780 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.780 DEBUG:boto:Method: GET 2023-10-18T20:24:20.781 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.782 DEBUG:boto:Data: 2023-10-18T20:24:20.783 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.783 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.784 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.785 DEBUG:boto:Params: {} 2023-10-18T20:24:20.786 DEBUG:boto:Token: None 2023-10-18T20:24:20.787 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.787 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA= 2023-10-18T20:24:20.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA=', 'Content-Length': '0'} 2023-10-18T20:24:20.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed32c02909044c4a-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.791 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.792 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.793 DEBUG:boto:path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.793 DEBUG:boto:auth_path=/bofrnf-55/c?versionId=null 2023-10-18T20:24:20.794 DEBUG:boto:Method: GET 2023-10-18T20:24:20.795 DEBUG:boto:Path: /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.796 DEBUG:boto:Data: 2023-10-18T20:24:20.796 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.797 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.797 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.798 DEBUG:boto:Params: {} 2023-10-18T20:24:20.798 DEBUG:boto:Token: None 2023-10-18T20:24:20.799 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c?versionId=null 2023-10-18T20:24:20.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA= 2023-10-18T20:24:20.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA=', 'Content-Length': '0'} 2023-10-18T20:24:20.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006642de39c70f677-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.802 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.803 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.803 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:24:20.803 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:24:20.804 DEBUG:boto:Method: GET 2023-10-18T20:24:20.804 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:24:20.804 DEBUG:boto:Data: 2023-10-18T20:24:20.805 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.806 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.806 DEBUG:boto:Params: {} 2023-10-18T20:24:20.806 DEBUG:boto:Token: None 2023-10-18T20:24:20.807 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.807 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8b0cbc73b7d984-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.850 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.851 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.851 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.852 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:24:20.852 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:24:20.852 DEBUG:boto:Method: GET 2023-10-18T20:24:20.853 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:24:20.853 DEBUG:boto:Data: 2023-10-18T20:24:20.853 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.854 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.854 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.855 DEBUG:boto:Params: {} 2023-10-18T20:24:20.855 DEBUG:boto:Token: None 2023-10-18T20:24:20.855 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.856 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f934a8567b60be42-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.859 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.859 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.860 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.860 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.860 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:24:20.861 DEBUG:boto:Data: 2023-10-18T20:24:20.861 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.861 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.862 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.862 DEBUG:boto:Params: {} 2023-10-18T20:24:20.863 DEBUG:boto:Token: None 2023-10-18T20:24:20.863 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c 2023-10-18T20:24:20.863 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sEi4yJccpmT9L5LeGUX2oZmVY88= 2023-10-18T20:24:20.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sEi4yJccpmT9L5LeGUX2oZmVY88=', 'Content-Length': '0'} 2023-10-18T20:24:20.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd96062112bca7c-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.866 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.866 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.867 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.867 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:24:20.867 DEBUG:boto:Data: 2023-10-18T20:24:20.868 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.868 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.869 DEBUG:boto:Params: {} 2023-10-18T20:24:20.869 DEBUG:boto:Token: None 2023-10-18T20:24:20.870 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c 2023-10-18T20:24:20.870 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sEi4yJccpmT9L5LeGUX2oZmVY88= 2023-10-18T20:24:20.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sEi4yJccpmT9L5LeGUX2oZmVY88=', 'Content-Length': '0'} 2023-10-18T20:24:20.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b47492ffd2a964-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:24:20.873 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.873 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.874 DEBUG:boto:Method: GET 2023-10-18T20:24:20.874 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:24:20.874 DEBUG:boto:Data: 2023-10-18T20:24:20.875 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.876 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.876 DEBUG:boto:Params: {} 2023-10-18T20:24:20.876 DEBUG:boto:Token: None 2023-10-18T20:24:20.877 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c 2023-10-18T20:24:20.877 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SiCEjAH22VfvZKFXKGShKNp1fiY= 2023-10-18T20:24:20.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SiCEjAH22VfvZKFXKGShKNp1fiY=', 'Content-Length': '0'} 2023-10-18T20:24:20.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff73cf761ad07ba-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.880 DEBUG:boto:path=/bofrnf-55/c 2023-10-18T20:24:20.881 DEBUG:boto:auth_path=/bofrnf-55/c 2023-10-18T20:24:20.881 DEBUG:boto:Method: GET 2023-10-18T20:24:20.881 DEBUG:boto:Path: /bofrnf-55/c 2023-10-18T20:24:20.882 DEBUG:boto:Data: 2023-10-18T20:24:20.882 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.883 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.883 DEBUG:boto:Params: {} 2023-10-18T20:24:20.884 DEBUG:boto:Token: None 2023-10-18T20:24:20.884 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/c 2023-10-18T20:24:20.885 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SiCEjAH22VfvZKFXKGShKNp1fiY= 2023-10-18T20:24:20.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SiCEjAH22VfvZKFXKGShKNp1fiY=', 'Content-Length': '0'} 2023-10-18T20:24:20.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b5ca264500940b-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.930 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.931 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.931 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:24:20.932 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:24:20.932 DEBUG:boto:Method: GET 2023-10-18T20:24:20.932 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:24:20.933 DEBUG:boto:Data: 2023-10-18T20:24:20.933 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.934 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.934 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.934 DEBUG:boto:Params: {} 2023-10-18T20:24:20.935 DEBUG:boto:Token: None 2023-10-18T20:24:20.935 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad4ea85558e3282c-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.939 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.939 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:20.940 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:20.940 DEBUG:boto:path=/bofrnf-55/?prefix=c 2023-10-18T20:24:20.940 DEBUG:boto:auth_path=/bofrnf-55/?prefix=c 2023-10-18T20:24:20.941 DEBUG:boto:Method: GET 2023-10-18T20:24:20.941 DEBUG:boto:Path: /bofrnf-55/?prefix=c 2023-10-18T20:24:20.941 DEBUG:boto:Data: 2023-10-18T20:24:20.942 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.942 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.943 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.943 DEBUG:boto:Params: {} 2023-10-18T20:24:20.944 DEBUG:boto:Token: None 2023-10-18T20:24:20.944 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/ 2023-10-18T20:24:20.944 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T20:24:20.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T20:24:20.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025eb8624fbba5fe3-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.947 DEBUG:boto:b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:20.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:24:20.948 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:20.949 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:20.949 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.949 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.950 DEBUG:boto:Method: GET 2023-10-18T20:24:20.950 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.951 DEBUG:boto:Data: 2023-10-18T20:24:20.951 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.951 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.952 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.952 DEBUG:boto:Params: {} 2023-10-18T20:24:20.953 DEBUG:boto:Token: None 2023-10-18T20:24:20.953 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.954 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI= 2023-10-18T20:24:20.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI=', 'Content-Length': '0'} 2023-10-18T20:24:20.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000492c08782f112c56-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.957 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:20.957 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:20.958 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.958 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.958 DEBUG:boto:Method: GET 2023-10-18T20:24:20.959 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.959 DEBUG:boto:Data: 2023-10-18T20:24:20.960 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.960 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.960 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.961 DEBUG:boto:Params: {} 2023-10-18T20:24:20.961 DEBUG:boto:Token: None 2023-10-18T20:24:20.962 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.962 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI= 2023-10-18T20:24:20.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI=', 'Content-Length': '0'} 2023-10-18T20:24:20.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005b645e36c73e5c5-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.965 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:20.965 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:20.966 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.966 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.966 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.967 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.967 DEBUG:boto:Data: 2023-10-18T20:24:20.968 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.968 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.968 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.969 DEBUG:boto:Params: {} 2023-10-18T20:24:20.969 DEBUG:boto:Token: None 2023-10-18T20:24:20.969 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.970 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4jv7kbyfhq1XrKReKOG0NC3Lmug= 2023-10-18T20:24:20.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4jv7kbyfhq1XrKReKOG0NC3Lmug=', 'Content-Length': '0'} 2023-10-18T20:24:20.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001066f6e2ee19d917-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.973 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:20.973 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:20.973 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.974 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.974 DEBUG:boto:Method: HEAD 2023-10-18T20:24:20.974 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.975 DEBUG:boto:Data: 2023-10-18T20:24:20.975 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.975 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.976 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.976 DEBUG:boto:Params: {} 2023-10-18T20:24:20.977 DEBUG:boto:Token: None 2023-10-18T20:24:20.977 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.977 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4jv7kbyfhq1XrKReKOG0NC3Lmug= 2023-10-18T20:24:20.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4jv7kbyfhq1XrKReKOG0NC3Lmug=', 'Content-Length': '0'} 2023-10-18T20:24:20.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2af80ed35b835c0-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:24:20.981 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:20.981 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:20.982 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.982 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.983 DEBUG:boto:Method: GET 2023-10-18T20:24:20.983 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.984 DEBUG:boto:Data: 2023-10-18T20:24:20.984 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.985 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:20.986 DEBUG:boto:Port: 8001 2023-10-18T20:24:20.986 DEBUG:boto:Params: {} 2023-10-18T20:24:20.987 DEBUG:boto:Token: None 2023-10-18T20:24:20.988 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.989 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI= 2023-10-18T20:24:20.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI=', 'Content-Length': '0'} 2023-10-18T20:24:20.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a456e4a8782307f7-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:20.992 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:20.992 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:20.992 DEBUG:boto:path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.993 DEBUG:boto:auth_path=/bofrnf-55/d?versionId=null 2023-10-18T20:24:20.993 DEBUG:boto:Method: GET 2023-10-18T20:24:20.994 DEBUG:boto:Path: /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.994 DEBUG:boto:Data: 2023-10-18T20:24:20.994 DEBUG:boto:Headers: {} 2023-10-18T20:24:20.995 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:20.995 DEBUG:boto:Port: 8000 2023-10-18T20:24:20.996 DEBUG:boto:Params: {} 2023-10-18T20:24:20.996 DEBUG:boto:Token: None 2023-10-18T20:24:20.996 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:20 GMT /bofrnf-55/d?versionId=null 2023-10-18T20:24:20.997 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI= 2023-10-18T20:24:20.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI=', 'Content-Length': '0'} 2023-10-18T20:24:20.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053f942e6534aa6dc-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.000 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.001 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.001 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:24:21.001 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:24:21.002 DEBUG:boto:Method: GET 2023-10-18T20:24:21.002 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:24:21.003 DEBUG:boto:Data: 2023-10-18T20:24:21.003 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.004 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.004 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.004 DEBUG:boto:Params: {} 2023-10-18T20:24:21.005 DEBUG:boto:Token: None 2023-10-18T20:24:21.005 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.005 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a37b88677885194c-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.050 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.051 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.051 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.052 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:24:21.052 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:24:21.052 DEBUG:boto:Method: GET 2023-10-18T20:24:21.053 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:24:21.053 DEBUG:boto:Data: 2023-10-18T20:24:21.054 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.054 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.054 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.055 DEBUG:boto:Params: {} 2023-10-18T20:24:21.055 DEBUG:boto:Token: None 2023-10-18T20:24:21.056 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.056 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097265c515256caa4-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.059 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.060 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:21.060 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:21.060 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.061 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:24:21.061 DEBUG:boto:Data: 2023-10-18T20:24:21.061 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.062 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.062 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.063 DEBUG:boto:Params: {} 2023-10-18T20:24:21.063 DEBUG:boto:Token: None 2023-10-18T20:24:21.063 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/d 2023-10-18T20:24:21.064 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jg1JA9Ain07Hfi2VLbmtuS6Tahw= 2023-10-18T20:24:21.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jg1JA9Ain07Hfi2VLbmtuS6Tahw=', 'Content-Length': '0'} 2023-10-18T20:24:21.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000743f6ce08f0e19e0-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.066 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:21.067 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:21.067 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.068 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:24:21.068 DEBUG:boto:Data: 2023-10-18T20:24:21.068 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.069 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.069 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.069 DEBUG:boto:Params: {} 2023-10-18T20:24:21.070 DEBUG:boto:Token: None 2023-10-18T20:24:21.070 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/d 2023-10-18T20:24:21.071 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jg1JA9Ain07Hfi2VLbmtuS6Tahw= 2023-10-18T20:24:21.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jg1JA9Ain07Hfi2VLbmtuS6Tahw=', 'Content-Length': '0'} 2023-10-18T20:24:21.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009314ebf438ccb496-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:24:21.074 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:21.074 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:21.075 DEBUG:boto:Method: GET 2023-10-18T20:24:21.075 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:24:21.075 DEBUG:boto:Data: 2023-10-18T20:24:21.076 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.076 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.077 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.077 DEBUG:boto:Params: {} 2023-10-18T20:24:21.077 DEBUG:boto:Token: None 2023-10-18T20:24:21.078 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/d 2023-10-18T20:24:21.078 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IckQ0iH/CJg7dUW+95WfkdyqxLs= 2023-10-18T20:24:21.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IckQ0iH/CJg7dUW+95WfkdyqxLs=', 'Content-Length': '0'} 2023-10-18T20:24:21.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062957882fbe18c3b-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.081 DEBUG:boto:path=/bofrnf-55/d 2023-10-18T20:24:21.081 DEBUG:boto:auth_path=/bofrnf-55/d 2023-10-18T20:24:21.082 DEBUG:boto:Method: GET 2023-10-18T20:24:21.082 DEBUG:boto:Path: /bofrnf-55/d 2023-10-18T20:24:21.082 DEBUG:boto:Data: 2023-10-18T20:24:21.083 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.083 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.084 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.084 DEBUG:boto:Params: {} 2023-10-18T20:24:21.084 DEBUG:boto:Token: None 2023-10-18T20:24:21.085 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/d 2023-10-18T20:24:21.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IckQ0iH/CJg7dUW+95WfkdyqxLs= 2023-10-18T20:24:21.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IckQ0iH/CJg7dUW+95WfkdyqxLs=', 'Content-Length': '0'} 2023-10-18T20:24:21.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fade565dca7ae4a2-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.130 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.131 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.131 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:24:21.132 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:24:21.132 DEBUG:boto:Method: GET 2023-10-18T20:24:21.132 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:24:21.133 DEBUG:boto:Data: 2023-10-18T20:24:21.133 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.134 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.134 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.134 DEBUG:boto:Params: {} 2023-10-18T20:24:21.135 DEBUG:boto:Token: None 2023-10-18T20:24:21.135 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.136 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000432fb6cad655c18b-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.139 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.140 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.140 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.141 DEBUG:boto:path=/bofrnf-55/?prefix=d 2023-10-18T20:24:21.141 DEBUG:boto:auth_path=/bofrnf-55/?prefix=d 2023-10-18T20:24:21.142 DEBUG:boto:Method: GET 2023-10-18T20:24:21.142 DEBUG:boto:Path: /bofrnf-55/?prefix=d 2023-10-18T20:24:21.142 DEBUG:boto:Data: 2023-10-18T20:24:21.143 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.143 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.144 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.144 DEBUG:boto:Params: {} 2023-10-18T20:24:21.145 DEBUG:boto:Token: None 2023-10-18T20:24:21.145 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.146 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022e2354d56aa7655-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.148 DEBUG:boto:b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:24:21.149 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.150 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.150 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.151 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.151 DEBUG:boto:Method: GET 2023-10-18T20:24:21.151 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.152 DEBUG:boto:Data: 2023-10-18T20:24:21.152 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.153 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.153 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.153 DEBUG:boto:Params: {} 2023-10-18T20:24:21.154 DEBUG:boto:Token: None 2023-10-18T20:24:21.154 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.155 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q= 2023-10-18T20:24:21.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q=', 'Content-Length': '0'} 2023-10-18T20:24:21.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048ce49a41b5186ae-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.157 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.158 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.158 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.159 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.159 DEBUG:boto:Method: GET 2023-10-18T20:24:21.160 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.160 DEBUG:boto:Data: 2023-10-18T20:24:21.160 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.161 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.161 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.161 DEBUG:boto:Params: {} 2023-10-18T20:24:21.162 DEBUG:boto:Token: None 2023-10-18T20:24:21.162 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q= 2023-10-18T20:24:21.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q=', 'Content-Length': '0'} 2023-10-18T20:24:21.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7a00e647be0e51-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.166 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.166 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.167 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.167 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.167 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.168 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.168 DEBUG:boto:Data: 2023-10-18T20:24:21.168 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.170 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.170 DEBUG:boto:Params: {} 2023-10-18T20:24:21.170 DEBUG:boto:Token: None 2023-10-18T20:24:21.171 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nqGZcqvuxeKqvak6R2RWMDmMIgA= 2023-10-18T20:24:21.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nqGZcqvuxeKqvak6R2RWMDmMIgA=', 'Content-Length': '0'} 2023-10-18T20:24:21.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6904bd91cc710b-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.174 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.174 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.175 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.175 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.176 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.176 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.176 DEBUG:boto:Data: 2023-10-18T20:24:21.177 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.177 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.177 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.178 DEBUG:boto:Params: {} 2023-10-18T20:24:21.178 DEBUG:boto:Token: None 2023-10-18T20:24:21.178 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.179 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nqGZcqvuxeKqvak6R2RWMDmMIgA= 2023-10-18T20:24:21.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nqGZcqvuxeKqvak6R2RWMDmMIgA=', 'Content-Length': '0'} 2023-10-18T20:24:21.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024ef9ae549869e60-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:24:21.182 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.183 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.183 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.183 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.184 DEBUG:boto:Method: GET 2023-10-18T20:24:21.184 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.185 DEBUG:boto:Data: 2023-10-18T20:24:21.185 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.186 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.186 DEBUG:boto:Params: {} 2023-10-18T20:24:21.186 DEBUG:boto:Token: None 2023-10-18T20:24:21.187 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.187 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q= 2023-10-18T20:24:21.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q=', 'Content-Length': '0'} 2023-10-18T20:24:21.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b31df91c1ec068-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.190 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.191 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.191 DEBUG:boto:path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.191 DEBUG:boto:auth_path=/bofrnf-55/e?versionId=null 2023-10-18T20:24:21.192 DEBUG:boto:Method: GET 2023-10-18T20:24:21.192 DEBUG:boto:Path: /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.193 DEBUG:boto:Data: 2023-10-18T20:24:21.193 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.194 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.194 DEBUG:boto:Params: {} 2023-10-18T20:24:21.195 DEBUG:boto:Token: None 2023-10-18T20:24:21.195 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e?versionId=null 2023-10-18T20:24:21.195 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q= 2023-10-18T20:24:21.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q=', 'Content-Length': '0'} 2023-10-18T20:24:21.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80f2c66581dc10e-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.198 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.199 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.199 DEBUG:boto:path=/bofrnf-55/?prefix=e 2023-10-18T20:24:21.200 DEBUG:boto:auth_path=/bofrnf-55/?prefix=e 2023-10-18T20:24:21.200 DEBUG:boto:Method: GET 2023-10-18T20:24:21.200 DEBUG:boto:Path: /bofrnf-55/?prefix=e 2023-10-18T20:24:21.201 DEBUG:boto:Data: 2023-10-18T20:24:21.201 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.202 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.202 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.202 DEBUG:boto:Params: {} 2023-10-18T20:24:21.203 DEBUG:boto:Token: None 2023-10-18T20:24:21.203 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.204 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3e361b17abb6b3a-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.207 DEBUG:boto:b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.207 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.208 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.208 DEBUG:boto:path=/bofrnf-55/?prefix=e 2023-10-18T20:24:21.209 DEBUG:boto:auth_path=/bofrnf-55/?prefix=e 2023-10-18T20:24:21.209 DEBUG:boto:Method: GET 2023-10-18T20:24:21.209 DEBUG:boto:Path: /bofrnf-55/?prefix=e 2023-10-18T20:24:21.210 DEBUG:boto:Data: 2023-10-18T20:24:21.210 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.211 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.211 DEBUG:boto:Params: {} 2023-10-18T20:24:21.212 DEBUG:boto:Token: None 2023-10-18T20:24:21.212 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005adcc1afbaee70ac-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.216 DEBUG:boto:b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.216 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.216 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.217 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.217 DEBUG:boto:Path: /bofrnf-55/e 2023-10-18T20:24:21.218 DEBUG:boto:Data: 2023-10-18T20:24:21.218 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.219 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.219 DEBUG:boto:Params: {} 2023-10-18T20:24:21.219 DEBUG:boto:Token: None 2023-10-18T20:24:21.220 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e 2023-10-18T20:24:21.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qab1GfcLJN1fMPiU3L9E2QLQ/YI= 2023-10-18T20:24:21.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qab1GfcLJN1fMPiU3L9E2QLQ/YI=', 'Content-Length': '0'} 2023-10-18T20:24:21.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160e2bd12084f5af-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.223 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.223 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.224 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.224 DEBUG:boto:Path: /bofrnf-55/e 2023-10-18T20:24:21.225 DEBUG:boto:Data: 2023-10-18T20:24:21.225 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.225 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.226 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.226 DEBUG:boto:Params: {} 2023-10-18T20:24:21.226 DEBUG:boto:Token: None 2023-10-18T20:24:21.227 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e 2023-10-18T20:24:21.227 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qab1GfcLJN1fMPiU3L9E2QLQ/YI= 2023-10-18T20:24:21.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qab1GfcLJN1fMPiU3L9E2QLQ/YI=', 'Content-Length': '0'} 2023-10-18T20:24:21.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139c31ae1892bc0f-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:24:21.230 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.231 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.231 DEBUG:boto:Method: GET 2023-10-18T20:24:21.231 DEBUG:boto:Path: /bofrnf-55/e 2023-10-18T20:24:21.232 DEBUG:boto:Data: 2023-10-18T20:24:21.232 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.233 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.233 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.233 DEBUG:boto:Params: {} 2023-10-18T20:24:21.234 DEBUG:boto:Token: None 2023-10-18T20:24:21.234 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e 2023-10-18T20:24:21.235 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4YMjfD4bfDWAvC86jdWyF87bks8= 2023-10-18T20:24:21.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4YMjfD4bfDWAvC86jdWyF87bks8=', 'Content-Length': '0'} 2023-10-18T20:24:21.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c292cd3edd79a7a2-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.278 DEBUG:boto:path=/bofrnf-55/e 2023-10-18T20:24:21.279 DEBUG:boto:auth_path=/bofrnf-55/e 2023-10-18T20:24:21.279 DEBUG:boto:Method: GET 2023-10-18T20:24:21.279 DEBUG:boto:Path: /bofrnf-55/e 2023-10-18T20:24:21.280 DEBUG:boto:Data: 2023-10-18T20:24:21.280 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.281 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.281 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.281 DEBUG:boto:Params: {} 2023-10-18T20:24:21.282 DEBUG:boto:Token: None 2023-10-18T20:24:21.282 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/e 2023-10-18T20:24:21.283 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4YMjfD4bfDWAvC86jdWyF87bks8= 2023-10-18T20:24:21.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4YMjfD4bfDWAvC86jdWyF87bks8=', 'Content-Length': '0'} 2023-10-18T20:24:21.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212a15d21f556a04-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.286 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.287 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.287 DEBUG:boto:path=/bofrnf-55/?prefix=e 2023-10-18T20:24:21.287 DEBUG:boto:auth_path=/bofrnf-55/?prefix=e 2023-10-18T20:24:21.288 DEBUG:boto:Method: GET 2023-10-18T20:24:21.288 DEBUG:boto:Path: /bofrnf-55/?prefix=e 2023-10-18T20:24:21.288 DEBUG:boto:Data: 2023-10-18T20:24:21.289 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.289 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.290 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.290 DEBUG:boto:Params: {} 2023-10-18T20:24:21.290 DEBUG:boto:Token: None 2023-10-18T20:24:21.291 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097077375783ee6ec-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.295 DEBUG:boto:b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.295 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.296 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.296 DEBUG:boto:path=/bofrnf-55/?prefix=e 2023-10-18T20:24:21.296 DEBUG:boto:auth_path=/bofrnf-55/?prefix=e 2023-10-18T20:24:21.297 DEBUG:boto:Method: GET 2023-10-18T20:24:21.297 DEBUG:boto:Path: /bofrnf-55/?prefix=e 2023-10-18T20:24:21.297 DEBUG:boto:Data: 2023-10-18T20:24:21.298 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.298 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.299 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.299 DEBUG:boto:Params: {} 2023-10-18T20:24:21.299 DEBUG:boto:Token: None 2023-10-18T20:24:21.300 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.300 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d1f8872adf1c0c5-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.304 DEBUG:boto:b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:24:21.304 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.305 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.305 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.306 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.306 DEBUG:boto:Method: GET 2023-10-18T20:24:21.307 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.307 DEBUG:boto:Data: 2023-10-18T20:24:21.307 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.308 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.308 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.308 DEBUG:boto:Params: {} 2023-10-18T20:24:21.309 DEBUG:boto:Token: None 2023-10-18T20:24:21.309 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.310 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA= 2023-10-18T20:24:21.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA=', 'Content-Length': '0'} 2023-10-18T20:24:21.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2d89ec10bfae8f-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.313 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.314 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.314 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.315 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.315 DEBUG:boto:Method: GET 2023-10-18T20:24:21.316 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.316 DEBUG:boto:Data: 2023-10-18T20:24:21.317 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.317 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.318 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.319 DEBUG:boto:Params: {} 2023-10-18T20:24:21.319 DEBUG:boto:Token: None 2023-10-18T20:24:21.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA= 2023-10-18T20:24:21.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA=', 'Content-Length': '0'} 2023-10-18T20:24:21.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053824e7f51d74b9c-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.366 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.367 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.367 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.367 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.368 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.368 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.369 DEBUG:boto:Data: 2023-10-18T20:24:21.369 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.369 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.370 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.370 DEBUG:boto:Params: {} 2023-10-18T20:24:21.370 DEBUG:boto:Token: None 2023-10-18T20:24:21.371 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.371 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8mA9voj8wYW2FW98EUW+HmHG4rM= 2023-10-18T20:24:21.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8mA9voj8wYW2FW98EUW+HmHG4rM=', 'Content-Length': '0'} 2023-10-18T20:24:21.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000267d057999f16c65-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.374 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.374 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.375 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.375 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.376 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.376 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.376 DEBUG:boto:Data: 2023-10-18T20:24:21.377 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.378 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.378 DEBUG:boto:Params: {} 2023-10-18T20:24:21.378 DEBUG:boto:Token: None 2023-10-18T20:24:21.379 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.379 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8mA9voj8wYW2FW98EUW+HmHG4rM= 2023-10-18T20:24:21.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8mA9voj8wYW2FW98EUW+HmHG4rM=', 'Content-Length': '0'} 2023-10-18T20:24:21.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae72e29d37395db-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:24:21.383 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.383 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.384 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.384 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.384 DEBUG:boto:Method: GET 2023-10-18T20:24:21.385 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.385 DEBUG:boto:Data: 2023-10-18T20:24:21.386 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.386 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.387 DEBUG:boto:Params: {} 2023-10-18T20:24:21.387 DEBUG:boto:Token: None 2023-10-18T20:24:21.388 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA= 2023-10-18T20:24:21.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA=', 'Content-Length': '0'} 2023-10-18T20:24:21.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4167d42109bc0fd-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.391 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.391 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.392 DEBUG:boto:path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.392 DEBUG:boto:auth_path=/bofrnf-55/f?versionId=null 2023-10-18T20:24:21.393 DEBUG:boto:Method: GET 2023-10-18T20:24:21.393 DEBUG:boto:Path: /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.393 DEBUG:boto:Data: 2023-10-18T20:24:21.394 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.394 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.395 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.395 DEBUG:boto:Params: {} 2023-10-18T20:24:21.395 DEBUG:boto:Token: None 2023-10-18T20:24:21.396 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f?versionId=null 2023-10-18T20:24:21.396 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA= 2023-10-18T20:24:21.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA=', 'Content-Length': '0'} 2023-10-18T20:24:21.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cda27cb496cbcd17-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.399 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.399 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.400 DEBUG:boto:path=/bofrnf-55/?prefix=f 2023-10-18T20:24:21.400 DEBUG:boto:auth_path=/bofrnf-55/?prefix=f 2023-10-18T20:24:21.401 DEBUG:boto:Method: GET 2023-10-18T20:24:21.401 DEBUG:boto:Path: /bofrnf-55/?prefix=f 2023-10-18T20:24:21.401 DEBUG:boto:Data: 2023-10-18T20:24:21.402 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.402 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.403 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.403 DEBUG:boto:Params: {} 2023-10-18T20:24:21.403 DEBUG:boto:Token: None 2023-10-18T20:24:21.404 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033899ee9d21a65cd-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.450 DEBUG:boto:b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.451 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.452 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.453 DEBUG:boto:path=/bofrnf-55/?prefix=f 2023-10-18T20:24:21.453 DEBUG:boto:auth_path=/bofrnf-55/?prefix=f 2023-10-18T20:24:21.454 DEBUG:boto:Method: GET 2023-10-18T20:24:21.455 DEBUG:boto:Path: /bofrnf-55/?prefix=f 2023-10-18T20:24:21.456 DEBUG:boto:Data: 2023-10-18T20:24:21.456 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.458 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.459 DEBUG:boto:Params: {} 2023-10-18T20:24:21.459 DEBUG:boto:Token: None 2023-10-18T20:24:21.460 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.461 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000673f8b435ca33c83-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.464 DEBUG:boto:b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.465 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.465 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.466 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.466 DEBUG:boto:Path: /bofrnf-55/f 2023-10-18T20:24:21.467 DEBUG:boto:Data: 2023-10-18T20:24:21.467 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.467 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.468 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.468 DEBUG:boto:Params: {} 2023-10-18T20:24:21.468 DEBUG:boto:Token: None 2023-10-18T20:24:21.469 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f 2023-10-18T20:24:21.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s2R/StI8reiQYXj7UADNHY8WnLo= 2023-10-18T20:24:21.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s2R/StI8reiQYXj7UADNHY8WnLo=', 'Content-Length': '0'} 2023-10-18T20:24:21.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c92505149650543c-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.472 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.472 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.473 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.473 DEBUG:boto:Path: /bofrnf-55/f 2023-10-18T20:24:21.474 DEBUG:boto:Data: 2023-10-18T20:24:21.474 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.475 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.475 DEBUG:boto:Params: {} 2023-10-18T20:24:21.475 DEBUG:boto:Token: None 2023-10-18T20:24:21.476 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f 2023-10-18T20:24:21.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s2R/StI8reiQYXj7UADNHY8WnLo= 2023-10-18T20:24:21.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s2R/StI8reiQYXj7UADNHY8WnLo=', 'Content-Length': '0'} 2023-10-18T20:24:21.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099f1684b0829b51-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:24:21.479 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.480 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.480 DEBUG:boto:Method: GET 2023-10-18T20:24:21.481 DEBUG:boto:Path: /bofrnf-55/f 2023-10-18T20:24:21.481 DEBUG:boto:Data: 2023-10-18T20:24:21.481 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.482 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.482 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.483 DEBUG:boto:Params: {} 2023-10-18T20:24:21.483 DEBUG:boto:Token: None 2023-10-18T20:24:21.483 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f 2023-10-18T20:24:21.484 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HubOixa8CKyavZ3TR+dZ1ZxLtZc= 2023-10-18T20:24:21.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HubOixa8CKyavZ3TR+dZ1ZxLtZc=', 'Content-Length': '0'} 2023-10-18T20:24:21.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007aeb56e1f580553-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.486 DEBUG:boto:path=/bofrnf-55/f 2023-10-18T20:24:21.487 DEBUG:boto:auth_path=/bofrnf-55/f 2023-10-18T20:24:21.487 DEBUG:boto:Method: GET 2023-10-18T20:24:21.488 DEBUG:boto:Path: /bofrnf-55/f 2023-10-18T20:24:21.488 DEBUG:boto:Data: 2023-10-18T20:24:21.488 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.489 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.489 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.489 DEBUG:boto:Params: {} 2023-10-18T20:24:21.490 DEBUG:boto:Token: None 2023-10-18T20:24:21.490 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/f 2023-10-18T20:24:21.491 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HubOixa8CKyavZ3TR+dZ1ZxLtZc= 2023-10-18T20:24:21.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HubOixa8CKyavZ3TR+dZ1ZxLtZc=', 'Content-Length': '0'} 2023-10-18T20:24:21.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4021df9bdf6b029-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.494 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.494 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.494 DEBUG:boto:path=/bofrnf-55/?prefix=f 2023-10-18T20:24:21.495 DEBUG:boto:auth_path=/bofrnf-55/?prefix=f 2023-10-18T20:24:21.495 DEBUG:boto:Method: GET 2023-10-18T20:24:21.496 DEBUG:boto:Path: /bofrnf-55/?prefix=f 2023-10-18T20:24:21.496 DEBUG:boto:Data: 2023-10-18T20:24:21.496 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.497 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.497 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.497 DEBUG:boto:Params: {} 2023-10-18T20:24:21.498 DEBUG:boto:Token: None 2023-10-18T20:24:21.498 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.499 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516a6e0bff154d93-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.502 DEBUG:boto:b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.502 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.503 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.503 DEBUG:boto:path=/bofrnf-55/?prefix=f 2023-10-18T20:24:21.503 DEBUG:boto:auth_path=/bofrnf-55/?prefix=f 2023-10-18T20:24:21.504 DEBUG:boto:Method: GET 2023-10-18T20:24:21.504 DEBUG:boto:Path: /bofrnf-55/?prefix=f 2023-10-18T20:24:21.505 DEBUG:boto:Data: 2023-10-18T20:24:21.505 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.505 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.506 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.506 DEBUG:boto:Params: {} 2023-10-18T20:24:21.506 DEBUG:boto:Token: None 2023-10-18T20:24:21.507 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.507 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016061c9bfe687155-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.510 DEBUG:boto:b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:24:21.511 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.512 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.512 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.512 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.513 DEBUG:boto:Method: GET 2023-10-18T20:24:21.513 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.514 DEBUG:boto:Data: 2023-10-18T20:24:21.514 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.514 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.515 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.515 DEBUG:boto:Params: {} 2023-10-18T20:24:21.516 DEBUG:boto:Token: None 2023-10-18T20:24:21.516 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ= 2023-10-18T20:24:21.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ=', 'Content-Length': '0'} 2023-10-18T20:24:21.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c93ef91b65cbe3b-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.519 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.520 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.520 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.520 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.521 DEBUG:boto:Method: GET 2023-10-18T20:24:21.521 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.522 DEBUG:boto:Data: 2023-10-18T20:24:21.522 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.523 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.523 DEBUG:boto:Params: {} 2023-10-18T20:24:21.524 DEBUG:boto:Token: None 2023-10-18T20:24:21.524 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ= 2023-10-18T20:24:21.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ=', 'Content-Length': '0'} 2023-10-18T20:24:21.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af48cdecec7e74b-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.527 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.527 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.528 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.528 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.529 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.529 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.529 DEBUG:boto:Data: 2023-10-18T20:24:21.530 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.530 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.531 DEBUG:boto:Params: {} 2023-10-18T20:24:21.531 DEBUG:boto:Token: None 2023-10-18T20:24:21.531 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1qnP6lr+/Cv2WWoUG4C9byn7oBg= 2023-10-18T20:24:21.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1qnP6lr+/Cv2WWoUG4C9byn7oBg=', 'Content-Length': '0'} 2023-10-18T20:24:21.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009849be6e41c5ff31-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.535 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.535 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.535 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.536 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.536 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.536 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.537 DEBUG:boto:Data: 2023-10-18T20:24:21.537 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.538 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.538 DEBUG:boto:Params: {} 2023-10-18T20:24:21.539 DEBUG:boto:Token: None 2023-10-18T20:24:21.539 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.540 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1qnP6lr+/Cv2WWoUG4C9byn7oBg= 2023-10-18T20:24:21.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1qnP6lr+/Cv2WWoUG4C9byn7oBg=', 'Content-Length': '0'} 2023-10-18T20:24:21.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07e6cc49637b6b5-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:24:21.543 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.543 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.543 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.544 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.544 DEBUG:boto:Method: GET 2023-10-18T20:24:21.545 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.545 DEBUG:boto:Data: 2023-10-18T20:24:21.545 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.546 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.546 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.546 DEBUG:boto:Params: {} 2023-10-18T20:24:21.547 DEBUG:boto:Token: None 2023-10-18T20:24:21.547 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.548 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ= 2023-10-18T20:24:21.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ=', 'Content-Length': '0'} 2023-10-18T20:24:21.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017ed8bdbf93c5bc9-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.551 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.551 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.551 DEBUG:boto:path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.552 DEBUG:boto:auth_path=/bofrnf-55/g?versionId=null 2023-10-18T20:24:21.552 DEBUG:boto:Method: GET 2023-10-18T20:24:21.553 DEBUG:boto:Path: /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.553 DEBUG:boto:Data: 2023-10-18T20:24:21.553 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.554 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.554 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.554 DEBUG:boto:Params: {} 2023-10-18T20:24:21.555 DEBUG:boto:Token: None 2023-10-18T20:24:21.555 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g?versionId=null 2023-10-18T20:24:21.556 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ= 2023-10-18T20:24:21.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ=', 'Content-Length': '0'} 2023-10-18T20:24:21.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e224af1826dce0-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.598 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.599 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.599 DEBUG:boto:path=/bofrnf-55/?prefix=g 2023-10-18T20:24:21.599 DEBUG:boto:auth_path=/bofrnf-55/?prefix=g 2023-10-18T20:24:21.600 DEBUG:boto:Method: GET 2023-10-18T20:24:21.600 DEBUG:boto:Path: /bofrnf-55/?prefix=g 2023-10-18T20:24:21.601 DEBUG:boto:Data: 2023-10-18T20:24:21.601 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.601 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.602 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.602 DEBUG:boto:Params: {} 2023-10-18T20:24:21.603 DEBUG:boto:Token: None 2023-10-18T20:24:21.603 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025abbfbd46c3cd91-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.607 DEBUG:boto:b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.607 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.608 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.608 DEBUG:boto:path=/bofrnf-55/?prefix=g 2023-10-18T20:24:21.609 DEBUG:boto:auth_path=/bofrnf-55/?prefix=g 2023-10-18T20:24:21.609 DEBUG:boto:Method: GET 2023-10-18T20:24:21.609 DEBUG:boto:Path: /bofrnf-55/?prefix=g 2023-10-18T20:24:21.610 DEBUG:boto:Data: 2023-10-18T20:24:21.610 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.611 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.611 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.611 DEBUG:boto:Params: {} 2023-10-18T20:24:21.612 DEBUG:boto:Token: None 2023-10-18T20:24:21.612 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3ec1226cf8b009c-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.616 DEBUG:boto:b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.616 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.617 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.617 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.617 DEBUG:boto:Path: /bofrnf-55/g 2023-10-18T20:24:21.618 DEBUG:boto:Data: 2023-10-18T20:24:21.618 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.618 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.619 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.619 DEBUG:boto:Params: {} 2023-10-18T20:24:21.620 DEBUG:boto:Token: None 2023-10-18T20:24:21.620 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g 2023-10-18T20:24:21.620 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4in5VvuIo9OLcUMacIlF2lc1P54= 2023-10-18T20:24:21.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4in5VvuIo9OLcUMacIlF2lc1P54=', 'Content-Length': '0'} 2023-10-18T20:24:21.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075a1730b9e8ceb5-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.623 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.623 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.624 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.624 DEBUG:boto:Path: /bofrnf-55/g 2023-10-18T20:24:21.624 DEBUG:boto:Data: 2023-10-18T20:24:21.625 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.625 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.626 DEBUG:boto:Params: {} 2023-10-18T20:24:21.626 DEBUG:boto:Token: None 2023-10-18T20:24:21.627 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g 2023-10-18T20:24:21.627 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4in5VvuIo9OLcUMacIlF2lc1P54= 2023-10-18T20:24:21.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4in5VvuIo9OLcUMacIlF2lc1P54=', 'Content-Length': '0'} 2023-10-18T20:24:21.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f6c0a11de669b7-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:24:21.631 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.632 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.632 DEBUG:boto:Method: GET 2023-10-18T20:24:21.632 DEBUG:boto:Path: /bofrnf-55/g 2023-10-18T20:24:21.633 DEBUG:boto:Data: 2023-10-18T20:24:21.633 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.634 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.634 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.634 DEBUG:boto:Params: {} 2023-10-18T20:24:21.635 DEBUG:boto:Token: None 2023-10-18T20:24:21.635 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g 2023-10-18T20:24:21.636 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VV6trtJKbYsV81bVb8DsgqE+D9E= 2023-10-18T20:24:21.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VV6trtJKbYsV81bVb8DsgqE+D9E=', 'Content-Length': '0'} 2023-10-18T20:24:21.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e881b329942fc71f-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.678 DEBUG:boto:path=/bofrnf-55/g 2023-10-18T20:24:21.679 DEBUG:boto:auth_path=/bofrnf-55/g 2023-10-18T20:24:21.679 DEBUG:boto:Method: GET 2023-10-18T20:24:21.679 DEBUG:boto:Path: /bofrnf-55/g 2023-10-18T20:24:21.680 DEBUG:boto:Data: 2023-10-18T20:24:21.680 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.681 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.681 DEBUG:boto:Params: {} 2023-10-18T20:24:21.682 DEBUG:boto:Token: None 2023-10-18T20:24:21.682 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/g 2023-10-18T20:24:21.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VV6trtJKbYsV81bVb8DsgqE+D9E= 2023-10-18T20:24:21.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VV6trtJKbYsV81bVb8DsgqE+D9E=', 'Content-Length': '0'} 2023-10-18T20:24:21.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cde247104dec81c-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.686 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.686 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.686 DEBUG:boto:path=/bofrnf-55/?prefix=g 2023-10-18T20:24:21.687 DEBUG:boto:auth_path=/bofrnf-55/?prefix=g 2023-10-18T20:24:21.687 DEBUG:boto:Method: GET 2023-10-18T20:24:21.687 DEBUG:boto:Path: /bofrnf-55/?prefix=g 2023-10-18T20:24:21.688 DEBUG:boto:Data: 2023-10-18T20:24:21.688 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.689 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.690 DEBUG:boto:Params: {} 2023-10-18T20:24:21.690 DEBUG:boto:Token: None 2023-10-18T20:24:21.690 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.691 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7397fe1c3121466-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.695 DEBUG:boto:b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.696 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.696 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.696 DEBUG:boto:path=/bofrnf-55/?prefix=g 2023-10-18T20:24:21.697 DEBUG:boto:auth_path=/bofrnf-55/?prefix=g 2023-10-18T20:24:21.697 DEBUG:boto:Method: GET 2023-10-18T20:24:21.698 DEBUG:boto:Path: /bofrnf-55/?prefix=g 2023-10-18T20:24:21.698 DEBUG:boto:Data: 2023-10-18T20:24:21.699 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.699 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.700 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.700 DEBUG:boto:Params: {} 2023-10-18T20:24:21.701 DEBUG:boto:Token: None 2023-10-18T20:24:21.701 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.702 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087cc6aecdbd044b2-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.705 DEBUG:boto:b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:24:21.706 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.706 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.707 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.707 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.707 DEBUG:boto:Method: GET 2023-10-18T20:24:21.708 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.708 DEBUG:boto:Data: 2023-10-18T20:24:21.709 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.709 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.709 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.710 DEBUG:boto:Params: {} 2023-10-18T20:24:21.710 DEBUG:boto:Token: None 2023-10-18T20:24:21.711 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.711 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk= 2023-10-18T20:24:21.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk=', 'Content-Length': '0'} 2023-10-18T20:24:21.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b53736eba4cc7e-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.714 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.715 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.715 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.715 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.716 DEBUG:boto:Method: GET 2023-10-18T20:24:21.716 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.717 DEBUG:boto:Data: 2023-10-18T20:24:21.717 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.717 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.718 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.718 DEBUG:boto:Params: {} 2023-10-18T20:24:21.719 DEBUG:boto:Token: None 2023-10-18T20:24:21.719 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.720 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk= 2023-10-18T20:24:21.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk=', 'Content-Length': '0'} 2023-10-18T20:24:21.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095c9322cc0d7ed31-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.723 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.723 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.724 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.724 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.724 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.725 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.725 DEBUG:boto:Data: 2023-10-18T20:24:21.726 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.726 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.726 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.727 DEBUG:boto:Params: {} 2023-10-18T20:24:21.727 DEBUG:boto:Token: None 2023-10-18T20:24:21.728 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Oqm6z2Z7LsDrqKrcP2dB+CT3kcQ= 2023-10-18T20:24:21.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Oqm6z2Z7LsDrqKrcP2dB+CT3kcQ=', 'Content-Length': '0'} 2023-10-18T20:24:21.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000874fd60ef79fe3ac-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.731 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.731 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.732 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.732 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.732 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.733 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.733 DEBUG:boto:Data: 2023-10-18T20:24:21.734 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.734 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.734 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.735 DEBUG:boto:Params: {} 2023-10-18T20:24:21.735 DEBUG:boto:Token: None 2023-10-18T20:24:21.736 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.736 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Oqm6z2Z7LsDrqKrcP2dB+CT3kcQ= 2023-10-18T20:24:21.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Oqm6z2Z7LsDrqKrcP2dB+CT3kcQ=', 'Content-Length': '0'} 2023-10-18T20:24:21.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44c043c4427993d-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:24:21.739 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.740 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.740 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.740 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.741 DEBUG:boto:Method: GET 2023-10-18T20:24:21.741 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.742 DEBUG:boto:Data: 2023-10-18T20:24:21.742 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.743 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.743 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.743 DEBUG:boto:Params: {} 2023-10-18T20:24:21.744 DEBUG:boto:Token: None 2023-10-18T20:24:21.744 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.745 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk= 2023-10-18T20:24:21.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk=', 'Content-Length': '0'} 2023-10-18T20:24:21.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ca852170adb6d4-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.790 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.791 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.791 DEBUG:boto:path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.791 DEBUG:boto:auth_path=/bofrnf-55/h?versionId=null 2023-10-18T20:24:21.792 DEBUG:boto:Method: GET 2023-10-18T20:24:21.792 DEBUG:boto:Path: /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.793 DEBUG:boto:Data: 2023-10-18T20:24:21.793 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.794 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.794 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.795 DEBUG:boto:Params: {} 2023-10-18T20:24:21.795 DEBUG:boto:Token: None 2023-10-18T20:24:21.795 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h?versionId=null 2023-10-18T20:24:21.796 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk= 2023-10-18T20:24:21.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk=', 'Content-Length': '0'} 2023-10-18T20:24:21.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eb79b8975ab0576-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.799 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.799 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.800 DEBUG:boto:path=/bofrnf-55/?prefix=h 2023-10-18T20:24:21.800 DEBUG:boto:auth_path=/bofrnf-55/?prefix=h 2023-10-18T20:24:21.801 DEBUG:boto:Method: GET 2023-10-18T20:24:21.801 DEBUG:boto:Path: /bofrnf-55/?prefix=h 2023-10-18T20:24:21.802 DEBUG:boto:Data: 2023-10-18T20:24:21.802 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.802 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.803 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.803 DEBUG:boto:Params: {} 2023-10-18T20:24:21.804 DEBUG:boto:Token: None 2023-10-18T20:24:21.804 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.805 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c5f5c86ffeb395e-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.808 DEBUG:boto:b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.809 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.809 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.809 DEBUG:boto:path=/bofrnf-55/?prefix=h 2023-10-18T20:24:21.810 DEBUG:boto:auth_path=/bofrnf-55/?prefix=h 2023-10-18T20:24:21.810 DEBUG:boto:Method: GET 2023-10-18T20:24:21.811 DEBUG:boto:Path: /bofrnf-55/?prefix=h 2023-10-18T20:24:21.811 DEBUG:boto:Data: 2023-10-18T20:24:21.812 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.812 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.813 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.813 DEBUG:boto:Params: {} 2023-10-18T20:24:21.814 DEBUG:boto:Token: None 2023-10-18T20:24:21.814 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.815 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007438b16e1b305340-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.818 DEBUG:boto:b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.818 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.819 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.819 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.819 DEBUG:boto:Path: /bofrnf-55/h 2023-10-18T20:24:21.820 DEBUG:boto:Data: 2023-10-18T20:24:21.820 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.821 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.821 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.821 DEBUG:boto:Params: {} 2023-10-18T20:24:21.822 DEBUG:boto:Token: None 2023-10-18T20:24:21.822 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h 2023-10-18T20:24:21.823 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W7vS7xrIqK76pI1e6vFD7DMH4Qs= 2023-10-18T20:24:21.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W7vS7xrIqK76pI1e6vFD7DMH4Qs=', 'Content-Length': '0'} 2023-10-18T20:24:21.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b6502b3bb955231-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.825 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.826 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.826 DEBUG:boto:Method: HEAD 2023-10-18T20:24:21.827 DEBUG:boto:Path: /bofrnf-55/h 2023-10-18T20:24:21.827 DEBUG:boto:Data: 2023-10-18T20:24:21.827 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.828 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.828 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.829 DEBUG:boto:Params: {} 2023-10-18T20:24:21.829 DEBUG:boto:Token: None 2023-10-18T20:24:21.830 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h 2023-10-18T20:24:21.830 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W7vS7xrIqK76pI1e6vFD7DMH4Qs= 2023-10-18T20:24:21.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W7vS7xrIqK76pI1e6vFD7DMH4Qs=', 'Content-Length': '0'} 2023-10-18T20:24:21.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e7334c101821d0-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:24:21.833 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.833 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.834 DEBUG:boto:Method: GET 2023-10-18T20:24:21.834 DEBUG:boto:Path: /bofrnf-55/h 2023-10-18T20:24:21.835 DEBUG:boto:Data: 2023-10-18T20:24:21.835 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.835 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.836 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.836 DEBUG:boto:Params: {} 2023-10-18T20:24:21.836 DEBUG:boto:Token: None 2023-10-18T20:24:21.837 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h 2023-10-18T20:24:21.837 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:loEfPDZEZ5s5Gqcv6XRNFq45/hs= 2023-10-18T20:24:21.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:loEfPDZEZ5s5Gqcv6XRNFq45/hs=', 'Content-Length': '0'} 2023-10-18T20:24:21.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007de1612eb7e2edf3-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.840 DEBUG:boto:path=/bofrnf-55/h 2023-10-18T20:24:21.841 DEBUG:boto:auth_path=/bofrnf-55/h 2023-10-18T20:24:21.841 DEBUG:boto:Method: GET 2023-10-18T20:24:21.842 DEBUG:boto:Path: /bofrnf-55/h 2023-10-18T20:24:21.842 DEBUG:boto:Data: 2023-10-18T20:24:21.843 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.843 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.844 DEBUG:boto:Params: {} 2023-10-18T20:24:21.844 DEBUG:boto:Token: None 2023-10-18T20:24:21.845 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/h 2023-10-18T20:24:21.845 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:loEfPDZEZ5s5Gqcv6XRNFq45/hs= 2023-10-18T20:24:21.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:loEfPDZEZ5s5Gqcv6XRNFq45/hs=', 'Content-Length': '0'} 2023-10-18T20:24:21.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f98d16a1d56142f0-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.848 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.848 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.849 DEBUG:boto:path=/bofrnf-55/?prefix=h 2023-10-18T20:24:21.849 DEBUG:boto:auth_path=/bofrnf-55/?prefix=h 2023-10-18T20:24:21.849 DEBUG:boto:Method: GET 2023-10-18T20:24:21.850 DEBUG:boto:Path: /bofrnf-55/?prefix=h 2023-10-18T20:24:21.850 DEBUG:boto:Data: 2023-10-18T20:24:21.851 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.851 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:21.852 DEBUG:boto:Port: 8001 2023-10-18T20:24:21.852 DEBUG:boto:Params: {} 2023-10-18T20:24:21.853 DEBUG:boto:Token: None 2023-10-18T20:24:21.853 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.853 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b46e8db5ba7bdee0-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.857 DEBUG:boto:b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.857 DEBUG:boto:path=/bofrnf-55/ 2023-10-18T20:24:21.858 DEBUG:boto:auth_path=/bofrnf-55/ 2023-10-18T20:24:21.858 DEBUG:boto:path=/bofrnf-55/?prefix=h 2023-10-18T20:24:21.859 DEBUG:boto:auth_path=/bofrnf-55/?prefix=h 2023-10-18T20:24:21.859 DEBUG:boto:Method: GET 2023-10-18T20:24:21.859 DEBUG:boto:Path: /bofrnf-55/?prefix=h 2023-10-18T20:24:21.860 DEBUG:boto:Data: 2023-10-18T20:24:21.860 DEBUG:boto:Headers: {} 2023-10-18T20:24:21.861 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:21.861 DEBUG:boto:Port: 8000 2023-10-18T20:24:21.861 DEBUG:boto:Params: {} 2023-10-18T20:24:21.862 DEBUG:boto:Token: None 2023-10-18T20:24:21.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:21 GMT /bofrnf-55/ 2023-10-18T20:24:21.863 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T20:24:21.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T20:24:21.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014f843139a16b54b-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:21.866 DEBUG:boto:b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:21.866 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-55 zones={a2, a1} 2023-10-18T20:24:21.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:21.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:21.868 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:24:21.913 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:21.914 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:21.954 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:21.952+0000 7efce4c91700 -1 req 10418709809151165008 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-36:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.3,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-36:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.3,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:21.954 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:21.952+0000 7efdc5e53700 -1 req 17951071748088388098 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-24:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.24,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-24:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.24,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:21.955 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:21.952+0000 7efdbc640700 -1 req 7809433190435413888 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-5:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.5,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-5:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.5,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:21.958 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:21.955+0000 7efcdac7d700 -1 req 16622378275756042683 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-41:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.10,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-41:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.10,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.000 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:21.997+0000 7efdb3e2f700 -1 req 11600398951189980837 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-27:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.27,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-27:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.27,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.004 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.002+0000 7efdae624700 -1 req 15539175400740723401 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-19:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.19,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-19:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.19,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.010 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.008+0000 7efd96df5700 -1 req 9945040015104871972 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-3:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.3,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-3:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.3,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.041 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.039+0000 7efd8b5de700 -1 req 1080517829428759886 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-30:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.30,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-30:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.30,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.048 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.046+0000 7efda3e0f700 -1 req 9728147470867133463 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-40:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.9,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-40:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.9,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.048 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.046+0000 7efd775b6700 -1 req 1993393129796747981 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-25:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.25,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-25:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.25,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.053 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.051+0000 7efd90de9700 -1 req 14791781324847460765 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-22:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.22,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-22:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.22,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:22.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:22.111 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:24:22.158 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:22.158 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:22.200 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:24:22.200 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.200 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding": "None", 2023-10-18T20:24:22.200 INFO:teuthology.orchestra.run.smithi082.stdout: "current_index": { 2023-10-18T20:24:22.201 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 2, 2023-10-18T20:24:22.201 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.201 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "Normal", 2023-10-18T20:24:22.201 INFO:teuthology.orchestra.run.smithi082.stdout: "normal": { 2023-10-18T20:24:22.201 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:24:22.202 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:22.202 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.202 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.203 INFO:teuthology.orchestra.run.smithi082.stdout: "logs": [ 2023-10-18T20:24:22.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.203 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 1, 2023-10-18T20:24:22.203 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.203 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:24:22.204 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:24:22.204 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 1, 2023-10-18T20:24:22.204 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.204 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:24:22.204 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:22.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.206 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 2, 2023-10-18T20:24:22.206 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.206 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:24:22.206 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:24:22.206 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 2, 2023-10-18T20:24:22.207 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.207 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:24:22.207 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:22.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.208 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:24:22.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.208 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:24:22.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:22.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:22.210 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:24:22.255 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:22.255 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:22.298 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.296+0000 7efd7c5c0700 -1 req 15075530772322273016 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-39:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.8,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-39:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.8,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.300 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.298+0000 7efd5e584700 -1 req 13649186578147700193 0.000999982s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-37:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.6,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-37:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.6,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.340 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.338+0000 7efd6a59c700 -1 req 8082273709273409527 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-2:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.2,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-2:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.2,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.342 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.340+0000 7efd4bd5f700 -1 req 15487544499056391101 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-29:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.29,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-29:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.29,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.345 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.342+0000 7efd7fdc7700 -1 req 17200455680627767505 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-23:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.23,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-23:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.23,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.385 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.382+0000 7efd3953a700 -1 req 2889084971211465227 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-28:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.28,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-28:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.28,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.388 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:24:22.386+0000 7efd3dd43700 -1 req 1055971798316952861 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-20:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.20,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-20:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.20,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T20:24:22.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:22.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:22.447 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:24:22.494 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:22.494 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:22.534 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:24:22.534 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.535 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding": "None", 2023-10-18T20:24:22.535 INFO:teuthology.orchestra.run.smithi082.stdout: "current_index": { 2023-10-18T20:24:22.535 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 2, 2023-10-18T20:24:22.535 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.535 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "Normal", 2023-10-18T20:24:22.536 INFO:teuthology.orchestra.run.smithi082.stdout: "normal": { 2023-10-18T20:24:22.536 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:24:22.536 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:22.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.537 INFO:teuthology.orchestra.run.smithi082.stdout: "logs": [ 2023-10-18T20:24:22.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.537 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 1, 2023-10-18T20:24:22.537 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.538 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:24:22.538 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:24:22.538 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 1, 2023-10-18T20:24:22.538 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.539 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:24:22.539 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:22.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.540 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 2, 2023-10-18T20:24:22.540 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.540 INFO:teuthology.orchestra.run.smithi082.stdout: "type": "InIndex", 2023-10-18T20:24:22.541 INFO:teuthology.orchestra.run.smithi082.stdout: "in_index": { 2023-10-18T20:24:22.541 INFO:teuthology.orchestra.run.smithi082.stdout: "gen": 2, 2023-10-18T20:24:22.541 INFO:teuthology.orchestra.run.smithi082.stdout: "layout": { 2023-10-18T20:24:22.541 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 3, 2023-10-18T20:24:22.541 INFO:teuthology.orchestra.run.smithi082.stdout: "hash_type": "Mod" 2023-10-18T20:24:22.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.543 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:24:22.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.543 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:24:22.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2023-10-18T20:24:22.548 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:22.549 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:22.549 DEBUG:boto:Method: PUT 2023-10-18T20:24:22.549 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:22.550 DEBUG:boto:Data: 2023-10-18T20:24:22.550 DEBUG:boto:Headers: {} 2023-10-18T20:24:22.551 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:22.551 DEBUG:boto:Port: 8000 2023-10-18T20:24:22.551 DEBUG:boto:Params: {} 2023-10-18T20:24:22.552 DEBUG:boto:Token: None 2023-10-18T20:24:22.553 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:24:22 GMT /bofrnf-56/ 2023-10-18T20:24:22.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Irk9NMcgn5637i4njjT4fH0gRc= 2023-10-18T20:24:22.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Irk9NMcgn5637i4njjT4fH0gRc=', 'Content-Length': '0'} 2023-10-18T20:24:22.567 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099ac27a1040d40b5-0065303ef6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:22.568 DEBUG:rgw_multi.tests:created bucket=bofrnf-56 2023-10-18T20:24:22.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:24:22.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:24:22.569 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:24:22.620 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:22.621 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:22.653 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:24:22.666 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:24:22.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.667 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660662.565748_962.1", 2023-10-18T20:24:22.667 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:24:22.565748Z" 2023-10-18T20:24:22.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.667 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:24:22.668 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:24:22.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.668 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:24:22.668 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:24:22.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.669 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:24:22.669 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:24:22.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.670 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:24:22.670 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660662.565748_962.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:24:22.671 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:24:22.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:24:22.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:24:22.672 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:24:22.721 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:22.721 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:22.749 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:22.747+0000 7fb7dcd3e780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:24:22.750 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:22.748+0000 7fb7dcd3e780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:24:22.762 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:24:22.763 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:24:22.763 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:24:22.763 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:24:22.763 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:24:22.763 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:24:22.764 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:24:22.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.764 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:24:22.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.765 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:24:22.765 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:24:22.765 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:24:22.765 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660662.565748_962.1", 2023-10-18T20:24:22.765 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:24:22.766 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:24:22.766 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:24:22.766 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:24:22.565748Z", 2023-10-18T20:24:22.766 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:24:22.766 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.767 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:24:22.767 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:24:22.767 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:24:22.768 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:24:22.768 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:24:22.768 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:24:22.768 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:24:22.768 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:24:22.769 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:24:22.769 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.769 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:24:22.770 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:24:22.770 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:24:22.770 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:24:22.770 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:24:22.770 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:24:22.771 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:24:22.771 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:24:22.771 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:24:22.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:24:22.772 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:24:22.772 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:24:22.772 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:24:22.772 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:24:22.773 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:24:22.773 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:24:22.773 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:24:22.773 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:24:22.774 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:24:22.774 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.774 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:24:22.774 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:24:22.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.775 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:24:22.775 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:24:22.775 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:24:22.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:24:22.775 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:24:22Z" 2023-10-18T20:24:22.776 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:24:22.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660662.565748_962.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:24:22.565748Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:24:22Z" } 2023-10-18T20:24:22.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660662.565748_962.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:24:22.565748Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:24:22.778 DEBUG:rgw_multi.tests:log_status={0: '1_1697660662.565748_962.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:24:22.778 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660662.565748_962.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:24:22.778 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:24:22.779 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:22.779 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:22.780 DEBUG:boto:Method: HEAD 2023-10-18T20:24:22.780 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:22.780 DEBUG:boto:Data: 2023-10-18T20:24:22.781 DEBUG:boto:Headers: {} 2023-10-18T20:24:22.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:22.782 DEBUG:boto:Port: 8000 2023-10-18T20:24:22.782 DEBUG:boto:Params: {} 2023-10-18T20:24:22.782 DEBUG:boto:Token: None 2023-10-18T20:24:22.783 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:22 GMT /bofrnf-56/ 2023-10-18T20:24:22.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ir6HztHuIe4nK3MbW3RKd6xHmP0= 2023-10-18T20:24:22.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ir6HztHuIe4nK3MbW3RKd6xHmP0=', 'Content-Length': '0'} 2023-10-18T20:24:22.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002ad4d9bf4cf45d7-0065303ef6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:22.789 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:22.789 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:22.790 DEBUG:boto:Method: PUT 2023-10-18T20:24:22.790 DEBUG:boto:Path: /bofrnf-56/a 2023-10-18T20:24:22.791 DEBUG:boto:Data: 2023-10-18T20:24:22.791 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:24:22.791 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:22.792 DEBUG:boto:Port: 8000 2023-10-18T20:24:22.792 DEBUG:boto:Params: {} 2023-10-18T20:24:22.793 DEBUG:boto:Token: None 2023-10-18T20:24:22.793 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:24:22 GMT /bofrnf-56/a 2023-10-18T20:24:22.793 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1bnB7S+QwiWwhzbVtzkVj6UD88o= 2023-10-18T20:24:22.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:24:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1bnB7S+QwiWwhzbVtzkVj6UD88o='} 2023-10-18T20:24:22.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d1e85fc89635001-0065303ef6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:22.800 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:22.800 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:22.801 DEBUG:boto:Method: HEAD 2023-10-18T20:24:22.801 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:22.802 DEBUG:boto:Data: 2023-10-18T20:24:22.802 DEBUG:boto:Headers: {} 2023-10-18T20:24:22.802 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:22.803 DEBUG:boto:Port: 8000 2023-10-18T20:24:22.803 DEBUG:boto:Params: {} 2023-10-18T20:24:22.804 DEBUG:boto:Token: None 2023-10-18T20:24:22.804 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:22 GMT /bofrnf-56/ 2023-10-18T20:24:22.805 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ir6HztHuIe4nK3MbW3RKd6xHmP0= 2023-10-18T20:24:22.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ir6HztHuIe4nK3MbW3RKd6xHmP0=', 'Content-Length': '0'} 2023-10-18T20:24:22.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002dcd8153e696108f-0065303ef6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:22.808 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:22.808 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:22.808 DEBUG:boto:Method: PUT 2023-10-18T20:24:22.809 DEBUG:boto:Path: /bofrnf-56/b 2023-10-18T20:24:22.809 DEBUG:boto:Data: 2023-10-18T20:24:22.810 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:24:22.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:22.810 DEBUG:boto:Port: 8000 2023-10-18T20:24:22.811 DEBUG:boto:Params: {} 2023-10-18T20:24:22.811 DEBUG:boto:Token: None 2023-10-18T20:24:22.812 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:24:22 GMT /bofrnf-56/b 2023-10-18T20:24:22.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ztzkWoZRB5oEprg6atAsXu7o1S8= 2023-10-18T20:24:22.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:24:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ztzkWoZRB5oEprg6atAsXu7o1S8='} 2023-10-18T20:24:22.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edc36e0e3bd2c010-0065303ef6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:22.855 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:22.855 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:22.856 DEBUG:boto:Method: HEAD 2023-10-18T20:24:22.856 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:22.856 DEBUG:boto:Data: 2023-10-18T20:24:22.857 DEBUG:boto:Headers: {} 2023-10-18T20:24:22.857 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:22.858 DEBUG:boto:Port: 8000 2023-10-18T20:24:22.858 DEBUG:boto:Params: {} 2023-10-18T20:24:22.858 DEBUG:boto:Token: None 2023-10-18T20:24:22.859 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:22 GMT /bofrnf-56/ 2023-10-18T20:24:22.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ir6HztHuIe4nK3MbW3RKd6xHmP0= 2023-10-18T20:24:22.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ir6HztHuIe4nK3MbW3RKd6xHmP0=', 'Content-Length': '0'} 2023-10-18T20:24:22.862 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062d03d2a9f801b62-0065303ef6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:22.862 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:22.863 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:22.863 DEBUG:boto:Method: PUT 2023-10-18T20:24:22.864 DEBUG:boto:Path: /bofrnf-56/c 2023-10-18T20:24:22.864 DEBUG:boto:Data: 2023-10-18T20:24:22.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:24:22.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:22.865 DEBUG:boto:Port: 8000 2023-10-18T20:24:22.866 DEBUG:boto:Params: {} 2023-10-18T20:24:22.866 DEBUG:boto:Token: None 2023-10-18T20:24:22.866 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:24:22 GMT /bofrnf-56/c 2023-10-18T20:24:22.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DLwwzlQdHGg8O3iFi2r6aAxQZ2c= 2023-10-18T20:24:22.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:24:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DLwwzlQdHGg8O3iFi2r6aAxQZ2c='} 2023-10-18T20:24:22.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c5b8c01eb1cd37f-0065303ef6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:22.872 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:22.873 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:22.873 DEBUG:boto:Method: HEAD 2023-10-18T20:24:22.874 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:22.874 DEBUG:boto:Data: 2023-10-18T20:24:22.874 DEBUG:boto:Headers: {} 2023-10-18T20:24:22.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:22.875 DEBUG:boto:Port: 8000 2023-10-18T20:24:22.875 DEBUG:boto:Params: {} 2023-10-18T20:24:22.876 DEBUG:boto:Token: None 2023-10-18T20:24:22.876 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:22 GMT /bofrnf-56/ 2023-10-18T20:24:22.877 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ir6HztHuIe4nK3MbW3RKd6xHmP0= 2023-10-18T20:24:22.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ir6HztHuIe4nK3MbW3RKd6xHmP0=', 'Content-Length': '0'} 2023-10-18T20:24:22.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000117bb6835c286018-0065303ef6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:22.880 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:22.880 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:22.881 DEBUG:boto:Method: PUT 2023-10-18T20:24:22.881 DEBUG:boto:Path: /bofrnf-56/d 2023-10-18T20:24:22.882 DEBUG:boto:Data: 2023-10-18T20:24:22.882 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:24:22.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:22.883 DEBUG:boto:Port: 8000 2023-10-18T20:24:22.883 DEBUG:boto:Params: {} 2023-10-18T20:24:22.884 DEBUG:boto:Token: None 2023-10-18T20:24:22.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:24:22 GMT /bofrnf-56/d 2023-10-18T20:24:22.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AX4EjUDN52mH5dH8R9THzy7Yu8c= 2023-10-18T20:24:22.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:24:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AX4EjUDN52mH5dH8R9THzy7Yu8c='} 2023-10-18T20:24:22.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b47fc6938fa5739-0065303ef6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:22.931 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-56 2023-10-18T20:24:22.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:22.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:22.932 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:24:22.989 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:22.989 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:23.022 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:23.020+0000 7f3dcbdbf780 1 waiting to reach incremental sync.. 2023-10-18T20:24:28.023 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:28.020+0000 7f3dcbdbf780 1 waiting to reach incremental sync.. 2023-10-18T20:24:33.023 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:33.021+0000 7f3dcbdbf780 1 waiting to reach incremental sync.. 2023-10-18T20:24:38.024 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:38.022+0000 7f3dcbdbf780 1 waiting to reach incremental sync.. 2023-10-18T20:24:43.025 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:43.024+0000 7f3dcbdbf780 1 bucket sync caught up with source: 2023-10-18T20:24:43.026 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.55.6, , , , 00000000001.57.6, , , 00000000001.30.6, , 00000000001.55.6] 2023-10-18T20:24:43.026 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.55.6, , , , 00000000001.57.6, , , 00000000001.30.6, , 00000000001.55.6] 2023-10-18T20:24:43.026 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:43.024+0000 7f3dcbdbf780 0 bucket checkpoint complete 2023-10-18T20:24:43.042 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-56 2023-10-18T20:24:43.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:43.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:43.044 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:24:43.106 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:43.106 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:43.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:43.141+0000 7f8220a86780 1 waiting to reach incremental sync.. 2023-10-18T20:24:48.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:48.142+0000 7f8220a86780 1 waiting to reach incremental sync.. 2023-10-18T20:24:53.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:53.142+0000 7f8220a86780 1 waiting to reach incremental sync.. 2023-10-18T20:24:58.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:58.144+0000 7f8220a86780 1 bucket sync caught up with source: 2023-10-18T20:24:58.147 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.710.6, , , , 00000000001.832.6, , , 00000000001.1186.6, , 00000000001.831.6] 2023-10-18T20:24:58.147 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.710.6, , , , 00000000001.832.6, , , 00000000001.1186.6, , 00000000001.831.6] 2023-10-18T20:24:58.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:58.144+0000 7f8220a86780 0 bucket checkpoint complete 2023-10-18T20:24:58.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-56 zones={a2, a1} 2023-10-18T20:24:58.163 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.163 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.164 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.164 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:58.165 DEBUG:boto:Data: 2023-10-18T20:24:58.165 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.165 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.166 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.166 DEBUG:boto:Params: {} 2023-10-18T20:24:58.167 DEBUG:boto:Token: None 2023-10-18T20:24:58.168 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+jKIdTTAGF60Z5vuLBheJf9YVYQ= 2023-10-18T20:24:58.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+jKIdTTAGF60Z5vuLBheJf9YVYQ=', 'Content-Length': '0'} 2023-10-18T20:24:58.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4d1a2b16adc5069-0065303f1a-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.172 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.172 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.173 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.173 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:58.173 DEBUG:boto:Data: 2023-10-18T20:24:58.174 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.174 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.175 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.175 DEBUG:boto:Params: {} 2023-10-18T20:24:58.175 DEBUG:boto:Token: None 2023-10-18T20:24:58.176 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.176 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+jKIdTTAGF60Z5vuLBheJf9YVYQ= 2023-10-18T20:24:58.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+jKIdTTAGF60Z5vuLBheJf9YVYQ=', 'Content-Length': '0'} 2023-10-18T20:24:58.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006311b564b2f28a69-0065303f1a-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.179 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:24:58.180 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.180 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.181 DEBUG:boto:path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.181 DEBUG:boto:auth_path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.181 DEBUG:boto:Method: GET 2023-10-18T20:24:58.182 DEBUG:boto:Path: /bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.182 DEBUG:boto:Data: 2023-10-18T20:24:58.183 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.183 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.183 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.184 DEBUG:boto:Params: {} 2023-10-18T20:24:58.184 DEBUG:boto:Token: None 2023-10-18T20:24:58.184 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/?versions 2023-10-18T20:24:58.185 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1HovJrqmiK3cLM9BGX5WpTgtwvY= 2023-10-18T20:24:58.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1HovJrqmiK3cLM9BGX5WpTgtwvY=', 'Content-Length': '0'} 2023-10-18T20:24:58.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ca1d7d8183e9a4a-0065303f1a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.188 DEBUG:boto:b'bofrnf-56999falseanulltrue2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.190 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:24:58.190 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:24:58.190 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:24:58.191 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:24:58.191 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:24:58.192 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.192 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.192 DEBUG:boto:path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.193 DEBUG:boto:auth_path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.193 DEBUG:boto:Method: GET 2023-10-18T20:24:58.194 DEBUG:boto:Path: /bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.194 DEBUG:boto:Data: 2023-10-18T20:24:58.194 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.195 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.195 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.195 DEBUG:boto:Params: {} 2023-10-18T20:24:58.196 DEBUG:boto:Token: None 2023-10-18T20:24:58.196 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/?versions 2023-10-18T20:24:58.197 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1HovJrqmiK3cLM9BGX5WpTgtwvY= 2023-10-18T20:24:58.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1HovJrqmiK3cLM9BGX5WpTgtwvY=', 'Content-Length': '0'} 2023-10-18T20:24:58.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cd309a132ef80ef-0065303f1a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.200 DEBUG:boto:b'bofrnf-56999falseanulltrue2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.201 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:24:58.201 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:24:58.202 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:24:58.202 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:24:58.203 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.203 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.203 DEBUG:boto:path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.204 DEBUG:boto:auth_path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.204 DEBUG:boto:Method: GET 2023-10-18T20:24:58.205 DEBUG:boto:Path: /bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.205 DEBUG:boto:Data: 2023-10-18T20:24:58.205 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.206 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.206 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.207 DEBUG:boto:Params: {} 2023-10-18T20:24:58.207 DEBUG:boto:Token: None 2023-10-18T20:24:58.207 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/?versions 2023-10-18T20:24:58.208 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1HovJrqmiK3cLM9BGX5WpTgtwvY= 2023-10-18T20:24:58.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1HovJrqmiK3cLM9BGX5WpTgtwvY=', 'Content-Length': '0'} 2023-10-18T20:24:58.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e630aaeb71966e0-0065303f1a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.211 DEBUG:boto:b'bofrnf-56999falseanulltrue2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.212 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.212 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.213 DEBUG:boto:path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.213 DEBUG:boto:auth_path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.213 DEBUG:boto:Method: GET 2023-10-18T20:24:58.214 DEBUG:boto:Path: /bofrnf-56/?versions&max-keys=999 2023-10-18T20:24:58.214 DEBUG:boto:Data: 2023-10-18T20:24:58.214 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.215 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.215 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.216 DEBUG:boto:Params: {} 2023-10-18T20:24:58.216 DEBUG:boto:Token: None 2023-10-18T20:24:58.216 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/?versions 2023-10-18T20:24:58.217 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1HovJrqmiK3cLM9BGX5WpTgtwvY= 2023-10-18T20:24:58.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1HovJrqmiK3cLM9BGX5WpTgtwvY=', 'Content-Length': '0'} 2023-10-18T20:24:58.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caa197e6c3743c76-0065303f1a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.220 DEBUG:boto:b'bofrnf-56999falseanulltrue2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:24:58.221 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.222 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.222 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.223 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.223 DEBUG:boto:Method: GET 2023-10-18T20:24:58.223 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.224 DEBUG:boto:Data: 2023-10-18T20:24:58.224 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.224 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.225 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.225 DEBUG:boto:Params: {} 2023-10-18T20:24:58.226 DEBUG:boto:Token: None 2023-10-18T20:24:58.226 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.227 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bHqySVYYXSaxtmgNMtxNb2NLGY8= 2023-10-18T20:24:58.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bHqySVYYXSaxtmgNMtxNb2NLGY8=', 'Content-Length': '0'} 2023-10-18T20:24:58.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4f1db9eb17941c-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.270 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.271 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.271 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.272 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.272 DEBUG:boto:Method: GET 2023-10-18T20:24:58.272 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.273 DEBUG:boto:Data: 2023-10-18T20:24:58.273 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.274 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.274 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.275 DEBUG:boto:Params: {} 2023-10-18T20:24:58.275 DEBUG:boto:Token: None 2023-10-18T20:24:58.275 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bHqySVYYXSaxtmgNMtxNb2NLGY8= 2023-10-18T20:24:58.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bHqySVYYXSaxtmgNMtxNb2NLGY8=', 'Content-Length': '0'} 2023-10-18T20:24:58.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c679b821a86c7f3-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.279 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.279 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.279 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.280 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.280 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.281 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.281 DEBUG:boto:Data: 2023-10-18T20:24:58.281 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.282 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.282 DEBUG:boto:Params: {} 2023-10-18T20:24:58.283 DEBUG:boto:Token: None 2023-10-18T20:24:58.283 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0eGTsDS3V9w+P77e5jJzP9KtasQ= 2023-10-18T20:24:58.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0eGTsDS3V9w+P77e5jJzP9KtasQ=', 'Content-Length': '0'} 2023-10-18T20:24:58.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003b3b3e106fe677c-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.287 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.287 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.288 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.288 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.288 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.289 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.289 DEBUG:boto:Data: 2023-10-18T20:24:58.290 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.290 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.290 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.291 DEBUG:boto:Params: {} 2023-10-18T20:24:58.291 DEBUG:boto:Token: None 2023-10-18T20:24:58.292 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.292 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0eGTsDS3V9w+P77e5jJzP9KtasQ= 2023-10-18T20:24:58.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0eGTsDS3V9w+P77e5jJzP9KtasQ=', 'Content-Length': '0'} 2023-10-18T20:24:58.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e5e08eac5e38f6-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:24:58.295 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.296 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.296 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.296 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.297 DEBUG:boto:Method: GET 2023-10-18T20:24:58.297 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.298 DEBUG:boto:Data: 2023-10-18T20:24:58.298 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.298 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.299 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.299 DEBUG:boto:Params: {} 2023-10-18T20:24:58.300 DEBUG:boto:Token: None 2023-10-18T20:24:58.300 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.300 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bHqySVYYXSaxtmgNMtxNb2NLGY8= 2023-10-18T20:24:58.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bHqySVYYXSaxtmgNMtxNb2NLGY8=', 'Content-Length': '0'} 2023-10-18T20:24:58.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1bcdacae48f911f-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.303 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.304 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.304 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.305 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:24:58.305 DEBUG:boto:Method: GET 2023-10-18T20:24:58.305 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.306 DEBUG:boto:Data: 2023-10-18T20:24:58.306 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.307 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.307 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.307 DEBUG:boto:Params: {} 2023-10-18T20:24:58.308 DEBUG:boto:Token: None 2023-10-18T20:24:58.308 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:24:58.309 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bHqySVYYXSaxtmgNMtxNb2NLGY8= 2023-10-18T20:24:58.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bHqySVYYXSaxtmgNMtxNb2NLGY8=', 'Content-Length': '0'} 2023-10-18T20:24:58.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000910f929be688956e-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.354 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.355 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.355 DEBUG:boto:path=/bofrnf-56/?prefix=a 2023-10-18T20:24:58.356 DEBUG:boto:auth_path=/bofrnf-56/?prefix=a 2023-10-18T20:24:58.356 DEBUG:boto:Method: GET 2023-10-18T20:24:58.356 DEBUG:boto:Path: /bofrnf-56/?prefix=a 2023-10-18T20:24:58.357 DEBUG:boto:Data: 2023-10-18T20:24:58.357 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.358 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.359 DEBUG:boto:Params: {} 2023-10-18T20:24:58.359 DEBUG:boto:Token: None 2023-10-18T20:24:58.359 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000956985eeb00c68e3-0065303f1a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.364 DEBUG:boto:b'bofrnf-56a1000falsea2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.364 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.365 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.365 DEBUG:boto:path=/bofrnf-56/?prefix=a 2023-10-18T20:24:58.365 DEBUG:boto:auth_path=/bofrnf-56/?prefix=a 2023-10-18T20:24:58.366 DEBUG:boto:Method: GET 2023-10-18T20:24:58.366 DEBUG:boto:Path: /bofrnf-56/?prefix=a 2023-10-18T20:24:58.367 DEBUG:boto:Data: 2023-10-18T20:24:58.367 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.368 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.368 DEBUG:boto:Params: {} 2023-10-18T20:24:58.369 DEBUG:boto:Token: None 2023-10-18T20:24:58.369 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f951e214f883901-0065303f1a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.373 DEBUG:boto:b'bofrnf-56a1000falsea2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.373 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.374 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.374 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.374 DEBUG:boto:Path: /bofrnf-56/a 2023-10-18T20:24:58.375 DEBUG:boto:Data: 2023-10-18T20:24:58.375 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.376 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.376 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.376 DEBUG:boto:Params: {} 2023-10-18T20:24:58.377 DEBUG:boto:Token: None 2023-10-18T20:24:58.377 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a 2023-10-18T20:24:58.378 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6KuEkaVYOCzyqmd0smzg3FPZnts= 2023-10-18T20:24:58.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6KuEkaVYOCzyqmd0smzg3FPZnts=', 'Content-Length': '0'} 2023-10-18T20:24:58.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070d9302a426f67c8-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.381 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.381 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.381 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.382 DEBUG:boto:Path: /bofrnf-56/a 2023-10-18T20:24:58.382 DEBUG:boto:Data: 2023-10-18T20:24:58.383 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.383 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.383 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.384 DEBUG:boto:Params: {} 2023-10-18T20:24:58.384 DEBUG:boto:Token: None 2023-10-18T20:24:58.385 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a 2023-10-18T20:24:58.385 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6KuEkaVYOCzyqmd0smzg3FPZnts= 2023-10-18T20:24:58.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6KuEkaVYOCzyqmd0smzg3FPZnts=', 'Content-Length': '0'} 2023-10-18T20:24:58.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2b52bcb3dcb445b-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:24:58.389 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.389 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.390 DEBUG:boto:Method: GET 2023-10-18T20:24:58.390 DEBUG:boto:Path: /bofrnf-56/a 2023-10-18T20:24:58.390 DEBUG:boto:Data: 2023-10-18T20:24:58.391 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.391 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.392 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.392 DEBUG:boto:Params: {} 2023-10-18T20:24:58.393 DEBUG:boto:Token: None 2023-10-18T20:24:58.393 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a 2023-10-18T20:24:58.394 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QJzTkQBqBXmTviqgikQelYA4kZ0= 2023-10-18T20:24:58.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QJzTkQBqBXmTviqgikQelYA4kZ0=', 'Content-Length': '0'} 2023-10-18T20:24:58.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b007f18372e4dc43-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.397 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:24:58.398 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:24:58.398 DEBUG:boto:Method: GET 2023-10-18T20:24:58.399 DEBUG:boto:Path: /bofrnf-56/a 2023-10-18T20:24:58.399 DEBUG:boto:Data: 2023-10-18T20:24:58.399 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.400 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.400 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.401 DEBUG:boto:Params: {} 2023-10-18T20:24:58.401 DEBUG:boto:Token: None 2023-10-18T20:24:58.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/a 2023-10-18T20:24:58.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QJzTkQBqBXmTviqgikQelYA4kZ0= 2023-10-18T20:24:58.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QJzTkQBqBXmTviqgikQelYA4kZ0=', 'Content-Length': '0'} 2023-10-18T20:24:58.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057487e4bd396c618-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.405 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.406 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.406 DEBUG:boto:path=/bofrnf-56/?prefix=a 2023-10-18T20:24:58.406 DEBUG:boto:auth_path=/bofrnf-56/?prefix=a 2023-10-18T20:24:58.407 DEBUG:boto:Method: GET 2023-10-18T20:24:58.407 DEBUG:boto:Path: /bofrnf-56/?prefix=a 2023-10-18T20:24:58.408 DEBUG:boto:Data: 2023-10-18T20:24:58.408 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.409 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.409 DEBUG:boto:Params: {} 2023-10-18T20:24:58.410 DEBUG:boto:Token: None 2023-10-18T20:24:58.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030960afd52ff9ae9-0065303f1a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.414 DEBUG:boto:b'bofrnf-56a1000falsea2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.415 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.415 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.416 DEBUG:boto:path=/bofrnf-56/?prefix=a 2023-10-18T20:24:58.416 DEBUG:boto:auth_path=/bofrnf-56/?prefix=a 2023-10-18T20:24:58.416 DEBUG:boto:Method: GET 2023-10-18T20:24:58.417 DEBUG:boto:Path: /bofrnf-56/?prefix=a 2023-10-18T20:24:58.417 DEBUG:boto:Data: 2023-10-18T20:24:58.418 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.419 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.419 DEBUG:boto:Params: {} 2023-10-18T20:24:58.419 DEBUG:boto:Token: None 2023-10-18T20:24:58.420 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079a669f752560d11-0065303f1a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.424 DEBUG:boto:b'bofrnf-56a1000falsea2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:24:58.425 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.426 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.426 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.427 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.427 DEBUG:boto:Method: GET 2023-10-18T20:24:58.427 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.428 DEBUG:boto:Data: 2023-10-18T20:24:58.428 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.429 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.429 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.429 DEBUG:boto:Params: {} 2023-10-18T20:24:58.430 DEBUG:boto:Token: None 2023-10-18T20:24:58.430 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.431 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:l95gKMlOBqs1d9C17L2WLI14opA= 2023-10-18T20:24:58.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:l95gKMlOBqs1d9C17L2WLI14opA=', 'Content-Length': '0'} 2023-10-18T20:24:58.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076efc69b13f27c0b-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.434 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.434 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.435 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.435 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.436 DEBUG:boto:Method: GET 2023-10-18T20:24:58.436 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.436 DEBUG:boto:Data: 2023-10-18T20:24:58.437 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.437 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.438 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.438 DEBUG:boto:Params: {} 2023-10-18T20:24:58.439 DEBUG:boto:Token: None 2023-10-18T20:24:58.439 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:l95gKMlOBqs1d9C17L2WLI14opA= 2023-10-18T20:24:58.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:l95gKMlOBqs1d9C17L2WLI14opA=', 'Content-Length': '0'} 2023-10-18T20:24:58.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b98bec5a3de6cd-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.443 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.443 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.444 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.444 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.444 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.445 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.445 DEBUG:boto:Data: 2023-10-18T20:24:58.446 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.446 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.447 DEBUG:boto:Params: {} 2023-10-18T20:24:58.447 DEBUG:boto:Token: None 2023-10-18T20:24:58.448 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hRAIOoj0JTx6+tMWXja6qsPcPPI= 2023-10-18T20:24:58.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hRAIOoj0JTx6+tMWXja6qsPcPPI=', 'Content-Length': '0'} 2023-10-18T20:24:58.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085c0a7ccbf907b59-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.451 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.451 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.452 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.452 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.453 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.453 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.453 DEBUG:boto:Data: 2023-10-18T20:24:58.454 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.455 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.455 DEBUG:boto:Params: {} 2023-10-18T20:24:58.455 DEBUG:boto:Token: None 2023-10-18T20:24:58.456 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hRAIOoj0JTx6+tMWXja6qsPcPPI= 2023-10-18T20:24:58.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hRAIOoj0JTx6+tMWXja6qsPcPPI=', 'Content-Length': '0'} 2023-10-18T20:24:58.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014849eb74b8e1e3d-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:24:58.465 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.465 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.466 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.466 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.467 DEBUG:boto:Method: GET 2023-10-18T20:24:58.467 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.467 DEBUG:boto:Data: 2023-10-18T20:24:58.468 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.468 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.469 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.469 DEBUG:boto:Params: {} 2023-10-18T20:24:58.470 DEBUG:boto:Token: None 2023-10-18T20:24:58.470 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.471 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:l95gKMlOBqs1d9C17L2WLI14opA= 2023-10-18T20:24:58.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:l95gKMlOBqs1d9C17L2WLI14opA=', 'Content-Length': '0'} 2023-10-18T20:24:58.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de247497de36f049-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.514 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.515 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.515 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.516 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:24:58.516 DEBUG:boto:Method: GET 2023-10-18T20:24:58.516 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.517 DEBUG:boto:Data: 2023-10-18T20:24:58.517 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.518 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.518 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.519 DEBUG:boto:Params: {} 2023-10-18T20:24:58.519 DEBUG:boto:Token: None 2023-10-18T20:24:58.520 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:24:58.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:l95gKMlOBqs1d9C17L2WLI14opA= 2023-10-18T20:24:58.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:l95gKMlOBqs1d9C17L2WLI14opA=', 'Content-Length': '0'} 2023-10-18T20:24:58.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bb0c3e7d559ad4c-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.524 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.524 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.524 DEBUG:boto:path=/bofrnf-56/?prefix=b 2023-10-18T20:24:58.525 DEBUG:boto:auth_path=/bofrnf-56/?prefix=b 2023-10-18T20:24:58.525 DEBUG:boto:Method: GET 2023-10-18T20:24:58.526 DEBUG:boto:Path: /bofrnf-56/?prefix=b 2023-10-18T20:24:58.526 DEBUG:boto:Data: 2023-10-18T20:24:58.526 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.527 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.527 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.528 DEBUG:boto:Params: {} 2023-10-18T20:24:58.528 DEBUG:boto:Token: None 2023-10-18T20:24:58.529 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.529 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000883e2b2e32135c96-0065303f1a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.533 DEBUG:boto:b'bofrnf-56b1000falseb2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.533 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.534 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.534 DEBUG:boto:path=/bofrnf-56/?prefix=b 2023-10-18T20:24:58.535 DEBUG:boto:auth_path=/bofrnf-56/?prefix=b 2023-10-18T20:24:58.535 DEBUG:boto:Method: GET 2023-10-18T20:24:58.536 DEBUG:boto:Path: /bofrnf-56/?prefix=b 2023-10-18T20:24:58.536 DEBUG:boto:Data: 2023-10-18T20:24:58.536 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.537 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.538 DEBUG:boto:Params: {} 2023-10-18T20:24:58.538 DEBUG:boto:Token: None 2023-10-18T20:24:58.539 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.539 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfb592fb14b6a528-0065303f1a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.543 DEBUG:boto:b'bofrnf-56b1000falseb2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.543 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.544 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.544 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.545 DEBUG:boto:Path: /bofrnf-56/b 2023-10-18T20:24:58.545 DEBUG:boto:Data: 2023-10-18T20:24:58.545 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.546 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.546 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.547 DEBUG:boto:Params: {} 2023-10-18T20:24:58.547 DEBUG:boto:Token: None 2023-10-18T20:24:58.547 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b 2023-10-18T20:24:58.548 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:79X4SRTEHtHwtR2VxrKZXdt57k4= 2023-10-18T20:24:58.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:79X4SRTEHtHwtR2VxrKZXdt57k4=', 'Content-Length': '0'} 2023-10-18T20:24:58.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d56fd191ea93b59f-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.551 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.551 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.552 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.552 DEBUG:boto:Path: /bofrnf-56/b 2023-10-18T20:24:58.553 DEBUG:boto:Data: 2023-10-18T20:24:58.553 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.553 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.554 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.554 DEBUG:boto:Params: {} 2023-10-18T20:24:58.555 DEBUG:boto:Token: None 2023-10-18T20:24:58.555 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b 2023-10-18T20:24:58.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:79X4SRTEHtHwtR2VxrKZXdt57k4= 2023-10-18T20:24:58.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:79X4SRTEHtHwtR2VxrKZXdt57k4=', 'Content-Length': '0'} 2023-10-18T20:24:58.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f53f87ee66f02423-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:24:58.559 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.560 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.560 DEBUG:boto:Method: GET 2023-10-18T20:24:58.561 DEBUG:boto:Path: /bofrnf-56/b 2023-10-18T20:24:58.561 DEBUG:boto:Data: 2023-10-18T20:24:58.562 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.563 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.564 DEBUG:boto:Params: {} 2023-10-18T20:24:58.564 DEBUG:boto:Token: None 2023-10-18T20:24:58.565 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b 2023-10-18T20:24:58.565 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:goigRDmjTtLMhTW4QvWhZK6fim8= 2023-10-18T20:24:58.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:goigRDmjTtLMhTW4QvWhZK6fim8=', 'Content-Length': '0'} 2023-10-18T20:24:58.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d3a3c3dbfc63c2-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.568 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:24:58.569 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:24:58.569 DEBUG:boto:Method: GET 2023-10-18T20:24:58.570 DEBUG:boto:Path: /bofrnf-56/b 2023-10-18T20:24:58.570 DEBUG:boto:Data: 2023-10-18T20:24:58.571 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.571 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.572 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.572 DEBUG:boto:Params: {} 2023-10-18T20:24:58.573 DEBUG:boto:Token: None 2023-10-18T20:24:58.573 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/b 2023-10-18T20:24:58.574 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:goigRDmjTtLMhTW4QvWhZK6fim8= 2023-10-18T20:24:58.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:goigRDmjTtLMhTW4QvWhZK6fim8=', 'Content-Length': '0'} 2023-10-18T20:24:58.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0d9fea9ec5801db-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.577 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.578 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.579 DEBUG:boto:path=/bofrnf-56/?prefix=b 2023-10-18T20:24:58.579 DEBUG:boto:auth_path=/bofrnf-56/?prefix=b 2023-10-18T20:24:58.580 DEBUG:boto:Method: GET 2023-10-18T20:24:58.581 DEBUG:boto:Path: /bofrnf-56/?prefix=b 2023-10-18T20:24:58.581 DEBUG:boto:Data: 2023-10-18T20:24:58.582 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.584 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.584 DEBUG:boto:Params: {} 2023-10-18T20:24:58.585 DEBUG:boto:Token: None 2023-10-18T20:24:58.586 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.587 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb3062c7be8c678b-0065303f1a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.591 DEBUG:boto:b'bofrnf-56b1000falseb2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.592 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.593 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.594 DEBUG:boto:path=/bofrnf-56/?prefix=b 2023-10-18T20:24:58.594 DEBUG:boto:auth_path=/bofrnf-56/?prefix=b 2023-10-18T20:24:58.595 DEBUG:boto:Method: GET 2023-10-18T20:24:58.596 DEBUG:boto:Path: /bofrnf-56/?prefix=b 2023-10-18T20:24:58.596 DEBUG:boto:Data: 2023-10-18T20:24:58.597 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.598 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.599 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.599 DEBUG:boto:Params: {} 2023-10-18T20:24:58.600 DEBUG:boto:Token: None 2023-10-18T20:24:58.601 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.601 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010c4c7ee8b31294a-0065303f1a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.605 DEBUG:boto:b'bofrnf-56b1000falseb2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:24:58.607 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.608 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.608 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.609 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.609 DEBUG:boto:Method: GET 2023-10-18T20:24:58.610 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.610 DEBUG:boto:Data: 2023-10-18T20:24:58.610 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.611 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.611 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.612 DEBUG:boto:Params: {} 2023-10-18T20:24:58.612 DEBUG:boto:Token: None 2023-10-18T20:24:58.613 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:njTVejgybl7uaXepyzHtbDPs71E= 2023-10-18T20:24:58.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:njTVejgybl7uaXepyzHtbDPs71E=', 'Content-Length': '0'} 2023-10-18T20:24:58.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037511fee80cb442-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.658 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.659 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.659 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.660 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.660 DEBUG:boto:Method: GET 2023-10-18T20:24:58.661 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.661 DEBUG:boto:Data: 2023-10-18T20:24:58.661 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.662 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.663 DEBUG:boto:Params: {} 2023-10-18T20:24:58.664 DEBUG:boto:Token: None 2023-10-18T20:24:58.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:njTVejgybl7uaXepyzHtbDPs71E= 2023-10-18T20:24:58.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:njTVejgybl7uaXepyzHtbDPs71E=', 'Content-Length': '0'} 2023-10-18T20:24:58.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000320e05ee2517376f-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.668 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.668 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.669 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.669 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.670 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.670 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.671 DEBUG:boto:Data: 2023-10-18T20:24:58.671 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.672 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.672 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.673 DEBUG:boto:Params: {} 2023-10-18T20:24:58.673 DEBUG:boto:Token: None 2023-10-18T20:24:58.674 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.674 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VFUUlntETe1W2GD84Vf7v/yC18I= 2023-10-18T20:24:58.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VFUUlntETe1W2GD84Vf7v/yC18I=', 'Content-Length': '0'} 2023-10-18T20:24:58.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7d2fd4f5aa05559-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.677 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.678 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.678 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.679 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.679 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.680 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.680 DEBUG:boto:Data: 2023-10-18T20:24:58.680 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.681 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.682 DEBUG:boto:Params: {} 2023-10-18T20:24:58.682 DEBUG:boto:Token: None 2023-10-18T20:24:58.683 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VFUUlntETe1W2GD84Vf7v/yC18I= 2023-10-18T20:24:58.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VFUUlntETe1W2GD84Vf7v/yC18I=', 'Content-Length': '0'} 2023-10-18T20:24:58.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0abf3680f455678-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:24:58.686 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.687 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.687 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.687 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.688 DEBUG:boto:Method: GET 2023-10-18T20:24:58.688 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.689 DEBUG:boto:Data: 2023-10-18T20:24:58.689 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.690 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.690 DEBUG:boto:Params: {} 2023-10-18T20:24:58.691 DEBUG:boto:Token: None 2023-10-18T20:24:58.691 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:njTVejgybl7uaXepyzHtbDPs71E= 2023-10-18T20:24:58.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:njTVejgybl7uaXepyzHtbDPs71E=', 'Content-Length': '0'} 2023-10-18T20:24:58.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304c52a10458fa60-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.695 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.696 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.696 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.696 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:24:58.697 DEBUG:boto:Method: GET 2023-10-18T20:24:58.697 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.698 DEBUG:boto:Data: 2023-10-18T20:24:58.698 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.699 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.699 DEBUG:boto:Params: {} 2023-10-18T20:24:58.700 DEBUG:boto:Token: None 2023-10-18T20:24:58.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:24:58.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:njTVejgybl7uaXepyzHtbDPs71E= 2023-10-18T20:24:58.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:njTVejgybl7uaXepyzHtbDPs71E=', 'Content-Length': '0'} 2023-10-18T20:24:58.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6c2b1a8d534530a-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.747 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.747 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.748 DEBUG:boto:path=/bofrnf-56/?prefix=c 2023-10-18T20:24:58.748 DEBUG:boto:auth_path=/bofrnf-56/?prefix=c 2023-10-18T20:24:58.748 DEBUG:boto:Method: GET 2023-10-18T20:24:58.749 DEBUG:boto:Path: /bofrnf-56/?prefix=c 2023-10-18T20:24:58.749 DEBUG:boto:Data: 2023-10-18T20:24:58.749 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.750 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.750 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.751 DEBUG:boto:Params: {} 2023-10-18T20:24:58.751 DEBUG:boto:Token: None 2023-10-18T20:24:58.751 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.752 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008267a3e43ad55d35-0065303f1a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.756 DEBUG:boto:b'bofrnf-56c1000falsec2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.756 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.757 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.757 DEBUG:boto:path=/bofrnf-56/?prefix=c 2023-10-18T20:24:58.757 DEBUG:boto:auth_path=/bofrnf-56/?prefix=c 2023-10-18T20:24:58.758 DEBUG:boto:Method: GET 2023-10-18T20:24:58.758 DEBUG:boto:Path: /bofrnf-56/?prefix=c 2023-10-18T20:24:58.759 DEBUG:boto:Data: 2023-10-18T20:24:58.759 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.759 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.760 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.760 DEBUG:boto:Params: {} 2023-10-18T20:24:58.760 DEBUG:boto:Token: None 2023-10-18T20:24:58.761 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.761 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8edc6ea0188feb3-0065303f1a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.765 DEBUG:boto:b'bofrnf-56c1000falsec2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.766 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.766 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.766 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.767 DEBUG:boto:Path: /bofrnf-56/c 2023-10-18T20:24:58.767 DEBUG:boto:Data: 2023-10-18T20:24:58.768 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.768 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.768 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.769 DEBUG:boto:Params: {} 2023-10-18T20:24:58.769 DEBUG:boto:Token: None 2023-10-18T20:24:58.769 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c 2023-10-18T20:24:58.770 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7xfBHvuH/VPLCTQiJCSXeev4uzo= 2023-10-18T20:24:58.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7xfBHvuH/VPLCTQiJCSXeev4uzo=', 'Content-Length': '0'} 2023-10-18T20:24:58.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898655d2fec22099-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.773 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.773 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.773 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.774 DEBUG:boto:Path: /bofrnf-56/c 2023-10-18T20:24:58.774 DEBUG:boto:Data: 2023-10-18T20:24:58.774 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.775 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.775 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.776 DEBUG:boto:Params: {} 2023-10-18T20:24:58.776 DEBUG:boto:Token: None 2023-10-18T20:24:58.776 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c 2023-10-18T20:24:58.777 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7xfBHvuH/VPLCTQiJCSXeev4uzo= 2023-10-18T20:24:58.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7xfBHvuH/VPLCTQiJCSXeev4uzo=', 'Content-Length': '0'} 2023-10-18T20:24:58.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000672986011194f159-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:24:58.780 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.780 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.781 DEBUG:boto:Method: GET 2023-10-18T20:24:58.781 DEBUG:boto:Path: /bofrnf-56/c 2023-10-18T20:24:58.782 DEBUG:boto:Data: 2023-10-18T20:24:58.782 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.782 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.783 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.783 DEBUG:boto:Params: {} 2023-10-18T20:24:58.783 DEBUG:boto:Token: None 2023-10-18T20:24:58.784 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c 2023-10-18T20:24:58.784 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VvT5wSu4K10a3bM6yEqWvY2ilV0= 2023-10-18T20:24:58.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VvT5wSu4K10a3bM6yEqWvY2ilV0=', 'Content-Length': '0'} 2023-10-18T20:24:58.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f58bee4f958cc2f-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.787 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:24:58.788 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:24:58.788 DEBUG:boto:Method: GET 2023-10-18T20:24:58.788 DEBUG:boto:Path: /bofrnf-56/c 2023-10-18T20:24:58.789 DEBUG:boto:Data: 2023-10-18T20:24:58.789 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.790 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.790 DEBUG:boto:Params: {} 2023-10-18T20:24:58.791 DEBUG:boto:Token: None 2023-10-18T20:24:58.791 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/c 2023-10-18T20:24:58.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VvT5wSu4K10a3bM6yEqWvY2ilV0= 2023-10-18T20:24:58.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VvT5wSu4K10a3bM6yEqWvY2ilV0=', 'Content-Length': '0'} 2023-10-18T20:24:58.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004754e2330e4acb81-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.795 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.795 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.796 DEBUG:boto:path=/bofrnf-56/?prefix=c 2023-10-18T20:24:58.796 DEBUG:boto:auth_path=/bofrnf-56/?prefix=c 2023-10-18T20:24:58.796 DEBUG:boto:Method: GET 2023-10-18T20:24:58.797 DEBUG:boto:Path: /bofrnf-56/?prefix=c 2023-10-18T20:24:58.797 DEBUG:boto:Data: 2023-10-18T20:24:58.797 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.798 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.798 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.799 DEBUG:boto:Params: {} 2023-10-18T20:24:58.799 DEBUG:boto:Token: None 2023-10-18T20:24:58.799 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.800 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016a8a4e0591ee349-0065303f1a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.846 DEBUG:boto:b'bofrnf-56c1000falsec2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.847 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.847 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.848 DEBUG:boto:path=/bofrnf-56/?prefix=c 2023-10-18T20:24:58.848 DEBUG:boto:auth_path=/bofrnf-56/?prefix=c 2023-10-18T20:24:58.848 DEBUG:boto:Method: GET 2023-10-18T20:24:58.849 DEBUG:boto:Path: /bofrnf-56/?prefix=c 2023-10-18T20:24:58.849 DEBUG:boto:Data: 2023-10-18T20:24:58.849 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.850 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.850 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.851 DEBUG:boto:Params: {} 2023-10-18T20:24:58.851 DEBUG:boto:Token: None 2023-10-18T20:24:58.851 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000729c2cd10c15a537-0065303f1a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.855 DEBUG:boto:b'bofrnf-56c1000falsec2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:24:58.856 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:58.857 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:58.857 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.858 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.858 DEBUG:boto:Method: GET 2023-10-18T20:24:58.858 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.859 DEBUG:boto:Data: 2023-10-18T20:24:58.859 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.860 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.860 DEBUG:boto:Params: {} 2023-10-18T20:24:58.861 DEBUG:boto:Token: None 2023-10-18T20:24:58.861 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.861 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e3kHLoZXwnZoWojIkZ47TqfUiQI= 2023-10-18T20:24:58.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e3kHLoZXwnZoWojIkZ47TqfUiQI=', 'Content-Length': '0'} 2023-10-18T20:24:58.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb01c544d6791823-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.864 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:58.865 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:58.865 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.866 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.866 DEBUG:boto:Method: GET 2023-10-18T20:24:58.866 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.867 DEBUG:boto:Data: 2023-10-18T20:24:58.867 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.868 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.868 DEBUG:boto:Params: {} 2023-10-18T20:24:58.869 DEBUG:boto:Token: None 2023-10-18T20:24:58.869 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.870 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e3kHLoZXwnZoWojIkZ47TqfUiQI= 2023-10-18T20:24:58.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e3kHLoZXwnZoWojIkZ47TqfUiQI=', 'Content-Length': '0'} 2023-10-18T20:24:58.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6f37bf5bad42415-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.914 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:58.915 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:58.915 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.915 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.916 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.916 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.916 DEBUG:boto:Data: 2023-10-18T20:24:58.917 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.918 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.918 DEBUG:boto:Params: {} 2023-10-18T20:24:58.918 DEBUG:boto:Token: None 2023-10-18T20:24:58.919 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.919 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ORSdp1q3+ruGbc2UhI+zlvznci4= 2023-10-18T20:24:58.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ORSdp1q3+ruGbc2UhI+zlvznci4=', 'Content-Length': '0'} 2023-10-18T20:24:58.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000574e71a6e0d6af6c-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.922 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:58.922 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:58.923 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.923 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.923 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.924 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.924 DEBUG:boto:Data: 2023-10-18T20:24:58.924 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.925 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.925 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.925 DEBUG:boto:Params: {} 2023-10-18T20:24:58.926 DEBUG:boto:Token: None 2023-10-18T20:24:58.926 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.927 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ORSdp1q3+ruGbc2UhI+zlvznci4= 2023-10-18T20:24:58.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ORSdp1q3+ruGbc2UhI+zlvznci4=', 'Content-Length': '0'} 2023-10-18T20:24:58.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007912e5c1ab0a7483-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:24:58.930 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:58.931 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:58.931 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.931 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.932 DEBUG:boto:Method: GET 2023-10-18T20:24:58.932 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.932 DEBUG:boto:Data: 2023-10-18T20:24:58.933 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.933 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.934 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.934 DEBUG:boto:Params: {} 2023-10-18T20:24:58.934 DEBUG:boto:Token: None 2023-10-18T20:24:58.935 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.935 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e3kHLoZXwnZoWojIkZ47TqfUiQI= 2023-10-18T20:24:58.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e3kHLoZXwnZoWojIkZ47TqfUiQI=', 'Content-Length': '0'} 2023-10-18T20:24:58.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004be90e6d33c146f7-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.938 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:58.939 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:58.939 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.939 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:24:58.940 DEBUG:boto:Method: GET 2023-10-18T20:24:58.940 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.941 DEBUG:boto:Data: 2023-10-18T20:24:58.941 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.942 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.942 DEBUG:boto:Params: {} 2023-10-18T20:24:58.942 DEBUG:boto:Token: None 2023-10-18T20:24:58.943 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:24:58.943 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e3kHLoZXwnZoWojIkZ47TqfUiQI= 2023-10-18T20:24:58.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e3kHLoZXwnZoWojIkZ47TqfUiQI=', 'Content-Length': '0'} 2023-10-18T20:24:58.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20d273912e33b7b-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.946 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.947 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.947 DEBUG:boto:path=/bofrnf-56/?prefix=d 2023-10-18T20:24:58.947 DEBUG:boto:auth_path=/bofrnf-56/?prefix=d 2023-10-18T20:24:58.948 DEBUG:boto:Method: GET 2023-10-18T20:24:58.948 DEBUG:boto:Path: /bofrnf-56/?prefix=d 2023-10-18T20:24:58.949 DEBUG:boto:Data: 2023-10-18T20:24:58.949 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.950 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.950 DEBUG:boto:Params: {} 2023-10-18T20:24:58.950 DEBUG:boto:Token: None 2023-10-18T20:24:58.951 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f55fb289e60a07b7-0065303f1a-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.955 DEBUG:boto:b'bofrnf-56d1000falsed2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.955 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:58.956 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:58.956 DEBUG:boto:path=/bofrnf-56/?prefix=d 2023-10-18T20:24:58.956 DEBUG:boto:auth_path=/bofrnf-56/?prefix=d 2023-10-18T20:24:58.957 DEBUG:boto:Method: GET 2023-10-18T20:24:58.957 DEBUG:boto:Path: /bofrnf-56/?prefix=d 2023-10-18T20:24:58.957 DEBUG:boto:Data: 2023-10-18T20:24:58.958 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.959 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.959 DEBUG:boto:Params: {} 2023-10-18T20:24:58.959 DEBUG:boto:Token: None 2023-10-18T20:24:58.960 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/ 2023-10-18T20:24:58.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI= 2023-10-18T20:24:58.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMDkSzopD3TY/6MSxfiVZOHiEAI=', 'Content-Length': '0'} 2023-10-18T20:24:58.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000894c510f3d8c73b5-0065303f1a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.964 DEBUG:boto:b'bofrnf-56d1000falsed2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:58.964 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:58.965 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:58.965 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.965 DEBUG:boto:Path: /bofrnf-56/d 2023-10-18T20:24:58.966 DEBUG:boto:Data: 2023-10-18T20:24:58.966 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.967 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.967 DEBUG:boto:Params: {} 2023-10-18T20:24:58.967 DEBUG:boto:Token: None 2023-10-18T20:24:58.968 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/d 2023-10-18T20:24:58.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F10rgVlvOT+LoSE7Q//NcqyZ4zU= 2023-10-18T20:24:58.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F10rgVlvOT+LoSE7Q//NcqyZ4zU=', 'Content-Length': '0'} 2023-10-18T20:24:58.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c70cc058101ffe19-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.971 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:58.971 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:58.972 DEBUG:boto:Method: HEAD 2023-10-18T20:24:58.972 DEBUG:boto:Path: /bofrnf-56/d 2023-10-18T20:24:58.973 DEBUG:boto:Data: 2023-10-18T20:24:58.973 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.973 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:58.974 DEBUG:boto:Port: 8000 2023-10-18T20:24:58.974 DEBUG:boto:Params: {} 2023-10-18T20:24:58.974 DEBUG:boto:Token: None 2023-10-18T20:24:58.975 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/d 2023-10-18T20:24:58.975 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F10rgVlvOT+LoSE7Q//NcqyZ4zU= 2023-10-18T20:24:58.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F10rgVlvOT+LoSE7Q//NcqyZ4zU=', 'Content-Length': '0'} 2023-10-18T20:24:58.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd1b4b5a417bdd2-0065303f1a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:58.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:24:58.979 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:58.979 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:58.980 DEBUG:boto:Method: GET 2023-10-18T20:24:58.980 DEBUG:boto:Path: /bofrnf-56/d 2023-10-18T20:24:58.980 DEBUG:boto:Data: 2023-10-18T20:24:58.981 DEBUG:boto:Headers: {} 2023-10-18T20:24:58.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:58.981 DEBUG:boto:Port: 8001 2023-10-18T20:24:58.982 DEBUG:boto:Params: {} 2023-10-18T20:24:58.982 DEBUG:boto:Token: None 2023-10-18T20:24:58.983 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:58 GMT /bofrnf-56/d 2023-10-18T20:24:58.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DN415jv/wlyF/uwspZz/5MMhxA8= 2023-10-18T20:24:58.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DN415jv/wlyF/uwspZz/5MMhxA8=', 'Content-Length': '0'} 2023-10-18T20:24:58.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc1abafa8d6c10a3-0065303f1a-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.030 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:24:59.031 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:24:59.031 DEBUG:boto:Method: GET 2023-10-18T20:24:59.032 DEBUG:boto:Path: /bofrnf-56/d 2023-10-18T20:24:59.032 DEBUG:boto:Data: 2023-10-18T20:24:59.032 DEBUG:boto:Headers: {} 2023-10-18T20:24:59.033 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.033 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.034 DEBUG:boto:Params: {} 2023-10-18T20:24:59.034 DEBUG:boto:Token: None 2023-10-18T20:24:59.035 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/d 2023-10-18T20:24:59.035 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o/5Wy+yOO/iUoLYn8G7MfWn5Scg= 2023-10-18T20:24:59.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o/5Wy+yOO/iUoLYn8G7MfWn5Scg=', 'Content-Length': '0'} 2023-10-18T20:24:59.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b87ba3609872c26-0065303f1b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.038 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:59.038 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:59.039 DEBUG:boto:path=/bofrnf-56/?prefix=d 2023-10-18T20:24:59.039 DEBUG:boto:auth_path=/bofrnf-56/?prefix=d 2023-10-18T20:24:59.039 DEBUG:boto:Method: GET 2023-10-18T20:24:59.040 DEBUG:boto:Path: /bofrnf-56/?prefix=d 2023-10-18T20:24:59.040 DEBUG:boto:Data: 2023-10-18T20:24:59.041 DEBUG:boto:Headers: {} 2023-10-18T20:24:59.041 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:24:59.041 DEBUG:boto:Port: 8001 2023-10-18T20:24:59.042 DEBUG:boto:Params: {} 2023-10-18T20:24:59.042 DEBUG:boto:Token: None 2023-10-18T20:24:59.043 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/ 2023-10-18T20:24:59.043 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Iu9rW+REAhAwnYV+fSOFEkgg3MQ= 2023-10-18T20:24:59.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Iu9rW+REAhAwnYV+fSOFEkgg3MQ=', 'Content-Length': '0'} 2023-10-18T20:24:59.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd7d2ec4552b23c-0065303f1b-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.047 DEBUG:boto:b'bofrnf-56d1000falsed2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:59.047 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:59.048 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:59.048 DEBUG:boto:path=/bofrnf-56/?prefix=d 2023-10-18T20:24:59.048 DEBUG:boto:auth_path=/bofrnf-56/?prefix=d 2023-10-18T20:24:59.049 DEBUG:boto:Method: GET 2023-10-18T20:24:59.049 DEBUG:boto:Path: /bofrnf-56/?prefix=d 2023-10-18T20:24:59.049 DEBUG:boto:Data: 2023-10-18T20:24:59.050 DEBUG:boto:Headers: {} 2023-10-18T20:24:59.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.051 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.051 DEBUG:boto:Params: {} 2023-10-18T20:24:59.051 DEBUG:boto:Token: None 2023-10-18T20:24:59.052 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/ 2023-10-18T20:24:59.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Iu9rW+REAhAwnYV+fSOFEkgg3MQ= 2023-10-18T20:24:59.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Iu9rW+REAhAwnYV+fSOFEkgg3MQ=', 'Content-Length': '0'} 2023-10-18T20:24:59.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0533b50ef6d0f00-0065303f1b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.056 DEBUG:boto:b'bofrnf-56d1000falsed2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:24:59.056 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-56 zones={a2, a1} 2023-10-18T20:24:59.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'bofrnf-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:59.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'bofrnf-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:59.057 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket bofrnf-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:24:59.113 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:59.113 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:59.155 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:24:59.156 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-56 2023-10-18T20:24:59.159 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 4 2023-10-18T20:24:59.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'bofrnf-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:59.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'bofrnf-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:24:59.196 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket bofrnf-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:24:59.245 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:59.245 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:59.281 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:24:59.281 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-56 2023-10-18T20:24:59.282 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 4 2023-10-18T20:24:59.309 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:59.310 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:59.310 DEBUG:boto:Method: HEAD 2023-10-18T20:24:59.310 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:59.311 DEBUG:boto:Data: 2023-10-18T20:24:59.311 DEBUG:boto:Headers: {} 2023-10-18T20:24:59.311 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.312 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.312 DEBUG:boto:Params: {} 2023-10-18T20:24:59.313 DEBUG:boto:Token: None 2023-10-18T20:24:59.313 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/ 2023-10-18T20:24:59.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gA6stDYeonFXr6zYlzlawnaWK2o= 2023-10-18T20:24:59.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gA6stDYeonFXr6zYlzlawnaWK2o=', 'Content-Length': '0'} 2023-10-18T20:24:59.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002958fb537c550e58-0065303f1b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.316 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:24:59.317 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:24:59.317 DEBUG:boto:Method: PUT 2023-10-18T20:24:59.317 DEBUG:boto:Path: /bofrnf-56/e 2023-10-18T20:24:59.318 DEBUG:boto:Data: 2023-10-18T20:24:59.318 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:24:59.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.319 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.319 DEBUG:boto:Params: {} 2023-10-18T20:24:59.319 DEBUG:boto:Token: None 2023-10-18T20:24:59.320 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/e 2023-10-18T20:24:59.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ow7cFSHT7CwrUzC683q5qaaVFjE= 2023-10-18T20:24:59.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ow7cFSHT7CwrUzC683q5qaaVFjE='} 2023-10-18T20:24:59.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b4ba78c7a9ba159-0065303f1b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.326 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:59.326 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:59.327 DEBUG:boto:Method: HEAD 2023-10-18T20:24:59.327 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:59.327 DEBUG:boto:Data: 2023-10-18T20:24:59.328 DEBUG:boto:Headers: {} 2023-10-18T20:24:59.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.329 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.329 DEBUG:boto:Params: {} 2023-10-18T20:24:59.329 DEBUG:boto:Token: None 2023-10-18T20:24:59.330 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/ 2023-10-18T20:24:59.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gA6stDYeonFXr6zYlzlawnaWK2o= 2023-10-18T20:24:59.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gA6stDYeonFXr6zYlzlawnaWK2o=', 'Content-Length': '0'} 2023-10-18T20:24:59.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090cfbe2f2a6166aa-0065303f1b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.333 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:24:59.333 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:24:59.334 DEBUG:boto:Method: PUT 2023-10-18T20:24:59.334 DEBUG:boto:Path: /bofrnf-56/f 2023-10-18T20:24:59.334 DEBUG:boto:Data: 2023-10-18T20:24:59.335 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:24:59.335 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.335 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.336 DEBUG:boto:Params: {} 2023-10-18T20:24:59.336 DEBUG:boto:Token: None 2023-10-18T20:24:59.337 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/f 2023-10-18T20:24:59.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/hQ5pXEO/Rt10RXCwO0tf0wVeJ8= 2023-10-18T20:24:59.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/hQ5pXEO/Rt10RXCwO0tf0wVeJ8='} 2023-10-18T20:24:59.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057bfe18b0de2b3c8-0065303f1b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.383 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:59.383 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:59.384 DEBUG:boto:Method: HEAD 2023-10-18T20:24:59.384 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:59.384 DEBUG:boto:Data: 2023-10-18T20:24:59.385 DEBUG:boto:Headers: {} 2023-10-18T20:24:59.385 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.385 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.386 DEBUG:boto:Params: {} 2023-10-18T20:24:59.386 DEBUG:boto:Token: None 2023-10-18T20:24:59.387 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/ 2023-10-18T20:24:59.387 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gA6stDYeonFXr6zYlzlawnaWK2o= 2023-10-18T20:24:59.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gA6stDYeonFXr6zYlzlawnaWK2o=', 'Content-Length': '0'} 2023-10-18T20:24:59.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009bdde17798884ef-0065303f1b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.390 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:24:59.390 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:24:59.391 DEBUG:boto:Method: PUT 2023-10-18T20:24:59.391 DEBUG:boto:Path: /bofrnf-56/g 2023-10-18T20:24:59.391 DEBUG:boto:Data: 2023-10-18T20:24:59.392 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:24:59.392 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.392 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.393 DEBUG:boto:Params: {} 2023-10-18T20:24:59.393 DEBUG:boto:Token: None 2023-10-18T20:24:59.394 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/g 2023-10-18T20:24:59.394 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uZcf8ZXwSNfhhCuHExRmwHucJOc= 2023-10-18T20:24:59.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uZcf8ZXwSNfhhCuHExRmwHucJOc='} 2023-10-18T20:24:59.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba2ebaa1bb05a4fe-0065303f1b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.400 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:24:59.400 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:24:59.400 DEBUG:boto:Method: HEAD 2023-10-18T20:24:59.401 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:24:59.401 DEBUG:boto:Data: 2023-10-18T20:24:59.401 DEBUG:boto:Headers: {} 2023-10-18T20:24:59.402 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.402 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.402 DEBUG:boto:Params: {} 2023-10-18T20:24:59.403 DEBUG:boto:Token: None 2023-10-18T20:24:59.403 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/ 2023-10-18T20:24:59.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gA6stDYeonFXr6zYlzlawnaWK2o= 2023-10-18T20:24:59.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gA6stDYeonFXr6zYlzlawnaWK2o=', 'Content-Length': '0'} 2023-10-18T20:24:59.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000399931fa0adc1ea4-0065303f1b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.406 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:24:59.407 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:24:59.407 DEBUG:boto:Method: PUT 2023-10-18T20:24:59.407 DEBUG:boto:Path: /bofrnf-56/h 2023-10-18T20:24:59.408 DEBUG:boto:Data: 2023-10-18T20:24:59.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:24:59.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:24:59.409 DEBUG:boto:Port: 8000 2023-10-18T20:24:59.409 DEBUG:boto:Params: {} 2023-10-18T20:24:59.409 DEBUG:boto:Token: None 2023-10-18T20:24:59.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:24:59 GMT /bofrnf-56/h 2023-10-18T20:24:59.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SINAya7SVSasEI4OtbBPZ0myYng= 2023-10-18T20:24:59.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:24:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SINAya7SVSasEI4OtbBPZ0myYng='} 2023-10-18T20:24:59.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028a94f3ce47dcba3-0065303f1b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:24:59.414 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-56 2023-10-18T20:24:59.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:59.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:24:59.416 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:24:59.472 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:24:59.472 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:24:59.504 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:24:59.503+0000 7fb8c94c3780 1 waiting to reach latest gen 2, on 0.. 2023-10-18T20:25:04.505 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:04.504+0000 7fb8c94c3780 1 waiting to reach latest gen 2, on 0.. 2023-10-18T20:25:09.507 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:09.504+0000 7fb8c94c3780 1 waiting to reach latest gen 2, on 0.. 2023-10-18T20:25:14.506 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:14.504+0000 7fb8c94c3780 1 waiting to reach latest gen 2, on 0.. 2023-10-18T20:25:19.507 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:19.505+0000 7fb8c94c3780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:25:24.508 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:24.505+0000 7fb8c94c3780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:25:29.508 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:29.507+0000 7fb8c94c3780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:25:34.509 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:34.508+0000 7fb8c94c3780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:25:39.510 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:39.509+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:25:39.510 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:25:39.511 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:25:44.511 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:44.509+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:25:44.511 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:25:44.511 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:25:49.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:49.510+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:25:49.512 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:25:49.512 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:25:54.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:54.511+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:25:54.513 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:25:54.513 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:25:59.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:25:59.512+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:25:59.513 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:25:59.514 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:04.514 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:04.512+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:04.514 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:26:04.514 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:09.515 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:09.512+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:09.515 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:26:09.515 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:14.515 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:14.513+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:14.516 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:26:14.516 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:19.516 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:19.514+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:19.517 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:26:19.517 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:24.517 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:24.516+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:24.518 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:26:24.518 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:29.518 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:29.517+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:29.519 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:26:29.519 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:34.519 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:34.517+0000 7fb8c94c3780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:34.519 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , ] 2023-10-18T20:26:34.519 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:39.520 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:39.518+0000 7fb8c94c3780 1 bucket sync caught up with source: 2023-10-18T20:26:39.520 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:39.521 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.34.6, 00000000002.27.6, 00000000002.32.6] 2023-10-18T20:26:39.521 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:39.518+0000 7fb8c94c3780 0 bucket checkpoint complete 2023-10-18T20:26:39.537 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-56 2023-10-18T20:26:39.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:26:39.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:26:39.539 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:26:39.599 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:26:39.599 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:26:39.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:39.634+0000 7f1a21135780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:39.635 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.710.6, , , , 00000000001.832.6, , , 00000000001.1186.6, , 00000000001.831.6] 2023-10-18T20:26:39.635 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.716.6, , , 00000000001.1196.6, 00000000001.832.6, , , 00000000002.1198.6, , 00000000002.842.6] 2023-10-18T20:26:44.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:44.635+0000 7f1a21135780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:44.637 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.710.6, , , , 00000000001.832.6, , , 00000000001.1186.6, , 00000000001.831.6] 2023-10-18T20:26:44.637 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.716.6, , , 00000000001.1196.6, 00000000001.832.6, , , 00000000002.1198.6, , 00000000002.842.6] 2023-10-18T20:26:49.638 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:49.637+0000 7f1a21135780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:49.639 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.710.6, , , , 00000000001.832.6, , , 00000000001.1186.6, , 00000000001.831.6] 2023-10-18T20:26:49.639 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.716.6, , , 00000000001.1196.6, 00000000001.832.6, , , 00000000002.1198.6, , 00000000002.842.6] 2023-10-18T20:26:54.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:54.638+0000 7f1a21135780 1 waiting for incremental sync to catch up: 2023-10-18T20:26:54.640 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.710.6, , , , 00000000001.832.6, , , 00000000001.1186.6, , 00000000001.831.6] 2023-10-18T20:26:54.640 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.716.6, , , 00000000001.1196.6, 00000000001.832.6, , , 00000000002.1198.6, , 00000000002.842.6] 2023-10-18T20:26:59.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:59.639+0000 7f1a21135780 1 bucket sync caught up with source: 2023-10-18T20:26:59.642 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.716.6, , , 00000000001.1196.6, 00000000001.832.6, , , 00000000002.1198.6, , 00000000002.842.6] 2023-10-18T20:26:59.642 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.716.6, , , 00000000001.1196.6, 00000000001.832.6, , , 00000000002.1198.6, , 00000000002.842.6] 2023-10-18T20:26:59.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:26:59.639+0000 7f1a21135780 0 bucket checkpoint complete 2023-10-18T20:26:59.659 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-56 zones={a2, a1} 2023-10-18T20:26:59.659 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.660 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.660 DEBUG:boto:Method: HEAD 2023-10-18T20:26:59.660 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:26:59.661 DEBUG:boto:Data: 2023-10-18T20:26:59.662 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.662 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.663 DEBUG:boto:Params: {} 2023-10-18T20:26:59.664 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:26:59.664 DEBUG:boto:Token: None 2023-10-18T20:26:59.665 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/ 2023-10-18T20:26:59.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PlTu3fcwUy/xwJKcctb82woQ6r8= 2023-10-18T20:26:59.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PlTu3fcwUy/xwJKcctb82woQ6r8=', 'Content-Length': '0'} 2023-10-18T20:26:59.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2bbf9d85b32c69b-0065303f93-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.670 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.671 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.671 DEBUG:boto:Method: HEAD 2023-10-18T20:26:59.672 DEBUG:boto:Path: /bofrnf-56/ 2023-10-18T20:26:59.672 DEBUG:boto:Data: 2023-10-18T20:26:59.673 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.673 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.673 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.674 DEBUG:boto:Params: {} 2023-10-18T20:26:59.674 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:26:59.675 DEBUG:boto:Token: None 2023-10-18T20:26:59.675 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/ 2023-10-18T20:26:59.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PlTu3fcwUy/xwJKcctb82woQ6r8= 2023-10-18T20:26:59.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PlTu3fcwUy/xwJKcctb82woQ6r8=', 'Content-Length': '0'} 2023-10-18T20:26:59.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072ab0dd3ad86f77f-0065303f93-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.680 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:26:59.680 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.681 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.681 DEBUG:boto:path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.682 DEBUG:boto:auth_path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.682 DEBUG:boto:Method: GET 2023-10-18T20:26:59.683 DEBUG:boto:Path: /bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.683 DEBUG:boto:Data: 2023-10-18T20:26:59.684 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.684 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.684 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.685 DEBUG:boto:Params: {} 2023-10-18T20:26:59.685 DEBUG:boto:Token: None 2023-10-18T20:26:59.686 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/?versions 2023-10-18T20:26:59.686 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AVeg0j+htcEy3MR0lvUsof5kqBE= 2023-10-18T20:26:59.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AVeg0j+htcEy3MR0lvUsof5kqBE=', 'Content-Length': '0'} 2023-10-18T20:26:59.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000137fbcd800934cd-0065303f93-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.690 DEBUG:boto:b'bofrnf-56999falseanulltrue2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:24:59.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:24:59.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:24:59.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:24:59.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:26:59.691 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:26:59.692 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:26:59.692 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:26:59.693 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:26:59.693 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-18T20:26:59.693 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-18T20:26:59.694 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-18T20:26:59.694 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-18T20:26:59.695 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:26:59.695 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.695 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.696 DEBUG:boto:path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.696 DEBUG:boto:auth_path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.696 DEBUG:boto:Method: GET 2023-10-18T20:26:59.697 DEBUG:boto:Path: /bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.697 DEBUG:boto:Data: 2023-10-18T20:26:59.698 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.698 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.699 DEBUG:boto:Params: {} 2023-10-18T20:26:59.699 DEBUG:boto:Token: None 2023-10-18T20:26:59.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/?versions 2023-10-18T20:26:59.700 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AVeg0j+htcEy3MR0lvUsof5kqBE= 2023-10-18T20:26:59.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AVeg0j+htcEy3MR0lvUsof5kqBE=', 'Content-Length': '0'} 2023-10-18T20:26:59.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000316d98be5ed85890-0065303f93-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.703 DEBUG:boto:b'bofrnf-56999falseanulltrue2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:24:59.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:24:59.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:24:59.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:24:59.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:26:59.704 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:26:59.705 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:26:59.705 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:26:59.706 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:26:59.706 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-18T20:26:59.706 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-18T20:26:59.707 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-18T20:26:59.707 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-18T20:26:59.707 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.708 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.708 DEBUG:boto:path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.709 DEBUG:boto:auth_path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.709 DEBUG:boto:Method: GET 2023-10-18T20:26:59.710 DEBUG:boto:Path: /bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.710 DEBUG:boto:Data: 2023-10-18T20:26:59.710 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.711 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.711 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.712 DEBUG:boto:Params: {} 2023-10-18T20:26:59.712 DEBUG:boto:Token: None 2023-10-18T20:26:59.713 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/?versions 2023-10-18T20:26:59.713 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AVeg0j+htcEy3MR0lvUsof5kqBE= 2023-10-18T20:26:59.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AVeg0j+htcEy3MR0lvUsof5kqBE=', 'Content-Length': '0'} 2023-10-18T20:26:59.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0f5cf2bb0148be6-0065303f93-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.717 DEBUG:boto:b'bofrnf-56999falseanulltrue2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:24:59.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:24:59.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:24:59.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:24:59.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:26:59.718 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.718 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.719 DEBUG:boto:path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.720 DEBUG:boto:auth_path=/bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.721 DEBUG:boto:Method: GET 2023-10-18T20:26:59.721 DEBUG:boto:Path: /bofrnf-56/?versions&max-keys=999 2023-10-18T20:26:59.721 DEBUG:boto:Data: 2023-10-18T20:26:59.722 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.722 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.723 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.723 DEBUG:boto:Params: {} 2023-10-18T20:26:59.724 DEBUG:boto:Token: None 2023-10-18T20:26:59.724 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/?versions 2023-10-18T20:26:59.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AVeg0j+htcEy3MR0lvUsof5kqBE= 2023-10-18T20:26:59.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AVeg0j+htcEy3MR0lvUsof5kqBE=', 'Content-Length': '0'} 2023-10-18T20:26:59.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b53c5c5246b4736-0065303f93-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.728 DEBUG:boto:b'bofrnf-56999falseanulltrue2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:24:59.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:24:59.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:24:59.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:24:59.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:26:59.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:26:59.729 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.730 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.730 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.731 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.731 DEBUG:boto:Method: GET 2023-10-18T20:26:59.731 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.732 DEBUG:boto:Data: 2023-10-18T20:26:59.732 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.733 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.733 DEBUG:boto:Params: {} 2023-10-18T20:26:59.734 DEBUG:boto:Token: None 2023-10-18T20:26:59.734 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SzSVJkuHEJEINGsVhQq9EICphm8= 2023-10-18T20:26:59.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SzSVJkuHEJEINGsVhQq9EICphm8=', 'Content-Length': '0'} 2023-10-18T20:26:59.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8e7a0195a1799a-0065303f93-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.739 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.739 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.739 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.740 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.740 DEBUG:boto:Method: GET 2023-10-18T20:26:59.741 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.741 DEBUG:boto:Data: 2023-10-18T20:26:59.741 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.742 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.742 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.743 DEBUG:boto:Params: {} 2023-10-18T20:26:59.743 DEBUG:boto:Token: None 2023-10-18T20:26:59.744 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.744 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SzSVJkuHEJEINGsVhQq9EICphm8= 2023-10-18T20:26:59.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SzSVJkuHEJEINGsVhQq9EICphm8=', 'Content-Length': '0'} 2023-10-18T20:26:59.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000899ebe92836a668d-0065303f93-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.748 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.748 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.749 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.749 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.749 DEBUG:boto:Method: HEAD 2023-10-18T20:26:59.750 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.750 DEBUG:boto:Data: 2023-10-18T20:26:59.751 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.751 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.752 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.752 DEBUG:boto:Params: {} 2023-10-18T20:26:59.752 DEBUG:boto:Token: None 2023-10-18T20:26:59.753 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.754 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WuRB0VKPEbv6568fRsZaKRKu+Ig= 2023-10-18T20:26:59.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WuRB0VKPEbv6568fRsZaKRKu+Ig=', 'Content-Length': '0'} 2023-10-18T20:26:59.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079c9fb67100a25f0-0065303f93-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.757 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.758 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.758 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.759 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.759 DEBUG:boto:Method: HEAD 2023-10-18T20:26:59.759 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.760 DEBUG:boto:Data: 2023-10-18T20:26:59.760 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.761 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.761 DEBUG:boto:Params: {} 2023-10-18T20:26:59.762 DEBUG:boto:Token: None 2023-10-18T20:26:59.762 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WuRB0VKPEbv6568fRsZaKRKu+Ig= 2023-10-18T20:26:59.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WuRB0VKPEbv6568fRsZaKRKu+Ig=', 'Content-Length': '0'} 2023-10-18T20:26:59.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004cb015ec4620761-0065303f93-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:26:59.767 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.767 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.767 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.768 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.768 DEBUG:boto:Method: GET 2023-10-18T20:26:59.769 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.769 DEBUG:boto:Data: 2023-10-18T20:26:59.770 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.771 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.771 DEBUG:boto:Params: {} 2023-10-18T20:26:59.771 DEBUG:boto:Token: None 2023-10-18T20:26:59.772 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SzSVJkuHEJEINGsVhQq9EICphm8= 2023-10-18T20:26:59.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SzSVJkuHEJEINGsVhQq9EICphm8=', 'Content-Length': '0'} 2023-10-18T20:26:59.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fd7c2e43d19b646-0065303f93-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.776 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.776 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.777 DEBUG:boto:path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.777 DEBUG:boto:auth_path=/bofrnf-56/a?versionId=null 2023-10-18T20:26:59.778 DEBUG:boto:Method: GET 2023-10-18T20:26:59.778 DEBUG:boto:Path: /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.779 DEBUG:boto:Data: 2023-10-18T20:26:59.779 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.780 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.780 DEBUG:boto:Params: {} 2023-10-18T20:26:59.781 DEBUG:boto:Token: None 2023-10-18T20:26:59.781 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a?versionId=null 2023-10-18T20:26:59.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SzSVJkuHEJEINGsVhQq9EICphm8= 2023-10-18T20:26:59.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SzSVJkuHEJEINGsVhQq9EICphm8=', 'Content-Length': '0'} 2023-10-18T20:26:59.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08c3baeeb16e1ee-0065303f93-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.826 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.827 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.828 DEBUG:boto:path=/bofrnf-56/?prefix=a 2023-10-18T20:26:59.828 DEBUG:boto:auth_path=/bofrnf-56/?prefix=a 2023-10-18T20:26:59.829 DEBUG:boto:Method: GET 2023-10-18T20:26:59.829 DEBUG:boto:Path: /bofrnf-56/?prefix=a 2023-10-18T20:26:59.830 DEBUG:boto:Data: 2023-10-18T20:26:59.830 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.831 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.831 DEBUG:boto:Params: {} 2023-10-18T20:26:59.832 DEBUG:boto:Token: None 2023-10-18T20:26:59.832 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/ 2023-10-18T20:26:59.833 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oJbzXnEo7EpIuY4FKh11OWMHgGY= 2023-10-18T20:26:59.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oJbzXnEo7EpIuY4FKh11OWMHgGY=', 'Content-Length': '0'} 2023-10-18T20:26:59.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a03ffc8f10c1706-0065303f93-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.836 DEBUG:boto:b'bofrnf-56a1000falsea2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:26:59.837 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.837 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.838 DEBUG:boto:path=/bofrnf-56/?prefix=a 2023-10-18T20:26:59.838 DEBUG:boto:auth_path=/bofrnf-56/?prefix=a 2023-10-18T20:26:59.839 DEBUG:boto:Method: GET 2023-10-18T20:26:59.839 DEBUG:boto:Path: /bofrnf-56/?prefix=a 2023-10-18T20:26:59.840 DEBUG:boto:Data: 2023-10-18T20:26:59.840 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.841 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.841 DEBUG:boto:Params: {} 2023-10-18T20:26:59.842 DEBUG:boto:Token: None 2023-10-18T20:26:59.842 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/ 2023-10-18T20:26:59.843 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oJbzXnEo7EpIuY4FKh11OWMHgGY= 2023-10-18T20:26:59.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oJbzXnEo7EpIuY4FKh11OWMHgGY=', 'Content-Length': '0'} 2023-10-18T20:26:59.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c28abb3f3a129c0-0065303f93-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.846 DEBUG:boto:b'bofrnf-56a1000falsea2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:26:59.846 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.847 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.847 DEBUG:boto:Method: HEAD 2023-10-18T20:26:59.848 DEBUG:boto:Path: /bofrnf-56/a 2023-10-18T20:26:59.848 DEBUG:boto:Data: 2023-10-18T20:26:59.848 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.849 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.849 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.850 DEBUG:boto:Params: {} 2023-10-18T20:26:59.850 DEBUG:boto:Token: None 2023-10-18T20:26:59.851 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a 2023-10-18T20:26:59.851 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lwexh38EM+WuhIDHEHvv3Ut6kwQ= 2023-10-18T20:26:59.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lwexh38EM+WuhIDHEHvv3Ut6kwQ=', 'Content-Length': '0'} 2023-10-18T20:26:59.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645cfd329873b749-0065303f93-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.855 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.855 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.856 DEBUG:boto:Method: HEAD 2023-10-18T20:26:59.856 DEBUG:boto:Path: /bofrnf-56/a 2023-10-18T20:26:59.857 DEBUG:boto:Data: 2023-10-18T20:26:59.857 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.857 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.858 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.858 DEBUG:boto:Params: {} 2023-10-18T20:26:59.859 DEBUG:boto:Token: None 2023-10-18T20:26:59.859 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a 2023-10-18T20:26:59.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lwexh38EM+WuhIDHEHvv3Ut6kwQ= 2023-10-18T20:26:59.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lwexh38EM+WuhIDHEHvv3Ut6kwQ=', 'Content-Length': '0'} 2023-10-18T20:26:59.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f448a6e3cc3788-0065303f93-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:26:59.863 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.863 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.864 DEBUG:boto:Method: GET 2023-10-18T20:26:59.864 DEBUG:boto:Path: /bofrnf-56/a 2023-10-18T20:26:59.864 DEBUG:boto:Data: 2023-10-18T20:26:59.865 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.866 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.866 DEBUG:boto:Params: {} 2023-10-18T20:26:59.866 DEBUG:boto:Token: None 2023-10-18T20:26:59.867 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a 2023-10-18T20:26:59.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:byAL+E4oT0jLnwjN3g/ZWutylww= 2023-10-18T20:26:59.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:byAL+E4oT0jLnwjN3g/ZWutylww=', 'Content-Length': '0'} 2023-10-18T20:26:59.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccf0eb370604859a-0065303f93-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.870 DEBUG:boto:path=/bofrnf-56/a 2023-10-18T20:26:59.871 DEBUG:boto:auth_path=/bofrnf-56/a 2023-10-18T20:26:59.871 DEBUG:boto:Method: GET 2023-10-18T20:26:59.872 DEBUG:boto:Path: /bofrnf-56/a 2023-10-18T20:26:59.873 DEBUG:boto:Data: 2023-10-18T20:26:59.873 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.874 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.875 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.876 DEBUG:boto:Params: {} 2023-10-18T20:26:59.876 DEBUG:boto:Token: None 2023-10-18T20:26:59.877 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/a 2023-10-18T20:26:59.878 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:byAL+E4oT0jLnwjN3g/ZWutylww= 2023-10-18T20:26:59.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:byAL+E4oT0jLnwjN3g/ZWutylww=', 'Content-Length': '0'} 2023-10-18T20:26:59.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ec3b9324c6ce2b-0065303f93-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.882 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.882 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.883 DEBUG:boto:path=/bofrnf-56/?prefix=a 2023-10-18T20:26:59.883 DEBUG:boto:auth_path=/bofrnf-56/?prefix=a 2023-10-18T20:26:59.884 DEBUG:boto:Method: GET 2023-10-18T20:26:59.884 DEBUG:boto:Path: /bofrnf-56/?prefix=a 2023-10-18T20:26:59.885 DEBUG:boto:Data: 2023-10-18T20:26:59.885 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.885 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.886 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.886 DEBUG:boto:Params: {} 2023-10-18T20:26:59.887 DEBUG:boto:Token: None 2023-10-18T20:26:59.887 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/ 2023-10-18T20:26:59.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oJbzXnEo7EpIuY4FKh11OWMHgGY= 2023-10-18T20:26:59.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oJbzXnEo7EpIuY4FKh11OWMHgGY=', 'Content-Length': '0'} 2023-10-18T20:26:59.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005858819f53eb77d2-0065303f93-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.891 DEBUG:boto:b'bofrnf-56a1000falsea2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:26:59.892 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:26:59.892 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:26:59.893 DEBUG:boto:path=/bofrnf-56/?prefix=a 2023-10-18T20:26:59.893 DEBUG:boto:auth_path=/bofrnf-56/?prefix=a 2023-10-18T20:26:59.893 DEBUG:boto:Method: GET 2023-10-18T20:26:59.894 DEBUG:boto:Path: /bofrnf-56/?prefix=a 2023-10-18T20:26:59.894 DEBUG:boto:Data: 2023-10-18T20:26:59.895 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.895 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.895 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.896 DEBUG:boto:Params: {} 2023-10-18T20:26:59.896 DEBUG:boto:Token: None 2023-10-18T20:26:59.897 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/ 2023-10-18T20:26:59.897 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oJbzXnEo7EpIuY4FKh11OWMHgGY= 2023-10-18T20:26:59.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oJbzXnEo7EpIuY4FKh11OWMHgGY=', 'Content-Length': '0'} 2023-10-18T20:26:59.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000774a09d1de6f0f4e-0065303f93-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.900 DEBUG:boto:b'bofrnf-56a1000falsea2023-10-18T20:24:22.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:26:59.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:26:59.901 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:26:59.902 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:26:59.902 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.903 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.903 DEBUG:boto:Method: GET 2023-10-18T20:26:59.904 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.904 DEBUG:boto:Data: 2023-10-18T20:26:59.905 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.905 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.905 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.906 DEBUG:boto:Params: {} 2023-10-18T20:26:59.906 DEBUG:boto:Token: None 2023-10-18T20:26:59.907 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.907 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YX84tsWg5AjTt0/jViPnNMvLKSw= 2023-10-18T20:26:59.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YX84tsWg5AjTt0/jViPnNMvLKSw=', 'Content-Length': '0'} 2023-10-18T20:26:59.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4864a34c6ce0d37-0065303f93-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.950 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:26:59.951 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:26:59.952 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.952 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.953 DEBUG:boto:Method: GET 2023-10-18T20:26:59.954 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.955 DEBUG:boto:Data: 2023-10-18T20:26:59.955 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.956 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.957 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.957 DEBUG:boto:Params: {} 2023-10-18T20:26:59.958 DEBUG:boto:Token: None 2023-10-18T20:26:59.959 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YX84tsWg5AjTt0/jViPnNMvLKSw= 2023-10-18T20:26:59.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YX84tsWg5AjTt0/jViPnNMvLKSw=', 'Content-Length': '0'} 2023-10-18T20:26:59.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29b8c82b9e171db-0065303f93-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.963 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:26:59.963 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:26:59.964 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.964 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.965 DEBUG:boto:Method: HEAD 2023-10-18T20:26:59.965 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.966 DEBUG:boto:Data: 2023-10-18T20:26:59.966 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.967 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.968 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.968 DEBUG:boto:Params: {} 2023-10-18T20:26:59.969 DEBUG:boto:Token: None 2023-10-18T20:26:59.970 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.971 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mtNxRbzq8eQ6ROV04/B9lzWw8UE= 2023-10-18T20:26:59.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mtNxRbzq8eQ6ROV04/B9lzWw8UE=', 'Content-Length': '0'} 2023-10-18T20:26:59.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e7efe1fccfb72f-0065303f93-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.975 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:26:59.975 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:26:59.975 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.976 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.976 DEBUG:boto:Method: HEAD 2023-10-18T20:26:59.977 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.977 DEBUG:boto:Data: 2023-10-18T20:26:59.977 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.978 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.978 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.979 DEBUG:boto:Params: {} 2023-10-18T20:26:59.979 DEBUG:boto:Token: None 2023-10-18T20:26:59.979 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.980 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mtNxRbzq8eQ6ROV04/B9lzWw8UE= 2023-10-18T20:26:59.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mtNxRbzq8eQ6ROV04/B9lzWw8UE=', 'Content-Length': '0'} 2023-10-18T20:26:59.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005036062754c556ca-0065303f93-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:26:59.983 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:26:59.984 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:26:59.984 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.985 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.985 DEBUG:boto:Method: GET 2023-10-18T20:26:59.985 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.986 DEBUG:boto:Data: 2023-10-18T20:26:59.986 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.987 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:26:59.987 DEBUG:boto:Port: 8001 2023-10-18T20:26:59.987 DEBUG:boto:Params: {} 2023-10-18T20:26:59.988 DEBUG:boto:Token: None 2023-10-18T20:26:59.988 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:26:59 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.989 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YX84tsWg5AjTt0/jViPnNMvLKSw= 2023-10-18T20:26:59.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:26:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YX84tsWg5AjTt0/jViPnNMvLKSw=', 'Content-Length': '0'} 2023-10-18T20:26:59.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004950266234d0901f-0065303f93-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:26:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:26:59.992 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:26:59.993 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:26:59.993 DEBUG:boto:path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.993 DEBUG:boto:auth_path=/bofrnf-56/b?versionId=null 2023-10-18T20:26:59.994 DEBUG:boto:Method: GET 2023-10-18T20:26:59.995 DEBUG:boto:Path: /bofrnf-56/b?versionId=null 2023-10-18T20:26:59.995 DEBUG:boto:Data: 2023-10-18T20:26:59.996 DEBUG:boto:Headers: {} 2023-10-18T20:26:59.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:26:59.997 DEBUG:boto:Port: 8000 2023-10-18T20:26:59.998 DEBUG:boto:Params: {} 2023-10-18T20:26:59.999 DEBUG:boto:Token: None 2023-10-18T20:27:00.000 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/b?versionId=null 2023-10-18T20:27:00.000 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9V0KQo9p6KHVVTUsZteeC/9V86I= 2023-10-18T20:27:00.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9V0KQo9p6KHVVTUsZteeC/9V86I=', 'Content-Length': '0'} 2023-10-18T20:27:00.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b05e0ddb1cc154-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.047 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.047 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.047 DEBUG:boto:path=/bofrnf-56/?prefix=b 2023-10-18T20:27:00.048 DEBUG:boto:auth_path=/bofrnf-56/?prefix=b 2023-10-18T20:27:00.048 DEBUG:boto:Method: GET 2023-10-18T20:27:00.049 DEBUG:boto:Path: /bofrnf-56/?prefix=b 2023-10-18T20:27:00.049 DEBUG:boto:Data: 2023-10-18T20:27:00.049 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.050 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.051 DEBUG:boto:Params: {} 2023-10-18T20:27:00.051 DEBUG:boto:Token: None 2023-10-18T20:27:00.052 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000441e433dad2934be-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.056 DEBUG:boto:b'bofrnf-56b1000falseb2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.057 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.058 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.059 DEBUG:boto:path=/bofrnf-56/?prefix=b 2023-10-18T20:27:00.060 DEBUG:boto:auth_path=/bofrnf-56/?prefix=b 2023-10-18T20:27:00.060 DEBUG:boto:Method: GET 2023-10-18T20:27:00.061 DEBUG:boto:Path: /bofrnf-56/?prefix=b 2023-10-18T20:27:00.062 DEBUG:boto:Data: 2023-10-18T20:27:00.063 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.063 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.064 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.065 DEBUG:boto:Params: {} 2023-10-18T20:27:00.066 DEBUG:boto:Token: None 2023-10-18T20:27:00.066 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.067 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000772fcc94e9832491-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.071 DEBUG:boto:b'bofrnf-56b1000falseb2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.071 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:27:00.072 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:27:00.073 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.074 DEBUG:boto:Path: /bofrnf-56/b 2023-10-18T20:27:00.074 DEBUG:boto:Data: 2023-10-18T20:27:00.075 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.076 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.076 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.077 DEBUG:boto:Params: {} 2023-10-18T20:27:00.078 DEBUG:boto:Token: None 2023-10-18T20:27:00.079 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/b 2023-10-18T20:27:00.079 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SvK46HWxv7cM3MVOKDOP0SO6BmA= 2023-10-18T20:27:00.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SvK46HWxv7cM3MVOKDOP0SO6BmA=', 'Content-Length': '0'} 2023-10-18T20:27:00.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006687ecf0318afec9-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.083 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:27:00.084 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:27:00.085 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.085 DEBUG:boto:Path: /bofrnf-56/b 2023-10-18T20:27:00.086 DEBUG:boto:Data: 2023-10-18T20:27:00.086 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.087 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.087 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.087 DEBUG:boto:Params: {} 2023-10-18T20:27:00.088 DEBUG:boto:Token: None 2023-10-18T20:27:00.088 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/b 2023-10-18T20:27:00.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SvK46HWxv7cM3MVOKDOP0SO6BmA= 2023-10-18T20:27:00.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SvK46HWxv7cM3MVOKDOP0SO6BmA=', 'Content-Length': '0'} 2023-10-18T20:27:00.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da29d39677609d45-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:27:00.092 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:27:00.092 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:27:00.093 DEBUG:boto:Method: GET 2023-10-18T20:27:00.093 DEBUG:boto:Path: /bofrnf-56/b 2023-10-18T20:27:00.093 DEBUG:boto:Data: 2023-10-18T20:27:00.094 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.094 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.095 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.095 DEBUG:boto:Params: {} 2023-10-18T20:27:00.095 DEBUG:boto:Token: None 2023-10-18T20:27:00.096 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/b 2023-10-18T20:27:00.096 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zJg/3cxbk2kNlauJWwxlzQVgTUU= 2023-10-18T20:27:00.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zJg/3cxbk2kNlauJWwxlzQVgTUU=', 'Content-Length': '0'} 2023-10-18T20:27:00.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f07b2b8e8afbc8c-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.142 DEBUG:boto:path=/bofrnf-56/b 2023-10-18T20:27:00.143 DEBUG:boto:auth_path=/bofrnf-56/b 2023-10-18T20:27:00.144 DEBUG:boto:Method: GET 2023-10-18T20:27:00.145 DEBUG:boto:Path: /bofrnf-56/b 2023-10-18T20:27:00.145 DEBUG:boto:Data: 2023-10-18T20:27:00.146 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.147 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.148 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.149 DEBUG:boto:Params: {} 2023-10-18T20:27:00.149 DEBUG:boto:Token: None 2023-10-18T20:27:00.150 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/b 2023-10-18T20:27:00.151 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zJg/3cxbk2kNlauJWwxlzQVgTUU= 2023-10-18T20:27:00.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zJg/3cxbk2kNlauJWwxlzQVgTUU=', 'Content-Length': '0'} 2023-10-18T20:27:00.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021c6e53c0acc9d49-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.156 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.156 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.157 DEBUG:boto:path=/bofrnf-56/?prefix=b 2023-10-18T20:27:00.157 DEBUG:boto:auth_path=/bofrnf-56/?prefix=b 2023-10-18T20:27:00.158 DEBUG:boto:Method: GET 2023-10-18T20:27:00.158 DEBUG:boto:Path: /bofrnf-56/?prefix=b 2023-10-18T20:27:00.159 DEBUG:boto:Data: 2023-10-18T20:27:00.160 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.161 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.161 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.162 DEBUG:boto:Params: {} 2023-10-18T20:27:00.163 DEBUG:boto:Token: None 2023-10-18T20:27:00.163 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb2a38b0b53ac9ee-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.167 DEBUG:boto:b'bofrnf-56b1000falseb2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.168 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.168 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.169 DEBUG:boto:path=/bofrnf-56/?prefix=b 2023-10-18T20:27:00.169 DEBUG:boto:auth_path=/bofrnf-56/?prefix=b 2023-10-18T20:27:00.170 DEBUG:boto:Method: GET 2023-10-18T20:27:00.171 DEBUG:boto:Path: /bofrnf-56/?prefix=b 2023-10-18T20:27:00.172 DEBUG:boto:Data: 2023-10-18T20:27:00.172 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.173 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.174 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.175 DEBUG:boto:Params: {} 2023-10-18T20:27:00.176 DEBUG:boto:Token: None 2023-10-18T20:27:00.176 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.177 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b09b0ad06375f74-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.222 DEBUG:boto:b'bofrnf-56b1000falseb2023-10-18T20:24:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:27:00.224 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.224 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.225 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.225 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.225 DEBUG:boto:Method: GET 2023-10-18T20:27:00.226 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.227 DEBUG:boto:Data: 2023-10-18T20:27:00.227 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.227 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.228 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.228 DEBUG:boto:Params: {} 2023-10-18T20:27:00.229 DEBUG:boto:Token: None 2023-10-18T20:27:00.229 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.230 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jwx4G2ZwEHa1Q+w49f61aURQxx4= 2023-10-18T20:27:00.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jwx4G2ZwEHa1Q+w49f61aURQxx4=', 'Content-Length': '0'} 2023-10-18T20:27:00.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c3941f1554cf917-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.233 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.234 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.234 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.235 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.235 DEBUG:boto:Method: GET 2023-10-18T20:27:00.235 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.236 DEBUG:boto:Data: 2023-10-18T20:27:00.236 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.237 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.237 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.238 DEBUG:boto:Params: {} 2023-10-18T20:27:00.238 DEBUG:boto:Token: None 2023-10-18T20:27:00.239 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.239 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jwx4G2ZwEHa1Q+w49f61aURQxx4= 2023-10-18T20:27:00.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jwx4G2ZwEHa1Q+w49f61aURQxx4=', 'Content-Length': '0'} 2023-10-18T20:27:00.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a74316ec2e221cf9-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.242 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.243 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.243 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.244 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.244 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.245 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.245 DEBUG:boto:Data: 2023-10-18T20:27:00.245 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.246 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.246 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.247 DEBUG:boto:Params: {} 2023-10-18T20:27:00.247 DEBUG:boto:Token: None 2023-10-18T20:27:00.248 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.248 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8zwZ8x6DZ8JBJ3vetCUcYWYHeo4= 2023-10-18T20:27:00.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8zwZ8x6DZ8JBJ3vetCUcYWYHeo4=', 'Content-Length': '0'} 2023-10-18T20:27:00.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b20e4550c6b7e6-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.251 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.251 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.252 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.252 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.253 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.253 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.253 DEBUG:boto:Data: 2023-10-18T20:27:00.254 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.254 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.255 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.255 DEBUG:boto:Params: {} 2023-10-18T20:27:00.256 DEBUG:boto:Token: None 2023-10-18T20:27:00.256 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.257 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8zwZ8x6DZ8JBJ3vetCUcYWYHeo4= 2023-10-18T20:27:00.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8zwZ8x6DZ8JBJ3vetCUcYWYHeo4=', 'Content-Length': '0'} 2023-10-18T20:27:00.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d172a15e33c1de8-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:27:00.260 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.261 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.261 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.261 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.262 DEBUG:boto:Method: GET 2023-10-18T20:27:00.262 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.263 DEBUG:boto:Data: 2023-10-18T20:27:00.263 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.264 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.264 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.265 DEBUG:boto:Params: {} 2023-10-18T20:27:00.265 DEBUG:boto:Token: None 2023-10-18T20:27:00.266 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.266 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jwx4G2ZwEHa1Q+w49f61aURQxx4= 2023-10-18T20:27:00.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jwx4G2ZwEHa1Q+w49f61aURQxx4=', 'Content-Length': '0'} 2023-10-18T20:27:00.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000785e3ff505fa9f87-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.269 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.270 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.270 DEBUG:boto:path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.271 DEBUG:boto:auth_path=/bofrnf-56/c?versionId=null 2023-10-18T20:27:00.271 DEBUG:boto:Method: GET 2023-10-18T20:27:00.272 DEBUG:boto:Path: /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.272 DEBUG:boto:Data: 2023-10-18T20:27:00.272 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.273 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.273 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.274 DEBUG:boto:Params: {} 2023-10-18T20:27:00.274 DEBUG:boto:Token: None 2023-10-18T20:27:00.275 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c?versionId=null 2023-10-18T20:27:00.275 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jwx4G2ZwEHa1Q+w49f61aURQxx4= 2023-10-18T20:27:00.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jwx4G2ZwEHa1Q+w49f61aURQxx4=', 'Content-Length': '0'} 2023-10-18T20:27:00.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ddecdf845d2c1b-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.279 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.279 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.279 DEBUG:boto:path=/bofrnf-56/?prefix=c 2023-10-18T20:27:00.280 DEBUG:boto:auth_path=/bofrnf-56/?prefix=c 2023-10-18T20:27:00.280 DEBUG:boto:Method: GET 2023-10-18T20:27:00.281 DEBUG:boto:Path: /bofrnf-56/?prefix=c 2023-10-18T20:27:00.281 DEBUG:boto:Data: 2023-10-18T20:27:00.282 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.282 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.283 DEBUG:boto:Params: {} 2023-10-18T20:27:00.283 DEBUG:boto:Token: None 2023-10-18T20:27:00.284 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6241b9c00948d4b-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.289 DEBUG:boto:b'bofrnf-56c1000falsec2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.289 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.290 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.290 DEBUG:boto:path=/bofrnf-56/?prefix=c 2023-10-18T20:27:00.291 DEBUG:boto:auth_path=/bofrnf-56/?prefix=c 2023-10-18T20:27:00.291 DEBUG:boto:Method: GET 2023-10-18T20:27:00.291 DEBUG:boto:Path: /bofrnf-56/?prefix=c 2023-10-18T20:27:00.292 DEBUG:boto:Data: 2023-10-18T20:27:00.292 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.293 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.293 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.294 DEBUG:boto:Params: {} 2023-10-18T20:27:00.294 DEBUG:boto:Token: None 2023-10-18T20:27:00.295 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.295 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224f2336e9e5b7bd-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.298 DEBUG:boto:b'bofrnf-56c1000falsec2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.299 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.299 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.300 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.300 DEBUG:boto:Path: /bofrnf-56/c 2023-10-18T20:27:00.300 DEBUG:boto:Data: 2023-10-18T20:27:00.301 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.301 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.302 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.302 DEBUG:boto:Params: {} 2023-10-18T20:27:00.303 DEBUG:boto:Token: None 2023-10-18T20:27:00.303 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c 2023-10-18T20:27:00.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BCxbJpKiLruDLW37/1/ycq8AUHc= 2023-10-18T20:27:00.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BCxbJpKiLruDLW37/1/ycq8AUHc=', 'Content-Length': '0'} 2023-10-18T20:27:00.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3f86f14a6aa5f37-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.307 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.307 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.307 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.308 DEBUG:boto:Path: /bofrnf-56/c 2023-10-18T20:27:00.308 DEBUG:boto:Data: 2023-10-18T20:27:00.309 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.309 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.310 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.310 DEBUG:boto:Params: {} 2023-10-18T20:27:00.311 DEBUG:boto:Token: None 2023-10-18T20:27:00.311 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c 2023-10-18T20:27:00.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BCxbJpKiLruDLW37/1/ycq8AUHc= 2023-10-18T20:27:00.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BCxbJpKiLruDLW37/1/ycq8AUHc=', 'Content-Length': '0'} 2023-10-18T20:27:00.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afffa23b2febd7c5-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:27:00.315 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.315 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.316 DEBUG:boto:Method: GET 2023-10-18T20:27:00.316 DEBUG:boto:Path: /bofrnf-56/c 2023-10-18T20:27:00.317 DEBUG:boto:Data: 2023-10-18T20:27:00.317 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.318 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.318 DEBUG:boto:Params: {} 2023-10-18T20:27:00.319 DEBUG:boto:Token: None 2023-10-18T20:27:00.319 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c 2023-10-18T20:27:00.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2OppTLzx+MUuIajJDdHe29St+xE= 2023-10-18T20:27:00.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2OppTLzx+MUuIajJDdHe29St+xE=', 'Content-Length': '0'} 2023-10-18T20:27:00.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d8138ac275c0ddf-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.366 DEBUG:boto:path=/bofrnf-56/c 2023-10-18T20:27:00.367 DEBUG:boto:auth_path=/bofrnf-56/c 2023-10-18T20:27:00.367 DEBUG:boto:Method: GET 2023-10-18T20:27:00.368 DEBUG:boto:Path: /bofrnf-56/c 2023-10-18T20:27:00.368 DEBUG:boto:Data: 2023-10-18T20:27:00.369 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.369 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.369 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.370 DEBUG:boto:Params: {} 2023-10-18T20:27:00.371 DEBUG:boto:Token: None 2023-10-18T20:27:00.371 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/c 2023-10-18T20:27:00.372 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2OppTLzx+MUuIajJDdHe29St+xE= 2023-10-18T20:27:00.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2OppTLzx+MUuIajJDdHe29St+xE=', 'Content-Length': '0'} 2023-10-18T20:27:00.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000440659266975835a-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.377 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.377 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.378 DEBUG:boto:path=/bofrnf-56/?prefix=c 2023-10-18T20:27:00.378 DEBUG:boto:auth_path=/bofrnf-56/?prefix=c 2023-10-18T20:27:00.379 DEBUG:boto:Method: GET 2023-10-18T20:27:00.379 DEBUG:boto:Path: /bofrnf-56/?prefix=c 2023-10-18T20:27:00.379 DEBUG:boto:Data: 2023-10-18T20:27:00.380 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.381 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.381 DEBUG:boto:Params: {} 2023-10-18T20:27:00.381 DEBUG:boto:Token: None 2023-10-18T20:27:00.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000842c0a22bdece896-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.387 DEBUG:boto:b'bofrnf-56c1000falsec2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.389 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.389 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.390 DEBUG:boto:path=/bofrnf-56/?prefix=c 2023-10-18T20:27:00.391 DEBUG:boto:auth_path=/bofrnf-56/?prefix=c 2023-10-18T20:27:00.391 DEBUG:boto:Method: GET 2023-10-18T20:27:00.392 DEBUG:boto:Path: /bofrnf-56/?prefix=c 2023-10-18T20:27:00.393 DEBUG:boto:Data: 2023-10-18T20:27:00.394 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.394 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.395 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.396 DEBUG:boto:Params: {} 2023-10-18T20:27:00.397 DEBUG:boto:Token: None 2023-10-18T20:27:00.397 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.398 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2dfd6477f5deff5-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.402 DEBUG:boto:b'bofrnf-56c1000falsec2023-10-18T20:24:22.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:27:00.404 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.404 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.405 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.406 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.407 DEBUG:boto:Method: GET 2023-10-18T20:27:00.407 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.408 DEBUG:boto:Data: 2023-10-18T20:27:00.409 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.410 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.410 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.411 DEBUG:boto:Params: {} 2023-10-18T20:27:00.412 DEBUG:boto:Token: None 2023-10-18T20:27:00.413 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.414 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JkJphRn/nbjdZ5YTusiytHyqvq8= 2023-10-18T20:27:00.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JkJphRn/nbjdZ5YTusiytHyqvq8=', 'Content-Length': '0'} 2023-10-18T20:27:00.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f1c0a33f75442a6-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.418 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.419 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.419 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.420 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.421 DEBUG:boto:Method: GET 2023-10-18T20:27:00.421 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.421 DEBUG:boto:Data: 2023-10-18T20:27:00.422 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.422 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.423 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.423 DEBUG:boto:Params: {} 2023-10-18T20:27:00.424 DEBUG:boto:Token: None 2023-10-18T20:27:00.424 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.425 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JkJphRn/nbjdZ5YTusiytHyqvq8= 2023-10-18T20:27:00.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JkJphRn/nbjdZ5YTusiytHyqvq8=', 'Content-Length': '0'} 2023-10-18T20:27:00.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b84b9a63f69f6e-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.428 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.428 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.429 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.429 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.430 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.430 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.431 DEBUG:boto:Data: 2023-10-18T20:27:00.431 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.432 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.432 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.432 DEBUG:boto:Params: {} 2023-10-18T20:27:00.433 DEBUG:boto:Token: None 2023-10-18T20:27:00.433 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.434 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xFCZVJEL7ST1bTUU7I8p3zKjxuw= 2023-10-18T20:27:00.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xFCZVJEL7ST1bTUU7I8p3zKjxuw=', 'Content-Length': '0'} 2023-10-18T20:27:00.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b67742fa45c25443-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.437 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.437 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.438 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.438 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.438 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.439 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.439 DEBUG:boto:Data: 2023-10-18T20:27:00.440 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.440 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.440 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.441 DEBUG:boto:Params: {} 2023-10-18T20:27:00.441 DEBUG:boto:Token: None 2023-10-18T20:27:00.442 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.442 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xFCZVJEL7ST1bTUU7I8p3zKjxuw= 2023-10-18T20:27:00.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xFCZVJEL7ST1bTUU7I8p3zKjxuw=', 'Content-Length': '0'} 2023-10-18T20:27:00.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6e56edc06100c82-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:27:00.445 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.446 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.446 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.447 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.447 DEBUG:boto:Method: GET 2023-10-18T20:27:00.447 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.448 DEBUG:boto:Data: 2023-10-18T20:27:00.448 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.448 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.449 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.449 DEBUG:boto:Params: {} 2023-10-18T20:27:00.450 DEBUG:boto:Token: None 2023-10-18T20:27:00.450 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.451 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JkJphRn/nbjdZ5YTusiytHyqvq8= 2023-10-18T20:27:00.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JkJphRn/nbjdZ5YTusiytHyqvq8=', 'Content-Length': '0'} 2023-10-18T20:27:00.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab849cd67099d6a0-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.453 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.454 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.454 DEBUG:boto:path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.455 DEBUG:boto:auth_path=/bofrnf-56/d?versionId=null 2023-10-18T20:27:00.455 DEBUG:boto:Method: GET 2023-10-18T20:27:00.456 DEBUG:boto:Path: /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.456 DEBUG:boto:Data: 2023-10-18T20:27:00.456 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.457 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.458 DEBUG:boto:Params: {} 2023-10-18T20:27:00.458 DEBUG:boto:Token: None 2023-10-18T20:27:00.459 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d?versionId=null 2023-10-18T20:27:00.459 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JkJphRn/nbjdZ5YTusiytHyqvq8= 2023-10-18T20:27:00.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JkJphRn/nbjdZ5YTusiytHyqvq8=', 'Content-Length': '0'} 2023-10-18T20:27:00.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed7687bd4bbcfafd-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.506 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.507 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.507 DEBUG:boto:path=/bofrnf-56/?prefix=d 2023-10-18T20:27:00.508 DEBUG:boto:auth_path=/bofrnf-56/?prefix=d 2023-10-18T20:27:00.508 DEBUG:boto:Method: GET 2023-10-18T20:27:00.509 DEBUG:boto:Path: /bofrnf-56/?prefix=d 2023-10-18T20:27:00.509 DEBUG:boto:Data: 2023-10-18T20:27:00.510 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.510 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.510 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.511 DEBUG:boto:Params: {} 2023-10-18T20:27:00.511 DEBUG:boto:Token: None 2023-10-18T20:27:00.512 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.513 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003323c1921466f27-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.517 DEBUG:boto:b'bofrnf-56d1000falsed2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.517 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.518 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.518 DEBUG:boto:path=/bofrnf-56/?prefix=d 2023-10-18T20:27:00.519 DEBUG:boto:auth_path=/bofrnf-56/?prefix=d 2023-10-18T20:27:00.519 DEBUG:boto:Method: GET 2023-10-18T20:27:00.519 DEBUG:boto:Path: /bofrnf-56/?prefix=d 2023-10-18T20:27:00.520 DEBUG:boto:Data: 2023-10-18T20:27:00.520 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.521 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.521 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.521 DEBUG:boto:Params: {} 2023-10-18T20:27:00.522 DEBUG:boto:Token: None 2023-10-18T20:27:00.522 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.523 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a871897b4a2e6885-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.526 DEBUG:boto:b'bofrnf-56d1000falsed2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.527 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.527 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.528 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.528 DEBUG:boto:Path: /bofrnf-56/d 2023-10-18T20:27:00.529 DEBUG:boto:Data: 2023-10-18T20:27:00.529 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.529 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.530 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.530 DEBUG:boto:Params: {} 2023-10-18T20:27:00.531 DEBUG:boto:Token: None 2023-10-18T20:27:00.531 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d 2023-10-18T20:27:00.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0T3FTn6yoXc49kY8JKGd5+25sO8= 2023-10-18T20:27:00.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0T3FTn6yoXc49kY8JKGd5+25sO8=', 'Content-Length': '0'} 2023-10-18T20:27:00.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f45daf4c13cc5cc2-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.535 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.535 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.535 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.536 DEBUG:boto:Path: /bofrnf-56/d 2023-10-18T20:27:00.536 DEBUG:boto:Data: 2023-10-18T20:27:00.537 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.537 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.538 DEBUG:boto:Params: {} 2023-10-18T20:27:00.538 DEBUG:boto:Token: None 2023-10-18T20:27:00.541 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d 2023-10-18T20:27:00.542 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0T3FTn6yoXc49kY8JKGd5+25sO8= 2023-10-18T20:27:00.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0T3FTn6yoXc49kY8JKGd5+25sO8=', 'Content-Length': '0'} 2023-10-18T20:27:00.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5a124f87695178-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:27:00.545 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.546 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.546 DEBUG:boto:Method: GET 2023-10-18T20:27:00.547 DEBUG:boto:Path: /bofrnf-56/d 2023-10-18T20:27:00.547 DEBUG:boto:Data: 2023-10-18T20:27:00.548 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.548 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.549 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.549 DEBUG:boto:Params: {} 2023-10-18T20:27:00.549 DEBUG:boto:Token: None 2023-10-18T20:27:00.550 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d 2023-10-18T20:27:00.550 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WWSsWq0bEX6rppThSaVHnaGz+Hs= 2023-10-18T20:27:00.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WWSsWq0bEX6rppThSaVHnaGz+Hs=', 'Content-Length': '0'} 2023-10-18T20:27:00.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191f4df1c7efb95f-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.594 DEBUG:boto:path=/bofrnf-56/d 2023-10-18T20:27:00.595 DEBUG:boto:auth_path=/bofrnf-56/d 2023-10-18T20:27:00.595 DEBUG:boto:Method: GET 2023-10-18T20:27:00.596 DEBUG:boto:Path: /bofrnf-56/d 2023-10-18T20:27:00.596 DEBUG:boto:Data: 2023-10-18T20:27:00.596 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.597 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.598 DEBUG:boto:Params: {} 2023-10-18T20:27:00.598 DEBUG:boto:Token: None 2023-10-18T20:27:00.599 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/d 2023-10-18T20:27:00.599 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WWSsWq0bEX6rppThSaVHnaGz+Hs= 2023-10-18T20:27:00.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WWSsWq0bEX6rppThSaVHnaGz+Hs=', 'Content-Length': '0'} 2023-10-18T20:27:00.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5dcfa5905ffa6a4-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.602 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.603 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.603 DEBUG:boto:path=/bofrnf-56/?prefix=d 2023-10-18T20:27:00.604 DEBUG:boto:auth_path=/bofrnf-56/?prefix=d 2023-10-18T20:27:00.604 DEBUG:boto:Method: GET 2023-10-18T20:27:00.604 DEBUG:boto:Path: /bofrnf-56/?prefix=d 2023-10-18T20:27:00.605 DEBUG:boto:Data: 2023-10-18T20:27:00.605 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.606 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.606 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.606 DEBUG:boto:Params: {} 2023-10-18T20:27:00.607 DEBUG:boto:Token: None 2023-10-18T20:27:00.607 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056da9701f6770c43-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.612 DEBUG:boto:b'bofrnf-56d1000falsed2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.612 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.613 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.613 DEBUG:boto:path=/bofrnf-56/?prefix=d 2023-10-18T20:27:00.614 DEBUG:boto:auth_path=/bofrnf-56/?prefix=d 2023-10-18T20:27:00.614 DEBUG:boto:Method: GET 2023-10-18T20:27:00.614 DEBUG:boto:Path: /bofrnf-56/?prefix=d 2023-10-18T20:27:00.615 DEBUG:boto:Data: 2023-10-18T20:27:00.615 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.616 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.616 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.616 DEBUG:boto:Params: {} 2023-10-18T20:27:00.617 DEBUG:boto:Token: None 2023-10-18T20:27:00.617 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.618 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643b4219cbf5d6ec-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.621 DEBUG:boto:b'bofrnf-56d1000falsed2023-10-18T20:24:22.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:27:00.622 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.622 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.622 DEBUG:boto:path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.623 DEBUG:boto:auth_path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.623 DEBUG:boto:Method: GET 2023-10-18T20:27:00.624 DEBUG:boto:Path: /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.624 DEBUG:boto:Data: 2023-10-18T20:27:00.624 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.625 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.626 DEBUG:boto:Params: {} 2023-10-18T20:27:00.626 DEBUG:boto:Token: None 2023-10-18T20:27:00.626 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.627 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MLgfwlYpLPmYoC4dwjN4gshkCzw= 2023-10-18T20:27:00.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MLgfwlYpLPmYoC4dwjN4gshkCzw=', 'Content-Length': '0'} 2023-10-18T20:27:00.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa93006687ef837-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.630 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.631 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.631 DEBUG:boto:path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.631 DEBUG:boto:auth_path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.632 DEBUG:boto:Method: GET 2023-10-18T20:27:00.632 DEBUG:boto:Path: /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.632 DEBUG:boto:Data: 2023-10-18T20:27:00.633 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.633 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.634 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.634 DEBUG:boto:Params: {} 2023-10-18T20:27:00.635 DEBUG:boto:Token: None 2023-10-18T20:27:00.635 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.635 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MLgfwlYpLPmYoC4dwjN4gshkCzw= 2023-10-18T20:27:00.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MLgfwlYpLPmYoC4dwjN4gshkCzw=', 'Content-Length': '0'} 2023-10-18T20:27:00.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fec4a718465dae0b-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.639 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.639 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.639 DEBUG:boto:path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.640 DEBUG:boto:auth_path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.640 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.641 DEBUG:boto:Path: /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.641 DEBUG:boto:Data: 2023-10-18T20:27:00.641 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.642 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.642 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.642 DEBUG:boto:Params: {} 2023-10-18T20:27:00.643 DEBUG:boto:Token: None 2023-10-18T20:27:00.643 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.644 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zkaqj8ehqKlRR5CWchWa13bLPeo= 2023-10-18T20:27:00.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zkaqj8ehqKlRR5CWchWa13bLPeo=', 'Content-Length': '0'} 2023-10-18T20:27:00.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee4356e95515b8d-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.647 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.648 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.648 DEBUG:boto:path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.648 DEBUG:boto:auth_path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.649 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.649 DEBUG:boto:Path: /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.650 DEBUG:boto:Data: 2023-10-18T20:27:00.650 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.651 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.651 DEBUG:boto:Params: {} 2023-10-18T20:27:00.652 DEBUG:boto:Token: None 2023-10-18T20:27:00.652 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zkaqj8ehqKlRR5CWchWa13bLPeo= 2023-10-18T20:27:00.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zkaqj8ehqKlRR5CWchWa13bLPeo=', 'Content-Length': '0'} 2023-10-18T20:27:00.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb48ef1b7572fdbe-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:27:00.656 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.656 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.657 DEBUG:boto:path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.657 DEBUG:boto:auth_path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.658 DEBUG:boto:Method: GET 2023-10-18T20:27:00.658 DEBUG:boto:Path: /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.659 DEBUG:boto:Data: 2023-10-18T20:27:00.659 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.660 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.660 DEBUG:boto:Params: {} 2023-10-18T20:27:00.661 DEBUG:boto:Token: None 2023-10-18T20:27:00.661 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MLgfwlYpLPmYoC4dwjN4gshkCzw= 2023-10-18T20:27:00.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MLgfwlYpLPmYoC4dwjN4gshkCzw=', 'Content-Length': '0'} 2023-10-18T20:27:00.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f053a540c5c1d070-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.706 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.707 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.707 DEBUG:boto:path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.707 DEBUG:boto:auth_path=/bofrnf-56/e?versionId=null 2023-10-18T20:27:00.708 DEBUG:boto:Method: GET 2023-10-18T20:27:00.708 DEBUG:boto:Path: /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.709 DEBUG:boto:Data: 2023-10-18T20:27:00.709 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.710 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.710 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.711 DEBUG:boto:Params: {} 2023-10-18T20:27:00.711 DEBUG:boto:Token: None 2023-10-18T20:27:00.711 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e?versionId=null 2023-10-18T20:27:00.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MLgfwlYpLPmYoC4dwjN4gshkCzw= 2023-10-18T20:27:00.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MLgfwlYpLPmYoC4dwjN4gshkCzw=', 'Content-Length': '0'} 2023-10-18T20:27:00.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a899f88b86cac7a-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.715 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.716 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.716 DEBUG:boto:path=/bofrnf-56/?prefix=e 2023-10-18T20:27:00.716 DEBUG:boto:auth_path=/bofrnf-56/?prefix=e 2023-10-18T20:27:00.717 DEBUG:boto:Method: GET 2023-10-18T20:27:00.717 DEBUG:boto:Path: /bofrnf-56/?prefix=e 2023-10-18T20:27:00.718 DEBUG:boto:Data: 2023-10-18T20:27:00.718 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.719 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.719 DEBUG:boto:Params: {} 2023-10-18T20:27:00.720 DEBUG:boto:Token: None 2023-10-18T20:27:00.720 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af361013cfdda2b1-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.724 DEBUG:boto:b'bofrnf-56e1000falsee2023-10-18T20:24:59.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.725 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.725 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.726 DEBUG:boto:path=/bofrnf-56/?prefix=e 2023-10-18T20:27:00.726 DEBUG:boto:auth_path=/bofrnf-56/?prefix=e 2023-10-18T20:27:00.726 DEBUG:boto:Method: GET 2023-10-18T20:27:00.727 DEBUG:boto:Path: /bofrnf-56/?prefix=e 2023-10-18T20:27:00.727 DEBUG:boto:Data: 2023-10-18T20:27:00.728 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.728 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.729 DEBUG:boto:Params: {} 2023-10-18T20:27:00.729 DEBUG:boto:Token: None 2023-10-18T20:27:00.730 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.730 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3f2f060953e4749-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.733 DEBUG:boto:b'bofrnf-56e1000falsee2023-10-18T20:24:59.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.734 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.734 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.735 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.735 DEBUG:boto:Path: /bofrnf-56/e 2023-10-18T20:27:00.736 DEBUG:boto:Data: 2023-10-18T20:27:00.736 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.736 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.737 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.737 DEBUG:boto:Params: {} 2023-10-18T20:27:00.738 DEBUG:boto:Token: None 2023-10-18T20:27:00.738 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e 2023-10-18T20:27:00.739 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1KKbosebQmSoPrc6uoaSspO9q6w= 2023-10-18T20:27:00.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1KKbosebQmSoPrc6uoaSspO9q6w=', 'Content-Length': '0'} 2023-10-18T20:27:00.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa8ef5f795361d1-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.742 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.742 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.742 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.743 DEBUG:boto:Path: /bofrnf-56/e 2023-10-18T20:27:00.743 DEBUG:boto:Data: 2023-10-18T20:27:00.744 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.744 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.744 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.745 DEBUG:boto:Params: {} 2023-10-18T20:27:00.745 DEBUG:boto:Token: None 2023-10-18T20:27:00.746 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e 2023-10-18T20:27:00.746 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1KKbosebQmSoPrc6uoaSspO9q6w= 2023-10-18T20:27:00.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1KKbosebQmSoPrc6uoaSspO9q6w=', 'Content-Length': '0'} 2023-10-18T20:27:00.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e8ab13ccd08242-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:27:00.750 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.750 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.750 DEBUG:boto:Method: GET 2023-10-18T20:27:00.751 DEBUG:boto:Path: /bofrnf-56/e 2023-10-18T20:27:00.751 DEBUG:boto:Data: 2023-10-18T20:27:00.752 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.752 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.753 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.753 DEBUG:boto:Params: {} 2023-10-18T20:27:00.753 DEBUG:boto:Token: None 2023-10-18T20:27:00.754 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e 2023-10-18T20:27:00.754 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aNxSsFLyXwpp/EynaTWIXO0jfsI= 2023-10-18T20:27:00.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aNxSsFLyXwpp/EynaTWIXO0jfsI=', 'Content-Length': '0'} 2023-10-18T20:27:00.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d31c1c74e8b381da-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.757 DEBUG:boto:path=/bofrnf-56/e 2023-10-18T20:27:00.758 DEBUG:boto:auth_path=/bofrnf-56/e 2023-10-18T20:27:00.758 DEBUG:boto:Method: GET 2023-10-18T20:27:00.759 DEBUG:boto:Path: /bofrnf-56/e 2023-10-18T20:27:00.759 DEBUG:boto:Data: 2023-10-18T20:27:00.760 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.760 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.760 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.761 DEBUG:boto:Params: {} 2023-10-18T20:27:00.761 DEBUG:boto:Token: None 2023-10-18T20:27:00.762 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/e 2023-10-18T20:27:00.762 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aNxSsFLyXwpp/EynaTWIXO0jfsI= 2023-10-18T20:27:00.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aNxSsFLyXwpp/EynaTWIXO0jfsI=', 'Content-Length': '0'} 2023-10-18T20:27:00.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3de6563ac2dee89-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.765 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.765 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.766 DEBUG:boto:path=/bofrnf-56/?prefix=e 2023-10-18T20:27:00.766 DEBUG:boto:auth_path=/bofrnf-56/?prefix=e 2023-10-18T20:27:00.767 DEBUG:boto:Method: GET 2023-10-18T20:27:00.767 DEBUG:boto:Path: /bofrnf-56/?prefix=e 2023-10-18T20:27:00.768 DEBUG:boto:Data: 2023-10-18T20:27:00.768 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.768 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.769 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.769 DEBUG:boto:Params: {} 2023-10-18T20:27:00.770 DEBUG:boto:Token: None 2023-10-18T20:27:00.770 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.771 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034a3b168fb1abca4-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.774 DEBUG:boto:b'bofrnf-56e1000falsee2023-10-18T20:24:59.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.775 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.775 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.776 DEBUG:boto:path=/bofrnf-56/?prefix=e 2023-10-18T20:27:00.776 DEBUG:boto:auth_path=/bofrnf-56/?prefix=e 2023-10-18T20:27:00.777 DEBUG:boto:Method: GET 2023-10-18T20:27:00.777 DEBUG:boto:Path: /bofrnf-56/?prefix=e 2023-10-18T20:27:00.777 DEBUG:boto:Data: 2023-10-18T20:27:00.778 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.778 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.779 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.779 DEBUG:boto:Params: {} 2023-10-18T20:27:00.779 DEBUG:boto:Token: None 2023-10-18T20:27:00.780 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8121690d6ca3772-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.783 DEBUG:boto:b'bofrnf-56e1000falsee2023-10-18T20:24:59.322Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:27:00.784 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.785 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.785 DEBUG:boto:path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.786 DEBUG:boto:auth_path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.786 DEBUG:boto:Method: GET 2023-10-18T20:27:00.786 DEBUG:boto:Path: /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.787 DEBUG:boto:Data: 2023-10-18T20:27:00.787 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.788 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.788 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.789 DEBUG:boto:Params: {} 2023-10-18T20:27:00.789 DEBUG:boto:Token: None 2023-10-18T20:27:00.789 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.790 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DtcG+jn8EboyLEFpq2CPgz3n/BU= 2023-10-18T20:27:00.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DtcG+jn8EboyLEFpq2CPgz3n/BU=', 'Content-Length': '0'} 2023-10-18T20:27:00.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f8b6231718c2600-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.793 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.794 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.794 DEBUG:boto:path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.795 DEBUG:boto:auth_path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.795 DEBUG:boto:Method: GET 2023-10-18T20:27:00.795 DEBUG:boto:Path: /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.796 DEBUG:boto:Data: 2023-10-18T20:27:00.796 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.797 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.797 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.797 DEBUG:boto:Params: {} 2023-10-18T20:27:00.798 DEBUG:boto:Token: None 2023-10-18T20:27:00.798 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DtcG+jn8EboyLEFpq2CPgz3n/BU= 2023-10-18T20:27:00.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DtcG+jn8EboyLEFpq2CPgz3n/BU=', 'Content-Length': '0'} 2023-10-18T20:27:00.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c775e79efdb74c4-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.842 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.843 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.843 DEBUG:boto:path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.844 DEBUG:boto:auth_path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.844 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.844 DEBUG:boto:Path: /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.845 DEBUG:boto:Data: 2023-10-18T20:27:00.845 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.846 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.846 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.846 DEBUG:boto:Params: {} 2023-10-18T20:27:00.847 DEBUG:boto:Token: None 2023-10-18T20:27:00.847 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3j6HLy82mYTvP1FTbyo2gAb+ilw= 2023-10-18T20:27:00.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3j6HLy82mYTvP1FTbyo2gAb+ilw=', 'Content-Length': '0'} 2023-10-18T20:27:00.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dc4aedf00576c1d-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.851 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.851 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.851 DEBUG:boto:path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.852 DEBUG:boto:auth_path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.852 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.853 DEBUG:boto:Path: /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.853 DEBUG:boto:Data: 2023-10-18T20:27:00.853 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.854 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.854 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.855 DEBUG:boto:Params: {} 2023-10-18T20:27:00.855 DEBUG:boto:Token: None 2023-10-18T20:27:00.856 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.856 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3j6HLy82mYTvP1FTbyo2gAb+ilw= 2023-10-18T20:27:00.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3j6HLy82mYTvP1FTbyo2gAb+ilw=', 'Content-Length': '0'} 2023-10-18T20:27:00.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f4ca936a6c01e98-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:27:00.859 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.860 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.860 DEBUG:boto:path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.861 DEBUG:boto:auth_path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.861 DEBUG:boto:Method: GET 2023-10-18T20:27:00.862 DEBUG:boto:Path: /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.862 DEBUG:boto:Data: 2023-10-18T20:27:00.862 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.863 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.863 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.864 DEBUG:boto:Params: {} 2023-10-18T20:27:00.864 DEBUG:boto:Token: None 2023-10-18T20:27:00.865 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.865 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DtcG+jn8EboyLEFpq2CPgz3n/BU= 2023-10-18T20:27:00.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DtcG+jn8EboyLEFpq2CPgz3n/BU=', 'Content-Length': '0'} 2023-10-18T20:27:00.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012be7ca964db1a5f-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.869 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.869 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.869 DEBUG:boto:path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.870 DEBUG:boto:auth_path=/bofrnf-56/f?versionId=null 2023-10-18T20:27:00.870 DEBUG:boto:Method: GET 2023-10-18T20:27:00.871 DEBUG:boto:Path: /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.871 DEBUG:boto:Data: 2023-10-18T20:27:00.871 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.872 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.872 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.873 DEBUG:boto:Params: {} 2023-10-18T20:27:00.873 DEBUG:boto:Token: None 2023-10-18T20:27:00.874 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f?versionId=null 2023-10-18T20:27:00.874 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DtcG+jn8EboyLEFpq2CPgz3n/BU= 2023-10-18T20:27:00.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DtcG+jn8EboyLEFpq2CPgz3n/BU=', 'Content-Length': '0'} 2023-10-18T20:27:00.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8feb51b304fe556-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.877 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.878 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.878 DEBUG:boto:path=/bofrnf-56/?prefix=f 2023-10-18T20:27:00.879 DEBUG:boto:auth_path=/bofrnf-56/?prefix=f 2023-10-18T20:27:00.879 DEBUG:boto:Method: GET 2023-10-18T20:27:00.880 DEBUG:boto:Path: /bofrnf-56/?prefix=f 2023-10-18T20:27:00.880 DEBUG:boto:Data: 2023-10-18T20:27:00.880 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.881 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.881 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.881 DEBUG:boto:Params: {} 2023-10-18T20:27:00.882 DEBUG:boto:Token: None 2023-10-18T20:27:00.883 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.883 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f10ea9f02f36b8c5-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.886 DEBUG:boto:b'bofrnf-56f1000falsef2023-10-18T20:24:59.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.887 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.888 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.888 DEBUG:boto:path=/bofrnf-56/?prefix=f 2023-10-18T20:27:00.888 DEBUG:boto:auth_path=/bofrnf-56/?prefix=f 2023-10-18T20:27:00.889 DEBUG:boto:Method: GET 2023-10-18T20:27:00.889 DEBUG:boto:Path: /bofrnf-56/?prefix=f 2023-10-18T20:27:00.890 DEBUG:boto:Data: 2023-10-18T20:27:00.890 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.891 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.891 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.891 DEBUG:boto:Params: {} 2023-10-18T20:27:00.892 DEBUG:boto:Token: None 2023-10-18T20:27:00.892 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.893 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000b58feb1d211d4e-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.896 DEBUG:boto:b'bofrnf-56f1000falsef2023-10-18T20:24:59.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.897 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.897 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.898 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.898 DEBUG:boto:Path: /bofrnf-56/f 2023-10-18T20:27:00.899 DEBUG:boto:Data: 2023-10-18T20:27:00.899 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.899 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.900 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.900 DEBUG:boto:Params: {} 2023-10-18T20:27:00.901 DEBUG:boto:Token: None 2023-10-18T20:27:00.901 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f 2023-10-18T20:27:00.902 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OkxN/9eILIsZssPIWRggHjCs1QM= 2023-10-18T20:27:00.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OkxN/9eILIsZssPIWRggHjCs1QM=', 'Content-Length': '0'} 2023-10-18T20:27:00.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007998bcf04ed773fd-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.904 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.905 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.905 DEBUG:boto:Method: HEAD 2023-10-18T20:27:00.906 DEBUG:boto:Path: /bofrnf-56/f 2023-10-18T20:27:00.906 DEBUG:boto:Data: 2023-10-18T20:27:00.907 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.907 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.907 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.908 DEBUG:boto:Params: {} 2023-10-18T20:27:00.908 DEBUG:boto:Token: None 2023-10-18T20:27:00.909 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f 2023-10-18T20:27:00.909 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OkxN/9eILIsZssPIWRggHjCs1QM= 2023-10-18T20:27:00.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OkxN/9eILIsZssPIWRggHjCs1QM=', 'Content-Length': '0'} 2023-10-18T20:27:00.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000052a211b0d1ea9c0-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:27:00.912 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.913 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.913 DEBUG:boto:Method: GET 2023-10-18T20:27:00.913 DEBUG:boto:Path: /bofrnf-56/f 2023-10-18T20:27:00.914 DEBUG:boto:Data: 2023-10-18T20:27:00.914 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.915 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.915 DEBUG:boto:Params: {} 2023-10-18T20:27:00.916 DEBUG:boto:Token: None 2023-10-18T20:27:00.916 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f 2023-10-18T20:27:00.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SWypuOOH3/iwlPEF75JtSVOMlrc= 2023-10-18T20:27:00.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SWypuOOH3/iwlPEF75JtSVOMlrc=', 'Content-Length': '0'} 2023-10-18T20:27:00.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061c62452a7de63e1-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.920 DEBUG:boto:path=/bofrnf-56/f 2023-10-18T20:27:00.921 DEBUG:boto:auth_path=/bofrnf-56/f 2023-10-18T20:27:00.921 DEBUG:boto:Method: GET 2023-10-18T20:27:00.921 DEBUG:boto:Path: /bofrnf-56/f 2023-10-18T20:27:00.922 DEBUG:boto:Data: 2023-10-18T20:27:00.922 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.923 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.924 DEBUG:boto:Params: {} 2023-10-18T20:27:00.924 DEBUG:boto:Token: None 2023-10-18T20:27:00.925 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/f 2023-10-18T20:27:00.925 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SWypuOOH3/iwlPEF75JtSVOMlrc= 2023-10-18T20:27:00.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SWypuOOH3/iwlPEF75JtSVOMlrc=', 'Content-Length': '0'} 2023-10-18T20:27:00.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f756b8f4dfd408d-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.928 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.929 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.929 DEBUG:boto:path=/bofrnf-56/?prefix=f 2023-10-18T20:27:00.929 DEBUG:boto:auth_path=/bofrnf-56/?prefix=f 2023-10-18T20:27:00.930 DEBUG:boto:Method: GET 2023-10-18T20:27:00.930 DEBUG:boto:Path: /bofrnf-56/?prefix=f 2023-10-18T20:27:00.931 DEBUG:boto:Data: 2023-10-18T20:27:00.931 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.932 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.932 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.933 DEBUG:boto:Params: {} 2023-10-18T20:27:00.933 DEBUG:boto:Token: None 2023-10-18T20:27:00.934 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.934 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba70ae460dc2fea9-0065303f94-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.938 DEBUG:boto:b'bofrnf-56f1000falsef2023-10-18T20:24:59.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.938 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:00.939 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:00.939 DEBUG:boto:path=/bofrnf-56/?prefix=f 2023-10-18T20:27:00.940 DEBUG:boto:auth_path=/bofrnf-56/?prefix=f 2023-10-18T20:27:00.940 DEBUG:boto:Method: GET 2023-10-18T20:27:00.941 DEBUG:boto:Path: /bofrnf-56/?prefix=f 2023-10-18T20:27:00.941 DEBUG:boto:Data: 2023-10-18T20:27:00.942 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.942 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.942 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.943 DEBUG:boto:Params: {} 2023-10-18T20:27:00.943 DEBUG:boto:Token: None 2023-10-18T20:27:00.944 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/ 2023-10-18T20:27:00.944 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0= 2023-10-18T20:27:00.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/0HgFsn4XJTWEM/odGRuKZ4Fi0=', 'Content-Length': '0'} 2023-10-18T20:27:00.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd3e0f9cf5f0a9af-0065303f94-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.951 DEBUG:boto:b'bofrnf-56f1000falsef2023-10-18T20:24:59.339Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:00.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:27:00.952 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:00.953 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:00.953 DEBUG:boto:path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:00.954 DEBUG:boto:auth_path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:00.954 DEBUG:boto:Method: GET 2023-10-18T20:27:00.954 DEBUG:boto:Path: /bofrnf-56/g?versionId=null 2023-10-18T20:27:00.955 DEBUG:boto:Data: 2023-10-18T20:27:00.955 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.956 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:00.956 DEBUG:boto:Port: 8001 2023-10-18T20:27:00.957 DEBUG:boto:Params: {} 2023-10-18T20:27:00.957 DEBUG:boto:Token: None 2023-10-18T20:27:00.958 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/g?versionId=null 2023-10-18T20:27:00.958 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ulBXY6opfEIpnU2zK7QFNunVZCI= 2023-10-18T20:27:00.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ulBXY6opfEIpnU2zK7QFNunVZCI=', 'Content-Length': '0'} 2023-10-18T20:27:00.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083c0576782558d1a-0065303f94-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:00.962 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:00.962 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:00.962 DEBUG:boto:path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:00.963 DEBUG:boto:auth_path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:00.963 DEBUG:boto:Method: GET 2023-10-18T20:27:00.964 DEBUG:boto:Path: /bofrnf-56/g?versionId=null 2023-10-18T20:27:00.964 DEBUG:boto:Data: 2023-10-18T20:27:00.965 DEBUG:boto:Headers: {} 2023-10-18T20:27:00.965 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:00.965 DEBUG:boto:Port: 8000 2023-10-18T20:27:00.966 DEBUG:boto:Params: {} 2023-10-18T20:27:00.966 DEBUG:boto:Token: None 2023-10-18T20:27:00.967 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:00 GMT /bofrnf-56/g?versionId=null 2023-10-18T20:27:00.967 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ulBXY6opfEIpnU2zK7QFNunVZCI= 2023-10-18T20:27:00.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ulBXY6opfEIpnU2zK7QFNunVZCI=', 'Content-Length': '0'} 2023-10-18T20:27:00.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f70915458e86861e-0065303f94-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.014 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:01.015 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:01.015 DEBUG:boto:path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:01.015 DEBUG:boto:auth_path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:01.016 DEBUG:boto:Method: HEAD 2023-10-18T20:27:01.016 DEBUG:boto:Path: /bofrnf-56/g?versionId=null 2023-10-18T20:27:01.017 DEBUG:boto:Data: 2023-10-18T20:27:01.017 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.018 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.018 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.018 DEBUG:boto:Params: {} 2023-10-18T20:27:01.019 DEBUG:boto:Token: None 2023-10-18T20:27:01.019 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/g?versionId=null 2023-10-18T20:27:01.020 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y/BFAIZ0K3eJ1A1RRBVIg5nFRXg= 2023-10-18T20:27:01.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y/BFAIZ0K3eJ1A1RRBVIg5nFRXg=', 'Content-Length': '0'} 2023-10-18T20:27:01.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000295e9b345ec56868-0065303f95-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.023 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:01.024 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:01.024 DEBUG:boto:path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:01.024 DEBUG:boto:auth_path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:01.025 DEBUG:boto:Method: HEAD 2023-10-18T20:27:01.025 DEBUG:boto:Path: /bofrnf-56/g?versionId=null 2023-10-18T20:27:01.026 DEBUG:boto:Data: 2023-10-18T20:27:01.026 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.027 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.027 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.028 DEBUG:boto:Params: {} 2023-10-18T20:27:01.028 DEBUG:boto:Token: None 2023-10-18T20:27:01.029 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/g?versionId=null 2023-10-18T20:27:01.029 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y/BFAIZ0K3eJ1A1RRBVIg5nFRXg= 2023-10-18T20:27:01.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y/BFAIZ0K3eJ1A1RRBVIg5nFRXg=', 'Content-Length': '0'} 2023-10-18T20:27:01.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686866839aa49fd9-0065303f95-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:27:01.033 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:01.033 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:01.034 DEBUG:boto:path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:01.034 DEBUG:boto:auth_path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:01.035 DEBUG:boto:Method: GET 2023-10-18T20:27:01.035 DEBUG:boto:Path: /bofrnf-56/g?versionId=null 2023-10-18T20:27:01.036 DEBUG:boto:Data: 2023-10-18T20:27:01.036 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.037 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.037 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.037 DEBUG:boto:Params: {} 2023-10-18T20:27:01.038 DEBUG:boto:Token: None 2023-10-18T20:27:01.039 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/g?versionId=null 2023-10-18T20:27:01.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nu4Pbr/2npsNfmGtKSQhxO5WcV8= 2023-10-18T20:27:01.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nu4Pbr/2npsNfmGtKSQhxO5WcV8=', 'Content-Length': '0'} 2023-10-18T20:27:01.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2b163244d04c879-0065303f95-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.086 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:01.087 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:01.087 DEBUG:boto:path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:01.088 DEBUG:boto:auth_path=/bofrnf-56/g?versionId=null 2023-10-18T20:27:01.088 DEBUG:boto:Method: GET 2023-10-18T20:27:01.089 DEBUG:boto:Path: /bofrnf-56/g?versionId=null 2023-10-18T20:27:01.089 DEBUG:boto:Data: 2023-10-18T20:27:01.089 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.090 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.090 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.091 DEBUG:boto:Params: {} 2023-10-18T20:27:01.091 DEBUG:boto:Token: None 2023-10-18T20:27:01.092 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/g?versionId=null 2023-10-18T20:27:01.092 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nu4Pbr/2npsNfmGtKSQhxO5WcV8= 2023-10-18T20:27:01.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nu4Pbr/2npsNfmGtKSQhxO5WcV8=', 'Content-Length': '0'} 2023-10-18T20:27:01.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0adae20f251faf4-0065303f95-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.095 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:01.096 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:01.096 DEBUG:boto:path=/bofrnf-56/?prefix=g 2023-10-18T20:27:01.097 DEBUG:boto:auth_path=/bofrnf-56/?prefix=g 2023-10-18T20:27:01.097 DEBUG:boto:Method: GET 2023-10-18T20:27:01.097 DEBUG:boto:Path: /bofrnf-56/?prefix=g 2023-10-18T20:27:01.098 DEBUG:boto:Data: 2023-10-18T20:27:01.098 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.099 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.099 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.100 DEBUG:boto:Params: {} 2023-10-18T20:27:01.100 DEBUG:boto:Token: None 2023-10-18T20:27:01.100 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/ 2023-10-18T20:27:01.101 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4= 2023-10-18T20:27:01.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4=', 'Content-Length': '0'} 2023-10-18T20:27:01.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7b760533175b2b-0065303f95-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.104 DEBUG:boto:b'bofrnf-56g1000falseg2023-10-18T20:24:59.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:01.105 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:01.106 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:01.106 DEBUG:boto:path=/bofrnf-56/?prefix=g 2023-10-18T20:27:01.106 DEBUG:boto:auth_path=/bofrnf-56/?prefix=g 2023-10-18T20:27:01.107 DEBUG:boto:Method: GET 2023-10-18T20:27:01.107 DEBUG:boto:Path: /bofrnf-56/?prefix=g 2023-10-18T20:27:01.108 DEBUG:boto:Data: 2023-10-18T20:27:01.108 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.109 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.109 DEBUG:boto:Params: {} 2023-10-18T20:27:01.110 DEBUG:boto:Token: None 2023-10-18T20:27:01.110 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/ 2023-10-18T20:27:01.111 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4= 2023-10-18T20:27:01.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4=', 'Content-Length': '0'} 2023-10-18T20:27:01.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000507bc22c1823fdde-0065303f95-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.114 DEBUG:boto:b'bofrnf-56g1000falseg2023-10-18T20:24:59.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:01.114 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:01.115 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:01.115 DEBUG:boto:Method: HEAD 2023-10-18T20:27:01.116 DEBUG:boto:Path: /bofrnf-56/g 2023-10-18T20:27:01.116 DEBUG:boto:Data: 2023-10-18T20:27:01.116 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.117 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.117 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.117 DEBUG:boto:Params: {} 2023-10-18T20:27:01.118 DEBUG:boto:Token: None 2023-10-18T20:27:01.118 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/g 2023-10-18T20:27:01.119 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:10RiUWHzeGzohIi7D0Ld1/1kHoQ= 2023-10-18T20:27:01.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:10RiUWHzeGzohIi7D0Ld1/1kHoQ=', 'Content-Length': '0'} 2023-10-18T20:27:01.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daaf89560a0825a9-0065303f95-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.121 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:01.122 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:01.122 DEBUG:boto:Method: HEAD 2023-10-18T20:27:01.123 DEBUG:boto:Path: /bofrnf-56/g 2023-10-18T20:27:01.123 DEBUG:boto:Data: 2023-10-18T20:27:01.124 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.124 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.124 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.125 DEBUG:boto:Params: {} 2023-10-18T20:27:01.125 DEBUG:boto:Token: None 2023-10-18T20:27:01.126 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/g 2023-10-18T20:27:01.126 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:10RiUWHzeGzohIi7D0Ld1/1kHoQ= 2023-10-18T20:27:01.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:10RiUWHzeGzohIi7D0Ld1/1kHoQ=', 'Content-Length': '0'} 2023-10-18T20:27:01.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191390c1c2bb7973-0065303f95-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:27:01.130 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:01.130 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:01.131 DEBUG:boto:Method: GET 2023-10-18T20:27:01.131 DEBUG:boto:Path: /bofrnf-56/g 2023-10-18T20:27:01.131 DEBUG:boto:Data: 2023-10-18T20:27:01.132 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.132 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.133 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.133 DEBUG:boto:Params: {} 2023-10-18T20:27:01.134 DEBUG:boto:Token: None 2023-10-18T20:27:01.134 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/g 2023-10-18T20:27:01.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pBqIkBw+RGYD1QE8ZZN9uv41iBM= 2023-10-18T20:27:01.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pBqIkBw+RGYD1QE8ZZN9uv41iBM=', 'Content-Length': '0'} 2023-10-18T20:27:01.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bab3e87a0821688-0065303f95-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.138 DEBUG:boto:path=/bofrnf-56/g 2023-10-18T20:27:01.138 DEBUG:boto:auth_path=/bofrnf-56/g 2023-10-18T20:27:01.139 DEBUG:boto:Method: GET 2023-10-18T20:27:01.139 DEBUG:boto:Path: /bofrnf-56/g 2023-10-18T20:27:01.139 DEBUG:boto:Data: 2023-10-18T20:27:01.140 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.140 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.141 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.141 DEBUG:boto:Params: {} 2023-10-18T20:27:01.142 DEBUG:boto:Token: None 2023-10-18T20:27:01.142 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/g 2023-10-18T20:27:01.143 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pBqIkBw+RGYD1QE8ZZN9uv41iBM= 2023-10-18T20:27:01.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pBqIkBw+RGYD1QE8ZZN9uv41iBM=', 'Content-Length': '0'} 2023-10-18T20:27:01.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001897bf5442c5e6cd-0065303f95-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.146 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:01.146 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:01.147 DEBUG:boto:path=/bofrnf-56/?prefix=g 2023-10-18T20:27:01.147 DEBUG:boto:auth_path=/bofrnf-56/?prefix=g 2023-10-18T20:27:01.148 DEBUG:boto:Method: GET 2023-10-18T20:27:01.148 DEBUG:boto:Path: /bofrnf-56/?prefix=g 2023-10-18T20:27:01.149 DEBUG:boto:Data: 2023-10-18T20:27:01.149 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.150 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.151 DEBUG:boto:Params: {} 2023-10-18T20:27:01.151 DEBUG:boto:Token: None 2023-10-18T20:27:01.151 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/ 2023-10-18T20:27:01.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4= 2023-10-18T20:27:01.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4=', 'Content-Length': '0'} 2023-10-18T20:27:01.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab62b2cf0c0a59d2-0065303f95-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.156 DEBUG:boto:b'bofrnf-56g1000falseg2023-10-18T20:24:59.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:01.156 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:01.157 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:01.157 DEBUG:boto:path=/bofrnf-56/?prefix=g 2023-10-18T20:27:01.158 DEBUG:boto:auth_path=/bofrnf-56/?prefix=g 2023-10-18T20:27:01.158 DEBUG:boto:Method: GET 2023-10-18T20:27:01.159 DEBUG:boto:Path: /bofrnf-56/?prefix=g 2023-10-18T20:27:01.159 DEBUG:boto:Data: 2023-10-18T20:27:01.159 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.160 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.160 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.161 DEBUG:boto:Params: {} 2023-10-18T20:27:01.161 DEBUG:boto:Token: None 2023-10-18T20:27:01.162 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/ 2023-10-18T20:27:01.162 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4= 2023-10-18T20:27:01.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4=', 'Content-Length': '0'} 2023-10-18T20:27:01.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0ae4d82240e9f95-0065303f95-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.165 DEBUG:boto:b'bofrnf-56g1000falseg2023-10-18T20:24:59.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:01.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:27:01.166 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.167 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.167 DEBUG:boto:path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.167 DEBUG:boto:auth_path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.168 DEBUG:boto:Method: GET 2023-10-18T20:27:01.168 DEBUG:boto:Path: /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.169 DEBUG:boto:Data: 2023-10-18T20:27:01.169 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.170 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.170 DEBUG:boto:Params: {} 2023-10-18T20:27:01.171 DEBUG:boto:Token: None 2023-10-18T20:27:01.171 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzDB45W4CHd9rP5MAJJtsB5+LX8= 2023-10-18T20:27:01.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzDB45W4CHd9rP5MAJJtsB5+LX8=', 'Content-Length': '0'} 2023-10-18T20:27:01.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021778a265840069b-0065303f95-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.218 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.219 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.219 DEBUG:boto:path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.220 DEBUG:boto:auth_path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.220 DEBUG:boto:Method: GET 2023-10-18T20:27:01.220 DEBUG:boto:Path: /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.221 DEBUG:boto:Data: 2023-10-18T20:27:01.221 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.222 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.222 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.223 DEBUG:boto:Params: {} 2023-10-18T20:27:01.223 DEBUG:boto:Token: None 2023-10-18T20:27:01.223 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.224 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzDB45W4CHd9rP5MAJJtsB5+LX8= 2023-10-18T20:27:01.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzDB45W4CHd9rP5MAJJtsB5+LX8=', 'Content-Length': '0'} 2023-10-18T20:27:01.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000913ae5c31e9b93e0-0065303f95-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.227 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.228 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.228 DEBUG:boto:path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.229 DEBUG:boto:auth_path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.229 DEBUG:boto:Method: HEAD 2023-10-18T20:27:01.229 DEBUG:boto:Path: /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.230 DEBUG:boto:Data: 2023-10-18T20:27:01.230 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.231 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.231 DEBUG:boto:Params: {} 2023-10-18T20:27:01.232 DEBUG:boto:Token: None 2023-10-18T20:27:01.232 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xZ2MhzwfMnZqFsNyak1NH0Nxqgc= 2023-10-18T20:27:01.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xZ2MhzwfMnZqFsNyak1NH0Nxqgc=', 'Content-Length': '0'} 2023-10-18T20:27:01.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1d540a567bcd879-0065303f95-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.235 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.236 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.236 DEBUG:boto:path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.237 DEBUG:boto:auth_path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.237 DEBUG:boto:Method: HEAD 2023-10-18T20:27:01.237 DEBUG:boto:Path: /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.238 DEBUG:boto:Data: 2023-10-18T20:27:01.238 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.239 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.239 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.239 DEBUG:boto:Params: {} 2023-10-18T20:27:01.240 DEBUG:boto:Token: None 2023-10-18T20:27:01.240 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.241 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xZ2MhzwfMnZqFsNyak1NH0Nxqgc= 2023-10-18T20:27:01.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xZ2MhzwfMnZqFsNyak1NH0Nxqgc=', 'Content-Length': '0'} 2023-10-18T20:27:01.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813abd1b69c1cae6-0065303f95-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:27:01.244 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.245 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.245 DEBUG:boto:path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.245 DEBUG:boto:auth_path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.246 DEBUG:boto:Method: GET 2023-10-18T20:27:01.246 DEBUG:boto:Path: /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.247 DEBUG:boto:Data: 2023-10-18T20:27:01.247 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.248 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.249 DEBUG:boto:Params: {} 2023-10-18T20:27:01.249 DEBUG:boto:Token: None 2023-10-18T20:27:01.250 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzDB45W4CHd9rP5MAJJtsB5+LX8= 2023-10-18T20:27:01.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzDB45W4CHd9rP5MAJJtsB5+LX8=', 'Content-Length': '0'} 2023-10-18T20:27:01.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006be8d6b1d72cdc62-0065303f95-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.253 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.254 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.254 DEBUG:boto:path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.254 DEBUG:boto:auth_path=/bofrnf-56/h?versionId=null 2023-10-18T20:27:01.255 DEBUG:boto:Method: GET 2023-10-18T20:27:01.255 DEBUG:boto:Path: /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.256 DEBUG:boto:Data: 2023-10-18T20:27:01.256 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.256 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.257 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.257 DEBUG:boto:Params: {} 2023-10-18T20:27:01.258 DEBUG:boto:Token: None 2023-10-18T20:27:01.258 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h?versionId=null 2023-10-18T20:27:01.259 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rzDB45W4CHd9rP5MAJJtsB5+LX8= 2023-10-18T20:27:01.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzDB45W4CHd9rP5MAJJtsB5+LX8=', 'Content-Length': '0'} 2023-10-18T20:27:01.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff898a053f668e57-0065303f95-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.306 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:01.307 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:01.307 DEBUG:boto:path=/bofrnf-56/?prefix=h 2023-10-18T20:27:01.308 DEBUG:boto:auth_path=/bofrnf-56/?prefix=h 2023-10-18T20:27:01.308 DEBUG:boto:Method: GET 2023-10-18T20:27:01.309 DEBUG:boto:Path: /bofrnf-56/?prefix=h 2023-10-18T20:27:01.309 DEBUG:boto:Data: 2023-10-18T20:27:01.310 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.310 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.311 DEBUG:boto:Params: {} 2023-10-18T20:27:01.311 DEBUG:boto:Token: None 2023-10-18T20:27:01.312 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/ 2023-10-18T20:27:01.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4= 2023-10-18T20:27:01.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4=', 'Content-Length': '0'} 2023-10-18T20:27:01.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a05009a54c30190-0065303f95-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.316 DEBUG:boto:b'bofrnf-56h1000falseh2023-10-18T20:24:59.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:01.317 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:01.318 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:01.318 DEBUG:boto:path=/bofrnf-56/?prefix=h 2023-10-18T20:27:01.318 DEBUG:boto:auth_path=/bofrnf-56/?prefix=h 2023-10-18T20:27:01.319 DEBUG:boto:Method: GET 2023-10-18T20:27:01.319 DEBUG:boto:Path: /bofrnf-56/?prefix=h 2023-10-18T20:27:01.320 DEBUG:boto:Data: 2023-10-18T20:27:01.320 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.320 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.321 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.321 DEBUG:boto:Params: {} 2023-10-18T20:27:01.322 DEBUG:boto:Token: None 2023-10-18T20:27:01.322 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/ 2023-10-18T20:27:01.323 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4= 2023-10-18T20:27:01.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4=', 'Content-Length': '0'} 2023-10-18T20:27:01.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8fea32e471fc87b-0065303f95-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.326 DEBUG:boto:b'bofrnf-56h1000falseh2023-10-18T20:24:59.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:01.327 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.327 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.328 DEBUG:boto:Method: HEAD 2023-10-18T20:27:01.328 DEBUG:boto:Path: /bofrnf-56/h 2023-10-18T20:27:01.328 DEBUG:boto:Data: 2023-10-18T20:27:01.329 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.329 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.330 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.330 DEBUG:boto:Params: {} 2023-10-18T20:27:01.330 DEBUG:boto:Token: None 2023-10-18T20:27:01.331 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h 2023-10-18T20:27:01.331 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6nLHuolqe6i/fMKNNaHksdcfPng= 2023-10-18T20:27:01.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6nLHuolqe6i/fMKNNaHksdcfPng=', 'Content-Length': '0'} 2023-10-18T20:27:01.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c4d6518691b7352-0065303f95-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.334 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.335 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.335 DEBUG:boto:Method: HEAD 2023-10-18T20:27:01.336 DEBUG:boto:Path: /bofrnf-56/h 2023-10-18T20:27:01.336 DEBUG:boto:Data: 2023-10-18T20:27:01.336 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.337 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.337 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.338 DEBUG:boto:Params: {} 2023-10-18T20:27:01.338 DEBUG:boto:Token: None 2023-10-18T20:27:01.339 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h 2023-10-18T20:27:01.339 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6nLHuolqe6i/fMKNNaHksdcfPng= 2023-10-18T20:27:01.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6nLHuolqe6i/fMKNNaHksdcfPng=', 'Content-Length': '0'} 2023-10-18T20:27:01.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa8978c1a48aa88c-0065303f95-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:27:01.342 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.343 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.343 DEBUG:boto:Method: GET 2023-10-18T20:27:01.344 DEBUG:boto:Path: /bofrnf-56/h 2023-10-18T20:27:01.344 DEBUG:boto:Data: 2023-10-18T20:27:01.345 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.345 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.345 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.346 DEBUG:boto:Params: {} 2023-10-18T20:27:01.346 DEBUG:boto:Token: None 2023-10-18T20:27:01.347 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h 2023-10-18T20:27:01.347 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2Gl1sQ7+EXCIXPbyyI8WdpmKCZY= 2023-10-18T20:27:01.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2Gl1sQ7+EXCIXPbyyI8WdpmKCZY=', 'Content-Length': '0'} 2023-10-18T20:27:01.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-56:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000889d8f459dde5243-0065303f95-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.390 DEBUG:boto:path=/bofrnf-56/h 2023-10-18T20:27:01.391 DEBUG:boto:auth_path=/bofrnf-56/h 2023-10-18T20:27:01.392 DEBUG:boto:Method: GET 2023-10-18T20:27:01.392 DEBUG:boto:Path: /bofrnf-56/h 2023-10-18T20:27:01.393 DEBUG:boto:Data: 2023-10-18T20:27:01.393 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.394 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.394 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.395 DEBUG:boto:Params: {} 2023-10-18T20:27:01.395 DEBUG:boto:Token: None 2023-10-18T20:27:01.396 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/h 2023-10-18T20:27:01.396 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2Gl1sQ7+EXCIXPbyyI8WdpmKCZY= 2023-10-18T20:27:01.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2Gl1sQ7+EXCIXPbyyI8WdpmKCZY=', 'Content-Length': '0'} 2023-10-18T20:27:01.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039aebd8e97837527-0065303f95-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.400 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:01.400 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:01.400 DEBUG:boto:path=/bofrnf-56/?prefix=h 2023-10-18T20:27:01.401 DEBUG:boto:auth_path=/bofrnf-56/?prefix=h 2023-10-18T20:27:01.401 DEBUG:boto:Method: GET 2023-10-18T20:27:01.402 DEBUG:boto:Path: /bofrnf-56/?prefix=h 2023-10-18T20:27:01.402 DEBUG:boto:Data: 2023-10-18T20:27:01.403 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.403 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:01.404 DEBUG:boto:Port: 8001 2023-10-18T20:27:01.404 DEBUG:boto:Params: {} 2023-10-18T20:27:01.405 DEBUG:boto:Token: None 2023-10-18T20:27:01.405 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/ 2023-10-18T20:27:01.406 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4= 2023-10-18T20:27:01.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4=', 'Content-Length': '0'} 2023-10-18T20:27:01.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8424406ebfaad56-0065303f95-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.409 DEBUG:boto:b'bofrnf-56h1000falseh2023-10-18T20:24:59.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:01.410 DEBUG:boto:path=/bofrnf-56/ 2023-10-18T20:27:01.410 DEBUG:boto:auth_path=/bofrnf-56/ 2023-10-18T20:27:01.411 DEBUG:boto:path=/bofrnf-56/?prefix=h 2023-10-18T20:27:01.411 DEBUG:boto:auth_path=/bofrnf-56/?prefix=h 2023-10-18T20:27:01.411 DEBUG:boto:Method: GET 2023-10-18T20:27:01.412 DEBUG:boto:Path: /bofrnf-56/?prefix=h 2023-10-18T20:27:01.412 DEBUG:boto:Data: 2023-10-18T20:27:01.413 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.413 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.413 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.414 DEBUG:boto:Params: {} 2023-10-18T20:27:01.414 DEBUG:boto:Token: None 2023-10-18T20:27:01.415 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-56/ 2023-10-18T20:27:01.415 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4= 2023-10-18T20:27:01.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W1xaFIM/H8IXUMYohNxCUfxcxO4=', 'Content-Length': '0'} 2023-10-18T20:27:01.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000659184d9d30d63cb-0065303f95-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.419 DEBUG:boto:b'bofrnf-56h1000falseh2023-10-18T20:24:59.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:27:01.420 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-56 zones={a2, a1} 2023-10-18T20:27:01.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2023-10-18T20:27:01.422 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:27:01.422 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:27:01.422 DEBUG:boto:Method: PUT 2023-10-18T20:27:01.423 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:27:01.423 DEBUG:boto:Data: 2023-10-18T20:27:01.424 DEBUG:boto:Headers: {} 2023-10-18T20:27:01.424 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:01.425 DEBUG:boto:Port: 8000 2023-10-18T20:27:01.425 DEBUG:boto:Params: {} 2023-10-18T20:27:01.425 DEBUG:boto:Token: None 2023-10-18T20:27:01.426 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:27:01 GMT /bofrnf-57/ 2023-10-18T20:27:01.426 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jd3Ai4pSMo9zSB2L27e1c2Zmp2I= 2023-10-18T20:27:01.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jd3Ai4pSMo9zSB2L27e1c2Zmp2I=', 'Content-Length': '0'} 2023-10-18T20:27:01.445 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4a27223b75fded2-0065303f95-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:27:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:01.445 DEBUG:rgw_multi.tests:created bucket=bofrnf-57 2023-10-18T20:27:01.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:27:01.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:27:01.447 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:27:01.502 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:27:01.502 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:27:01.538 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:27:01.551 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:27:01.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:27:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660821.442511_992.1", 2023-10-18T20:27:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:27:01.442511Z" 2023-10-18T20:27:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:27:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:27:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:27:01.553 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:27:01.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:27:01.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:27:01.553 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:27:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:27:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:27:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:27:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:27:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:27:01.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:27:01.555 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:27:01.556 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660821.442511_992.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:27:01.556 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:27:01.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:27:01.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:27:01.557 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:27:01.607 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:27:01.607 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:27:01.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:01.633+0000 7f7581c10780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:27:01.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:01.634+0000 7f7581c10780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:27:01.648 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:27:01.649 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:27:01.649 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:27:01.649 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:27:01.649 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:27:01.649 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:27:01.650 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:27:01.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:27:01.650 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:27:01.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:27:01.650 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:27:01.651 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:27:01.651 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:27:01.651 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660821.442511_992.1", 2023-10-18T20:27:01.651 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:27:01.651 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:27:01.652 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:27:01.652 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:27:01.442511Z", 2023-10-18T20:27:01.652 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:27:01.652 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:27:01.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:27:01.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:27:01.653 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:27:01.653 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:27:01.653 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:27:01.653 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:27:01.654 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:27:01.654 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:27:01.654 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:27:01.654 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:27:01.655 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:27:01.655 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:27:01.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:27:01.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:27:01.655 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:27:01.656 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:27:01.656 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:27:01.656 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:27:01.656 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:27:01.656 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:27:01.657 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:27:01.657 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:27:01.657 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:27:01.657 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:27:01.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:27:01.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:27:01.658 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:27:01.658 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:27:01.658 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:27:01.658 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:27:01.658 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:27:01.659 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:27:01.659 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:27:01.659 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:27:01.659 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:27:01.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:27:01.660 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:27:01.660 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:27:01.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:27:01.660 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:27:01.660 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:27:01.660 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:27:01.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:27:01.661 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:27:01Z" 2023-10-18T20:27:01.661 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:27:01.662 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660821.442511_992.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:27:01.442511Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:27:01Z" } 2023-10-18T20:27:01.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660821.442511_992.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:27:01.442511Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:27:01.663 DEBUG:rgw_multi.tests:log_status={0: '1_1697660821.442511_992.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:27:01.663 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660821.442511_992.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:27:01.664 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:27:01.664 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-18T20:27:01.665 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T20:27:01.665 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:27:01.663+0000 7efdf696a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 123669) UID: 0 2023-10-18T20:27:01.665 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:27:01.663+0000 7efdfabd37c0 -1 shutting down 2023-10-18T20:27:05.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T20:27:07.766 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-18T20:27:07.767 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:27:07.768 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:27:07.768 DEBUG:boto:Method: HEAD 2023-10-18T20:27:07.769 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:27:07.770 DEBUG:boto:Data: 2023-10-18T20:27:07.771 DEBUG:boto:Headers: {} 2023-10-18T20:27:07.771 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:07.772 DEBUG:boto:Port: 8000 2023-10-18T20:27:07.773 DEBUG:boto:Params: {} 2023-10-18T20:27:07.774 DEBUG:boto:Token: None 2023-10-18T20:27:07.774 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:07 GMT /bofrnf-57/ 2023-10-18T20:27:07.775 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AsIBKFCBsiaW747TQ70BnzEaeoU= 2023-10-18T20:27:07.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AsIBKFCBsiaW747TQ70BnzEaeoU=', 'Content-Length': '0'} 2023-10-18T20:27:07.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000926e2be5345140ef-0065303f9b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:07.781 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:27:07.781 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:27:07.782 DEBUG:boto:Method: PUT 2023-10-18T20:27:07.783 DEBUG:boto:Path: /bofrnf-57/a 2023-10-18T20:27:07.783 DEBUG:boto:Data: 2023-10-18T20:27:07.784 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:27:07.785 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:07.785 DEBUG:boto:Port: 8000 2023-10-18T20:27:07.786 DEBUG:boto:Params: {} 2023-10-18T20:27:07.786 DEBUG:boto:Token: None 2023-10-18T20:27:07.787 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:27:07 GMT /bofrnf-57/a 2023-10-18T20:27:07.788 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wPQifg5qRHsiTJM9RqhFBZOxo+w= 2023-10-18T20:27:07.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:27:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wPQifg5qRHsiTJM9RqhFBZOxo+w='} 2023-10-18T20:27:07.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d98e2b572349bd70-0065303f9b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:07.796 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:27:07.797 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:27:07.797 DEBUG:boto:Method: HEAD 2023-10-18T20:27:07.798 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:27:07.798 DEBUG:boto:Data: 2023-10-18T20:27:07.798 DEBUG:boto:Headers: {} 2023-10-18T20:27:07.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:07.799 DEBUG:boto:Port: 8000 2023-10-18T20:27:07.799 DEBUG:boto:Params: {} 2023-10-18T20:27:07.800 DEBUG:boto:Token: None 2023-10-18T20:27:07.800 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:07 GMT /bofrnf-57/ 2023-10-18T20:27:07.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AsIBKFCBsiaW747TQ70BnzEaeoU= 2023-10-18T20:27:07.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AsIBKFCBsiaW747TQ70BnzEaeoU=', 'Content-Length': '0'} 2023-10-18T20:27:07.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027c4cc36336355b2-0065303f9b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:07.805 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:27:07.805 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:27:07.805 DEBUG:boto:Method: PUT 2023-10-18T20:27:07.806 DEBUG:boto:Path: /bofrnf-57/b 2023-10-18T20:27:07.806 DEBUG:boto:Data: 2023-10-18T20:27:07.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:27:07.808 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:07.809 DEBUG:boto:Port: 8000 2023-10-18T20:27:07.810 DEBUG:boto:Params: {} 2023-10-18T20:27:07.811 DEBUG:boto:Token: None 2023-10-18T20:27:07.811 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:27:07 GMT /bofrnf-57/b 2023-10-18T20:27:07.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CATXA6HBqbnbSt52RRjsryCkYCU= 2023-10-18T20:27:07.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:27:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CATXA6HBqbnbSt52RRjsryCkYCU='} 2023-10-18T20:27:07.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047e5fb9ef83e694e-0065303f9b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:07.827 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:27:07.827 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:27:07.828 DEBUG:boto:Method: HEAD 2023-10-18T20:27:07.828 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:27:07.829 DEBUG:boto:Data: 2023-10-18T20:27:07.829 DEBUG:boto:Headers: {} 2023-10-18T20:27:07.829 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:07.830 DEBUG:boto:Port: 8000 2023-10-18T20:27:07.830 DEBUG:boto:Params: {} 2023-10-18T20:27:07.831 DEBUG:boto:Token: None 2023-10-18T20:27:07.831 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:07 GMT /bofrnf-57/ 2023-10-18T20:27:07.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AsIBKFCBsiaW747TQ70BnzEaeoU= 2023-10-18T20:27:07.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AsIBKFCBsiaW747TQ70BnzEaeoU=', 'Content-Length': '0'} 2023-10-18T20:27:07.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028538c2fe8e3c289-0065303f9b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:07.835 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:27:07.836 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:27:07.836 DEBUG:boto:Method: PUT 2023-10-18T20:27:07.837 DEBUG:boto:Path: /bofrnf-57/c 2023-10-18T20:27:07.837 DEBUG:boto:Data: 2023-10-18T20:27:07.837 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:27:07.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:07.838 DEBUG:boto:Port: 8000 2023-10-18T20:27:07.839 DEBUG:boto:Params: {} 2023-10-18T20:27:07.839 DEBUG:boto:Token: None 2023-10-18T20:27:07.840 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:27:07 GMT /bofrnf-57/c 2023-10-18T20:27:07.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L7fdQ91pjNHYZTt9BcKx9ab4JqE= 2023-10-18T20:27:07.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:27:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L7fdQ91pjNHYZTt9BcKx9ab4JqE='} 2023-10-18T20:27:07.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018b7a1089c7c40f5-0065303f9b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:07.883 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:27:07.884 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:27:07.884 DEBUG:boto:Method: HEAD 2023-10-18T20:27:07.884 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:27:07.885 DEBUG:boto:Data: 2023-10-18T20:27:07.885 DEBUG:boto:Headers: {} 2023-10-18T20:27:07.886 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:07.886 DEBUG:boto:Port: 8000 2023-10-18T20:27:07.886 DEBUG:boto:Params: {} 2023-10-18T20:27:07.887 DEBUG:boto:Token: None 2023-10-18T20:27:07.887 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:07 GMT /bofrnf-57/ 2023-10-18T20:27:07.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AsIBKFCBsiaW747TQ70BnzEaeoU= 2023-10-18T20:27:07.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AsIBKFCBsiaW747TQ70BnzEaeoU=', 'Content-Length': '0'} 2023-10-18T20:27:07.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003d63c21bcb2cbc4-0065303f9b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:07.891 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:27:07.892 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:27:07.892 DEBUG:boto:Method: PUT 2023-10-18T20:27:07.893 DEBUG:boto:Path: /bofrnf-57/d 2023-10-18T20:27:07.893 DEBUG:boto:Data: 2023-10-18T20:27:07.893 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:27:07.894 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:07.894 DEBUG:boto:Port: 8000 2023-10-18T20:27:07.895 DEBUG:boto:Params: {} 2023-10-18T20:27:07.895 DEBUG:boto:Token: None 2023-10-18T20:27:07.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:27:07 GMT /bofrnf-57/d 2023-10-18T20:27:07.896 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lkJAabm/oqVdFppbPZ4uktLZ2Bk= 2023-10-18T20:27:07.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:27:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lkJAabm/oqVdFppbPZ4uktLZ2Bk='} 2023-10-18T20:27:07.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027d037370e71c708-0065303f9b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:07.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'bofrnf-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:27:07.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'bofrnf-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:27:07.904 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket bofrnf-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:27:07.960 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:27:07.961 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:27:08.005 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:27:08.005 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-57 2023-10-18T20:27:08.009 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 4 2023-10-18T20:27:08.044 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:27:08.044 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:27:08.045 DEBUG:boto:Method: HEAD 2023-10-18T20:27:08.045 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:27:08.046 DEBUG:boto:Data: 2023-10-18T20:27:08.046 DEBUG:boto:Headers: {} 2023-10-18T20:27:08.047 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:08.047 DEBUG:boto:Port: 8000 2023-10-18T20:27:08.047 DEBUG:boto:Params: {} 2023-10-18T20:27:08.048 DEBUG:boto:Token: None 2023-10-18T20:27:08.048 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:08 GMT /bofrnf-57/ 2023-10-18T20:27:08.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6+fx4kWFcxIhMrbqTAD9BayxyGk= 2023-10-18T20:27:08.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6+fx4kWFcxIhMrbqTAD9BayxyGk=', 'Content-Length': '0'} 2023-10-18T20:27:08.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000970aae0b202856a6-0065303f9c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:08.052 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:27:08.052 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:27:08.053 DEBUG:boto:Method: PUT 2023-10-18T20:27:08.053 DEBUG:boto:Path: /bofrnf-57/e 2023-10-18T20:27:08.054 DEBUG:boto:Data: 2023-10-18T20:27:08.054 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:27:08.054 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:08.055 DEBUG:boto:Port: 8000 2023-10-18T20:27:08.055 DEBUG:boto:Params: {} 2023-10-18T20:27:08.056 DEBUG:boto:Token: None 2023-10-18T20:27:08.056 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:27:08 GMT /bofrnf-57/e 2023-10-18T20:27:08.057 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lGKXbRuBEYJoJuRqvDCGcJZvhX8= 2023-10-18T20:27:08.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:27:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lGKXbRuBEYJoJuRqvDCGcJZvhX8='} 2023-10-18T20:27:08.103 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000128375501f082dc3-0065303f9c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:08.103 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:27:08.104 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:27:08.104 DEBUG:boto:Method: HEAD 2023-10-18T20:27:08.105 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:27:08.105 DEBUG:boto:Data: 2023-10-18T20:27:08.105 DEBUG:boto:Headers: {} 2023-10-18T20:27:08.106 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:08.106 DEBUG:boto:Port: 8000 2023-10-18T20:27:08.107 DEBUG:boto:Params: {} 2023-10-18T20:27:08.107 DEBUG:boto:Token: None 2023-10-18T20:27:08.107 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:08 GMT /bofrnf-57/ 2023-10-18T20:27:08.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6+fx4kWFcxIhMrbqTAD9BayxyGk= 2023-10-18T20:27:08.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6+fx4kWFcxIhMrbqTAD9BayxyGk=', 'Content-Length': '0'} 2023-10-18T20:27:08.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040b98ded2944967e-0065303f9c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:08.111 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:27:08.111 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:27:08.112 DEBUG:boto:Method: PUT 2023-10-18T20:27:08.112 DEBUG:boto:Path: /bofrnf-57/f 2023-10-18T20:27:08.112 DEBUG:boto:Data: 2023-10-18T20:27:08.113 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:27:08.113 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:08.114 DEBUG:boto:Port: 8000 2023-10-18T20:27:08.114 DEBUG:boto:Params: {} 2023-10-18T20:27:08.115 DEBUG:boto:Token: None 2023-10-18T20:27:08.115 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:27:08 GMT /bofrnf-57/f 2023-10-18T20:27:08.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OBLUqCUS7Tda6sKxDeyTGqW4sIE= 2023-10-18T20:27:08.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:27:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OBLUqCUS7Tda6sKxDeyTGqW4sIE='} 2023-10-18T20:27:08.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000137ff9409f8a63cb-0065303f9c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:08.123 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:27:08.123 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:27:08.124 DEBUG:boto:Method: HEAD 2023-10-18T20:27:08.124 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:27:08.125 DEBUG:boto:Data: 2023-10-18T20:27:08.125 DEBUG:boto:Headers: {} 2023-10-18T20:27:08.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:08.126 DEBUG:boto:Port: 8000 2023-10-18T20:27:08.126 DEBUG:boto:Params: {} 2023-10-18T20:27:08.127 DEBUG:boto:Token: None 2023-10-18T20:27:08.127 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:08 GMT /bofrnf-57/ 2023-10-18T20:27:08.128 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6+fx4kWFcxIhMrbqTAD9BayxyGk= 2023-10-18T20:27:08.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6+fx4kWFcxIhMrbqTAD9BayxyGk=', 'Content-Length': '0'} 2023-10-18T20:27:08.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e527cb491a8adb4a-0065303f9c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:08.131 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:27:08.131 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:27:08.131 DEBUG:boto:Method: PUT 2023-10-18T20:27:08.132 DEBUG:boto:Path: /bofrnf-57/g 2023-10-18T20:27:08.132 DEBUG:boto:Data: 2023-10-18T20:27:08.133 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:27:08.133 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:08.133 DEBUG:boto:Port: 8000 2023-10-18T20:27:08.134 DEBUG:boto:Params: {} 2023-10-18T20:27:08.134 DEBUG:boto:Token: None 2023-10-18T20:27:08.135 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:27:08 GMT /bofrnf-57/g 2023-10-18T20:27:08.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:raMR2rs3Vyzri5m1RPgbknyIfns= 2023-10-18T20:27:08.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:27:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:raMR2rs3Vyzri5m1RPgbknyIfns='} 2023-10-18T20:27:08.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000648fd06029350b24-0065303f9c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:08.183 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:27:08.184 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:27:08.184 DEBUG:boto:Method: HEAD 2023-10-18T20:27:08.185 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:27:08.185 DEBUG:boto:Data: 2023-10-18T20:27:08.185 DEBUG:boto:Headers: {} 2023-10-18T20:27:08.186 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:08.186 DEBUG:boto:Port: 8000 2023-10-18T20:27:08.187 DEBUG:boto:Params: {} 2023-10-18T20:27:08.187 DEBUG:boto:Token: None 2023-10-18T20:27:08.187 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:27:08 GMT /bofrnf-57/ 2023-10-18T20:27:08.188 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6+fx4kWFcxIhMrbqTAD9BayxyGk= 2023-10-18T20:27:08.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:27:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6+fx4kWFcxIhMrbqTAD9BayxyGk=', 'Content-Length': '0'} 2023-10-18T20:27:08.190 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c570ebe05b5b29c2-0065303f9c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:08.191 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:27:08.191 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:27:08.192 DEBUG:boto:Method: PUT 2023-10-18T20:27:08.192 DEBUG:boto:Path: /bofrnf-57/h 2023-10-18T20:27:08.193 DEBUG:boto:Data: 2023-10-18T20:27:08.193 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:27:08.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:27:08.194 DEBUG:boto:Port: 8000 2023-10-18T20:27:08.194 DEBUG:boto:Params: {} 2023-10-18T20:27:08.195 DEBUG:boto:Token: None 2023-10-18T20:27:08.195 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:27:08 GMT /bofrnf-57/h 2023-10-18T20:27:08.196 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JgMJgzWM8UBLdKRE32lq8P+lBGE= 2023-10-18T20:27:08.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:27:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JgMJgzWM8UBLdKRE32lq8P+lBGE='} 2023-10-18T20:27:08.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c28bf297d99afe55-0065303f9c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:27:08.200 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-18T20:27:08.201 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T20:27:08.204 INFO:tasks.rgw.c1.client.1:Started 2023-10-18T20:27:08.204 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:08.226 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:27:08.226 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:27:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T20:27:08.229 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T20:27:09.230 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:27:09.254 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:27:09.254 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:27:09.267 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 16461 0 --:--:-- --:--:-- --:--:-- 16461 2023-10-18T20:27:09.268 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T20:27:09.268 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-57 2023-10-18T20:27:09.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:27:09.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:27:09.269 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:27:09.369 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:27:09.369 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:27:09.407 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:09.405+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:11.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-18T20:27:14.408 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:14.407+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:19.409 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:19.407+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:24.409 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:24.408+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:29.410 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:29.408+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:34.410 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:34.409+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:39.411 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:39.409+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:44.411 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:44.410+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:49.412 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:49.409+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:54.413 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:54.410+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:27:59.413 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:27:59.411+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:04.414 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:04.412+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:09.414 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:09.413+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:14.415 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:14.413+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:19.415 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:19.414+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:24.416 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:24.415+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:29.416 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:29.415+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:34.417 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:34.416+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:39.418 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:39.416+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:44.418 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:44.416+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:49.419 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:49.417+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:54.420 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:54.418+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:28:59.421 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:28:59.419+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:29:04.421 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:04.420+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:29:09.422 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:09.420+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:29:14.423 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:14.421+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:29:19.423 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:19.422+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:29:24.424 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:24.422+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:29:29.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:29.423+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:29:34.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:34.422+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:29:39.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:39.423+0000 7fec111ed780 1 waiting to reach incremental sync.. 2023-10-18T20:29:44.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:44.424+0000 7fec111ed780 1 bucket sync caught up with source: 2023-10-18T20:29:44.427 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000003.107.6, 00000000002.44.6, 00000000002.46.6] 2023-10-18T20:29:44.427 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000003.107.6, 00000000002.44.6, 00000000002.46.6] 2023-10-18T20:29:44.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:44.424+0000 7fec111ed780 0 bucket checkpoint complete 2023-10-18T20:29:44.444 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-57 2023-10-18T20:29:44.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:29:44.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:29:44.447 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:29:44.511 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:29:44.511 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:29:44.548 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:44.545+0000 7f75db288780 1 waiting for incremental sync to catch up: 2023-10-18T20:29:44.548 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.921.6, , , 00000000001.609.6, 00000000001.928.6, , , 00000000002.678.6, , 00000000001.43.6] 2023-10-18T20:29:44.548 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.921.6, , , 00000000001.609.6, 00000000001.928.6, , , 00000000002.678.6, , 00000000002.48.6] 2023-10-18T20:29:49.549 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:49.547+0000 7f75db288780 1 waiting for incremental sync to catch up: 2023-10-18T20:29:49.549 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.921.6, , , 00000000001.609.6, 00000000001.928.6, , , 00000000002.678.6, , 00000000001.43.6] 2023-10-18T20:29:49.550 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.921.6, , , 00000000001.609.6, 00000000001.928.6, , , 00000000002.678.6, , 00000000002.48.6] 2023-10-18T20:29:54.550 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:54.549+0000 7f75db288780 1 waiting for incremental sync to catch up: 2023-10-18T20:29:54.551 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.921.6, , , 00000000001.609.6, 00000000001.928.6, , , 00000000002.678.6, , 00000000001.43.6] 2023-10-18T20:29:54.551 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.921.6, , , 00000000001.609.6, 00000000001.928.6, , , 00000000002.678.6, , 00000000002.48.6] 2023-10-18T20:29:59.551 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:59.550+0000 7f75db288780 1 bucket sync caught up with source: 2023-10-18T20:29:59.552 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.921.6, , , 00000000001.609.6, 00000000001.928.6, , , 00000000002.678.6, , 00000000002.48.6] 2023-10-18T20:29:59.552 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.921.6, , , 00000000001.609.6, 00000000001.928.6, , , 00000000002.678.6, , 00000000002.48.6] 2023-10-18T20:29:59.552 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:29:59.550+0000 7f75db288780 0 bucket checkpoint complete 2023-10-18T20:29:59.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-57 zones={a2, a1} 2023-10-18T20:29:59.569 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.570 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.570 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.571 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:29:59.571 DEBUG:boto:Data: 2023-10-18T20:29:59.572 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.572 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.572 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.573 DEBUG:boto:Params: {} 2023-10-18T20:29:59.574 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:29:59.574 DEBUG:boto:Token: None 2023-10-18T20:29:59.575 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/ 2023-10-18T20:29:59.575 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HUsrAVo37vtlFDAzlV8KgGSiZ1c= 2023-10-18T20:29:59.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HUsrAVo37vtlFDAzlV8KgGSiZ1c=', 'Content-Length': '0'} 2023-10-18T20:29:59.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7547b9639ad8cd6-0065304047-1a24-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.582 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.583 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.583 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.584 DEBUG:boto:Path: /bofrnf-57/ 2023-10-18T20:29:59.584 DEBUG:boto:Data: 2023-10-18T20:29:59.584 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.585 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.586 DEBUG:boto:Params: {} 2023-10-18T20:29:59.589 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:29:59.589 DEBUG:boto:Token: None 2023-10-18T20:29:59.590 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/ 2023-10-18T20:29:59.590 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HUsrAVo37vtlFDAzlV8KgGSiZ1c= 2023-10-18T20:29:59.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HUsrAVo37vtlFDAzlV8KgGSiZ1c=', 'Content-Length': '0'} 2023-10-18T20:29:59.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5931a6e430ffd79-0065304047-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:29:59.597 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.597 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.598 DEBUG:boto:path=/bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.598 DEBUG:boto:auth_path=/bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.599 DEBUG:boto:Method: GET 2023-10-18T20:29:59.599 DEBUG:boto:Path: /bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.600 DEBUG:boto:Data: 2023-10-18T20:29:59.600 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.600 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.601 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.601 DEBUG:boto:Params: {} 2023-10-18T20:29:59.602 DEBUG:boto:Token: None 2023-10-18T20:29:59.602 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/?versions 2023-10-18T20:29:59.603 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNtWP2nKEC6VYoy1VJ4o8E3NyDg= 2023-10-18T20:29:59.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNtWP2nKEC6VYoy1VJ4o8E3NyDg=', 'Content-Length': '0'} 2023-10-18T20:29:59.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cdb6501fc1d949a-0065304047-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.607 DEBUG:boto:b'bofrnf-57999falseanulltrue2023-10-18T20:27:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:27:07.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:27:07.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:27:07.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:27:08.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:27:08.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:27:08.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:27:08.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.608 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:29:59.609 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:29:59.609 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:29:59.609 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:29:59.610 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-18T20:29:59.610 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-18T20:29:59.611 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-18T20:29:59.611 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-18T20:29:59.611 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:29:59.612 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.612 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.613 DEBUG:boto:path=/bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.613 DEBUG:boto:auth_path=/bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.614 DEBUG:boto:Method: GET 2023-10-18T20:29:59.614 DEBUG:boto:Path: /bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.614 DEBUG:boto:Data: 2023-10-18T20:29:59.615 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.615 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.616 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.616 DEBUG:boto:Params: {} 2023-10-18T20:29:59.617 DEBUG:boto:Token: None 2023-10-18T20:29:59.617 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/?versions 2023-10-18T20:29:59.617 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNtWP2nKEC6VYoy1VJ4o8E3NyDg= 2023-10-18T20:29:59.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNtWP2nKEC6VYoy1VJ4o8E3NyDg=', 'Content-Length': '0'} 2023-10-18T20:29:59.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054143a7d27c83bf8-0065304047-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.621 DEBUG:boto:b'bofrnf-57999falseanulltrue2023-10-18T20:27:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:27:07.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:27:07.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:27:07.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:27:08.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:27:08.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:27:08.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:27:08.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.622 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-18T20:29:59.622 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-18T20:29:59.622 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-18T20:29:59.623 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-18T20:29:59.623 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-18T20:29:59.623 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-18T20:29:59.624 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-18T20:29:59.624 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-18T20:29:59.625 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.625 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.625 DEBUG:boto:path=/bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.626 DEBUG:boto:auth_path=/bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.626 DEBUG:boto:Method: GET 2023-10-18T20:29:59.627 DEBUG:boto:Path: /bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.627 DEBUG:boto:Data: 2023-10-18T20:29:59.628 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.628 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.629 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.629 DEBUG:boto:Params: {} 2023-10-18T20:29:59.630 DEBUG:boto:Token: None 2023-10-18T20:29:59.630 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/?versions 2023-10-18T20:29:59.631 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNtWP2nKEC6VYoy1VJ4o8E3NyDg= 2023-10-18T20:29:59.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNtWP2nKEC6VYoy1VJ4o8E3NyDg=', 'Content-Length': '0'} 2023-10-18T20:29:59.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003831c3ea35a45e39-0065304047-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.634 DEBUG:boto:b'bofrnf-57999falseanulltrue2023-10-18T20:27:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:27:07.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:27:07.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:27:07.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:27:08.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:27:08.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:27:08.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:27:08.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.635 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.636 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.636 DEBUG:boto:path=/bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.637 DEBUG:boto:auth_path=/bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.637 DEBUG:boto:Method: GET 2023-10-18T20:29:59.638 DEBUG:boto:Path: /bofrnf-57/?versions&max-keys=999 2023-10-18T20:29:59.638 DEBUG:boto:Data: 2023-10-18T20:29:59.639 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.639 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.640 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.640 DEBUG:boto:Params: {} 2023-10-18T20:29:59.641 DEBUG:boto:Token: None 2023-10-18T20:29:59.641 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/?versions 2023-10-18T20:29:59.642 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNtWP2nKEC6VYoy1VJ4o8E3NyDg= 2023-10-18T20:29:59.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNtWP2nKEC6VYoy1VJ4o8E3NyDg=', 'Content-Length': '0'} 2023-10-18T20:29:59.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000473d90fa98fd6794-0065304047-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.645 DEBUG:boto:b'bofrnf-57999falseanulltrue2023-10-18T20:27:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:27:07.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:27:07.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:27:07.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:27:08.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:27:08.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:27:08.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:27:08.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:29:59.647 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.647 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.648 DEBUG:boto:path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.648 DEBUG:boto:auth_path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.649 DEBUG:boto:Method: GET 2023-10-18T20:29:59.649 DEBUG:boto:Path: /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.650 DEBUG:boto:Data: 2023-10-18T20:29:59.650 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.651 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.652 DEBUG:boto:Params: {} 2023-10-18T20:29:59.652 DEBUG:boto:Token: None 2023-10-18T20:29:59.653 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.654 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PWfASIRoFgXxpQo8lqXpFYFfWp0= 2023-10-18T20:29:59.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PWfASIRoFgXxpQo8lqXpFYFfWp0=', 'Content-Length': '0'} 2023-10-18T20:29:59.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d019c21b55ce73b-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.657 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.658 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.658 DEBUG:boto:path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.659 DEBUG:boto:auth_path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.659 DEBUG:boto:Method: GET 2023-10-18T20:29:59.660 DEBUG:boto:Path: /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.660 DEBUG:boto:Data: 2023-10-18T20:29:59.661 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.662 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.662 DEBUG:boto:Params: {} 2023-10-18T20:29:59.663 DEBUG:boto:Token: None 2023-10-18T20:29:59.663 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PWfASIRoFgXxpQo8lqXpFYFfWp0= 2023-10-18T20:29:59.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PWfASIRoFgXxpQo8lqXpFYFfWp0=', 'Content-Length': '0'} 2023-10-18T20:29:59.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c9221f4899e4d21-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.667 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.668 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.668 DEBUG:boto:path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.669 DEBUG:boto:auth_path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.669 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.670 DEBUG:boto:Path: /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.670 DEBUG:boto:Data: 2023-10-18T20:29:59.671 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.671 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.672 DEBUG:boto:Params: {} 2023-10-18T20:29:59.672 DEBUG:boto:Token: None 2023-10-18T20:29:59.673 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OrGeMMUT5MBe9xEESZslFrwajG8= 2023-10-18T20:29:59.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OrGeMMUT5MBe9xEESZslFrwajG8=', 'Content-Length': '0'} 2023-10-18T20:29:59.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087f1162c2a0b43b3-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.676 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.677 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.677 DEBUG:boto:path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.678 DEBUG:boto:auth_path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.678 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.679 DEBUG:boto:Path: /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.679 DEBUG:boto:Data: 2023-10-18T20:29:59.680 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.680 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.681 DEBUG:boto:Params: {} 2023-10-18T20:29:59.681 DEBUG:boto:Token: None 2023-10-18T20:29:59.682 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.682 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OrGeMMUT5MBe9xEESZslFrwajG8= 2023-10-18T20:29:59.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OrGeMMUT5MBe9xEESZslFrwajG8=', 'Content-Length': '0'} 2023-10-18T20:29:59.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da1dec935cb94d5b-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:29:59.686 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.686 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.687 DEBUG:boto:path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.687 DEBUG:boto:auth_path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.688 DEBUG:boto:Method: GET 2023-10-18T20:29:59.688 DEBUG:boto:Path: /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.688 DEBUG:boto:Data: 2023-10-18T20:29:59.689 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.690 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.690 DEBUG:boto:Params: {} 2023-10-18T20:29:59.691 DEBUG:boto:Token: None 2023-10-18T20:29:59.691 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PWfASIRoFgXxpQo8lqXpFYFfWp0= 2023-10-18T20:29:59.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PWfASIRoFgXxpQo8lqXpFYFfWp0=', 'Content-Length': '0'} 2023-10-18T20:29:59.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000771a71f575f753e0-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.695 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.695 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.696 DEBUG:boto:path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.696 DEBUG:boto:auth_path=/bofrnf-57/a?versionId=null 2023-10-18T20:29:59.696 DEBUG:boto:Method: GET 2023-10-18T20:29:59.697 DEBUG:boto:Path: /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.697 DEBUG:boto:Data: 2023-10-18T20:29:59.698 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.699 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.699 DEBUG:boto:Params: {} 2023-10-18T20:29:59.699 DEBUG:boto:Token: None 2023-10-18T20:29:59.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a?versionId=null 2023-10-18T20:29:59.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PWfASIRoFgXxpQo8lqXpFYFfWp0= 2023-10-18T20:29:59.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PWfASIRoFgXxpQo8lqXpFYFfWp0=', 'Content-Length': '0'} 2023-10-18T20:29:59.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c77f3cf7ca58f7-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.704 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.704 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.704 DEBUG:boto:path=/bofrnf-57/?prefix=a 2023-10-18T20:29:59.705 DEBUG:boto:auth_path=/bofrnf-57/?prefix=a 2023-10-18T20:29:59.705 DEBUG:boto:Method: GET 2023-10-18T20:29:59.706 DEBUG:boto:Path: /bofrnf-57/?prefix=a 2023-10-18T20:29:59.706 DEBUG:boto:Data: 2023-10-18T20:29:59.707 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.707 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.708 DEBUG:boto:Params: {} 2023-10-18T20:29:59.708 DEBUG:boto:Token: None 2023-10-18T20:29:59.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/ 2023-10-18T20:29:59.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ= 2023-10-18T20:29:59.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ=', 'Content-Length': '0'} 2023-10-18T20:29:59.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b4a5e1ff1ac0014-0065304047-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.712 DEBUG:boto:b'bofrnf-57a1000falsea2023-10-18T20:27:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.713 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.714 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.714 DEBUG:boto:path=/bofrnf-57/?prefix=a 2023-10-18T20:29:59.714 DEBUG:boto:auth_path=/bofrnf-57/?prefix=a 2023-10-18T20:29:59.715 DEBUG:boto:Method: GET 2023-10-18T20:29:59.715 DEBUG:boto:Path: /bofrnf-57/?prefix=a 2023-10-18T20:29:59.716 DEBUG:boto:Data: 2023-10-18T20:29:59.716 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.717 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.717 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.718 DEBUG:boto:Params: {} 2023-10-18T20:29:59.718 DEBUG:boto:Token: None 2023-10-18T20:29:59.719 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/ 2023-10-18T20:29:59.719 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ= 2023-10-18T20:29:59.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ=', 'Content-Length': '0'} 2023-10-18T20:29:59.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000316630a2126daacb-0065304047-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.766 DEBUG:boto:b'bofrnf-57a1000falsea2023-10-18T20:27:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.767 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.768 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.768 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.769 DEBUG:boto:Path: /bofrnf-57/a 2023-10-18T20:29:59.769 DEBUG:boto:Data: 2023-10-18T20:29:59.770 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.771 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.771 DEBUG:boto:Params: {} 2023-10-18T20:29:59.772 DEBUG:boto:Token: None 2023-10-18T20:29:59.772 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a 2023-10-18T20:29:59.773 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EO2hqC4/XqGKhxHo7wx1O7RCvDs= 2023-10-18T20:29:59.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EO2hqC4/XqGKhxHo7wx1O7RCvDs=', 'Content-Length': '0'} 2023-10-18T20:29:59.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f99d62e7b136fa8-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.776 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.777 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.777 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.778 DEBUG:boto:Path: /bofrnf-57/a 2023-10-18T20:29:59.778 DEBUG:boto:Data: 2023-10-18T20:29:59.778 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.779 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.780 DEBUG:boto:Params: {} 2023-10-18T20:29:59.780 DEBUG:boto:Token: None 2023-10-18T20:29:59.780 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a 2023-10-18T20:29:59.781 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EO2hqC4/XqGKhxHo7wx1O7RCvDs= 2023-10-18T20:29:59.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EO2hqC4/XqGKhxHo7wx1O7RCvDs=', 'Content-Length': '0'} 2023-10-18T20:29:59.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cf2c192373b6232-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-18T20:29:59.784 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.785 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.785 DEBUG:boto:Method: GET 2023-10-18T20:29:59.786 DEBUG:boto:Path: /bofrnf-57/a 2023-10-18T20:29:59.786 DEBUG:boto:Data: 2023-10-18T20:29:59.787 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.787 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.788 DEBUG:boto:Params: {} 2023-10-18T20:29:59.788 DEBUG:boto:Token: None 2023-10-18T20:29:59.789 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a 2023-10-18T20:29:59.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rvXrCmduQiPcAh21D3LmBt6TOOE= 2023-10-18T20:29:59.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rvXrCmduQiPcAh21D3LmBt6TOOE=', 'Content-Length': '0'} 2023-10-18T20:29:59.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a443dece36b8048-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.792 DEBUG:boto:path=/bofrnf-57/a 2023-10-18T20:29:59.793 DEBUG:boto:auth_path=/bofrnf-57/a 2023-10-18T20:29:59.793 DEBUG:boto:Method: GET 2023-10-18T20:29:59.794 DEBUG:boto:Path: /bofrnf-57/a 2023-10-18T20:29:59.797 DEBUG:boto:Data: 2023-10-18T20:29:59.797 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.798 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.798 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.799 DEBUG:boto:Params: {} 2023-10-18T20:29:59.799 DEBUG:boto:Token: None 2023-10-18T20:29:59.801 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/a 2023-10-18T20:29:59.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rvXrCmduQiPcAh21D3LmBt6TOOE= 2023-10-18T20:29:59.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rvXrCmduQiPcAh21D3LmBt6TOOE=', 'Content-Length': '0'} 2023-10-18T20:29:59.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1a14b4562281c93-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.805 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.805 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.806 DEBUG:boto:path=/bofrnf-57/?prefix=a 2023-10-18T20:29:59.806 DEBUG:boto:auth_path=/bofrnf-57/?prefix=a 2023-10-18T20:29:59.806 DEBUG:boto:Method: GET 2023-10-18T20:29:59.807 DEBUG:boto:Path: /bofrnf-57/?prefix=a 2023-10-18T20:29:59.807 DEBUG:boto:Data: 2023-10-18T20:29:59.808 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.808 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.809 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.809 DEBUG:boto:Params: {} 2023-10-18T20:29:59.810 DEBUG:boto:Token: None 2023-10-18T20:29:59.810 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/ 2023-10-18T20:29:59.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ= 2023-10-18T20:29:59.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ=', 'Content-Length': '0'} 2023-10-18T20:29:59.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eddef484599641ee-0065304047-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.814 DEBUG:boto:b'bofrnf-57a1000falsea2023-10-18T20:27:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.815 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.815 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.816 DEBUG:boto:path=/bofrnf-57/?prefix=a 2023-10-18T20:29:59.816 DEBUG:boto:auth_path=/bofrnf-57/?prefix=a 2023-10-18T20:29:59.817 DEBUG:boto:Method: GET 2023-10-18T20:29:59.817 DEBUG:boto:Path: /bofrnf-57/?prefix=a 2023-10-18T20:29:59.817 DEBUG:boto:Data: 2023-10-18T20:29:59.818 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.818 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.819 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.819 DEBUG:boto:Params: {} 2023-10-18T20:29:59.820 DEBUG:boto:Token: None 2023-10-18T20:29:59.820 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/ 2023-10-18T20:29:59.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ= 2023-10-18T20:29:59.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ=', 'Content-Length': '0'} 2023-10-18T20:29:59.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000851f0513da92c639-0065304047-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.824 DEBUG:boto:b'bofrnf-57a1000falsea2023-10-18T20:27:07.790Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:29:59.825 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.825 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.825 DEBUG:boto:path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.826 DEBUG:boto:auth_path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.826 DEBUG:boto:Method: GET 2023-10-18T20:29:59.827 DEBUG:boto:Path: /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.827 DEBUG:boto:Data: 2023-10-18T20:29:59.828 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.828 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.829 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.829 DEBUG:boto:Params: {} 2023-10-18T20:29:59.830 DEBUG:boto:Token: None 2023-10-18T20:29:59.830 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.831 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqMSneVE25LE72yGSQ5tl6wvs8A= 2023-10-18T20:29:59.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqMSneVE25LE72yGSQ5tl6wvs8A=', 'Content-Length': '0'} 2023-10-18T20:29:59.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013279730cbd01c76-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.834 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.835 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.835 DEBUG:boto:path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.835 DEBUG:boto:auth_path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.836 DEBUG:boto:Method: GET 2023-10-18T20:29:59.836 DEBUG:boto:Path: /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.837 DEBUG:boto:Data: 2023-10-18T20:29:59.837 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.838 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.838 DEBUG:boto:Params: {} 2023-10-18T20:29:59.839 DEBUG:boto:Token: None 2023-10-18T20:29:59.839 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqMSneVE25LE72yGSQ5tl6wvs8A= 2023-10-18T20:29:59.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqMSneVE25LE72yGSQ5tl6wvs8A=', 'Content-Length': '0'} 2023-10-18T20:29:59.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d35021a11668ac5-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.843 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.843 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.844 DEBUG:boto:path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.844 DEBUG:boto:auth_path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.844 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.845 DEBUG:boto:Path: /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.845 DEBUG:boto:Data: 2023-10-18T20:29:59.846 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.846 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.846 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.847 DEBUG:boto:Params: {} 2023-10-18T20:29:59.847 DEBUG:boto:Token: None 2023-10-18T20:29:59.848 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I1juiBAyIAyptHziPzKEspn6Ypw= 2023-10-18T20:29:59.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I1juiBAyIAyptHziPzKEspn6Ypw=', 'Content-Length': '0'} 2023-10-18T20:29:59.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04510688b5f9dd3-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.851 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.852 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.852 DEBUG:boto:path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.852 DEBUG:boto:auth_path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.853 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.853 DEBUG:boto:Path: /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.854 DEBUG:boto:Data: 2023-10-18T20:29:59.854 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.854 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.855 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.855 DEBUG:boto:Params: {} 2023-10-18T20:29:59.856 DEBUG:boto:Token: None 2023-10-18T20:29:59.856 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.857 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I1juiBAyIAyptHziPzKEspn6Ypw= 2023-10-18T20:29:59.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I1juiBAyIAyptHziPzKEspn6Ypw=', 'Content-Length': '0'} 2023-10-18T20:29:59.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a20b755db46178ee-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:29:59.860 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.860 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.861 DEBUG:boto:path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.861 DEBUG:boto:auth_path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.862 DEBUG:boto:Method: GET 2023-10-18T20:29:59.862 DEBUG:boto:Path: /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.863 DEBUG:boto:Data: 2023-10-18T20:29:59.863 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.863 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.864 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.864 DEBUG:boto:Params: {} 2023-10-18T20:29:59.865 DEBUG:boto:Token: None 2023-10-18T20:29:59.865 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.866 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqMSneVE25LE72yGSQ5tl6wvs8A= 2023-10-18T20:29:59.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqMSneVE25LE72yGSQ5tl6wvs8A=', 'Content-Length': '0'} 2023-10-18T20:29:59.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009e49a42d704ea84-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.869 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.869 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.870 DEBUG:boto:path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.870 DEBUG:boto:auth_path=/bofrnf-57/b?versionId=null 2023-10-18T20:29:59.871 DEBUG:boto:Method: GET 2023-10-18T20:29:59.871 DEBUG:boto:Path: /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.871 DEBUG:boto:Data: 2023-10-18T20:29:59.872 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.872 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.873 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.873 DEBUG:boto:Params: {} 2023-10-18T20:29:59.874 DEBUG:boto:Token: None 2023-10-18T20:29:59.874 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b?versionId=null 2023-10-18T20:29:59.875 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqMSneVE25LE72yGSQ5tl6wvs8A= 2023-10-18T20:29:59.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqMSneVE25LE72yGSQ5tl6wvs8A=', 'Content-Length': '0'} 2023-10-18T20:29:59.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4f6b129cbfb1f13-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.918 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.919 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.919 DEBUG:boto:path=/bofrnf-57/?prefix=b 2023-10-18T20:29:59.920 DEBUG:boto:auth_path=/bofrnf-57/?prefix=b 2023-10-18T20:29:59.920 DEBUG:boto:Method: GET 2023-10-18T20:29:59.921 DEBUG:boto:Path: /bofrnf-57/?prefix=b 2023-10-18T20:29:59.921 DEBUG:boto:Data: 2023-10-18T20:29:59.921 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.922 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.922 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.923 DEBUG:boto:Params: {} 2023-10-18T20:29:59.923 DEBUG:boto:Token: None 2023-10-18T20:29:59.924 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/ 2023-10-18T20:29:59.924 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ= 2023-10-18T20:29:59.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ=', 'Content-Length': '0'} 2023-10-18T20:29:59.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4b9d1d43418d96f-0065304047-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.928 DEBUG:boto:b'bofrnf-57b1000falseb2023-10-18T20:27:07.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.928 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.929 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.929 DEBUG:boto:path=/bofrnf-57/?prefix=b 2023-10-18T20:29:59.930 DEBUG:boto:auth_path=/bofrnf-57/?prefix=b 2023-10-18T20:29:59.930 DEBUG:boto:Method: GET 2023-10-18T20:29:59.931 DEBUG:boto:Path: /bofrnf-57/?prefix=b 2023-10-18T20:29:59.932 DEBUG:boto:Data: 2023-10-18T20:29:59.932 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.932 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.933 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.933 DEBUG:boto:Params: {} 2023-10-18T20:29:59.934 DEBUG:boto:Token: None 2023-10-18T20:29:59.934 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/ 2023-10-18T20:29:59.935 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ= 2023-10-18T20:29:59.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ=', 'Content-Length': '0'} 2023-10-18T20:29:59.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8bf2fc27cb7c489-0065304047-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.938 DEBUG:boto:b'bofrnf-57b1000falseb2023-10-18T20:27:07.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:29:59.939 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.939 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.940 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.940 DEBUG:boto:Path: /bofrnf-57/b 2023-10-18T20:29:59.941 DEBUG:boto:Data: 2023-10-18T20:29:59.941 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.942 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.942 DEBUG:boto:Params: {} 2023-10-18T20:29:59.943 DEBUG:boto:Token: None 2023-10-18T20:29:59.943 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b 2023-10-18T20:29:59.944 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oAvGXx71P+RZd4766AttmCQ85OQ= 2023-10-18T20:29:59.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oAvGXx71P+RZd4766AttmCQ85OQ=', 'Content-Length': '0'} 2023-10-18T20:29:59.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb801cafc1ba2efc-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.946 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.947 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.947 DEBUG:boto:Method: HEAD 2023-10-18T20:29:59.948 DEBUG:boto:Path: /bofrnf-57/b 2023-10-18T20:29:59.948 DEBUG:boto:Data: 2023-10-18T20:29:59.949 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.950 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.950 DEBUG:boto:Params: {} 2023-10-18T20:29:59.951 DEBUG:boto:Token: None 2023-10-18T20:29:59.951 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b 2023-10-18T20:29:59.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oAvGXx71P+RZd4766AttmCQ85OQ= 2023-10-18T20:29:59.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oAvGXx71P+RZd4766AttmCQ85OQ=', 'Content-Length': '0'} 2023-10-18T20:29:59.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b058907a4b911ae-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-18T20:29:59.955 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.955 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.956 DEBUG:boto:Method: GET 2023-10-18T20:29:59.956 DEBUG:boto:Path: /bofrnf-57/b 2023-10-18T20:29:59.957 DEBUG:boto:Data: 2023-10-18T20:29:59.957 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.958 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.958 DEBUG:boto:Params: {} 2023-10-18T20:29:59.959 DEBUG:boto:Token: None 2023-10-18T20:29:59.959 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b 2023-10-18T20:29:59.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:StQlOi2uKsa7TAO2P2KowGgBdEM= 2023-10-18T20:29:59.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:StQlOi2uKsa7TAO2P2KowGgBdEM=', 'Content-Length': '0'} 2023-10-18T20:29:59.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009accc67608c22a6f-0065304047-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.963 DEBUG:boto:path=/bofrnf-57/b 2023-10-18T20:29:59.963 DEBUG:boto:auth_path=/bofrnf-57/b 2023-10-18T20:29:59.964 DEBUG:boto:Method: GET 2023-10-18T20:29:59.964 DEBUG:boto:Path: /bofrnf-57/b 2023-10-18T20:29:59.965 DEBUG:boto:Data: 2023-10-18T20:29:59.965 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:29:59.966 DEBUG:boto:Port: 8000 2023-10-18T20:29:59.966 DEBUG:boto:Params: {} 2023-10-18T20:29:59.967 DEBUG:boto:Token: None 2023-10-18T20:29:59.967 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/b 2023-10-18T20:29:59.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:StQlOi2uKsa7TAO2P2KowGgBdEM= 2023-10-18T20:29:59.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:StQlOi2uKsa7TAO2P2KowGgBdEM=', 'Content-Length': '0'} 2023-10-18T20:29:59.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fda58ecf09f34ccb-0065304047-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:29:59.971 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:29:59.971 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:29:59.972 DEBUG:boto:path=/bofrnf-57/?prefix=b 2023-10-18T20:29:59.972 DEBUG:boto:auth_path=/bofrnf-57/?prefix=b 2023-10-18T20:29:59.973 DEBUG:boto:Method: GET 2023-10-18T20:29:59.973 DEBUG:boto:Path: /bofrnf-57/?prefix=b 2023-10-18T20:29:59.973 DEBUG:boto:Data: 2023-10-18T20:29:59.974 DEBUG:boto:Headers: {} 2023-10-18T20:29:59.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:29:59.975 DEBUG:boto:Port: 8001 2023-10-18T20:29:59.975 DEBUG:boto:Params: {} 2023-10-18T20:29:59.976 DEBUG:boto:Token: None 2023-10-18T20:29:59.976 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:29:59 GMT /bofrnf-57/ 2023-10-18T20:29:59.977 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ= 2023-10-18T20:29:59.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:29:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jo2QF3uxb0CSBcNxQDftfI9g+NQ=', 'Content-Length': '0'} 2023-10-18T20:29:59.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008c220a65ed2ee43-0065304047-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:29:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.022 DEBUG:boto:b'bofrnf-57b1000falseb2023-10-18T20:27:07.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.023 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.024 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.024 DEBUG:boto:path=/bofrnf-57/?prefix=b 2023-10-18T20:30:00.025 DEBUG:boto:auth_path=/bofrnf-57/?prefix=b 2023-10-18T20:30:00.025 DEBUG:boto:Method: GET 2023-10-18T20:30:00.026 DEBUG:boto:Path: /bofrnf-57/?prefix=b 2023-10-18T20:30:00.026 DEBUG:boto:Data: 2023-10-18T20:30:00.026 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.027 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.027 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.028 DEBUG:boto:Params: {} 2023-10-18T20:30:00.028 DEBUG:boto:Token: None 2023-10-18T20:30:00.029 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.029 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ad3ac2fb24cc374-0065304048-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.033 DEBUG:boto:b'bofrnf-57b1000falseb2023-10-18T20:27:07.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:30:00.034 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.034 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.035 DEBUG:boto:path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.035 DEBUG:boto:auth_path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.036 DEBUG:boto:Method: GET 2023-10-18T20:30:00.036 DEBUG:boto:Path: /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.036 DEBUG:boto:Data: 2023-10-18T20:30:00.037 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.037 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.038 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.038 DEBUG:boto:Params: {} 2023-10-18T20:30:00.039 DEBUG:boto:Token: None 2023-10-18T20:30:00.039 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AGbiRpKzrMdJap7c0GmZuop60so= 2023-10-18T20:30:00.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AGbiRpKzrMdJap7c0GmZuop60so=', 'Content-Length': '0'} 2023-10-18T20:30:00.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e49e308b88952776-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.043 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.043 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.044 DEBUG:boto:path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.044 DEBUG:boto:auth_path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.045 DEBUG:boto:Method: GET 2023-10-18T20:30:00.045 DEBUG:boto:Path: /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.045 DEBUG:boto:Data: 2023-10-18T20:30:00.046 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.046 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.047 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.047 DEBUG:boto:Params: {} 2023-10-18T20:30:00.048 DEBUG:boto:Token: None 2023-10-18T20:30:00.048 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AGbiRpKzrMdJap7c0GmZuop60so= 2023-10-18T20:30:00.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AGbiRpKzrMdJap7c0GmZuop60so=', 'Content-Length': '0'} 2023-10-18T20:30:00.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a03f328529215b01-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.052 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.053 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.053 DEBUG:boto:path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.053 DEBUG:boto:auth_path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.054 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.054 DEBUG:boto:Path: /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.055 DEBUG:boto:Data: 2023-10-18T20:30:00.055 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.056 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.056 DEBUG:boto:Params: {} 2023-10-18T20:30:00.057 DEBUG:boto:Token: None 2023-10-18T20:30:00.057 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.058 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CDy7vWIAPTUQUAO5377uoaLwvIk= 2023-10-18T20:30:00.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CDy7vWIAPTUQUAO5377uoaLwvIk=', 'Content-Length': '0'} 2023-10-18T20:30:00.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072cc14401d5c587e-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.061 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.061 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.062 DEBUG:boto:path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.062 DEBUG:boto:auth_path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.063 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.063 DEBUG:boto:Path: /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.063 DEBUG:boto:Data: 2023-10-18T20:30:00.064 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.065 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.065 DEBUG:boto:Params: {} 2023-10-18T20:30:00.065 DEBUG:boto:Token: None 2023-10-18T20:30:00.066 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.066 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CDy7vWIAPTUQUAO5377uoaLwvIk= 2023-10-18T20:30:00.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CDy7vWIAPTUQUAO5377uoaLwvIk=', 'Content-Length': '0'} 2023-10-18T20:30:00.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b21f3ab2781312fe-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:30:00.070 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.070 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.071 DEBUG:boto:path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.071 DEBUG:boto:auth_path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.072 DEBUG:boto:Method: GET 2023-10-18T20:30:00.072 DEBUG:boto:Path: /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.073 DEBUG:boto:Data: 2023-10-18T20:30:00.073 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.074 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.074 DEBUG:boto:Params: {} 2023-10-18T20:30:00.075 DEBUG:boto:Token: None 2023-10-18T20:30:00.075 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.076 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AGbiRpKzrMdJap7c0GmZuop60so= 2023-10-18T20:30:00.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AGbiRpKzrMdJap7c0GmZuop60so=', 'Content-Length': '0'} 2023-10-18T20:30:00.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b1a25fc971d3cb-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.079 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.079 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.080 DEBUG:boto:path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.080 DEBUG:boto:auth_path=/bofrnf-57/c?versionId=null 2023-10-18T20:30:00.081 DEBUG:boto:Method: GET 2023-10-18T20:30:00.081 DEBUG:boto:Path: /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.081 DEBUG:boto:Data: 2023-10-18T20:30:00.082 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.083 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.083 DEBUG:boto:Params: {} 2023-10-18T20:30:00.084 DEBUG:boto:Token: None 2023-10-18T20:30:00.084 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c?versionId=null 2023-10-18T20:30:00.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AGbiRpKzrMdJap7c0GmZuop60so= 2023-10-18T20:30:00.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AGbiRpKzrMdJap7c0GmZuop60so=', 'Content-Length': '0'} 2023-10-18T20:30:00.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e93a9af06855340-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.088 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.088 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.089 DEBUG:boto:path=/bofrnf-57/?prefix=c 2023-10-18T20:30:00.089 DEBUG:boto:auth_path=/bofrnf-57/?prefix=c 2023-10-18T20:30:00.090 DEBUG:boto:Method: GET 2023-10-18T20:30:00.090 DEBUG:boto:Path: /bofrnf-57/?prefix=c 2023-10-18T20:30:00.091 DEBUG:boto:Data: 2023-10-18T20:30:00.091 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.092 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.092 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.092 DEBUG:boto:Params: {} 2023-10-18T20:30:00.093 DEBUG:boto:Token: None 2023-10-18T20:30:00.093 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.094 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003219d94ab5aaa5b8-0065304048-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.142 DEBUG:boto:b'bofrnf-57c1000falsec2023-10-18T20:27:07.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.143 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.144 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.144 DEBUG:boto:path=/bofrnf-57/?prefix=c 2023-10-18T20:30:00.145 DEBUG:boto:auth_path=/bofrnf-57/?prefix=c 2023-10-18T20:30:00.145 DEBUG:boto:Method: GET 2023-10-18T20:30:00.145 DEBUG:boto:Path: /bofrnf-57/?prefix=c 2023-10-18T20:30:00.146 DEBUG:boto:Data: 2023-10-18T20:30:00.146 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.147 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.147 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.148 DEBUG:boto:Params: {} 2023-10-18T20:30:00.148 DEBUG:boto:Token: None 2023-10-18T20:30:00.149 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.149 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000519004fac62f3bd0-0065304048-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.153 DEBUG:boto:b'bofrnf-57c1000falsec2023-10-18T20:27:07.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.153 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.154 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.154 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.155 DEBUG:boto:Path: /bofrnf-57/c 2023-10-18T20:30:00.155 DEBUG:boto:Data: 2023-10-18T20:30:00.155 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.156 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.156 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.156 DEBUG:boto:Params: {} 2023-10-18T20:30:00.157 DEBUG:boto:Token: None 2023-10-18T20:30:00.157 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c 2023-10-18T20:30:00.158 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mFKKgMW2e9i0lC8iNFNdZQfBU0c= 2023-10-18T20:30:00.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mFKKgMW2e9i0lC8iNFNdZQfBU0c=', 'Content-Length': '0'} 2023-10-18T20:30:00.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bd3d5afb195958a-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.161 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.161 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.162 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.162 DEBUG:boto:Path: /bofrnf-57/c 2023-10-18T20:30:00.163 DEBUG:boto:Data: 2023-10-18T20:30:00.163 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.163 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.164 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.164 DEBUG:boto:Params: {} 2023-10-18T20:30:00.165 DEBUG:boto:Token: None 2023-10-18T20:30:00.165 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c 2023-10-18T20:30:00.165 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mFKKgMW2e9i0lC8iNFNdZQfBU0c= 2023-10-18T20:30:00.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mFKKgMW2e9i0lC8iNFNdZQfBU0c=', 'Content-Length': '0'} 2023-10-18T20:30:00.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b022412c7163ef9-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-18T20:30:00.169 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.169 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.170 DEBUG:boto:Method: GET 2023-10-18T20:30:00.170 DEBUG:boto:Path: /bofrnf-57/c 2023-10-18T20:30:00.171 DEBUG:boto:Data: 2023-10-18T20:30:00.171 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.172 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.172 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.173 DEBUG:boto:Params: {} 2023-10-18T20:30:00.173 DEBUG:boto:Token: None 2023-10-18T20:30:00.174 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c 2023-10-18T20:30:00.174 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xk15rJi+anAG8rkVg0odqM64HI0= 2023-10-18T20:30:00.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xk15rJi+anAG8rkVg0odqM64HI0=', 'Content-Length': '0'} 2023-10-18T20:30:00.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002654ef850b1ef60f-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.177 DEBUG:boto:path=/bofrnf-57/c 2023-10-18T20:30:00.178 DEBUG:boto:auth_path=/bofrnf-57/c 2023-10-18T20:30:00.178 DEBUG:boto:Method: GET 2023-10-18T20:30:00.179 DEBUG:boto:Path: /bofrnf-57/c 2023-10-18T20:30:00.179 DEBUG:boto:Data: 2023-10-18T20:30:00.179 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.180 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.180 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.181 DEBUG:boto:Params: {} 2023-10-18T20:30:00.181 DEBUG:boto:Token: None 2023-10-18T20:30:00.182 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/c 2023-10-18T20:30:00.182 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xk15rJi+anAG8rkVg0odqM64HI0= 2023-10-18T20:30:00.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xk15rJi+anAG8rkVg0odqM64HI0=', 'Content-Length': '0'} 2023-10-18T20:30:00.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d68d0cf4328014d-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.185 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.186 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.186 DEBUG:boto:path=/bofrnf-57/?prefix=c 2023-10-18T20:30:00.187 DEBUG:boto:auth_path=/bofrnf-57/?prefix=c 2023-10-18T20:30:00.187 DEBUG:boto:Method: GET 2023-10-18T20:30:00.188 DEBUG:boto:Path: /bofrnf-57/?prefix=c 2023-10-18T20:30:00.188 DEBUG:boto:Data: 2023-10-18T20:30:00.189 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.189 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.189 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.190 DEBUG:boto:Params: {} 2023-10-18T20:30:00.190 DEBUG:boto:Token: None 2023-10-18T20:30:00.191 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.191 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000700b989bd273ac44-0065304048-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.195 DEBUG:boto:b'bofrnf-57c1000falsec2023-10-18T20:27:07.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.195 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.196 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.196 DEBUG:boto:path=/bofrnf-57/?prefix=c 2023-10-18T20:30:00.197 DEBUG:boto:auth_path=/bofrnf-57/?prefix=c 2023-10-18T20:30:00.197 DEBUG:boto:Method: GET 2023-10-18T20:30:00.197 DEBUG:boto:Path: /bofrnf-57/?prefix=c 2023-10-18T20:30:00.198 DEBUG:boto:Data: 2023-10-18T20:30:00.198 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.199 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.200 DEBUG:boto:Params: {} 2023-10-18T20:30:00.200 DEBUG:boto:Token: None 2023-10-18T20:30:00.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc23690b2100600f-0065304048-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.204 DEBUG:boto:b'bofrnf-57c1000falsec2023-10-18T20:27:07.842Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:30:00.205 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.206 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.206 DEBUG:boto:path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.206 DEBUG:boto:auth_path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.207 DEBUG:boto:Method: GET 2023-10-18T20:30:00.207 DEBUG:boto:Path: /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.208 DEBUG:boto:Data: 2023-10-18T20:30:00.208 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.209 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.209 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.209 DEBUG:boto:Params: {} 2023-10-18T20:30:00.210 DEBUG:boto:Token: None 2023-10-18T20:30:00.210 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3stWNSTAyP3aEfhWyle5dUWT+kc= 2023-10-18T20:30:00.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3stWNSTAyP3aEfhWyle5dUWT+kc=', 'Content-Length': '0'} 2023-10-18T20:30:00.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ef7a1efab740762-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.214 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.215 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.215 DEBUG:boto:path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.215 DEBUG:boto:auth_path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.216 DEBUG:boto:Method: GET 2023-10-18T20:30:00.216 DEBUG:boto:Path: /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.217 DEBUG:boto:Data: 2023-10-18T20:30:00.217 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.218 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.219 DEBUG:boto:Params: {} 2023-10-18T20:30:00.219 DEBUG:boto:Token: None 2023-10-18T20:30:00.219 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3stWNSTAyP3aEfhWyle5dUWT+kc= 2023-10-18T20:30:00.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3stWNSTAyP3aEfhWyle5dUWT+kc=', 'Content-Length': '0'} 2023-10-18T20:30:00.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b4d04992d9ef30f-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.223 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.224 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.224 DEBUG:boto:path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.224 DEBUG:boto:auth_path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.225 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.225 DEBUG:boto:Path: /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.226 DEBUG:boto:Data: 2023-10-18T20:30:00.226 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.227 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.227 DEBUG:boto:Params: {} 2023-10-18T20:30:00.228 DEBUG:boto:Token: None 2023-10-18T20:30:00.228 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.229 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+lOEe0g1PWWwMdJC98ynbsTM6gA= 2023-10-18T20:30:00.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+lOEe0g1PWWwMdJC98ynbsTM6gA=', 'Content-Length': '0'} 2023-10-18T20:30:00.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff34bcd1c1e1e29d-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.232 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.232 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.233 DEBUG:boto:path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.233 DEBUG:boto:auth_path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.233 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.234 DEBUG:boto:Path: /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.234 DEBUG:boto:Data: 2023-10-18T20:30:00.235 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.235 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.236 DEBUG:boto:Params: {} 2023-10-18T20:30:00.236 DEBUG:boto:Token: None 2023-10-18T20:30:00.237 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+lOEe0g1PWWwMdJC98ynbsTM6gA= 2023-10-18T20:30:00.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+lOEe0g1PWWwMdJC98ynbsTM6gA=', 'Content-Length': '0'} 2023-10-18T20:30:00.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006da84474cdad618b-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:30:00.241 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.241 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.242 DEBUG:boto:path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.242 DEBUG:boto:auth_path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.243 DEBUG:boto:Method: GET 2023-10-18T20:30:00.243 DEBUG:boto:Path: /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.244 DEBUG:boto:Data: 2023-10-18T20:30:00.244 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.245 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.245 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.245 DEBUG:boto:Params: {} 2023-10-18T20:30:00.246 DEBUG:boto:Token: None 2023-10-18T20:30:00.246 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.247 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3stWNSTAyP3aEfhWyle5dUWT+kc= 2023-10-18T20:30:00.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3stWNSTAyP3aEfhWyle5dUWT+kc=', 'Content-Length': '0'} 2023-10-18T20:30:00.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022dce5041d0be636-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.290 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.291 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.291 DEBUG:boto:path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.292 DEBUG:boto:auth_path=/bofrnf-57/d?versionId=null 2023-10-18T20:30:00.292 DEBUG:boto:Method: GET 2023-10-18T20:30:00.293 DEBUG:boto:Path: /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.293 DEBUG:boto:Data: 2023-10-18T20:30:00.294 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.295 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.295 DEBUG:boto:Params: {} 2023-10-18T20:30:00.296 DEBUG:boto:Token: None 2023-10-18T20:30:00.296 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d?versionId=null 2023-10-18T20:30:00.297 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3stWNSTAyP3aEfhWyle5dUWT+kc= 2023-10-18T20:30:00.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3stWNSTAyP3aEfhWyle5dUWT+kc=', 'Content-Length': '0'} 2023-10-18T20:30:00.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bcdebee8942f543-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.300 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.300 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.301 DEBUG:boto:path=/bofrnf-57/?prefix=d 2023-10-18T20:30:00.301 DEBUG:boto:auth_path=/bofrnf-57/?prefix=d 2023-10-18T20:30:00.302 DEBUG:boto:Method: GET 2023-10-18T20:30:00.302 DEBUG:boto:Path: /bofrnf-57/?prefix=d 2023-10-18T20:30:00.302 DEBUG:boto:Data: 2023-10-18T20:30:00.303 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.303 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.304 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.304 DEBUG:boto:Params: {} 2023-10-18T20:30:00.305 DEBUG:boto:Token: None 2023-10-18T20:30:00.305 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.306 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e16ad8f055e93eba-0065304048-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.309 DEBUG:boto:b'bofrnf-57d1000falsed2023-10-18T20:27:07.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.310 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.311 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.311 DEBUG:boto:path=/bofrnf-57/?prefix=d 2023-10-18T20:30:00.312 DEBUG:boto:auth_path=/bofrnf-57/?prefix=d 2023-10-18T20:30:00.312 DEBUG:boto:Method: GET 2023-10-18T20:30:00.313 DEBUG:boto:Path: /bofrnf-57/?prefix=d 2023-10-18T20:30:00.313 DEBUG:boto:Data: 2023-10-18T20:30:00.313 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.314 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.315 DEBUG:boto:Params: {} 2023-10-18T20:30:00.316 DEBUG:boto:Token: None 2023-10-18T20:30:00.316 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.317 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d351c051eb2504b-0065304048-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.366 DEBUG:boto:b'bofrnf-57d1000falsed2023-10-18T20:27:07.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.367 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.368 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.368 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.369 DEBUG:boto:Path: /bofrnf-57/d 2023-10-18T20:30:00.369 DEBUG:boto:Data: 2023-10-18T20:30:00.369 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.370 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.371 DEBUG:boto:Params: {} 2023-10-18T20:30:00.371 DEBUG:boto:Token: None 2023-10-18T20:30:00.372 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d 2023-10-18T20:30:00.372 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SPKb1JrrqvRPo7U4Z6eNPn0poQQ= 2023-10-18T20:30:00.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SPKb1JrrqvRPo7U4Z6eNPn0poQQ=', 'Content-Length': '0'} 2023-10-18T20:30:00.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ac3c524ea7b55fc-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.375 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.376 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.376 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.377 DEBUG:boto:Path: /bofrnf-57/d 2023-10-18T20:30:00.377 DEBUG:boto:Data: 2023-10-18T20:30:00.377 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.378 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.378 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.379 DEBUG:boto:Params: {} 2023-10-18T20:30:00.379 DEBUG:boto:Token: None 2023-10-18T20:30:00.379 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d 2023-10-18T20:30:00.380 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SPKb1JrrqvRPo7U4Z6eNPn0poQQ= 2023-10-18T20:30:00.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SPKb1JrrqvRPo7U4Z6eNPn0poQQ=', 'Content-Length': '0'} 2023-10-18T20:30:00.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000465ac3b24861f636-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-18T20:30:00.383 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.384 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.384 DEBUG:boto:Method: GET 2023-10-18T20:30:00.385 DEBUG:boto:Path: /bofrnf-57/d 2023-10-18T20:30:00.385 DEBUG:boto:Data: 2023-10-18T20:30:00.385 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.386 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.387 DEBUG:boto:Params: {} 2023-10-18T20:30:00.387 DEBUG:boto:Token: None 2023-10-18T20:30:00.388 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d 2023-10-18T20:30:00.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZkuMEIATs6h/q6TuzgkgmFA9VYo= 2023-10-18T20:30:00.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZkuMEIATs6h/q6TuzgkgmFA9VYo=', 'Content-Length': '0'} 2023-10-18T20:30:00.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000717f9d6b1c72fe97-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.392 DEBUG:boto:path=/bofrnf-57/d 2023-10-18T20:30:00.392 DEBUG:boto:auth_path=/bofrnf-57/d 2023-10-18T20:30:00.393 DEBUG:boto:Method: GET 2023-10-18T20:30:00.394 DEBUG:boto:Path: /bofrnf-57/d 2023-10-18T20:30:00.394 DEBUG:boto:Data: 2023-10-18T20:30:00.395 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.395 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.396 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.396 DEBUG:boto:Params: {} 2023-10-18T20:30:00.397 DEBUG:boto:Token: None 2023-10-18T20:30:00.397 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/d 2023-10-18T20:30:00.398 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZkuMEIATs6h/q6TuzgkgmFA9VYo= 2023-10-18T20:30:00.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZkuMEIATs6h/q6TuzgkgmFA9VYo=', 'Content-Length': '0'} 2023-10-18T20:30:00.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba89a21059ba16cc-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.401 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.402 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.402 DEBUG:boto:path=/bofrnf-57/?prefix=d 2023-10-18T20:30:00.403 DEBUG:boto:auth_path=/bofrnf-57/?prefix=d 2023-10-18T20:30:00.403 DEBUG:boto:Method: GET 2023-10-18T20:30:00.404 DEBUG:boto:Path: /bofrnf-57/?prefix=d 2023-10-18T20:30:00.404 DEBUG:boto:Data: 2023-10-18T20:30:00.405 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.405 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.406 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.406 DEBUG:boto:Params: {} 2023-10-18T20:30:00.407 DEBUG:boto:Token: None 2023-10-18T20:30:00.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a6a278c951545b-0065304048-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.458 DEBUG:boto:b'bofrnf-57d1000falsed2023-10-18T20:27:07.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.459 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.460 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.460 DEBUG:boto:path=/bofrnf-57/?prefix=d 2023-10-18T20:30:00.461 DEBUG:boto:auth_path=/bofrnf-57/?prefix=d 2023-10-18T20:30:00.461 DEBUG:boto:Method: GET 2023-10-18T20:30:00.461 DEBUG:boto:Path: /bofrnf-57/?prefix=d 2023-10-18T20:30:00.462 DEBUG:boto:Data: 2023-10-18T20:30:00.462 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.463 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.464 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.464 DEBUG:boto:Params: {} 2023-10-18T20:30:00.464 DEBUG:boto:Token: None 2023-10-18T20:30:00.465 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.465 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005390d222c366036e-0065304048-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.469 DEBUG:boto:b'bofrnf-57d1000falsed2023-10-18T20:27:07.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:30:00.470 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.471 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.471 DEBUG:boto:path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.472 DEBUG:boto:auth_path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.472 DEBUG:boto:Method: GET 2023-10-18T20:30:00.473 DEBUG:boto:Path: /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.473 DEBUG:boto:Data: 2023-10-18T20:30:00.474 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.475 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.475 DEBUG:boto:Params: {} 2023-10-18T20:30:00.475 DEBUG:boto:Token: None 2023-10-18T20:30:00.476 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BCcYgHCvziBUCQbdVbdSUJOBVQI= 2023-10-18T20:30:00.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BCcYgHCvziBUCQbdVbdSUJOBVQI=', 'Content-Length': '0'} 2023-10-18T20:30:00.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab7ff9220f89760-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.480 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.480 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.481 DEBUG:boto:path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.481 DEBUG:boto:auth_path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.482 DEBUG:boto:Method: GET 2023-10-18T20:30:00.483 DEBUG:boto:Path: /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.483 DEBUG:boto:Data: 2023-10-18T20:30:00.483 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.484 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.484 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.485 DEBUG:boto:Params: {} 2023-10-18T20:30:00.486 DEBUG:boto:Token: None 2023-10-18T20:30:00.486 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.487 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BCcYgHCvziBUCQbdVbdSUJOBVQI= 2023-10-18T20:30:00.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BCcYgHCvziBUCQbdVbdSUJOBVQI=', 'Content-Length': '0'} 2023-10-18T20:30:00.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092072889dfd2b738-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.490 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.491 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.491 DEBUG:boto:path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.491 DEBUG:boto:auth_path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.492 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.492 DEBUG:boto:Path: /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.493 DEBUG:boto:Data: 2023-10-18T20:30:00.493 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.494 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.495 DEBUG:boto:Params: {} 2023-10-18T20:30:00.495 DEBUG:boto:Token: None 2023-10-18T20:30:00.496 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.496 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0U5mWc2q3OMUyKuxOMh2yxynll8= 2023-10-18T20:30:00.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0U5mWc2q3OMUyKuxOMh2yxynll8=', 'Content-Length': '0'} 2023-10-18T20:30:00.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4929f3d19eaf4e-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.499 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.499 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.500 DEBUG:boto:path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.500 DEBUG:boto:auth_path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.501 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.501 DEBUG:boto:Path: /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.501 DEBUG:boto:Data: 2023-10-18T20:30:00.502 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.502 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.503 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.503 DEBUG:boto:Params: {} 2023-10-18T20:30:00.504 DEBUG:boto:Token: None 2023-10-18T20:30:00.504 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.504 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0U5mWc2q3OMUyKuxOMh2yxynll8= 2023-10-18T20:30:00.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0U5mWc2q3OMUyKuxOMh2yxynll8=', 'Content-Length': '0'} 2023-10-18T20:30:00.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000230cd90934c6fed9-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:30:00.508 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.508 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.509 DEBUG:boto:path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.509 DEBUG:boto:auth_path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.509 DEBUG:boto:Method: GET 2023-10-18T20:30:00.510 DEBUG:boto:Path: /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.510 DEBUG:boto:Data: 2023-10-18T20:30:00.511 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.511 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.512 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.512 DEBUG:boto:Params: {} 2023-10-18T20:30:00.513 DEBUG:boto:Token: None 2023-10-18T20:30:00.513 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.514 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BCcYgHCvziBUCQbdVbdSUJOBVQI= 2023-10-18T20:30:00.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BCcYgHCvziBUCQbdVbdSUJOBVQI=', 'Content-Length': '0'} 2023-10-18T20:30:00.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5db59e9106da433-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.558 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.559 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.559 DEBUG:boto:path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.560 DEBUG:boto:auth_path=/bofrnf-57/e?versionId=null 2023-10-18T20:30:00.560 DEBUG:boto:Method: GET 2023-10-18T20:30:00.561 DEBUG:boto:Path: /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.561 DEBUG:boto:Data: 2023-10-18T20:30:00.561 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.562 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.563 DEBUG:boto:Params: {} 2023-10-18T20:30:00.563 DEBUG:boto:Token: None 2023-10-18T20:30:00.564 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e?versionId=null 2023-10-18T20:30:00.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BCcYgHCvziBUCQbdVbdSUJOBVQI= 2023-10-18T20:30:00.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BCcYgHCvziBUCQbdVbdSUJOBVQI=', 'Content-Length': '0'} 2023-10-18T20:30:00.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018bd8e3f2695f7e2-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.568 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.568 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.569 DEBUG:boto:path=/bofrnf-57/?prefix=e 2023-10-18T20:30:00.569 DEBUG:boto:auth_path=/bofrnf-57/?prefix=e 2023-10-18T20:30:00.569 DEBUG:boto:Method: GET 2023-10-18T20:30:00.570 DEBUG:boto:Path: /bofrnf-57/?prefix=e 2023-10-18T20:30:00.570 DEBUG:boto:Data: 2023-10-18T20:30:00.571 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.571 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.572 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.572 DEBUG:boto:Params: {} 2023-10-18T20:30:00.572 DEBUG:boto:Token: None 2023-10-18T20:30:00.573 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.573 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b4a86142627ad7b-0065304048-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.577 DEBUG:boto:b'bofrnf-57e1000falsee2023-10-18T20:27:08.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.578 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.578 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.579 DEBUG:boto:path=/bofrnf-57/?prefix=e 2023-10-18T20:30:00.579 DEBUG:boto:auth_path=/bofrnf-57/?prefix=e 2023-10-18T20:30:00.579 DEBUG:boto:Method: GET 2023-10-18T20:30:00.581 DEBUG:boto:Path: /bofrnf-57/?prefix=e 2023-10-18T20:30:00.581 DEBUG:boto:Data: 2023-10-18T20:30:00.582 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.583 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.583 DEBUG:boto:Params: {} 2023-10-18T20:30:00.584 DEBUG:boto:Token: None 2023-10-18T20:30:00.584 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdf2ca4525b2d89e-0065304048-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.630 DEBUG:boto:b'bofrnf-57e1000falsee2023-10-18T20:27:08.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.632 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.632 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.633 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.633 DEBUG:boto:Path: /bofrnf-57/e 2023-10-18T20:30:00.633 DEBUG:boto:Data: 2023-10-18T20:30:00.634 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.634 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.635 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.635 DEBUG:boto:Params: {} 2023-10-18T20:30:00.636 DEBUG:boto:Token: None 2023-10-18T20:30:00.636 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e 2023-10-18T20:30:00.636 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZewN8N7qA9HbJDimJ89IgxYpvLc= 2023-10-18T20:30:00.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZewN8N7qA9HbJDimJ89IgxYpvLc=', 'Content-Length': '0'} 2023-10-18T20:30:00.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c03c89e08f3378-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.639 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.640 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.640 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.641 DEBUG:boto:Path: /bofrnf-57/e 2023-10-18T20:30:00.641 DEBUG:boto:Data: 2023-10-18T20:30:00.641 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.642 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.642 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.642 DEBUG:boto:Params: {} 2023-10-18T20:30:00.643 DEBUG:boto:Token: None 2023-10-18T20:30:00.643 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e 2023-10-18T20:30:00.644 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZewN8N7qA9HbJDimJ89IgxYpvLc= 2023-10-18T20:30:00.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZewN8N7qA9HbJDimJ89IgxYpvLc=', 'Content-Length': '0'} 2023-10-18T20:30:00.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008482bcf5347e0fa-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-18T20:30:00.647 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.647 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.648 DEBUG:boto:Method: GET 2023-10-18T20:30:00.648 DEBUG:boto:Path: /bofrnf-57/e 2023-10-18T20:30:00.649 DEBUG:boto:Data: 2023-10-18T20:30:00.649 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.650 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.650 DEBUG:boto:Params: {} 2023-10-18T20:30:00.651 DEBUG:boto:Token: None 2023-10-18T20:30:00.651 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e 2023-10-18T20:30:00.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NPfJvgUfA0DMLEhuodzoqqZgxVU= 2023-10-18T20:30:00.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NPfJvgUfA0DMLEhuodzoqqZgxVU=', 'Content-Length': '0'} 2023-10-18T20:30:00.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001475a7b71ed57688-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.655 DEBUG:boto:path=/bofrnf-57/e 2023-10-18T20:30:00.655 DEBUG:boto:auth_path=/bofrnf-57/e 2023-10-18T20:30:00.656 DEBUG:boto:Method: GET 2023-10-18T20:30:00.656 DEBUG:boto:Path: /bofrnf-57/e 2023-10-18T20:30:00.657 DEBUG:boto:Data: 2023-10-18T20:30:00.657 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.657 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.658 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.658 DEBUG:boto:Params: {} 2023-10-18T20:30:00.659 DEBUG:boto:Token: None 2023-10-18T20:30:00.659 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/e 2023-10-18T20:30:00.660 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NPfJvgUfA0DMLEhuodzoqqZgxVU= 2023-10-18T20:30:00.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NPfJvgUfA0DMLEhuodzoqqZgxVU=', 'Content-Length': '0'} 2023-10-18T20:30:00.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2143b37b42a3e93-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.663 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.663 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.664 DEBUG:boto:path=/bofrnf-57/?prefix=e 2023-10-18T20:30:00.664 DEBUG:boto:auth_path=/bofrnf-57/?prefix=e 2023-10-18T20:30:00.665 DEBUG:boto:Method: GET 2023-10-18T20:30:00.665 DEBUG:boto:Path: /bofrnf-57/?prefix=e 2023-10-18T20:30:00.665 DEBUG:boto:Data: 2023-10-18T20:30:00.666 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.666 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.667 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.667 DEBUG:boto:Params: {} 2023-10-18T20:30:00.668 DEBUG:boto:Token: None 2023-10-18T20:30:00.668 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060ab8a8ab97c96ef-0065304048-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.714 DEBUG:boto:b'bofrnf-57e1000falsee2023-10-18T20:27:08.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.715 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.716 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.716 DEBUG:boto:path=/bofrnf-57/?prefix=e 2023-10-18T20:30:00.717 DEBUG:boto:auth_path=/bofrnf-57/?prefix=e 2023-10-18T20:30:00.717 DEBUG:boto:Method: GET 2023-10-18T20:30:00.717 DEBUG:boto:Path: /bofrnf-57/?prefix=e 2023-10-18T20:30:00.718 DEBUG:boto:Data: 2023-10-18T20:30:00.718 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.719 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.720 DEBUG:boto:Params: {} 2023-10-18T20:30:00.720 DEBUG:boto:Token: None 2023-10-18T20:30:00.721 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edde75a598857877-0065304048-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.725 DEBUG:boto:b'bofrnf-57e1000falsee2023-10-18T20:27:08.058Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:30:00.726 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.726 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.727 DEBUG:boto:path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.727 DEBUG:boto:auth_path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.727 DEBUG:boto:Method: GET 2023-10-18T20:30:00.728 DEBUG:boto:Path: /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.728 DEBUG:boto:Data: 2023-10-18T20:30:00.729 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.729 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.730 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.730 DEBUG:boto:Params: {} 2023-10-18T20:30:00.731 DEBUG:boto:Token: None 2023-10-18T20:30:00.731 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.732 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NpTEyfrXd8hBIvedKHDnqILfIcY= 2023-10-18T20:30:00.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NpTEyfrXd8hBIvedKHDnqILfIcY=', 'Content-Length': '0'} 2023-10-18T20:30:00.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b385e8796abfd6b-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.735 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.735 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.736 DEBUG:boto:path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.736 DEBUG:boto:auth_path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.736 DEBUG:boto:Method: GET 2023-10-18T20:30:00.737 DEBUG:boto:Path: /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.737 DEBUG:boto:Data: 2023-10-18T20:30:00.738 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.739 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.739 DEBUG:boto:Params: {} 2023-10-18T20:30:00.740 DEBUG:boto:Token: None 2023-10-18T20:30:00.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.741 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NpTEyfrXd8hBIvedKHDnqILfIcY= 2023-10-18T20:30:00.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NpTEyfrXd8hBIvedKHDnqILfIcY=', 'Content-Length': '0'} 2023-10-18T20:30:00.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007342f94896ede6ed-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.744 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.744 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.745 DEBUG:boto:path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.745 DEBUG:boto:auth_path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.746 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.746 DEBUG:boto:Path: /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.746 DEBUG:boto:Data: 2023-10-18T20:30:00.747 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.748 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.748 DEBUG:boto:Params: {} 2023-10-18T20:30:00.748 DEBUG:boto:Token: None 2023-10-18T20:30:00.749 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1j3dU/dhYA0N0BLkcs3a8Ne+lYs= 2023-10-18T20:30:00.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1j3dU/dhYA0N0BLkcs3a8Ne+lYs=', 'Content-Length': '0'} 2023-10-18T20:30:00.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d4e39a8c021fbd-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.752 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.753 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.753 DEBUG:boto:path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.754 DEBUG:boto:auth_path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.754 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.754 DEBUG:boto:Path: /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.755 DEBUG:boto:Data: 2023-10-18T20:30:00.755 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.756 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.756 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.756 DEBUG:boto:Params: {} 2023-10-18T20:30:00.757 DEBUG:boto:Token: None 2023-10-18T20:30:00.757 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.758 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1j3dU/dhYA0N0BLkcs3a8Ne+lYs= 2023-10-18T20:30:00.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1j3dU/dhYA0N0BLkcs3a8Ne+lYs=', 'Content-Length': '0'} 2023-10-18T20:30:00.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eeba92c27b11e24-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:30:00.761 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.762 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.762 DEBUG:boto:path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.763 DEBUG:boto:auth_path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.763 DEBUG:boto:Method: GET 2023-10-18T20:30:00.763 DEBUG:boto:Path: /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.764 DEBUG:boto:Data: 2023-10-18T20:30:00.764 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.765 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.765 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.765 DEBUG:boto:Params: {} 2023-10-18T20:30:00.766 DEBUG:boto:Token: None 2023-10-18T20:30:00.766 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.767 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NpTEyfrXd8hBIvedKHDnqILfIcY= 2023-10-18T20:30:00.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NpTEyfrXd8hBIvedKHDnqILfIcY=', 'Content-Length': '0'} 2023-10-18T20:30:00.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087d0543edd59c03-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.771 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.771 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.771 DEBUG:boto:path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.772 DEBUG:boto:auth_path=/bofrnf-57/f?versionId=null 2023-10-18T20:30:00.772 DEBUG:boto:Method: GET 2023-10-18T20:30:00.773 DEBUG:boto:Path: /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.773 DEBUG:boto:Data: 2023-10-18T20:30:00.774 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.774 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.774 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.775 DEBUG:boto:Params: {} 2023-10-18T20:30:00.775 DEBUG:boto:Token: None 2023-10-18T20:30:00.776 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f?versionId=null 2023-10-18T20:30:00.776 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NpTEyfrXd8hBIvedKHDnqILfIcY= 2023-10-18T20:30:00.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NpTEyfrXd8hBIvedKHDnqILfIcY=', 'Content-Length': '0'} 2023-10-18T20:30:00.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d1c40c43f7dbe1-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.780 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.780 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.780 DEBUG:boto:path=/bofrnf-57/?prefix=f 2023-10-18T20:30:00.781 DEBUG:boto:auth_path=/bofrnf-57/?prefix=f 2023-10-18T20:30:00.781 DEBUG:boto:Method: GET 2023-10-18T20:30:00.782 DEBUG:boto:Path: /bofrnf-57/?prefix=f 2023-10-18T20:30:00.782 DEBUG:boto:Data: 2023-10-18T20:30:00.783 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.783 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.784 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.784 DEBUG:boto:Params: {} 2023-10-18T20:30:00.784 DEBUG:boto:Token: None 2023-10-18T20:30:00.785 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.785 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008857cb5a56722d68-0065304048-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.789 DEBUG:boto:b'bofrnf-57f1000falsef2023-10-18T20:27:08.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.790 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.790 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.791 DEBUG:boto:path=/bofrnf-57/?prefix=f 2023-10-18T20:30:00.791 DEBUG:boto:auth_path=/bofrnf-57/?prefix=f 2023-10-18T20:30:00.791 DEBUG:boto:Method: GET 2023-10-18T20:30:00.792 DEBUG:boto:Path: /bofrnf-57/?prefix=f 2023-10-18T20:30:00.792 DEBUG:boto:Data: 2023-10-18T20:30:00.793 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.793 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.794 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.794 DEBUG:boto:Params: {} 2023-10-18T20:30:00.794 DEBUG:boto:Token: None 2023-10-18T20:30:00.795 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.795 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030aa396dac98575d-0065304048-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.842 DEBUG:boto:b'bofrnf-57f1000falsef2023-10-18T20:27:08.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.843 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.844 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.844 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.844 DEBUG:boto:Path: /bofrnf-57/f 2023-10-18T20:30:00.845 DEBUG:boto:Data: 2023-10-18T20:30:00.845 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.846 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.846 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.846 DEBUG:boto:Params: {} 2023-10-18T20:30:00.847 DEBUG:boto:Token: None 2023-10-18T20:30:00.847 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f 2023-10-18T20:30:00.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sQJkBNNcyGfjz3hJkZZRWPkFEIA= 2023-10-18T20:30:00.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sQJkBNNcyGfjz3hJkZZRWPkFEIA=', 'Content-Length': '0'} 2023-10-18T20:30:00.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cac464529c1e97d2-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.851 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.852 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.852 DEBUG:boto:Method: HEAD 2023-10-18T20:30:00.852 DEBUG:boto:Path: /bofrnf-57/f 2023-10-18T20:30:00.853 DEBUG:boto:Data: 2023-10-18T20:30:00.853 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.854 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.854 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.854 DEBUG:boto:Params: {} 2023-10-18T20:30:00.855 DEBUG:boto:Token: None 2023-10-18T20:30:00.855 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f 2023-10-18T20:30:00.856 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sQJkBNNcyGfjz3hJkZZRWPkFEIA= 2023-10-18T20:30:00.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sQJkBNNcyGfjz3hJkZZRWPkFEIA=', 'Content-Length': '0'} 2023-10-18T20:30:00.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c61dc508808a898-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-18T20:30:00.859 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.860 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.860 DEBUG:boto:Method: GET 2023-10-18T20:30:00.860 DEBUG:boto:Path: /bofrnf-57/f 2023-10-18T20:30:00.861 DEBUG:boto:Data: 2023-10-18T20:30:00.861 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.862 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.862 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.863 DEBUG:boto:Params: {} 2023-10-18T20:30:00.863 DEBUG:boto:Token: None 2023-10-18T20:30:00.864 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f 2023-10-18T20:30:00.864 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qVYwr9OP17pTYbgKz2fo4d5ZWsk= 2023-10-18T20:30:00.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qVYwr9OP17pTYbgKz2fo4d5ZWsk=', 'Content-Length': '0'} 2023-10-18T20:30:00.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3d08d5652a2cd4-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.914 DEBUG:boto:path=/bofrnf-57/f 2023-10-18T20:30:00.915 DEBUG:boto:auth_path=/bofrnf-57/f 2023-10-18T20:30:00.916 DEBUG:boto:Method: GET 2023-10-18T20:30:00.916 DEBUG:boto:Path: /bofrnf-57/f 2023-10-18T20:30:00.916 DEBUG:boto:Data: 2023-10-18T20:30:00.917 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.918 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.918 DEBUG:boto:Params: {} 2023-10-18T20:30:00.919 DEBUG:boto:Token: None 2023-10-18T20:30:00.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/f 2023-10-18T20:30:00.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qVYwr9OP17pTYbgKz2fo4d5ZWsk= 2023-10-18T20:30:00.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qVYwr9OP17pTYbgKz2fo4d5ZWsk=', 'Content-Length': '0'} 2023-10-18T20:30:00.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c92d9f728287e6-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.924 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.924 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.924 DEBUG:boto:path=/bofrnf-57/?prefix=f 2023-10-18T20:30:00.925 DEBUG:boto:auth_path=/bofrnf-57/?prefix=f 2023-10-18T20:30:00.925 DEBUG:boto:Method: GET 2023-10-18T20:30:00.926 DEBUG:boto:Path: /bofrnf-57/?prefix=f 2023-10-18T20:30:00.926 DEBUG:boto:Data: 2023-10-18T20:30:00.927 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.927 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.928 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.928 DEBUG:boto:Params: {} 2023-10-18T20:30:00.928 DEBUG:boto:Token: None 2023-10-18T20:30:00.929 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b41091ce0769556f-0065304048-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.933 DEBUG:boto:b'bofrnf-57f1000falsef2023-10-18T20:27:08.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.934 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:00.934 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:00.935 DEBUG:boto:path=/bofrnf-57/?prefix=f 2023-10-18T20:30:00.935 DEBUG:boto:auth_path=/bofrnf-57/?prefix=f 2023-10-18T20:30:00.935 DEBUG:boto:Method: GET 2023-10-18T20:30:00.936 DEBUG:boto:Path: /bofrnf-57/?prefix=f 2023-10-18T20:30:00.936 DEBUG:boto:Data: 2023-10-18T20:30:00.937 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.937 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.938 DEBUG:boto:Params: {} 2023-10-18T20:30:00.938 DEBUG:boto:Token: None 2023-10-18T20:30:00.939 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/ 2023-10-18T20:30:00.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I= 2023-10-18T20:30:00.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wEQfrx9d4eF34xtqx/ojHSjJ/6I=', 'Content-Length': '0'} 2023-10-18T20:30:00.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000624295cc79e89fa5-0065304048-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.943 DEBUG:boto:b'bofrnf-57f1000falsef2023-10-18T20:27:08.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:00.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:30:00.944 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:00.944 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:00.945 DEBUG:boto:path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:00.945 DEBUG:boto:auth_path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:00.946 DEBUG:boto:Method: GET 2023-10-18T20:30:00.946 DEBUG:boto:Path: /bofrnf-57/g?versionId=null 2023-10-18T20:30:00.947 DEBUG:boto:Data: 2023-10-18T20:30:00.947 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.947 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:00.948 DEBUG:boto:Port: 8001 2023-10-18T20:30:00.948 DEBUG:boto:Params: {} 2023-10-18T20:30:00.949 DEBUG:boto:Token: None 2023-10-18T20:30:00.949 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/g?versionId=null 2023-10-18T20:30:00.950 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ktDYG2vC6fnsC2y6zit03TQjkeY= 2023-10-18T20:30:00.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ktDYG2vC6fnsC2y6zit03TQjkeY=', 'Content-Length': '0'} 2023-10-18T20:30:00.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000975618424d4f717f-0065304048-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:00.953 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:00.954 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:00.954 DEBUG:boto:path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:00.954 DEBUG:boto:auth_path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:00.955 DEBUG:boto:Method: GET 2023-10-18T20:30:00.955 DEBUG:boto:Path: /bofrnf-57/g?versionId=null 2023-10-18T20:30:00.956 DEBUG:boto:Data: 2023-10-18T20:30:00.956 DEBUG:boto:Headers: {} 2023-10-18T20:30:00.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:00.957 DEBUG:boto:Port: 8000 2023-10-18T20:30:00.957 DEBUG:boto:Params: {} 2023-10-18T20:30:00.958 DEBUG:boto:Token: None 2023-10-18T20:30:00.958 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:00 GMT /bofrnf-57/g?versionId=null 2023-10-18T20:30:00.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ktDYG2vC6fnsC2y6zit03TQjkeY= 2023-10-18T20:30:00.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ktDYG2vC6fnsC2y6zit03TQjkeY=', 'Content-Length': '0'} 2023-10-18T20:30:00.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029075d6820ede571-0065304048-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.002 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:01.003 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:01.003 DEBUG:boto:path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:01.004 DEBUG:boto:auth_path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:01.004 DEBUG:boto:Method: HEAD 2023-10-18T20:30:01.005 DEBUG:boto:Path: /bofrnf-57/g?versionId=null 2023-10-18T20:30:01.005 DEBUG:boto:Data: 2023-10-18T20:30:01.005 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.006 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.006 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.007 DEBUG:boto:Params: {} 2023-10-18T20:30:01.007 DEBUG:boto:Token: None 2023-10-18T20:30:01.008 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/g?versionId=null 2023-10-18T20:30:01.008 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Zg7QJu2oWYzbAOEcwiM4sobUSQ= 2023-10-18T20:30:01.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Zg7QJu2oWYzbAOEcwiM4sobUSQ=', 'Content-Length': '0'} 2023-10-18T20:30:01.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e283749f3eb0cd70-0065304049-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.011 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:01.012 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:01.012 DEBUG:boto:path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:01.013 DEBUG:boto:auth_path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:01.013 DEBUG:boto:Method: HEAD 2023-10-18T20:30:01.013 DEBUG:boto:Path: /bofrnf-57/g?versionId=null 2023-10-18T20:30:01.014 DEBUG:boto:Data: 2023-10-18T20:30:01.014 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.015 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.015 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.015 DEBUG:boto:Params: {} 2023-10-18T20:30:01.016 DEBUG:boto:Token: None 2023-10-18T20:30:01.016 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/g?versionId=null 2023-10-18T20:30:01.017 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Zg7QJu2oWYzbAOEcwiM4sobUSQ= 2023-10-18T20:30:01.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Zg7QJu2oWYzbAOEcwiM4sobUSQ=', 'Content-Length': '0'} 2023-10-18T20:30:01.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099aa8be6b0e8e11-0065304049-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:30:01.020 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:01.021 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:01.021 DEBUG:boto:path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:01.022 DEBUG:boto:auth_path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:01.022 DEBUG:boto:Method: GET 2023-10-18T20:30:01.022 DEBUG:boto:Path: /bofrnf-57/g?versionId=null 2023-10-18T20:30:01.023 DEBUG:boto:Data: 2023-10-18T20:30:01.023 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.024 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.024 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.025 DEBUG:boto:Params: {} 2023-10-18T20:30:01.025 DEBUG:boto:Token: None 2023-10-18T20:30:01.026 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/g?versionId=null 2023-10-18T20:30:01.026 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qIrvbA7H7v8X8NkHW5011iV+Ru8= 2023-10-18T20:30:01.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qIrvbA7H7v8X8NkHW5011iV+Ru8=', 'Content-Length': '0'} 2023-10-18T20:30:01.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ff0350c86971a4c-0065304049-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.029 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:01.030 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:01.030 DEBUG:boto:path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:01.031 DEBUG:boto:auth_path=/bofrnf-57/g?versionId=null 2023-10-18T20:30:01.031 DEBUG:boto:Method: GET 2023-10-18T20:30:01.031 DEBUG:boto:Path: /bofrnf-57/g?versionId=null 2023-10-18T20:30:01.032 DEBUG:boto:Data: 2023-10-18T20:30:01.032 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.033 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.033 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.034 DEBUG:boto:Params: {} 2023-10-18T20:30:01.034 DEBUG:boto:Token: None 2023-10-18T20:30:01.035 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/g?versionId=null 2023-10-18T20:30:01.035 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qIrvbA7H7v8X8NkHW5011iV+Ru8= 2023-10-18T20:30:01.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qIrvbA7H7v8X8NkHW5011iV+Ru8=', 'Content-Length': '0'} 2023-10-18T20:30:01.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a416d15b9bbd0c73-0065304049-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.038 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:01.039 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:01.039 DEBUG:boto:path=/bofrnf-57/?prefix=g 2023-10-18T20:30:01.039 DEBUG:boto:auth_path=/bofrnf-57/?prefix=g 2023-10-18T20:30:01.040 DEBUG:boto:Method: GET 2023-10-18T20:30:01.040 DEBUG:boto:Path: /bofrnf-57/?prefix=g 2023-10-18T20:30:01.041 DEBUG:boto:Data: 2023-10-18T20:30:01.041 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.042 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.042 DEBUG:boto:Params: {} 2023-10-18T20:30:01.043 DEBUG:boto:Token: None 2023-10-18T20:30:01.043 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/ 2023-10-18T20:30:01.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo= 2023-10-18T20:30:01.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo=', 'Content-Length': '0'} 2023-10-18T20:30:01.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a59bbbe8902f01c4-0065304049-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.047 DEBUG:boto:b'bofrnf-57g1000falseg2023-10-18T20:27:08.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:01.048 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:01.049 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:01.049 DEBUG:boto:path=/bofrnf-57/?prefix=g 2023-10-18T20:30:01.049 DEBUG:boto:auth_path=/bofrnf-57/?prefix=g 2023-10-18T20:30:01.050 DEBUG:boto:Method: GET 2023-10-18T20:30:01.050 DEBUG:boto:Path: /bofrnf-57/?prefix=g 2023-10-18T20:30:01.051 DEBUG:boto:Data: 2023-10-18T20:30:01.051 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.052 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.052 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.052 DEBUG:boto:Params: {} 2023-10-18T20:30:01.053 DEBUG:boto:Token: None 2023-10-18T20:30:01.053 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/ 2023-10-18T20:30:01.054 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo= 2023-10-18T20:30:01.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo=', 'Content-Length': '0'} 2023-10-18T20:30:01.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5b9fc1613c5de22-0065304049-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.057 DEBUG:boto:b'bofrnf-57g1000falseg2023-10-18T20:27:08.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:01.058 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:01.058 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:01.059 DEBUG:boto:Method: HEAD 2023-10-18T20:30:01.059 DEBUG:boto:Path: /bofrnf-57/g 2023-10-18T20:30:01.060 DEBUG:boto:Data: 2023-10-18T20:30:01.060 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.060 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.061 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.061 DEBUG:boto:Params: {} 2023-10-18T20:30:01.062 DEBUG:boto:Token: None 2023-10-18T20:30:01.062 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/g 2023-10-18T20:30:01.063 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VuZTMMvZ3YmK1Ls9ehW1/lYHAXw= 2023-10-18T20:30:01.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VuZTMMvZ3YmK1Ls9ehW1/lYHAXw=', 'Content-Length': '0'} 2023-10-18T20:30:01.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000021777919cf8485a-0065304049-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.066 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:01.066 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:01.067 DEBUG:boto:Method: HEAD 2023-10-18T20:30:01.067 DEBUG:boto:Path: /bofrnf-57/g 2023-10-18T20:30:01.067 DEBUG:boto:Data: 2023-10-18T20:30:01.068 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.068 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.069 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.069 DEBUG:boto:Params: {} 2023-10-18T20:30:01.069 DEBUG:boto:Token: None 2023-10-18T20:30:01.070 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/g 2023-10-18T20:30:01.070 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VuZTMMvZ3YmK1Ls9ehW1/lYHAXw= 2023-10-18T20:30:01.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VuZTMMvZ3YmK1Ls9ehW1/lYHAXw=', 'Content-Length': '0'} 2023-10-18T20:30:01.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed5fa92d913cf545-0065304049-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-18T20:30:01.074 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:01.075 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:01.075 DEBUG:boto:Method: GET 2023-10-18T20:30:01.075 DEBUG:boto:Path: /bofrnf-57/g 2023-10-18T20:30:01.076 DEBUG:boto:Data: 2023-10-18T20:30:01.076 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.077 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.077 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.078 DEBUG:boto:Params: {} 2023-10-18T20:30:01.078 DEBUG:boto:Token: None 2023-10-18T20:30:01.078 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/g 2023-10-18T20:30:01.079 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1whr1FHzFeqWGw5UlNg9tyJfiYc= 2023-10-18T20:30:01.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1whr1FHzFeqWGw5UlNg9tyJfiYc=', 'Content-Length': '0'} 2023-10-18T20:30:01.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005cfa5b1efd04310-0065304049-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.122 DEBUG:boto:path=/bofrnf-57/g 2023-10-18T20:30:01.123 DEBUG:boto:auth_path=/bofrnf-57/g 2023-10-18T20:30:01.123 DEBUG:boto:Method: GET 2023-10-18T20:30:01.124 DEBUG:boto:Path: /bofrnf-57/g 2023-10-18T20:30:01.124 DEBUG:boto:Data: 2023-10-18T20:30:01.125 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.126 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.126 DEBUG:boto:Params: {} 2023-10-18T20:30:01.126 DEBUG:boto:Token: None 2023-10-18T20:30:01.127 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/g 2023-10-18T20:30:01.127 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1whr1FHzFeqWGw5UlNg9tyJfiYc= 2023-10-18T20:30:01.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1whr1FHzFeqWGw5UlNg9tyJfiYc=', 'Content-Length': '0'} 2023-10-18T20:30:01.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b11c340aa0d73ad-0065304049-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.131 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:01.132 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:01.132 DEBUG:boto:path=/bofrnf-57/?prefix=g 2023-10-18T20:30:01.132 DEBUG:boto:auth_path=/bofrnf-57/?prefix=g 2023-10-18T20:30:01.133 DEBUG:boto:Method: GET 2023-10-18T20:30:01.133 DEBUG:boto:Path: /bofrnf-57/?prefix=g 2023-10-18T20:30:01.134 DEBUG:boto:Data: 2023-10-18T20:30:01.134 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.135 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.135 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.135 DEBUG:boto:Params: {} 2023-10-18T20:30:01.136 DEBUG:boto:Token: None 2023-10-18T20:30:01.136 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/ 2023-10-18T20:30:01.137 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo= 2023-10-18T20:30:01.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo=', 'Content-Length': '0'} 2023-10-18T20:30:01.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000399c66e12fdbe013-0065304049-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.141 DEBUG:boto:b'bofrnf-57g1000falseg2023-10-18T20:27:08.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:01.141 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:01.142 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:01.142 DEBUG:boto:path=/bofrnf-57/?prefix=g 2023-10-18T20:30:01.143 DEBUG:boto:auth_path=/bofrnf-57/?prefix=g 2023-10-18T20:30:01.143 DEBUG:boto:Method: GET 2023-10-18T20:30:01.144 DEBUG:boto:Path: /bofrnf-57/?prefix=g 2023-10-18T20:30:01.144 DEBUG:boto:Data: 2023-10-18T20:30:01.145 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.145 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.145 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.146 DEBUG:boto:Params: {} 2023-10-18T20:30:01.146 DEBUG:boto:Token: None 2023-10-18T20:30:01.147 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/ 2023-10-18T20:30:01.147 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo= 2023-10-18T20:30:01.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo=', 'Content-Length': '0'} 2023-10-18T20:30:01.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7b1a3603ff70921-0065304049-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.194 DEBUG:boto:b'bofrnf-57g1000falseg2023-10-18T20:27:08.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:01.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:30:01.196 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.196 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.197 DEBUG:boto:path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.197 DEBUG:boto:auth_path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.197 DEBUG:boto:Method: GET 2023-10-18T20:30:01.198 DEBUG:boto:Path: /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.198 DEBUG:boto:Data: 2023-10-18T20:30:01.199 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.200 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.200 DEBUG:boto:Params: {} 2023-10-18T20:30:01.200 DEBUG:boto:Token: None 2023-10-18T20:30:01.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9lXUct/fBxxAbCmQDiD2eTrPFr0= 2023-10-18T20:30:01.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9lXUct/fBxxAbCmQDiD2eTrPFr0=', 'Content-Length': '0'} 2023-10-18T20:30:01.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e38718b5e6b5c07b-0065304049-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.205 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.205 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.206 DEBUG:boto:path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.206 DEBUG:boto:auth_path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.207 DEBUG:boto:Method: GET 2023-10-18T20:30:01.207 DEBUG:boto:Path: /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.208 DEBUG:boto:Data: 2023-10-18T20:30:01.208 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.209 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.209 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.209 DEBUG:boto:Params: {} 2023-10-18T20:30:01.210 DEBUG:boto:Token: None 2023-10-18T20:30:01.210 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9lXUct/fBxxAbCmQDiD2eTrPFr0= 2023-10-18T20:30:01.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9lXUct/fBxxAbCmQDiD2eTrPFr0=', 'Content-Length': '0'} 2023-10-18T20:30:01.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000756616bd2d274bf1-0065304049-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.214 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.214 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.215 DEBUG:boto:path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.215 DEBUG:boto:auth_path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.216 DEBUG:boto:Method: HEAD 2023-10-18T20:30:01.216 DEBUG:boto:Path: /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.217 DEBUG:boto:Data: 2023-10-18T20:30:01.217 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.217 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.218 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.218 DEBUG:boto:Params: {} 2023-10-18T20:30:01.219 DEBUG:boto:Token: None 2023-10-18T20:30:01.219 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.219 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dcul2zxDloWvj+5QgO/2IGMi2BY= 2023-10-18T20:30:01.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dcul2zxDloWvj+5QgO/2IGMi2BY=', 'Content-Length': '0'} 2023-10-18T20:30:01.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9c6bc4dc60b8bb-0065304049-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.222 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.223 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.223 DEBUG:boto:path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.224 DEBUG:boto:auth_path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.224 DEBUG:boto:Method: HEAD 2023-10-18T20:30:01.224 DEBUG:boto:Path: /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.225 DEBUG:boto:Data: 2023-10-18T20:30:01.225 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.226 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.226 DEBUG:boto:Params: {} 2023-10-18T20:30:01.227 DEBUG:boto:Token: None 2023-10-18T20:30:01.227 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.228 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dcul2zxDloWvj+5QgO/2IGMi2BY= 2023-10-18T20:30:01.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dcul2zxDloWvj+5QgO/2IGMi2BY=', 'Content-Length': '0'} 2023-10-18T20:30:01.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fa6ec71313fe66e-0065304049-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:30:01.233 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.233 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.234 DEBUG:boto:path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.234 DEBUG:boto:auth_path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.235 DEBUG:boto:Method: GET 2023-10-18T20:30:01.235 DEBUG:boto:Path: /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.235 DEBUG:boto:Data: 2023-10-18T20:30:01.236 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.236 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.237 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.237 DEBUG:boto:Params: {} 2023-10-18T20:30:01.238 DEBUG:boto:Token: None 2023-10-18T20:30:01.238 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.239 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9lXUct/fBxxAbCmQDiD2eTrPFr0= 2023-10-18T20:30:01.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9lXUct/fBxxAbCmQDiD2eTrPFr0=', 'Content-Length': '0'} 2023-10-18T20:30:01.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec457e2bdba9fc4-0065304049-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.242 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.242 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.243 DEBUG:boto:path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.243 DEBUG:boto:auth_path=/bofrnf-57/h?versionId=null 2023-10-18T20:30:01.244 DEBUG:boto:Method: GET 2023-10-18T20:30:01.244 DEBUG:boto:Path: /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.244 DEBUG:boto:Data: 2023-10-18T20:30:01.245 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.245 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.246 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.246 DEBUG:boto:Params: {} 2023-10-18T20:30:01.247 DEBUG:boto:Token: None 2023-10-18T20:30:01.247 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h?versionId=null 2023-10-18T20:30:01.248 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9lXUct/fBxxAbCmQDiD2eTrPFr0= 2023-10-18T20:30:01.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9lXUct/fBxxAbCmQDiD2eTrPFr0=', 'Content-Length': '0'} 2023-10-18T20:30:01.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f941db18c02af209-0065304049-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.251 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:01.251 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:01.252 DEBUG:boto:path=/bofrnf-57/?prefix=h 2023-10-18T20:30:01.252 DEBUG:boto:auth_path=/bofrnf-57/?prefix=h 2023-10-18T20:30:01.253 DEBUG:boto:Method: GET 2023-10-18T20:30:01.253 DEBUG:boto:Path: /bofrnf-57/?prefix=h 2023-10-18T20:30:01.253 DEBUG:boto:Data: 2023-10-18T20:30:01.254 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.254 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.255 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.255 DEBUG:boto:Params: {} 2023-10-18T20:30:01.256 DEBUG:boto:Token: None 2023-10-18T20:30:01.256 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/ 2023-10-18T20:30:01.257 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo= 2023-10-18T20:30:01.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo=', 'Content-Length': '0'} 2023-10-18T20:30:01.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078141d8800eee4b9-0065304049-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.260 DEBUG:boto:b'bofrnf-57h1000falseh2023-10-18T20:27:08.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:01.261 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:01.261 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:01.262 DEBUG:boto:path=/bofrnf-57/?prefix=h 2023-10-18T20:30:01.262 DEBUG:boto:auth_path=/bofrnf-57/?prefix=h 2023-10-18T20:30:01.263 DEBUG:boto:Method: GET 2023-10-18T20:30:01.263 DEBUG:boto:Path: /bofrnf-57/?prefix=h 2023-10-18T20:30:01.264 DEBUG:boto:Data: 2023-10-18T20:30:01.264 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.264 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.265 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.265 DEBUG:boto:Params: {} 2023-10-18T20:30:01.266 DEBUG:boto:Token: None 2023-10-18T20:30:01.266 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/ 2023-10-18T20:30:01.267 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo= 2023-10-18T20:30:01.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo=', 'Content-Length': '0'} 2023-10-18T20:30:01.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f12a8f947dca47ef-0065304049-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.270 DEBUG:boto:b'bofrnf-57h1000falseh2023-10-18T20:27:08.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:01.270 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.271 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.271 DEBUG:boto:Method: HEAD 2023-10-18T20:30:01.272 DEBUG:boto:Path: /bofrnf-57/h 2023-10-18T20:30:01.272 DEBUG:boto:Data: 2023-10-18T20:30:01.273 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.273 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.273 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.274 DEBUG:boto:Params: {} 2023-10-18T20:30:01.274 DEBUG:boto:Token: None 2023-10-18T20:30:01.275 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h 2023-10-18T20:30:01.275 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+8i4voNsMb4ZbYJz28Ohs+LCJ0U= 2023-10-18T20:30:01.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+8i4voNsMb4ZbYJz28Ohs+LCJ0U=', 'Content-Length': '0'} 2023-10-18T20:30:01.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006efba7cf5b9f6a90-0065304049-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.278 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.279 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.279 DEBUG:boto:Method: HEAD 2023-10-18T20:30:01.279 DEBUG:boto:Path: /bofrnf-57/h 2023-10-18T20:30:01.280 DEBUG:boto:Data: 2023-10-18T20:30:01.280 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.281 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.281 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.282 DEBUG:boto:Params: {} 2023-10-18T20:30:01.282 DEBUG:boto:Token: None 2023-10-18T20:30:01.282 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h 2023-10-18T20:30:01.283 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+8i4voNsMb4ZbYJz28Ohs+LCJ0U= 2023-10-18T20:30:01.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+8i4voNsMb4ZbYJz28Ohs+LCJ0U=', 'Content-Length': '0'} 2023-10-18T20:30:01.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adf9362150bf89d3-0065304049-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-18T20:30:01.286 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.287 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.287 DEBUG:boto:Method: GET 2023-10-18T20:30:01.288 DEBUG:boto:Path: /bofrnf-57/h 2023-10-18T20:30:01.288 DEBUG:boto:Data: 2023-10-18T20:30:01.288 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.289 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.289 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.290 DEBUG:boto:Params: {} 2023-10-18T20:30:01.290 DEBUG:boto:Token: None 2023-10-18T20:30:01.291 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h 2023-10-18T20:30:01.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ByxChiNWRou97DoLlmwtrSrIYSQ= 2023-10-18T20:30:01.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ByxChiNWRou97DoLlmwtrSrIYSQ=', 'Content-Length': '0'} 2023-10-18T20:30:01.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-57:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000854c9eb03d5c4325-0065304049-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.294 DEBUG:boto:path=/bofrnf-57/h 2023-10-18T20:30:01.295 DEBUG:boto:auth_path=/bofrnf-57/h 2023-10-18T20:30:01.295 DEBUG:boto:Method: GET 2023-10-18T20:30:01.296 DEBUG:boto:Path: /bofrnf-57/h 2023-10-18T20:30:01.296 DEBUG:boto:Data: 2023-10-18T20:30:01.296 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.297 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.297 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.298 DEBUG:boto:Params: {} 2023-10-18T20:30:01.298 DEBUG:boto:Token: None 2023-10-18T20:30:01.299 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/h 2023-10-18T20:30:01.299 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ByxChiNWRou97DoLlmwtrSrIYSQ= 2023-10-18T20:30:01.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ByxChiNWRou97DoLlmwtrSrIYSQ=', 'Content-Length': '0'} 2023-10-18T20:30:01.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:27:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9a677d4f97401e6-0065304049-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.302 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:01.302 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:01.303 DEBUG:boto:path=/bofrnf-57/?prefix=h 2023-10-18T20:30:01.303 DEBUG:boto:auth_path=/bofrnf-57/?prefix=h 2023-10-18T20:30:01.304 DEBUG:boto:Method: GET 2023-10-18T20:30:01.304 DEBUG:boto:Path: /bofrnf-57/?prefix=h 2023-10-18T20:30:01.305 DEBUG:boto:Data: 2023-10-18T20:30:01.305 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.305 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.306 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.306 DEBUG:boto:Params: {} 2023-10-18T20:30:01.307 DEBUG:boto:Token: None 2023-10-18T20:30:01.307 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/ 2023-10-18T20:30:01.308 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo= 2023-10-18T20:30:01.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo=', 'Content-Length': '0'} 2023-10-18T20:30:01.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049fd2c4170d5a1ec-0065304049-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.311 DEBUG:boto:b'bofrnf-57h1000falseh2023-10-18T20:27:08.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:01.312 DEBUG:boto:path=/bofrnf-57/ 2023-10-18T20:30:01.312 DEBUG:boto:auth_path=/bofrnf-57/ 2023-10-18T20:30:01.313 DEBUG:boto:path=/bofrnf-57/?prefix=h 2023-10-18T20:30:01.313 DEBUG:boto:auth_path=/bofrnf-57/?prefix=h 2023-10-18T20:30:01.314 DEBUG:boto:Method: GET 2023-10-18T20:30:01.314 DEBUG:boto:Path: /bofrnf-57/?prefix=h 2023-10-18T20:30:01.314 DEBUG:boto:Data: 2023-10-18T20:30:01.315 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.315 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.316 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.316 DEBUG:boto:Params: {} 2023-10-18T20:30:01.317 DEBUG:boto:Token: None 2023-10-18T20:30:01.317 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-57/ 2023-10-18T20:30:01.318 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo= 2023-10-18T20:30:01.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBwBchiYNWOHduKa55rj5BW1Obo=', 'Content-Length': '0'} 2023-10-18T20:30:01.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f3eaecefb92c61f-0065304049-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.321 DEBUG:boto:b'bofrnf-57h1000falseh2023-10-18T20:27:08.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:01.322 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-57 zones={a2, a1} 2023-10-18T20:30:01.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2023-10-18T20:30:01.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:01.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:01.325 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:01.381 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:01.382 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:01.417 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:30:01.430 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:30:01.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.431 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660828.012858_999.1", 2023-10-18T20:30:01.431 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:27:08.012858Z" 2023-10-18T20:30:01.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.431 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:30:01.432 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:30:01.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.432 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:01.433 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:30:01.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.433 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:01.433 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:30:01.434 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.434 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:30:01.435 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697660828.012858_999.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:01.435 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:30:01.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:01.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:01.437 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:01.487 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:01.488 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:01.516 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:01.514+0000 7f450a3ff780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:30:01.516 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:01.515+0000 7f450a3ff780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:30:01.529 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:01.529 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:30:01.529 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:30:01.529 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:30:01.530 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:30:01.530 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:30:01.530 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.531 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:30:01.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.531 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:30:01.531 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:01.531 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:01.532 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697660828.012858_999.1", 2023-10-18T20:30:01.532 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:01.532 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:30:01.532 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:01.533 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:27:08.012858Z", 2023-10-18T20:30:01.533 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.533 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.534 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:30:01.534 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:01.534 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:01.534 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:30:01.534 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:01.535 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:01.535 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:01.535 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:30:01.535 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.535 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.536 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:30:01.536 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:01.536 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:01.537 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:01.537 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:01.537 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:01.537 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:01.538 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:30:01.538 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.538 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.539 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:30:01.539 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:01.539 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:01.539 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:01.539 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:01.540 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:01.540 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:01.540 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:30:01.540 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.541 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.541 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:01.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.541 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:30:01.541 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:30:01.542 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:30:01.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.542 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:30:01Z" 2023-10-18T20:30:01.542 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:01.543 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697660828.012858_999.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:27:08.012858Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:30:01Z" } 2023-10-18T20:30:01.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660828.012858_999.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:27:08.012858Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:30:01.544 DEBUG:rgw_multi.tests:log_status={0: '1_1697660828.012858_999.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:01.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1697660828.012858_999.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:01.545 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:30:01.545 DEBUG:boto:path=/ 2023-10-18T20:30:01.546 DEBUG:boto:auth_path=/ 2023-10-18T20:30:01.546 DEBUG:boto:Method: GET 2023-10-18T20:30:01.547 DEBUG:boto:Path: / 2023-10-18T20:30:01.547 DEBUG:boto:Data: 2023-10-18T20:30:01.547 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.548 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.548 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.549 DEBUG:boto:Params: {} 2023-10-18T20:30:01.549 DEBUG:boto:Token: None 2023-10-18T20:30:01.550 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT / 2023-10-18T20:30:01.550 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aoLQ5V3I1SAU3bQjQPBJQ0iGCsA= 2023-10-18T20:30:01.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aoLQ5V3I1SAU3bQjQPBJQ0iGCsA=', 'Content-Length': '0'} 2023-10-18T20:30:01.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb5f4b0fed2af9b-0065304049-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.555 DEBUG:boto:path=/ 2023-10-18T20:30:01.556 DEBUG:boto:auth_path=/ 2023-10-18T20:30:01.556 DEBUG:boto:Method: GET 2023-10-18T20:30:01.556 DEBUG:boto:Path: / 2023-10-18T20:30:01.557 DEBUG:boto:Data: 2023-10-18T20:30:01.557 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:01.558 DEBUG:boto:Port: 8001 2023-10-18T20:30:01.558 DEBUG:boto:Params: {} 2023-10-18T20:30:01.559 DEBUG:boto:Token: None 2023-10-18T20:30:01.559 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:01 GMT / 2023-10-18T20:30:01.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aoLQ5V3I1SAU3bQjQPBJQ0iGCsA= 2023-10-18T20:30:01.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aoLQ5V3I1SAU3bQjQPBJQ0iGCsA=', 'Content-Length': '0'} 2023-10-18T20:30:01.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4925626549a971-0065304049-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-10-18T20:30:01.564 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:01.565 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:01.565 DEBUG:boto:Method: PUT 2023-10-18T20:30:01.566 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:01.566 DEBUG:boto:Data: 2023-10-18T20:30:01.566 DEBUG:boto:Headers: {} 2023-10-18T20:30:01.567 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:01.567 DEBUG:boto:Port: 8000 2023-10-18T20:30:01.568 DEBUG:boto:Params: {} 2023-10-18T20:30:01.568 DEBUG:boto:Token: None 2023-10-18T20:30:01.569 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:30:01 GMT /bofrnf-58/ 2023-10-18T20:30:01.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w1XDAxqjoyTlZiWJt5bethh3IHA= 2023-10-18T20:30:01.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w1XDAxqjoyTlZiWJt5bethh3IHA=', 'Content-Length': '0'} 2023-10-18T20:30:01.584 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d09b8597e254cc67-0065304049-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:01.584 DEBUG:rgw_multi.tests:created bucket=bofrnf-58 2023-10-18T20:30:01.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:01.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:01.586 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:01.635 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:01.635 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:01.664 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:30:01.676 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:30:01.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.676 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:30:01.677 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:30:01.581996Z" 2023-10-18T20:30:01.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.677 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:30:01.677 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:30:01.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.678 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:01.678 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:30:01.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.679 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:01.679 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:30:01.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.680 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:30:01.681 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:01.681 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:30:01.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:01.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:01.682 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:01.731 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:01.731 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:01.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:01.760+0000 7f92e55a6780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:30:01.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:01.760+0000 7f92e55a6780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:30:01.774 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:01.774 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:30:01.775 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:30:01.775 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:30:01.775 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:30:01.775 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:30:01.775 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.776 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:30:01.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.776 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:30:01.776 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:01.777 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:01.777 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.575918_1013.1", 2023-10-18T20:30:01.777 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:01.777 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:30:01.778 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:01.778 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.575918Z", 2023-10-18T20:30:01.778 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.778 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.779 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:30:01.779 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:01.779 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:01.779 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:30:01.780 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:01.780 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:01.780 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:01.780 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:30:01.781 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.781 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.781 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:30:01.782 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:01.782 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:01.782 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:01.782 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:01.782 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:01.783 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:01.783 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:30:01.783 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:01.784 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:30:01.784 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:01.784 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:01.785 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:01.785 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:01.785 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:01.785 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:01.785 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:30:01.786 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:01.786 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.786 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:01.786 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:01.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.787 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:30:01.787 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:30:01.787 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:30:01.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:01.787 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:30:01Z" 2023-10-18T20:30:01.788 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:01.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.575918_1013.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.575918Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:30:01Z" } 2023-10-18T20:30:01.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.575918_1013.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.575918Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:30:01.790 DEBUG:rgw_multi.tests:log_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:01.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.575918_1013.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:01.791 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661001.581996_1019.1 target=1_1697661001.575918_1013.1 2023-10-18T20:30:06.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:06.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:06.794 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:06.851 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:06.851 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:06.882 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:06.881+0000 7f2bb5fda780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:30:06.883 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:06.882+0000 7f2bb5fda780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:30:06.896 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:06.897 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:30:06.897 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:30:06.897 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:30:06.897 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:30:06.898 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:30:06.898 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:06.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:06.898 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:30:06.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:06.899 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:30:06.899 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:06.899 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:06.899 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:30:06.900 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:06.900 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:30:06.900 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:06.900 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:30:06.901 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:06.901 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:06.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:06.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:06.902 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:30:06.902 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:06.902 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:06.902 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:30:06.903 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:06.903 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:06.903 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:06.903 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:30:06.903 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:06.904 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:06.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:06.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:06.904 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:30:06.904 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:06.905 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:06.905 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:06.905 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:06.905 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:06.906 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:06.906 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:30:06.906 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:06.906 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:06.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:06.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:06.909 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:30:06.909 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:06.910 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:06.910 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:06.910 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:06.910 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:06.911 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:06.911 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:30:06.911 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:06.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:06.912 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:06.912 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:06.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:06.913 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:30:06.913 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:30:06.913 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:30:06.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:06.914 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:30:06Z" 2023-10-18T20:30:06.914 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:06.915 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:30:06Z" } 2023-10-18T20:30:06.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:30:06.916 DEBUG:rgw_multi.tests:log_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:06.916 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:06.917 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:30:06.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:06.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:06.918 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:06.971 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:06.971 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:07.011 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:07.011 INFO:teuthology.orchestra.run.smithi082.stdout: "objs": [ 2023-10-18T20:30:07.012 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000001", 2023-10-18T20:30:07.012 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000005", 2023-10-18T20:30:07.012 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000012", 2023-10-18T20:30:07.012 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000004", 2023-10-18T20:30:07.012 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000011", 2023-10-18T20:30:07.013 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000000", 2023-10-18T20:30:07.013 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000010", 2023-10-18T20:30:07.013 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000003", 2023-10-18T20:30:07.013 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000013", 2023-10-18T20:30:07.013 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000002", 2023-10-18T20:30:07.014 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000006" 2023-10-18T20:30:07.014 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:07.014 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:07.015 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.016 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.016 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.016 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.017 DEBUG:boto:Data: 2023-10-18T20:30:07.017 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.018 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.018 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.019 DEBUG:boto:Params: {} 2023-10-18T20:30:07.019 DEBUG:boto:Token: None 2023-10-18T20:30:07.020 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.020 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003691d5d36693c4e8-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.024 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:07.024 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:07.024 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.025 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:07.025 DEBUG:boto:Data: 2023-10-18T20:30:07.026 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.026 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.026 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.027 DEBUG:boto:Params: {} 2023-10-18T20:30:07.027 DEBUG:boto:Token: None 2023-10-18T20:30:07.028 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:07.028 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BUcMUuGcc5aw2Kg4DLrx/IxRZS0= 2023-10-18T20:30:07.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BUcMUuGcc5aw2Kg4DLrx/IxRZS0='} 2023-10-18T20:30:07.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003b554645c99d65b-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.035 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.036 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.036 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.036 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.037 DEBUG:boto:Data: 2023-10-18T20:30:07.037 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.038 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.038 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.038 DEBUG:boto:Params: {} 2023-10-18T20:30:07.039 DEBUG:boto:Token: None 2023-10-18T20:30:07.039 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083d55cb84a2147f4-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.043 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:07.044 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:07.044 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.045 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:07.045 DEBUG:boto:Data: 2023-10-18T20:30:07.045 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.046 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.046 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.047 DEBUG:boto:Params: {} 2023-10-18T20:30:07.047 DEBUG:boto:Token: None 2023-10-18T20:30:07.048 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:07.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZAYD1tRTUYUyki95a/NgvSAQa18= 2023-10-18T20:30:07.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZAYD1tRTUYUyki95a/NgvSAQa18='} 2023-10-18T20:30:07.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077151c4f39989fdf-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.095 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.096 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.096 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.097 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.097 DEBUG:boto:Data: 2023-10-18T20:30:07.097 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.098 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.099 DEBUG:boto:Params: {} 2023-10-18T20:30:07.099 DEBUG:boto:Token: None 2023-10-18T20:30:07.100 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.103 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ab09f6f18a5885a-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.103 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:07.104 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:07.104 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.105 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:07.105 DEBUG:boto:Data: 2023-10-18T20:30:07.106 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.106 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.106 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.107 DEBUG:boto:Params: {} 2023-10-18T20:30:07.107 DEBUG:boto:Token: None 2023-10-18T20:30:07.108 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:07.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3VUnqcxEFL3nJ3XZTT23RsxO6mM= 2023-10-18T20:30:07.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3VUnqcxEFL3nJ3XZTT23RsxO6mM='} 2023-10-18T20:30:07.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009092aa9cd7b5c69-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.116 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.116 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.117 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.118 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.119 DEBUG:boto:Data: 2023-10-18T20:30:07.119 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.120 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.121 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.122 DEBUG:boto:Params: {} 2023-10-18T20:30:07.122 DEBUG:boto:Token: None 2023-10-18T20:30:07.123 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.123 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a80150c0bcb692a9-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.127 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:07.127 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:07.128 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.128 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:07.129 DEBUG:boto:Data: 2023-10-18T20:30:07.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.129 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.130 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.130 DEBUG:boto:Params: {} 2023-10-18T20:30:07.131 DEBUG:boto:Token: None 2023-10-18T20:30:07.131 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:07.132 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZhoHTGmtHmSRtGrOioJ1ugUd81U= 2023-10-18T20:30:07.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZhoHTGmtHmSRtGrOioJ1ugUd81U='} 2023-10-18T20:30:07.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f4b5e59dd131ed4-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.176 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.176 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.177 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.177 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.178 DEBUG:boto:Data: 2023-10-18T20:30:07.179 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.179 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.180 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.181 DEBUG:boto:Params: {} 2023-10-18T20:30:07.181 DEBUG:boto:Token: None 2023-10-18T20:30:07.182 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a10cdc6f22873778-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.186 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:07.187 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:07.187 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.188 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:07.188 DEBUG:boto:Data: 2023-10-18T20:30:07.189 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.189 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.189 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.190 DEBUG:boto:Params: {} 2023-10-18T20:30:07.190 DEBUG:boto:Token: None 2023-10-18T20:30:07.191 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:07.191 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/N7KcOeyNk5AhMVraRXc+i308sg= 2023-10-18T20:30:07.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/N7KcOeyNk5AhMVraRXc+i308sg='} 2023-10-18T20:30:07.197 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b991c5653c4dfa6-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.198 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.198 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.199 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.200 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.200 DEBUG:boto:Data: 2023-10-18T20:30:07.201 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.202 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.203 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.203 DEBUG:boto:Params: {} 2023-10-18T20:30:07.204 DEBUG:boto:Token: None 2023-10-18T20:30:07.205 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.205 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f68f7836b008a70-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.209 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:07.209 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:07.210 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.210 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:07.210 DEBUG:boto:Data: 2023-10-18T20:30:07.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.211 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.211 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.212 DEBUG:boto:Params: {} 2023-10-18T20:30:07.212 DEBUG:boto:Token: None 2023-10-18T20:30:07.213 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:07.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SnMPczxUF0fqkygZdk+D2PvzYn8= 2023-10-18T20:30:07.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SnMPczxUF0fqkygZdk+D2PvzYn8='} 2023-10-18T20:30:07.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f357511c85c2d37a-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.219 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.219 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.220 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.220 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.221 DEBUG:boto:Data: 2023-10-18T20:30:07.221 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.221 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.222 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.222 DEBUG:boto:Params: {} 2023-10-18T20:30:07.223 DEBUG:boto:Token: None 2023-10-18T20:30:07.223 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.224 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0946bfc5e2bdcbe-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.227 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:07.228 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:07.228 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.229 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:07.229 DEBUG:boto:Data: 2023-10-18T20:30:07.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.230 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.231 DEBUG:boto:Params: {} 2023-10-18T20:30:07.231 DEBUG:boto:Token: None 2023-10-18T20:30:07.232 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:07.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NkcP3kIZpfSBzOSwkNfL3zIu0bc= 2023-10-18T20:30:07.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NkcP3kIZpfSBzOSwkNfL3zIu0bc='} 2023-10-18T20:30:07.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b154739bf995022d-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.275 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.276 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.276 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.277 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.277 DEBUG:boto:Data: 2023-10-18T20:30:07.277 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.278 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.278 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.279 DEBUG:boto:Params: {} 2023-10-18T20:30:07.279 DEBUG:boto:Token: None 2023-10-18T20:30:07.280 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.280 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee0d22ce089f3fdc-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.283 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:07.284 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:07.284 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.284 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:07.285 DEBUG:boto:Data: 2023-10-18T20:30:07.285 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.286 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.286 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.286 DEBUG:boto:Params: {} 2023-10-18T20:30:07.287 DEBUG:boto:Token: None 2023-10-18T20:30:07.287 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:07.288 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W/UZTXdV6GJHbRfstGWwT2e7Jss= 2023-10-18T20:30:07.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W/UZTXdV6GJHbRfstGWwT2e7Jss='} 2023-10-18T20:30:07.293 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6b23f5345f17559-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.293 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.294 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.294 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.295 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.295 DEBUG:boto:Data: 2023-10-18T20:30:07.296 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.296 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.296 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.297 DEBUG:boto:Params: {} 2023-10-18T20:30:07.298 DEBUG:boto:Token: None 2023-10-18T20:30:07.299 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.300 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025d6e74b687ae6c9-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.304 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:07.304 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:07.305 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.306 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:07.306 DEBUG:boto:Data: 2023-10-18T20:30:07.307 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.307 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.308 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.308 DEBUG:boto:Params: {} 2023-10-18T20:30:07.308 DEBUG:boto:Token: None 2023-10-18T20:30:07.309 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:07.309 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DNACINyhNrTr1dm8W1HMU7tfOCY= 2023-10-18T20:30:07.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DNACINyhNrTr1dm8W1HMU7tfOCY='} 2023-10-18T20:30:07.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043831070154d5c38-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.359 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:07.360 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:07.360 DEBUG:boto:Method: HEAD 2023-10-18T20:30:07.361 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:07.361 DEBUG:boto:Data: 2023-10-18T20:30:07.362 DEBUG:boto:Headers: {} 2023-10-18T20:30:07.362 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.363 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.363 DEBUG:boto:Params: {} 2023-10-18T20:30:07.363 DEBUG:boto:Token: None 2023-10-18T20:30:07.364 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/ 2023-10-18T20:30:07.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw= 2023-10-18T20:30:07.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D/hdPnZRmM81T05KREB8wRgjjvw=', 'Content-Length': '0'} 2023-10-18T20:30:07.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cc8039e2bad93ad-006530404f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.367 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:07.368 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:07.368 DEBUG:boto:Method: PUT 2023-10-18T20:30:07.369 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:07.369 DEBUG:boto:Data: 2023-10-18T20:30:07.370 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:07.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:07.370 DEBUG:boto:Port: 8000 2023-10-18T20:30:07.371 DEBUG:boto:Params: {} 2023-10-18T20:30:07.371 DEBUG:boto:Token: None 2023-10-18T20:30:07.372 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:07 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:07.372 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jZ3qyCaI2jVRjtbwLhBvJe2WEdo= 2023-10-18T20:30:07.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jZ3qyCaI2jVRjtbwLhBvJe2WEdo='} 2023-10-18T20:30:07.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc90f84930f936f3-006530404f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:07.378 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-58 2023-10-18T20:30:07.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:30:07.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:30:07.380 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:30:07.437 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:07.437 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:07.471 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:07.470+0000 7f627f24d780 1 waiting to reach incremental sync.. 2023-10-18T20:30:12.473 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:12.472+0000 7f627f24d780 1 bucket sync caught up with source: 2023-10-18T20:30:12.474 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.62.6, 00000000001.36.6, 00000000001.46.6, 00000000001.36.6, 00000000001.49.6, 00000000001.75.6, 00000000001.30.6, 00000000001.44.6, , 00000000001.38.6, 00000000001.36.6] 2023-10-18T20:30:12.474 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.62.6, 00000000001.36.6, 00000000001.46.6, 00000000001.36.6, 00000000001.49.6, 00000000001.75.6, 00000000001.30.6, 00000000001.44.6, , 00000000001.38.6, 00000000001.36.6] 2023-10-18T20:30:12.474 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:12.472+0000 7f627f24d780 0 bucket checkpoint complete 2023-10-18T20:30:12.491 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-58 2023-10-18T20:30:12.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:30:12.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:30:12.493 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:30:12.555 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:12.556 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:12.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:12.592+0000 7f2f906cd780 1 waiting to reach incremental sync.. 2023-10-18T20:30:17.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:17.594+0000 7f2f906cd780 1 bucket sync caught up with source: 2023-10-18T20:30:17.596 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.603.6, 00000000001.297.6, 00000000001.309.6, 00000000001.692.6, 00000000001.934.6, 00000000001.500.6, 00000000001.920.6, 00000000001.310.6, , 00000000001.1172.6, 00000000001.1173.6] 2023-10-18T20:30:17.597 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.603.6, 00000000001.297.6, 00000000001.309.6, 00000000001.692.6, 00000000001.934.6, 00000000001.500.6, 00000000001.920.6, 00000000001.310.6, , 00000000001.1172.6, 00000000001.1173.6] 2023-10-18T20:30:17.597 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:17.594+0000 7f2f906cd780 0 bucket checkpoint complete 2023-10-18T20:30:17.613 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-58 zones={a2, a1} 2023-10-18T20:30:17.614 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.614 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.615 DEBUG:boto:Method: HEAD 2023-10-18T20:30:17.615 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:17.616 DEBUG:boto:Data: 2023-10-18T20:30:17.616 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.616 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.617 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.618 DEBUG:boto:Params: {} 2023-10-18T20:30:17.618 DEBUG:boto:Token: None 2023-10-18T20:30:17.619 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/ 2023-10-18T20:30:17.619 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tI4uEf/EZk7YJIHP6T9JUNvaaxc= 2023-10-18T20:30:17.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tI4uEf/EZk7YJIHP6T9JUNvaaxc=', 'Content-Length': '0'} 2023-10-18T20:30:17.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef8f69842276d404-0065304059-1a24-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.623 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.623 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.624 DEBUG:boto:Method: HEAD 2023-10-18T20:30:17.624 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:17.625 DEBUG:boto:Data: 2023-10-18T20:30:17.625 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.626 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.626 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.629 DEBUG:boto:Params: {} 2023-10-18T20:30:17.629 DEBUG:boto:Token: None 2023-10-18T20:30:17.630 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/ 2023-10-18T20:30:17.630 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tI4uEf/EZk7YJIHP6T9JUNvaaxc= 2023-10-18T20:30:17.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tI4uEf/EZk7YJIHP6T9JUNvaaxc=', 'Content-Length': '0'} 2023-10-18T20:30:17.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000828df3e760da073a-0065304059-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.634 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:30:17.635 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.635 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.636 DEBUG:boto:path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.636 DEBUG:boto:auth_path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.637 DEBUG:boto:Method: GET 2023-10-18T20:30:17.637 DEBUG:boto:Path: /bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.638 DEBUG:boto:Data: 2023-10-18T20:30:17.638 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.639 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.639 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.640 DEBUG:boto:Params: {} 2023-10-18T20:30:17.640 DEBUG:boto:Token: None 2023-10-18T20:30:17.641 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/?versions 2023-10-18T20:30:17.641 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MkAn5Z3V1vp8ES/efufPppE3UeM= 2023-10-18T20:30:17.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MkAn5Z3V1vp8ES/efufPppE3UeM=', 'Content-Length': '0'} 2023-10-18T20:30:17.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f06c4c54ea642a7-0065304059-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.645 DEBUG:boto:b'bofrnf-58999falseobj00000000000000000000nulltrue2023-10-18T20:30:07.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-18T20:30:07.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-18T20:30:07.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-18T20:30:07.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-18T20:30:07.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-18T20:30:07.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-18T20:30:07.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-18T20:30:07.193Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-18T20:30:07.311Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-18T20:30:07.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:17.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-18T20:30:17.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-18T20:30:17.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-18T20:30:17.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-18T20:30:17.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-18T20:30:17.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-18T20:30:17.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-18T20:30:17.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-18T20:30:17.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-18T20:30:17.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-18T20:30:17.650 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:30:17.651 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.651 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.651 DEBUG:boto:path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.652 DEBUG:boto:auth_path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.653 DEBUG:boto:Method: GET 2023-10-18T20:30:17.653 DEBUG:boto:Path: /bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.654 DEBUG:boto:Data: 2023-10-18T20:30:17.655 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.656 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.657 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.657 DEBUG:boto:Params: {} 2023-10-18T20:30:17.658 DEBUG:boto:Token: None 2023-10-18T20:30:17.659 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/?versions 2023-10-18T20:30:17.659 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MkAn5Z3V1vp8ES/efufPppE3UeM= 2023-10-18T20:30:17.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MkAn5Z3V1vp8ES/efufPppE3UeM=', 'Content-Length': '0'} 2023-10-18T20:30:17.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af9d2c0d30fc2414-0065304059-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.664 DEBUG:boto:b'bofrnf-58999falseobj00000000000000000000nulltrue2023-10-18T20:30:07.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-18T20:30:07.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-18T20:30:07.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-18T20:30:07.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-18T20:30:07.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-18T20:30:07.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-18T20:30:07.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-18T20:30:07.193Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-18T20:30:07.311Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-18T20:30:07.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:17.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-18T20:30:17.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-18T20:30:17.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-18T20:30:17.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-18T20:30:17.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-18T20:30:17.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-18T20:30:17.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-18T20:30:17.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-18T20:30:17.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-18T20:30:17.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-18T20:30:17.672 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.672 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.673 DEBUG:boto:path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.673 DEBUG:boto:auth_path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.674 DEBUG:boto:Method: GET 2023-10-18T20:30:17.674 DEBUG:boto:Path: /bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.675 DEBUG:boto:Data: 2023-10-18T20:30:17.675 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.675 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.676 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.676 DEBUG:boto:Params: {} 2023-10-18T20:30:17.677 DEBUG:boto:Token: None 2023-10-18T20:30:17.677 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/?versions 2023-10-18T20:30:17.678 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MkAn5Z3V1vp8ES/efufPppE3UeM= 2023-10-18T20:30:17.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MkAn5Z3V1vp8ES/efufPppE3UeM=', 'Content-Length': '0'} 2023-10-18T20:30:17.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa00f88e437c9744-0065304059-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.682 DEBUG:boto:b'bofrnf-58999falseobj00000000000000000000nulltrue2023-10-18T20:30:07.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-18T20:30:07.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-18T20:30:07.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-18T20:30:07.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-18T20:30:07.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-18T20:30:07.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-18T20:30:07.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-18T20:30:07.193Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-18T20:30:07.311Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-18T20:30:07.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:17.683 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.683 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.684 DEBUG:boto:path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.684 DEBUG:boto:auth_path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.684 DEBUG:boto:Method: GET 2023-10-18T20:30:17.685 DEBUG:boto:Path: /bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:17.685 DEBUG:boto:Data: 2023-10-18T20:30:17.686 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.687 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.687 DEBUG:boto:Params: {} 2023-10-18T20:30:17.687 DEBUG:boto:Token: None 2023-10-18T20:30:17.688 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/?versions 2023-10-18T20:30:17.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MkAn5Z3V1vp8ES/efufPppE3UeM= 2023-10-18T20:30:17.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MkAn5Z3V1vp8ES/efufPppE3UeM=', 'Content-Length': '0'} 2023-10-18T20:30:17.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb33623fc824a83e-0065304059-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.693 DEBUG:boto:b'bofrnf-58999falseobj00000000000000000000nulltrue2023-10-18T20:30:07.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-18T20:30:07.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-18T20:30:07.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-18T20:30:07.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-18T20:30:07.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-18T20:30:07.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-18T20:30:07.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-18T20:30:07.193Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-18T20:30:07.311Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-18T20:30:07.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:17.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-18T20:30:17.694 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.695 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.695 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.695 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.696 DEBUG:boto:Method: GET 2023-10-18T20:30:17.696 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.697 DEBUG:boto:Data: 2023-10-18T20:30:17.697 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.697 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.698 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.698 DEBUG:boto:Params: {} 2023-10-18T20:30:17.699 DEBUG:boto:Token: None 2023-10-18T20:30:17.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c1s8yZ907im5Nv8rT9qkAdv0Uvc= 2023-10-18T20:30:17.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c1s8yZ907im5Nv8rT9qkAdv0Uvc=', 'Content-Length': '0'} 2023-10-18T20:30:17.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddca880a402c2686-0065304059-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.706 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.706 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.707 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.707 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.708 DEBUG:boto:Method: GET 2023-10-18T20:30:17.708 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.709 DEBUG:boto:Data: 2023-10-18T20:30:17.709 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.710 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.710 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.711 DEBUG:boto:Params: {} 2023-10-18T20:30:17.711 DEBUG:boto:Token: None 2023-10-18T20:30:17.712 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.713 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c1s8yZ907im5Nv8rT9qkAdv0Uvc= 2023-10-18T20:30:17.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c1s8yZ907im5Nv8rT9qkAdv0Uvc=', 'Content-Length': '0'} 2023-10-18T20:30:17.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000895c03c9ca8b8d2e-0065304059-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.758 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.759 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.759 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.760 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.760 DEBUG:boto:Method: HEAD 2023-10-18T20:30:17.761 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.761 DEBUG:boto:Data: 2023-10-18T20:30:17.761 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.762 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.763 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.763 DEBUG:boto:Params: {} 2023-10-18T20:30:17.763 DEBUG:boto:Token: None 2023-10-18T20:30:17.764 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hLPNxVX1rT3OnmxFsgqfd0fYZGM= 2023-10-18T20:30:17.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hLPNxVX1rT3OnmxFsgqfd0fYZGM=', 'Content-Length': '0'} 2023-10-18T20:30:17.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000383d4aeb1f0ee78f-0065304059-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.767 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.768 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.769 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.770 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.770 DEBUG:boto:Method: HEAD 2023-10-18T20:30:17.771 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.771 DEBUG:boto:Data: 2023-10-18T20:30:17.771 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.772 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.772 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.773 DEBUG:boto:Params: {} 2023-10-18T20:30:17.773 DEBUG:boto:Token: None 2023-10-18T20:30:17.773 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hLPNxVX1rT3OnmxFsgqfd0fYZGM= 2023-10-18T20:30:17.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hLPNxVX1rT3OnmxFsgqfd0fYZGM=', 'Content-Length': '0'} 2023-10-18T20:30:17.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000739d11844b5f9ca2-0065304059-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-18T20:30:17.778 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.778 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.779 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.779 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.779 DEBUG:boto:Method: GET 2023-10-18T20:30:17.780 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.780 DEBUG:boto:Data: 2023-10-18T20:30:17.781 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.781 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.782 DEBUG:boto:Params: {} 2023-10-18T20:30:17.782 DEBUG:boto:Token: None 2023-10-18T20:30:17.783 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c1s8yZ907im5Nv8rT9qkAdv0Uvc= 2023-10-18T20:30:17.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c1s8yZ907im5Nv8rT9qkAdv0Uvc=', 'Content-Length': '0'} 2023-10-18T20:30:17.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f8d30e0bed874f-0065304059-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.826 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.827 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.827 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.828 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.828 DEBUG:boto:Method: GET 2023-10-18T20:30:17.828 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.829 DEBUG:boto:Data: 2023-10-18T20:30:17.829 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.830 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.830 DEBUG:boto:Params: {} 2023-10-18T20:30:17.831 DEBUG:boto:Token: None 2023-10-18T20:30:17.833 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:17.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c1s8yZ907im5Nv8rT9qkAdv0Uvc= 2023-10-18T20:30:17.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c1s8yZ907im5Nv8rT9qkAdv0Uvc=', 'Content-Length': '0'} 2023-10-18T20:30:17.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a92a4d4a0bc9f019-0065304059-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.837 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.837 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.838 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.838 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.839 DEBUG:boto:Method: GET 2023-10-18T20:30:17.839 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.840 DEBUG:boto:Data: 2023-10-18T20:30:17.840 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.841 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.841 DEBUG:boto:Params: {} 2023-10-18T20:30:17.842 DEBUG:boto:Token: None 2023-10-18T20:30:17.842 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/ 2023-10-18T20:30:17.843 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kM3n47Y2L9xRClUquQyyM0Bv7bo= 2023-10-18T20:30:17.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kM3n47Y2L9xRClUquQyyM0Bv7bo=', 'Content-Length': '0'} 2023-10-18T20:30:17.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000317699527cbe3f61-0065304059-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.846 DEBUG:boto:b'bofrnf-58obj000000000000000000001000falseobj000000000000000000002023-10-18T20:30:07.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:17.847 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.847 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.848 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.848 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.849 DEBUG:boto:Method: GET 2023-10-18T20:30:17.849 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.849 DEBUG:boto:Data: 2023-10-18T20:30:17.850 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.850 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.851 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.851 DEBUG:boto:Params: {} 2023-10-18T20:30:17.851 DEBUG:boto:Token: None 2023-10-18T20:30:17.852 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/ 2023-10-18T20:30:17.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kM3n47Y2L9xRClUquQyyM0Bv7bo= 2023-10-18T20:30:17.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kM3n47Y2L9xRClUquQyyM0Bv7bo=', 'Content-Length': '0'} 2023-10-18T20:30:17.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b28cce99aa6f0934-0065304059-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.856 DEBUG:boto:b'bofrnf-58obj000000000000000000001000falseobj000000000000000000002023-10-18T20:30:07.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:17.856 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.857 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.857 DEBUG:boto:Method: HEAD 2023-10-18T20:30:17.858 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.858 DEBUG:boto:Data: 2023-10-18T20:30:17.858 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.859 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.859 DEBUG:boto:Params: {} 2023-10-18T20:30:17.861 DEBUG:boto:Token: None 2023-10-18T20:30:17.861 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Hvaz6ygEoQnAcg2rOiplBgyfp9A= 2023-10-18T20:30:17.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Hvaz6ygEoQnAcg2rOiplBgyfp9A=', 'Content-Length': '0'} 2023-10-18T20:30:17.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003761d80d13d7d6dc-0065304059-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.865 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.865 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.866 DEBUG:boto:Method: HEAD 2023-10-18T20:30:17.866 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.867 DEBUG:boto:Data: 2023-10-18T20:30:17.867 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.868 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.869 DEBUG:boto:Params: {} 2023-10-18T20:30:17.869 DEBUG:boto:Token: None 2023-10-18T20:30:17.870 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.871 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Hvaz6ygEoQnAcg2rOiplBgyfp9A= 2023-10-18T20:30:17.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Hvaz6ygEoQnAcg2rOiplBgyfp9A=', 'Content-Length': '0'} 2023-10-18T20:30:17.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a716491cb3e3563-0065304059-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-18T20:30:17.874 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.875 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.875 DEBUG:boto:Method: GET 2023-10-18T20:30:17.876 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.876 DEBUG:boto:Data: 2023-10-18T20:30:17.877 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.877 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.878 DEBUG:boto:Params: {} 2023-10-18T20:30:17.878 DEBUG:boto:Token: None 2023-10-18T20:30:17.879 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.879 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:017bKQhss57PcltgywH5MT/XTP8= 2023-10-18T20:30:17.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:017bKQhss57PcltgywH5MT/XTP8=', 'Content-Length': '0'} 2023-10-18T20:30:17.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a092dc83914dba68-0065304059-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.883 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.884 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.885 DEBUG:boto:Method: GET 2023-10-18T20:30:17.886 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.886 DEBUG:boto:Data: 2023-10-18T20:30:17.887 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.888 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.889 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.889 DEBUG:boto:Params: {} 2023-10-18T20:30:17.890 DEBUG:boto:Token: None 2023-10-18T20:30:17.891 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:17.892 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:017bKQhss57PcltgywH5MT/XTP8= 2023-10-18T20:30:17.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:017bKQhss57PcltgywH5MT/XTP8=', 'Content-Length': '0'} 2023-10-18T20:30:17.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b260cc4b1ab6b9-0065304059-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.897 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.897 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.898 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.898 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.899 DEBUG:boto:Method: GET 2023-10-18T20:30:17.899 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.899 DEBUG:boto:Data: 2023-10-18T20:30:17.900 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.901 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.901 DEBUG:boto:Params: {} 2023-10-18T20:30:17.902 DEBUG:boto:Token: None 2023-10-18T20:30:17.902 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/ 2023-10-18T20:30:17.904 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kM3n47Y2L9xRClUquQyyM0Bv7bo= 2023-10-18T20:30:17.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kM3n47Y2L9xRClUquQyyM0Bv7bo=', 'Content-Length': '0'} 2023-10-18T20:30:17.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bcf7a4e358fe45e-0065304059-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.908 DEBUG:boto:b'bofrnf-58obj000000000000000000001000falseobj000000000000000000002023-10-18T20:30:07.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:17.909 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:17.909 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:17.909 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.910 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.910 DEBUG:boto:Method: GET 2023-10-18T20:30:17.911 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:17.911 DEBUG:boto:Data: 2023-10-18T20:30:17.912 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.913 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.913 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.913 DEBUG:boto:Params: {} 2023-10-18T20:30:17.915 DEBUG:boto:Token: None 2023-10-18T20:30:17.915 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/ 2023-10-18T20:30:17.916 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kM3n47Y2L9xRClUquQyyM0Bv7bo= 2023-10-18T20:30:17.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kM3n47Y2L9xRClUquQyyM0Bv7bo=', 'Content-Length': '0'} 2023-10-18T20:30:17.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cd1f1dca3740e09-0065304059-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.920 DEBUG:boto:b'bofrnf-58obj000000000000000000001000falseobj000000000000000000002023-10-18T20:30:07.374Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:17.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-18T20:30:17.921 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:17.921 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:17.922 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.922 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.923 DEBUG:boto:Method: GET 2023-10-18T20:30:17.923 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.924 DEBUG:boto:Data: 2023-10-18T20:30:17.924 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.925 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.925 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.926 DEBUG:boto:Params: {} 2023-10-18T20:30:17.927 DEBUG:boto:Token: None 2023-10-18T20:30:17.928 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wUAtGaJJ50ZVEC2T+PJx++Y2rOQ= 2023-10-18T20:30:17.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wUAtGaJJ50ZVEC2T+PJx++Y2rOQ=', 'Content-Length': '0'} 2023-10-18T20:30:17.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c2853f646a1d479-0065304059-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.933 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:17.934 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:17.935 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.935 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.935 DEBUG:boto:Method: GET 2023-10-18T20:30:17.936 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.936 DEBUG:boto:Data: 2023-10-18T20:30:17.937 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:17.938 DEBUG:boto:Port: 8000 2023-10-18T20:30:17.938 DEBUG:boto:Params: {} 2023-10-18T20:30:17.939 DEBUG:boto:Token: None 2023-10-18T20:30:17.939 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.940 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wUAtGaJJ50ZVEC2T+PJx++Y2rOQ= 2023-10-18T20:30:17.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wUAtGaJJ50ZVEC2T+PJx++Y2rOQ=', 'Content-Length': '0'} 2023-10-18T20:30:17.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d081f4f4c5f7d61-0065304059-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:17.990 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:17.991 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:17.991 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.992 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.992 DEBUG:boto:Method: HEAD 2023-10-18T20:30:17.993 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.993 DEBUG:boto:Data: 2023-10-18T20:30:17.993 DEBUG:boto:Headers: {} 2023-10-18T20:30:17.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:17.994 DEBUG:boto:Port: 8001 2023-10-18T20:30:17.995 DEBUG:boto:Params: {} 2023-10-18T20:30:17.995 DEBUG:boto:Token: None 2023-10-18T20:30:17.996 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:17 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:17.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sxfWeI0IrYh1d/4Sgww0BRr+Nio= 2023-10-18T20:30:17.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sxfWeI0IrYh1d/4Sgww0BRr+Nio=', 'Content-Length': '0'} 2023-10-18T20:30:17.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d1137b0367bd72c-0065304059-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.000 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.000 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.001 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.001 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.001 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.002 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.002 DEBUG:boto:Data: 2023-10-18T20:30:18.003 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.003 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.004 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.004 DEBUG:boto:Params: {} 2023-10-18T20:30:18.004 DEBUG:boto:Token: None 2023-10-18T20:30:18.005 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.005 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oLBpsNvva8qoaVBNUNNYkgtb5N8= 2023-10-18T20:30:18.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oLBpsNvva8qoaVBNUNNYkgtb5N8=', 'Content-Length': '0'} 2023-10-18T20:30:18.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000637e0776c0417a0c-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-18T20:30:18.009 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.010 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.010 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.011 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.011 DEBUG:boto:Method: GET 2023-10-18T20:30:18.011 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.012 DEBUG:boto:Data: 2023-10-18T20:30:18.013 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.013 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.014 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.015 DEBUG:boto:Params: {} 2023-10-18T20:30:18.016 DEBUG:boto:Token: None 2023-10-18T20:30:18.017 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.018 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bgm6nYEJEUaAgGQmahblHFwpDi8= 2023-10-18T20:30:18.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bgm6nYEJEUaAgGQmahblHFwpDi8=', 'Content-Length': '0'} 2023-10-18T20:30:18.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d754f5453814e3f4-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.062 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.063 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.064 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.064 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.064 DEBUG:boto:Method: GET 2023-10-18T20:30:18.065 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.065 DEBUG:boto:Data: 2023-10-18T20:30:18.066 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.067 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.067 DEBUG:boto:Params: {} 2023-10-18T20:30:18.068 DEBUG:boto:Token: None 2023-10-18T20:30:18.068 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:18.069 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bgm6nYEJEUaAgGQmahblHFwpDi8= 2023-10-18T20:30:18.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bgm6nYEJEUaAgGQmahblHFwpDi8=', 'Content-Length': '0'} 2023-10-18T20:30:18.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007623ddea80eee6b5-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.073 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.073 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.074 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.074 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.075 DEBUG:boto:Method: GET 2023-10-18T20:30:18.075 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.076 DEBUG:boto:Data: 2023-10-18T20:30:18.076 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.077 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.077 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.078 DEBUG:boto:Params: {} 2023-10-18T20:30:18.079 DEBUG:boto:Token: None 2023-10-18T20:30:18.079 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.080 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bbdb2bab7857c7-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.086 DEBUG:boto:b'bofrnf-58obj000000000000000000011000falseobj000000000000000000012023-10-18T20:30:07.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.087 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.087 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.088 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.088 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.089 DEBUG:boto:Method: GET 2023-10-18T20:30:18.089 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.090 DEBUG:boto:Data: 2023-10-18T20:30:18.090 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.091 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.091 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.091 DEBUG:boto:Params: {} 2023-10-18T20:30:18.092 DEBUG:boto:Token: None 2023-10-18T20:30:18.093 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.093 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d655c633d95ba31-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.097 DEBUG:boto:b'bofrnf-58obj000000000000000000011000falseobj000000000000000000012023-10-18T20:30:07.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.097 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.098 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.098 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.099 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.099 DEBUG:boto:Data: 2023-10-18T20:30:18.099 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.100 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.100 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.100 DEBUG:boto:Params: {} 2023-10-18T20:30:18.101 DEBUG:boto:Token: None 2023-10-18T20:30:18.101 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.102 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z8/p54MlzXscfYlcmbnWWQnEzus= 2023-10-18T20:30:18.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z8/p54MlzXscfYlcmbnWWQnEzus=', 'Content-Length': '0'} 2023-10-18T20:30:18.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000732feaebd1126a9d-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.105 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.105 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.105 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.106 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.106 DEBUG:boto:Data: 2023-10-18T20:30:18.107 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.107 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.107 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.108 DEBUG:boto:Params: {} 2023-10-18T20:30:18.108 DEBUG:boto:Token: None 2023-10-18T20:30:18.109 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.109 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z8/p54MlzXscfYlcmbnWWQnEzus= 2023-10-18T20:30:18.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z8/p54MlzXscfYlcmbnWWQnEzus=', 'Content-Length': '0'} 2023-10-18T20:30:18.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1233f3fbd90cb63-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-18T20:30:18.112 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.113 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.113 DEBUG:boto:Method: GET 2023-10-18T20:30:18.114 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.114 DEBUG:boto:Data: 2023-10-18T20:30:18.114 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.115 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.115 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.116 DEBUG:boto:Params: {} 2023-10-18T20:30:18.116 DEBUG:boto:Token: None 2023-10-18T20:30:18.117 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.117 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RvdczFk1soLgX2chiy7tZTP+m0Y= 2023-10-18T20:30:18.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RvdczFk1soLgX2chiy7tZTP+m0Y=', 'Content-Length': '0'} 2023-10-18T20:30:18.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a5c31832bfeb254-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.121 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.121 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.121 DEBUG:boto:Method: GET 2023-10-18T20:30:18.122 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.122 DEBUG:boto:Data: 2023-10-18T20:30:18.123 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.123 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.123 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.124 DEBUG:boto:Params: {} 2023-10-18T20:30:18.124 DEBUG:boto:Token: None 2023-10-18T20:30:18.125 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:18.125 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RvdczFk1soLgX2chiy7tZTP+m0Y= 2023-10-18T20:30:18.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RvdczFk1soLgX2chiy7tZTP+m0Y=', 'Content-Length': '0'} 2023-10-18T20:30:18.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063aa485a7240e822-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.128 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.129 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.129 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.129 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.130 DEBUG:boto:Method: GET 2023-10-18T20:30:18.130 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.131 DEBUG:boto:Data: 2023-10-18T20:30:18.131 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.131 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.132 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.132 DEBUG:boto:Params: {} 2023-10-18T20:30:18.133 DEBUG:boto:Token: None 2023-10-18T20:30:18.133 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.134 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388b4f3c21d19a49-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.138 DEBUG:boto:b'bofrnf-58obj000000000000000000011000falseobj000000000000000000012023-10-18T20:30:07.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.139 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.140 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.140 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.140 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.141 DEBUG:boto:Method: GET 2023-10-18T20:30:18.141 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:18.141 DEBUG:boto:Data: 2023-10-18T20:30:18.142 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.143 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.143 DEBUG:boto:Params: {} 2023-10-18T20:30:18.144 DEBUG:boto:Token: None 2023-10-18T20:30:18.144 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ae6bd561997c4d0-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.148 DEBUG:boto:b'bofrnf-58obj000000000000000000011000falseobj000000000000000000012023-10-18T20:30:07.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-18T20:30:18.149 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.150 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.150 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.150 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.151 DEBUG:boto:Method: GET 2023-10-18T20:30:18.151 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.152 DEBUG:boto:Data: 2023-10-18T20:30:18.152 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.152 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.153 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.153 DEBUG:boto:Params: {} 2023-10-18T20:30:18.154 DEBUG:boto:Token: None 2023-10-18T20:30:18.154 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.155 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ssjCRzxhQxk9adx/1y3eZO/8ncQ= 2023-10-18T20:30:18.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ssjCRzxhQxk9adx/1y3eZO/8ncQ=', 'Content-Length': '0'} 2023-10-18T20:30:18.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099ff68cb3676d6d4-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.158 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.159 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.159 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.159 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.160 DEBUG:boto:Method: GET 2023-10-18T20:30:18.160 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.161 DEBUG:boto:Data: 2023-10-18T20:30:18.161 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.161 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.162 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.162 DEBUG:boto:Params: {} 2023-10-18T20:30:18.163 DEBUG:boto:Token: None 2023-10-18T20:30:18.163 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ssjCRzxhQxk9adx/1y3eZO/8ncQ= 2023-10-18T20:30:18.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ssjCRzxhQxk9adx/1y3eZO/8ncQ=', 'Content-Length': '0'} 2023-10-18T20:30:18.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d9f4d0174755d26-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.166 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.167 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.167 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.168 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.168 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.168 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.169 DEBUG:boto:Data: 2023-10-18T20:30:18.169 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.170 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.170 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.170 DEBUG:boto:Params: {} 2023-10-18T20:30:18.171 DEBUG:boto:Token: None 2023-10-18T20:30:18.171 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.172 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RuB1Bc6+xnI+p9Pgn+zp0fYy9xY= 2023-10-18T20:30:18.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RuB1Bc6+xnI+p9Pgn+zp0fYy9xY=', 'Content-Length': '0'} 2023-10-18T20:30:18.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ca6cac96205b79-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.174 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.175 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.175 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.175 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.176 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.176 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.177 DEBUG:boto:Data: 2023-10-18T20:30:18.177 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.177 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.178 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.178 DEBUG:boto:Params: {} 2023-10-18T20:30:18.179 DEBUG:boto:Token: None 2023-10-18T20:30:18.179 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RuB1Bc6+xnI+p9Pgn+zp0fYy9xY= 2023-10-18T20:30:18.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RuB1Bc6+xnI+p9Pgn+zp0fYy9xY=', 'Content-Length': '0'} 2023-10-18T20:30:18.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011648ea0fc0df107-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-18T20:30:18.183 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.184 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.184 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.184 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.185 DEBUG:boto:Method: GET 2023-10-18T20:30:18.185 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.186 DEBUG:boto:Data: 2023-10-18T20:30:18.186 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.187 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.187 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.187 DEBUG:boto:Params: {} 2023-10-18T20:30:18.188 DEBUG:boto:Token: None 2023-10-18T20:30:18.188 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.189 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ssjCRzxhQxk9adx/1y3eZO/8ncQ= 2023-10-18T20:30:18.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ssjCRzxhQxk9adx/1y3eZO/8ncQ=', 'Content-Length': '0'} 2023-10-18T20:30:18.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fc0353908cd37ef-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.192 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.192 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.193 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.193 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.194 DEBUG:boto:Method: GET 2023-10-18T20:30:18.194 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.195 DEBUG:boto:Data: 2023-10-18T20:30:18.195 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.195 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.196 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.196 DEBUG:boto:Params: {} 2023-10-18T20:30:18.197 DEBUG:boto:Token: None 2023-10-18T20:30:18.197 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:18.198 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ssjCRzxhQxk9adx/1y3eZO/8ncQ= 2023-10-18T20:30:18.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ssjCRzxhQxk9adx/1y3eZO/8ncQ=', 'Content-Length': '0'} 2023-10-18T20:30:18.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000836809d1391b1bbe-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.201 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.201 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.202 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.202 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.203 DEBUG:boto:Method: GET 2023-10-18T20:30:18.203 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.203 DEBUG:boto:Data: 2023-10-18T20:30:18.204 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.204 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.204 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.205 DEBUG:boto:Params: {} 2023-10-18T20:30:18.205 DEBUG:boto:Token: None 2023-10-18T20:30:18.206 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.206 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016a9c229eb6df709-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.210 DEBUG:boto:b'bofrnf-58obj000000000000000000021000falseobj000000000000000000022023-10-18T20:30:07.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.211 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.211 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.212 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.212 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.212 DEBUG:boto:Method: GET 2023-10-18T20:30:18.213 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.213 DEBUG:boto:Data: 2023-10-18T20:30:18.214 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.214 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.214 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.215 DEBUG:boto:Params: {} 2023-10-18T20:30:18.215 DEBUG:boto:Token: None 2023-10-18T20:30:18.216 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.216 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b417250fe6aaac8a-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.220 DEBUG:boto:b'bofrnf-58obj000000000000000000021000falseobj000000000000000000022023-10-18T20:30:07.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.220 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.221 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.221 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.222 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.222 DEBUG:boto:Data: 2023-10-18T20:30:18.223 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.223 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.223 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.224 DEBUG:boto:Params: {} 2023-10-18T20:30:18.224 DEBUG:boto:Token: None 2023-10-18T20:30:18.225 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.225 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:080K6L6pAbVXz0YoLi5aEWsHv/Y= 2023-10-18T20:30:18.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:080K6L6pAbVXz0YoLi5aEWsHv/Y=', 'Content-Length': '0'} 2023-10-18T20:30:18.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c91342b2faccd08-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.228 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.229 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.229 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.229 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.230 DEBUG:boto:Data: 2023-10-18T20:30:18.230 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.231 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.231 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.231 DEBUG:boto:Params: {} 2023-10-18T20:30:18.232 DEBUG:boto:Token: None 2023-10-18T20:30:18.232 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.233 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:080K6L6pAbVXz0YoLi5aEWsHv/Y= 2023-10-18T20:30:18.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:080K6L6pAbVXz0YoLi5aEWsHv/Y=', 'Content-Length': '0'} 2023-10-18T20:30:18.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4d5466108ad556d-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-18T20:30:18.236 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.237 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.237 DEBUG:boto:Method: GET 2023-10-18T20:30:18.238 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.238 DEBUG:boto:Data: 2023-10-18T20:30:18.238 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.239 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.239 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.240 DEBUG:boto:Params: {} 2023-10-18T20:30:18.240 DEBUG:boto:Token: None 2023-10-18T20:30:18.241 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.241 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XMb83lcMu3LNBkzYM2IEM7VlC4s= 2023-10-18T20:30:18.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XMb83lcMu3LNBkzYM2IEM7VlC4s=', 'Content-Length': '0'} 2023-10-18T20:30:18.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d98410e4f4d580cc-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.244 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.245 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.245 DEBUG:boto:Method: GET 2023-10-18T20:30:18.246 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.246 DEBUG:boto:Data: 2023-10-18T20:30:18.246 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.247 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.247 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.249 DEBUG:boto:Params: {} 2023-10-18T20:30:18.250 DEBUG:boto:Token: None 2023-10-18T20:30:18.250 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:18.251 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XMb83lcMu3LNBkzYM2IEM7VlC4s= 2023-10-18T20:30:18.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XMb83lcMu3LNBkzYM2IEM7VlC4s=', 'Content-Length': '0'} 2023-10-18T20:30:18.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b8fc7edeadee24e-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.298 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.299 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.299 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.300 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.300 DEBUG:boto:Method: GET 2023-10-18T20:30:18.300 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.301 DEBUG:boto:Data: 2023-10-18T20:30:18.301 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.301 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.302 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.302 DEBUG:boto:Params: {} 2023-10-18T20:30:18.303 DEBUG:boto:Token: None 2023-10-18T20:30:18.303 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000492108218ab6cfca-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.309 DEBUG:boto:b'bofrnf-58obj000000000000000000021000falseobj000000000000000000022023-10-18T20:30:07.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.310 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.310 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.310 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.311 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.311 DEBUG:boto:Method: GET 2023-10-18T20:30:18.312 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:18.312 DEBUG:boto:Data: 2023-10-18T20:30:18.313 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.313 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.313 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.314 DEBUG:boto:Params: {} 2023-10-18T20:30:18.314 DEBUG:boto:Token: None 2023-10-18T20:30:18.315 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cf32f3dcf79c7dc-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.319 DEBUG:boto:b'bofrnf-58obj000000000000000000021000falseobj000000000000000000022023-10-18T20:30:07.290Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-18T20:30:18.320 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.321 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.321 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.322 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.322 DEBUG:boto:Method: GET 2023-10-18T20:30:18.323 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.323 DEBUG:boto:Data: 2023-10-18T20:30:18.324 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.324 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.324 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.325 DEBUG:boto:Params: {} 2023-10-18T20:30:18.325 DEBUG:boto:Token: None 2023-10-18T20:30:18.326 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.326 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ByXPfAD1J/lTTlFCiwp5Vvkk/T0= 2023-10-18T20:30:18.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ByXPfAD1J/lTTlFCiwp5Vvkk/T0=', 'Content-Length': '0'} 2023-10-18T20:30:18.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e28d2b84ba171c51-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.330 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.330 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.330 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.331 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.331 DEBUG:boto:Method: GET 2023-10-18T20:30:18.332 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.332 DEBUG:boto:Data: 2023-10-18T20:30:18.333 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.334 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.334 DEBUG:boto:Params: {} 2023-10-18T20:30:18.335 DEBUG:boto:Token: None 2023-10-18T20:30:18.335 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.335 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ByXPfAD1J/lTTlFCiwp5Vvkk/T0= 2023-10-18T20:30:18.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ByXPfAD1J/lTTlFCiwp5Vvkk/T0=', 'Content-Length': '0'} 2023-10-18T20:30:18.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7ccff9c684df56-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.339 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.339 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.340 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.340 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.340 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.341 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.341 DEBUG:boto:Data: 2023-10-18T20:30:18.342 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.342 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.343 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.343 DEBUG:boto:Params: {} 2023-10-18T20:30:18.344 DEBUG:boto:Token: None 2023-10-18T20:30:18.344 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.345 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QlWjZyvLiXiEPk7nK+xxw8TQ9cc= 2023-10-18T20:30:18.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QlWjZyvLiXiEPk7nK+xxw8TQ9cc=', 'Content-Length': '0'} 2023-10-18T20:30:18.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018dec8f2b13aca67-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.348 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.348 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.349 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.349 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.349 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.350 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.350 DEBUG:boto:Data: 2023-10-18T20:30:18.351 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.351 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.351 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.352 DEBUG:boto:Params: {} 2023-10-18T20:30:18.352 DEBUG:boto:Token: None 2023-10-18T20:30:18.353 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.353 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QlWjZyvLiXiEPk7nK+xxw8TQ9cc= 2023-10-18T20:30:18.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QlWjZyvLiXiEPk7nK+xxw8TQ9cc=', 'Content-Length': '0'} 2023-10-18T20:30:18.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a56bbd47f316c2f-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-18T20:30:18.357 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.357 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.357 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.358 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.358 DEBUG:boto:Method: GET 2023-10-18T20:30:18.359 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.359 DEBUG:boto:Data: 2023-10-18T20:30:18.359 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.360 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.360 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.361 DEBUG:boto:Params: {} 2023-10-18T20:30:18.361 DEBUG:boto:Token: None 2023-10-18T20:30:18.362 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.362 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ByXPfAD1J/lTTlFCiwp5Vvkk/T0= 2023-10-18T20:30:18.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ByXPfAD1J/lTTlFCiwp5Vvkk/T0=', 'Content-Length': '0'} 2023-10-18T20:30:18.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eecbdd1059111f59-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.406 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.407 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.408 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.408 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.409 DEBUG:boto:Method: GET 2023-10-18T20:30:18.409 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.409 DEBUG:boto:Data: 2023-10-18T20:30:18.410 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.410 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.411 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.411 DEBUG:boto:Params: {} 2023-10-18T20:30:18.412 DEBUG:boto:Token: None 2023-10-18T20:30:18.412 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:18.413 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ByXPfAD1J/lTTlFCiwp5Vvkk/T0= 2023-10-18T20:30:18.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ByXPfAD1J/lTTlFCiwp5Vvkk/T0=', 'Content-Length': '0'} 2023-10-18T20:30:18.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7c6f117ed170c5-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.416 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.416 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.417 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.418 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.418 DEBUG:boto:Method: GET 2023-10-18T20:30:18.419 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.419 DEBUG:boto:Data: 2023-10-18T20:30:18.419 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.420 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.420 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.421 DEBUG:boto:Params: {} 2023-10-18T20:30:18.421 DEBUG:boto:Token: None 2023-10-18T20:30:18.422 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.422 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004410e7e4d38e41f6-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.426 DEBUG:boto:b'bofrnf-58obj000000000000000000031000falseobj000000000000000000032023-10-18T20:30:07.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.427 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.427 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.428 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.428 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.428 DEBUG:boto:Method: GET 2023-10-18T20:30:18.429 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.430 DEBUG:boto:Data: 2023-10-18T20:30:18.430 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.431 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.431 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.431 DEBUG:boto:Params: {} 2023-10-18T20:30:18.432 DEBUG:boto:Token: None 2023-10-18T20:30:18.432 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.433 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007032cfe26f12ba4d-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.437 DEBUG:boto:b'bofrnf-58obj000000000000000000031000falseobj000000000000000000032023-10-18T20:30:07.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.437 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.438 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.438 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.439 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.439 DEBUG:boto:Data: 2023-10-18T20:30:18.439 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.440 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.440 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.441 DEBUG:boto:Params: {} 2023-10-18T20:30:18.441 DEBUG:boto:Token: None 2023-10-18T20:30:18.441 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.442 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GXT7/PcUiaXDMDMb4zM7Fqmg0Aw= 2023-10-18T20:30:18.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GXT7/PcUiaXDMDMb4zM7Fqmg0Aw=', 'Content-Length': '0'} 2023-10-18T20:30:18.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000658d34991cbd2fa2-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.445 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.445 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.446 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.446 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.447 DEBUG:boto:Data: 2023-10-18T20:30:18.447 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.447 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.448 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.448 DEBUG:boto:Params: {} 2023-10-18T20:30:18.449 DEBUG:boto:Token: None 2023-10-18T20:30:18.449 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.450 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GXT7/PcUiaXDMDMb4zM7Fqmg0Aw= 2023-10-18T20:30:18.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GXT7/PcUiaXDMDMb4zM7Fqmg0Aw=', 'Content-Length': '0'} 2023-10-18T20:30:18.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad34b4c009d5dc9-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-18T20:30:18.453 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.453 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.454 DEBUG:boto:Method: GET 2023-10-18T20:30:18.454 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.457 DEBUG:boto:Data: 2023-10-18T20:30:18.458 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.458 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.458 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.459 DEBUG:boto:Params: {} 2023-10-18T20:30:18.459 DEBUG:boto:Token: None 2023-10-18T20:30:18.460 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WgC9jrSiaUD4Ub7GGd8Bihn3yBk= 2023-10-18T20:30:18.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WgC9jrSiaUD4Ub7GGd8Bihn3yBk=', 'Content-Length': '0'} 2023-10-18T20:30:18.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b7c7c5285d4aaf-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.463 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.464 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.464 DEBUG:boto:Method: GET 2023-10-18T20:30:18.465 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.465 DEBUG:boto:Data: 2023-10-18T20:30:18.465 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.466 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.466 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.467 DEBUG:boto:Params: {} 2023-10-18T20:30:18.467 DEBUG:boto:Token: None 2023-10-18T20:30:18.468 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:18.468 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WgC9jrSiaUD4Ub7GGd8Bihn3yBk= 2023-10-18T20:30:18.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WgC9jrSiaUD4Ub7GGd8Bihn3yBk=', 'Content-Length': '0'} 2023-10-18T20:30:18.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b0e32c39615560f-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.471 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.472 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.472 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.473 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.473 DEBUG:boto:Method: GET 2023-10-18T20:30:18.473 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.474 DEBUG:boto:Data: 2023-10-18T20:30:18.474 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.475 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.475 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.475 DEBUG:boto:Params: {} 2023-10-18T20:30:18.476 DEBUG:boto:Token: None 2023-10-18T20:30:18.476 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.477 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a7a4923fac74faf-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.481 DEBUG:boto:b'bofrnf-58obj000000000000000000031000falseobj000000000000000000032023-10-18T20:30:07.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.481 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.482 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.483 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.483 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.483 DEBUG:boto:Method: GET 2023-10-18T20:30:18.484 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:18.484 DEBUG:boto:Data: 2023-10-18T20:30:18.485 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.485 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.486 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.486 DEBUG:boto:Params: {} 2023-10-18T20:30:18.486 DEBUG:boto:Token: None 2023-10-18T20:30:18.487 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.487 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ca9050ce671cddd-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.491 DEBUG:boto:b'bofrnf-58obj000000000000000000031000falseobj000000000000000000032023-10-18T20:30:07.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-18T20:30:18.492 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.492 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.493 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.493 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.494 DEBUG:boto:Method: GET 2023-10-18T20:30:18.494 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.494 DEBUG:boto:Data: 2023-10-18T20:30:18.495 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.495 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.496 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.496 DEBUG:boto:Params: {} 2023-10-18T20:30:18.497 DEBUG:boto:Token: None 2023-10-18T20:30:18.497 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.498 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fn19GE8v3rLEICNZT2asJ026sRA= 2023-10-18T20:30:18.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fn19GE8v3rLEICNZT2asJ026sRA=', 'Content-Length': '0'} 2023-10-18T20:30:18.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099399405416602cd-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.501 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.501 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.502 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.502 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.503 DEBUG:boto:Method: GET 2023-10-18T20:30:18.503 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.504 DEBUG:boto:Data: 2023-10-18T20:30:18.504 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.504 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.505 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.505 DEBUG:boto:Params: {} 2023-10-18T20:30:18.506 DEBUG:boto:Token: None 2023-10-18T20:30:18.506 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.507 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fn19GE8v3rLEICNZT2asJ026sRA= 2023-10-18T20:30:18.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fn19GE8v3rLEICNZT2asJ026sRA=', 'Content-Length': '0'} 2023-10-18T20:30:18.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d3918366a12442-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.510 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.510 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.511 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.511 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.511 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.512 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.512 DEBUG:boto:Data: 2023-10-18T20:30:18.512 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.513 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.513 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.514 DEBUG:boto:Params: {} 2023-10-18T20:30:18.514 DEBUG:boto:Token: None 2023-10-18T20:30:18.515 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.515 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDQGt//18QwxhM/jrc/iugL07F8= 2023-10-18T20:30:18.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDQGt//18QwxhM/jrc/iugL07F8=', 'Content-Length': '0'} 2023-10-18T20:30:18.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f8abb717c40900-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.518 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.518 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.519 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.519 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.520 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.520 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.520 DEBUG:boto:Data: 2023-10-18T20:30:18.521 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.521 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.522 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.522 DEBUG:boto:Params: {} 2023-10-18T20:30:18.522 DEBUG:boto:Token: None 2023-10-18T20:30:18.523 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.523 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDQGt//18QwxhM/jrc/iugL07F8= 2023-10-18T20:30:18.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDQGt//18QwxhM/jrc/iugL07F8=', 'Content-Length': '0'} 2023-10-18T20:30:18.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003448cbaf5d466828-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-18T20:30:18.527 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.527 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.528 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.528 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.528 DEBUG:boto:Method: GET 2023-10-18T20:30:18.529 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.529 DEBUG:boto:Data: 2023-10-18T20:30:18.530 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.530 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.531 DEBUG:boto:Params: {} 2023-10-18T20:30:18.531 DEBUG:boto:Token: None 2023-10-18T20:30:18.532 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fn19GE8v3rLEICNZT2asJ026sRA= 2023-10-18T20:30:18.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fn19GE8v3rLEICNZT2asJ026sRA=', 'Content-Length': '0'} 2023-10-18T20:30:18.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1dbab77527b75c-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.536 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.536 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.536 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.537 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.537 DEBUG:boto:Method: GET 2023-10-18T20:30:18.538 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.538 DEBUG:boto:Data: 2023-10-18T20:30:18.539 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.539 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.539 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.540 DEBUG:boto:Params: {} 2023-10-18T20:30:18.540 DEBUG:boto:Token: None 2023-10-18T20:30:18.541 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:18.541 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fn19GE8v3rLEICNZT2asJ026sRA= 2023-10-18T20:30:18.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fn19GE8v3rLEICNZT2asJ026sRA=', 'Content-Length': '0'} 2023-10-18T20:30:18.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccaa8f8b70c5433f-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.545 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.545 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.546 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.546 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.546 DEBUG:boto:Method: GET 2023-10-18T20:30:18.547 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.547 DEBUG:boto:Data: 2023-10-18T20:30:18.548 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.548 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.548 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.549 DEBUG:boto:Params: {} 2023-10-18T20:30:18.549 DEBUG:boto:Token: None 2023-10-18T20:30:18.550 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.550 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032f2dada8031ca37-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.594 DEBUG:boto:b'bofrnf-58obj000000000000000000041000falseobj000000000000000000042023-10-18T20:30:07.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.595 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.596 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.596 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.597 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.597 DEBUG:boto:Method: GET 2023-10-18T20:30:18.597 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.598 DEBUG:boto:Data: 2023-10-18T20:30:18.598 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.599 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.599 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.599 DEBUG:boto:Params: {} 2023-10-18T20:30:18.600 DEBUG:boto:Token: None 2023-10-18T20:30:18.600 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.601 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099c4a90e955db2f5-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.604 DEBUG:boto:b'bofrnf-58obj000000000000000000041000falseobj000000000000000000042023-10-18T20:30:07.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.605 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.605 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.606 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.606 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.607 DEBUG:boto:Data: 2023-10-18T20:30:18.607 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.607 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.608 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.608 DEBUG:boto:Params: {} 2023-10-18T20:30:18.609 DEBUG:boto:Token: None 2023-10-18T20:30:18.609 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.610 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mXFQ5xiqRtXj6GLymPD82IxJfSY= 2023-10-18T20:30:18.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mXFQ5xiqRtXj6GLymPD82IxJfSY=', 'Content-Length': '0'} 2023-10-18T20:30:18.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135657e2c5eee61a-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.612 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.613 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.613 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.614 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.614 DEBUG:boto:Data: 2023-10-18T20:30:18.614 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.615 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.615 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.616 DEBUG:boto:Params: {} 2023-10-18T20:30:18.616 DEBUG:boto:Token: None 2023-10-18T20:30:18.617 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.617 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mXFQ5xiqRtXj6GLymPD82IxJfSY= 2023-10-18T20:30:18.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mXFQ5xiqRtXj6GLymPD82IxJfSY=', 'Content-Length': '0'} 2023-10-18T20:30:18.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa9b466165ea0fe0-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-18T20:30:18.620 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.621 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.621 DEBUG:boto:Method: GET 2023-10-18T20:30:18.622 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.622 DEBUG:boto:Data: 2023-10-18T20:30:18.623 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.623 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.623 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.624 DEBUG:boto:Params: {} 2023-10-18T20:30:18.624 DEBUG:boto:Token: None 2023-10-18T20:30:18.625 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.625 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7c1uZCvxZulemIJppK+cNz+7jDw= 2023-10-18T20:30:18.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7c1uZCvxZulemIJppK+cNz+7jDw=', 'Content-Length': '0'} 2023-10-18T20:30:18.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa4ef250c0fc0a3-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.628 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.629 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.629 DEBUG:boto:Method: GET 2023-10-18T20:30:18.630 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.630 DEBUG:boto:Data: 2023-10-18T20:30:18.631 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.631 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.631 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.632 DEBUG:boto:Params: {} 2023-10-18T20:30:18.632 DEBUG:boto:Token: None 2023-10-18T20:30:18.633 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:18.633 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7c1uZCvxZulemIJppK+cNz+7jDw= 2023-10-18T20:30:18.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7c1uZCvxZulemIJppK+cNz+7jDw=', 'Content-Length': '0'} 2023-10-18T20:30:18.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba59a7a5bee142a-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.637 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.637 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.638 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.638 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.638 DEBUG:boto:Method: GET 2023-10-18T20:30:18.639 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.639 DEBUG:boto:Data: 2023-10-18T20:30:18.640 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.640 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.641 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.641 DEBUG:boto:Params: {} 2023-10-18T20:30:18.641 DEBUG:boto:Token: None 2023-10-18T20:30:18.642 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.642 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afe4e36bf88cc2d4-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.646 DEBUG:boto:b'bofrnf-58obj000000000000000000041000falseobj000000000000000000042023-10-18T20:30:07.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.647 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.647 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.647 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.648 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.648 DEBUG:boto:Method: GET 2023-10-18T20:30:18.649 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:18.649 DEBUG:boto:Data: 2023-10-18T20:30:18.650 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.650 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.651 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.651 DEBUG:boto:Params: {} 2023-10-18T20:30:18.652 DEBUG:boto:Token: None 2023-10-18T20:30:18.653 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004be7ac0cfe03b736-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.657 DEBUG:boto:b'bofrnf-58obj000000000000000000041000falseobj000000000000000000042023-10-18T20:30:07.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-18T20:30:18.658 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.658 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.659 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.659 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.660 DEBUG:boto:Method: GET 2023-10-18T20:30:18.660 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.661 DEBUG:boto:Data: 2023-10-18T20:30:18.661 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.662 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.662 DEBUG:boto:Params: {} 2023-10-18T20:30:18.663 DEBUG:boto:Token: None 2023-10-18T20:30:18.663 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gsOCS0jWMgwe4GsjcuVQ5D/EYaE= 2023-10-18T20:30:18.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gsOCS0jWMgwe4GsjcuVQ5D/EYaE=', 'Content-Length': '0'} 2023-10-18T20:30:18.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d214868ebacd7a50-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.669 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.669 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.670 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.670 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.671 DEBUG:boto:Method: GET 2023-10-18T20:30:18.671 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.672 DEBUG:boto:Data: 2023-10-18T20:30:18.672 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.672 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.673 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.673 DEBUG:boto:Params: {} 2023-10-18T20:30:18.674 DEBUG:boto:Token: None 2023-10-18T20:30:18.674 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.675 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gsOCS0jWMgwe4GsjcuVQ5D/EYaE= 2023-10-18T20:30:18.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gsOCS0jWMgwe4GsjcuVQ5D/EYaE=', 'Content-Length': '0'} 2023-10-18T20:30:18.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a88a46c27a30b5ca-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.678 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.678 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.679 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.679 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.679 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.680 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.680 DEBUG:boto:Data: 2023-10-18T20:30:18.681 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.681 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.682 DEBUG:boto:Params: {} 2023-10-18T20:30:18.682 DEBUG:boto:Token: None 2023-10-18T20:30:18.683 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2ST1mfM28nP4DbETvoThpeU64ZY= 2023-10-18T20:30:18.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2ST1mfM28nP4DbETvoThpeU64ZY=', 'Content-Length': '0'} 2023-10-18T20:30:18.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e6c6acba616f69-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.686 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.686 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.687 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.687 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.688 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.688 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.688 DEBUG:boto:Data: 2023-10-18T20:30:18.689 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.690 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.690 DEBUG:boto:Params: {} 2023-10-18T20:30:18.690 DEBUG:boto:Token: None 2023-10-18T20:30:18.691 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.691 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2ST1mfM28nP4DbETvoThpeU64ZY= 2023-10-18T20:30:18.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2ST1mfM28nP4DbETvoThpeU64ZY=', 'Content-Length': '0'} 2023-10-18T20:30:18.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000367c650be7ab14a7-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-18T20:30:18.695 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.695 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.696 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.697 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.697 DEBUG:boto:Method: GET 2023-10-18T20:30:18.697 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.698 DEBUG:boto:Data: 2023-10-18T20:30:18.698 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.699 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.699 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.700 DEBUG:boto:Params: {} 2023-10-18T20:30:18.700 DEBUG:boto:Token: None 2023-10-18T20:30:18.701 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gsOCS0jWMgwe4GsjcuVQ5D/EYaE= 2023-10-18T20:30:18.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gsOCS0jWMgwe4GsjcuVQ5D/EYaE=', 'Content-Length': '0'} 2023-10-18T20:30:18.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099b040531ddfbbd3-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.704 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.705 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.705 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.705 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.706 DEBUG:boto:Method: GET 2023-10-18T20:30:18.706 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.707 DEBUG:boto:Data: 2023-10-18T20:30:18.707 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.708 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.708 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.708 DEBUG:boto:Params: {} 2023-10-18T20:30:18.709 DEBUG:boto:Token: None 2023-10-18T20:30:18.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:18.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gsOCS0jWMgwe4GsjcuVQ5D/EYaE= 2023-10-18T20:30:18.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gsOCS0jWMgwe4GsjcuVQ5D/EYaE=', 'Content-Length': '0'} 2023-10-18T20:30:18.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6c4812e42ea998b-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.713 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.714 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.714 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.715 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.715 DEBUG:boto:Method: GET 2023-10-18T20:30:18.715 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.716 DEBUG:boto:Data: 2023-10-18T20:30:18.716 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.717 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.717 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.718 DEBUG:boto:Params: {} 2023-10-18T20:30:18.718 DEBUG:boto:Token: None 2023-10-18T20:30:18.719 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.719 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066558d36dbff6676-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.722 DEBUG:boto:b'bofrnf-58obj000000000000000000051000falseobj000000000000000000052023-10-18T20:30:07.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.723 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.724 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.724 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.725 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.725 DEBUG:boto:Method: GET 2023-10-18T20:30:18.725 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.726 DEBUG:boto:Data: 2023-10-18T20:30:18.726 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.727 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.727 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.728 DEBUG:boto:Params: {} 2023-10-18T20:30:18.728 DEBUG:boto:Token: None 2023-10-18T20:30:18.729 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.729 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001efa69c91885122c-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.733 DEBUG:boto:b'bofrnf-58obj000000000000000000051000falseobj000000000000000000052023-10-18T20:30:07.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.734 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.734 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.735 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.735 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.736 DEBUG:boto:Data: 2023-10-18T20:30:18.736 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.736 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.737 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.737 DEBUG:boto:Params: {} 2023-10-18T20:30:18.738 DEBUG:boto:Token: None 2023-10-18T20:30:18.738 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.739 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dy+VEXTVxOO5v5nVRPLTfE+BIJM= 2023-10-18T20:30:18.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dy+VEXTVxOO5v5nVRPLTfE+BIJM=', 'Content-Length': '0'} 2023-10-18T20:30:18.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e75d7527b7392d-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.742 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.742 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.743 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.743 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.743 DEBUG:boto:Data: 2023-10-18T20:30:18.744 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.744 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.745 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.745 DEBUG:boto:Params: {} 2023-10-18T20:30:18.746 DEBUG:boto:Token: None 2023-10-18T20:30:18.746 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.747 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dy+VEXTVxOO5v5nVRPLTfE+BIJM= 2023-10-18T20:30:18.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dy+VEXTVxOO5v5nVRPLTfE+BIJM=', 'Content-Length': '0'} 2023-10-18T20:30:18.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3b22338f9870e93-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-18T20:30:18.750 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.750 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.751 DEBUG:boto:Method: GET 2023-10-18T20:30:18.751 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.752 DEBUG:boto:Data: 2023-10-18T20:30:18.752 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.753 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.753 DEBUG:boto:Params: {} 2023-10-18T20:30:18.754 DEBUG:boto:Token: None 2023-10-18T20:30:18.754 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CZXlEZd7mPa6VUsh2OWhBy9PCII= 2023-10-18T20:30:18.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZXlEZd7mPa6VUsh2OWhBy9PCII=', 'Content-Length': '0'} 2023-10-18T20:30:18.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f133ca05396096b-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.758 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.758 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.759 DEBUG:boto:Method: GET 2023-10-18T20:30:18.759 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.760 DEBUG:boto:Data: 2023-10-18T20:30:18.760 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.760 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.761 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.761 DEBUG:boto:Params: {} 2023-10-18T20:30:18.762 DEBUG:boto:Token: None 2023-10-18T20:30:18.762 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:18.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CZXlEZd7mPa6VUsh2OWhBy9PCII= 2023-10-18T20:30:18.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZXlEZd7mPa6VUsh2OWhBy9PCII=', 'Content-Length': '0'} 2023-10-18T20:30:18.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf7d7bd830a8b9c1-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.766 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.766 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.767 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.767 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.768 DEBUG:boto:Method: GET 2023-10-18T20:30:18.768 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.768 DEBUG:boto:Data: 2023-10-18T20:30:18.769 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.769 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.770 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.770 DEBUG:boto:Params: {} 2023-10-18T20:30:18.771 DEBUG:boto:Token: None 2023-10-18T20:30:18.771 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004db401e1930294d-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.775 DEBUG:boto:b'bofrnf-58obj000000000000000000051000falseobj000000000000000000052023-10-18T20:30:07.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.776 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.776 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.777 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.777 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.778 DEBUG:boto:Method: GET 2023-10-18T20:30:18.778 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:18.779 DEBUG:boto:Data: 2023-10-18T20:30:18.779 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.780 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.780 DEBUG:boto:Params: {} 2023-10-18T20:30:18.781 DEBUG:boto:Token: None 2023-10-18T20:30:18.781 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312c726af56689aa-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.786 DEBUG:boto:b'bofrnf-58obj000000000000000000051000falseobj000000000000000000052023-10-18T20:30:07.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-18T20:30:18.787 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.787 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.788 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.788 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.788 DEBUG:boto:Method: GET 2023-10-18T20:30:18.789 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.789 DEBUG:boto:Data: 2023-10-18T20:30:18.790 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.791 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.791 DEBUG:boto:Params: {} 2023-10-18T20:30:18.792 DEBUG:boto:Token: None 2023-10-18T20:30:18.792 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.793 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:58cHbcEt1WZPFX/UsYgCUjGkplg= 2023-10-18T20:30:18.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:58cHbcEt1WZPFX/UsYgCUjGkplg=', 'Content-Length': '0'} 2023-10-18T20:30:18.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004240e76fdd323f48-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.796 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.797 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.797 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.797 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.798 DEBUG:boto:Method: GET 2023-10-18T20:30:18.798 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.799 DEBUG:boto:Data: 2023-10-18T20:30:18.799 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.800 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.800 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.801 DEBUG:boto:Params: {} 2023-10-18T20:30:18.801 DEBUG:boto:Token: None 2023-10-18T20:30:18.801 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.802 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:58cHbcEt1WZPFX/UsYgCUjGkplg= 2023-10-18T20:30:18.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:58cHbcEt1WZPFX/UsYgCUjGkplg=', 'Content-Length': '0'} 2023-10-18T20:30:18.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9ea98fd8a1d61ca-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.805 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.806 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.806 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.806 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.807 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.807 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.808 DEBUG:boto:Data: 2023-10-18T20:30:18.808 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.808 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.809 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.809 DEBUG:boto:Params: {} 2023-10-18T20:30:18.810 DEBUG:boto:Token: None 2023-10-18T20:30:18.810 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s5aH0BWn29aLh848btIjW7HwuY8= 2023-10-18T20:30:18.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s5aH0BWn29aLh848btIjW7HwuY8=', 'Content-Length': '0'} 2023-10-18T20:30:18.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caec0682ef924b80-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.813 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.814 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.815 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.815 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.816 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.816 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.817 DEBUG:boto:Data: 2023-10-18T20:30:18.817 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.817 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.818 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.818 DEBUG:boto:Params: {} 2023-10-18T20:30:18.819 DEBUG:boto:Token: None 2023-10-18T20:30:18.819 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.820 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s5aH0BWn29aLh848btIjW7HwuY8= 2023-10-18T20:30:18.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s5aH0BWn29aLh848btIjW7HwuY8=', 'Content-Length': '0'} 2023-10-18T20:30:18.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3977c41ee637d13-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-18T20:30:18.823 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.824 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.824 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.825 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.825 DEBUG:boto:Method: GET 2023-10-18T20:30:18.825 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.826 DEBUG:boto:Data: 2023-10-18T20:30:18.826 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.827 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.827 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.828 DEBUG:boto:Params: {} 2023-10-18T20:30:18.828 DEBUG:boto:Token: None 2023-10-18T20:30:18.829 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.829 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:58cHbcEt1WZPFX/UsYgCUjGkplg= 2023-10-18T20:30:18.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:58cHbcEt1WZPFX/UsYgCUjGkplg=', 'Content-Length': '0'} 2023-10-18T20:30:18.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e80e7f987f7fabbf-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.878 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.879 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.880 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.881 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.881 DEBUG:boto:Method: GET 2023-10-18T20:30:18.882 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.882 DEBUG:boto:Data: 2023-10-18T20:30:18.883 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.883 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.883 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.884 DEBUG:boto:Params: {} 2023-10-18T20:30:18.885 DEBUG:boto:Token: None 2023-10-18T20:30:18.885 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:18.886 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:58cHbcEt1WZPFX/UsYgCUjGkplg= 2023-10-18T20:30:18.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:58cHbcEt1WZPFX/UsYgCUjGkplg=', 'Content-Length': '0'} 2023-10-18T20:30:18.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf235881f6858089-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.890 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.890 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.891 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:18.891 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:18.892 DEBUG:boto:Method: GET 2023-10-18T20:30:18.892 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:18.893 DEBUG:boto:Data: 2023-10-18T20:30:18.893 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.894 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.894 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.895 DEBUG:boto:Params: {} 2023-10-18T20:30:18.895 DEBUG:boto:Token: None 2023-10-18T20:30:18.896 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.896 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089318333fe75e14c-006530405a-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.900 DEBUG:boto:b'bofrnf-58obj000000000000000000061000falseobj000000000000000000062023-10-18T20:30:07.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.901 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.901 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.901 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:18.902 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:18.902 DEBUG:boto:Method: GET 2023-10-18T20:30:18.903 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:18.903 DEBUG:boto:Data: 2023-10-18T20:30:18.904 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.904 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.905 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.905 DEBUG:boto:Params: {} 2023-10-18T20:30:18.905 DEBUG:boto:Token: None 2023-10-18T20:30:18.906 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/ 2023-10-18T20:30:18.907 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU= 2023-10-18T20:30:18.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AP0HqluFH6OUPS48fz2XZ6FpQEU=', 'Content-Length': '0'} 2023-10-18T20:30:18.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b9023e81a0a6692-006530405a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.910 DEBUG:boto:b'bofrnf-58obj000000000000000000061000falseobj000000000000000000062023-10-18T20:30:07.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:18.911 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.911 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.912 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.912 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.913 DEBUG:boto:Data: 2023-10-18T20:30:18.913 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.913 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.914 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.914 DEBUG:boto:Params: {} 2023-10-18T20:30:18.915 DEBUG:boto:Token: None 2023-10-18T20:30:18.915 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.916 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oQz1DRxKiq5/BUhafaURZTKYOMQ= 2023-10-18T20:30:18.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oQz1DRxKiq5/BUhafaURZTKYOMQ=', 'Content-Length': '0'} 2023-10-18T20:30:18.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f029f0786230f4d0-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.920 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.921 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.922 DEBUG:boto:Method: HEAD 2023-10-18T20:30:18.922 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.923 DEBUG:boto:Data: 2023-10-18T20:30:18.924 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.924 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.925 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.926 DEBUG:boto:Params: {} 2023-10-18T20:30:18.927 DEBUG:boto:Token: None 2023-10-18T20:30:18.927 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.928 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oQz1DRxKiq5/BUhafaURZTKYOMQ= 2023-10-18T20:30:18.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oQz1DRxKiq5/BUhafaURZTKYOMQ=', 'Content-Length': '0'} 2023-10-18T20:30:18.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dbcb3e2489a5e21-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-18T20:30:18.933 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.934 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.935 DEBUG:boto:Method: GET 2023-10-18T20:30:18.936 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.937 DEBUG:boto:Data: 2023-10-18T20:30:18.937 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.938 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:18.939 DEBUG:boto:Port: 8001 2023-10-18T20:30:18.939 DEBUG:boto:Params: {} 2023-10-18T20:30:18.940 DEBUG:boto:Token: None 2023-10-18T20:30:18.940 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.941 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e43s2SVy7AwpiJurX5+jmrl5DTQ= 2023-10-18T20:30:18.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e43s2SVy7AwpiJurX5+jmrl5DTQ=', 'Content-Length': '0'} 2023-10-18T20:30:18.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e059564c90ffcfdf-006530405a-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.986 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.987 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.988 DEBUG:boto:Method: GET 2023-10-18T20:30:18.989 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.990 DEBUG:boto:Data: 2023-10-18T20:30:18.990 DEBUG:boto:Headers: {} 2023-10-18T20:30:18.991 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:18.991 DEBUG:boto:Port: 8000 2023-10-18T20:30:18.992 DEBUG:boto:Params: {} 2023-10-18T20:30:18.993 DEBUG:boto:Token: None 2023-10-18T20:30:18.993 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:18 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:18.994 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e43s2SVy7AwpiJurX5+jmrl5DTQ= 2023-10-18T20:30:18.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e43s2SVy7AwpiJurX5+jmrl5DTQ=', 'Content-Length': '0'} 2023-10-18T20:30:18.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7e668b95056380-006530405a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:18.997 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:18.998 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:18.998 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:18.998 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:18.999 DEBUG:boto:Method: GET 2023-10-18T20:30:18.999 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:19.000 DEBUG:boto:Data: 2023-10-18T20:30:19.000 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.001 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.001 DEBUG:boto:Params: {} 2023-10-18T20:30:19.002 DEBUG:boto:Token: None 2023-10-18T20:30:19.002 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f93da27715f7ad30-006530405b-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.008 DEBUG:boto:b'bofrnf-58obj000000000000000000061000falseobj000000000000000000062023-10-18T20:30:07.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.010 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.010 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.011 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:19.011 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:19.011 DEBUG:boto:Method: GET 2023-10-18T20:30:19.012 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:19.012 DEBUG:boto:Data: 2023-10-18T20:30:19.013 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.013 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.014 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.015 DEBUG:boto:Params: {} 2023-10-18T20:30:19.015 DEBUG:boto:Token: None 2023-10-18T20:30:19.015 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.016 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c204c944886392cd-006530405b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.020 DEBUG:boto:b'bofrnf-58obj000000000000000000061000falseobj000000000000000000062023-10-18T20:30:07.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-18T20:30:19.021 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.021 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.022 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.022 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.023 DEBUG:boto:Method: GET 2023-10-18T20:30:19.023 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.024 DEBUG:boto:Data: 2023-10-18T20:30:19.024 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.025 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.025 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.026 DEBUG:boto:Params: {} 2023-10-18T20:30:19.027 DEBUG:boto:Token: None 2023-10-18T20:30:19.027 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.028 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TUBF/N+IkTOscL6rgCIXbritUdc= 2023-10-18T20:30:19.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TUBF/N+IkTOscL6rgCIXbritUdc=', 'Content-Length': '0'} 2023-10-18T20:30:19.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344f7a6bca9f602c-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.031 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.031 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.032 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.032 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.033 DEBUG:boto:Method: GET 2023-10-18T20:30:19.033 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.034 DEBUG:boto:Data: 2023-10-18T20:30:19.034 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.035 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.036 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.036 DEBUG:boto:Params: {} 2023-10-18T20:30:19.037 DEBUG:boto:Token: None 2023-10-18T20:30:19.038 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TUBF/N+IkTOscL6rgCIXbritUdc= 2023-10-18T20:30:19.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TUBF/N+IkTOscL6rgCIXbritUdc=', 'Content-Length': '0'} 2023-10-18T20:30:19.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a88988df18e1618-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.082 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.083 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.084 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.084 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.085 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.085 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.086 DEBUG:boto:Data: 2023-10-18T20:30:19.086 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.087 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.087 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.088 DEBUG:boto:Params: {} 2023-10-18T20:30:19.088 DEBUG:boto:Token: None 2023-10-18T20:30:19.089 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CiTKIUc7X6tTj2bWsAY1cH7qBEE= 2023-10-18T20:30:19.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CiTKIUc7X6tTj2bWsAY1cH7qBEE=', 'Content-Length': '0'} 2023-10-18T20:30:19.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000690d1f3394c41362-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.092 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.093 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.093 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.093 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.094 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.094 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.095 DEBUG:boto:Data: 2023-10-18T20:30:19.095 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.095 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.096 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.096 DEBUG:boto:Params: {} 2023-10-18T20:30:19.097 DEBUG:boto:Token: None 2023-10-18T20:30:19.097 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.098 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CiTKIUc7X6tTj2bWsAY1cH7qBEE= 2023-10-18T20:30:19.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CiTKIUc7X6tTj2bWsAY1cH7qBEE=', 'Content-Length': '0'} 2023-10-18T20:30:19.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d1c9c090e1e56c2-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-18T20:30:19.103 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.104 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.104 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.105 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.106 DEBUG:boto:Method: GET 2023-10-18T20:30:19.106 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.107 DEBUG:boto:Data: 2023-10-18T20:30:19.107 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.108 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.108 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.109 DEBUG:boto:Params: {} 2023-10-18T20:30:19.109 DEBUG:boto:Token: None 2023-10-18T20:30:19.110 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.111 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TUBF/N+IkTOscL6rgCIXbritUdc= 2023-10-18T20:30:19.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TUBF/N+IkTOscL6rgCIXbritUdc=', 'Content-Length': '0'} 2023-10-18T20:30:19.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046aa00cfeb8ffb34-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.114 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.114 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.115 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.115 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.116 DEBUG:boto:Method: GET 2023-10-18T20:30:19.116 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.117 DEBUG:boto:Data: 2023-10-18T20:30:19.117 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.117 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.118 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.118 DEBUG:boto:Params: {} 2023-10-18T20:30:19.119 DEBUG:boto:Token: None 2023-10-18T20:30:19.119 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:19.120 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TUBF/N+IkTOscL6rgCIXbritUdc= 2023-10-18T20:30:19.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TUBF/N+IkTOscL6rgCIXbritUdc=', 'Content-Length': '0'} 2023-10-18T20:30:19.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c1e45e17e82e5c3-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.123 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.123 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.124 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.125 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.125 DEBUG:boto:Method: GET 2023-10-18T20:30:19.126 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.127 DEBUG:boto:Data: 2023-10-18T20:30:19.128 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.128 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.129 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.130 DEBUG:boto:Params: {} 2023-10-18T20:30:19.131 DEBUG:boto:Token: None 2023-10-18T20:30:19.132 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006204d131cc727997-006530405b-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.137 DEBUG:boto:b'bofrnf-58obj000000000000000000101000falseobj000000000000000000102023-10-18T20:30:07.193Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.139 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.139 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.140 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.141 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.142 DEBUG:boto:Method: GET 2023-10-18T20:30:19.143 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.143 DEBUG:boto:Data: 2023-10-18T20:30:19.144 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.145 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.146 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.147 DEBUG:boto:Params: {} 2023-10-18T20:30:19.148 DEBUG:boto:Token: None 2023-10-18T20:30:19.149 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.150 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065eee248b4da3341-006530405b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.154 DEBUG:boto:b'bofrnf-58obj000000000000000000101000falseobj000000000000000000102023-10-18T20:30:07.193Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.155 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.156 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.157 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.158 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.158 DEBUG:boto:Data: 2023-10-18T20:30:19.159 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.160 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.161 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.161 DEBUG:boto:Params: {} 2023-10-18T20:30:19.162 DEBUG:boto:Token: None 2023-10-18T20:30:19.163 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pN+hzQ9k43y0QsTgEbsdtQDnMtc= 2023-10-18T20:30:19.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pN+hzQ9k43y0QsTgEbsdtQDnMtc=', 'Content-Length': '0'} 2023-10-18T20:30:19.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7dd228e2e9d606b-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.167 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.167 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.168 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.168 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.168 DEBUG:boto:Data: 2023-10-18T20:30:19.169 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.170 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.170 DEBUG:boto:Params: {} 2023-10-18T20:30:19.170 DEBUG:boto:Token: None 2023-10-18T20:30:19.171 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pN+hzQ9k43y0QsTgEbsdtQDnMtc= 2023-10-18T20:30:19.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pN+hzQ9k43y0QsTgEbsdtQDnMtc=', 'Content-Length': '0'} 2023-10-18T20:30:19.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009bf8d245698840d-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-18T20:30:19.176 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.176 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.177 DEBUG:boto:Method: GET 2023-10-18T20:30:19.178 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.179 DEBUG:boto:Data: 2023-10-18T20:30:19.180 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.180 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.181 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.182 DEBUG:boto:Params: {} 2023-10-18T20:30:19.182 DEBUG:boto:Token: None 2023-10-18T20:30:19.183 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bLcNKbKf8p2Yl8KV6vas9wO58yw= 2023-10-18T20:30:19.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bLcNKbKf8p2Yl8KV6vas9wO58yw=', 'Content-Length': '0'} 2023-10-18T20:30:19.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd725d7348e4f3fe-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.186 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.187 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.187 DEBUG:boto:Method: GET 2023-10-18T20:30:19.188 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.188 DEBUG:boto:Data: 2023-10-18T20:30:19.188 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.189 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.189 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.190 DEBUG:boto:Params: {} 2023-10-18T20:30:19.190 DEBUG:boto:Token: None 2023-10-18T20:30:19.191 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:19.191 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bLcNKbKf8p2Yl8KV6vas9wO58yw= 2023-10-18T20:30:19.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bLcNKbKf8p2Yl8KV6vas9wO58yw=', 'Content-Length': '0'} 2023-10-18T20:30:19.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003374a2aa0ef4bb37-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.195 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.195 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.196 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.196 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.196 DEBUG:boto:Method: GET 2023-10-18T20:30:19.197 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.198 DEBUG:boto:Data: 2023-10-18T20:30:19.198 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.199 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.200 DEBUG:boto:Params: {} 2023-10-18T20:30:19.200 DEBUG:boto:Token: None 2023-10-18T20:30:19.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f9c5eb9a7acf1d7-006530405b-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.205 DEBUG:boto:b'bofrnf-58obj000000000000000000101000falseobj000000000000000000102023-10-18T20:30:07.193Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.206 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.206 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.206 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.207 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.208 DEBUG:boto:Method: GET 2023-10-18T20:30:19.209 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:19.209 DEBUG:boto:Data: 2023-10-18T20:30:19.210 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.211 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.212 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.213 DEBUG:boto:Params: {} 2023-10-18T20:30:19.213 DEBUG:boto:Token: None 2023-10-18T20:30:19.214 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003431ab1300e6e553-006530405b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.262 DEBUG:boto:b'bofrnf-58obj000000000000000000101000falseobj000000000000000000102023-10-18T20:30:07.193Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-18T20:30:19.264 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.265 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.266 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.266 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.267 DEBUG:boto:Method: GET 2023-10-18T20:30:19.268 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.268 DEBUG:boto:Data: 2023-10-18T20:30:19.269 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.270 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.270 DEBUG:boto:Params: {} 2023-10-18T20:30:19.271 DEBUG:boto:Token: None 2023-10-18T20:30:19.271 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.272 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V/Ml6UydOQVleMT3EfeY0Wncr7g= 2023-10-18T20:30:19.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V/Ml6UydOQVleMT3EfeY0Wncr7g=', 'Content-Length': '0'} 2023-10-18T20:30:19.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e942fed635c8ed42-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.278 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.279 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.279 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.280 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.281 DEBUG:boto:Method: GET 2023-10-18T20:30:19.281 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.281 DEBUG:boto:Data: 2023-10-18T20:30:19.282 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.283 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.283 DEBUG:boto:Params: {} 2023-10-18T20:30:19.284 DEBUG:boto:Token: None 2023-10-18T20:30:19.284 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.285 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V/Ml6UydOQVleMT3EfeY0Wncr7g= 2023-10-18T20:30:19.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V/Ml6UydOQVleMT3EfeY0Wncr7g=', 'Content-Length': '0'} 2023-10-18T20:30:19.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5905b6eda870283-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.289 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.289 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.290 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.290 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.290 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.291 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.291 DEBUG:boto:Data: 2023-10-18T20:30:19.292 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.292 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.292 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.293 DEBUG:boto:Params: {} 2023-10-18T20:30:19.294 DEBUG:boto:Token: None 2023-10-18T20:30:19.294 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.295 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zQQcJwfNiRbCRCPeloMghwaDTpc= 2023-10-18T20:30:19.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zQQcJwfNiRbCRCPeloMghwaDTpc=', 'Content-Length': '0'} 2023-10-18T20:30:19.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000254c686d6186a00f-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.299 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.299 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.300 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.300 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.301 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.301 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.301 DEBUG:boto:Data: 2023-10-18T20:30:19.302 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.303 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.303 DEBUG:boto:Params: {} 2023-10-18T20:30:19.304 DEBUG:boto:Token: None 2023-10-18T20:30:19.304 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zQQcJwfNiRbCRCPeloMghwaDTpc= 2023-10-18T20:30:19.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zQQcJwfNiRbCRCPeloMghwaDTpc=', 'Content-Length': '0'} 2023-10-18T20:30:19.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1b7a0dd7668ee12-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-18T20:30:19.309 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.309 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.310 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.310 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.311 DEBUG:boto:Method: GET 2023-10-18T20:30:19.311 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.311 DEBUG:boto:Data: 2023-10-18T20:30:19.312 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.313 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.313 DEBUG:boto:Params: {} 2023-10-18T20:30:19.314 DEBUG:boto:Token: None 2023-10-18T20:30:19.314 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V/Ml6UydOQVleMT3EfeY0Wncr7g= 2023-10-18T20:30:19.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V/Ml6UydOQVleMT3EfeY0Wncr7g=', 'Content-Length': '0'} 2023-10-18T20:30:19.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093980df17918ff3e-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.318 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.318 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.319 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.319 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.320 DEBUG:boto:Method: GET 2023-10-18T20:30:19.320 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.320 DEBUG:boto:Data: 2023-10-18T20:30:19.321 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.321 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.322 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.322 DEBUG:boto:Params: {} 2023-10-18T20:30:19.323 DEBUG:boto:Token: None 2023-10-18T20:30:19.323 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:19.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V/Ml6UydOQVleMT3EfeY0Wncr7g= 2023-10-18T20:30:19.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V/Ml6UydOQVleMT3EfeY0Wncr7g=', 'Content-Length': '0'} 2023-10-18T20:30:19.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde7a605eeeddfb7-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.327 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.327 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.328 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.328 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.329 DEBUG:boto:Method: GET 2023-10-18T20:30:19.330 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.331 DEBUG:boto:Data: 2023-10-18T20:30:19.331 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.332 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.333 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.334 DEBUG:boto:Params: {} 2023-10-18T20:30:19.334 DEBUG:boto:Token: None 2023-10-18T20:30:19.335 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.336 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a218095b27d5068-006530405b-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.339 DEBUG:boto:b'bofrnf-58obj000000000000000000111000falseobj000000000000000000112023-10-18T20:30:07.311Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.340 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.340 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.341 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.341 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.342 DEBUG:boto:Method: GET 2023-10-18T20:30:19.342 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.343 DEBUG:boto:Data: 2023-10-18T20:30:19.343 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.344 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.344 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.345 DEBUG:boto:Params: {} 2023-10-18T20:30:19.345 DEBUG:boto:Token: None 2023-10-18T20:30:19.346 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.346 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b6ea4496c0d671f-006530405b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.350 DEBUG:boto:b'bofrnf-58obj000000000000000000111000falseobj000000000000000000112023-10-18T20:30:07.311Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.350 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.351 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.351 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.352 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.352 DEBUG:boto:Data: 2023-10-18T20:30:19.353 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.353 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.353 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.354 DEBUG:boto:Params: {} 2023-10-18T20:30:19.354 DEBUG:boto:Token: None 2023-10-18T20:30:19.355 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.355 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/ErAT6nAw7oJBDAmiRF64nX6mOg= 2023-10-18T20:30:19.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/ErAT6nAw7oJBDAmiRF64nX6mOg=', 'Content-Length': '0'} 2023-10-18T20:30:19.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da499166beb761e-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.358 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.359 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.359 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.360 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.360 DEBUG:boto:Data: 2023-10-18T20:30:19.360 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.361 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.361 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.362 DEBUG:boto:Params: {} 2023-10-18T20:30:19.362 DEBUG:boto:Token: None 2023-10-18T20:30:19.362 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.363 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/ErAT6nAw7oJBDAmiRF64nX6mOg= 2023-10-18T20:30:19.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/ErAT6nAw7oJBDAmiRF64nX6mOg=', 'Content-Length': '0'} 2023-10-18T20:30:19.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc792d457d34751a-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-18T20:30:19.366 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.367 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.367 DEBUG:boto:Method: GET 2023-10-18T20:30:19.368 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.368 DEBUG:boto:Data: 2023-10-18T20:30:19.369 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.369 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.369 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.370 DEBUG:boto:Params: {} 2023-10-18T20:30:19.370 DEBUG:boto:Token: None 2023-10-18T20:30:19.371 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.371 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P+0X/0GkmGtbhPZPl80xNdNVoXs= 2023-10-18T20:30:19.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P+0X/0GkmGtbhPZPl80xNdNVoXs=', 'Content-Length': '0'} 2023-10-18T20:30:19.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057f0bb929b9390a5-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.414 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.415 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.415 DEBUG:boto:Method: GET 2023-10-18T20:30:19.416 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.416 DEBUG:boto:Data: 2023-10-18T20:30:19.417 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.418 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.418 DEBUG:boto:Params: {} 2023-10-18T20:30:19.419 DEBUG:boto:Token: None 2023-10-18T20:30:19.419 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:19.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P+0X/0GkmGtbhPZPl80xNdNVoXs= 2023-10-18T20:30:19.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P+0X/0GkmGtbhPZPl80xNdNVoXs=', 'Content-Length': '0'} 2023-10-18T20:30:19.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005534e01f2ffa459a-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.423 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.423 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.424 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.424 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.425 DEBUG:boto:Method: GET 2023-10-18T20:30:19.425 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.425 DEBUG:boto:Data: 2023-10-18T20:30:19.426 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.427 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.427 DEBUG:boto:Params: {} 2023-10-18T20:30:19.428 DEBUG:boto:Token: None 2023-10-18T20:30:19.428 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.429 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c36a7aa7f9904f6-006530405b-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.433 DEBUG:boto:b'bofrnf-58obj000000000000000000111000falseobj000000000000000000112023-10-18T20:30:07.311Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.434 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.434 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.435 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.436 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.437 DEBUG:boto:Method: GET 2023-10-18T20:30:19.437 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:19.438 DEBUG:boto:Data: 2023-10-18T20:30:19.439 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.440 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.440 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.441 DEBUG:boto:Params: {} 2023-10-18T20:30:19.442 DEBUG:boto:Token: None 2023-10-18T20:30:19.443 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.443 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000894a4ba2b4305cd6-006530405b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.486 DEBUG:boto:b'bofrnf-58obj000000000000000000111000falseobj000000000000000000112023-10-18T20:30:07.311Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-18T20:30:19.488 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.488 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.489 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.489 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.490 DEBUG:boto:Method: GET 2023-10-18T20:30:19.490 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.491 DEBUG:boto:Data: 2023-10-18T20:30:19.491 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.492 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.492 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.492 DEBUG:boto:Params: {} 2023-10-18T20:30:19.493 DEBUG:boto:Token: None 2023-10-18T20:30:19.494 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.494 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fh7nhLl6j4NpYqHGWYqk5i8INpw= 2023-10-18T20:30:19.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fh7nhLl6j4NpYqHGWYqk5i8INpw=', 'Content-Length': '0'} 2023-10-18T20:30:19.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008719f6285679f5ae-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.498 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.498 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.499 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.499 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.500 DEBUG:boto:Method: GET 2023-10-18T20:30:19.501 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.501 DEBUG:boto:Data: 2023-10-18T20:30:19.502 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.504 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.504 DEBUG:boto:Params: {} 2023-10-18T20:30:19.505 DEBUG:boto:Token: None 2023-10-18T20:30:19.505 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.506 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fh7nhLl6j4NpYqHGWYqk5i8INpw= 2023-10-18T20:30:19.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fh7nhLl6j4NpYqHGWYqk5i8INpw=', 'Content-Length': '0'} 2023-10-18T20:30:19.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f9c265eaaed632-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.510 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.511 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.512 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.512 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.513 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.514 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.515 DEBUG:boto:Data: 2023-10-18T20:30:19.515 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.517 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.518 DEBUG:boto:Params: {} 2023-10-18T20:30:19.519 DEBUG:boto:Token: None 2023-10-18T20:30:19.519 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ujsx4FIJr+xjQRCpA5iu/m6yuhc= 2023-10-18T20:30:19.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ujsx4FIJr+xjQRCpA5iu/m6yuhc=', 'Content-Length': '0'} 2023-10-18T20:30:19.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c287a3e0a414986b-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.525 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.525 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.526 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.527 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.527 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.528 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.529 DEBUG:boto:Data: 2023-10-18T20:30:19.529 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.530 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.531 DEBUG:boto:Params: {} 2023-10-18T20:30:19.531 DEBUG:boto:Token: None 2023-10-18T20:30:19.532 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ujsx4FIJr+xjQRCpA5iu/m6yuhc= 2023-10-18T20:30:19.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ujsx4FIJr+xjQRCpA5iu/m6yuhc=', 'Content-Length': '0'} 2023-10-18T20:30:19.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000590a1a0ee8429743-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-18T20:30:19.536 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.536 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.537 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.537 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.538 DEBUG:boto:Method: GET 2023-10-18T20:30:19.538 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.539 DEBUG:boto:Data: 2023-10-18T20:30:19.539 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.541 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.542 DEBUG:boto:Params: {} 2023-10-18T20:30:19.543 DEBUG:boto:Token: None 2023-10-18T20:30:19.543 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fh7nhLl6j4NpYqHGWYqk5i8INpw= 2023-10-18T20:30:19.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fh7nhLl6j4NpYqHGWYqk5i8INpw=', 'Content-Length': '0'} 2023-10-18T20:30:19.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbf964569472a637-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.548 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.549 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.549 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.550 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.550 DEBUG:boto:Method: GET 2023-10-18T20:30:19.551 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.551 DEBUG:boto:Data: 2023-10-18T20:30:19.552 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.553 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.553 DEBUG:boto:Params: {} 2023-10-18T20:30:19.554 DEBUG:boto:Token: None 2023-10-18T20:30:19.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:19.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fh7nhLl6j4NpYqHGWYqk5i8INpw= 2023-10-18T20:30:19.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fh7nhLl6j4NpYqHGWYqk5i8INpw=', 'Content-Length': '0'} 2023-10-18T20:30:19.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aaa1720bf02fcba-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.559 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.559 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.560 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.560 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.560 DEBUG:boto:Method: GET 2023-10-18T20:30:19.561 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.561 DEBUG:boto:Data: 2023-10-18T20:30:19.562 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.563 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.563 DEBUG:boto:Params: {} 2023-10-18T20:30:19.564 DEBUG:boto:Token: None 2023-10-18T20:30:19.564 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.565 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006282bf61e1bbe3f8-006530405b-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.568 DEBUG:boto:b'bofrnf-58obj000000000000000000121000falseobj000000000000000000122023-10-18T20:30:07.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.569 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.570 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.570 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.571 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.571 DEBUG:boto:Method: GET 2023-10-18T20:30:19.572 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.572 DEBUG:boto:Data: 2023-10-18T20:30:19.573 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.573 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.574 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.574 DEBUG:boto:Params: {} 2023-10-18T20:30:19.575 DEBUG:boto:Token: None 2023-10-18T20:30:19.575 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d0d980ababba192-006530405b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.579 DEBUG:boto:b'bofrnf-58obj000000000000000000121000falseobj000000000000000000122023-10-18T20:30:07.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.580 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.580 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.581 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.581 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.582 DEBUG:boto:Data: 2023-10-18T20:30:19.582 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.583 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.583 DEBUG:boto:Params: {} 2023-10-18T20:30:19.584 DEBUG:boto:Token: None 2023-10-18T20:30:19.584 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eCiawY7NxDTowT4BflvqMW81MfE= 2023-10-18T20:30:19.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eCiawY7NxDTowT4BflvqMW81MfE=', 'Content-Length': '0'} 2023-10-18T20:30:19.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c76dddc2de56d17-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.587 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.588 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.588 DEBUG:boto:Method: HEAD 2023-10-18T20:30:19.589 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.589 DEBUG:boto:Data: 2023-10-18T20:30:19.590 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.590 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.591 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.591 DEBUG:boto:Params: {} 2023-10-18T20:30:19.592 DEBUG:boto:Token: None 2023-10-18T20:30:19.593 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.594 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eCiawY7NxDTowT4BflvqMW81MfE= 2023-10-18T20:30:19.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eCiawY7NxDTowT4BflvqMW81MfE=', 'Content-Length': '0'} 2023-10-18T20:30:19.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c185d1a76b1d467-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-18T20:30:19.598 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.599 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.599 DEBUG:boto:Method: GET 2023-10-18T20:30:19.600 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.600 DEBUG:boto:Data: 2023-10-18T20:30:19.601 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.601 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.602 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.603 DEBUG:boto:Params: {} 2023-10-18T20:30:19.604 DEBUG:boto:Token: None 2023-10-18T20:30:19.605 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OYVyLKvnEN13VcdW7bCoM3LZPtw= 2023-10-18T20:30:19.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OYVyLKvnEN13VcdW7bCoM3LZPtw=', 'Content-Length': '0'} 2023-10-18T20:30:19.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fcb1e11411b6079-006530405b-1a24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.610 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.611 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.611 DEBUG:boto:Method: GET 2023-10-18T20:30:19.612 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.613 DEBUG:boto:Data: 2023-10-18T20:30:19.614 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.614 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.615 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.615 DEBUG:boto:Params: {} 2023-10-18T20:30:19.616 DEBUG:boto:Token: None 2023-10-18T20:30:19.616 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:19.617 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OYVyLKvnEN13VcdW7bCoM3LZPtw= 2023-10-18T20:30:19.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OYVyLKvnEN13VcdW7bCoM3LZPtw=', 'Content-Length': '0'} 2023-10-18T20:30:19.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daaea749961aaa31-006530405b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.662 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.663 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.663 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.664 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.664 DEBUG:boto:Method: GET 2023-10-18T20:30:19.665 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.665 DEBUG:boto:Data: 2023-10-18T20:30:19.665 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.666 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:19.666 DEBUG:boto:Port: 8001 2023-10-18T20:30:19.667 DEBUG:boto:Params: {} 2023-10-18T20:30:19.667 DEBUG:boto:Token: None 2023-10-18T20:30:19.668 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000051a96a714c74828-006530405b-1a24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.672 DEBUG:boto:b'bofrnf-58obj000000000000000000121000falseobj000000000000000000122023-10-18T20:30:07.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.672 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:19.673 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:19.673 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.674 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.674 DEBUG:boto:Method: GET 2023-10-18T20:30:19.675 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:19.675 DEBUG:boto:Data: 2023-10-18T20:30:19.676 DEBUG:boto:Headers: {} 2023-10-18T20:30:19.676 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:19.677 DEBUG:boto:Port: 8000 2023-10-18T20:30:19.677 DEBUG:boto:Params: {} 2023-10-18T20:30:19.678 DEBUG:boto:Token: None 2023-10-18T20:30:19.678 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:19 GMT /bofrnf-58/ 2023-10-18T20:30:19.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ= 2023-10-18T20:30:19.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fENf4F4sSi/OqZoO1LEeN2hDjAQ=', 'Content-Length': '0'} 2023-10-18T20:30:19.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0e95047478931f2-006530405b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:19.683 DEBUG:boto:b'bofrnf-58obj000000000000000000121000falseobj000000000000000000122023-10-18T20:30:07.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:19.686 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-58 zones={a2, a1} 2023-10-18T20:30:19.688 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-18T20:30:19.688 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T20:30:19.690 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:30:19.687+0000 7fb095976700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 156477) UID: 0 2023-10-18T20:30:19.690 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:30:19.687+0000 7fb099bdf7c0 -1 shutting down 2023-10-18T20:30:25.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T20:30:25.784 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-18T20:30:25.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:25.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:25.786 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-58 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:30:25.846 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:25.846 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:25.890 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:30:25.891 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-58 2023-10-18T20:30:25.894 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 10 2023-10-18T20:30:25.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:25.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:25.931 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:25.984 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:25.984 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:26.024 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:26.024 INFO:teuthology.orchestra.run.smithi082.stdout: "objs": [ 2023-10-18T20:30:26.025 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000007", 2023-10-18T20:30:26.025 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000016", 2023-10-18T20:30:26.025 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000024", 2023-10-18T20:30:26.025 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000020", 2023-10-18T20:30:26.025 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000028", 2023-10-18T20:30:26.026 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000000", 2023-10-18T20:30:26.026 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000032", 2023-10-18T20:30:26.029 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000001", 2023-10-18T20:30:26.029 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000010", 2023-10-18T20:30:26.030 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000019", 2023-10-18T20:30:26.030 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000013", 2023-10-18T20:30:26.030 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000021", 2023-10-18T20:30:26.030 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000030", 2023-10-18T20:30:26.031 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000026", 2023-10-18T20:30:26.031 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000033", 2023-10-18T20:30:26.031 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000004", 2023-10-18T20:30:26.032 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000076" 2023-10-18T20:30:26.032 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:26.032 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:26.033 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.034 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.034 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.035 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.035 DEBUG:boto:Data: 2023-10-18T20:30:26.036 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.036 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.037 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.037 DEBUG:boto:Params: {} 2023-10-18T20:30:26.038 DEBUG:boto:Token: None 2023-10-18T20:30:26.038 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a454eb06f4e634c-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.043 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:26.043 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:26.044 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.044 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:26.045 DEBUG:boto:Data: 2023-10-18T20:30:26.045 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.046 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.047 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.047 DEBUG:boto:Params: {} 2023-10-18T20:30:26.048 DEBUG:boto:Token: None 2023-10-18T20:30:26.048 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:26.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dLpl54XyEZEjNdnS6sSXwdaFQ7E= 2023-10-18T20:30:26.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dLpl54XyEZEjNdnS6sSXwdaFQ7E='} 2023-10-18T20:30:26.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054689257691e70f8-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.055 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.055 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.056 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.056 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.056 DEBUG:boto:Data: 2023-10-18T20:30:26.057 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.057 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.058 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.058 DEBUG:boto:Params: {} 2023-10-18T20:30:26.059 DEBUG:boto:Token: None 2023-10-18T20:30:26.059 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.060 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005440f2e417123eec-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.064 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:26.064 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:26.065 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.065 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:26.066 DEBUG:boto:Data: 2023-10-18T20:30:26.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.067 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.068 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.068 DEBUG:boto:Params: {} 2023-10-18T20:30:26.069 DEBUG:boto:Token: None 2023-10-18T20:30:26.070 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:26.070 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7PuAdyVhfIri7eMmBSU4aJkda2M= 2023-10-18T20:30:26.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7PuAdyVhfIri7eMmBSU4aJkda2M='} 2023-10-18T20:30:26.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a2e7104b5434b7c-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.119 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.120 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.120 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.121 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.122 DEBUG:boto:Data: 2023-10-18T20:30:26.125 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.126 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.127 DEBUG:boto:Params: {} 2023-10-18T20:30:26.128 DEBUG:boto:Token: None 2023-10-18T20:30:26.128 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.129 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f713dc9c15a568aa-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.133 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:26.134 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:26.134 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.135 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:26.135 DEBUG:boto:Data: 2023-10-18T20:30:26.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.136 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.136 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.137 DEBUG:boto:Params: {} 2023-10-18T20:30:26.138 DEBUG:boto:Token: None 2023-10-18T20:30:26.138 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:26.139 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bwyyg6giRgQXZZZps+IlZaFpRoM= 2023-10-18T20:30:26.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bwyyg6giRgQXZZZps+IlZaFpRoM='} 2023-10-18T20:30:26.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f97c9dee62da847c-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.145 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.145 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.146 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.146 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.146 DEBUG:boto:Data: 2023-10-18T20:30:26.147 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.147 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.148 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.148 DEBUG:boto:Params: {} 2023-10-18T20:30:26.149 DEBUG:boto:Token: None 2023-10-18T20:30:26.149 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.149 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d75f96002b7ee45-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.153 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:26.154 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:26.154 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.155 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:26.155 DEBUG:boto:Data: 2023-10-18T20:30:26.156 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.156 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.156 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.157 DEBUG:boto:Params: {} 2023-10-18T20:30:26.157 DEBUG:boto:Token: None 2023-10-18T20:30:26.158 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:26.158 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ceNnju9ZYlBxL2h0w8+Tj8uDjLo= 2023-10-18T20:30:26.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ceNnju9ZYlBxL2h0w8+Tj8uDjLo='} 2023-10-18T20:30:26.165 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d98464419f9630af-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.165 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.166 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.166 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.167 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.167 DEBUG:boto:Data: 2023-10-18T20:30:26.167 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.168 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.168 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.169 DEBUG:boto:Params: {} 2023-10-18T20:30:26.169 DEBUG:boto:Token: None 2023-10-18T20:30:26.170 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.170 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.173 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e94ee084cdb4753f-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.174 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:26.174 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:26.175 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.175 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:26.175 DEBUG:boto:Data: 2023-10-18T20:30:26.176 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.176 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.177 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.177 DEBUG:boto:Params: {} 2023-10-18T20:30:26.177 DEBUG:boto:Token: None 2023-10-18T20:30:26.178 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:26.178 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kiEjjH7OhvaGp0iBwG7ZwP7jthQ= 2023-10-18T20:30:26.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kiEjjH7OhvaGp0iBwG7ZwP7jthQ='} 2023-10-18T20:30:26.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ffd2338a83269e1-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.227 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.228 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.228 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.228 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.229 DEBUG:boto:Data: 2023-10-18T20:30:26.229 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.230 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.233 DEBUG:boto:Params: {} 2023-10-18T20:30:26.233 DEBUG:boto:Token: None 2023-10-18T20:30:26.234 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.234 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.237 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000981ec19135f3d023-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.238 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:26.239 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:26.239 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.240 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:26.240 DEBUG:boto:Data: 2023-10-18T20:30:26.240 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.241 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.241 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.242 DEBUG:boto:Params: {} 2023-10-18T20:30:26.242 DEBUG:boto:Token: None 2023-10-18T20:30:26.243 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:26.243 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YvODfG9dXGG6+FHsoP41YVC4fTc= 2023-10-18T20:30:26.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YvODfG9dXGG6+FHsoP41YVC4fTc='} 2023-10-18T20:30:26.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8490696237db539-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.249 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.250 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.250 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.251 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.251 DEBUG:boto:Data: 2023-10-18T20:30:26.252 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.252 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.252 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.253 DEBUG:boto:Params: {} 2023-10-18T20:30:26.253 DEBUG:boto:Token: None 2023-10-18T20:30:26.254 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.254 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007df85adb804a5f67-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.258 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:26.258 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:26.259 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.259 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:26.260 DEBUG:boto:Data: 2023-10-18T20:30:26.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.260 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.261 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.261 DEBUG:boto:Params: {} 2023-10-18T20:30:26.262 DEBUG:boto:Token: None 2023-10-18T20:30:26.262 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:26.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:04US1e11BmsAXkR8U230sAsFwYk= 2023-10-18T20:30:26.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:04US1e11BmsAXkR8U230sAsFwYk='} 2023-10-18T20:30:26.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c28a524e841d4e3a-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.269 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.269 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.270 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.270 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.271 DEBUG:boto:Data: 2023-10-18T20:30:26.271 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.271 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.272 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.272 DEBUG:boto:Params: {} 2023-10-18T20:30:26.273 DEBUG:boto:Token: None 2023-10-18T20:30:26.273 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.274 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.277 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b15d8de3a4a93161-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.278 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:26.278 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:26.278 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.279 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:26.279 DEBUG:boto:Data: 2023-10-18T20:30:26.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.280 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.280 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.281 DEBUG:boto:Params: {} 2023-10-18T20:30:26.281 DEBUG:boto:Token: None 2023-10-18T20:30:26.282 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:26.282 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lEOm/mHD/W0Lh7knP5nZIuQ2MjE= 2023-10-18T20:30:26.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lEOm/mHD/W0Lh7knP5nZIuQ2MjE='} 2023-10-18T20:30:26.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000996fb960862c6d8-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.331 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.332 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.332 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.333 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.333 DEBUG:boto:Data: 2023-10-18T20:30:26.334 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.334 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.334 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.335 DEBUG:boto:Params: {} 2023-10-18T20:30:26.335 DEBUG:boto:Token: None 2023-10-18T20:30:26.336 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.336 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.339 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdad46de7f5d2790-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.340 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:26.341 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:26.341 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.341 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:26.342 DEBUG:boto:Data: 2023-10-18T20:30:26.342 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.343 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.343 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.344 DEBUG:boto:Params: {} 2023-10-18T20:30:26.344 DEBUG:boto:Token: None 2023-10-18T20:30:26.344 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:26.345 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qqCCLyDAnmBc2T2yl58/BOtNziU= 2023-10-18T20:30:26.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qqCCLyDAnmBc2T2yl58/BOtNziU='} 2023-10-18T20:30:26.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000830fdb0dffafce06-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.351 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.351 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.352 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.352 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.352 DEBUG:boto:Data: 2023-10-18T20:30:26.353 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.353 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.354 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.354 DEBUG:boto:Params: {} 2023-10-18T20:30:26.355 DEBUG:boto:Token: None 2023-10-18T20:30:26.355 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b982bc34061bce81-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.359 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:26.360 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:26.360 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.360 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:26.361 DEBUG:boto:Data: 2023-10-18T20:30:26.361 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.362 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.362 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.363 DEBUG:boto:Params: {} 2023-10-18T20:30:26.363 DEBUG:boto:Token: None 2023-10-18T20:30:26.364 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:26.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z/Unv052FSJOGmO3R2PoDgppYv4= 2023-10-18T20:30:26.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z/Unv052FSJOGmO3R2PoDgppYv4='} 2023-10-18T20:30:26.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d530721c6a68fce7-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.408 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.408 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.408 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.409 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.409 DEBUG:boto:Data: 2023-10-18T20:30:26.410 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.410 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.411 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.411 DEBUG:boto:Params: {} 2023-10-18T20:30:26.412 DEBUG:boto:Token: None 2023-10-18T20:30:26.412 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.413 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d22dbfe1aa6d43c2-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.417 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:26.417 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:26.418 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.418 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:26.419 DEBUG:boto:Data: 2023-10-18T20:30:26.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.420 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.420 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.421 DEBUG:boto:Params: {} 2023-10-18T20:30:26.421 DEBUG:boto:Token: None 2023-10-18T20:30:26.422 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:26.422 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2xIZ4hIvb5+LlAuXx1JdNoJEuik= 2023-10-18T20:30:26.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2xIZ4hIvb5+LlAuXx1JdNoJEuik='} 2023-10-18T20:30:26.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061cd753de86408d6-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.429 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.430 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.430 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.431 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.431 DEBUG:boto:Data: 2023-10-18T20:30:26.431 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.432 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.433 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.433 DEBUG:boto:Params: {} 2023-10-18T20:30:26.434 DEBUG:boto:Token: None 2023-10-18T20:30:26.434 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.435 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005edea274597dfb15-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.438 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:26.439 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:26.440 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.441 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:26.441 DEBUG:boto:Data: 2023-10-18T20:30:26.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.443 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.444 DEBUG:boto:Params: {} 2023-10-18T20:30:26.444 DEBUG:boto:Token: None 2023-10-18T20:30:26.445 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:26.445 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8k+asVTOGBuAMuT4iMXeITHI8Bk= 2023-10-18T20:30:26.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8k+asVTOGBuAMuT4iMXeITHI8Bk='} 2023-10-18T20:30:26.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000178eddfceae6443-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.453 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.454 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.454 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.455 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.455 DEBUG:boto:Data: 2023-10-18T20:30:26.456 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.456 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.456 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.457 DEBUG:boto:Params: {} 2023-10-18T20:30:26.457 DEBUG:boto:Token: None 2023-10-18T20:30:26.458 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.458 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065b743c0aea6c7b8-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.462 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:26.463 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:26.463 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.463 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:26.464 DEBUG:boto:Data: 2023-10-18T20:30:26.464 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.465 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.465 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.466 DEBUG:boto:Params: {} 2023-10-18T20:30:26.466 DEBUG:boto:Token: None 2023-10-18T20:30:26.467 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:26.467 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/ladI3sMcKhlvEFSEGPWfHZ8+q0= 2023-10-18T20:30:26.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/ladI3sMcKhlvEFSEGPWfHZ8+q0='} 2023-10-18T20:30:26.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e8d68b5d17619a8-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.512 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.512 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.512 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.513 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.513 DEBUG:boto:Data: 2023-10-18T20:30:26.514 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.514 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.514 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.515 DEBUG:boto:Params: {} 2023-10-18T20:30:26.516 DEBUG:boto:Token: None 2023-10-18T20:30:26.516 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.517 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.521 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ae2304a241882ac-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.522 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:26.523 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:26.524 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.524 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:26.525 DEBUG:boto:Data: 2023-10-18T20:30:26.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.526 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.526 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.527 DEBUG:boto:Params: {} 2023-10-18T20:30:26.527 DEBUG:boto:Token: None 2023-10-18T20:30:26.528 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:26.528 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UVgL+ynl3VgbWJnkVkYCKT1uem0= 2023-10-18T20:30:26.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UVgL+ynl3VgbWJnkVkYCKT1uem0='} 2023-10-18T20:30:26.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007245ce7ec0da6b2b-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.536 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.537 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.537 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.538 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.538 DEBUG:boto:Data: 2023-10-18T20:30:26.539 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.539 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.539 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.540 DEBUG:boto:Params: {} 2023-10-18T20:30:26.541 DEBUG:boto:Token: None 2023-10-18T20:30:26.541 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.542 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd0503233411e86e-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.546 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:26.546 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:26.547 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.547 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:26.548 DEBUG:boto:Data: 2023-10-18T20:30:26.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.548 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.549 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.549 DEBUG:boto:Params: {} 2023-10-18T20:30:26.550 DEBUG:boto:Token: None 2023-10-18T20:30:26.550 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:26.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1ACrJ2UEw1MxPYetQWK6QhorQ+I= 2023-10-18T20:30:26.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1ACrJ2UEw1MxPYetQWK6QhorQ+I='} 2023-10-18T20:30:26.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005baefece0cfa762f-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.558 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.558 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.559 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.559 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.560 DEBUG:boto:Data: 2023-10-18T20:30:26.560 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.560 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.561 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.561 DEBUG:boto:Params: {} 2023-10-18T20:30:26.562 DEBUG:boto:Token: None 2023-10-18T20:30:26.562 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.563 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bedfe48fbec930d-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.566 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:26.567 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:26.567 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.568 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:26.568 DEBUG:boto:Data: 2023-10-18T20:30:26.569 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.569 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.569 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.570 DEBUG:boto:Params: {} 2023-10-18T20:30:26.570 DEBUG:boto:Token: None 2023-10-18T20:30:26.571 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:26.571 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hs/ovbGJ71DbQ03REqTFeVvOcdg= 2023-10-18T20:30:26.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hs/ovbGJ71DbQ03REqTFeVvOcdg='} 2023-10-18T20:30:26.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000228877a8d0ae76ec-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:26.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:26.616 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-58 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:30:26.669 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:26.670 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:26.711 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:30:26.711 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-58 2023-10-18T20:30:26.717 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 22 2023-10-18T20:30:26.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:26.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:26.760 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:26.811 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:26.812 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:26.851 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:26.851 INFO:teuthology.orchestra.run.smithi082.stdout: "objs": [ 2023-10-18T20:30:26.851 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000005", 2023-10-18T20:30:26.852 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000010", 2023-10-18T20:30:26.852 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000016", 2023-10-18T20:30:26.852 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000021", 2023-10-18T20:30:26.852 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000004", 2023-10-18T20:30:26.852 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000025", 2023-10-18T20:30:26.853 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000030", 2023-10-18T20:30:26.853 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000015", 2023-10-18T20:30:26.853 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000019", 2023-10-18T20:30:26.853 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000003", 2023-10-18T20:30:26.853 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000009", 2023-10-18T20:30:26.854 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000014", 2023-10-18T20:30:26.854 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000012", 2023-10-18T20:30:26.854 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000002", 2023-10-18T20:30:26.854 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000000", 2023-10-18T20:30:26.855 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000006", 2023-10-18T20:30:26.855 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000011", 2023-10-18T20:30:26.855 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000001", 2023-10-18T20:30:26.855 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000007" 2023-10-18T20:30:26.856 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:26.856 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:26.857 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.857 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.858 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.858 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.859 DEBUG:boto:Data: 2023-10-18T20:30:26.859 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.860 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.860 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.860 DEBUG:boto:Params: {} 2023-10-18T20:30:26.861 DEBUG:boto:Token: None 2023-10-18T20:30:26.861 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b29cb691d2bc409a-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.866 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:26.866 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:26.867 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.867 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:26.867 DEBUG:boto:Data: 2023-10-18T20:30:26.868 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.869 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.869 DEBUG:boto:Params: {} 2023-10-18T20:30:26.869 DEBUG:boto:Token: None 2023-10-18T20:30:26.870 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:26.870 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VOncAwVK/xwUt0Xsu9lN8lMfquw= 2023-10-18T20:30:26.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VOncAwVK/xwUt0Xsu9lN8lMfquw='} 2023-10-18T20:30:26.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d761fdff675097c5-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.876 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.877 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.877 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.877 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.878 DEBUG:boto:Data: 2023-10-18T20:30:26.878 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.879 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.880 DEBUG:boto:Params: {} 2023-10-18T20:30:26.880 DEBUG:boto:Token: None 2023-10-18T20:30:26.880 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.881 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad34969d7a48e0e1-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.884 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:26.885 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:26.885 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.886 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:26.886 DEBUG:boto:Data: 2023-10-18T20:30:26.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.887 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.887 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.888 DEBUG:boto:Params: {} 2023-10-18T20:30:26.888 DEBUG:boto:Token: None 2023-10-18T20:30:26.889 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:26.889 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ohuOITmBgnnkkbjGnBuYLI2OyeY= 2023-10-18T20:30:26.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ohuOITmBgnnkkbjGnBuYLI2OyeY='} 2023-10-18T20:30:26.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000256f4a8a4550d395-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.895 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.895 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.895 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.896 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.896 DEBUG:boto:Data: 2023-10-18T20:30:26.897 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.897 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.897 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.898 DEBUG:boto:Params: {} 2023-10-18T20:30:26.898 DEBUG:boto:Token: None 2023-10-18T20:30:26.899 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.899 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087818090703b71ce-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.903 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:26.903 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:26.904 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.904 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:26.905 DEBUG:boto:Data: 2023-10-18T20:30:26.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.905 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.906 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.906 DEBUG:boto:Params: {} 2023-10-18T20:30:26.907 DEBUG:boto:Token: None 2023-10-18T20:30:26.907 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:26.908 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OLy5YBMGIJ12VGqE0AoIxBGuMLc= 2023-10-18T20:30:26.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OLy5YBMGIJ12VGqE0AoIxBGuMLc='} 2023-10-18T20:30:26.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5d4d80d7a192ab2-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.914 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.914 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.914 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.915 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.915 DEBUG:boto:Data: 2023-10-18T20:30:26.916 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.916 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.916 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.917 DEBUG:boto:Params: {} 2023-10-18T20:30:26.917 DEBUG:boto:Token: None 2023-10-18T20:30:26.918 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.918 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027470f82377aad29-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.922 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:26.922 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:26.923 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.923 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:26.923 DEBUG:boto:Data: 2023-10-18T20:30:26.924 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.924 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.925 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.925 DEBUG:boto:Params: {} 2023-10-18T20:30:26.925 DEBUG:boto:Token: None 2023-10-18T20:30:26.926 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:26.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wwgk+1RfPdqt3mqVCaZa1w3Bwtc= 2023-10-18T20:30:26.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wwgk+1RfPdqt3mqVCaZa1w3Bwtc='} 2023-10-18T20:30:26.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9771ce6703c0fbd-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.971 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.972 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.972 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.972 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.973 DEBUG:boto:Data: 2023-10-18T20:30:26.973 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.974 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.974 DEBUG:boto:Params: {} 2023-10-18T20:30:26.975 DEBUG:boto:Token: None 2023-10-18T20:30:26.975 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.976 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.979 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c758642fdaecfdc-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.979 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:26.980 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:26.980 DEBUG:boto:Method: PUT 2023-10-18T20:30:26.981 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:26.981 DEBUG:boto:Data: 2023-10-18T20:30:26.981 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:26.982 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.982 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.983 DEBUG:boto:Params: {} 2023-10-18T20:30:26.983 DEBUG:boto:Token: None 2023-10-18T20:30:26.984 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:26.984 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WbbK003SaKr1K7r5v7TOqfSxSbY= 2023-10-18T20:30:26.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WbbK003SaKr1K7r5v7TOqfSxSbY='} 2023-10-18T20:30:26.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000672e82aa20507538-0065304062-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:26.991 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:26.992 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:26.992 DEBUG:boto:Method: HEAD 2023-10-18T20:30:26.993 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:26.993 DEBUG:boto:Data: 2023-10-18T20:30:26.994 DEBUG:boto:Headers: {} 2023-10-18T20:30:26.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:26.994 DEBUG:boto:Port: 8000 2023-10-18T20:30:26.995 DEBUG:boto:Params: {} 2023-10-18T20:30:26.995 DEBUG:boto:Token: None 2023-10-18T20:30:26.996 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:26 GMT /bofrnf-58/ 2023-10-18T20:30:26.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI= 2023-10-18T20:30:26.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fHMTKw/WixliuEuQE7UrQs+E+CI=', 'Content-Length': '0'} 2023-10-18T20:30:26.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4e18464aff3c184-0065304062-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.000 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:27.000 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:27.001 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.001 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:27.002 DEBUG:boto:Data: 2023-10-18T20:30:27.002 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.003 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.003 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.004 DEBUG:boto:Params: {} 2023-10-18T20:30:27.004 DEBUG:boto:Token: None 2023-10-18T20:30:27.005 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:27.005 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KenCc4M+3FqP0QF0QJSG3JpVLlg= 2023-10-18T20:30:27.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KenCc4M+3FqP0QF0QJSG3JpVLlg='} 2023-10-18T20:30:27.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef0c1d02cc5cb334-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.055 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.056 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.056 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.056 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.057 DEBUG:boto:Data: 2023-10-18T20:30:27.057 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.058 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.058 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.059 DEBUG:boto:Params: {} 2023-10-18T20:30:27.059 DEBUG:boto:Token: None 2023-10-18T20:30:27.060 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.060 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4e37ac6f198bd61-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.065 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:27.065 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:27.066 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.066 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:27.067 DEBUG:boto:Data: 2023-10-18T20:30:27.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.067 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.068 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.068 DEBUG:boto:Params: {} 2023-10-18T20:30:27.069 DEBUG:boto:Token: None 2023-10-18T20:30:27.070 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:27.070 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UTQgvawSP8PGlWvRjTtL+vj4z0I= 2023-10-18T20:30:27.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UTQgvawSP8PGlWvRjTtL+vj4z0I='} 2023-10-18T20:30:27.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000850580584448c44-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.079 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.079 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.080 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.080 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.081 DEBUG:boto:Data: 2023-10-18T20:30:27.081 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.082 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.083 DEBUG:boto:Params: {} 2023-10-18T20:30:27.083 DEBUG:boto:Token: None 2023-10-18T20:30:27.083 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.084 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.087 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7a0fb88b5433e45-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.088 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:27.088 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:27.089 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.089 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:27.090 DEBUG:boto:Data: 2023-10-18T20:30:27.090 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.090 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.091 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.092 DEBUG:boto:Params: {} 2023-10-18T20:30:27.092 DEBUG:boto:Token: None 2023-10-18T20:30:27.093 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:27.094 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zNm4lgDVfvq/41QPles9EnsQ5OA= 2023-10-18T20:30:27.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zNm4lgDVfvq/41QPles9EnsQ5OA='} 2023-10-18T20:30:27.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b71cb0001c5da903-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.139 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.140 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.140 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.140 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.141 DEBUG:boto:Data: 2023-10-18T20:30:27.141 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.142 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.144 DEBUG:boto:Params: {} 2023-10-18T20:30:27.144 DEBUG:boto:Token: None 2023-10-18T20:30:27.145 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.145 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000634620f99528efe-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.151 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:27.151 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:27.152 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.152 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:27.153 DEBUG:boto:Data: 2023-10-18T20:30:27.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.154 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.154 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.155 DEBUG:boto:Params: {} 2023-10-18T20:30:27.155 DEBUG:boto:Token: None 2023-10-18T20:30:27.155 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:27.156 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1fL/TJGGUb1yLFfUmS2dN8y1u08= 2023-10-18T20:30:27.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1fL/TJGGUb1yLFfUmS2dN8y1u08='} 2023-10-18T20:30:27.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1542856f2fb434b-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.162 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.162 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.163 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.163 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.164 DEBUG:boto:Data: 2023-10-18T20:30:27.164 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.164 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.165 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.165 DEBUG:boto:Params: {} 2023-10-18T20:30:27.166 DEBUG:boto:Token: None 2023-10-18T20:30:27.166 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.167 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082960e6d08c443ff-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.170 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:27.171 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:27.171 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.172 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:27.172 DEBUG:boto:Data: 2023-10-18T20:30:27.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.173 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.173 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.174 DEBUG:boto:Params: {} 2023-10-18T20:30:27.174 DEBUG:boto:Token: None 2023-10-18T20:30:27.175 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:27.175 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7Rpdjluv2ovXiVirBQTQ18XrQRg= 2023-10-18T20:30:27.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7Rpdjluv2ovXiVirBQTQ18XrQRg='} 2023-10-18T20:30:27.181 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004895f963b9f9524f-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.182 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.183 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.183 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.183 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.184 DEBUG:boto:Data: 2023-10-18T20:30:27.184 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.184 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.185 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.185 DEBUG:boto:Params: {} 2023-10-18T20:30:27.186 DEBUG:boto:Token: None 2023-10-18T20:30:27.186 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.187 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.190 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007642d6b063302211-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.190 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:27.191 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:27.191 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.191 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:27.192 DEBUG:boto:Data: 2023-10-18T20:30:27.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.193 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.193 DEBUG:boto:Params: {} 2023-10-18T20:30:27.194 DEBUG:boto:Token: None 2023-10-18T20:30:27.194 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:27.195 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3K2pvNtF3EvBe3o1xW+c+MpHXq4= 2023-10-18T20:30:27.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3K2pvNtF3EvBe3o1xW+c+MpHXq4='} 2023-10-18T20:30:27.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009718f74973c47e46-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.239 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.240 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.240 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.240 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.241 DEBUG:boto:Data: 2023-10-18T20:30:27.241 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.242 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.242 DEBUG:boto:Params: {} 2023-10-18T20:30:27.243 DEBUG:boto:Token: None 2023-10-18T20:30:27.243 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcb9d30ce4ba1ba7-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.248 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:27.248 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:27.249 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.249 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:27.249 DEBUG:boto:Data: 2023-10-18T20:30:27.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.250 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.251 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.252 DEBUG:boto:Params: {} 2023-10-18T20:30:27.252 DEBUG:boto:Token: None 2023-10-18T20:30:27.252 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:27.253 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wtpQGvQrO2zZ4e5mvfSmS1fiF+o= 2023-10-18T20:30:27.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wtpQGvQrO2zZ4e5mvfSmS1fiF+o='} 2023-10-18T20:30:27.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000080a2ac186d1aa54-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.260 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.261 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.261 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.262 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.262 DEBUG:boto:Data: 2023-10-18T20:30:27.263 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.263 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.263 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.264 DEBUG:boto:Params: {} 2023-10-18T20:30:27.264 DEBUG:boto:Token: None 2023-10-18T20:30:27.265 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.265 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000552ab07176a27343-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.269 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:27.269 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:27.270 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.270 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:27.271 DEBUG:boto:Data: 2023-10-18T20:30:27.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.271 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.272 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.272 DEBUG:boto:Params: {} 2023-10-18T20:30:27.273 DEBUG:boto:Token: None 2023-10-18T20:30:27.273 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:27.274 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MS5WbwezhtLBVOiQ8Pkj0IHQogU= 2023-10-18T20:30:27.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MS5WbwezhtLBVOiQ8Pkj0IHQogU='} 2023-10-18T20:30:27.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abd4d6c6157024ad-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.323 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.324 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.324 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.325 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.325 DEBUG:boto:Data: 2023-10-18T20:30:27.325 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.326 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.326 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.327 DEBUG:boto:Params: {} 2023-10-18T20:30:27.327 DEBUG:boto:Token: None 2023-10-18T20:30:27.328 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e91da5dc0e7546d8-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.332 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:27.333 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:27.334 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.335 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:27.335 DEBUG:boto:Data: 2023-10-18T20:30:27.336 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.337 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.337 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.338 DEBUG:boto:Params: {} 2023-10-18T20:30:27.339 DEBUG:boto:Token: None 2023-10-18T20:30:27.340 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:27.341 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7DPUmHYinzyvKVl4oqLdXh3ySMQ= 2023-10-18T20:30:27.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7DPUmHYinzyvKVl4oqLdXh3ySMQ='} 2023-10-18T20:30:27.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000344b21518fdc65-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.349 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.349 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.350 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.350 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.351 DEBUG:boto:Data: 2023-10-18T20:30:27.351 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.351 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.352 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.352 DEBUG:boto:Params: {} 2023-10-18T20:30:27.353 DEBUG:boto:Token: None 2023-10-18T20:30:27.353 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.354 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.357 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000815056d77dabb0fa-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.358 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:27.358 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:27.359 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.359 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:27.359 DEBUG:boto:Data: 2023-10-18T20:30:27.360 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.360 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.361 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.361 DEBUG:boto:Params: {} 2023-10-18T20:30:27.362 DEBUG:boto:Token: None 2023-10-18T20:30:27.362 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:27.362 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VvOAzxD2CxhLfdRXUOkbJfQ46hk= 2023-10-18T20:30:27.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VvOAzxD2CxhLfdRXUOkbJfQ46hk='} 2023-10-18T20:30:27.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d8c2711e2823338-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.407 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.408 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.408 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.408 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.409 DEBUG:boto:Data: 2023-10-18T20:30:27.409 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.410 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.410 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.410 DEBUG:boto:Params: {} 2023-10-18T20:30:27.411 DEBUG:boto:Token: None 2023-10-18T20:30:27.411 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.412 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021f2141930434057-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.416 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:27.416 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:27.416 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.417 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:27.417 DEBUG:boto:Data: 2023-10-18T20:30:27.418 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.419 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.419 DEBUG:boto:Params: {} 2023-10-18T20:30:27.419 DEBUG:boto:Token: None 2023-10-18T20:30:27.420 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:27.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ATACThb+jt1DByMM+n28EazgULY= 2023-10-18T20:30:27.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ATACThb+jt1DByMM+n28EazgULY='} 2023-10-18T20:30:27.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd920999a723b638-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.428 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.428 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.428 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.429 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.429 DEBUG:boto:Data: 2023-10-18T20:30:27.430 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.430 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.431 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.431 DEBUG:boto:Params: {} 2023-10-18T20:30:27.431 DEBUG:boto:Token: None 2023-10-18T20:30:27.432 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.432 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009970c0f88d595a43-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.436 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:27.436 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:27.437 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.437 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:27.438 DEBUG:boto:Data: 2023-10-18T20:30:27.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.439 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.439 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.439 DEBUG:boto:Params: {} 2023-10-18T20:30:27.440 DEBUG:boto:Token: None 2023-10-18T20:30:27.440 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:27.441 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nmWOG05wOR2PtADvfiCincdLVEM= 2023-10-18T20:30:27.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nmWOG05wOR2PtADvfiCincdLVEM='} 2023-10-18T20:30:27.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b267daf8b738dea-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.491 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.492 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.492 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.492 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.493 DEBUG:boto:Data: 2023-10-18T20:30:27.493 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.494 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.494 DEBUG:boto:Params: {} 2023-10-18T20:30:27.495 DEBUG:boto:Token: None 2023-10-18T20:30:27.495 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.496 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d78c99cfc15d53b6-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.500 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:27.500 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:27.501 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.501 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:27.502 DEBUG:boto:Data: 2023-10-18T20:30:27.502 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.503 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.503 DEBUG:boto:Params: {} 2023-10-18T20:30:27.504 DEBUG:boto:Token: None 2023-10-18T20:30:27.504 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:27.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0zQYNzCilBOJmXllRs7605LDNfU= 2023-10-18T20:30:27.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0zQYNzCilBOJmXllRs7605LDNfU='} 2023-10-18T20:30:27.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000456053295867529b-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:27.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:27.512 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-58 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:30:27.565 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:27.565 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:27.609 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:30:27.610 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-58 2023-10-18T20:30:27.616 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 27 2023-10-18T20:30:27.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:27.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:27.662 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:27.714 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:27.714 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:27.754 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:27.755 INFO:teuthology.orchestra.run.smithi082.stdout: "objs": [ 2023-10-18T20:30:27.755 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000058", 2023-10-18T20:30:27.755 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000026", 2023-10-18T20:30:27.755 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000011", 2023-10-18T20:30:27.755 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000046", 2023-10-18T20:30:27.756 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000014", 2023-10-18T20:30:27.756 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000050", 2023-10-18T20:30:27.756 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000032", 2023-10-18T20:30:27.756 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000002", 2023-10-18T20:30:27.756 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000039", 2023-10-18T20:30:27.757 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000007", 2023-10-18T20:30:27.757 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000006", 2023-10-18T20:30:27.757 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000040", 2023-10-18T20:30:27.757 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000009", 2023-10-18T20:30:27.757 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000045", 2023-10-18T20:30:27.758 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000013", 2023-10-18T20:30:27.758 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000012", 2023-10-18T20:30:27.758 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000033", 2023-10-18T20:30:27.758 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000001", 2023-10-18T20:30:27.759 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000000", 2023-10-18T20:30:27.759 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000021", 2023-10-18T20:30:27.759 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000003", 2023-10-18T20:30:27.759 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000055", 2023-10-18T20:30:27.759 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000008" 2023-10-18T20:30:27.760 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:27.760 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:27.761 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.761 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.762 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.762 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.762 DEBUG:boto:Data: 2023-10-18T20:30:27.763 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.764 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.764 DEBUG:boto:Params: {} 2023-10-18T20:30:27.764 DEBUG:boto:Token: None 2023-10-18T20:30:27.765 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.765 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000015b7b0eb46a40bb-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.769 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:27.769 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:27.770 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.770 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:27.771 DEBUG:boto:Data: 2023-10-18T20:30:27.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.771 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.772 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.772 DEBUG:boto:Params: {} 2023-10-18T20:30:27.772 DEBUG:boto:Token: None 2023-10-18T20:30:27.773 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:27.773 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9BidssqnbQtVWZSnz+kHFJoWhOY= 2023-10-18T20:30:27.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9BidssqnbQtVWZSnz+kHFJoWhOY='} 2023-10-18T20:30:27.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061b2f4fa51186ab8-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.779 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.779 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.780 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.780 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.780 DEBUG:boto:Data: 2023-10-18T20:30:27.781 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.781 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.782 DEBUG:boto:Params: {} 2023-10-18T20:30:27.782 DEBUG:boto:Token: None 2023-10-18T20:30:27.783 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077352e86ac304a8b-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.787 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:27.787 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:27.788 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.788 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:27.788 DEBUG:boto:Data: 2023-10-18T20:30:27.789 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.790 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.790 DEBUG:boto:Params: {} 2023-10-18T20:30:27.790 DEBUG:boto:Token: None 2023-10-18T20:30:27.791 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:27.791 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:etZjjRS8uXalOwMW7TDgsDfCtHY= 2023-10-18T20:30:27.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:etZjjRS8uXalOwMW7TDgsDfCtHY='} 2023-10-18T20:30:27.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2cddbc2c6f9b816-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.835 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.835 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.836 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.836 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.836 DEBUG:boto:Data: 2023-10-18T20:30:27.837 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.837 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.838 DEBUG:boto:Params: {} 2023-10-18T20:30:27.838 DEBUG:boto:Token: None 2023-10-18T20:30:27.839 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.843 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f2982c99fed3bf2-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.843 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:27.844 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:27.844 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.844 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:27.845 DEBUG:boto:Data: 2023-10-18T20:30:27.845 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.846 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.846 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.846 DEBUG:boto:Params: {} 2023-10-18T20:30:27.847 DEBUG:boto:Token: None 2023-10-18T20:30:27.847 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:27.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xA3ZBy8h8gEySHNrcJlv4NRAe8E= 2023-10-18T20:30:27.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xA3ZBy8h8gEySHNrcJlv4NRAe8E='} 2023-10-18T20:30:27.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001225457351508226-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.855 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.855 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.856 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.856 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.857 DEBUG:boto:Data: 2023-10-18T20:30:27.857 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.858 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.859 DEBUG:boto:Params: {} 2023-10-18T20:30:27.859 DEBUG:boto:Token: None 2023-10-18T20:30:27.860 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a99619fcd4b8598-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.865 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:27.866 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:27.866 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.867 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:27.867 DEBUG:boto:Data: 2023-10-18T20:30:27.868 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.869 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.869 DEBUG:boto:Params: {} 2023-10-18T20:30:27.869 DEBUG:boto:Token: None 2023-10-18T20:30:27.870 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:27.871 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LtVDtOc8PZ1o8lbP7piuf3PGCX0= 2023-10-18T20:30:27.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LtVDtOc8PZ1o8lbP7piuf3PGCX0='} 2023-10-18T20:30:27.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cde24e44517b10f9-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.915 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.916 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.916 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.917 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.918 DEBUG:boto:Data: 2023-10-18T20:30:27.918 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.919 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.920 DEBUG:boto:Params: {} 2023-10-18T20:30:27.920 DEBUG:boto:Token: None 2023-10-18T20:30:27.922 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.923 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a75b68eca72012b7-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.927 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:27.927 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:27.928 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.928 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:27.929 DEBUG:boto:Data: 2023-10-18T20:30:27.929 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.930 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.930 DEBUG:boto:Params: {} 2023-10-18T20:30:27.931 DEBUG:boto:Token: None 2023-10-18T20:30:27.931 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:27.932 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w5rgPu8ef7t6bGP+Rz7TSz8EIlw= 2023-10-18T20:30:27.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w5rgPu8ef7t6bGP+Rz7TSz8EIlw='} 2023-10-18T20:30:27.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfc799bab3604506-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.939 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.939 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.939 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.940 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.940 DEBUG:boto:Data: 2023-10-18T20:30:27.941 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.941 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.942 DEBUG:boto:Params: {} 2023-10-18T20:30:27.942 DEBUG:boto:Token: None 2023-10-18T20:30:27.943 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.943 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d79421911e18e59-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.947 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:27.947 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:27.947 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.948 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:27.948 DEBUG:boto:Data: 2023-10-18T20:30:27.949 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.949 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.950 DEBUG:boto:Params: {} 2023-10-18T20:30:27.950 DEBUG:boto:Token: None 2023-10-18T20:30:27.951 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:27.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6tZqO/smHBY1E0SON9VoM1hFTF8= 2023-10-18T20:30:27.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6tZqO/smHBY1E0SON9VoM1hFTF8='} 2023-10-18T20:30:27.957 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbceaa372adaba70-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.958 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:27.958 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:27.959 DEBUG:boto:Method: HEAD 2023-10-18T20:30:27.959 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:27.960 DEBUG:boto:Data: 2023-10-18T20:30:27.960 DEBUG:boto:Headers: {} 2023-10-18T20:30:27.961 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.961 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.961 DEBUG:boto:Params: {} 2023-10-18T20:30:27.962 DEBUG:boto:Token: None 2023-10-18T20:30:27.962 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/ 2023-10-18T20:30:27.963 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0= 2023-10-18T20:30:27.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZrbgLUpzRiktL6iyLyQEBhVlvE0=', 'Content-Length': '0'} 2023-10-18T20:30:27.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e5958a09e2e72bb-0065304063-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:27.967 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:27.967 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:27.968 DEBUG:boto:Method: PUT 2023-10-18T20:30:27.968 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:27.968 DEBUG:boto:Data: 2023-10-18T20:30:27.969 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:27.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:27.970 DEBUG:boto:Port: 8000 2023-10-18T20:30:27.970 DEBUG:boto:Params: {} 2023-10-18T20:30:27.971 DEBUG:boto:Token: None 2023-10-18T20:30:27.971 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:27 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:27.972 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KenCc4M+3FqP0QF0QJSG3JpVLlg= 2023-10-18T20:30:27.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KenCc4M+3FqP0QF0QJSG3JpVLlg='} 2023-10-18T20:30:28.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a21b74e068c11c37-0065304063-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.015 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.015 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.016 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.016 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.017 DEBUG:boto:Data: 2023-10-18T20:30:28.017 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.017 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.018 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.018 DEBUG:boto:Params: {} 2023-10-18T20:30:28.019 DEBUG:boto:Token: None 2023-10-18T20:30:28.019 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.020 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041a3ecd0db926fb1-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.023 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:28.024 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:28.024 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.025 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:28.025 DEBUG:boto:Data: 2023-10-18T20:30:28.026 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.026 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.026 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.027 DEBUG:boto:Params: {} 2023-10-18T20:30:28.027 DEBUG:boto:Token: None 2023-10-18T20:30:28.028 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:28.028 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:saJKGEn2TCI7weAWp0uEJ3fuGOA= 2023-10-18T20:30:28.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:saJKGEn2TCI7weAWp0uEJ3fuGOA='} 2023-10-18T20:30:28.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d1876f42df348bc-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.034 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.035 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.035 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.036 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.036 DEBUG:boto:Data: 2023-10-18T20:30:28.036 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.037 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.037 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.038 DEBUG:boto:Params: {} 2023-10-18T20:30:28.038 DEBUG:boto:Token: None 2023-10-18T20:30:28.039 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f49a8001c43e7ec-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.043 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:28.043 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:28.044 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.044 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:28.044 DEBUG:boto:Data: 2023-10-18T20:30:28.045 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.045 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.046 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.046 DEBUG:boto:Params: {} 2023-10-18T20:30:28.047 DEBUG:boto:Token: None 2023-10-18T20:30:28.047 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:28.047 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nZT1gohKdZRv5ND/42zvrJwQbws= 2023-10-18T20:30:28.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nZT1gohKdZRv5ND/42zvrJwQbws='} 2023-10-18T20:30:28.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d45185ab96fad058-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.095 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.096 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.096 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.096 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.097 DEBUG:boto:Data: 2023-10-18T20:30:28.097 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.098 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.098 DEBUG:boto:Params: {} 2023-10-18T20:30:28.099 DEBUG:boto:Token: None 2023-10-18T20:30:28.099 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.103 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008377ca15cd0bbd1b-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.103 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:28.104 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:28.104 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.105 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:28.105 DEBUG:boto:Data: 2023-10-18T20:30:28.106 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.106 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.106 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.107 DEBUG:boto:Params: {} 2023-10-18T20:30:28.107 DEBUG:boto:Token: None 2023-10-18T20:30:28.108 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:28.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vl8p1MUsoOpwvRFpqYnSfC5kK1Q= 2023-10-18T20:30:28.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vl8p1MUsoOpwvRFpqYnSfC5kK1Q='} 2023-10-18T20:30:28.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066f7e7e71f55efa2-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.115 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.116 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.116 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.116 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.117 DEBUG:boto:Data: 2023-10-18T20:30:28.117 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.117 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.118 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.118 DEBUG:boto:Params: {} 2023-10-18T20:30:28.119 DEBUG:boto:Token: None 2023-10-18T20:30:28.119 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.120 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000694ad4ae3a78388a-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.123 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:28.124 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:28.124 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.125 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:28.125 DEBUG:boto:Data: 2023-10-18T20:30:28.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.126 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.126 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.127 DEBUG:boto:Params: {} 2023-10-18T20:30:28.127 DEBUG:boto:Token: None 2023-10-18T20:30:28.128 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:28.128 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XenmjCcxDXji147Tp30n9u9lI2I= 2023-10-18T20:30:28.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XenmjCcxDXji147Tp30n9u9lI2I='} 2023-10-18T20:30:28.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d675e74fd3763998-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.176 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.176 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.177 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.178 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.178 DEBUG:boto:Data: 2023-10-18T20:30:28.179 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.180 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.180 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.181 DEBUG:boto:Params: {} 2023-10-18T20:30:28.182 DEBUG:boto:Token: None 2023-10-18T20:30:28.183 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000219d8f3657d4ad62-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.190 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:28.191 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:28.191 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.192 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:28.192 DEBUG:boto:Data: 2023-10-18T20:30:28.193 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.193 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.194 DEBUG:boto:Params: {} 2023-10-18T20:30:28.194 DEBUG:boto:Token: None 2023-10-18T20:30:28.195 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:28.195 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LJkt/yvmP297e4fO1x7kyykDecQ= 2023-10-18T20:30:28.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LJkt/yvmP297e4fO1x7kyykDecQ='} 2023-10-18T20:30:28.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000663125e7309bdad6-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.203 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.204 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.204 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.205 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.205 DEBUG:boto:Data: 2023-10-18T20:30:28.205 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.206 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.206 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.207 DEBUG:boto:Params: {} 2023-10-18T20:30:28.207 DEBUG:boto:Token: None 2023-10-18T20:30:28.208 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.209 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3f52389f60ea9ab-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.213 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:28.213 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:28.214 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.214 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:28.215 DEBUG:boto:Data: 2023-10-18T20:30:28.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.216 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.216 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.217 DEBUG:boto:Params: {} 2023-10-18T20:30:28.218 DEBUG:boto:Token: None 2023-10-18T20:30:28.219 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:28.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UYOkPhWEVZFmY8KI6MoaevNKYQY= 2023-10-18T20:30:28.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UYOkPhWEVZFmY8KI6MoaevNKYQY='} 2023-10-18T20:30:28.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007521d8f92dd22503-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.268 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.269 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.269 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.270 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.271 DEBUG:boto:Data: 2023-10-18T20:30:28.271 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.272 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.273 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.274 DEBUG:boto:Params: {} 2023-10-18T20:30:28.275 DEBUG:boto:Token: None 2023-10-18T20:30:28.275 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b986fd7834bcb9d3-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.280 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:28.281 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:28.281 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.282 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:28.282 DEBUG:boto:Data: 2023-10-18T20:30:28.285 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.286 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.287 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.287 DEBUG:boto:Params: {} 2023-10-18T20:30:28.288 DEBUG:boto:Token: None 2023-10-18T20:30:28.288 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:28.289 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dOerGFPkhK3wu9fZvceWpfTsCbc= 2023-10-18T20:30:28.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dOerGFPkhK3wu9fZvceWpfTsCbc='} 2023-10-18T20:30:28.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004057df9c90d50155-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.296 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.297 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.297 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.298 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.298 DEBUG:boto:Data: 2023-10-18T20:30:28.299 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.299 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.300 DEBUG:boto:Params: {} 2023-10-18T20:30:28.300 DEBUG:boto:Token: None 2023-10-18T20:30:28.301 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caf51619152a8ee5-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.306 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:28.306 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:28.307 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.307 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:28.307 DEBUG:boto:Data: 2023-10-18T20:30:28.308 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.308 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.309 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.309 DEBUG:boto:Params: {} 2023-10-18T20:30:28.310 DEBUG:boto:Token: None 2023-10-18T20:30:28.310 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:28.311 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mcp9WiLnOsEPjP42mjSB1Ws//bU= 2023-10-18T20:30:28.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mcp9WiLnOsEPjP42mjSB1Ws//bU='} 2023-10-18T20:30:28.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a36786ad35d9a68-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.355 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.356 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.356 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.357 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.357 DEBUG:boto:Data: 2023-10-18T20:30:28.357 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.358 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.359 DEBUG:boto:Params: {} 2023-10-18T20:30:28.359 DEBUG:boto:Token: None 2023-10-18T20:30:28.360 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c309e0f1eef9f2da-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.364 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:28.365 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:28.365 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.366 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:28.366 DEBUG:boto:Data: 2023-10-18T20:30:28.367 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.368 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.368 DEBUG:boto:Params: {} 2023-10-18T20:30:28.369 DEBUG:boto:Token: None 2023-10-18T20:30:28.369 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:28.370 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6HMVIyUisPvGrppMXwZ1FOf/QuU= 2023-10-18T20:30:28.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6HMVIyUisPvGrppMXwZ1FOf/QuU='} 2023-10-18T20:30:28.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a40378e32feb9bd-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.376 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.376 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.376 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.377 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.377 DEBUG:boto:Data: 2023-10-18T20:30:28.378 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.378 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.379 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.379 DEBUG:boto:Params: {} 2023-10-18T20:30:28.380 DEBUG:boto:Token: None 2023-10-18T20:30:28.380 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.381 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086dd68eb48b2eace-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.385 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:28.385 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:28.386 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.387 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:28.387 DEBUG:boto:Data: 2023-10-18T20:30:28.388 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.388 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.389 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.389 DEBUG:boto:Params: {} 2023-10-18T20:30:28.390 DEBUG:boto:Token: None 2023-10-18T20:30:28.390 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:28.391 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YDgKeDFezWpawhvsWbvdE6rNJUI= 2023-10-18T20:30:28.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YDgKeDFezWpawhvsWbvdE6rNJUI='} 2023-10-18T20:30:28.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f02cf459ee0c7882-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.436 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.437 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.438 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.438 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.439 DEBUG:boto:Data: 2023-10-18T20:30:28.440 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.441 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.442 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.443 DEBUG:boto:Params: {} 2023-10-18T20:30:28.443 DEBUG:boto:Token: None 2023-10-18T20:30:28.444 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.445 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084131adc7446ec41-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.450 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:28.451 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:28.452 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.452 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:28.453 DEBUG:boto:Data: 2023-10-18T20:30:28.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.455 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.456 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.457 DEBUG:boto:Params: {} 2023-10-18T20:30:28.457 DEBUG:boto:Token: None 2023-10-18T20:30:28.458 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:28.458 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tIJ9K6ONb9X4r+agRCPqTuBv5HA= 2023-10-18T20:30:28.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tIJ9K6ONb9X4r+agRCPqTuBv5HA='} 2023-10-18T20:30:28.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ce6c2a1504f4341-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.467 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.467 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.467 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.468 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.468 DEBUG:boto:Data: 2023-10-18T20:30:28.469 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.469 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.470 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.470 DEBUG:boto:Params: {} 2023-10-18T20:30:28.470 DEBUG:boto:Token: None 2023-10-18T20:30:28.471 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.471 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fce2810f01a1340-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.475 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:28.476 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:28.476 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.477 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:28.477 DEBUG:boto:Data: 2023-10-18T20:30:28.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.478 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.478 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.479 DEBUG:boto:Params: {} 2023-10-18T20:30:28.479 DEBUG:boto:Token: None 2023-10-18T20:30:28.480 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:28.480 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SET0ogA9fnmhlSZtQGxja4Kz9ao= 2023-10-18T20:30:28.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SET0ogA9fnmhlSZtQGxja4Kz9ao='} 2023-10-18T20:30:28.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af0278fc5aa2ef31-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.527 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.528 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.528 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.528 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.529 DEBUG:boto:Data: 2023-10-18T20:30:28.529 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.530 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.531 DEBUG:boto:Params: {} 2023-10-18T20:30:28.531 DEBUG:boto:Token: None 2023-10-18T20:30:28.532 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4367292d461d211-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.536 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:28.536 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:28.537 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.537 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:28.538 DEBUG:boto:Data: 2023-10-18T20:30:28.538 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.538 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.539 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.539 DEBUG:boto:Params: {} 2023-10-18T20:30:28.540 DEBUG:boto:Token: None 2023-10-18T20:30:28.540 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:28.541 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WCK/b2a0zr+cFSbBpUzwvJWiTB8= 2023-10-18T20:30:28.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WCK/b2a0zr+cFSbBpUzwvJWiTB8='} 2023-10-18T20:30:28.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033f419afedb53576-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.547 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.547 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.548 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.548 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.548 DEBUG:boto:Data: 2023-10-18T20:30:28.549 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.549 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.550 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.550 DEBUG:boto:Params: {} 2023-10-18T20:30:28.551 DEBUG:boto:Token: None 2023-10-18T20:30:28.551 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ed46a19dbb3c3da-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.555 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:28.556 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:28.556 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.556 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:28.557 DEBUG:boto:Data: 2023-10-18T20:30:28.557 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.558 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.559 DEBUG:boto:Params: {} 2023-10-18T20:30:28.559 DEBUG:boto:Token: None 2023-10-18T20:30:28.560 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:28.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y8/DVwxdlHjfKpKOqfZc815TitI= 2023-10-18T20:30:28.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y8/DVwxdlHjfKpKOqfZc815TitI='} 2023-10-18T20:30:28.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0a15908ff371026-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:28.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:28.608 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-58 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:30:28.660 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:28.661 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:28.705 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:30:28.706 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-58 2023-10-18T20:30:28.714 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 35 2023-10-18T20:30:28.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:28.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:28.764 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:28.816 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:28.816 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:28.855 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:28.856 INFO:teuthology.orchestra.run.smithi082.stdout: "objs": [ 2023-10-18T20:30:28.856 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000006", 2023-10-18T20:30:28.856 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000059", 2023-10-18T20:30:28.856 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000027", 2023-10-18T20:30:28.857 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000004", 2023-10-18T20:30:28.857 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000011", 2023-10-18T20:30:28.857 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000002", 2023-10-18T20:30:28.857 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000019", 2023-10-18T20:30:28.857 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000049", 2023-10-18T20:30:28.858 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000017", 2023-10-18T20:30:28.858 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000008", 2023-10-18T20:30:28.858 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000015", 2023-10-18T20:30:28.858 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000031", 2023-10-18T20:30:28.859 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000022", 2023-10-18T20:30:28.859 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000013", 2023-10-18T20:30:28.859 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000020", 2023-10-18T20:30:28.859 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000037", 2023-10-18T20:30:28.859 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000005", 2023-10-18T20:30:28.860 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000012", 2023-10-18T20:30:28.860 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000003", 2023-10-18T20:30:28.860 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000042", 2023-10-18T20:30:28.860 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000010", 2023-10-18T20:30:28.860 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000001", 2023-10-18T20:30:28.861 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000009", 2023-10-18T20:30:28.861 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000025", 2023-10-18T20:30:28.861 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000016", 2023-10-18T20:30:28.861 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000000", 2023-10-18T20:30:28.862 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000014", 2023-10-18T20:30:28.862 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000030", 2023-10-18T20:30:28.862 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000021" 2023-10-18T20:30:28.862 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:28.862 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:28.864 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.864 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.865 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.865 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.865 DEBUG:boto:Data: 2023-10-18T20:30:28.866 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.867 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.867 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.868 DEBUG:boto:Params: {} 2023-10-18T20:30:28.868 DEBUG:boto:Token: None 2023-10-18T20:30:28.869 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.870 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.873 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ff192c219eeb89c-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.874 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:28.874 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:28.875 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.875 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:28.876 DEBUG:boto:Data: 2023-10-18T20:30:28.876 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.877 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.878 DEBUG:boto:Params: {} 2023-10-18T20:30:28.878 DEBUG:boto:Token: None 2023-10-18T20:30:28.879 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:28.879 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JsB0tDsgFJnOJaaU8WyQoSr1y64= 2023-10-18T20:30:28.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JsB0tDsgFJnOJaaU8WyQoSr1y64='} 2023-10-18T20:30:28.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6051fda1cca7a7f-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.885 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.885 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.886 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.886 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.887 DEBUG:boto:Data: 2023-10-18T20:30:28.887 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.887 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.888 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.888 DEBUG:boto:Params: {} 2023-10-18T20:30:28.889 DEBUG:boto:Token: None 2023-10-18T20:30:28.889 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.889 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.893 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007eba7c748f99a30e-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.894 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:28.894 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:28.895 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.895 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:28.896 DEBUG:boto:Data: 2023-10-18T20:30:28.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.897 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.897 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.898 DEBUG:boto:Params: {} 2023-10-18T20:30:28.898 DEBUG:boto:Token: None 2023-10-18T20:30:28.899 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:28.899 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mc2P//3Ot9OJoAQWsb54KDa2zss= 2023-10-18T20:30:28.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mc2P//3Ot9OJoAQWsb54KDa2zss='} 2023-10-18T20:30:28.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000493ed8b1550e87e3-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.906 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.907 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.907 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.908 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.908 DEBUG:boto:Data: 2023-10-18T20:30:28.909 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.909 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.909 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.910 DEBUG:boto:Params: {} 2023-10-18T20:30:28.910 DEBUG:boto:Token: None 2023-10-18T20:30:28.911 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.911 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000782fd9e7ba587896-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.915 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:28.916 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:28.916 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.917 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:28.917 DEBUG:boto:Data: 2023-10-18T20:30:28.918 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.919 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.919 DEBUG:boto:Params: {} 2023-10-18T20:30:28.920 DEBUG:boto:Token: None 2023-10-18T20:30:28.920 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:28.921 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ieSGbJZ15nXpd6Q+zIuA44BlMuw= 2023-10-18T20:30:28.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ieSGbJZ15nXpd6Q+zIuA44BlMuw='} 2023-10-18T20:30:28.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3b54a04d74b0c08-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.971 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.972 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.972 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.973 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.973 DEBUG:boto:Data: 2023-10-18T20:30:28.973 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.974 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.975 DEBUG:boto:Params: {} 2023-10-18T20:30:28.975 DEBUG:boto:Token: None 2023-10-18T20:30:28.976 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.976 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:28.980 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f290e55e436853f-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.980 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:28.981 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:28.981 DEBUG:boto:Method: PUT 2023-10-18T20:30:28.982 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:28.982 DEBUG:boto:Data: 2023-10-18T20:30:28.982 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:28.983 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.983 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.984 DEBUG:boto:Params: {} 2023-10-18T20:30:28.984 DEBUG:boto:Token: None 2023-10-18T20:30:28.985 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:28.985 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y8/DVwxdlHjfKpKOqfZc815TitI= 2023-10-18T20:30:28.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y8/DVwxdlHjfKpKOqfZc815TitI='} 2023-10-18T20:30:28.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c1ce9e1ec0cad00-0065304064-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:28.993 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:28.994 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:28.994 DEBUG:boto:Method: HEAD 2023-10-18T20:30:28.995 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:28.995 DEBUG:boto:Data: 2023-10-18T20:30:28.995 DEBUG:boto:Headers: {} 2023-10-18T20:30:28.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:28.996 DEBUG:boto:Port: 8000 2023-10-18T20:30:28.997 DEBUG:boto:Params: {} 2023-10-18T20:30:28.997 DEBUG:boto:Token: None 2023-10-18T20:30:28.998 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:28 GMT /bofrnf-58/ 2023-10-18T20:30:28.998 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho= 2023-10-18T20:30:28.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YeYShaYnYYQHg7p3lJb0B2Tcpho=', 'Content-Length': '0'} 2023-10-18T20:30:29.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f71fdc73bd08e527-0065304064-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.003 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:29.004 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:29.004 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.005 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:29.005 DEBUG:boto:Data: 2023-10-18T20:30:29.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.006 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.006 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.007 DEBUG:boto:Params: {} 2023-10-18T20:30:29.007 DEBUG:boto:Token: None 2023-10-18T20:30:29.008 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:29.008 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uNQHhmUC7NTv+23a3OUdpibUmIU= 2023-10-18T20:30:29.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uNQHhmUC7NTv+23a3OUdpibUmIU='} 2023-10-18T20:30:29.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aed72feaa7bc5ec-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.055 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.056 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.056 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.057 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.057 DEBUG:boto:Data: 2023-10-18T20:30:29.057 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.058 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.058 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.059 DEBUG:boto:Params: {} 2023-10-18T20:30:29.059 DEBUG:boto:Token: None 2023-10-18T20:30:29.060 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.060 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021208463372592e4-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.064 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:29.065 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:29.065 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.066 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:29.066 DEBUG:boto:Data: 2023-10-18T20:30:29.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.067 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.068 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.068 DEBUG:boto:Params: {} 2023-10-18T20:30:29.070 DEBUG:boto:Token: None 2023-10-18T20:30:29.070 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:29.071 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9mtpe+Ua2KEyEa6pmjErFSSfwgk= 2023-10-18T20:30:29.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9mtpe+Ua2KEyEa6pmjErFSSfwgk='} 2023-10-18T20:30:29.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076f80769cf7f24ea-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.077 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.077 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.078 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.078 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.079 DEBUG:boto:Data: 2023-10-18T20:30:29.079 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.079 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.080 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.080 DEBUG:boto:Params: {} 2023-10-18T20:30:29.081 DEBUG:boto:Token: None 2023-10-18T20:30:29.081 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.081 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002104dbf807f822da-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.086 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:29.087 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:29.087 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.087 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:29.088 DEBUG:boto:Data: 2023-10-18T20:30:29.088 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.089 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.089 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.089 DEBUG:boto:Params: {} 2023-10-18T20:30:29.090 DEBUG:boto:Token: None 2023-10-18T20:30:29.090 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:29.091 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:antPbqw7CU7uQds2fGkcoUeRatU= 2023-10-18T20:30:29.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:antPbqw7CU7uQds2fGkcoUeRatU='} 2023-10-18T20:30:29.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006674617b2b87cfe3-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.139 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.140 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.140 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.141 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.141 DEBUG:boto:Data: 2023-10-18T20:30:29.141 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.142 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.143 DEBUG:boto:Params: {} 2023-10-18T20:30:29.143 DEBUG:boto:Token: None 2023-10-18T20:30:29.144 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037a019f2018f124e-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.148 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:29.149 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:29.149 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.150 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:29.150 DEBUG:boto:Data: 2023-10-18T20:30:29.150 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.151 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.152 DEBUG:boto:Params: {} 2023-10-18T20:30:29.152 DEBUG:boto:Token: None 2023-10-18T20:30:29.153 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:29.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZBdA2LYl4Z3MZqwoNw3QdtEMTDw= 2023-10-18T20:30:29.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZBdA2LYl4Z3MZqwoNw3QdtEMTDw='} 2023-10-18T20:30:29.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063fd32adf8215f1a-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.160 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.161 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.161 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.162 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.162 DEBUG:boto:Data: 2023-10-18T20:30:29.162 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.163 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.163 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.164 DEBUG:boto:Params: {} 2023-10-18T20:30:29.164 DEBUG:boto:Token: None 2023-10-18T20:30:29.165 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.165 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070cd44e12596c7fc-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.169 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:29.169 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:29.170 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.170 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:29.171 DEBUG:boto:Data: 2023-10-18T20:30:29.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.171 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.172 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.172 DEBUG:boto:Params: {} 2023-10-18T20:30:29.173 DEBUG:boto:Token: None 2023-10-18T20:30:29.173 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:29.174 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GU2RLkw+YfLxbNozh5eZZzbonSU= 2023-10-18T20:30:29.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GU2RLkw+YfLxbNozh5eZZzbonSU='} 2023-10-18T20:30:29.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b73852df48b31c3b-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.219 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.220 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.220 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.220 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.221 DEBUG:boto:Data: 2023-10-18T20:30:29.221 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.222 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.222 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.223 DEBUG:boto:Params: {} 2023-10-18T20:30:29.223 DEBUG:boto:Token: None 2023-10-18T20:30:29.224 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.224 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b5a188e62a2fcd-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.228 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:29.228 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:29.229 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.229 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:29.230 DEBUG:boto:Data: 2023-10-18T20:30:29.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.231 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.231 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.231 DEBUG:boto:Params: {} 2023-10-18T20:30:29.232 DEBUG:boto:Token: None 2023-10-18T20:30:29.232 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:29.233 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oAud3eyRcjEMinAMwC8Co7fmZCU= 2023-10-18T20:30:29.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oAud3eyRcjEMinAMwC8Co7fmZCU='} 2023-10-18T20:30:29.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000988e71f37d7ef2e9-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.240 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.240 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.241 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.241 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.242 DEBUG:boto:Data: 2023-10-18T20:30:29.242 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.243 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.243 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.243 DEBUG:boto:Params: {} 2023-10-18T20:30:29.244 DEBUG:boto:Token: None 2023-10-18T20:30:29.244 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.245 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ec32cd9845f070b-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.249 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:29.249 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:29.250 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.250 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:29.250 DEBUG:boto:Data: 2023-10-18T20:30:29.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.251 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.252 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.252 DEBUG:boto:Params: {} 2023-10-18T20:30:29.253 DEBUG:boto:Token: None 2023-10-18T20:30:29.253 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:29.253 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uNxmx/febTzehHcUfEM/wV6eadU= 2023-10-18T20:30:29.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uNxmx/febTzehHcUfEM/wV6eadU='} 2023-10-18T20:30:29.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3a9ca5d31a4caf2-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.299 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.300 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.300 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.300 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.301 DEBUG:boto:Data: 2023-10-18T20:30:29.301 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.302 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.302 DEBUG:boto:Params: {} 2023-10-18T20:30:29.303 DEBUG:boto:Token: None 2023-10-18T20:30:29.303 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aee384857a975fab-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.308 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:29.308 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:29.309 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.309 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:29.309 DEBUG:boto:Data: 2023-10-18T20:30:29.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.311 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.313 DEBUG:boto:Params: {} 2023-10-18T20:30:29.314 DEBUG:boto:Token: None 2023-10-18T20:30:29.314 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:29.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g020yEJ/4T03Z++1DDqwx42wXM4= 2023-10-18T20:30:29.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g020yEJ/4T03Z++1DDqwx42wXM4='} 2023-10-18T20:30:29.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008669e0c988a46c29-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.322 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.322 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.323 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.323 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.323 DEBUG:boto:Data: 2023-10-18T20:30:29.324 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.324 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.325 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.325 DEBUG:boto:Params: {} 2023-10-18T20:30:29.325 DEBUG:boto:Token: None 2023-10-18T20:30:29.326 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.326 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000100c411468d89f89-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.330 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:29.331 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:29.331 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.332 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:29.332 DEBUG:boto:Data: 2023-10-18T20:30:29.332 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.333 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.334 DEBUG:boto:Params: {} 2023-10-18T20:30:29.334 DEBUG:boto:Token: None 2023-10-18T20:30:29.335 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:29.335 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3imzIVPzidPeofwpICA3/RzTsE4= 2023-10-18T20:30:29.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3imzIVPzidPeofwpICA3/RzTsE4='} 2023-10-18T20:30:29.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f546414ad9bc0e67-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.379 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.380 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.380 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.380 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.381 DEBUG:boto:Data: 2023-10-18T20:30:29.381 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.382 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.382 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.382 DEBUG:boto:Params: {} 2023-10-18T20:30:29.383 DEBUG:boto:Token: None 2023-10-18T20:30:29.383 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.384 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4ea44376ca3896d-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.388 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:29.389 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:29.389 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.389 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:29.390 DEBUG:boto:Data: 2023-10-18T20:30:29.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.391 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.391 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.391 DEBUG:boto:Params: {} 2023-10-18T20:30:29.392 DEBUG:boto:Token: None 2023-10-18T20:30:29.392 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:29.393 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A8eNAaeWqq9lhSkWOnbWg4kgMCU= 2023-10-18T20:30:29.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A8eNAaeWqq9lhSkWOnbWg4kgMCU='} 2023-10-18T20:30:29.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f173aa0bfb771545-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.399 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.399 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.400 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.400 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.401 DEBUG:boto:Data: 2023-10-18T20:30:29.401 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.402 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.402 DEBUG:boto:Params: {} 2023-10-18T20:30:29.403 DEBUG:boto:Token: None 2023-10-18T20:30:29.403 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cce69e2b3db8ce1-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.408 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:29.408 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:29.409 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.409 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:29.409 DEBUG:boto:Data: 2023-10-18T20:30:29.410 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.411 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.411 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.411 DEBUG:boto:Params: {} 2023-10-18T20:30:29.412 DEBUG:boto:Token: None 2023-10-18T20:30:29.412 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:29.413 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9VqzRWo6VXLOD4p9s6x+FDqC13M= 2023-10-18T20:30:29.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9VqzRWo6VXLOD4p9s6x+FDqC13M='} 2023-10-18T20:30:29.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008487ea4adc302c84-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.459 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.460 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.460 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.460 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.461 DEBUG:boto:Data: 2023-10-18T20:30:29.461 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.462 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.463 DEBUG:boto:Params: {} 2023-10-18T20:30:29.463 DEBUG:boto:Token: None 2023-10-18T20:30:29.463 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.467 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6d5899edf2991d0-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.468 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:29.468 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:29.469 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.469 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:29.470 DEBUG:boto:Data: 2023-10-18T20:30:29.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.471 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.472 DEBUG:boto:Params: {} 2023-10-18T20:30:29.472 DEBUG:boto:Token: None 2023-10-18T20:30:29.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:29.473 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3z23+1UITazeFaDoKufzzh+9Zvo= 2023-10-18T20:30:29.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3z23+1UITazeFaDoKufzzh+9Zvo='} 2023-10-18T20:30:29.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000614039997a0b9d38-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.480 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.480 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.481 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.481 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.482 DEBUG:boto:Data: 2023-10-18T20:30:29.482 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.482 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.483 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.483 DEBUG:boto:Params: {} 2023-10-18T20:30:29.484 DEBUG:boto:Token: None 2023-10-18T20:30:29.484 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.485 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb473204913096ae-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.489 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:29.490 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:29.490 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.490 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:29.491 DEBUG:boto:Data: 2023-10-18T20:30:29.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.492 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.492 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.492 DEBUG:boto:Params: {} 2023-10-18T20:30:29.493 DEBUG:boto:Token: None 2023-10-18T20:30:29.493 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:29.494 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SdvKzdS0rRHavQB79frIDr1COsc= 2023-10-18T20:30:29.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SdvKzdS0rRHavQB79frIDr1COsc='} 2023-10-18T20:30:29.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007bfa1c48d2c74f83-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.539 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.540 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.540 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.540 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.541 DEBUG:boto:Data: 2023-10-18T20:30:29.541 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.542 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.543 DEBUG:boto:Params: {} 2023-10-18T20:30:29.543 DEBUG:boto:Token: None 2023-10-18T20:30:29.544 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c42babf30da066f-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.549 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:29.549 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:29.550 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.550 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:29.550 DEBUG:boto:Data: 2023-10-18T20:30:29.551 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.551 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.552 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.552 DEBUG:boto:Params: {} 2023-10-18T20:30:29.553 DEBUG:boto:Token: None 2023-10-18T20:30:29.553 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:29.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8ImMfKvZbOCtf747QvmSaDFONtw= 2023-10-18T20:30:29.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8ImMfKvZbOCtf747QvmSaDFONtw='} 2023-10-18T20:30:29.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7e4d42c60bb77e4-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.560 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.560 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.560 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.561 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.561 DEBUG:boto:Data: 2023-10-18T20:30:29.562 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.563 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.563 DEBUG:boto:Params: {} 2023-10-18T20:30:29.563 DEBUG:boto:Token: None 2023-10-18T20:30:29.564 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9b12352524b3f9a-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.568 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:29.569 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:29.569 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.569 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:29.570 DEBUG:boto:Data: 2023-10-18T20:30:29.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.571 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.571 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.571 DEBUG:boto:Params: {} 2023-10-18T20:30:29.572 DEBUG:boto:Token: None 2023-10-18T20:30:29.572 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:29.573 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qj+vdcbewFa2ZzB7Mh/s9dW716E= 2023-10-18T20:30:29.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qj+vdcbewFa2ZzB7Mh/s9dW716E='} 2023-10-18T20:30:29.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bc1063e4bff2e47-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.623 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.624 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.624 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.624 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.625 DEBUG:boto:Data: 2023-10-18T20:30:29.625 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.626 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.626 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.626 DEBUG:boto:Params: {} 2023-10-18T20:30:29.627 DEBUG:boto:Token: None 2023-10-18T20:30:29.627 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.628 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007f1962eebafc062-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.632 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:29.632 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:29.633 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.633 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:29.633 DEBUG:boto:Data: 2023-10-18T20:30:29.634 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.634 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.635 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.635 DEBUG:boto:Params: {} 2023-10-18T20:30:29.636 DEBUG:boto:Token: None 2023-10-18T20:30:29.636 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:29.636 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WQ0YVOo6O8iguZ5K1ho83JxTCXM= 2023-10-18T20:30:29.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WQ0YVOo6O8iguZ5K1ho83JxTCXM='} 2023-10-18T20:30:29.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0e0822d03867138-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.643 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.644 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.644 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.644 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.645 DEBUG:boto:Data: 2023-10-18T20:30:29.645 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.646 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.646 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.646 DEBUG:boto:Params: {} 2023-10-18T20:30:29.647 DEBUG:boto:Token: None 2023-10-18T20:30:29.647 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.648 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f316814e48b057ac-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.651 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:29.652 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:29.652 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.653 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:29.653 DEBUG:boto:Data: 2023-10-18T20:30:29.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.654 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.655 DEBUG:boto:Params: {} 2023-10-18T20:30:29.655 DEBUG:boto:Token: None 2023-10-18T20:30:29.656 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:29.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SB9MABH0yYf5JjIPUxUuv2nUbBE= 2023-10-18T20:30:29.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SB9MABH0yYf5JjIPUxUuv2nUbBE='} 2023-10-18T20:30:29.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f98e78968ae7b11e-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.707 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.708 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.708 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.709 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.709 DEBUG:boto:Data: 2023-10-18T20:30:29.709 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.710 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.710 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.711 DEBUG:boto:Params: {} 2023-10-18T20:30:29.711 DEBUG:boto:Token: None 2023-10-18T20:30:29.711 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf386b3925303c74-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.716 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:29.716 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:29.716 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.717 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:29.717 DEBUG:boto:Data: 2023-10-18T20:30:29.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.718 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.718 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.719 DEBUG:boto:Params: {} 2023-10-18T20:30:29.719 DEBUG:boto:Token: None 2023-10-18T20:30:29.720 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:29.720 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aD4Z64G8JxGtjItvqCk7j+/BHB8= 2023-10-18T20:30:29.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aD4Z64G8JxGtjItvqCk7j+/BHB8='} 2023-10-18T20:30:29.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079f731963fd61e5a-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.727 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.728 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.728 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.729 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.729 DEBUG:boto:Data: 2023-10-18T20:30:29.729 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.730 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.730 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.731 DEBUG:boto:Params: {} 2023-10-18T20:30:29.731 DEBUG:boto:Token: None 2023-10-18T20:30:29.732 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.732 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e3d92cc3fec7be0-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.736 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:29.737 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:29.737 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.738 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:29.738 DEBUG:boto:Data: 2023-10-18T20:30:29.738 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.739 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.739 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.740 DEBUG:boto:Params: {} 2023-10-18T20:30:29.740 DEBUG:boto:Token: None 2023-10-18T20:30:29.741 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:29.741 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lljnlSNPDZLPTbbtE1578xZoNDk= 2023-10-18T20:30:29.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lljnlSNPDZLPTbbtE1578xZoNDk='} 2023-10-18T20:30:29.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab351c782aee5f7d-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.787 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.788 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.788 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.788 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.789 DEBUG:boto:Data: 2023-10-18T20:30:29.789 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.790 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.791 DEBUG:boto:Params: {} 2023-10-18T20:30:29.791 DEBUG:boto:Token: None 2023-10-18T20:30:29.791 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079998d5039b4cfbe-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.796 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:29.797 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:29.797 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.798 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:29.798 DEBUG:boto:Data: 2023-10-18T20:30:29.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.799 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.800 DEBUG:boto:Params: {} 2023-10-18T20:30:29.800 DEBUG:boto:Token: None 2023-10-18T20:30:29.801 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:29.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T6GbNLgUT+/jn/gMpRr+9sp2ZIY= 2023-10-18T20:30:29.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T6GbNLgUT+/jn/gMpRr+9sp2ZIY='} 2023-10-18T20:30:29.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a2645c7e4c74959-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.808 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.808 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.809 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.809 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.810 DEBUG:boto:Data: 2023-10-18T20:30:29.810 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.811 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.811 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.811 DEBUG:boto:Params: {} 2023-10-18T20:30:29.812 DEBUG:boto:Token: None 2023-10-18T20:30:29.812 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.813 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075b64673ccbf7a84-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.817 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:29.817 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:29.817 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.818 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:29.818 DEBUG:boto:Data: 2023-10-18T20:30:29.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.819 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.819 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.820 DEBUG:boto:Params: {} 2023-10-18T20:30:29.820 DEBUG:boto:Token: None 2023-10-18T20:30:29.821 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:29.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oMkcQIBkXx3k9Q4CHlwfrXN8E34= 2023-10-18T20:30:29.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oMkcQIBkXx3k9Q4CHlwfrXN8E34='} 2023-10-18T20:30:29.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a20c28c510634b5a-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.867 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.868 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.868 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.868 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.869 DEBUG:boto:Data: 2023-10-18T20:30:29.869 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.870 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.870 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.870 DEBUG:boto:Params: {} 2023-10-18T20:30:29.871 DEBUG:boto:Token: None 2023-10-18T20:30:29.872 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a352bfec8e243494-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.876 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:29.876 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:29.877 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.877 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:29.878 DEBUG:boto:Data: 2023-10-18T20:30:29.878 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.879 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.879 DEBUG:boto:Params: {} 2023-10-18T20:30:29.880 DEBUG:boto:Token: None 2023-10-18T20:30:29.880 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:29.881 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ML8sKAsIBLWejEMqzataZxI6Iw0= 2023-10-18T20:30:29.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ML8sKAsIBLWejEMqzataZxI6Iw0='} 2023-10-18T20:30:29.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029a256d29b5b7e59-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.887 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:29.887 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:29.888 DEBUG:boto:Method: HEAD 2023-10-18T20:30:29.889 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:29.889 DEBUG:boto:Data: 2023-10-18T20:30:29.890 DEBUG:boto:Headers: {} 2023-10-18T20:30:29.891 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.891 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.892 DEBUG:boto:Params: {} 2023-10-18T20:30:29.892 DEBUG:boto:Token: None 2023-10-18T20:30:29.893 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/ 2023-10-18T20:30:29.894 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs= 2023-10-18T20:30:29.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utfhvv1m5PbG0KkqE1MmgdSqWfs=', 'Content-Length': '0'} 2023-10-18T20:30:29.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009f0db0d9fb7863f-0065304065-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.897 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:29.898 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:29.898 DEBUG:boto:Method: PUT 2023-10-18T20:30:29.899 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:29.899 DEBUG:boto:Data: 2023-10-18T20:30:29.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:29.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:29.900 DEBUG:boto:Port: 8000 2023-10-18T20:30:29.901 DEBUG:boto:Params: {} 2023-10-18T20:30:29.901 DEBUG:boto:Token: None 2023-10-18T20:30:29.902 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:29 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:29.902 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AtgpLgSL+7ZuQ76phOc7MYidgAg= 2023-10-18T20:30:29.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AtgpLgSL+7ZuQ76phOc7MYidgAg='} 2023-10-18T20:30:29.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001dceb6aee53cbbcf-0065304065-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:29.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:29.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:29.948 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-58 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:30:30.001 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:30.001 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:30.053 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:30:30.053 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-58 2023-10-18T20:30:30.064 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 43 2023-10-18T20:30:30.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:30.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:30.122 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:30.174 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:30.174 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:30.214 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:30.215 INFO:teuthology.orchestra.run.smithi082.stdout: "objs": [ 2023-10-18T20:30:30.215 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000019", 2023-10-18T20:30:30.215 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000006", 2023-10-18T20:30:30.216 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000467", 2023-10-18T20:30:30.216 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000005", 2023-10-18T20:30:30.216 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000056", 2023-10-18T20:30:30.216 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000004", 2023-10-18T20:30:30.217 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000023", 2023-10-18T20:30:30.217 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000317", 2023-10-18T20:30:30.217 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000022", 2023-10-18T20:30:30.217 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000000", 2023-10-18T20:30:30.217 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000021", 2023-10-18T20:30:30.218 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000009", 2023-10-18T20:30:30.218 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000271", 2023-10-18T20:30:30.218 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000008", 2023-10-18T20:30:30.218 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000018", 2023-10-18T20:30:30.218 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000007", 2023-10-18T20:30:30.219 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000017", 2023-10-18T20:30:30.219 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000162", 2023-10-18T20:30:30.219 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000016", 2023-10-18T20:30:30.219 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000003", 2023-10-18T20:30:30.219 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000423", 2023-10-18T20:30:30.220 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000002", 2023-10-18T20:30:30.220 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000012", 2023-10-18T20:30:30.220 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000001", 2023-10-18T20:30:30.220 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000011", 2023-10-18T20:30:30.221 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000283", 2023-10-18T20:30:30.221 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000010", 2023-10-18T20:30:30.221 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000061", 2023-10-18T20:30:30.221 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000050", 2023-10-18T20:30:30.221 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000029", 2023-10-18T20:30:30.222 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000133" 2023-10-18T20:30:30.222 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:30.222 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:30.223 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.224 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.224 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.225 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.225 DEBUG:boto:Data: 2023-10-18T20:30:30.226 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.226 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.227 DEBUG:boto:Params: {} 2023-10-18T20:30:30.227 DEBUG:boto:Token: None 2023-10-18T20:30:30.228 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.228 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bd36fdd9d5e02e6-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.233 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:30.234 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:30.234 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.234 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:30.235 DEBUG:boto:Data: 2023-10-18T20:30:30.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.236 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.236 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.237 DEBUG:boto:Params: {} 2023-10-18T20:30:30.237 DEBUG:boto:Token: None 2023-10-18T20:30:30.237 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:30.238 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QgChO7yMZQQGQYtrtcQI+hPyZPk= 2023-10-18T20:30:30.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QgChO7yMZQQGQYtrtcQI+hPyZPk='} 2023-10-18T20:30:30.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000045af45df556c9d88-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.244 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.245 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.245 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.246 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.246 DEBUG:boto:Data: 2023-10-18T20:30:30.247 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.247 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.248 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.248 DEBUG:boto:Params: {} 2023-10-18T20:30:30.249 DEBUG:boto:Token: None 2023-10-18T20:30:30.249 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ee2287388131ccd-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.254 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:30.254 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:30.255 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.255 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:30.256 DEBUG:boto:Data: 2023-10-18T20:30:30.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.256 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.257 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.257 DEBUG:boto:Params: {} 2023-10-18T20:30:30.258 DEBUG:boto:Token: None 2023-10-18T20:30:30.258 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:30.259 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gMwBy8gm2gQHQCWafhEwzXZrh5s= 2023-10-18T20:30:30.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gMwBy8gm2gQHQCWafhEwzXZrh5s='} 2023-10-18T20:30:30.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098fa49954c032e2c-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.265 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.266 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.267 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.267 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.268 DEBUG:boto:Data: 2023-10-18T20:30:30.268 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.269 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.270 DEBUG:boto:Params: {} 2023-10-18T20:30:30.270 DEBUG:boto:Token: None 2023-10-18T20:30:30.271 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.271 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.275 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd8f7417f3e34ab0-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.276 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:30.277 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:30.277 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.278 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:30.278 DEBUG:boto:Data: 2023-10-18T20:30:30.279 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.279 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.280 DEBUG:boto:Params: {} 2023-10-18T20:30:30.280 DEBUG:boto:Token: None 2023-10-18T20:30:30.281 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:30.281 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NzD0ZajuAROfS5RnhXXxpPLp2Xo= 2023-10-18T20:30:30.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NzD0ZajuAROfS5RnhXXxpPLp2Xo='} 2023-10-18T20:30:30.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddb993f18e5d0de0-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.331 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.332 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.332 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.332 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.333 DEBUG:boto:Data: 2023-10-18T20:30:30.333 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.334 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.338 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.338 DEBUG:boto:Params: {} 2023-10-18T20:30:30.339 DEBUG:boto:Token: None 2023-10-18T20:30:30.339 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baddd1dd06858dbf-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.344 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:30.344 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:30.344 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.345 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:30.345 DEBUG:boto:Data: 2023-10-18T20:30:30.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.346 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.347 DEBUG:boto:Params: {} 2023-10-18T20:30:30.347 DEBUG:boto:Token: None 2023-10-18T20:30:30.348 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:30.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xw94yDeFRcb0ss/llPuV3NBJ4JY= 2023-10-18T20:30:30.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xw94yDeFRcb0ss/llPuV3NBJ4JY='} 2023-10-18T20:30:30.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023007c3dddb80979-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.355 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.355 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.356 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.356 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.356 DEBUG:boto:Data: 2023-10-18T20:30:30.357 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.357 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.358 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.358 DEBUG:boto:Params: {} 2023-10-18T20:30:30.359 DEBUG:boto:Token: None 2023-10-18T20:30:30.359 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.359 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000256f88fa65562ff5-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.364 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:30.364 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:30.365 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.366 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:30.366 DEBUG:boto:Data: 2023-10-18T20:30:30.367 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.368 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.369 DEBUG:boto:Params: {} 2023-10-18T20:30:30.369 DEBUG:boto:Token: None 2023-10-18T20:30:30.369 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:30.370 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2sEJiibxhpdfJ9ZmcGCAcG1kIRM= 2023-10-18T20:30:30.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2sEJiibxhpdfJ9ZmcGCAcG1kIRM='} 2023-10-18T20:30:30.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c61ae9ff09887c67-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.415 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.416 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.416 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.416 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.417 DEBUG:boto:Data: 2023-10-18T20:30:30.417 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.418 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.419 DEBUG:boto:Params: {} 2023-10-18T20:30:30.419 DEBUG:boto:Token: None 2023-10-18T20:30:30.419 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e67387363b997620-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.424 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:30.424 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:30.425 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.425 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:30.425 DEBUG:boto:Data: 2023-10-18T20:30:30.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.427 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.427 DEBUG:boto:Params: {} 2023-10-18T20:30:30.427 DEBUG:boto:Token: None 2023-10-18T20:30:30.428 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:30.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jFVYEl/T6UdQetoySCH+5AJF3F4= 2023-10-18T20:30:30.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jFVYEl/T6UdQetoySCH+5AJF3F4='} 2023-10-18T20:30:30.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1c23a71a610b303-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.436 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.436 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.436 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.437 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.437 DEBUG:boto:Data: 2023-10-18T20:30:30.438 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.438 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.438 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.439 DEBUG:boto:Params: {} 2023-10-18T20:30:30.439 DEBUG:boto:Token: None 2023-10-18T20:30:30.440 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9ab7cc0ac6c428d-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.444 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:30.445 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:30.445 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.445 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:30.446 DEBUG:boto:Data: 2023-10-18T20:30:30.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.447 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.447 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.448 DEBUG:boto:Params: {} 2023-10-18T20:30:30.448 DEBUG:boto:Token: None 2023-10-18T20:30:30.448 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:30.449 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mB9KA4DPQYBa/jPrHIOSiXwzimU= 2023-10-18T20:30:30.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mB9KA4DPQYBa/jPrHIOSiXwzimU='} 2023-10-18T20:30:30.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8bceb20b2f0f749-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.495 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.496 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.496 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.496 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.497 DEBUG:boto:Data: 2023-10-18T20:30:30.497 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.497 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.498 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.498 DEBUG:boto:Params: {} 2023-10-18T20:30:30.499 DEBUG:boto:Token: None 2023-10-18T20:30:30.499 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.500 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064a18680be86521b-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.504 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:30.504 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:30.505 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.505 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:30.506 DEBUG:boto:Data: 2023-10-18T20:30:30.506 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.506 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.507 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.507 DEBUG:boto:Params: {} 2023-10-18T20:30:30.508 DEBUG:boto:Token: None 2023-10-18T20:30:30.508 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:30.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:x7at5D9ZH/qoaaFcuM4quxSafsg= 2023-10-18T20:30:30.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x7at5D9ZH/qoaaFcuM4quxSafsg='} 2023-10-18T20:30:30.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6dca73f25615426-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.516 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.516 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.517 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.517 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.518 DEBUG:boto:Data: 2023-10-18T20:30:30.518 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.519 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.519 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.519 DEBUG:boto:Params: {} 2023-10-18T20:30:30.520 DEBUG:boto:Token: None 2023-10-18T20:30:30.520 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.521 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020d5fafafd49a288-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.525 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:30.525 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:30.526 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.526 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:30.527 DEBUG:boto:Data: 2023-10-18T20:30:30.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.528 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.528 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.528 DEBUG:boto:Params: {} 2023-10-18T20:30:30.529 DEBUG:boto:Token: None 2023-10-18T20:30:30.529 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:30.530 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dyva8r5HfmVXgnZKbKkr0n9VgHI= 2023-10-18T20:30:30.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dyva8r5HfmVXgnZKbKkr0n9VgHI='} 2023-10-18T20:30:30.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca1a1725c5664121-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.575 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.576 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.576 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.576 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.577 DEBUG:boto:Data: 2023-10-18T20:30:30.577 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.578 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.578 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.579 DEBUG:boto:Params: {} 2023-10-18T20:30:30.579 DEBUG:boto:Token: None 2023-10-18T20:30:30.580 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.580 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005defbd3755217f7e-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.585 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:30.585 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:30.585 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.586 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:30.586 DEBUG:boto:Data: 2023-10-18T20:30:30.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.587 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.587 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.588 DEBUG:boto:Params: {} 2023-10-18T20:30:30.588 DEBUG:boto:Token: None 2023-10-18T20:30:30.589 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:30.589 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2FMN6tfF0ucGtGP2t+cDB9EEiS0= 2023-10-18T20:30:30.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2FMN6tfF0ucGtGP2t+cDB9EEiS0='} 2023-10-18T20:30:30.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe9ac2a8ca5c9725-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.596 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.597 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.597 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.598 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.598 DEBUG:boto:Data: 2023-10-18T20:30:30.598 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.599 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.599 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.600 DEBUG:boto:Params: {} 2023-10-18T20:30:30.600 DEBUG:boto:Token: None 2023-10-18T20:30:30.601 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.601 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.604 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9540923d2fb7fd1-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.605 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:30.605 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:30.606 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.606 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:30.607 DEBUG:boto:Data: 2023-10-18T20:30:30.607 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.608 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.608 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.608 DEBUG:boto:Params: {} 2023-10-18T20:30:30.609 DEBUG:boto:Token: None 2023-10-18T20:30:30.609 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:30.610 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Og82/PCogcR8n6Sj7pYoFx+J69A= 2023-10-18T20:30:30.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Og82/PCogcR8n6Sj7pYoFx+J69A='} 2023-10-18T20:30:30.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e3008e68e718adc-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.656 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.657 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.657 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.657 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.658 DEBUG:boto:Data: 2023-10-18T20:30:30.658 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.659 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.659 DEBUG:boto:Params: {} 2023-10-18T20:30:30.660 DEBUG:boto:Token: None 2023-10-18T20:30:30.660 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.661 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035ad7f07cc288aab-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.666 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:30.666 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:30.667 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.667 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:30.667 DEBUG:boto:Data: 2023-10-18T20:30:30.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.668 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.669 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.669 DEBUG:boto:Params: {} 2023-10-18T20:30:30.669 DEBUG:boto:Token: None 2023-10-18T20:30:30.670 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:30.670 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+B0CpP5hJgO+B/ev4QM3uHtXogE= 2023-10-18T20:30:30.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+B0CpP5hJgO+B/ev4QM3uHtXogE='} 2023-10-18T20:30:30.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e03c742240eaaff8-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.678 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.678 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.679 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.679 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.679 DEBUG:boto:Data: 2023-10-18T20:30:30.680 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.680 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.681 DEBUG:boto:Params: {} 2023-10-18T20:30:30.681 DEBUG:boto:Token: None 2023-10-18T20:30:30.682 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.682 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031c2cb96107d51b2-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.686 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:30.687 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:30.687 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.688 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:30.688 DEBUG:boto:Data: 2023-10-18T20:30:30.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.689 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.690 DEBUG:boto:Params: {} 2023-10-18T20:30:30.690 DEBUG:boto:Token: None 2023-10-18T20:30:30.691 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:30.691 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gw2oZxy2yPqFR/vKLC9Ign0gR8Q= 2023-10-18T20:30:30.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gw2oZxy2yPqFR/vKLC9Ign0gR8Q='} 2023-10-18T20:30:30.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004050f48004857591-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.698 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.699 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.699 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.700 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.700 DEBUG:boto:Data: 2023-10-18T20:30:30.700 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.701 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.701 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.701 DEBUG:boto:Params: {} 2023-10-18T20:30:30.702 DEBUG:boto:Token: None 2023-10-18T20:30:30.702 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.703 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005485b09658036c4c-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.707 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:30.708 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:30.708 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.708 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:30.709 DEBUG:boto:Data: 2023-10-18T20:30:30.709 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.710 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.710 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.710 DEBUG:boto:Params: {} 2023-10-18T20:30:30.711 DEBUG:boto:Token: None 2023-10-18T20:30:30.711 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:30.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O/jjEGRDUrPn4nCAmjPKct2IvXU= 2023-10-18T20:30:30.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/jjEGRDUrPn4nCAmjPKct2IvXU='} 2023-10-18T20:30:30.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3d41735c332c161-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.755 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.756 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.756 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.757 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.757 DEBUG:boto:Data: 2023-10-18T20:30:30.757 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.758 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.758 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.759 DEBUG:boto:Params: {} 2023-10-18T20:30:30.759 DEBUG:boto:Token: None 2023-10-18T20:30:30.759 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.760 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001187d7e194bd1f50-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.765 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:30.765 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:30.765 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.766 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:30.766 DEBUG:boto:Data: 2023-10-18T20:30:30.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.767 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.767 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.768 DEBUG:boto:Params: {} 2023-10-18T20:30:30.768 DEBUG:boto:Token: None 2023-10-18T20:30:30.769 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:30.769 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T+Vh8PzfKCZcuDugepYHVA3T9eM= 2023-10-18T20:30:30.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T+Vh8PzfKCZcuDugepYHVA3T9eM='} 2023-10-18T20:30:30.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bbf20f7f31f1fdb-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.776 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.777 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.777 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.778 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.778 DEBUG:boto:Data: 2023-10-18T20:30:30.779 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.779 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.780 DEBUG:boto:Params: {} 2023-10-18T20:30:30.780 DEBUG:boto:Token: None 2023-10-18T20:30:30.781 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.781 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019ce992e2e1b6f10-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.785 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:30.786 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:30.786 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.787 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:30.787 DEBUG:boto:Data: 2023-10-18T20:30:30.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.788 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.788 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.788 DEBUG:boto:Params: {} 2023-10-18T20:30:30.789 DEBUG:boto:Token: None 2023-10-18T20:30:30.789 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:30.790 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RjtXJdkuklMnlDPqawFmP1YSZPk= 2023-10-18T20:30:30.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RjtXJdkuklMnlDPqawFmP1YSZPk='} 2023-10-18T20:30:30.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b51034a9a1b14d2-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.835 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.835 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.836 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.836 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.837 DEBUG:boto:Data: 2023-10-18T20:30:30.837 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.838 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.838 DEBUG:boto:Params: {} 2023-10-18T20:30:30.839 DEBUG:boto:Token: None 2023-10-18T20:30:30.839 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-18T20:30:30.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8b1f584541744f9-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.844 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:30.845 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:30.845 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.845 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:30.846 DEBUG:boto:Data: 2023-10-18T20:30:30.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.847 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.847 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.847 DEBUG:boto:Params: {} 2023-10-18T20:30:30.848 DEBUG:boto:Token: None 2023-10-18T20:30:30.848 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:30.849 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1+77ritQr0ZzEXL3P9g36vQ5cWU= 2023-10-18T20:30:30.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1+77ritQr0ZzEXL3P9g36vQ5cWU='} 2023-10-18T20:30:30.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6278d09e49daaec-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.854 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.855 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.855 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.856 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.856 DEBUG:boto:Data: 2023-10-18T20:30:30.856 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.857 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.857 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.858 DEBUG:boto:Params: {} 2023-10-18T20:30:30.858 DEBUG:boto:Token: None 2023-10-18T20:30:30.859 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.862 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086047a2987d0e181-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.863 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:30.864 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:30.864 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.864 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:30.865 DEBUG:boto:Data: 2023-10-18T20:30:30.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.866 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.866 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.866 DEBUG:boto:Params: {} 2023-10-18T20:30:30.867 DEBUG:boto:Token: None 2023-10-18T20:30:30.867 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:30.868 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:35ICmUDlkeMOtBHTJYNIFbTLMp8= 2023-10-18T20:30:30.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:35ICmUDlkeMOtBHTJYNIFbTLMp8='} 2023-10-18T20:30:30.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fcf141e4929273f-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.911 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.912 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.912 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.912 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.913 DEBUG:boto:Data: 2023-10-18T20:30:30.913 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.914 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.914 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.915 DEBUG:boto:Params: {} 2023-10-18T20:30:30.915 DEBUG:boto:Token: None 2023-10-18T20:30:30.915 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.916 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b053afa64b15bbaf-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.920 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:30.920 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:30.920 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.921 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:30.921 DEBUG:boto:Data: 2023-10-18T20:30:30.921 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.922 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.922 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.923 DEBUG:boto:Params: {} 2023-10-18T20:30:30.923 DEBUG:boto:Token: None 2023-10-18T20:30:30.924 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:30.924 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C+J4nI3LZQdt9GLbYfFXHW7T1lQ= 2023-10-18T20:30:30.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C+J4nI3LZQdt9GLbYfFXHW7T1lQ='} 2023-10-18T20:30:30.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3063abfafe062cc-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.930 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:30.930 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:30.931 DEBUG:boto:Method: HEAD 2023-10-18T20:30:30.931 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:30.932 DEBUG:boto:Data: 2023-10-18T20:30:30.932 DEBUG:boto:Headers: {} 2023-10-18T20:30:30.932 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.933 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.933 DEBUG:boto:Params: {} 2023-10-18T20:30:30.934 DEBUG:boto:Token: None 2023-10-18T20:30:30.934 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/ 2023-10-18T20:30:30.935 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI= 2023-10-18T20:30:30.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j+9fQIS8drMatKVW7Y2f6aSTHeI=', 'Content-Length': '0'} 2023-10-18T20:30:30.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbe791b6f084258a-0065304066-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:30.958 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:30.958 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:30.959 DEBUG:boto:Method: PUT 2023-10-18T20:30:30.959 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:30.959 DEBUG:boto:Data: 2023-10-18T20:30:30.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:30.960 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:30.961 DEBUG:boto:Port: 8000 2023-10-18T20:30:30.961 DEBUG:boto:Params: {} 2023-10-18T20:30:30.961 DEBUG:boto:Token: None 2023-10-18T20:30:30.962 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:30 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:30.963 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hPD1Rbc3o1MiqOKv04Bo4JI7asY= 2023-10-18T20:30:30.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hPD1Rbc3o1MiqOKv04Bo4JI7asY='} 2023-10-18T20:30:31.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005469d53d8eb77975-0065304066-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.011 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.012 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.012 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.013 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.013 DEBUG:boto:Data: 2023-10-18T20:30:31.013 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.014 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.015 DEBUG:boto:Params: {} 2023-10-18T20:30:31.015 DEBUG:boto:Token: None 2023-10-18T20:30:31.016 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.016 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008142cd880c7e5171-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.021 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:31.021 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:31.021 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.022 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:31.022 DEBUG:boto:Data: 2023-10-18T20:30:31.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.023 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.023 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.024 DEBUG:boto:Params: {} 2023-10-18T20:30:31.024 DEBUG:boto:Token: None 2023-10-18T20:30:31.025 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:31.025 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dJ4DrHucbIOToECmmJjcGV+cTIM= 2023-10-18T20:30:31.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dJ4DrHucbIOToECmmJjcGV+cTIM='} 2023-10-18T20:30:31.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090ffef55e1769122-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.031 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.032 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.032 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.033 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.033 DEBUG:boto:Data: 2023-10-18T20:30:31.033 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.034 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.034 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.035 DEBUG:boto:Params: {} 2023-10-18T20:30:31.035 DEBUG:boto:Token: None 2023-10-18T20:30:31.036 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052a850cf0602a437-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.040 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:31.040 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:31.041 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.041 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:31.042 DEBUG:boto:Data: 2023-10-18T20:30:31.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.043 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.043 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.044 DEBUG:boto:Params: {} 2023-10-18T20:30:31.044 DEBUG:boto:Token: None 2023-10-18T20:30:31.044 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:31.045 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2Ojri9xbA77KbIOPT9V7qs94ZRQ= 2023-10-18T20:30:31.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2Ojri9xbA77KbIOPT9V7qs94ZRQ='} 2023-10-18T20:30:31.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094ec74b40c143c54-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.091 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.092 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.092 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.093 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.093 DEBUG:boto:Data: 2023-10-18T20:30:31.093 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.094 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.094 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.095 DEBUG:boto:Params: {} 2023-10-18T20:30:31.095 DEBUG:boto:Token: None 2023-10-18T20:30:31.096 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.096 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000477839beae4e7c78-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.102 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:31.102 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:31.103 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.103 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:31.104 DEBUG:boto:Data: 2023-10-18T20:30:31.104 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.105 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.105 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.105 DEBUG:boto:Params: {} 2023-10-18T20:30:31.106 DEBUG:boto:Token: None 2023-10-18T20:30:31.106 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:31.107 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E5OtA+29wU/h/MBZtzNRmq87Osw= 2023-10-18T20:30:31.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E5OtA+29wU/h/MBZtzNRmq87Osw='} 2023-10-18T20:30:31.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dab61ffe6b10624-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.113 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.113 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.113 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.114 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.114 DEBUG:boto:Data: 2023-10-18T20:30:31.115 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.115 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.115 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.116 DEBUG:boto:Params: {} 2023-10-18T20:30:31.116 DEBUG:boto:Token: None 2023-10-18T20:30:31.117 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.117 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006152460fbba3db0c-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.121 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:31.121 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:31.122 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.122 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:31.123 DEBUG:boto:Data: 2023-10-18T20:30:31.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.124 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.124 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.124 DEBUG:boto:Params: {} 2023-10-18T20:30:31.125 DEBUG:boto:Token: None 2023-10-18T20:30:31.125 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:31.126 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5svH8zTzMr3fT3VXUaGqcItWIBs= 2023-10-18T20:30:31.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5svH8zTzMr3fT3VXUaGqcItWIBs='} 2023-10-18T20:30:31.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2ab2fd53ef8d8ac-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.133 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.133 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.134 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.134 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.135 DEBUG:boto:Data: 2023-10-18T20:30:31.135 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.135 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.136 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.136 DEBUG:boto:Params: {} 2023-10-18T20:30:31.137 DEBUG:boto:Token: None 2023-10-18T20:30:31.137 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.138 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000b45d900003fdff-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.142 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:31.142 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:31.143 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.143 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:31.143 DEBUG:boto:Data: 2023-10-18T20:30:31.144 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.144 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.145 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.145 DEBUG:boto:Params: {} 2023-10-18T20:30:31.146 DEBUG:boto:Token: None 2023-10-18T20:30:31.146 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:31.146 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RXCYkf493J4/BT8ZHBPyTdl2N6o= 2023-10-18T20:30:31.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RXCYkf493J4/BT8ZHBPyTdl2N6o='} 2023-10-18T20:30:31.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c4e28bb2d701762-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.195 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.196 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.196 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.197 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.197 DEBUG:boto:Data: 2023-10-18T20:30:31.197 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.198 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.198 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.199 DEBUG:boto:Params: {} 2023-10-18T20:30:31.199 DEBUG:boto:Token: None 2023-10-18T20:30:31.200 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.200 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000647ff88a092e2538-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.204 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:31.205 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:31.205 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.206 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:31.206 DEBUG:boto:Data: 2023-10-18T20:30:31.206 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.207 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.207 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.208 DEBUG:boto:Params: {} 2023-10-18T20:30:31.210 DEBUG:boto:Token: None 2023-10-18T20:30:31.211 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:31.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ajuYOc2oyMpBs4R7ZG2i7aOLvHY= 2023-10-18T20:30:31.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ajuYOc2oyMpBs4R7ZG2i7aOLvHY='} 2023-10-18T20:30:31.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001dfad24f041f983c-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.218 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.219 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.219 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.220 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.220 DEBUG:boto:Data: 2023-10-18T20:30:31.221 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.221 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.221 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.222 DEBUG:boto:Params: {} 2023-10-18T20:30:31.222 DEBUG:boto:Token: None 2023-10-18T20:30:31.223 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.223 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a3ac8aec9b10325-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.227 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:31.228 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:31.228 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.229 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:31.229 DEBUG:boto:Data: 2023-10-18T20:30:31.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.230 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.231 DEBUG:boto:Params: {} 2023-10-18T20:30:31.231 DEBUG:boto:Token: None 2023-10-18T20:30:31.232 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:31.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sfmw2wOMJEeqchCYJqY4RKpzV0g= 2023-10-18T20:30:31.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sfmw2wOMJEeqchCYJqY4RKpzV0g='} 2023-10-18T20:30:31.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000781608f5fcf63c87-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.279 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.280 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.280 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.280 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.281 DEBUG:boto:Data: 2023-10-18T20:30:31.281 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.282 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.282 DEBUG:boto:Params: {} 2023-10-18T20:30:31.283 DEBUG:boto:Token: None 2023-10-18T20:30:31.283 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a49f18d07fb293d1-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.288 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:31.289 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:31.289 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.289 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:31.290 DEBUG:boto:Data: 2023-10-18T20:30:31.290 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.291 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.291 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.291 DEBUG:boto:Params: {} 2023-10-18T20:30:31.292 DEBUG:boto:Token: None 2023-10-18T20:30:31.292 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:31.293 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NERPp5AA6ejPE72hM/FjJUUSCt4= 2023-10-18T20:30:31.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NERPp5AA6ejPE72hM/FjJUUSCt4='} 2023-10-18T20:30:31.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094c0b34a9b755728-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:31.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:31.299 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-58 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:30:31.352 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:31.352 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:31.405 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:30:31.406 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-58 2023-10-18T20:30:31.416 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 54 2023-10-18T20:30:31.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:31.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:31.477 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:31.528 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:31.528 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:31.570 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:31.570 INFO:teuthology.orchestra.run.smithi082.stdout: "objs": [ 2023-10-18T20:30:31.570 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000060", 2023-10-18T20:30:31.571 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000000", 2023-10-18T20:30:31.571 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000011", 2023-10-18T20:30:31.571 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000001", 2023-10-18T20:30:31.571 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000033", 2023-10-18T20:30:31.572 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000065", 2023-10-18T20:30:31.572 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000005", 2023-10-18T20:30:31.572 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000016", 2023-10-18T20:30:31.572 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000048", 2023-10-18T20:30:31.572 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000015", 2023-10-18T20:30:31.573 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000026", 2023-10-18T20:30:31.573 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000030", 2023-10-18T20:30:31.573 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000062", 2023-10-18T20:30:31.573 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000009", 2023-10-18T20:30:31.573 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000040", 2023-10-18T20:30:31.574 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000024", 2023-10-18T20:30:31.574 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000012", 2023-10-18T20:30:31.574 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000002", 2023-10-18T20:30:31.574 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000013", 2023-10-18T20:30:31.575 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000045", 2023-10-18T20:30:31.575 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000006", 2023-10-18T20:30:31.575 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000017", 2023-10-18T20:30:31.575 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000007", 2023-10-18T20:30:31.576 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000039", 2023-10-18T20:30:31.576 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000027", 2023-10-18T20:30:31.576 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000010", 2023-10-18T20:30:31.576 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000021", 2023-10-18T20:30:31.576 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000053", 2023-10-18T20:30:31.577 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000020", 2023-10-18T20:30:31.577 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000004", 2023-10-18T20:30:31.577 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000036", 2023-10-18T20:30:31.577 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000003", 2023-10-18T20:30:31.577 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000014", 2023-10-18T20:30:31.578 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000019", 2023-10-18T20:30:31.578 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000057", 2023-10-18T20:30:31.578 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000018", 2023-10-18T20:30:31.578 INFO:teuthology.orchestra.run.smithi082.stdout: "obj00000000000000000008" 2023-10-18T20:30:31.579 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:31.579 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:31.580 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.580 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.581 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.581 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.582 DEBUG:boto:Data: 2023-10-18T20:30:31.582 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.583 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.583 DEBUG:boto:Params: {} 2023-10-18T20:30:31.584 DEBUG:boto:Token: None 2023-10-18T20:30:31.584 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.584 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005034c2852ecba24f-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.589 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:31.589 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:31.590 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.590 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:31.591 DEBUG:boto:Data: 2023-10-18T20:30:31.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.591 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.592 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.592 DEBUG:boto:Params: {} 2023-10-18T20:30:31.593 DEBUG:boto:Token: None 2023-10-18T20:30:31.593 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:31.594 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xL74CgtuQOwIyU7EpJNanrskVzo= 2023-10-18T20:30:31.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xL74CgtuQOwIyU7EpJNanrskVzo='} 2023-10-18T20:30:31.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074e1cb54d45305b4-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.639 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.640 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.640 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.640 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.641 DEBUG:boto:Data: 2023-10-18T20:30:31.641 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.642 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.642 DEBUG:boto:Params: {} 2023-10-18T20:30:31.643 DEBUG:boto:Token: None 2023-10-18T20:30:31.643 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.644 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4efcdd663a9bed2-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.648 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:31.648 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:31.649 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.649 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:31.650 DEBUG:boto:Data: 2023-10-18T20:30:31.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.650 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.651 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.651 DEBUG:boto:Params: {} 2023-10-18T20:30:31.651 DEBUG:boto:Token: None 2023-10-18T20:30:31.652 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:31.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/NpJlaOWxo9vghwQ8BUjC2YHrNY= 2023-10-18T20:30:31.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/NpJlaOWxo9vghwQ8BUjC2YHrNY='} 2023-10-18T20:30:31.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f8bb63d7ca57e77-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.658 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.659 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.659 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.660 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.660 DEBUG:boto:Data: 2023-10-18T20:30:31.660 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.661 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.662 DEBUG:boto:Params: {} 2023-10-18T20:30:31.662 DEBUG:boto:Token: None 2023-10-18T20:30:31.662 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.663 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4fe2d54c2ae0e5a-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.667 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:31.667 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:31.670 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.671 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:31.671 DEBUG:boto:Data: 2023-10-18T20:30:31.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.672 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.672 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.673 DEBUG:boto:Params: {} 2023-10-18T20:30:31.673 DEBUG:boto:Token: None 2023-10-18T20:30:31.674 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:31.674 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lRxc4yyvrsVvdxvp6w6f1P1unwo= 2023-10-18T20:30:31.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lRxc4yyvrsVvdxvp6w6f1P1unwo='} 2023-10-18T20:30:31.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020389b1d47bd51ba-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.719 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.720 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.720 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.720 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.721 DEBUG:boto:Data: 2023-10-18T20:30:31.721 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.722 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.722 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.723 DEBUG:boto:Params: {} 2023-10-18T20:30:31.723 DEBUG:boto:Token: None 2023-10-18T20:30:31.724 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.724 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005380750a5f0790c8-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.729 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:31.729 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:31.730 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.730 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:31.731 DEBUG:boto:Data: 2023-10-18T20:30:31.731 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.732 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.732 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.733 DEBUG:boto:Params: {} 2023-10-18T20:30:31.733 DEBUG:boto:Token: None 2023-10-18T20:30:31.734 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:31.734 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qnoANX4dksjYrD7k5MdIrbkUJGQ= 2023-10-18T20:30:31.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qnoANX4dksjYrD7k5MdIrbkUJGQ='} 2023-10-18T20:30:31.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da7a023b966cc9ef-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.740 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.741 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.741 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.742 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.742 DEBUG:boto:Data: 2023-10-18T20:30:31.743 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.743 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.744 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.744 DEBUG:boto:Params: {} 2023-10-18T20:30:31.745 DEBUG:boto:Token: None 2023-10-18T20:30:31.745 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.746 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008663b1ba5cc5e327-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.751 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:31.751 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:31.752 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.752 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:31.753 DEBUG:boto:Data: 2023-10-18T20:30:31.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.754 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.755 DEBUG:boto:Params: {} 2023-10-18T20:30:31.755 DEBUG:boto:Token: None 2023-10-18T20:30:31.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:31.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:POhEsaAcoaSpsiyrnuIBkaf9Pwo= 2023-10-18T20:30:31.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:POhEsaAcoaSpsiyrnuIBkaf9Pwo='} 2023-10-18T20:30:31.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000204b42a758dd5db1-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.804 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.804 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.805 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.805 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.805 DEBUG:boto:Data: 2023-10-18T20:30:31.806 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.806 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.807 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.807 DEBUG:boto:Params: {} 2023-10-18T20:30:31.808 DEBUG:boto:Token: None 2023-10-18T20:30:31.808 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.809 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090968ddcd61f78bc-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.814 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:31.814 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:31.814 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.815 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:31.815 DEBUG:boto:Data: 2023-10-18T20:30:31.816 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.816 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.817 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.817 DEBUG:boto:Params: {} 2023-10-18T20:30:31.817 DEBUG:boto:Token: None 2023-10-18T20:30:31.818 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:31.818 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pGTzyyfdRI6e7/o9EF8sclfQK+g= 2023-10-18T20:30:31.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pGTzyyfdRI6e7/o9EF8sclfQK+g='} 2023-10-18T20:30:31.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005af3dd8614db4045-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.825 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.825 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.826 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.826 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.826 DEBUG:boto:Data: 2023-10-18T20:30:31.827 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.827 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.828 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.828 DEBUG:boto:Params: {} 2023-10-18T20:30:31.828 DEBUG:boto:Token: None 2023-10-18T20:30:31.829 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.829 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a47aeee473cab174-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.834 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:31.834 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:31.835 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.835 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:31.836 DEBUG:boto:Data: 2023-10-18T20:30:31.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.836 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.837 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.837 DEBUG:boto:Params: {} 2023-10-18T20:30:31.838 DEBUG:boto:Token: None 2023-10-18T20:30:31.838 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:31.839 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xgHhc0sgSkbJsfKaChrzIvxCvmU= 2023-10-18T20:30:31.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xgHhc0sgSkbJsfKaChrzIvxCvmU='} 2023-10-18T20:30:31.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ce6bde96de2c7e2-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.846 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.846 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.847 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.847 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.848 DEBUG:boto:Data: 2023-10-18T20:30:31.848 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.849 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.849 DEBUG:boto:Params: {} 2023-10-18T20:30:31.850 DEBUG:boto:Token: None 2023-10-18T20:30:31.850 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.851 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b6419b6a41871a5-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.855 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:31.855 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:31.856 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.856 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:31.857 DEBUG:boto:Data: 2023-10-18T20:30:31.857 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.858 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.858 DEBUG:boto:Params: {} 2023-10-18T20:30:31.859 DEBUG:boto:Token: None 2023-10-18T20:30:31.859 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:31.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TVrh9gC2wz8bvIBuzcuIwG1aYIs= 2023-10-18T20:30:31.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TVrh9gC2wz8bvIBuzcuIwG1aYIs='} 2023-10-18T20:30:31.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7c1fc5a62380b56-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.903 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.904 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.904 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.905 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.905 DEBUG:boto:Data: 2023-10-18T20:30:31.906 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.906 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.906 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.907 DEBUG:boto:Params: {} 2023-10-18T20:30:31.907 DEBUG:boto:Token: None 2023-10-18T20:30:31.908 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.908 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a5ffee53f3188bb-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.913 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:31.913 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:31.914 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.914 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:31.914 DEBUG:boto:Data: 2023-10-18T20:30:31.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.916 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.916 DEBUG:boto:Params: {} 2023-10-18T20:30:31.917 DEBUG:boto:Token: None 2023-10-18T20:30:31.917 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:31.918 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5svH8zTzMr3fT3VXUaGqcItWIBs= 2023-10-18T20:30:31.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5svH8zTzMr3fT3VXUaGqcItWIBs='} 2023-10-18T20:30:31.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a4df59d7f38f4a8-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.925 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.925 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.926 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.926 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.927 DEBUG:boto:Data: 2023-10-18T20:30:31.927 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.927 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.928 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.928 DEBUG:boto:Params: {} 2023-10-18T20:30:31.929 DEBUG:boto:Token: None 2023-10-18T20:30:31.929 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fef6bcbf75ec6584-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.934 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:31.934 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:31.935 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.935 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:31.936 DEBUG:boto:Data: 2023-10-18T20:30:31.936 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.936 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.937 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.937 DEBUG:boto:Params: {} 2023-10-18T20:30:31.938 DEBUG:boto:Token: None 2023-10-18T20:30:31.938 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:31.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yiS5rF2/s/a9teUtDDDKMMc2hZU= 2023-10-18T20:30:31.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yiS5rF2/s/a9teUtDDDKMMc2hZU='} 2023-10-18T20:30:31.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056be41916e8dc821-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.983 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:31.985 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:31.985 DEBUG:boto:Method: HEAD 2023-10-18T20:30:31.986 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:31.986 DEBUG:boto:Data: 2023-10-18T20:30:31.987 DEBUG:boto:Headers: {} 2023-10-18T20:30:31.987 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.987 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.988 DEBUG:boto:Params: {} 2023-10-18T20:30:31.988 DEBUG:boto:Token: None 2023-10-18T20:30:31.989 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/ 2023-10-18T20:30:31.989 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ= 2023-10-18T20:30:31.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:enVyqsKq/mpkxXwH5+YKd/eZ8wQ=', 'Content-Length': '0'} 2023-10-18T20:30:31.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a25df4d70399e8f-0065304067-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:31.994 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:31.995 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:31.995 DEBUG:boto:Method: PUT 2023-10-18T20:30:31.995 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:31.996 DEBUG:boto:Data: 2023-10-18T20:30:31.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:31.997 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:31.997 DEBUG:boto:Port: 8000 2023-10-18T20:30:31.997 DEBUG:boto:Params: {} 2023-10-18T20:30:31.998 DEBUG:boto:Token: None 2023-10-18T20:30:31.998 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:31 GMT /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:31.999 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KT8fAVvqNJxUgycyutUFwXCnWxg= 2023-10-18T20:30:31.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KT8fAVvqNJxUgycyutUFwXCnWxg='} 2023-10-18T20:30:32.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001814104eba257e2f-0065304067-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.005 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.005 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.006 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.006 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.006 DEBUG:boto:Data: 2023-10-18T20:30:32.007 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.008 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.008 DEBUG:boto:Params: {} 2023-10-18T20:30:32.009 DEBUG:boto:Token: None 2023-10-18T20:30:32.009 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002663006341d17049-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.014 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:32.014 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:32.015 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.015 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:32.015 DEBUG:boto:Data: 2023-10-18T20:30:32.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.016 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.017 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.017 DEBUG:boto:Params: {} 2023-10-18T20:30:32.017 DEBUG:boto:Token: None 2023-10-18T20:30:32.018 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:32.018 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/iOv7jF+v2lfOYNz7+t0NVTBWTQ= 2023-10-18T20:30:32.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/iOv7jF+v2lfOYNz7+t0NVTBWTQ='} 2023-10-18T20:30:32.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac72317e4858867e-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.025 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.026 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.026 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.027 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.027 DEBUG:boto:Data: 2023-10-18T20:30:32.028 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.028 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.028 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.029 DEBUG:boto:Params: {} 2023-10-18T20:30:32.029 DEBUG:boto:Token: None 2023-10-18T20:30:32.030 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fe3b2df5c5bce1e-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.035 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:32.035 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:32.036 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.036 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:32.036 DEBUG:boto:Data: 2023-10-18T20:30:32.037 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.037 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.038 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.038 DEBUG:boto:Params: {} 2023-10-18T20:30:32.039 DEBUG:boto:Token: None 2023-10-18T20:30:32.039 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:32.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pJGzLrwNq9lKRbDALSwixbSZiYs= 2023-10-18T20:30:32.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pJGzLrwNq9lKRbDALSwixbSZiYs='} 2023-10-18T20:30:32.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027ab43031acc93da-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.083 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.084 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.085 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.085 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.085 DEBUG:boto:Data: 2023-10-18T20:30:32.086 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.086 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.087 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.087 DEBUG:boto:Params: {} 2023-10-18T20:30:32.088 DEBUG:boto:Token: None 2023-10-18T20:30:32.088 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076cf9d57958903e7-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.094 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:32.095 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:32.095 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.096 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:32.097 DEBUG:boto:Data: 2023-10-18T20:30:32.097 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.099 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.099 DEBUG:boto:Params: {} 2023-10-18T20:30:32.100 DEBUG:boto:Token: None 2023-10-18T20:30:32.101 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:32.101 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sXpjuHeRRedu6u3ZImm/qZJSuC0= 2023-10-18T20:30:32.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sXpjuHeRRedu6u3ZImm/qZJSuC0='} 2023-10-18T20:30:32.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075f8f29dfe1ebcec-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.109 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.109 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.110 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.110 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.111 DEBUG:boto:Data: 2023-10-18T20:30:32.111 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.112 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.112 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.113 DEBUG:boto:Params: {} 2023-10-18T20:30:32.113 DEBUG:boto:Token: None 2023-10-18T20:30:32.113 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.114 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be8d2dddd34aacd1-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.118 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:32.119 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:32.119 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.120 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:32.120 DEBUG:boto:Data: 2023-10-18T20:30:32.121 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.123 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.123 DEBUG:boto:Params: {} 2023-10-18T20:30:32.124 DEBUG:boto:Token: None 2023-10-18T20:30:32.125 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:32.126 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y+L+QZ7PaYJ9kiPQ/I1hLu4+ZQI= 2023-10-18T20:30:32.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y+L+QZ7PaYJ9kiPQ/I1hLu4+ZQI='} 2023-10-18T20:30:32.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c05337102d3e7063-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.135 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.135 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.136 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.136 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.137 DEBUG:boto:Data: 2023-10-18T20:30:32.137 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.138 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.138 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.138 DEBUG:boto:Params: {} 2023-10-18T20:30:32.139 DEBUG:boto:Token: None 2023-10-18T20:30:32.139 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.140 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a95b01d0863ef2ac-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.145 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:32.145 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:32.146 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.146 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:32.146 DEBUG:boto:Data: 2023-10-18T20:30:32.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.147 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.148 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.148 DEBUG:boto:Params: {} 2023-10-18T20:30:32.149 DEBUG:boto:Token: None 2023-10-18T20:30:32.149 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:32.149 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FuojyZU9gvRV9Fd/aKI0o69lTTM= 2023-10-18T20:30:32.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FuojyZU9gvRV9Fd/aKI0o69lTTM='} 2023-10-18T20:30:32.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a646ced8cc534bd3-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.195 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.196 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.196 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.197 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.197 DEBUG:boto:Data: 2023-10-18T20:30:32.197 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.198 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.198 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.199 DEBUG:boto:Params: {} 2023-10-18T20:30:32.199 DEBUG:boto:Token: None 2023-10-18T20:30:32.200 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.200 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058cb46e6300bd3ce-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.205 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:32.205 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:32.206 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.206 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:32.207 DEBUG:boto:Data: 2023-10-18T20:30:32.207 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.208 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.208 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.209 DEBUG:boto:Params: {} 2023-10-18T20:30:32.209 DEBUG:boto:Token: None 2023-10-18T20:30:32.210 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:32.210 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qZWbYTL5pieQyO6ZghyaFFcCKBM= 2023-10-18T20:30:32.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qZWbYTL5pieQyO6ZghyaFFcCKBM='} 2023-10-18T20:30:32.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a6e35a92952b343-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.217 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.218 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.218 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.219 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.219 DEBUG:boto:Data: 2023-10-18T20:30:32.219 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.220 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.220 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.220 DEBUG:boto:Params: {} 2023-10-18T20:30:32.221 DEBUG:boto:Token: None 2023-10-18T20:30:32.221 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.222 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.225 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078dfd39a79fcfe63-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.226 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:32.227 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:32.227 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.228 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:32.228 DEBUG:boto:Data: 2023-10-18T20:30:32.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.229 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.229 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.230 DEBUG:boto:Params: {} 2023-10-18T20:30:32.230 DEBUG:boto:Token: None 2023-10-18T20:30:32.230 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:32.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:x2Hs0w+IrzkiEzSqq4TqPKW5/Jc= 2023-10-18T20:30:32.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x2Hs0w+IrzkiEzSqq4TqPKW5/Jc='} 2023-10-18T20:30:32.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a714e6997e0ded71-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.279 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.280 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.280 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.280 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.281 DEBUG:boto:Data: 2023-10-18T20:30:32.281 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.281 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.282 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.282 DEBUG:boto:Params: {} 2023-10-18T20:30:32.283 DEBUG:boto:Token: None 2023-10-18T20:30:32.283 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f14a078c21469c1a-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.288 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:32.289 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:32.289 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.290 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:32.290 DEBUG:boto:Data: 2023-10-18T20:30:32.290 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.291 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.291 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.292 DEBUG:boto:Params: {} 2023-10-18T20:30:32.292 DEBUG:boto:Token: None 2023-10-18T20:30:32.293 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:32.293 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D5L/6tdtvm/fwblhLiv6SOjLvAQ= 2023-10-18T20:30:32.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D5L/6tdtvm/fwblhLiv6SOjLvAQ='} 2023-10-18T20:30:32.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfc0289b42562153-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.300 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.300 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.301 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.301 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.302 DEBUG:boto:Data: 2023-10-18T20:30:32.302 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.303 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.303 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.304 DEBUG:boto:Params: {} 2023-10-18T20:30:32.304 DEBUG:boto:Token: None 2023-10-18T20:30:32.305 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.309 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbb99bf3de7851fd-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.310 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:32.310 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:32.311 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.311 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:32.312 DEBUG:boto:Data: 2023-10-18T20:30:32.312 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.313 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.313 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.314 DEBUG:boto:Params: {} 2023-10-18T20:30:32.314 DEBUG:boto:Token: None 2023-10-18T20:30:32.315 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:32.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:69vyyfDaUb0hIm1lDxjhudFvYec= 2023-10-18T20:30:32.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:69vyyfDaUb0hIm1lDxjhudFvYec='} 2023-10-18T20:30:32.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d01d991b4dfae24-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.359 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.360 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.360 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.361 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.361 DEBUG:boto:Data: 2023-10-18T20:30:32.361 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.362 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.362 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.363 DEBUG:boto:Params: {} 2023-10-18T20:30:32.363 DEBUG:boto:Token: None 2023-10-18T20:30:32.364 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000accf64de9493c243-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.369 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:32.369 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:32.369 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.370 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:32.370 DEBUG:boto:Data: 2023-10-18T20:30:32.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.371 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.371 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.372 DEBUG:boto:Params: {} 2023-10-18T20:30:32.372 DEBUG:boto:Token: None 2023-10-18T20:30:32.373 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:32.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SRgWcdKncoFCIayKEveapmzytvw= 2023-10-18T20:30:32.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SRgWcdKncoFCIayKEveapmzytvw='} 2023-10-18T20:30:32.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fce92dffedf5218e-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.381 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.382 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.383 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.383 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.383 DEBUG:boto:Data: 2023-10-18T20:30:32.384 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.384 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.385 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.385 DEBUG:boto:Params: {} 2023-10-18T20:30:32.385 DEBUG:boto:Token: None 2023-10-18T20:30:32.386 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.386 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033318aa22643b0a2-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.391 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:32.392 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:32.393 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.394 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:32.394 DEBUG:boto:Data: 2023-10-18T20:30:32.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.396 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.397 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.397 DEBUG:boto:Params: {} 2023-10-18T20:30:32.398 DEBUG:boto:Token: None 2023-10-18T20:30:32.398 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:32.399 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GQgHHFfbaiMXTQWGakHVlQyiLbk= 2023-10-18T20:30:32.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GQgHHFfbaiMXTQWGakHVlQyiLbk='} 2023-10-18T20:30:32.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004667778d4c170491-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.448 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.448 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.449 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.449 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.450 DEBUG:boto:Data: 2023-10-18T20:30:32.450 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.451 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.452 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.452 DEBUG:boto:Params: {} 2023-10-18T20:30:32.453 DEBUG:boto:Token: None 2023-10-18T20:30:32.453 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.454 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dc9ee48715067d9-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.458 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:32.459 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:32.459 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.460 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:32.460 DEBUG:boto:Data: 2023-10-18T20:30:32.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.461 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.461 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.462 DEBUG:boto:Params: {} 2023-10-18T20:30:32.462 DEBUG:boto:Token: None 2023-10-18T20:30:32.463 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:32.463 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KQYTGouTFX/zqxvFxJc4tjo/lFA= 2023-10-18T20:30:32.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KQYTGouTFX/zqxvFxJc4tjo/lFA='} 2023-10-18T20:30:32.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007e988794a57a7bd-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.471 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.471 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.471 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.472 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.472 DEBUG:boto:Data: 2023-10-18T20:30:32.472 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.473 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.473 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.474 DEBUG:boto:Params: {} 2023-10-18T20:30:32.474 DEBUG:boto:Token: None 2023-10-18T20:30:32.475 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.475 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009988d657e73f6677-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.480 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:32.481 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:32.481 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.482 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:32.483 DEBUG:boto:Data: 2023-10-18T20:30:32.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.484 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.485 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.485 DEBUG:boto:Params: {} 2023-10-18T20:30:32.485 DEBUG:boto:Token: None 2023-10-18T20:30:32.486 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:32.487 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L3ti3/jYuvE0mI6EK+Mh2DTfT/A= 2023-10-18T20:30:32.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L3ti3/jYuvE0mI6EK+Mh2DTfT/A='} 2023-10-18T20:30:32.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000045eb53b621dda2ec-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.535 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.536 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.536 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.537 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.537 DEBUG:boto:Data: 2023-10-18T20:30:32.538 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.538 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.539 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.539 DEBUG:boto:Params: {} 2023-10-18T20:30:32.540 DEBUG:boto:Token: None 2023-10-18T20:30:32.540 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.541 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088407ec8e387d867-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.546 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:32.546 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:32.547 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.547 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:32.547 DEBUG:boto:Data: 2023-10-18T20:30:32.548 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.548 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.549 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.549 DEBUG:boto:Params: {} 2023-10-18T20:30:32.549 DEBUG:boto:Token: None 2023-10-18T20:30:32.550 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:32.550 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aV8ZDJnWu/N1fs/4pNfxerJn4Ls= 2023-10-18T20:30:32.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aV8ZDJnWu/N1fs/4pNfxerJn4Ls='} 2023-10-18T20:30:32.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d50713ddb339c32c-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.557 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.558 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.558 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.559 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.559 DEBUG:boto:Data: 2023-10-18T20:30:32.559 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.560 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.560 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.561 DEBUG:boto:Params: {} 2023-10-18T20:30:32.561 DEBUG:boto:Token: None 2023-10-18T20:30:32.562 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.562 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa3eecf18973e521-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.566 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:32.567 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:32.567 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.568 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:32.569 DEBUG:boto:Data: 2023-10-18T20:30:32.569 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.570 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.571 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.571 DEBUG:boto:Params: {} 2023-10-18T20:30:32.572 DEBUG:boto:Token: None 2023-10-18T20:30:32.573 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:32.573 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QQhdNL6vjcQM9MqyazuUyyJhwQk= 2023-10-18T20:30:32.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QQhdNL6vjcQM9MqyazuUyyJhwQk='} 2023-10-18T20:30:32.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e46715fa460f04ae-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.619 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.620 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.620 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.621 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.621 DEBUG:boto:Data: 2023-10-18T20:30:32.622 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.622 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.622 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.623 DEBUG:boto:Params: {} 2023-10-18T20:30:32.623 DEBUG:boto:Token: None 2023-10-18T20:30:32.624 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ae7a78471afe91b-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.628 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:32.629 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:32.629 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.630 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:32.630 DEBUG:boto:Data: 2023-10-18T20:30:32.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.631 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.632 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.632 DEBUG:boto:Params: {} 2023-10-18T20:30:32.633 DEBUG:boto:Token: None 2023-10-18T20:30:32.634 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:32.635 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XQplGcAjZJMW1+PMVTemkwCGCZw= 2023-10-18T20:30:32.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XQplGcAjZJMW1+PMVTemkwCGCZw='} 2023-10-18T20:30:32.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c026a9cb54341e4-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.641 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.642 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.642 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.643 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.643 DEBUG:boto:Data: 2023-10-18T20:30:32.644 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.644 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.644 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.645 DEBUG:boto:Params: {} 2023-10-18T20:30:32.645 DEBUG:boto:Token: None 2023-10-18T20:30:32.646 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.646 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008ccdf3a6173c116-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.651 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:32.652 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:32.652 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.653 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:32.654 DEBUG:boto:Data: 2023-10-18T20:30:32.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.655 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.656 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.657 DEBUG:boto:Params: {} 2023-10-18T20:30:32.657 DEBUG:boto:Token: None 2023-10-18T20:30:32.658 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:32.659 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fCU5Ac9RzbeGE6no4TsqQizsv7s= 2023-10-18T20:30:32.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fCU5Ac9RzbeGE6no4TsqQizsv7s='} 2023-10-18T20:30:32.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac3a5cfba676510c-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.704 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.704 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.704 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.705 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.705 DEBUG:boto:Data: 2023-10-18T20:30:32.706 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.707 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.707 DEBUG:boto:Params: {} 2023-10-18T20:30:32.708 DEBUG:boto:Token: None 2023-10-18T20:30:32.708 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd96838e72d7560f-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.714 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:32.714 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:32.715 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.715 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:32.716 DEBUG:boto:Data: 2023-10-18T20:30:32.716 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.716 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.717 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.717 DEBUG:boto:Params: {} 2023-10-18T20:30:32.718 DEBUG:boto:Token: None 2023-10-18T20:30:32.718 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:32.719 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:264hDvdsoeOlNoEVAzrtsBo0/TM= 2023-10-18T20:30:32.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:264hDvdsoeOlNoEVAzrtsBo0/TM='} 2023-10-18T20:30:32.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078988124662cc5fe-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.726 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.726 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.727 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.727 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.728 DEBUG:boto:Data: 2023-10-18T20:30:32.728 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.729 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.729 DEBUG:boto:Params: {} 2023-10-18T20:30:32.730 DEBUG:boto:Token: None 2023-10-18T20:30:32.730 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.731 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030b7f16a35b5bbe8-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.735 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:32.736 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:32.736 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.737 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:32.737 DEBUG:boto:Data: 2023-10-18T20:30:32.737 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.738 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.739 DEBUG:boto:Params: {} 2023-10-18T20:30:32.739 DEBUG:boto:Token: None 2023-10-18T20:30:32.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:32.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:12D0UE46TsqMd77pU/rArFWQj0I= 2023-10-18T20:30:32.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:12D0UE46TsqMd77pU/rArFWQj0I='} 2023-10-18T20:30:32.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027b76e435872d466-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.783 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.784 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.784 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.785 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.785 DEBUG:boto:Data: 2023-10-18T20:30:32.785 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.786 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.786 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.787 DEBUG:boto:Params: {} 2023-10-18T20:30:32.787 DEBUG:boto:Token: None 2023-10-18T20:30:32.788 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.788 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.792 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fffff7e77bdcf824-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.793 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:32.793 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:32.794 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.794 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:32.794 DEBUG:boto:Data: 2023-10-18T20:30:32.795 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.795 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.796 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.796 DEBUG:boto:Params: {} 2023-10-18T20:30:32.796 DEBUG:boto:Token: None 2023-10-18T20:30:32.797 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:32.797 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:niHcQj9kcfK+lxnAE3LJaxqvku8= 2023-10-18T20:30:32.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:niHcQj9kcfK+lxnAE3LJaxqvku8='} 2023-10-18T20:30:32.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7b1c39cd07f501e-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.805 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.805 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.806 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.806 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.807 DEBUG:boto:Data: 2023-10-18T20:30:32.807 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.807 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.808 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.808 DEBUG:boto:Params: {} 2023-10-18T20:30:32.809 DEBUG:boto:Token: None 2023-10-18T20:30:32.809 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.810 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdf086067c1ee165-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.814 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:32.814 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:32.815 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.815 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:32.816 DEBUG:boto:Data: 2023-10-18T20:30:32.816 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.817 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.817 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.817 DEBUG:boto:Params: {} 2023-10-18T20:30:32.818 DEBUG:boto:Token: None 2023-10-18T20:30:32.818 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:32.819 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r9biVRT8q4ucJcvbRudbFG54NgM= 2023-10-18T20:30:32.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r9biVRT8q4ucJcvbRudbFG54NgM='} 2023-10-18T20:30:32.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000678d33e1305ed47f-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.863 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.864 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.864 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.864 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.865 DEBUG:boto:Data: 2023-10-18T20:30:32.865 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.866 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.866 DEBUG:boto:Params: {} 2023-10-18T20:30:32.867 DEBUG:boto:Token: None 2023-10-18T20:30:32.868 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.868 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009488bcf7e83beb96-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.873 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:32.874 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:32.874 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.874 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:32.875 DEBUG:boto:Data: 2023-10-18T20:30:32.876 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.876 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.877 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.878 DEBUG:boto:Params: {} 2023-10-18T20:30:32.878 DEBUG:boto:Token: None 2023-10-18T20:30:32.879 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:32.879 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xQUiJYoLD7ZhC6T+LyiJRkelAtA= 2023-10-18T20:30:32.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xQUiJYoLD7ZhC6T+LyiJRkelAtA='} 2023-10-18T20:30:32.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9cc682bae671a6e-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.886 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:32.887 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:32.887 DEBUG:boto:Method: HEAD 2023-10-18T20:30:32.887 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:32.888 DEBUG:boto:Data: 2023-10-18T20:30:32.888 DEBUG:boto:Headers: {} 2023-10-18T20:30:32.889 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.889 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.889 DEBUG:boto:Params: {} 2023-10-18T20:30:32.890 DEBUG:boto:Token: None 2023-10-18T20:30:32.890 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/ 2023-10-18T20:30:32.891 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk= 2023-10-18T20:30:32.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qkPN/SVvBfrQxmG0cLwV5dEyaCk=', 'Content-Length': '0'} 2023-10-18T20:30:32.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba51a4bba2e89ea6-0065304068-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.895 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:32.896 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:32.896 DEBUG:boto:Method: PUT 2023-10-18T20:30:32.896 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:32.897 DEBUG:boto:Data: 2023-10-18T20:30:32.897 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:30:32.897 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:32.898 DEBUG:boto:Port: 8000 2023-10-18T20:30:32.898 DEBUG:boto:Params: {} 2023-10-18T20:30:32.899 DEBUG:boto:Token: None 2023-10-18T20:30:32.899 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:30:32 GMT /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:32.900 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9erIT2NyRJl7eDmR3ycHnKmORM= 2023-10-18T20:30:32.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:30:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9erIT2NyRJl7eDmR3ycHnKmORM='} 2023-10-18T20:30:32.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff323f047ac95cb3-0065304068-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:32.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:32.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:30:32.948 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-58 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:30:33.000 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:33.001 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:33.065 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.062+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.066 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.062+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.066 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.063+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket shard: 2 2023-10-18T20:30:33.066 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.063+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position=00000000001.46.6 2023-10-18T20:30:33.066 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.063+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.067 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.063+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket shard: 2 2023-10-18T20:30:33.067 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.063+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position=00000000001.36.6 2023-10-18T20:30:33.067 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.063+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.067 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.063+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.067 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.063+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket shard: 2 2023-10-18T20:30:33.068 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.063+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position=00000000001.36.6 2023-10-18T20:30:33.069 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.064+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.069 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.064+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.069 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.064+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket shard: 2 2023-10-18T20:30:33.069 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.064+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-18T20:30:33.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.070 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket shard: 2 2023-10-18T20:30:33.071 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket shard: 2 2023-10-18T20:30:33.071 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket shard: 2 2023-10-18T20:30:33.071 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position=00000000001.75.6 2023-10-18T20:30:33.071 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-18T20:30:33.071 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.065+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position=00000000001.62.6 2023-10-18T20:30:33.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.066+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket shard: 2 2023-10-18T20:30:33.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.066+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position=00000000001.38.6 2023-10-18T20:30:33.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.066+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket shard: 2 2023-10-18T20:30:33.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.066+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 2 2023-10-18T20:30:33.072 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.066+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket shard: 2 2023-10-18T20:30:33.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.066+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position=00000000001.30.6 2023-10-18T20:30:33.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.066+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.066+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position=00000000001.36.6 2023-10-18T20:30:33.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.115+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.115+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.115+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.115+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.115+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket shard: 0 2023-10-18T20:30:33.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.115+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket shard: 0 2023-10-18T20:30:33.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.115+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.115+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.116+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket shard: 0 2023-10-18T20:30:33.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.116+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.116+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.116+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket shard: 0 2023-10-18T20:30:33.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.116+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket shard: 0 2023-10-18T20:30:33.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 0 2023-10-18T20:30:33.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket shard: 0 2023-10-18T20:30:33.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket shard: 0 2023-10-18T20:30:33.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket shard: 0 2023-10-18T20:30:33.129 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.129 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.129 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket shard: 0 2023-10-18T20:30:33.129 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.130 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.130 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.119+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-18T20:30:33.130 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.119+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5/obj00000000000000000000 2023-10-18T20:30:33.130 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.119+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: entry->timestamp=2023-10-18T20:30:26.553117+0000 2023-10-18T20:30:33.130 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.119+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-18T20:30:33.131 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.119+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000000]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000000[0] 2023-10-18T20:30:33.131 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.119+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-18T20:30:33.131 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket shard: 0 2023-10-18T20:30:33.131 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket shard: 0 2023-10-18T20:30:33.132 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket shard: 0 2023-10-18T20:30:33.132 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket shard: 0 2023-10-18T20:30:33.132 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.132 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-18T20:30:33.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000010 2023-10-18T20:30:33.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: entry->timestamp=2023-10-18T20:30:26.160546+0000 2023-10-18T20:30:33.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-10-18T20:30:33.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: parsed entry: id=00000000001.38.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-10-18T20:30:33.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6/obj00000000000000000032 2023-10-18T20:30:33.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: entry->timestamp=2023-10-18T20:30:26.264975+0000 2023-10-18T20:30:33.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.38.6 op=0 op_state=1 2023-10-18T20:30:33.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: parsed entry: id=00000000001.110.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-18T20:30:33.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9/obj00000000000000000019 2023-10-18T20:30:33.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: entry->timestamp=2023-10-18T20:30:26.573571+0000 2023-10-18T20:30:33.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.110.6 op=0 op_state=1 2023-10-18T20:30:33.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-10-18T20:30:33.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position=00000000002.46.6 2023-10-18T20:30:33.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000032[0] 2023-10-18T20:30:33.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position=00000000001.38.6 2023-10-18T20:30:33.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-18T20:30:33.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11/obj00000000000000000021 2023-10-18T20:30:33.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: entry->timestamp=2023-10-18T20:30:26.469593+0000 2023-10-18T20:30:33.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.41.6 op=0 op_state=1 2023-10-18T20:30:33.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000019]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000019[0] 2023-10-18T20:30:33.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position=00000000001.110.6 2023-10-18T20:30:33.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000010 2023-10-18T20:30:33.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000021]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000021[0] 2023-10-18T20:30:33.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.120+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position=00000000001.41.6 2023-10-18T20:30:33.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.121+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket shard: 0 2023-10-18T20:30:33.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.121+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket shard: 0 2023-10-18T20:30:33.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.121+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.121+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.122+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:33.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.122+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.141 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.124+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-18T20:30:33.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.124+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15/obj00000000000000000004 2023-10-18T20:30:33.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.124+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: entry->timestamp=2023-10-18T20:30:26.447815+0000 2023-10-18T20:30:33.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.124+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-10-18T20:30:33.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.124+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000004]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000004[0] 2023-10-18T20:30:33.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.124+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position=00000000002.36.6 2023-10-18T20:30:33.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.124+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000001.46.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2023-10-18T20:30:33.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.124+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000076 2023-10-18T20:30:33.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.124+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:26.141311+0000 2023-10-18T20:30:33.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.46.6 op=0 op_state=1 2023-10-18T20:30:33.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000076]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000076[0] 2023-10-18T20:30:33.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000001.46.6 2023-10-18T20:30:33.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-18T20:30:33.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13/obj00000000000000000026 2023-10-18T20:30:33.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: entry->timestamp=2023-10-18T20:30:26.073164+0000 2023-10-18T20:30:33.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-10-18T20:30:33.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000026]: : injecting data sync error on key=obj00000000000000000026 2023-10-18T20:30:33.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position=00000000001.37.6 2023-10-18T20:30:33.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000026]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.125+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000026]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13/obj00000000000000000026 2023-10-18T20:30:33.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.133+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000000]: success 2023-10-18T20:30:33.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.133+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5:1 marker=00000000002.52.6 timestamp=2023-10-18T20:30:26.553117+0000 2023-10-18T20:30:33.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.133+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000019]: success 2023-10-18T20:30:33.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.133+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9:1 marker=00000000001.110.6 timestamp=2023-10-18T20:30:26.573571+0000 2023-10-18T20:30:33.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.133+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000032]: success 2023-10-18T20:30:33.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.133+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6:1 marker=00000000001.38.6 timestamp=2023-10-18T20:30:26.264975+0000 2023-10-18T20:30:33.161 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-18T20:30:33.161 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0/obj00000000000000000007 2023-10-18T20:30:33.161 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: entry->timestamp=2023-10-18T20:30:26.284607+0000 2023-10-18T20:30:33.161 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-18T20:30:33.162 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000007]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000007[0] 2023-10-18T20:30:33.162 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-18T20:30:33.162 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-10-18T20:30:33.162 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2/obj00000000000000000024 2023-10-18T20:30:33.163 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: entry->timestamp=2023-10-18T20:30:26.180798+0000 2023-10-18T20:30:33.163 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-10-18T20:30:33.163 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-18T20:30:33.163 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3/obj00000000000000000020 2023-10-18T20:30:33.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: entry->timestamp=2023-10-18T20:30:26.530259+0000 2023-10-18T20:30:33.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.85.6 op=0 op_state=1 2023-10-18T20:30:33.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000024]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000024[0] 2023-10-18T20:30:33.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position=00000000001.43.6 2023-10-18T20:30:33.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000020]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000020[0] 2023-10-18T20:30:33.165 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position=00000000001.85.6 2023-10-18T20:30:33.165 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.160+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-18T20:30:33.166 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.160+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12/obj00000000000000000030 2023-10-18T20:30:33.166 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.160+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: entry->timestamp=2023-10-18T20:30:26.347305+0000 2023-10-18T20:30:33.166 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.160+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.81.6 op=0 op_state=1 2023-10-18T20:30:33.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.160+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000030]: : injecting data sync error on key=obj00000000000000000030 2023-10-18T20:30:33.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.160+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position=00000000001.81.6 2023-10-18T20:30:33.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.160+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000030]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.160+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000030]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12/obj00000000000000000030 2023-10-18T20:30:33.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-18T20:30:33.168 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10/obj00000000000000000013 2023-10-18T20:30:33.168 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: entry->timestamp=2023-10-18T20:30:26.366141+0000 2023-10-18T20:30:33.168 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.83.6 op=0 op_state=1 2023-10-18T20:30:33.168 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000013]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000013[0] 2023-10-18T20:30:33.169 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position=00000000001.83.6 2023-10-18T20:30:33.169 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-18T20:30:33.169 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14/obj00000000000000000033 2023-10-18T20:30:33.169 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: entry->timestamp=2023-10-18T20:30:26.051255+0000 2023-10-18T20:30:33.169 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-10-18T20:30:33.170 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000033]: : injecting data sync error on key=obj00000000000000000033 2023-10-18T20:30:33.170 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-18T20:30:33.170 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000033]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.170 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.163+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000033]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14/obj00000000000000000033 2023-10-18T20:30:33.172 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-10-18T20:30:33.172 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4/obj00000000000000000028 2023-10-18T20:30:33.172 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: entry->timestamp=2023-10-18T20:30:26.245833+0000 2023-10-18T20:30:33.173 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.63.6 op=0 op_state=1 2023-10-18T20:30:33.173 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-18T20:30:33.173 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7/obj00000000000000000001 2023-10-18T20:30:33.173 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: entry->timestamp=2023-10-18T20:30:26.424615+0000 2023-10-18T20:30:33.174 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-10-18T20:30:33.174 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000028]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000028[0] 2023-10-18T20:30:33.174 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position=00000000001.63.6 2023-10-18T20:30:33.174 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000001]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000001[0] 2023-10-18T20:30:33.174 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.164+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position=00000000002.65.6 2023-10-18T20:30:33.175 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.165+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: a sync operation returned error 2023-10-18T20:30:33.175 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.165+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: backing out with sync_status=-5 2023-10-18T20:30:33.175 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.165+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.175 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.165+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.175 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.167+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.176 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.168+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: a sync operation returned error 2023-10-18T20:30:33.176 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.168+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: backing out with sync_status=-5 2023-10-18T20:30:33.176 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.168+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.176 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.168+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.177 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.168+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 0 2023-10-18T20:30:33.177 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.168+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.177 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.169+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.177 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.169+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket shard: 0 2023-10-18T20:30:33.177 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.169+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.178 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.169+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000007]: success 2023-10-18T20:30:33.178 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.169+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0:1 marker=00000000001.49.6 timestamp=2023-10-18T20:30:26.284607+0000 2023-10-18T20:30:33.178 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.170+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000020]: success 2023-10-18T20:30:33.178 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.170+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3:1 marker=00000000001.85.6 timestamp=2023-10-18T20:30:26.530259+0000 2023-10-18T20:30:33.179 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.172+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000021]: success 2023-10-18T20:30:33.179 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.172+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11:1 marker=00000000001.41.6 timestamp=2023-10-18T20:30:26.469593+0000 2023-10-18T20:30:33.180 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.173+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000024]: success 2023-10-18T20:30:33.180 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.173+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2:1 marker=00000000001.43.6 timestamp=2023-10-18T20:30:26.180798+0000 2023-10-18T20:30:33.180 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.173+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000004]: success 2023-10-18T20:30:33.180 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.173+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15:1 marker=00000000002.36.6 timestamp=2023-10-18T20:30:26.447815+0000 2023-10-18T20:30:33.181 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.176+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000076]: success 2023-10-18T20:30:33.181 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.176+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16:1 marker=00000000001.46.6 timestamp=2023-10-18T20:30:26.141311+0000 2023-10-18T20:30:33.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.202+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: a sync operation returned error 2023-10-18T20:30:33.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.202+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: backing out with sync_status=-5 2023-10-18T20:30:33.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.202+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.202+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.203+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.204+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket shard: 0 2023-10-18T20:30:33.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.204+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.206+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: a sync operation returned error 2023-10-18T20:30:33.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.206+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: backing out with sync_status=-5 2023-10-18T20:30:33.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.206+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.206+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.206+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.206+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket shard: 0 2023-10-18T20:30:33.210 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.206+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.212 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.210+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-18T20:30:33.212 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.210+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000010 2023-10-18T20:30:33.212 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.210+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: entry->timestamp=2023-10-18T20:30:26.160546+0000 2023-10-18T20:30:33.212 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.210+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-10-18T20:30:33.213 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.210+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-10-18T20:30:33.213 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.210+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position=00000000002.46.6 2023-10-18T20:30:33.213 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.210+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.213 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.210+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000010 2023-10-18T20:30:33.214 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.211+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000028]: success 2023-10-18T20:30:33.214 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.211+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4:1 marker=00000000001.63.6 timestamp=2023-10-18T20:30:26.245833+0000 2023-10-18T20:30:33.214 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.212+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-18T20:30:33.215 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.212+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13/obj00000000000000000026 2023-10-18T20:30:33.215 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.212+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: entry->timestamp=2023-10-18T20:30:26.073164+0000 2023-10-18T20:30:33.215 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.212+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-10-18T20:30:33.215 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.212+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000026]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000026[0] 2023-10-18T20:30:33.216 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.212+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position=00000000001.37.6 2023-10-18T20:30:33.216 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.213+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000001]: success 2023-10-18T20:30:33.216 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.213+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7:1 marker=00000000002.65.6 timestamp=2023-10-18T20:30:26.424615+0000 2023-10-18T20:30:33.217 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.214+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000013]: success 2023-10-18T20:30:33.217 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.214+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10:1 marker=00000000001.83.6 timestamp=2023-10-18T20:30:26.366141+0000 2023-10-18T20:30:33.220 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.218+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000026]: success 2023-10-18T20:30:33.220 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.218+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13:1 marker=00000000001.37.6 timestamp=2023-10-18T20:30:26.073164+0000 2023-10-18T20:30:33.248 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.246+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-18T20:30:33.248 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.246+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12/obj00000000000000000030 2023-10-18T20:30:33.249 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.246+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: entry->timestamp=2023-10-18T20:30:26.347305+0000 2023-10-18T20:30:33.249 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.246+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.81.6 op=0 op_state=1 2023-10-18T20:30:33.249 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.246+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000030]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000030[0] 2023-10-18T20:30:33.249 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.246+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position=00000000001.81.6 2023-10-18T20:30:33.251 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.249+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-18T20:30:33.251 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.249+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14/obj00000000000000000033 2023-10-18T20:30:33.251 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.249+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: entry->timestamp=2023-10-18T20:30:26.051255+0000 2023-10-18T20:30:33.252 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.249+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-10-18T20:30:33.252 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.249+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000033]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000033[0] 2023-10-18T20:30:33.252 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.249+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-18T20:30:33.254 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.252+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: a sync operation returned error 2023-10-18T20:30:33.254 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.252+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: backing out with sync_status=-5 2023-10-18T20:30:33.254 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.252+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.254 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.252+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.255 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.252+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.255 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.252+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 0 2023-10-18T20:30:33.255 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.252+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.254+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000033]: success 2023-10-18T20:30:33.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.254+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14:1 marker=00000000001.44.6 timestamp=2023-10-18T20:30:26.051255+0000 2023-10-18T20:30:33.294 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.291+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000030]: success 2023-10-18T20:30:33.294 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.291+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12:1 marker=00000000001.81.6 timestamp=2023-10-18T20:30:26.347305+0000 2023-10-18T20:30:33.297 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.295+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-18T20:30:33.297 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.295+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000010 2023-10-18T20:30:33.298 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.295+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: entry->timestamp=2023-10-18T20:30:26.160546+0000 2023-10-18T20:30:33.298 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.295+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-10-18T20:30:33.298 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.295+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000010[0] 2023-10-18T20:30:33.298 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.295+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position=00000000002.46.6 2023-10-18T20:30:33.342 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.339+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000010]: success 2023-10-18T20:30:33.342 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.339+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8:1 marker=00000000002.46.6 timestamp=2023-10-18T20:30:26.160546+0000 2023-10-18T20:30:33.347 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.344+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.347 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.344+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.344+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.345+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket shard: 0 2023-10-18T20:30:33.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.345+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.345+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket shard: 0 2023-10-18T20:30:33.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.345+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.345+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket shard: 0 2023-10-18T20:30:33.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.345+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.345+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.350 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.345+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.352 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket shard: 0 2023-10-18T20:30:33.352 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket shard: 0 2023-10-18T20:30:33.352 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.353 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.353 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.353 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.353 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.354 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.354 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.354 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-18T20:30:33.354 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0/obj00000000000000000005 2023-10-18T20:30:33.355 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: entry->timestamp=2023-10-18T20:30:27.506925+0000 2023-10-18T20:30:33.355 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-18T20:30:33.355 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000005]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000005[0] 2023-10-18T20:30:33.355 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.346+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-18T20:30:33.356 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket shard: 0 2023-10-18T20:30:33.356 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket shard: 0 2023-10-18T20:30:33.356 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket shard: 0 2023-10-18T20:30:33.356 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket shard: 0 2023-10-18T20:30:33.357 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket shard: 0 2023-10-18T20:30:33.357 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.357 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.357 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.357 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.358 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.358 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-18T20:30:33.358 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2/obj00000000000000000016 2023-10-18T20:30:33.358 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: entry->timestamp=2023-10-18T20:30:26.909798+0000 2023-10-18T20:30:33.359 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-10-18T20:30:33.359 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.359 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.359 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.347+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.360 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.360 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000016]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000016[0] 2023-10-18T20:30:33.360 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position=00000000001.53.6 2023-10-18T20:30:33.360 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-18T20:30:33.360 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5/obj00000000000000000025 2023-10-18T20:30:33.361 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: entry->timestamp=2023-10-18T20:30:26.891244+0000 2023-10-18T20:30:33.361 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-10-18T20:30:33.361 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.361 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000025]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000025[0] 2023-10-18T20:30:33.361 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position=00000000001.60.6 2023-10-18T20:30:33.363 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.363 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.363 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.363 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.348+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.349+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 0 2023-10-18T20:30:33.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.349+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.349+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket shard: 0 2023-10-18T20:30:33.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.349+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.365 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket shard: 0 2023-10-18T20:30:33.365 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.365 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket shard: 0 2023-10-18T20:30:33.365 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket shard: 0 2023-10-18T20:30:33.366 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket shard: 0 2023-10-18T20:30:33.366 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket shard: 0 2023-10-18T20:30:33.366 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket shard: 0 2023-10-18T20:30:33.366 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.366 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.367 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.367 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.367 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.367 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:33.368 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.350+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.368 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-18T20:30:33.368 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7/obj00000000000000000015 2023-10-18T20:30:33.368 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: entry->timestamp=2023-10-18T20:30:27.158214+0000 2023-10-18T20:30:33.369 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-10-18T20:30:33.369 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-18T20:30:33.369 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6/obj00000000000000000030 2023-10-18T20:30:33.369 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: entry->timestamp=2023-10-18T20:30:27.197036+0000 2023-10-18T20:30:33.370 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-10-18T20:30:33.370 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000015]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000015[0] 2023-10-18T20:30:33.370 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position=00000000002.46.6 2023-10-18T20:30:33.370 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000030]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000030[0] 2023-10-18T20:30:33.371 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position=00000000002.41.6 2023-10-18T20:30:33.371 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-18T20:30:33.371 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9/obj00000000000000000003 2023-10-18T20:30:33.371 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: entry->timestamp=2023-10-18T20:30:27.007258+0000 2023-10-18T20:30:33.371 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-18T20:30:33.372 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000003]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000003[0] 2023-10-18T20:30:33.372 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.351+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-18T20:30:33.372 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: parsed entry: id=00000000001.29.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-18T20:30:33.372 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11/obj00000000000000000014 2023-10-18T20:30:33.373 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: entry->timestamp=2023-10-18T20:30:26.872477+0000 2023-10-18T20:30:33.373 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.29.6 op=0 op_state=1 2023-10-18T20:30:33.373 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000014]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000014[0] 2023-10-18T20:30:33.373 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position=00000000001.29.6 2023-10-18T20:30:33.373 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-18T20:30:33.374 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12/obj00000000000000000012 2023-10-18T20:30:33.374 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: entry->timestamp=2023-10-18T20:30:26.986257+0000 2023-10-18T20:30:33.374 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-10-18T20:30:33.375 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-18T20:30:33.375 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000019 2023-10-18T20:30:33.375 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: entry->timestamp=2023-10-18T20:30:27.443246+0000 2023-10-18T20:30:33.376 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-18T20:30:33.376 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000012]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000012[0] 2023-10-18T20:30:33.376 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position=00000000002.58.6 2023-10-18T20:30:33.376 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000019]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000019[0] 2023-10-18T20:30:33.377 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.352+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-18T20:30:33.377 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: parsed entry: id=00000000001.33.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-18T20:30:33.377 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10/obj00000000000000000009 2023-10-18T20:30:33.377 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: entry->timestamp=2023-10-18T20:30:26.928877+0000 2023-10-18T20:30:33.378 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.33.6 op=0 op_state=1 2023-10-18T20:30:33.378 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: parsed entry: id=00000000002.29.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-18T20:30:33.378 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17/obj00000000000000000001 2023-10-18T20:30:33.379 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: entry->timestamp=2023-10-18T20:30:27.364742+0000 2023-10-18T20:30:33.379 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.29.6 op=0 op_state=1 2023-10-18T20:30:33.379 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: parsed entry: id=00000000002.27.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-18T20:30:33.379 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14/obj00000000000000000000 2023-10-18T20:30:33.379 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: entry->timestamp=2023-10-18T20:30:27.073942+0000 2023-10-18T20:30:33.380 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.27.6 op=0 op_state=1 2023-10-18T20:30:33.380 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000009]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000009[0] 2023-10-18T20:30:33.380 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position=00000000001.33.6 2023-10-18T20:30:33.380 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000001[0] 2023-10-18T20:30:33.381 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position=00000000002.29.6 2023-10-18T20:30:33.381 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000000]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000000[0] 2023-10-18T20:30:33.381 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.355+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position=00000000002.27.6 2023-10-18T20:30:33.381 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.356+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000030]: success 2023-10-18T20:30:33.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.356+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6:2 marker=00000000002.41.6 timestamp=2023-10-18T20:30:27.197036+0000 2023-10-18T20:30:33.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.356+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000019]: success 2023-10-18T20:30:33.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.356+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8:2 marker=00000000002.43.6 timestamp=2023-10-18T20:30:27.443246+0000 2023-10-18T20:30:33.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.360+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000001]: success 2023-10-18T20:30:33.383 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.361+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17:2 marker=00000000002.29.6 timestamp=2023-10-18T20:30:27.364742+0000 2023-10-18T20:30:33.383 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.361+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000000]: success 2023-10-18T20:30:33.383 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.361+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14:2 marker=00000000002.27.6 timestamp=2023-10-18T20:30:27.073942+0000 2023-10-18T20:30:33.384 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.361+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000016]: success 2023-10-18T20:30:33.384 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.361+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2:2 marker=00000000001.53.6 timestamp=2023-10-18T20:30:26.909798+0000 2023-10-18T20:30:33.384 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.362+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000012]: success 2023-10-18T20:30:33.385 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.362+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12:2 marker=00000000002.58.6 timestamp=2023-10-18T20:30:26.986257+0000 2023-10-18T20:30:33.385 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.365+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000025]: success 2023-10-18T20:30:33.385 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.365+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5:2 marker=00000000001.60.6 timestamp=2023-10-18T20:30:26.891244+0000 2023-10-18T20:30:33.385 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.367+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000003]: success 2023-10-18T20:30:33.386 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.367+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9:2 marker=00000000002.52.6 timestamp=2023-10-18T20:30:27.007258+0000 2023-10-18T20:30:33.389 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.387+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-18T20:30:33.389 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.387+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1/obj00000000000000000010 2023-10-18T20:30:33.389 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.387+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: entry->timestamp=2023-10-18T20:30:27.422598+0000 2023-10-18T20:30:33.390 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.387+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-10-18T20:30:33.390 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.387+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000010]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000010[0] 2023-10-18T20:30:33.390 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.387+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position=00000000002.49.6 2023-10-18T20:30:33.391 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.388+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-18T20:30:33.391 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.388+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4/obj00000000000000000004 2023-10-18T20:30:33.391 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.388+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: entry->timestamp=2023-10-18T20:30:27.177219+0000 2023-10-18T20:30:33.392 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.388+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-10-18T20:30:33.392 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.388+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000004]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000004[0] 2023-10-18T20:30:33.392 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.388+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position=00000000002.53.6 2023-10-18T20:30:33.392 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.389+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000010]: success 2023-10-18T20:30:33.393 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.389+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1:2 marker=00000000002.49.6 timestamp=2023-10-18T20:30:27.422598+0000 2023-10-18T20:30:33.393 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.389+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000004]: success 2023-10-18T20:30:33.394 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.389+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4:2 marker=00000000002.53.6 timestamp=2023-10-18T20:30:27.177219+0000 2023-10-18T20:30:33.394 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.391+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-18T20:30:33.394 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.391+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3/obj00000000000000000021 2023-10-18T20:30:33.394 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.391+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: entry->timestamp=2023-10-18T20:30:27.255303+0000 2023-10-18T20:30:33.395 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.391+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-18T20:30:33.395 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.391+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000021]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000021[0] 2023-10-18T20:30:33.395 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.391+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-18T20:30:33.396 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.392+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000021]: success 2023-10-18T20:30:33.396 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.392+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3:2 marker=00000000002.45.6 timestamp=2023-10-18T20:30:27.255303+0000 2023-10-18T20:30:33.396 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.394+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000005]: success 2023-10-18T20:30:33.397 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.394+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0:2 marker=00000000002.51.6 timestamp=2023-10-18T20:30:27.506925+0000 2023-10-18T20:30:33.398 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-18T20:30:33.398 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13/obj00000000000000000002 2023-10-18T20:30:33.398 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: entry->timestamp=2023-10-18T20:30:27.276179+0000 2023-10-18T20:30:33.399 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-18T20:30:33.399 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000002]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000002[0] 2023-10-18T20:30:33.399 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-18T20:30:33.399 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-18T20:30:33.399 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15/obj00000000000000000006 2023-10-18T20:30:33.400 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: entry->timestamp=2023-10-18T20:30:27.343793+0000 2023-10-18T20:30:33.400 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-18T20:30:33.400 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-18T20:30:33.400 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000011 2023-10-18T20:30:33.401 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:27.096906+0000 2023-10-18T20:30:33.401 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-10-18T20:30:33.401 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000006]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000006[0] 2023-10-18T20:30:33.401 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-18T20:30:33.402 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000011]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000011[0] 2023-10-18T20:30:33.402 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.395+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000002.38.6 2023-10-18T20:30:33.405 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.402+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000015]: success 2023-10-18T20:30:33.405 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.402+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7:2 marker=00000000002.46.6 timestamp=2023-10-18T20:30:27.158214+0000 2023-10-18T20:30:33.407 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.404+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000009]: success 2023-10-18T20:30:33.407 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.404+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10:2 marker=00000000001.33.6 timestamp=2023-10-18T20:30:26.928877+0000 2023-10-18T20:30:33.407 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.405+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000014]: success 2023-10-18T20:30:33.408 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.405+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11:2 marker=00000000001.29.6 timestamp=2023-10-18T20:30:26.872477+0000 2023-10-18T20:30:33.445 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.442+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000011]: success 2023-10-18T20:30:33.445 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.442+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16:2 marker=00000000002.38.6 timestamp=2023-10-18T20:30:27.096906+0000 2023-10-18T20:30:33.446 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.444+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000002]: success 2023-10-18T20:30:33.446 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.444+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13:2 marker=00000000002.47.6 timestamp=2023-10-18T20:30:27.276179+0000 2023-10-18T20:30:33.449 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.447+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000006]: success 2023-10-18T20:30:33.450 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.447+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15:2 marker=00000000002.40.6 timestamp=2023-10-18T20:30:27.343793+0000 2023-10-18T20:30:33.454 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.451+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.454 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.451+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.454 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.452+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.455 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.452+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket shard: 0 2023-10-18T20:30:33.455 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.452+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.455 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.452+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket shard: 0 2023-10-18T20:30:33.455 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.452+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.456 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.452+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.457 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.452+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket shard: 0 2023-10-18T20:30:33.457 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.452+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.457 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.457 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.458 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.458 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.458 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket shard: 0 2023-10-18T20:30:33.458 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.459 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.459 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket shard: 0 2023-10-18T20:30:33.459 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket shard: 0 2023-10-18T20:30:33.459 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.460 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.460 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.453+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.462 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.454+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket shard: 0 2023-10-18T20:30:33.462 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.454+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.463 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.454+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.463 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.454+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.463 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.454+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.463 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.454+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.463 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.454+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket shard: 0 2023-10-18T20:30:33.464 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.454+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.464 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.454+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.464 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.455+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket shard: 0 2023-10-18T20:30:33.464 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.455+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.465 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.455+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.465 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.455+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.465 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.455+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket shard: 0 2023-10-18T20:30:33.465 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.455+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket shard: 0 2023-10-18T20:30:33.465 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.455+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.466 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.455+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.466 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.455+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.466 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 0 2023-10-18T20:30:33.466 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.467 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket shard: 0 2023-10-18T20:30:33.467 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket shard: 0 2023-10-18T20:30:33.467 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:33.467 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket shard: 0 2023-10-18T20:30:33.467 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.468 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.468 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.468 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.468 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.468 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.469 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.456+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.469 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.469 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.469 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-18T20:30:33.469 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12/obj00000000000000000009 2023-10-18T20:30:33.470 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: entry->timestamp=2023-10-18T20:30:27.793809+0000 2023-10-18T20:30:33.470 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-18T20:30:33.470 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000009]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000009[0] 2023-10-18T20:30:33.470 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-18T20:30:33.471 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket shard: 0 2023-10-18T20:30:33.471 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.457+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.471 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket shard: 0 2023-10-18T20:30:33.471 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.471 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: sync status for source bucket shard: 0 2023-10-18T20:30:33.472 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.472 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-10-18T20:30:33.472 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13/obj00000000000000000045 2023-10-18T20:30:33.472 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: entry->timestamp=2023-10-18T20:30:28.542920+0000 2023-10-18T20:30:33.472 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-10-18T20:30:33.473 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-10-18T20:30:33.473 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000039 2023-10-18T20:30:33.473 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: entry->timestamp=2023-10-18T20:30:28.223186+0000 2023-10-18T20:30:33.473 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.40.6 op=0 op_state=1 2023-10-18T20:30:33.473 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000045]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000045[0] 2023-10-18T20:30:33.474 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position=00000000001.58.6 2023-10-18T20:30:33.474 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-18T20:30:33.474 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14/obj00000000000000000013 2023-10-18T20:30:33.474 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: entry->timestamp=2023-10-18T20:30:27.933800+0000 2023-10-18T20:30:33.475 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.458+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-18T20:30:33.475 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000039]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000039[0] 2023-10-18T20:30:33.475 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position=00000000001.40.6 2023-10-18T20:30:33.475 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000013]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000013[0] 2023-10-18T20:30:33.476 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-18T20:30:33.476 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket shard: 0 2023-10-18T20:30:33.476 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket shard: 0 2023-10-18T20:30:33.476 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.476 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.477 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: sync status for source bucket shard: 0 2023-10-18T20:30:33.477 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: sync status for source bucket shard: 0 2023-10-18T20:30:33.477 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-18T20:30:33.477 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000033 2023-10-18T20:30:33.477 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:28.291287+0000 2023-10-18T20:30:33.478 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-10-18T20:30:33.478 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-18T20:30:33.478 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17/obj00000000000000000001 2023-10-18T20:30:33.478 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: entry->timestamp=2023-10-18T20:30:28.461417+0000 2023-10-18T20:30:33.479 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-10-18T20:30:33.479 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.479 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.479 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000033]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000033[0] 2023-10-18T20:30:33.479 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000002.39.6 2023-10-18T20:30:33.480 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000001[0] 2023-10-18T20:30:33.480 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.459+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position=00000000002.58.6 2023-10-18T20:30:33.481 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000009]: success 2023-10-18T20:30:33.481 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12:3 marker=00000000002.69.6 timestamp=2023-10-18T20:30:27.793809+0000 2023-10-18T20:30:33.481 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-18T20:30:33.482 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19/obj00000000000000000021 2023-10-18T20:30:33.482 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: entry->timestamp=2023-10-18T20:30:28.562420+0000 2023-10-18T20:30:33.482 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-10-18T20:30:33.482 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2023-10-18T20:30:33.482 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21/obj00000000000000000055 2023-10-18T20:30:33.483 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: entry->timestamp=2023-10-18T20:30:27.872840+0000 2023-10-18T20:30:33.483 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-10-18T20:30:33.483 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000021]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000021[0] 2023-10-18T20:30:33.483 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: listing bilog for incremental sync; position=00000000002.63.6 2023-10-18T20:30:33.484 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-18T20:30:33.484 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18/obj00000000000000000000 2023-10-18T20:30:33.484 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: entry->timestamp=2023-10-18T20:30:28.313238+0000 2023-10-18T20:30:33.484 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-18T20:30:33.484 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000055]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000055[0] 2023-10-18T20:30:33.485 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: listing bilog for incremental sync; position=00000000001.56.6 2023-10-18T20:30:33.485 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000000]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000000[0] 2023-10-18T20:30:33.485 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.463+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-18T20:30:33.485 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.464+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000033]: success 2023-10-18T20:30:33.485 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.464+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16:3 marker=00000000002.39.6 timestamp=2023-10-18T20:30:28.291287+0000 2023-10-18T20:30:33.486 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.466+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000013]: success 2023-10-18T20:30:33.486 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.466+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14:3 marker=00000000002.51.6 timestamp=2023-10-18T20:30:27.933800+0000 2023-10-18T20:30:33.486 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.466+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000001]: success 2023-10-18T20:30:33.486 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.466+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17:3 marker=00000000002.58.6 timestamp=2023-10-18T20:30:28.461417+0000 2023-10-18T20:30:33.487 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.466+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000021]: success 2023-10-18T20:30:33.487 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.466+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19:3 marker=00000000002.63.6 timestamp=2023-10-18T20:30:28.562420+0000 2023-10-18T20:30:33.487 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.466+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000000]: success 2023-10-18T20:30:33.487 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.466+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18:3 marker=00000000002.51.6 timestamp=2023-10-18T20:30:28.313238+0000 2023-10-18T20:30:33.487 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.470+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000045]: success 2023-10-18T20:30:33.488 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.470+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13:3 marker=00000000001.58.6 timestamp=2023-10-18T20:30:28.542920+0000 2023-10-18T20:30:33.496 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-18T20:30:33.497 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1/obj00000000000000000026 2023-10-18T20:30:33.497 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: entry->timestamp=2023-10-18T20:30:28.393515+0000 2023-10-18T20:30:33.497 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-10-18T20:30:33.498 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000026]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000026[0] 2023-10-18T20:30:33.498 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position=00000000002.63.6 2023-10-18T20:30:33.498 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2023-10-18T20:30:33.498 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0/obj00000000000000000058 2023-10-18T20:30:33.499 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: entry->timestamp=2023-10-18T20:30:28.372095+0000 2023-10-18T20:30:33.499 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.40.6 op=0 op_state=1 2023-10-18T20:30:33.499 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-18T20:30:33.499 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4/obj00000000000000000014 2023-10-18T20:30:33.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: entry->timestamp=2023-10-18T20:30:28.197733+0000 2023-10-18T20:30:33.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-18T20:30:33.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000058]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000058[0] 2023-10-18T20:30:33.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position=00000000001.40.6 2023-10-18T20:30:33.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000014]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000014[0] 2023-10-18T20:30:33.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.494+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-18T20:30:33.505 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-18T20:30:33.505 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5/obj00000000000000000050 2023-10-18T20:30:33.506 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: entry->timestamp=2023-10-18T20:30:27.775317+0000 2023-10-18T20:30:33.506 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-10-18T20:30:33.506 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000050]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000050[0] 2023-10-18T20:30:33.506 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position=00000000001.56.6 2023-10-18T20:30:33.506 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-10-18T20:30:33.507 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6/obj00000000000000000032 2023-10-18T20:30:33.507 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: entry->timestamp=2023-10-18T20:30:27.953086+0000 2023-10-18T20:30:33.507 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-10-18T20:30:33.507 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-18T20:30:33.508 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7/obj00000000000000000002 2023-10-18T20:30:33.508 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: entry->timestamp=2023-10-18T20:30:28.110419+0000 2023-10-18T20:30:33.508 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-10-18T20:30:33.508 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000032[0] 2023-10-18T20:30:33.509 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position=00000000002.34.6 2023-10-18T20:30:33.509 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000026]: success 2023-10-18T20:30:33.509 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1:3 marker=00000000002.63.6 timestamp=2023-10-18T20:30:28.393515+0000 2023-10-18T20:30:33.509 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000002]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000002[0] 2023-10-18T20:30:33.509 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position=00000000002.71.6 2023-10-18T20:30:33.510 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000014]: success 2023-10-18T20:30:33.510 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.496+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4:3 marker=00000000002.48.6 timestamp=2023-10-18T20:30:28.197733+0000 2023-10-18T20:30:33.510 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.498+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000032]: success 2023-10-18T20:30:33.510 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.498+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6:3 marker=00000000002.34.6 timestamp=2023-10-18T20:30:27.953086+0000 2023-10-18T20:30:33.511 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.498+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000002]: success 2023-10-18T20:30:33.511 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.498+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7:3 marker=00000000002.71.6 timestamp=2023-10-18T20:30:28.110419+0000 2023-10-18T20:30:33.511 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-10-18T20:30:33.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11/obj00000000000000000040 2023-10-18T20:30:33.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: entry->timestamp=2023-10-18T20:30:28.049773+0000 2023-10-18T20:30:33.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-10-18T20:30:33.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000040]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000040[0] 2023-10-18T20:30:33.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position=00000000001.56.6 2023-10-18T20:30:33.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-18T20:30:33.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10/obj00000000000000000006 2023-10-18T20:30:33.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: entry->timestamp=2023-10-18T20:30:28.130319+0000 2023-10-18T20:30:33.514 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-10-18T20:30:33.514 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-18T20:30:33.514 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15/obj00000000000000000012 2023-10-18T20:30:33.515 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: entry->timestamp=2023-10-18T20:30:27.849799+0000 2023-10-18T20:30:33.515 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-10-18T20:30:33.515 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000006]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000006[0] 2023-10-18T20:30:33.516 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position=00000000002.37.6 2023-10-18T20:30:33.516 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000012]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000012[0] 2023-10-18T20:30:33.517 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.499+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position=00000000002.34.6 2023-10-18T20:30:33.517 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.501+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000006]: success 2023-10-18T20:30:33.517 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.501+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10:3 marker=00000000002.37.6 timestamp=2023-10-18T20:30:28.130319+0000 2023-10-18T20:30:33.518 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.501+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000012]: success 2023-10-18T20:30:33.519 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.501+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15:3 marker=00000000002.34.6 timestamp=2023-10-18T20:30:27.849799+0000 2023-10-18T20:30:33.519 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-18T20:30:33.520 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20/obj00000000000000000003 2023-10-18T20:30:33.520 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: entry->timestamp=2023-10-18T20:30:27.973671+0000 2023-10-18T20:30:33.520 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-18T20:30:33.521 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000003]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000003[0] 2023-10-18T20:30:33.521 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-18T20:30:33.522 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-18T20:30:33.522 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22/obj00000000000000000008 2023-10-18T20:30:33.523 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: entry->timestamp=2023-10-18T20:30:28.482088+0000 2023-10-18T20:30:33.523 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-10-18T20:30:33.524 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000008]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000008[0] 2023-10-18T20:30:33.524 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.502+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: listing bilog for incremental sync; position=00000000002.58.6 2023-10-18T20:30:33.524 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.503+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: parsed entry: id=00000000001.38.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2023-10-18T20:30:33.525 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.503+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3/obj00000000000000000046 2023-10-18T20:30:33.525 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.503+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: entry->timestamp=2023-10-18T20:30:28.030762+0000 2023-10-18T20:30:33.526 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.503+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.38.6 op=0 op_state=1 2023-10-18T20:30:33.526 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.503+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000046]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000046[0] 2023-10-18T20:30:33.526 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.503+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position=00000000001.38.6 2023-10-18T20:30:33.528 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.504+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000003]: success 2023-10-18T20:30:33.528 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.504+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20:3 marker=00000000002.54.6 timestamp=2023-10-18T20:30:27.973671+0000 2023-10-18T20:30:33.529 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.508+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000039]: success 2023-10-18T20:30:33.529 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.508+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8:3 marker=00000000001.40.6 timestamp=2023-10-18T20:30:28.223186+0000 2023-10-18T20:30:33.530 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.510+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000055]: success 2023-10-18T20:30:33.530 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.510+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21:3 marker=00000000001.56.6 timestamp=2023-10-18T20:30:27.872840+0000 2023-10-18T20:30:33.542 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.539+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000058]: success 2023-10-18T20:30:33.542 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.540+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0:3 marker=00000000001.40.6 timestamp=2023-10-18T20:30:28.372095+0000 2023-10-18T20:30:33.545 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.543+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000050]: success 2023-10-18T20:30:33.545 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.543+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5:3 marker=00000000001.56.6 timestamp=2023-10-18T20:30:27.775317+0000 2023-10-18T20:30:33.549 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.547+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000040]: success 2023-10-18T20:30:33.549 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.547+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11:3 marker=00000000001.56.6 timestamp=2023-10-18T20:30:28.049773+0000 2023-10-18T20:30:33.552 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.550+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000046]: success 2023-10-18T20:30:33.552 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.550+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3:3 marker=00000000001.38.6 timestamp=2023-10-18T20:30:28.030762+0000 2023-10-18T20:30:33.553 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.551+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000008]: success 2023-10-18T20:30:33.553 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.551+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22:3 marker=00000000002.58.6 timestamp=2023-10-18T20:30:28.482088+0000 2023-10-18T20:30:33.568 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.569 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.569 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.569 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket shard: 0 2023-10-18T20:30:33.569 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.570 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket shard: 0 2023-10-18T20:30:33.570 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.573 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.574 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket shard: 0 2023-10-18T20:30:33.574 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.574 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.574 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket shard: 0 2023-10-18T20:30:33.575 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.566+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.575 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.567+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.575 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.567+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.575 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket shard: 0 2023-10-18T20:30:33.575 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.576 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket shard: 0 2023-10-18T20:30:33.576 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket shard: 0 2023-10-18T20:30:33.576 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.576 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.568+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket shard: 0 2023-10-18T20:30:33.579 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.579 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket shard: 0 2023-10-18T20:30:33.579 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.580 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket shard: 0 2023-10-18T20:30:33.580 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket shard: 0 2023-10-18T20:30:33.580 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 0 2023-10-18T20:30:33.580 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket shard: 0 2023-10-18T20:30:33.580 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket shard: 0 2023-10-18T20:30:33.581 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.581 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.581 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.581 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.582 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.582 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.582 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.569+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.582 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.570+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.583 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.570+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.583 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.570+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.583 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.570+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.583 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.570+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.583 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.570+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.570+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.570+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.570+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.588 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: sync status for source bucket shard: 0 2023-10-18T20:30:33.588 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket shard: 0 2023-10-18T20:30:33.588 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.589 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.589 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: parsed entry: id=00000000001.77.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-10-18T20:30:33.589 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7/obj00000000000000000049 2023-10-18T20:30:33.589 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: entry->timestamp=2023-10-18T20:30:29.555942+0000 2023-10-18T20:30:33.589 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.77.6 op=0 op_state=1 2023-10-18T20:30:33.590 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000049]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000049[0] 2023-10-18T20:30:33.590 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position=00000000001.77.6 2023-10-18T20:30:33.590 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: sync status for source bucket shard: 0 2023-10-18T20:30:33.590 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: sync status for source bucket shard: 0 2023-10-18T20:30:33.591 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket shard: 0 2023-10-18T20:30:33.591 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.591 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.591 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.591 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.592 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.592 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.572+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.592 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.573+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket shard: 0 2023-10-18T20:30:33.592 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: sync status for source bucket shard: 0 2023-10-18T20:30:33.593 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: sync status for source bucket shard: 0 2023-10-18T20:30:33.593 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket shard: 0 2023-10-18T20:30:33.593 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: sync status for source bucket shard: 0 2023-10-18T20:30:33.593 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket shard: 0 2023-10-18T20:30:33.593 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: sync status for source bucket shard: 0 2023-10-18T20:30:33.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: sync status for source bucket shard: 0 2023-10-18T20:30:33.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-18T20:30:33.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13/obj00000000000000000013 2023-10-18T20:30:33.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: entry->timestamp=2023-10-18T20:30:29.658694+0000 2023-10-18T20:30:33.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-18T20:30:33.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000013]: : injecting data sync error on key=obj00000000000000000013 2023-10-18T20:30:33.597 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-18T20:30:33.597 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000013]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.597 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000013]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13/obj00000000000000000013 2023-10-18T20:30:33.597 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:33.597 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.574+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.598 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2023-10-18T20:30:33.598 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15/obj00000000000000000037 2023-10-18T20:30:33.598 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: entry->timestamp=2023-10-18T20:30:29.093022+0000 2023-10-18T20:30:33.598 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.71.6 op=0 op_state=1 2023-10-18T20:30:33.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000037]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000037[0] 2023-10-18T20:30:33.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position=00000000001.71.6 2023-10-18T20:30:33.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-18T20:30:33.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25/obj00000000000000000000 2023-10-18T20:30:33.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: entry->timestamp=2023-10-18T20:30:29.337356+0000 2023-10-18T20:30:33.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-18T20:30:33.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-18T20:30:33.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23/obj00000000000000000025 2023-10-18T20:30:33.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: entry->timestamp=2023-10-18T20:30:29.176087+0000 2023-10-18T20:30:33.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-18T20:30:33.601 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-18T20:30:33.601 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12/obj00000000000000000022 2023-10-18T20:30:33.601 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: entry->timestamp=2023-10-18T20:30:29.823952+0000 2023-10-18T20:30:33.601 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.72.6 op=0 op_state=1 2023-10-18T20:30:33.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:entry[obj00000000000000000000]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000000[0] 2023-10-18T20:30:33.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-18T20:30:33.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000025]: : injecting data sync error on key=obj00000000000000000025 2023-10-18T20:30:33.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-18T20:30:33.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000022]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000022[0] 2023-10-18T20:30:33.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position=00000000002.72.6 2023-10-18T20:30:33.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000025]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.576+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000025]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23/obj00000000000000000025 2023-10-18T20:30:33.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.577+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: sync status for source bucket shard: 0 2023-10-18T20:30:33.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.577+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.604 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-18T20:30:33.604 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18/obj00000000000000000003 2023-10-18T20:30:33.604 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: entry->timestamp=2023-10-18T20:30:29.414948+0000 2023-10-18T20:30:33.604 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-10-18T20:30:33.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000003]: : injecting data sync error on key=obj00000000000000000003 2023-10-18T20:30:33.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position=00000000002.63.6 2023-10-18T20:30:33.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-18T20:30:33.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27/obj00000000000000000030 2023-10-18T20:30:33.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: entry->timestamp=2023-10-18T20:30:29.155170+0000 2023-10-18T20:30:33.606 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-18T20:30:33.606 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-18T20:30:33.606 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.579+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000005 2023-10-18T20:30:33.606 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:29.638569+0000 2023-10-18T20:30:33.607 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-18T20:30:33.607 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-18T20:30:33.607 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17/obj00000000000000000012 2023-10-18T20:30:33.608 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: entry->timestamp=2023-10-18T20:30:29.011594+0000 2023-10-18T20:30:33.608 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-18T20:30:33.608 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-18T20:30:33.608 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26/obj00000000000000000014 2023-10-18T20:30:33.608 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: entry->timestamp=2023-10-18T20:30:29.475146+0000 2023-10-18T20:30:33.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-18T20:30:33.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-18T20:30:33.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24/obj00000000000000000016 2023-10-18T20:30:33.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: entry->timestamp=2023-10-18T20:30:28.901250+0000 2023-10-18T20:30:33.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-10-18T20:30:33.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000003]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000003]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18/obj00000000000000000003 2023-10-18T20:30:33.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:entry[obj00000000000000000030]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000030[0] 2023-10-18T20:30:33.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-18T20:30:33.611 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: a sync operation returned error 2023-10-18T20:30:33.611 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000005]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000005[0] 2023-10-18T20:30:33.611 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-18T20:30:33.611 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000012]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000012[0] 2023-10-18T20:30:33.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-18T20:30:33.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:entry[obj00000000000000000014]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000014[0] 2023-10-18T20:30:33.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-18T20:30:33.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:entry[obj00000000000000000016]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000016[0] 2023-10-18T20:30:33.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: listing bilog for incremental sync; position=00000000002.55.6 2023-10-18T20:30:33.613 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: backing out with sync_status=-5 2023-10-18T20:30:33.613 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.613 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.580+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.613 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.582+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:entry[obj00000000000000000000]: success 2023-10-18T20:30:33.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.582+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25:4 marker=00000000002.69.6 timestamp=2023-10-18T20:30:29.337356+0000 2023-10-18T20:30:33.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.583+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: a sync operation returned error 2023-10-18T20:30:33.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.583+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: backing out with sync_status=-5 2023-10-18T20:30:33.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.583+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.583+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.615 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.583+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.615 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.584+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:entry[obj00000000000000000016]: success 2023-10-18T20:30:33.615 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.584+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24:4 marker=00000000002.55.6 timestamp=2023-10-18T20:30:28.901250+0000 2023-10-18T20:30:33.615 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.584+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000012]: success 2023-10-18T20:30:33.615 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.584+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17:4 marker=00000000002.69.6 timestamp=2023-10-18T20:30:29.011594+0000 2023-10-18T20:30:33.616 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.584+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:entry[obj00000000000000000030]: success 2023-10-18T20:30:33.616 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.584+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27:4 marker=00000000002.57.6 timestamp=2023-10-18T20:30:29.155170+0000 2023-10-18T20:30:33.616 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.584+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000049]: success 2023-10-18T20:30:33.616 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.584+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7:4 marker=00000000001.77.6 timestamp=2023-10-18T20:30:29.555942+0000 2023-10-18T20:30:33.617 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.585+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket shard: 0 2023-10-18T20:30:33.617 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.585+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.617 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.585+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000005]: success 2023-10-18T20:30:33.617 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.585+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16:4 marker=00000000002.48.6 timestamp=2023-10-18T20:30:29.638569+0000 2023-10-18T20:30:33.617 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.585+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:entry[obj00000000000000000014]: success 2023-10-18T20:30:33.618 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.585+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26:4 marker=00000000002.61.6 timestamp=2023-10-18T20:30:29.475146+0000 2023-10-18T20:30:33.622 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.587+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.622 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.588+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: sync status for source bucket shard: 0 2023-10-18T20:30:33.622 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.588+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-18T20:30:33.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0/obj00000000000000000006 2023-10-18T20:30:33.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: entry->timestamp=2023-10-18T20:30:29.904686+0000 2023-10-18T20:30:33.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-10-18T20:30:33.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000006]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000006[0] 2023-10-18T20:30:33.624 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position=00000000002.35.6 2023-10-18T20:30:33.624 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-18T20:30:33.624 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5/obj00000000000000000002 2023-10-18T20:30:33.624 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: entry->timestamp=2023-10-18T20:30:29.317039+0000 2023-10-18T20:30:33.625 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-18T20:30:33.625 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-18T20:30:33.625 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4/obj00000000000000000011 2023-10-18T20:30:33.625 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: entry->timestamp=2023-10-18T20:30:29.495848+0000 2023-10-18T20:30:33.625 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-10-18T20:30:33.626 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-10-18T20:30:33.626 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-18T20:30:33.626 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000011]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000011[0] 2023-10-18T20:30:33.626 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position=00000000002.59.6 2023-10-18T20:30:33.627 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.627 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.608+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000002]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5/obj00000000000000000002 2023-10-18T20:30:33.627 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.609+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: parsed entry: id=00000000001.75.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-10-18T20:30:33.627 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.609+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2/obj00000000000000000027 2023-10-18T20:30:33.627 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.609+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: entry->timestamp=2023-10-18T20:30:29.395120+0000 2023-10-18T20:30:33.628 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.609+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.75.6 op=0 op_state=1 2023-10-18T20:30:33.628 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.609+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000027]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000027[0] 2023-10-18T20:30:33.628 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.609+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position=00000000001.75.6 2023-10-18T20:30:33.628 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000011]: success 2023-10-18T20:30:33.629 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4:4 marker=00000000002.59.6 timestamp=2023-10-18T20:30:29.495848+0000 2023-10-18T20:30:33.629 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-10-18T20:30:33.629 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1/obj00000000000000000059 2023-10-18T20:30:33.629 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: entry->timestamp=2023-10-18T20:30:28.881315+0000 2023-10-18T20:30:33.629 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.71.6 op=0 op_state=1 2023-10-18T20:30:33.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-18T20:30:33.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14/obj00000000000000000020 2023-10-18T20:30:33.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: entry->timestamp=2023-10-18T20:30:28.922991+0000 2023-10-18T20:30:33.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-10-18T20:30:33.631 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000059]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000059[0] 2023-10-18T20:30:33.631 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position=00000000001.71.6 2023-10-18T20:30:33.633 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000006]: success 2023-10-18T20:30:33.633 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0:4 marker=00000000002.35.6 timestamp=2023-10-18T20:30:29.904686+0000 2023-10-18T20:30:33.633 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000020]: : injecting data sync error on key=obj00000000000000000020 2023-10-18T20:30:33.634 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position=00000000002.38.6 2023-10-18T20:30:33.634 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000020]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.634 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.611+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000020]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14/obj00000000000000000020 2023-10-18T20:30:33.634 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: parsed entry: id=00000000001.46.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-10-18T20:30:33.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11/obj00000000000000000031 2023-10-18T20:30:33.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: entry->timestamp=2023-10-18T20:30:29.073266+0000 2023-10-18T20:30:33.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.46.6 op=0 op_state=1 2023-10-18T20:30:33.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000031]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000031[0] 2023-10-18T20:30:33.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position=00000000001.46.6 2023-10-18T20:30:33.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-18T20:30:33.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9/obj00000000000000000008 2023-10-18T20:30:33.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: entry->timestamp=2023-10-18T20:30:29.743383+0000 2023-10-18T20:30:33.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-10-18T20:30:33.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-18T20:30:33.637 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10/obj00000000000000000015 2023-10-18T20:30:33.637 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: entry->timestamp=2023-10-18T20:30:29.722306+0000 2023-10-18T20:30:33.637 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-10-18T20:30:33.637 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000008]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000008[0] 2023-10-18T20:30:33.638 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position=00000000002.70.6 2023-10-18T20:30:33.638 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000015]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000015[0] 2023-10-18T20:30:33.638 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.613+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position=00000000002.39.6 2023-10-18T20:30:33.638 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.614+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: parsed entry: id=00000000001.70.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-18T20:30:33.638 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.614+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000017 2023-10-18T20:30:33.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.614+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: entry->timestamp=2023-10-18T20:30:29.883057+0000 2023-10-18T20:30:33.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.614+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.70.6 op=0 op_state=1 2023-10-18T20:30:33.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.614+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000017]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000017[0] 2023-10-18T20:30:33.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.614+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position=00000000001.70.6 2023-10-18T20:30:33.639 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2023-10-18T20:30:33.640 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19/obj00000000000000000042 2023-10-18T20:30:33.640 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: entry->timestamp=2023-10-18T20:30:29.255745+0000 2023-10-18T20:30:33.640 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-10-18T20:30:33.640 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000042]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000042[0] 2023-10-18T20:30:33.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: listing bilog for incremental sync; position=00000000001.42.6 2023-10-18T20:30:33.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000008]: success 2023-10-18T20:30:33.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9:4 marker=00000000002.70.6 timestamp=2023-10-18T20:30:29.743383+0000 2023-10-18T20:30:33.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000015]: success 2023-10-18T20:30:33.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.616+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10:4 marker=00000000002.39.6 timestamp=2023-10-18T20:30:29.722306+0000 2023-10-18T20:30:33.642 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.618+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-18T20:30:33.642 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.618+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22/obj00000000000000000009 2023-10-18T20:30:33.642 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.618+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: entry->timestamp=2023-10-18T20:30:29.575245+0000 2023-10-18T20:30:33.642 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.618+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-18T20:30:33.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.618+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000009]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000009[0] 2023-10-18T20:30:33.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.618+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-18T20:30:33.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-18T20:30:33.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20/obj00000000000000000010 2023-10-18T20:30:33.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: entry->timestamp=2023-10-18T20:30:29.235040+0000 2023-10-18T20:30:33.644 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.73.6 op=0 op_state=1 2023-10-18T20:30:33.644 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000010]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000010[0] 2023-10-18T20:30:33.644 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position=00000000002.73.6 2023-10-18T20:30:33.644 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-18T20:30:33.644 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21/obj00000000000000000001 2023-10-18T20:30:33.645 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: entry->timestamp=2023-10-18T20:30:29.803167+0000 2023-10-18T20:30:33.645 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-10-18T20:30:33.645 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000001]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000001[0] 2023-10-18T20:30:33.645 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.619+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: listing bilog for incremental sync; position=00000000002.71.6 2023-10-18T20:30:33.647 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000009]: success 2023-10-18T20:30:33.648 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22:4 marker=00000000002.56.6 timestamp=2023-10-18T20:30:29.575245+0000 2023-10-18T20:30:33.648 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-18T20:30:33.648 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28/obj00000000000000000021 2023-10-18T20:30:33.648 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: entry->timestamp=2023-10-18T20:30:28.987763+0000 2023-10-18T20:30:33.649 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-10-18T20:30:33.649 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000010]: success 2023-10-18T20:30:33.649 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20:4 marker=00000000002.73.6 timestamp=2023-10-18T20:30:29.235040+0000 2023-10-18T20:30:33.649 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000021]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000021[0] 2023-10-18T20:30:33.650 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.621+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: listing bilog for incremental sync; position=00000000002.78.6 2023-10-18T20:30:33.650 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.623+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000001]: success 2023-10-18T20:30:33.650 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.623+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21:4 marker=00000000002.71.6 timestamp=2023-10-18T20:30:29.803167+0000 2023-10-18T20:30:33.650 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.623+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000017]: success 2023-10-18T20:30:33.650 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.623+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8:4 marker=00000000001.70.6 timestamp=2023-10-18T20:30:29.883057+0000 2023-10-18T20:30:33.651 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.624+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: a sync operation returned error 2023-10-18T20:30:33.651 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.624+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: backing out with sync_status=-5 2023-10-18T20:30:33.651 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.624+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.651 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.624+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.651 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.624+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000021]: success 2023-10-18T20:30:33.652 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.624+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28:4 marker=00000000002.78.6 timestamp=2023-10-18T20:30:28.987763+0000 2023-10-18T20:30:33.652 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.625+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000037]: success 2023-10-18T20:30:33.652 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.625+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15:4 marker=00000000001.71.6 timestamp=2023-10-18T20:30:29.093022+0000 2023-10-18T20:30:33.652 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.627+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-18T20:30:33.653 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.627+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13/obj00000000000000000013 2023-10-18T20:30:33.653 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.627+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: entry->timestamp=2023-10-18T20:30:29.658694+0000 2023-10-18T20:30:33.653 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.627+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-18T20:30:33.653 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.627+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000013]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000013[0] 2023-10-18T20:30:33.654 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.627+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-18T20:30:33.654 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.627+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.654 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.628+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket shard: 0 2023-10-18T20:30:33.654 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.628+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.654 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.629+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000013]: success 2023-10-18T20:30:33.655 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.629+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13:4 marker=00000000002.48.6 timestamp=2023-10-18T20:30:29.658694+0000 2023-10-18T20:30:33.655 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.631+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000022]: success 2023-10-18T20:30:33.655 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.631+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12:4 marker=00000000002.72.6 timestamp=2023-10-18T20:30:29.823952+0000 2023-10-18T20:30:33.655 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.631+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-18T20:30:33.656 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.631+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23/obj00000000000000000025 2023-10-18T20:30:33.656 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.631+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: entry->timestamp=2023-10-18T20:30:29.176087+0000 2023-10-18T20:30:33.656 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.631+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-18T20:30:33.656 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.631+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000025]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000025[0] 2023-10-18T20:30:33.656 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.631+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-18T20:30:33.657 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.633+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000025]: success 2023-10-18T20:30:33.657 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.633+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23:4 marker=00000000002.40.6 timestamp=2023-10-18T20:30:29.176087+0000 2023-10-18T20:30:33.658 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.651+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: a sync operation returned error 2023-10-18T20:30:33.658 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.651+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: backing out with sync_status=-5 2023-10-18T20:30:33.658 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.651+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.659 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.651+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.659 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.651+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.659 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.652+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket shard: 0 2023-10-18T20:30:33.659 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.652+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.660 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.654+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: a sync operation returned error 2023-10-18T20:30:33.660 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.654+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: backing out with sync_status=-5 2023-10-18T20:30:33.660 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.654+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.660 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.654+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.660 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.654+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.661 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.655+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket shard: 0 2023-10-18T20:30:33.661 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.655+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.661 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.655+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000027]: success 2023-10-18T20:30:33.661 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.655+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2:4 marker=00000000001.75.6 timestamp=2023-10-18T20:30:29.395120+0000 2023-10-18T20:30:33.662 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.660+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000059]: success 2023-10-18T20:30:33.662 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.660+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1:4 marker=00000000001.71.6 timestamp=2023-10-18T20:30:28.881315+0000 2023-10-18T20:30:33.664 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.661+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000031]: success 2023-10-18T20:30:33.664 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.661+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11:4 marker=00000000001.46.6 timestamp=2023-10-18T20:30:29.073266+0000 2023-10-18T20:30:33.666 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.663+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000042]: success 2023-10-18T20:30:33.666 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.663+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19:4 marker=00000000001.42.6 timestamp=2023-10-18T20:30:29.255745+0000 2023-10-18T20:30:33.674 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.672+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-18T20:30:33.674 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.672+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18/obj00000000000000000003 2023-10-18T20:30:33.674 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.672+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: entry->timestamp=2023-10-18T20:30:29.414948+0000 2023-10-18T20:30:33.675 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.672+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-10-18T20:30:33.675 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.672+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000003]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000003[0] 2023-10-18T20:30:33.675 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.672+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position=00000000002.63.6 2023-10-18T20:30:33.676 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.673+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000003]: success 2023-10-18T20:30:33.676 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.673+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18:4 marker=00000000002.63.6 timestamp=2023-10-18T20:30:29.414948+0000 2023-10-18T20:30:33.696 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.694+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-18T20:30:33.696 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.694+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5/obj00000000000000000002 2023-10-18T20:30:33.697 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.694+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: entry->timestamp=2023-10-18T20:30:29.317039+0000 2023-10-18T20:30:33.697 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.694+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-18T20:30:33.697 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.694+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000002]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000002[0] 2023-10-18T20:30:33.697 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.694+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-18T20:30:33.698 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.696+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000002]: success 2023-10-18T20:30:33.698 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.696+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5:4 marker=00000000002.68.6 timestamp=2023-10-18T20:30:29.317039+0000 2023-10-18T20:30:33.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.698+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-18T20:30:33.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.698+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14/obj00000000000000000020 2023-10-18T20:30:33.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.698+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: entry->timestamp=2023-10-18T20:30:28.922991+0000 2023-10-18T20:30:33.701 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.698+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-10-18T20:30:33.701 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.698+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000020]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000020[0] 2023-10-18T20:30:33.701 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.698+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position=00000000002.38.6 2023-10-18T20:30:33.701 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.699+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000020]: success 2023-10-18T20:30:33.702 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.699+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14:4 marker=00000000002.38.6 timestamp=2023-10-18T20:30:28.922991+0000 2023-10-18T20:30:33.747 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.747 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.747 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.748 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket shard: 0 2023-10-18T20:30:33.748 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.748 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket shard: 0 2023-10-18T20:30:33.748 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.748 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.749 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket shard: 0 2023-10-18T20:30:33.749 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.749 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket shard: 0 2023-10-18T20:30:33.749 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.744+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.750 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.745+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.750 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.745+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.752 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.745+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket shard: 0 2023-10-18T20:30:33.752 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.745+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.753 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.745+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket shard: 0 2023-10-18T20:30:33.753 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.745+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.753 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.753 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.754 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.754 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.754 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.754 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.754 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket shard: 0 2023-10-18T20:30:33.755 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket shard: 0 2023-10-18T20:30:33.755 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.755 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.755 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 0 2023-10-18T20:30:33.756 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.746+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.756 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket shard: 0 2023-10-18T20:30:33.756 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket shard: 0 2023-10-18T20:30:33.756 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.756 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.757 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.757 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket shard: 0 2023-10-18T20:30:33.757 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.757 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.758 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.758 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.758 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.758 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-18T20:30:33.758 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5/obj00000000000000000004 2023-10-18T20:30:33.759 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: entry->timestamp=2023-10-18T20:30:30.531979+0000 2023-10-18T20:30:33.759 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-18T20:30:33.759 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.747+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000004]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000004[0] 2023-10-18T20:30:33.759 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-18T20:30:33.760 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: parsed entry: id=00000000002.92.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-18T20:30:33.760 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1/obj00000000000000000006 2023-10-18T20:30:33.760 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: entry->timestamp=2023-10-18T20:30:30.870070+0000 2023-10-18T20:30:33.760 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.92.6 op=0 op_state=1 2023-10-18T20:30:33.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000006]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000006[0] 2023-10-18T20:30:33.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position=00000000002.92.6 2023-10-18T20:30:33.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket shard: 0 2023-10-18T20:30:33.762 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.762 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.762 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.762 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.748+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.763 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:33.763 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.763 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: sync status for source bucket shard: 0 2023-10-18T20:30:33.763 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket shard: 0 2023-10-18T20:30:33.763 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket shard: 0 2023-10-18T20:30:33.764 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket shard: 0 2023-10-18T20:30:33.764 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket shard: 0 2023-10-18T20:30:33.764 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: sync status for source bucket shard: 0 2023-10-18T20:30:33.764 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket shard: 0 2023-10-18T20:30:33.764 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.765 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.765 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.765 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.765 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.766 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.766 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.766 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.749+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.766 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.766 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.767 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.767 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.767 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-18T20:30:33.767 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9/obj00000000000000000000 2023-10-18T20:30:33.768 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: entry->timestamp=2023-10-18T20:30:30.964776+0000 2023-10-18T20:30:33.768 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-18T20:30:33.768 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.768 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-18T20:30:33.769 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000022 2023-10-18T20:30:33.769 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: entry->timestamp=2023-10-18T20:30:30.350297+0000 2023-10-18T20:30:33.769 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-18T20:30:33.772 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000000]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000000[0] 2023-10-18T20:30:33.772 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-18T20:30:33.773 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000022]: : injecting data sync error on key=obj00000000000000000022 2023-10-18T20:30:33.773 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.750+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-18T20:30:33.773 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000022]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.773 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000022]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000022 2023-10-18T20:30:33.774 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-18T20:30:33.774 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10/obj00000000000000000021 2023-10-18T20:30:33.774 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: entry->timestamp=2023-10-18T20:30:30.591392+0000 2023-10-18T20:30:33.774 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-18T20:30:33.775 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000021]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000021[0] 2023-10-18T20:30:33.775 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-18T20:30:33.775 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-18T20:30:33.775 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14/obj00000000000000000018 2023-10-18T20:30:33.776 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: entry->timestamp=2023-10-18T20:30:30.926286+0000 2023-10-18T20:30:33.776 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.83.6 op=0 op_state=1 2023-10-18T20:30:33.776 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000018]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000018[0] 2023-10-18T20:30:33.776 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.751+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position=00000000002.83.6 2023-10-18T20:30:33.776 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: sync status for source bucket shard: 0 2023-10-18T20:30:33.777 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: sync status for source bucket shard: 0 2023-10-18T20:30:33.777 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: sync status for source bucket shard: 0 2023-10-18T20:30:33.777 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.777 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.778 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.778 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-18T20:30:33.778 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18/obj00000000000000000016 2023-10-18T20:30:33.778 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: entry->timestamp=2023-10-18T20:30:30.430418+0000 2023-10-18T20:30:33.779 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-18T20:30:33.779 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000016]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000016[0] 2023-10-18T20:30:33.779 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-18T20:30:33.779 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-18T20:30:33.779 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19/obj00000000000000000003 2023-10-18T20:30:33.780 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: entry->timestamp=2023-10-18T20:30:30.283504+0000 2023-10-18T20:30:33.780 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-10-18T20:30:33.780 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000004]: success 2023-10-18T20:30:33.780 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5:5 marker=00000000002.42.6 timestamp=2023-10-18T20:30:30.531979+0000 2023-10-18T20:30:33.781 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000006]: success 2023-10-18T20:30:33.781 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1:5 marker=00000000002.92.6 timestamp=2023-10-18T20:30:30.870070+0000 2023-10-18T20:30:33.781 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2023-10-18T20:30:33.781 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17/obj00000000000000000162 2023-10-18T20:30:33.781 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: entry->timestamp=2023-10-18T20:30:31.108982+0000 2023-10-18T20:30:33.782 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.78.6 op=0 op_state=1 2023-10-18T20:30:33.782 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000003]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000003[0] 2023-10-18T20:30:33.782 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: listing bilog for incremental sync; position=00000000002.67.6 2023-10-18T20:30:33.782 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000162]: : injecting data sync error on key=obj00000000000000000162 2023-10-18T20:30:33.782 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position=00000000001.78.6 2023-10-18T20:30:33.783 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: parsed entry: id=00000000001.65.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-10-18T20:30:33.783 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20/obj00000000000000000423 2023-10-18T20:30:33.783 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: entry->timestamp=2023-10-18T20:30:30.240139+0000 2023-10-18T20:30:33.783 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.754+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.65.6 op=0 op_state=1 2023-10-18T20:30:33.784 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000162]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.784 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000162]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17/obj00000000000000000162 2023-10-18T20:30:33.784 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000423]: : injecting data sync error on key=obj00000000000000000423 2023-10-18T20:30:33.784 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position=00000000001.65.6 2023-10-18T20:30:33.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: sync status for source bucket shard: 0 2023-10-18T20:30:33.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000423]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000423]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20/obj00000000000000000423 2023-10-18T20:30:33.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.786 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000021]: success 2023-10-18T20:30:33.786 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10:5 marker=00000000002.47.6 timestamp=2023-10-18T20:30:30.591392+0000 2023-10-18T20:30:33.786 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:entry[obj00000000000000000000]: success 2023-10-18T20:30:33.786 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.755+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9:5 marker=00000000002.69.6 timestamp=2023-10-18T20:30:30.964776+0000 2023-10-18T20:30:33.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket shard: 0 2023-10-18T20:30:33.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: sync status for source bucket shard: 0 2023-10-18T20:30:33.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: sync status for source bucket shard: 0 2023-10-18T20:30:33.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-18T20:30:33.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15/obj00000000000000000007 2023-10-18T20:30:33.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: entry->timestamp=2023-10-18T20:30:30.672589+0000 2023-10-18T20:30:33.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-18T20:30:33.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.790 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000007]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000007[0] 2023-10-18T20:30:33.790 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.756+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-18T20:30:33.790 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.758+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: sync status for source bucket shard: 0 2023-10-18T20:30:33.790 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.758+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.791 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.758+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: sync status for source bucket shard: 0 2023-10-18T20:30:33.791 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.758+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.791 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.760+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: sync status for source bucket shard: 0 2023-10-18T20:30:33.791 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.760+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.791 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.761+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000003]: success 2023-10-18T20:30:33.792 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.761+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19:5 marker=00000000002.67.6 timestamp=2023-10-18T20:30:30.283504+0000 2023-10-18T20:30:33.792 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.761+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000007]: success 2023-10-18T20:30:33.792 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.761+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15:5 marker=00000000002.56.6 timestamp=2023-10-18T20:30:30.672589+0000 2023-10-18T20:30:33.792 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.761+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000016]: success 2023-10-18T20:30:33.793 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.761+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18:5 marker=00000000002.42.6 timestamp=2023-10-18T20:30:30.430418+0000 2023-10-18T20:30:33.793 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-18T20:30:33.793 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28/obj00000000000000000050 2023-10-18T20:30:33.793 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: entry->timestamp=2023-10-18T20:30:30.771456+0000 2023-10-18T20:30:33.793 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-10-18T20:30:33.794 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000050]: : injecting data sync error on key=obj00000000000000000050 2023-10-18T20:30:33.794 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: listing bilog for incremental sync; position=00000000002.44.6 2023-10-18T20:30:33.794 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000050]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.794 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000050]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28/obj00000000000000000050 2023-10-18T20:30:33.795 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-10-18T20:30:33.795 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25/obj00000000000000000283 2023-10-18T20:30:33.795 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: entry->timestamp=2023-10-18T20:30:31.148673+0000 2023-10-18T20:30:33.795 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-18T20:30:33.795 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:entry[obj00000000000000000283]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000283[0] 2023-10-18T20:30:33.796 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.762+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-18T20:30:33.796 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.763+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: parsed entry: id=00000000001.55.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-10-18T20:30:33.796 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.763+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30/obj00000000000000000133 2023-10-18T20:30:33.796 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.763+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: entry->timestamp=2023-10-18T20:30:30.850838+0000 2023-10-18T20:30:33.797 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.763+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.55.6 op=0 op_state=1 2023-10-18T20:30:33.797 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.763+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:entry[obj00000000000000000133]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000133[0] 2023-10-18T20:30:33.797 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.763+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: listing bilog for incremental sync; position=00000000001.55.6 2023-10-18T20:30:33.800 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-18T20:30:33.801 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0/obj00000000000000000019 2023-10-18T20:30:33.801 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: entry->timestamp=2023-10-18T20:30:30.451126+0000 2023-10-18T20:30:33.801 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-18T20:30:33.801 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-10-18T20:30:33.802 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-18T20:30:33.802 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-18T20:30:33.802 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3/obj00000000000000000005 2023-10-18T20:30:33.802 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: entry->timestamp=2023-10-18T20:30:30.792124+0000 2023-10-18T20:30:33.802 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-10-18T20:30:33.803 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: parsed entry: id=00000000001.76.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-10-18T20:30:33.803 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2/obj00000000000000000467 2023-10-18T20:30:33.803 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: entry->timestamp=2023-10-18T20:30:31.027459+0000 2023-10-18T20:30:33.803 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.76.6 op=0 op_state=1 2023-10-18T20:30:33.804 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.804 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000019]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0/obj00000000000000000019 2023-10-18T20:30:33.804 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000005]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000005[0] 2023-10-18T20:30:33.804 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position=00000000002.64.6 2023-10-18T20:30:33.804 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000467]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000467[0] 2023-10-18T20:30:33.805 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.786+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position=00000000001.76.6 2023-10-18T20:30:33.805 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.788+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000005]: success 2023-10-18T20:30:33.805 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.788+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3:5 marker=00000000002.64.6 timestamp=2023-10-18T20:30:30.792124+0000 2023-10-18T20:30:33.805 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2023-10-18T20:30:33.806 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7/obj00000000000000000317 2023-10-18T20:30:33.806 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: entry->timestamp=2023-10-18T20:30:31.294926+0000 2023-10-18T20:30:33.806 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-10-18T20:30:33.806 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000317]: : injecting data sync error on key=obj00000000000000000317 2023-10-18T20:30:33.806 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position=00000000001.43.6 2023-10-18T20:30:33.807 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-18T20:30:33.807 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11/obj00000000000000000009 2023-10-18T20:30:33.807 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: entry->timestamp=2023-10-18T20:30:30.693410+0000 2023-10-18T20:30:33.807 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-18T20:30:33.808 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000317]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.808 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000317]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7/obj00000000000000000317 2023-10-18T20:30:33.808 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000009]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000009[0] 2023-10-18T20:30:33.808 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.789+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-18T20:30:33.809 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.790+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-10-18T20:30:33.809 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.790+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6/obj00000000000000000023 2023-10-18T20:30:33.809 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.790+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: entry->timestamp=2023-10-18T20:30:30.261228+0000 2023-10-18T20:30:33.809 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.790+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-10-18T20:30:33.810 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.790+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000023]: : injecting data sync error on key=obj00000000000000000023 2023-10-18T20:30:33.810 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.790+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position=00000000002.64.6 2023-10-18T20:30:33.810 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.790+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000023]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.810 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.790+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000023]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6/obj00000000000000000023 2023-10-18T20:30:33.811 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.791+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000009]: success 2023-10-18T20:30:33.811 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.791+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11:5 marker=00000000002.61.6 timestamp=2023-10-18T20:30:30.693410+0000 2023-10-18T20:30:33.811 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-18T20:30:33.811 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22/obj00000000000000000012 2023-10-18T20:30:33.811 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: entry->timestamp=2023-10-18T20:30:31.213786+0000 2023-10-18T20:30:33.812 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-10-18T20:30:33.812 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000012]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000012[0] 2023-10-18T20:30:33.812 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: listing bilog for incremental sync; position=00000000002.44.6 2023-10-18T20:30:33.812 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: a sync operation returned error 2023-10-18T20:30:33.813 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: backing out with sync_status=-5 2023-10-18T20:30:33.813 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.813 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.793+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.813 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.794+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.813 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.794+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 0 2023-10-18T20:30:33.814 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.794+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.814 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.794+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000012]: success 2023-10-18T20:30:33.814 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.794+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22:5 marker=00000000002.44.6 timestamp=2023-10-18T20:30:31.213786+0000 2023-10-18T20:30:33.814 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.795+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-18T20:30:33.814 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.795+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000017 2023-10-18T20:30:33.815 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.795+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:30.612083+0000 2023-10-18T20:30:33.815 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.795+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-18T20:30:33.815 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.795+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2023-10-18T20:30:33.815 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.795+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-18T20:30:33.815 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.795+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.816 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.795+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000017 2023-10-18T20:30:33.816 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.798+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: parsed entry: id=00000000002.116.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-18T20:30:33.816 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.798+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24/obj00000000000000000011 2023-10-18T20:30:33.816 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.798+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: entry->timestamp=2023-10-18T20:30:30.510795+0000 2023-10-18T20:30:33.817 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.798+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.116.6 op=0 op_state=1 2023-10-18T20:30:33.817 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.798+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:entry[obj00000000000000000011]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000011[0] 2023-10-18T20:30:33.817 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.798+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: listing bilog for incremental sync; position=00000000002.116.6 2023-10-18T20:30:33.820 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.799+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: a sync operation returned error 2023-10-18T20:30:33.821 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.799+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: backing out with sync_status=-5 2023-10-18T20:30:33.821 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.799+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.821 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.799+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.821 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.799+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.821 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.799+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:entry[obj00000000000000000011]: success 2023-10-18T20:30:33.822 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.799+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24:5 marker=00000000002.116.6 timestamp=2023-10-18T20:30:30.510795+0000 2023-10-18T20:30:33.822 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.799+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket shard: 0 2023-10-18T20:30:33.822 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.799+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.823 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-18T20:30:33.823 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21/obj00000000000000000002 2023-10-18T20:30:33.823 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: entry->timestamp=2023-10-18T20:30:31.128065+0000 2023-10-18T20:30:33.823 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-10-18T20:30:33.824 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000002]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000002[0] 2023-10-18T20:30:33.824 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: listing bilog for incremental sync; position=00000000002.41.6 2023-10-18T20:30:33.824 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: parsed entry: id=00000000001.118.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-10-18T20:30:33.824 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12/obj00000000000000000271 2023-10-18T20:30:33.825 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: entry->timestamp=2023-10-18T20:30:31.234381+0000 2023-10-18T20:30:33.825 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.118.6 op=0 op_state=1 2023-10-18T20:30:33.825 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000271]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000271[0] 2023-10-18T20:30:33.825 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.801+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position=00000000001.118.6 2023-10-18T20:30:33.826 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-10-18T20:30:33.826 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27/obj00000000000000000061 2023-10-18T20:30:33.826 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: entry->timestamp=2023-10-18T20:30:30.714009+0000 2023-10-18T20:30:33.826 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-10-18T20:30:33.826 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-18T20:30:33.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23/obj00000000000000000001 2023-10-18T20:30:33.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: entry->timestamp=2023-10-18T20:30:30.372318+0000 2023-10-18T20:30:33.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-18T20:30:33.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: a sync operation returned error 2023-10-18T20:30:33.828 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:entry[obj00000000000000000061]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000061[0] 2023-10-18T20:30:33.828 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: listing bilog for incremental sync; position=00000000001.39.6 2023-10-18T20:30:33.828 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000001]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000001[0] 2023-10-18T20:30:33.828 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-18T20:30:33.828 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: backing out with sync_status=-5 2023-10-18T20:30:33.829 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.829 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.802+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.829 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.803+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-10-18T20:30:33.829 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.803+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29/obj00000000000000000029 2023-10-18T20:30:33.829 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.803+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: entry->timestamp=2023-10-18T20:30:31.047138+0000 2023-10-18T20:30:33.830 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.803+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-10-18T20:30:33.830 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.803+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:entry[obj00000000000000000029]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000029[0] 2023-10-18T20:30:33.830 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.803+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: listing bilog for incremental sync; position=00000000001.66.6 2023-10-18T20:30:33.830 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.803+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000002]: success 2023-10-18T20:30:33.831 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.803+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21:5 marker=00000000002.41.6 timestamp=2023-10-18T20:30:31.128065+0000 2023-10-18T20:30:33.831 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.804+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: a sync operation returned error 2023-10-18T20:30:33.831 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.804+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: backing out with sync_status=-5 2023-10-18T20:30:33.831 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.804+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.831 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.804+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.832 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.805+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.832 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.806+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket shard: 0 2023-10-18T20:30:33.832 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.806+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.832 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.806+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000001]: success 2023-10-18T20:30:33.832 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.806+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23:5 marker=00000000002.48.6 timestamp=2023-10-18T20:30:30.372318+0000 2023-10-18T20:30:33.833 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.808+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:entry[obj00000000000000000018]: success 2023-10-18T20:30:33.833 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.808+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14:5 marker=00000000002.83.6 timestamp=2023-10-18T20:30:30.926286+0000 2023-10-18T20:30:33.833 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.809+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000271]: success 2023-10-18T20:30:33.833 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.809+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12:5 marker=00000000001.118.6 timestamp=2023-10-18T20:30:31.234381+0000 2023-10-18T20:30:33.834 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.809+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.834 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.810+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: sync status for source bucket shard: 0 2023-10-18T20:30:33.834 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.810+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.834 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.812+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:entry[obj00000000000000000133]: success 2023-10-18T20:30:33.834 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.812+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30:5 marker=00000000001.55.6 timestamp=2023-10-18T20:30:30.850838+0000 2023-10-18T20:30:33.835 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.812+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:entry[obj00000000000000000283]: success 2023-10-18T20:30:33.835 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.812+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25:5 marker=00000000001.49.6 timestamp=2023-10-18T20:30:31.148673+0000 2023-10-18T20:30:33.838 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.828+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: a sync operation returned error 2023-10-18T20:30:33.839 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.828+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: backing out with sync_status=-5 2023-10-18T20:30:33.839 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.828+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.839 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.828+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.839 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.828+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.840 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.828+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket shard: 0 2023-10-18T20:30:33.840 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.828+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.840 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.831+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: a sync operation returned error 2023-10-18T20:30:33.840 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.831+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: backing out with sync_status=-5 2023-10-18T20:30:33.840 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.831+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.841 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.831+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.841 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.831+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.841 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.831+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000467]: success 2023-10-18T20:30:33.841 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.831+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2:5 marker=00000000001.76.6 timestamp=2023-10-18T20:30:31.027459+0000 2023-10-18T20:30:33.841 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.832+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket shard: 0 2023-10-18T20:30:33.842 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.832+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.842 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.832+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: a sync operation returned error 2023-10-18T20:30:33.842 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.832+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: backing out with sync_status=-5 2023-10-18T20:30:33.842 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.832+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.843 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.832+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.843 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.833+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.843 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.833+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket shard: 0 2023-10-18T20:30:33.843 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.833+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.843 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.836+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-18T20:30:33.844 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.836+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000022 2023-10-18T20:30:33.844 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.836+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: entry->timestamp=2023-10-18T20:30:30.350297+0000 2023-10-18T20:30:33.844 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.836+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-18T20:30:33.844 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.836+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000022]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000022[0] 2023-10-18T20:30:33.844 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.836+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-18T20:30:33.846 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.837+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: a sync operation returned error 2023-10-18T20:30:33.846 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.837+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: backing out with sync_status=-5 2023-10-18T20:30:33.846 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.837+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.846 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.837+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.847 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.837+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000022]: success 2023-10-18T20:30:33.847 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.837+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8:5 marker=00000000002.43.6 timestamp=2023-10-18T20:30:30.350297+0000 2023-10-18T20:30:33.847 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.838+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.847 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.838+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:33.848 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.838+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.848 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.842+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: parsed entry: id=00000000001.78.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2023-10-18T20:30:33.848 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.842+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17/obj00000000000000000162 2023-10-18T20:30:33.848 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.842+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: entry->timestamp=2023-10-18T20:30:31.108982+0000 2023-10-18T20:30:33.848 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.842+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.78.6 op=0 op_state=1 2023-10-18T20:30:33.849 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.842+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000162]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000162[0] 2023-10-18T20:30:33.849 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.842+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position=00000000001.78.6 2023-10-18T20:30:33.850 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.848+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: parsed entry: id=00000000001.65.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-10-18T20:30:33.850 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.848+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20/obj00000000000000000423 2023-10-18T20:30:33.851 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.848+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: entry->timestamp=2023-10-18T20:30:30.240139+0000 2023-10-18T20:30:33.851 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.848+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.65.6 op=0 op_state=1 2023-10-18T20:30:33.851 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.848+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000423]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000423[0] 2023-10-18T20:30:33.851 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.848+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position=00000000001.65.6 2023-10-18T20:30:33.852 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.849+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:entry[obj00000000000000000061]: success 2023-10-18T20:30:33.852 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.849+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27:5 marker=00000000001.39.6 timestamp=2023-10-18T20:30:30.714009+0000 2023-10-18T20:30:33.853 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.851+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:entry[obj00000000000000000029]: success 2023-10-18T20:30:33.853 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.851+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29:5 marker=00000000001.66.6 timestamp=2023-10-18T20:30:31.047138+0000 2023-10-18T20:30:33.855 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.853+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-18T20:30:33.855 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.853+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28/obj00000000000000000050 2023-10-18T20:30:33.855 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.853+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: entry->timestamp=2023-10-18T20:30:30.771456+0000 2023-10-18T20:30:33.856 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.853+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-10-18T20:30:33.856 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.853+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000050]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000050[0] 2023-10-18T20:30:33.856 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.853+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: listing bilog for incremental sync; position=00000000002.44.6 2023-10-18T20:30:33.857 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.854+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000050]: success 2023-10-18T20:30:33.857 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.855+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28:5 marker=00000000002.44.6 timestamp=2023-10-18T20:30:30.771456+0000 2023-10-18T20:30:33.873 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.871+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-18T20:30:33.874 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.871+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0/obj00000000000000000019 2023-10-18T20:30:33.874 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.871+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: entry->timestamp=2023-10-18T20:30:30.451126+0000 2023-10-18T20:30:33.874 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.871+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-18T20:30:33.874 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.871+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000019]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000019[0] 2023-10-18T20:30:33.875 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.871+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-18T20:30:33.875 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.872+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000019]: success 2023-10-18T20:30:33.875 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.872+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0:5 marker=00000000002.54.6 timestamp=2023-10-18T20:30:30.451126+0000 2023-10-18T20:30:33.876 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.874+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2023-10-18T20:30:33.876 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.874+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7/obj00000000000000000317 2023-10-18T20:30:33.877 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.874+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: entry->timestamp=2023-10-18T20:30:31.294926+0000 2023-10-18T20:30:33.877 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.874+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-10-18T20:30:33.877 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.874+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000317]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000317[0] 2023-10-18T20:30:33.877 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.874+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position=00000000001.43.6 2023-10-18T20:30:33.878 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.876+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-10-18T20:30:33.878 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.876+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6/obj00000000000000000023 2023-10-18T20:30:33.878 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.876+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: entry->timestamp=2023-10-18T20:30:30.261228+0000 2023-10-18T20:30:33.879 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.876+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-10-18T20:30:33.879 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.876+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000023]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000023[0] 2023-10-18T20:30:33.879 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.876+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position=00000000002.64.6 2023-10-18T20:30:33.882 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-18T20:30:33.882 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000017 2023-10-18T20:30:33.882 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:30.612083+0000 2023-10-18T20:30:33.883 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-18T20:30:33.883 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2023-10-18T20:30:33.883 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-18T20:30:33.883 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.884 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000017 2023-10-18T20:30:33.884 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000023]: success 2023-10-18T20:30:33.884 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.880+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6:5 marker=00000000002.64.6 timestamp=2023-10-18T20:30:30.261228+0000 2023-10-18T20:30:33.891 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.889+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000162]: success 2023-10-18T20:30:33.891 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.889+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17:5 marker=00000000001.78.6 timestamp=2023-10-18T20:30:31.108982+0000 2023-10-18T20:30:33.897 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.895+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000423]: success 2023-10-18T20:30:33.897 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.895+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20:5 marker=00000000001.65.6 timestamp=2023-10-18T20:30:30.240139+0000 2023-10-18T20:30:33.920 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.918+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000317]: success 2023-10-18T20:30:33.921 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.918+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7:5 marker=00000000001.43.6 timestamp=2023-10-18T20:30:31.294926+0000 2023-10-18T20:30:33.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.922+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: a sync operation returned error 2023-10-18T20:30:33.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.922+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: backing out with sync_status=-5 2023-10-18T20:30:33.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.922+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:33.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.922+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:33.925 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.922+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:33.925 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.922+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:33.926 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.922+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:33.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.964+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-18T20:30:33.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.964+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000017 2023-10-18T20:30:33.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.964+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:30.612083+0000 2023-10-18T20:30:33.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.964+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-18T20:30:33.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.964+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2023-10-18T20:30:33.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.964+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-18T20:30:33.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.964+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:33.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:33.964+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000017 2023-10-18T20:30:34.008 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.006+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: a sync operation returned error 2023-10-18T20:30:34.008 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.006+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: backing out with sync_status=-5 2023-10-18T20:30:34.009 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.006+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:34.009 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.006+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:34.009 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.006+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.010 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.006+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:34.010 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.006+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.050 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.048+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-18T20:30:34.051 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.048+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000017 2023-10-18T20:30:34.051 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.048+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:30.612083+0000 2023-10-18T20:30:34.051 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.048+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-18T20:30:34.052 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.048+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000017[0] 2023-10-18T20:30:34.052 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.048+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-18T20:30:34.053 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.049+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000017]: success 2023-10-18T20:30:34.053 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.049+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16:5 marker=00000000002.57.6 timestamp=2023-10-18T20:30:30.612083+0000 2023-10-18T20:30:34.099 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.100 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.100 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.100 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: sync status for source bucket shard: 0 2023-10-18T20:30:34.101 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.101 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: sync status for source bucket shard: 0 2023-10-18T20:30:34.102 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: sync status for source bucket shard: 0 2023-10-18T20:30:34.102 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.103 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.103 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.103 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.104 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.096+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.106 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.097+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: sync status for source bucket shard: 0 2023-10-18T20:30:34.107 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.097+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: sync status for source bucket shard: 0 2023-10-18T20:30:34.107 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.097+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.107 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.097+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.107 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.097+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.097+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.097+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.097+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.097+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: sync status for source bucket shard: 0 2023-10-18T20:30:34.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: sync status for source bucket shard: 0 2023-10-18T20:30:34.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-18T20:30:34.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4/obj00000000000000000033 2023-10-18T20:30:34.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: entry->timestamp=2023-10-18T20:30:32.799608+0000 2023-10-18T20:30:34.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-10-18T20:30:34.111 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.111 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: parsed entry: id=00000000001.55.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2023-10-18T20:30:34.111 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0/obj00000000000000000060 2023-10-18T20:30:34.111 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: entry->timestamp=2023-10-18T20:30:32.403090+0000 2023-10-18T20:30:34.111 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.55.6 op=0 op_state=1 2023-10-18T20:30:34.112 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000033]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000033[0] 2023-10-18T20:30:34.112 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: listing bilog for incremental sync; position=00000000002.76.6 2023-10-18T20:30:34.112 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000060]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000060[0] 2023-10-18T20:30:34.112 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.098+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: listing bilog for incremental sync; position=00000000001.55.6 2023-10-18T20:30:34.113 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.113 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: sync status for source bucket shard: 0 2023-10-18T20:30:34.113 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: sync status for source bucket shard: 0 2023-10-18T20:30:34.113 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: sync status for source bucket shard: 0 2023-10-18T20:30:34.114 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: sync status for source bucket shard: 0 2023-10-18T20:30:34.114 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.114 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:9]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.114 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.115 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.115 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: sync status for source bucket shard: 0 2023-10-18T20:30:34.115 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.115 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.115 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.116 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: sync status for source bucket shard: 0 2023-10-18T20:30:34.116 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.116 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.099+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: sync status for source bucket shard: 0 2023-10-18T20:30:34.116 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.100+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.116 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.100+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.117 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.100+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: sync status for source bucket shard: 0 2023-10-18T20:30:34.117 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.100+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:14]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.117 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.100+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.117 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.100+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.117 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.100+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.118 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.100+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: sync status for source bucket shard: 0 2023-10-18T20:30:34.118 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.100+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.118 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.101+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: sync status for source bucket shard: 0 2023-10-18T20:30:34.118 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.101+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.101+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: sync status for source bucket shard: 0 2023-10-18T20:30:34.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.101+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: sync status for source bucket shard: 0 2023-10-18T20:30:34.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.101+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.101+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.119 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2023-10-18T20:30:34.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8/obj00000000000000000048 2023-10-18T20:30:34.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: entry->timestamp=2023-10-18T20:30:32.128085+0000 2023-10-18T20:30:34.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-18T20:30:34.120 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.121 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000048]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000048[0] 2023-10-18T20:30:34.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-18T20:30:34.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:34.122 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: sync status for source bucket shard: 0 2023-10-18T20:30:34.123 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.123 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.123 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-18T20:30:34.123 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7/obj00000000000000000016 2023-10-18T20:30:34.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: entry->timestamp=2023-10-18T20:30:32.576239+0000 2023-10-18T20:30:34.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-10-18T20:30:34.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-18T20:30:34.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3/obj00000000000000000001 2023-10-18T20:30:34.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: entry->timestamp=2023-10-18T20:30:31.840825+0000 2023-10-18T20:30:34.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-18T20:30:34.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000016]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000016[0] 2023-10-18T20:30:34.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: listing bilog for incremental sync; position=00000000002.66.6 2023-10-18T20:30:34.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.103+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000001]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000001[0] 2023-10-18T20:30:34.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.104+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-18T20:30:34.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.104+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.104+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.104+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.104+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:entry[obj00000000000000000033]: success 2023-10-18T20:30:34.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.104+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:4:6 marker=00000000002.76.6 timestamp=2023-10-18T20:30:32.799608+0000 2023-10-18T20:30:34.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: sync status for source bucket shard: 0 2023-10-18T20:30:34.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: sync status for source bucket shard: 0 2023-10-18T20:30:34.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: sync status for source bucket shard: 0 2023-10-18T20:30:34.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: sync status for source bucket shard: 0 2023-10-18T20:30:34.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: sync status for source bucket shard: 0 2023-10-18T20:30:34.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: sync status for source bucket shard: 0 2023-10-18T20:30:34.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: sync status for source bucket shard: 0 2023-10-18T20:30:34.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: sync status for source bucket shard: 0 2023-10-18T20:30:34.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: sync status for source bucket shard: 0 2023-10-18T20:30:34.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: parsed entry: id=00000000001.68.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-10-18T20:30:34.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5/obj00000000000000000065 2023-10-18T20:30:34.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: entry->timestamp=2023-10-18T20:30:32.637722+0000 2023-10-18T20:30:34.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.68.6 op=0 op_state=1 2023-10-18T20:30:34.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000065]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000065[0] 2023-10-18T20:30:34.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: listing bilog for incremental sync; position=00000000001.68.6 2023-10-18T20:30:34.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-18T20:30:34.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10/obj00000000000000000026 2023-10-18T20:30:34.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.107+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: entry->timestamp=2023-10-18T20:30:32.020265+0000 2023-10-18T20:30:34.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-10-18T20:30:34.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-18T20:30:34.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11/obj00000000000000000030 2023-10-18T20:30:34.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: entry->timestamp=2023-10-18T20:30:32.742308+0000 2023-10-18T20:30:34.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-10-18T20:30:34.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: parsed entry: id=00000000001.62.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2023-10-18T20:30:34.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12/obj00000000000000000062 2023-10-18T20:30:34.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: entry->timestamp=2023-10-18T20:30:31.736585+0000 2023-10-18T20:30:34.141 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.62.6 op=0 op_state=1 2023-10-18T20:30:34.141 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-18T20:30:34.141 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17/obj00000000000000000002 2023-10-18T20:30:34.141 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: entry->timestamp=2023-10-18T20:30:31.919980+0000 2023-10-18T20:30:34.141 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-18T20:30:34.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-18T20:30:34.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18/obj00000000000000000013 2023-10-18T20:30:34.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: entry->timestamp=2023-10-18T20:30:32.103734+0000 2023-10-18T20:30:34.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.83.6 op=0 op_state=1 2023-10-18T20:30:34.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000026]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000026[0] 2023-10-18T20:30:34.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: listing bilog for incremental sync; position=00000000002.49.6 2023-10-18T20:30:34.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000030]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000030[0] 2023-10-18T20:30:34.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: listing bilog for incremental sync; position=00000000002.53.6 2023-10-18T20:30:34.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000062]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000062[0] 2023-10-18T20:30:34.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: listing bilog for incremental sync; position=00000000001.62.6 2023-10-18T20:30:34.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000002]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000002[0] 2023-10-18T20:30:34.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-18T20:30:34.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-10-18T20:30:34.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15/obj00000000000000000024 2023-10-18T20:30:34.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: entry->timestamp=2023-10-18T20:30:32.820989+0000 2023-10-18T20:30:34.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-10-18T20:30:34.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000013]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000013[0] 2023-10-18T20:30:34.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: listing bilog for incremental sync; position=00000000002.83.6 2023-10-18T20:30:34.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.108+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: sync status for source bucket shard: 0 2023-10-18T20:30:34.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000024]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000024[0] 2023-10-18T20:30:34.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: listing bilog for incremental sync; position=00000000002.66.6 2023-10-18T20:30:34.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.148 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.148 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.148 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.148 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.148 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-18T20:30:34.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20/obj00000000000000000006 2023-10-18T20:30:34.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: entry->timestamp=2023-10-18T20:30:31.862226+0000 2023-10-18T20:30:34.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-10-18T20:30:34.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: parsed entry: id=00000000002.101.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-18T20:30:34.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13/obj00000000000000000009 2023-10-18T20:30:34.150 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: entry->timestamp=2023-10-18T20:30:32.720856+0000 2023-10-18T20:30:34.150 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.101.6 op=0 op_state=1 2023-10-18T20:30:34.150 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000006]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000006[0] 2023-10-18T20:30:34.150 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: listing bilog for incremental sync; position=00000000002.64.6 2023-10-18T20:30:34.151 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:entry[obj00000000000000000016]: success 2023-10-18T20:30:34.151 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.109+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:7:6 marker=00000000002.66.6 timestamp=2023-10-18T20:30:32.576239+0000 2023-10-18T20:30:34.151 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.110+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000009]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000009[0] 2023-10-18T20:30:34.151 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.110+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: listing bilog for incremental sync; position=00000000002.101.6 2023-10-18T20:30:34.151 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: sync status for source bucket shard: 0 2023-10-18T20:30:34.152 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.152 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]: sync status for source bucket shard: 0 2023-10-18T20:30:34.152 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]: sync status for source bucket shard: 0 2023-10-18T20:30:34.152 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]: sync status for source bucket shard: 0 2023-10-18T20:30:34.152 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]: sync status for source bucket shard: 0 2023-10-18T20:30:34.153 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:36]: sync status for source bucket shard: 0 2023-10-18T20:30:34.153 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.153 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.153 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.154 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.154 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.112+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:36]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:36]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.154 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.113+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:entry[obj00000000000000000060]: success 2023-10-18T20:30:34.154 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.113+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:0:6 marker=00000000001.55.6 timestamp=2023-10-18T20:30:32.403090+0000 2023-10-18T20:30:34.155 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:entry[obj00000000000000000001]: success 2023-10-18T20:30:34.155 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:3:6 marker=00000000002.50.6 timestamp=2023-10-18T20:30:31.840825+0000 2023-10-18T20:30:34.155 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:entry[obj00000000000000000030]: success 2023-10-18T20:30:34.155 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:11:6 marker=00000000002.53.6 timestamp=2023-10-18T20:30:32.742308+0000 2023-10-18T20:30:34.155 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:entry[obj00000000000000000013]: success 2023-10-18T20:30:34.156 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:18:6 marker=00000000002.83.6 timestamp=2023-10-18T20:30:32.103734+0000 2023-10-18T20:30:34.156 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:entry[obj00000000000000000026]: success 2023-10-18T20:30:34.156 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:10:6 marker=00000000002.49.6 timestamp=2023-10-18T20:30:32.020265+0000 2023-10-18T20:30:34.156 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:entry[obj00000000000000000006]: success 2023-10-18T20:30:34.157 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:20:6 marker=00000000002.64.6 timestamp=2023-10-18T20:30:31.862226+0000 2023-10-18T20:30:34.157 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:entry[obj00000000000000000024]: success 2023-10-18T20:30:34.157 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:15:6 marker=00000000002.66.6 timestamp=2023-10-18T20:30:32.820989+0000 2023-10-18T20:30:34.157 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:entry[obj00000000000000000009]: success 2023-10-18T20:30:34.157 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:13:6 marker=00000000002.101.6 timestamp=2023-10-18T20:30:32.720856+0000 2023-10-18T20:30:34.158 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-10-18T20:30:34.158 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23/obj00000000000000000039 2023-10-18T20:30:34.158 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: entry->timestamp=2023-10-18T20:30:32.317609+0000 2023-10-18T20:30:34.159 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-10-18T20:30:34.159 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000039]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000039[0] 2023-10-18T20:30:34.160 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: listing bilog for incremental sync; position=00000000002.77.6 2023-10-18T20:30:34.160 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:entry[obj00000000000000000002]: success 2023-10-18T20:30:34.160 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.117+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:17:6 marker=00000000002.68.6 timestamp=2023-10-18T20:30:31.919980+0000 2023-10-18T20:30:34.161 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-18T20:30:34.161 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35/obj00000000000000000018 2023-10-18T20:30:34.161 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: entry->timestamp=2023-10-18T20:30:32.902015+0000 2023-10-18T20:30:34.162 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-18T20:30:34.162 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]: parsed entry: id=00000000002.99.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-18T20:30:34.162 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31/obj00000000000000000003 2023-10-18T20:30:34.163 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]: entry->timestamp=2023-10-18T20:30:32.661814+0000 2023-10-18T20:30:34.163 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.99.6 op=0 op_state=1 2023-10-18T20:30:34.163 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2023-10-18T20:30:34.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34/obj00000000000000000057 2023-10-18T20:30:34.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]: entry->timestamp=2023-10-18T20:30:31.820867+0000 2023-10-18T20:30:34.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-10-18T20:30:34.165 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:entry[obj00000000000000000018]: : injecting data sync error on key=obj00000000000000000018 2023-10-18T20:30:34.165 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-18T20:30:34.165 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:entry[obj00000000000000000003]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000003[0] 2023-10-18T20:30:34.166 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]: listing bilog for incremental sync; position=00000000002.99.6 2023-10-18T20:30:34.166 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:entry[obj00000000000000000057]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000057[0] 2023-10-18T20:30:34.166 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]: listing bilog for incremental sync; position=00000000001.53.6 2023-10-18T20:30:34.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:entry[obj00000000000000000018]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:34.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.118+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:entry[obj00000000000000000018]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35/obj00000000000000000018 2023-10-18T20:30:34.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.123+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:entry[obj00000000000000000039]: success 2023-10-18T20:30:34.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.123+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:23:6 marker=00000000002.77.6 timestamp=2023-10-18T20:30:32.317609+0000 2023-10-18T20:30:34.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.123+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:entry[obj00000000000000000065]: success 2023-10-18T20:30:34.168 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.123+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:5:6 marker=00000000001.68.6 timestamp=2023-10-18T20:30:32.637722+0000 2023-10-18T20:30:34.168 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.123+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:entry[obj00000000000000000062]: success 2023-10-18T20:30:34.168 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.123+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:12:6 marker=00000000001.62.6 timestamp=2023-10-18T20:30:31.736585+0000 2023-10-18T20:30:34.168 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.124+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:entry[obj00000000000000000003]: success 2023-10-18T20:30:34.169 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.124+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:31:6 marker=00000000002.99.6 timestamp=2023-10-18T20:30:32.661814+0000 2023-10-18T20:30:34.169 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-18T20:30:34.169 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32/obj00000000000000000014 2023-10-18T20:30:34.169 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]: entry->timestamp=2023-10-18T20:30:31.758704+0000 2023-10-18T20:30:34.170 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-10-18T20:30:34.170 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:entry[obj00000000000000000014]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000014[0] 2023-10-18T20:30:34.171 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]: listing bilog for incremental sync; position=00000000002.74.6 2023-10-18T20:30:34.171 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-18T20:30:34.171 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33/obj00000000000000000019 2023-10-18T20:30:34.171 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]: entry->timestamp=2023-10-18T20:30:32.488877+0000 2023-10-18T20:30:34.171 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-10-18T20:30:34.172 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:entry[obj00000000000000000019]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000019[0] 2023-10-18T20:30:34.172 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.126+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]: listing bilog for incremental sync; position=00000000002.44.6 2023-10-18T20:30:34.172 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.128+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:entry[obj00000000000000000014]: success 2023-10-18T20:30:34.172 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.128+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:32:6 marker=00000000002.74.6 timestamp=2023-10-18T20:30:31.758704+0000 2023-10-18T20:30:34.173 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.128+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:entry[obj00000000000000000019]: success 2023-10-18T20:30:34.173 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.128+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:33:6 marker=00000000002.44.6 timestamp=2023-10-18T20:30:32.488877+0000 2023-10-18T20:30:34.177 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.140+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-18T20:30:34.177 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.140+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1/obj00000000000000000000 2023-10-18T20:30:34.177 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.140+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: entry->timestamp=2023-10-18T20:30:32.881421+0000 2023-10-18T20:30:34.178 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.140+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-18T20:30:34.178 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.140+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000000]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000000[0] 2023-10-18T20:30:34.178 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.140+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-18T20:30:34.178 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.141+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:entry[obj00000000000000000000]: success 2023-10-18T20:30:34.178 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.141+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:1:6 marker=00000000002.68.6 timestamp=2023-10-18T20:30:32.881421+0000 2023-10-18T20:30:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-18T20:30:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2/obj00000000000000000011 2023-10-18T20:30:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: entry->timestamp=2023-10-18T20:30:32.465781+0000 2023-10-18T20:30:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-18T20:30:34.180 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000011[0] 2023-10-18T20:30:34.180 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-18T20:30:34.180 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-18T20:30:34.180 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6/obj00000000000000000005 2023-10-18T20:30:34.180 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: entry->timestamp=2023-10-18T20:30:31.942358+0000 2023-10-18T20:30:34.181 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-18T20:30:34.181 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000005]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000005[0] 2023-10-18T20:30:34.181 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.145+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-18T20:30:34.181 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.146+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:entry[obj00000000000000000011]: success 2023-10-18T20:30:34.181 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.146+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:2:6 marker=00000000002.52.6 timestamp=2023-10-18T20:30:32.465781+0000 2023-10-18T20:30:34.182 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.147+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:entry[obj00000000000000000005]: success 2023-10-18T20:30:34.182 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.147+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:6:6 marker=00000000002.42.6 timestamp=2023-10-18T20:30:31.942358+0000 2023-10-18T20:30:34.182 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.148+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-18T20:30:34.182 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.148+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000012 2023-10-18T20:30:34.183 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.148+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:32.552770+0000 2023-10-18T20:30:34.183 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.148+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-18T20:30:34.183 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.148+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2023-10-18T20:30:34.183 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.148+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-18T20:30:34.183 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.148+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:30:34.184 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.148+0000 7fe0d0d94780 0 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000012]: ERROR: failed to sync object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000012 2023-10-18T20:30:34.184 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-10-18T20:30:34.184 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24/obj00000000000000000027 2023-10-18T20:30:34.185 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: entry->timestamp=2023-10-18T20:30:32.295322+0000 2023-10-18T20:30:34.185 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-18T20:30:34.185 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:entry[obj00000000000000000027]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000027[0] 2023-10-18T20:30:34.186 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-18T20:30:34.186 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-18T20:30:34.186 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22/obj00000000000000000007 2023-10-18T20:30:34.186 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: entry->timestamp=2023-10-18T20:30:32.041511+0000 2023-10-18T20:30:34.187 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-10-18T20:30:34.187 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000007]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000007[0] 2023-10-18T20:30:34.187 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.150+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: listing bilog for incremental sync; position=00000000002.49.6 2023-10-18T20:30:34.187 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.151+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:entry[obj00000000000000000048]: success 2023-10-18T20:30:34.187 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.151+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:8:6 marker=00000000001.49.6 timestamp=2023-10-18T20:30:32.128085+0000 2023-10-18T20:30:34.188 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.152+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:entry[obj00000000000000000027]: success 2023-10-18T20:30:34.188 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.152+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:24:6 marker=00000000002.50.6 timestamp=2023-10-18T20:30:32.295322+0000 2023-10-18T20:30:34.188 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.152+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:entry[obj00000000000000000007]: success 2023-10-18T20:30:34.188 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.152+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:22:6 marker=00000000002.49.6 timestamp=2023-10-18T20:30:32.041511+0000 2023-10-18T20:30:34.189 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-10-18T20:30:34.189 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27/obj00000000000000000053 2023-10-18T20:30:34.189 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: entry->timestamp=2023-10-18T20:30:32.001026+0000 2023-10-18T20:30:34.189 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.85.6 op=0 op_state=1 2023-10-18T20:30:34.189 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-18T20:30:34.190 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26/obj00000000000000000021 2023-10-18T20:30:34.190 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: entry->timestamp=2023-10-18T20:30:32.151935+0000 2023-10-18T20:30:34.190 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-18T20:30:34.190 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:entry[obj00000000000000000053]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000053[0] 2023-10-18T20:30:34.190 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: listing bilog for incremental sync; position=00000000002.85.6 2023-10-18T20:30:34.191 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:entry[obj00000000000000000021]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000021[0] 2023-10-18T20:30:34.191 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.153+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-18T20:30:34.191 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-10-18T20:30:34.191 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19/obj00000000000000000045 2023-10-18T20:30:34.192 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: entry->timestamp=2023-10-18T20:30:32.375793+0000 2023-10-18T20:30:34.192 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.85.6 op=0 op_state=1 2023-10-18T20:30:34.192 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000045]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000045[0] 2023-10-18T20:30:34.192 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: listing bilog for incremental sync; position=00000000002.85.6 2023-10-18T20:30:34.192 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-18T20:30:34.193 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21/obj00000000000000000017 2023-10-18T20:30:34.193 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: entry->timestamp=2023-10-18T20:30:31.596172+0000 2023-10-18T20:30:34.193 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-10-18T20:30:34.193 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:entry[obj00000000000000000021]: success 2023-10-18T20:30:34.193 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:26:6 marker=00000000002.56.6 timestamp=2023-10-18T20:30:32.151935+0000 2023-10-18T20:30:34.194 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000017]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000017[0] 2023-10-18T20:30:34.194 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.155+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: listing bilog for incremental sync; position=00000000002.49.6 2023-10-18T20:30:34.194 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.156+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:entry[obj00000000000000000045]: success 2023-10-18T20:30:34.194 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.156+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:19:6 marker=00000000002.85.6 timestamp=2023-10-18T20:30:32.375793+0000 2023-10-18T20:30:34.195 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.157+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2023-10-18T20:30:34.195 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.157+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30/obj00000000000000000036 2023-10-18T20:30:34.195 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.157+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: entry->timestamp=2023-10-18T20:30:31.654605+0000 2023-10-18T20:30:34.195 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.157+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-10-18T20:30:34.196 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.157+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:entry[obj00000000000000000036]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000036[0] 2023-10-18T20:30:34.196 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.157+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: listing bilog for incremental sync; position=00000000001.54.6 2023-10-18T20:30:34.196 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.157+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:entry[obj00000000000000000017]: success 2023-10-18T20:30:34.196 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.157+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:21:6 marker=00000000002.49.6 timestamp=2023-10-18T20:30:31.596172+0000 2023-10-18T20:30:34.196 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-18T20:30:34.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29/obj00000000000000000004 2023-10-18T20:30:34.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: entry->timestamp=2023-10-18T20:30:32.212382+0000 2023-10-18T20:30:34.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.83.6 op=0 op_state=1 2023-10-18T20:30:34.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.158+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:entry[obj00000000000000000004]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000004[0] 2023-10-18T20:30:34.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.158+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: listing bilog for incremental sync; position=00000000002.83.6 2023-10-18T20:30:34.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-18T20:30:34.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25/obj00000000000000000010 2023-10-18T20:30:34.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: entry->timestamp=2023-10-18T20:30:31.676501+0000 2023-10-18T20:30:34.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-18T20:30:34.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:entry[obj00000000000000000010]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000010[0] 2023-10-18T20:30:34.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-18T20:30:34.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-18T20:30:34.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28/obj00000000000000000020 2023-10-18T20:30:34.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: entry->timestamp=2023-10-18T20:30:32.233287+0000 2023-10-18T20:30:34.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-18T20:30:34.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000020]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000020[0] 2023-10-18T20:30:34.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.159+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-18T20:30:34.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.160+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:entry[obj00000000000000000004]: success 2023-10-18T20:30:34.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.160+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:29:6 marker=00000000002.83.6 timestamp=2023-10-18T20:30:32.212382+0000 2023-10-18T20:30:34.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.160+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:entry[obj00000000000000000010]: success 2023-10-18T20:30:34.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.160+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:25:6 marker=00000000002.47.6 timestamp=2023-10-18T20:30:31.676501+0000 2023-10-18T20:30:34.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.160+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:entry[obj00000000000000000020]: success 2023-10-18T20:30:34.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.160+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:28:6 marker=00000000002.51.6 timestamp=2023-10-18T20:30:32.233287+0000 2023-10-18T20:30:34.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.162+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: a sync operation returned error 2023-10-18T20:30:34.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.162+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: backing out with sync_status=-5 2023-10-18T20:30:34.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.162+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:34.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.162+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:34.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.162+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.162+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: sync status for source bucket shard: 0 2023-10-18T20:30:34.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.162+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.165+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:entry[obj00000000000000000057]: success 2023-10-18T20:30:34.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.165+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:34:6 marker=00000000001.53.6 timestamp=2023-10-18T20:30:31.820867+0000 2023-10-18T20:30:34.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.190+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: a sync operation returned error 2023-10-18T20:30:34.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.190+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: backing out with sync_status=-5 2023-10-18T20:30:34.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.190+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: incremental sync on bucket failed, retcode=-5 2023-10-18T20:30:34.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.190+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: ERROR: incremental sync failed. error: -5 2023-10-18T20:30:34.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.190+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:30:34.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.190+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: sync status for source bucket shard: 0 2023-10-18T20:30:34.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.190+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position= 2023-10-18T20:30:34.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.198+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:entry[obj00000000000000000053]: success 2023-10-18T20:30:34.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.198+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:27:6 marker=00000000002.85.6 timestamp=2023-10-18T20:30:32.001026+0000 2023-10-18T20:30:34.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.201+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:entry[obj00000000000000000036]: success 2023-10-18T20:30:34.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.201+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:30:6 marker=00000000001.54.6 timestamp=2023-10-18T20:30:31.654605+0000 2023-10-18T20:30:34.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.204+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-18T20:30:34.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.204+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35/obj00000000000000000018 2023-10-18T20:30:34.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.204+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: entry->timestamp=2023-10-18T20:30:32.902015+0000 2023-10-18T20:30:34.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.204+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-18T20:30:34.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.204+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:entry[obj00000000000000000018]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000018[0] 2023-10-18T20:30:34.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.204+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-18T20:30:34.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.205+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:entry[obj00000000000000000018]: success 2023-10-18T20:30:34.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.205+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:35:6 marker=00000000002.48.6 timestamp=2023-10-18T20:30:32.902015+0000 2023-10-18T20:30:34.234 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.232+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-18T20:30:34.234 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.232+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: syncing object: bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16/obj00000000000000000012 2023-10-18T20:30:34.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.232+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: entry->timestamp=2023-10-18T20:30:32.552770+0000 2023-10-18T20:30:34.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.232+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-18T20:30:34.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.232+0000 7fe0d0d94780 5 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000012]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-58[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27])/obj00000000000000000012[0] 2023-10-18T20:30:34.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.232+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-18T20:30:34.236 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.233+0000 7fe0d0d94780 10 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:entry[obj00000000000000000012]: success 2023-10-18T20:30:34.236 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:34.233+0000 7fe0d0d94780 20 RGW-SYNC:bucket[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27<-bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]:inc_sync[bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16]: updating marker marker_oid=bucket.sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27:16:6 marker=00000000002.57.6 timestamp=2023-10-18T20:30:32.552770+0000 2023-10-18T20:30:34.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:34.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:34.298 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:34.387 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:34.387 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:34.420 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:30:34.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.420 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000000", 2023-10-18T20:30:34.421 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.421 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.421 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.421 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 17 2023-10-18T20:30:34.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.422 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.422 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.422 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.422 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.423 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.423 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.881421Z", 2023-10-18T20:30:34.423 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.423 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.423 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.424 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.424 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.424 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.424 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.425 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.425 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.12235268824477669998", 2023-10-18T20:30:34.425 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.426 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.426 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.427 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000001", 2023-10-18T20:30:34.427 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.427 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.427 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.428 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 15 2023-10-18T20:30:34.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.428 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.428 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.428 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.429 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.429 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.429 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.840825Z", 2023-10-18T20:30:34.429 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.430 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.430 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.430 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.430 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.430 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.431 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.431 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.431 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.6694246706401429474", 2023-10-18T20:30:34.431 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.432 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.432 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.433 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000002", 2023-10-18T20:30:34.433 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.433 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.433 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.433 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 13 2023-10-18T20:30:34.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.434 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.434 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.434 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.434 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.435 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.435 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.919980Z", 2023-10-18T20:30:34.435 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.435 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.436 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.436 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.436 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.437 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.437 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.437 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.438 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.1895441074985956520", 2023-10-18T20:30:34.438 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.438 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.438 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000003", 2023-10-18T20:30:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.439 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12 2023-10-18T20:30:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.440 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.661814Z", 2023-10-18T20:30:34.441 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.442 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.442 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.442 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.442 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.443 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.12410333959072141580", 2023-10-18T20:30:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.445 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000004", 2023-10-18T20:30:34.445 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.445 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.446 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.446 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 15 2023-10-18T20:30:34.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.447 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.447 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.448 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.448 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.449 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.449 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.212382Z", 2023-10-18T20:30:34.449 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.449 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.450 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.450 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.450 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.450 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.450 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.451 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.451 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.7669126216117564227", 2023-10-18T20:30:34.451 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.451 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.452 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.452 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000005", 2023-10-18T20:30:34.453 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.453 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.453 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.453 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 17 2023-10-18T20:30:34.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.454 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.454 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.454 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.454 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.455 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.457 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.942358Z", 2023-10-18T20:30:34.457 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.458 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.458 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.458 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.459 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.459 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.459 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.459 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.460 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.6250505425717676065", 2023-10-18T20:30:34.460 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.460 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.460 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.461 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000006", 2023-10-18T20:30:34.462 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.462 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.462 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.463 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12 2023-10-18T20:30:34.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.463 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.463 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.464 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.464 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.464 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.464 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.862226Z", 2023-10-18T20:30:34.464 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.465 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.465 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.466 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.466 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.466 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.467 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.467 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.468 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.13241141844571065174", 2023-10-18T20:30:34.468 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.468 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.469 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.470 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000007", 2023-10-18T20:30:34.470 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.470 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.471 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.471 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:30:34.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.472 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.472 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.472 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.473 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.473 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.474 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.041511Z", 2023-10-18T20:30:34.474 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.474 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.474 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.474 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.475 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.475 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.475 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.475 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.476 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.2858452069110289370", 2023-10-18T20:30:34.476 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.476 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.476 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.477 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000008", 2023-10-18T20:30:34.477 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.478 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.478 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.478 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:30:34.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.479 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.479 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.479 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.479 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.480 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.480 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.743383Z", 2023-10-18T20:30:34.480 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.480 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.480 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.481 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.481 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.481 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.481 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.481 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.482 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.12336798056693260157", 2023-10-18T20:30:34.482 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.482 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.483 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.483 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000009", 2023-10-18T20:30:34.483 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.484 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.484 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.484 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12 2023-10-18T20:30:34.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.484 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.485 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.485 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.485 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.485 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.485 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.720856Z", 2023-10-18T20:30:34.486 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.486 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.486 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.486 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.486 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.487 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.487 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.487 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.487 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.8689837474344388094", 2023-10-18T20:30:34.488 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.488 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.488 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.489 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000010", 2023-10-18T20:30:34.489 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.489 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.489 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.490 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10 2023-10-18T20:30:34.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.490 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.490 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.490 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.491 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.491 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.491 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.676501Z", 2023-10-18T20:30:34.491 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.491 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.492 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.492 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.492 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.492 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.492 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.493 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.493 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.2321776157969437114", 2023-10-18T20:30:34.493 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.493 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.494 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.494 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000011", 2023-10-18T20:30:34.494 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.495 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.495 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.495 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 19 2023-10-18T20:30:34.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.495 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.496 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.496 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.496 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.496 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.496 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.465781Z", 2023-10-18T20:30:34.497 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.497 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.497 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.497 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.497 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.498 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.498 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.498 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.498 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.570136882358757309", 2023-10-18T20:30:34.499 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.499 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.499 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.500 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000012", 2023-10-18T20:30:34.500 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.500 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.500 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.500 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 14 2023-10-18T20:30:34.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.501 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.501 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.501 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.501 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.502 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.502 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.552770Z", 2023-10-18T20:30:34.502 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.502 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.503 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.503 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.503 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.503 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.503 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.504 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.504 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.15350259697831232300", 2023-10-18T20:30:34.504 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.504 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.505 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.505 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000013", 2023-10-18T20:30:34.505 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.506 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.506 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.506 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:30:34.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.506 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.507 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.507 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.507 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.507 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.507 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.103734Z", 2023-10-18T20:30:34.508 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.508 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.508 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.508 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.508 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.509 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.509 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.509 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.509 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.8500811057049025772", 2023-10-18T20:30:34.510 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.510 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.510 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.511 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000014", 2023-10-18T20:30:34.511 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.511 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.511 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.511 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 15 2023-10-18T20:30:34.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.512 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.512 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.512 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.512 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.513 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.513 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.758704Z", 2023-10-18T20:30:34.513 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.513 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.513 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.514 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.514 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.514 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.514 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.514 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.515 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.2327026918984867249", 2023-10-18T20:30:34.515 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.515 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.516 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.516 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000015", 2023-10-18T20:30:34.516 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.517 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.517 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.517 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:30:34.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.517 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.518 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.518 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.518 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.518 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.518 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.722306Z", 2023-10-18T20:30:34.519 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.519 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.519 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.519 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.519 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.520 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.520 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.520 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.520 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.8788547719222664794", 2023-10-18T20:30:34.521 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.521 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.521 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.522 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000016", 2023-10-18T20:30:34.522 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.522 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.522 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.522 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10 2023-10-18T20:30:34.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.523 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.523 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.523 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.524 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.524 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.524 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.576239Z", 2023-10-18T20:30:34.524 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.524 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.525 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.525 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.525 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.525 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.525 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.526 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.526 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.16458147527910163630", 2023-10-18T20:30:34.526 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.526 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.527 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.527 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000017", 2023-10-18T20:30:34.527 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.528 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.528 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.528 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:30:34.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.528 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.529 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.529 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.529 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.529 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.529 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.596172Z", 2023-10-18T20:30:34.530 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.530 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.530 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.530 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.531 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.531 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.531 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.531 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.532 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.8422236343359440308", 2023-10-18T20:30:34.532 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.532 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.532 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.533 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000018", 2023-10-18T20:30:34.533 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.533 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.533 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.534 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 25 2023-10-18T20:30:34.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.534 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.534 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.534 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.535 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.535 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.535 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.902015Z", 2023-10-18T20:30:34.535 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.535 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.536 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.536 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.536 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.536 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.536 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.537 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.537 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.18388829516979592371", 2023-10-18T20:30:34.537 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.537 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.538 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.538 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000019", 2023-10-18T20:30:34.538 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.539 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.539 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.539 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 41 2023-10-18T20:30:34.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.539 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.540 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.540 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.540 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.540 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.540 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.488877Z", 2023-10-18T20:30:34.541 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.541 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.541 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.541 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.542 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.542 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.542 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.542 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.543 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.5038212649861358316", 2023-10-18T20:30:34.543 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.543 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.543 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.544 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000020", 2023-10-18T20:30:34.544 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.544 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.544 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.545 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:30:34.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.545 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.545 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.545 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.546 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.546 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.546 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.233287Z", 2023-10-18T20:30:34.546 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.547 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.547 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.547 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.547 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.547 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.548 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.548 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.548 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.12039501250787405169", 2023-10-18T20:30:34.548 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.549 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.549 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.549 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000021", 2023-10-18T20:30:34.550 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.550 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.550 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.550 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 16 2023-10-18T20:30:34.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.552 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.151935Z", 2023-10-18T20:30:34.552 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.552 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.552 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.552 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.554 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.11981491289202052051", 2023-10-18T20:30:34.554 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.554 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.554 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000022", 2023-10-18T20:30:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:30:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.350297Z", 2023-10-18T20:30:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.2522152396482152825", 2023-10-18T20:30:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000023", 2023-10-18T20:30:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10 2023-10-18T20:30:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.563 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.261228Z", 2023-10-18T20:30:34.563 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.563 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.563 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.563 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.564 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.564 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.564 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.564 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.11023203943516941868", 2023-10-18T20:30:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000024", 2023-10-18T20:30:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.567 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.567 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:30:34.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.567 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.567 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.820989Z", 2023-10-18T20:30:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.7461677199894959231", 2023-10-18T20:30:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.571 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.571 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.571 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000025", 2023-10-18T20:30:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:30:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.573 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.573 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.573 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.574 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.574 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.574 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.176087Z", 2023-10-18T20:30:34.574 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.13202393426587491387", 2023-10-18T20:30:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000026", 2023-10-18T20:30:34.578 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.578 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.578 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.578 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:30:34.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.020265Z", 2023-10-18T20:30:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.12426048740318021246", 2023-10-18T20:30:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000027", 2023-10-18T20:30:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 15 2023-10-18T20:30:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.295322Z", 2023-10-18T20:30:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.13817088304070664531", 2023-10-18T20:30:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000028", 2023-10-18T20:30:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:26.245833Z", 2023-10-18T20:30:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.593 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.13279152218188920121", 2023-10-18T20:30:34.593 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.593 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.593 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000029", 2023-10-18T20:30:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.047138Z", 2023-10-18T20:30:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.597 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.597 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.597 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.597 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.597 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.598 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.598 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.598 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.10731080328762637396", 2023-10-18T20:30:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000030", 2023-10-18T20:30:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:30:34.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.601 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.601 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.601 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.601 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.742308Z", 2023-10-18T20:30:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.603 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.603 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.603 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.603 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.603 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.604 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.2861877323743024230", 2023-10-18T20:30:34.604 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.604 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.604 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000031", 2023-10-18T20:30:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.606 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.606 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:30:34.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.606 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.607 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.607 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.607 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.607 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.607 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.073266Z", 2023-10-18T20:30:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.8572610041733915882", 2023-10-18T20:30:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.611 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000032", 2023-10-18T20:30:34.611 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.611 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.611 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.611 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:27.953086Z", 2023-10-18T20:30:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.614 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.614 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.614 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.614 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.614 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.14685862588820666992", 2023-10-18T20:30:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.616 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000033", 2023-10-18T20:30:34.616 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.616 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:30:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.618 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.618 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.618 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.618 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.799608Z", 2023-10-18T20:30:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.15542418867312611358", 2023-10-18T20:30:34.621 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.621 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.621 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000036", 2023-10-18T20:30:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.654605Z", 2023-10-18T20:30:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.625 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.625 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.625 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.625 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.625 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.5731875325959569015", 2023-10-18T20:30:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000037", 2023-10-18T20:30:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.093022Z", 2023-10-18T20:30:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.630 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.630 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.630 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.630 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.7382632870806278115", 2023-10-18T20:30:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000039", 2023-10-18T20:30:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:30:34.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.634 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.634 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.634 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.634 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.317609Z", 2023-10-18T20:30:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.5548955435725205028", 2023-10-18T20:30:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.638 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000040", 2023-10-18T20:30:34.638 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.638 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:30:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:28.049773Z", 2023-10-18T20:30:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.642 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.642 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.642 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.15299156381173207128", 2023-10-18T20:30:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000042", 2023-10-18T20:30:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.255745Z", 2023-10-18T20:30:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.647 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.647 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.647 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.647 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.647 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.11793179600860007154", 2023-10-18T20:30:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000045", 2023-10-18T20:30:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.650 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.650 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 17 2023-10-18T20:30:34.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.650 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.650 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.375793Z", 2023-10-18T20:30:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.18224147944363598222", 2023-10-18T20:30:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.654 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.654 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000046", 2023-10-18T20:30:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:30:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:28.030762Z", 2023-10-18T20:30:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.658 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.658 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.658 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.658 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.658 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.9014085445297064124", 2023-10-18T20:30:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.661 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000048", 2023-10-18T20:30:34.661 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:30:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.663 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.663 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.663 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.663 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.663 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.128085Z", 2023-10-18T20:30:34.664 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.664 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.664 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.664 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.665 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.665 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.665 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.665 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.13858481020967219299", 2023-10-18T20:30:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000049", 2023-10-18T20:30:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.668 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.668 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.668 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.668 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.669 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.669 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.669 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.555942Z", 2023-10-18T20:30:34.669 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.669 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.14403870795304237028", 2023-10-18T20:30:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.672 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.672 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000050", 2023-10-18T20:30:34.672 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 22 2023-10-18T20:30:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.771456Z", 2023-10-18T20:30:34.675 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.675 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.675 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.675 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.675 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.676 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.676 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.676 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.3152274513490550747", 2023-10-18T20:30:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000053", 2023-10-18T20:30:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.679 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:30:34.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.679 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.679 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.679 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.001026Z", 2023-10-18T20:30:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.1735029686760996399", 2023-10-18T20:30:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.683 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.683 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000055", 2023-10-18T20:30:34.683 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:27.872840Z", 2023-10-18T20:30:34.686 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.686 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.686 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.686 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.14835506177843597561", 2023-10-18T20:30:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.689 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000057", 2023-10-18T20:30:34.689 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.689 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.689 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:30:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.820867Z", 2023-10-18T20:30:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.692 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.692 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.692 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.692 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.692 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.6553825450699472965", 2023-10-18T20:30:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000058", 2023-10-18T20:30:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.695 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.695 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.695 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.696 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.696 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.696 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.696 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.696 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:28.372095Z", 2023-10-18T20:30:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.699 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.4197415936589740477", 2023-10-18T20:30:34.699 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.699 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.699 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000059", 2023-10-18T20:30:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.701 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:34.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.701 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.701 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.701 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.702 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.702 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.702 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:28.881315Z", 2023-10-18T20:30:34.702 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.703 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.703 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.703 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.703 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.703 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.11963003006825953919", 2023-10-18T20:30:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.705 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.705 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.705 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000060", 2023-10-18T20:30:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:30:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.707 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.707 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.707 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.707 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.707 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.403090Z", 2023-10-18T20:30:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.710 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.5073154954006824081", 2023-10-18T20:30:34.710 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.710 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.710 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000061", 2023-10-18T20:30:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 16 2023-10-18T20:30:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.714009Z", 2023-10-18T20:30:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.715 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.715 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.715 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.15263850556741042529", 2023-10-18T20:30:34.715 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000062", 2023-10-18T20:30:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.719 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.736585Z", 2023-10-18T20:30:34.719 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.719 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.719 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.719 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.721 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.15742897902377028079", 2023-10-18T20:30:34.721 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.721 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.721 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000065", 2023-10-18T20:30:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:30:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.724 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.724 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.724 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.724 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.637722Z", 2023-10-18T20:30:34.724 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.10088743336551858660", 2023-10-18T20:30:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.727 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.727 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.727 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000076", 2023-10-18T20:30:34.728 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.728 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.728 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.728 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:30:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.730 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.730 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:26.141311Z", 2023-10-18T20:30:34.730 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.730 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.730 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.731 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.731 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.731 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.731 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.731 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.17977417459742442620", 2023-10-18T20:30:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.733 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000133", 2023-10-18T20:30:34.733 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.733 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:30:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.850838Z", 2023-10-18T20:30:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.737 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.737 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.737 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.13125614712624360172", 2023-10-18T20:30:34.737 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.738 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.738 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.739 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000162", 2023-10-18T20:30:34.739 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.739 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.739 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.739 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:30:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.108982Z", 2023-10-18T20:30:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.743 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.3290831704439195172", 2023-10-18T20:30:34.743 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.743 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.743 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000271", 2023-10-18T20:30:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 13 2023-10-18T20:30:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.234381Z", 2023-10-18T20:30:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.747 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.747 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.747 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.747 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.8653113586642992263", 2023-10-18T20:30:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000283", 2023-10-18T20:30:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:30:34.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.751 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.751 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.751 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.751 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.752 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.752 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.148673Z", 2023-10-18T20:30:34.752 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.752 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.752 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.753 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.753 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.753 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.753 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.753 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.754 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.2039612465631729506", 2023-10-18T20:30:34.754 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.754 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.754 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.755 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000317", 2023-10-18T20:30:34.755 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.755 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.756 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.756 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:34.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.756 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.756 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.757 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.757 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.757 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.757 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.294926Z", 2023-10-18T20:30:34.757 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.758 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.758 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.758 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.758 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.758 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.759 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.759 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.759 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.10718764246158890792", 2023-10-18T20:30:34.759 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.760 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.760 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.762 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000423", 2023-10-18T20:30:34.762 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.762 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.762 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.762 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:30:34.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.763 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.763 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.763 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.763 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.764 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.764 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.240139Z", 2023-10-18T20:30:34.764 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.764 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.765 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.765 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.765 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.765 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.765 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.766 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.766 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.5021308935054597512", 2023-10-18T20:30:34.766 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.766 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.767 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.767 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000467", 2023-10-18T20:30:34.767 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.768 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.768 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:30:34.768 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:34.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.768 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.769 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.769 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.769 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.769 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.769 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.027459Z", 2023-10-18T20:30:34.770 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.770 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.770 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.770 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.770 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.771 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.771 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.771 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.771 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.10448332812656742690", 2023-10-18T20:30:34.772 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.772 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.772 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:34.772 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:30:34.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:34.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:34.774 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:34.828 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:34.828 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:34.858 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:30:34.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.859 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000000", 2023-10-18T20:30:34.859 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.860 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.860 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.860 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:30:34.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.860 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.861 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.861 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.861 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.861 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.861 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.881421Z", 2023-10-18T20:30:34.862 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.862 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.862 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.862 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.863 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.863 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.863 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.863 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.864 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_ggbQyLc3k7DqFpvRfCdjXUGzhPwgkDS", 2023-10-18T20:30:34.864 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.864 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.864 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.865 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000001", 2023-10-18T20:30:34.865 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.865 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.865 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.866 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.866 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.866 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.869 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.869 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.870 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.870 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.840825Z", 2023-10-18T20:30:34.870 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.870 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.870 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.871 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.871 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.871 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.871 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.871 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.872 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_BUqkUJThMgNLL2Y-jso1D3e4ULr-rkH", 2023-10-18T20:30:34.872 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.872 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.872 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.873 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000002", 2023-10-18T20:30:34.873 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.873 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.874 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.874 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:30:34.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.874 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.874 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.875 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.875 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.875 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.875 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.919980Z", 2023-10-18T20:30:34.875 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.876 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.876 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.876 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.876 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.876 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.877 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.877 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.877 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_U83h5eGiPi-xNLncPp0eXVU_qeWsBpR", 2023-10-18T20:30:34.878 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.878 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.878 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.879 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000003", 2023-10-18T20:30:34.879 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.879 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.879 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.879 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.880 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.880 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.880 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.880 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.881 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.881 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.661814Z", 2023-10-18T20:30:34.881 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.881 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.881 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.882 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.882 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.882 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.882 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.883 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.883 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_dQ0cbpXJLrmwn_W7dupXU976ijiQ5fM", 2023-10-18T20:30:34.883 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.883 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.884 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.884 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000004", 2023-10-18T20:30:34.884 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.885 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.885 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.885 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:30:34.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.885 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.886 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.886 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.886 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.886 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.886 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.212382Z", 2023-10-18T20:30:34.887 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.887 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.887 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.887 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.887 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.888 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.888 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.888 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.888 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_Vnmu9VwYuhuhUPFNF4kmZFDHoCK09Nm", 2023-10-18T20:30:34.889 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.889 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.889 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.890 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000005", 2023-10-18T20:30:34.890 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.890 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.890 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.891 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.891 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.891 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.891 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.892 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.892 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.892 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.942358Z", 2023-10-18T20:30:34.892 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.892 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.893 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.893 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.893 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.893 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.893 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.894 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.894 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_XRI-y4NBLgrp_4eZUIRycbVJFIntEZh", 2023-10-18T20:30:34.894 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.894 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.895 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.895 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000006", 2023-10-18T20:30:34.895 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.896 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.896 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.896 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.896 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.897 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.897 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.897 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.897 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.897 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.862226Z", 2023-10-18T20:30:34.898 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.898 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.898 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.898 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.899 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.899 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.899 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.899 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.900 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_BYWolTuFPVLtK1sCgLO4ISuAhUwy6kX", 2023-10-18T20:30:34.900 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.900 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.900 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.901 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000007", 2023-10-18T20:30:34.901 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.901 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.901 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.902 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.902 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.902 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.903 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.903 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.903 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.903 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.041511Z", 2023-10-18T20:30:34.903 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.904 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.904 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.904 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.904 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.904 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.905 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.905 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.905 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_2An5So0-rAL7nBjzpb9_FIbeO2XLdIF", 2023-10-18T20:30:34.905 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.906 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.906 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.906 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000008", 2023-10-18T20:30:34.907 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.907 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.907 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.907 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:30:34.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.908 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.908 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.908 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.908 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.908 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.909 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.743383Z", 2023-10-18T20:30:34.909 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.909 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.909 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.909 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.910 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.910 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.910 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.910 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.911 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_1NrvxK4mTOelOTXTK7g3Vh43OpaBO6E", 2023-10-18T20:30:34.911 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.911 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.911 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000009", 2023-10-18T20:30:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.912 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.913 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.913 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:34.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.913 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.913 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.914 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.914 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.914 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.914 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.720856Z", 2023-10-18T20:30:34.914 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.915 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.915 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.915 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.915 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.915 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.916 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.916 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.916 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_141h5IVhdfahqnYIiTAnInl4Eej755g", 2023-10-18T20:30:34.916 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.917 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.917 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.917 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000010", 2023-10-18T20:30:34.918 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.918 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.918 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.918 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:34.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.919 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.919 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.919 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.919 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.919 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.920 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.676501Z", 2023-10-18T20:30:34.920 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.920 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.920 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.921 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.921 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.921 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.921 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.921 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.922 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_JDYbiRbnKKrS90TKsgkWcg5LaNuz9_k", 2023-10-18T20:30:34.922 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.922 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.922 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.923 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000011", 2023-10-18T20:30:34.923 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.924 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.924 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.924 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10 2023-10-18T20:30:34.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.924 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.925 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.925 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.925 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.925 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.925 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.465781Z", 2023-10-18T20:30:34.926 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.926 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.926 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.926 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.926 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.927 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.927 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.927 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.927 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_dH6MYpWvfCF0SIqxAJ5mBFtJtY0fEb8", 2023-10-18T20:30:34.928 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.928 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.928 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.929 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000012", 2023-10-18T20:30:34.929 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.929 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.929 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.929 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:34.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.930 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.930 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.930 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.930 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.931 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.931 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.552770Z", 2023-10-18T20:30:34.931 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.931 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.931 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.932 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.932 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.932 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.932 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.932 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.933 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_2pPpV03UM5SJP6iLPD5QnXAAZbvt-wc", 2023-10-18T20:30:34.933 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.933 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.933 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.934 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000013", 2023-10-18T20:30:34.934 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.934 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.935 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.935 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:34.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.935 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.935 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.936 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.936 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.936 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.936 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.103734Z", 2023-10-18T20:30:34.937 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.937 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.937 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.937 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.937 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.938 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.938 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.938 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.938 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_oFOLCdSwEKpRcuu2flAIJH6ArIOoLdB", 2023-10-18T20:30:34.939 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.939 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.939 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.940 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000014", 2023-10-18T20:30:34.940 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.940 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.940 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.940 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:34.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.941 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.941 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.941 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.941 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.942 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.942 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.758704Z", 2023-10-18T20:30:34.942 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.942 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.942 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.943 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.943 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.943 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.943 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.943 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.944 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_gbR0n2MVAngg0Sdb02cFPSvaAJfbS13", 2023-10-18T20:30:34.944 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.944 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.944 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.945 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000015", 2023-10-18T20:30:34.945 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.945 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.946 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.946 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:30:34.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.946 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.946 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.947 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.947 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.947 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.947 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.722306Z", 2023-10-18T20:30:34.948 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.948 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.948 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.948 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.948 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.949 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.949 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.949 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.949 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_M4ySAfYukYiBEeDAGmIkPQAUii15iMr", 2023-10-18T20:30:34.950 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.950 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.950 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.951 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000016", 2023-10-18T20:30:34.951 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.951 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.951 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.951 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:34.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.952 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.952 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.952 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.952 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.953 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:34.953 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.576239Z", 2023-10-18T20:30:34.953 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.953 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.953 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.954 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.954 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.954 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.954 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.954 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.955 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_DCAlhp8S1F1I0HyHJdxNEIl5ZxcsqGf", 2023-10-18T20:30:34.955 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.955 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.955 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.956 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000017", 2023-10-18T20:30:34.956 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.956 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.957 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.957 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:34.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.957 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.957 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.958 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.958 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.958 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.958 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.596172Z", 2023-10-18T20:30:34.959 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.959 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.959 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.959 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.959 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.960 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.960 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.960 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.960 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_BBsns5E8qrh6p11W6baZHTljuY2mUEU", 2023-10-18T20:30:34.961 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.961 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.961 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.962 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000018", 2023-10-18T20:30:34.962 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.962 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.962 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.962 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 18 2023-10-18T20:30:34.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.963 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.963 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.963 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.963 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.964 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.964 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.902015Z", 2023-10-18T20:30:34.964 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.964 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.964 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.965 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.965 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.965 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.965 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.965 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.966 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_qKbCcEr6s7-3UlewDU2Dxgdt5AobXuI", 2023-10-18T20:30:34.966 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.966 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.967 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.967 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000019", 2023-10-18T20:30:34.967 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.968 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.968 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.968 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 35 2023-10-18T20:30:34.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.968 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.969 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.969 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.969 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.969 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:34.969 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.488877Z", 2023-10-18T20:30:34.970 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.970 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.970 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.970 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.970 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.971 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.971 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.971 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.971 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_buD_u4x9f682kiIljqz7BtefK7uI4GB", 2023-10-18T20:30:34.972 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.972 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.972 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.973 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000020", 2023-10-18T20:30:34.973 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.973 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.973 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.973 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:34.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.974 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.974 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.974 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.974 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.975 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.975 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.233287Z", 2023-10-18T20:30:34.975 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.975 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.976 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.976 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.976 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.976 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.976 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.977 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.977 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_6BAt4OZhoktfHQvpIUbleusReRz1LuC", 2023-10-18T20:30:34.977 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.977 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.978 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.978 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000021", 2023-10-18T20:30:34.978 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.979 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.979 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.979 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:30:34.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.979 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.151935Z", 2023-10-18T20:30:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_vbd0bH0cfb5KN_HTQqzndB1zqUx3HmL", 2023-10-18T20:30:34.983 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.983 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.983 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.984 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000022", 2023-10-18T20:30:34.984 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.984 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.984 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.984 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:30:34.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.985 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.985 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.985 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.985 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.986 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.986 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.350297Z", 2023-10-18T20:30:34.986 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.986 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.987 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.987 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.987 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.987 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.987 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.988 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.988 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_BStaYX2opj7xo9mADZ5-pLgS4sbUG4Y", 2023-10-18T20:30:34.988 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.988 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.989 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.989 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000023", 2023-10-18T20:30:34.989 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.990 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.990 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.990 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:30:34.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.990 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.991 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.991 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.991 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.991 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.991 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.261228Z", 2023-10-18T20:30:34.992 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.992 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.992 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.992 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.992 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.993 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.993 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.993 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.993 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_MFOENseVGa-iCIo1W2gNV48FesHJXmg", 2023-10-18T20:30:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:34.995 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000024", 2023-10-18T20:30:34.995 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:34.995 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:34.995 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:34.996 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:34.997 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:34.997 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:34.997 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.820989Z", 2023-10-18T20:30:34.997 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:34.998 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:34.999 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:34.999 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:34.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:34.999 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_ihxF2PypVQbG8piZCySX3lKYdX4ktyZ", 2023-10-18T20:30:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000025", 2023-10-18T20:30:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.001 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.002 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.176087Z", 2023-10-18T20:30:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.003 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.004 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_rG7mUi8x7ttEakUbqiZph6IOkvESAx5", 2023-10-18T20:30:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.005 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.006 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000026", 2023-10-18T20:30:35.006 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.007 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.007 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.007 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:30:35.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.007 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.008 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.008 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.008 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.008 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.008 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.020265Z", 2023-10-18T20:30:35.009 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.009 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.009 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.009 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.009 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.010 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.010 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.010 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.010 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_3HKQcbaNwzmC04FoLQdsNzFWisKJuwA", 2023-10-18T20:30:35.011 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.011 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.011 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.012 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000027", 2023-10-18T20:30:35.012 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.012 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.012 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.012 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12 2023-10-18T20:30:35.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.013 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.013 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.013 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.013 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.014 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.014 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.295322Z", 2023-10-18T20:30:35.014 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.014 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.015 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.015 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.015 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.015 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.015 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.016 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.016 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_cMMG29feHdcbQpQldXC9wnUPz1xNI7M", 2023-10-18T20:30:35.016 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.016 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.017 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.017 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000028", 2023-10-18T20:30:35.017 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.018 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.018 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.018 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:30:35.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.018 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.019 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.019 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.019 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.019 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.019 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:26.245833Z", 2023-10-18T20:30:35.020 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.020 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.020 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.020 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.021 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.021 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.021 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.021 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.022 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_49_WUnKNz-Zk6ZmGHz-gvIlXatn2yPg", 2023-10-18T20:30:35.022 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.022 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.022 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.023 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000029", 2023-10-18T20:30:35.023 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.023 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.023 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.024 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:35.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.024 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.024 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.024 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.025 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.025 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:35.025 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.047138Z", 2023-10-18T20:30:35.025 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.025 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.026 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.026 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.026 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.026 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.027 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.027 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.027 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_7MuZiOO5axrl4Nb2G5UJs6GlaGpfuR-", 2023-10-18T20:30:35.027 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.028 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.028 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.028 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000030", 2023-10-18T20:30:35.029 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.029 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.029 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.029 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:35.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.030 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.030 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.030 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.030 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.030 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.031 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.742308Z", 2023-10-18T20:30:35.031 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.031 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.031 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.031 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.032 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.032 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.032 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.032 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.033 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_9A_AarKpYfLCjuerYIaAqgM3jqvpg7I", 2023-10-18T20:30:35.033 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.033 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.033 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.034 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000031", 2023-10-18T20:30:35.034 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.034 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.035 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.035 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:35.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.035 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.035 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.036 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.036 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.036 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:35.036 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.073266Z", 2023-10-18T20:30:35.036 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.037 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.037 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.037 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.037 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.037 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.038 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.038 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.038 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_accxVXRVDC1bbP-dKbsisLX8F5wjOgs", 2023-10-18T20:30:35.038 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.039 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.039 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.039 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000032", 2023-10-18T20:30:35.040 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.040 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.040 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.040 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:30:35.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.041 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.041 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.041 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.041 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.041 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:35.042 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:27.953086Z", 2023-10-18T20:30:35.042 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.042 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.042 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.043 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.043 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.043 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.043 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.043 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.044 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_T35djuXoCyxe4VMsOyU6v5P8tdy_gOs", 2023-10-18T20:30:35.044 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.044 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.045 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.045 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000033", 2023-10-18T20:30:35.045 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.046 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.046 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.046 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:35.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.046 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.047 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.047 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.047 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.047 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:35.047 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.799608Z", 2023-10-18T20:30:35.048 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.048 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.048 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.048 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.048 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.049 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.049 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.049 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.050 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_-3rjdNYKxPylLEWjNANuom2fHmL5kfW", 2023-10-18T20:30:35.050 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.050 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.050 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.051 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000036", 2023-10-18T20:30:35.051 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.051 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.051 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.052 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:35.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.052 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.052 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.052 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.053 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.053 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.053 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.654605Z", 2023-10-18T20:30:35.053 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.053 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.054 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.054 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.054 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.054 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.054 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.055 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.055 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_hnMt5ZAlk6c6lh7XtDEKa0ogpUh4Esq", 2023-10-18T20:30:35.055 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.055 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.056 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.056 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000037", 2023-10-18T20:30:35.056 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.057 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.057 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.057 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:35.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.057 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.058 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.058 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.058 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.058 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.059 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.093022Z", 2023-10-18T20:30:35.059 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.059 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.059 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.059 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.060 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.060 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.060 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.060 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.061 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_kw-KMDEVUCaLbWZDI0VH_PoPcGKXOSl", 2023-10-18T20:30:35.061 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.061 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.061 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.062 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000039", 2023-10-18T20:30:35.062 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.062 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.063 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.063 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:35.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.063 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.063 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.064 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.064 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.064 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:35.064 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.317609Z", 2023-10-18T20:30:35.064 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.065 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.065 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.065 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.065 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.065 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.066 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.066 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.066 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_OxJtWEFXaW9oXBKLNwVD8OPWTO9s3ly", 2023-10-18T20:30:35.066 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.067 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.067 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.067 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000040", 2023-10-18T20:30:35.068 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.068 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.068 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.068 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:30:35.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.069 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.069 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.069 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.069 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.070 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:35.070 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:28.049773Z", 2023-10-18T20:30:35.070 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.070 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.070 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.071 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.071 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.071 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.071 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.073 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.074 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_4AsNRqQcUWKZOJ2lC-xIloTXivUduJf", 2023-10-18T20:30:35.074 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.074 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.074 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.075 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000042", 2023-10-18T20:30:35.075 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.075 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.076 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.076 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:35.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.076 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.076 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.077 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.077 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.077 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:35.077 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.255745Z", 2023-10-18T20:30:35.077 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.078 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.078 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.078 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.078 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.078 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.079 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.079 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.079 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_CX8A86byfknlzOJpWq5zAWXtfbuMrVi", 2023-10-18T20:30:35.079 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.080 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.080 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.081 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000045", 2023-10-18T20:30:35.081 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.081 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.081 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.082 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:30:35.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.082 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.083 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.083 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.083 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.084 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.084 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.375793Z", 2023-10-18T20:30:35.085 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.085 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.085 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.085 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.086 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.086 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.086 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.086 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.087 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_8UtcSNlC-1NDJXEydaiMRTjs2TtPjZQ", 2023-10-18T20:30:35.087 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.087 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.087 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.088 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000046", 2023-10-18T20:30:35.088 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.088 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.088 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.089 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:30:35.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.089 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.089 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.090 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.090 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.090 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.090 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:28.030762Z", 2023-10-18T20:30:35.090 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.091 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.091 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.091 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.091 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.091 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.092 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.092 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.092 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_gYmyjHznZHD1S_XMBHmIwIDD8wvTHiQ", 2023-10-18T20:30:35.092 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.093 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.093 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.094 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000048", 2023-10-18T20:30:35.094 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.094 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.094 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.094 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:30:35.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.095 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.095 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.095 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.095 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.096 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.096 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.128085Z", 2023-10-18T20:30:35.096 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.096 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.096 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.097 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.097 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.097 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.097 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.097 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.098 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_cGeppmPzRp89Z4I78oHKZtvHLx09xr3", 2023-10-18T20:30:35.098 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.098 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.098 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.099 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000049", 2023-10-18T20:30:35.099 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.099 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.100 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.100 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:35.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.100 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.100 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.101 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.101 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.101 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:35.101 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:29.555942Z", 2023-10-18T20:30:35.101 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.102 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.102 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.102 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.102 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.102 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.103 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.103 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.103 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_O7wuqhCx7BmLMLw7jpZQjjuEjNveW6Z", 2023-10-18T20:30:35.103 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.104 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.104 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.104 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000050", 2023-10-18T20:30:35.105 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.105 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.105 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.105 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 17 2023-10-18T20:30:35.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.106 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.106 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.106 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.106 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.107 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:35.107 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.771456Z", 2023-10-18T20:30:35.107 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.107 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.107 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.108 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.108 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.108 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.108 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.108 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.109 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_8RHVEsVqF9boRaiFDsTkLxyrmm97wzJ", 2023-10-18T20:30:35.109 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.109 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.109 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.110 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000053", 2023-10-18T20:30:35.110 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.110 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.111 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.111 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:35.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.111 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.111 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.112 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.112 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.112 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:35.112 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.001026Z", 2023-10-18T20:30:35.113 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.113 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.113 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.113 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.113 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.114 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.114 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.114 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.115 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_W8eIL4M8nChuJ-V1PbG2Ai6D9FkOrBw", 2023-10-18T20:30:35.115 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.115 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.115 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.116 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000055", 2023-10-18T20:30:35.116 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.116 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.116 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.117 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:35.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.117 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.117 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.117 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.118 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.118 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.118 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:27.872840Z", 2023-10-18T20:30:35.118 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.118 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.119 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.119 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.119 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.119 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.119 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.120 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.120 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_GZmu43FsMUFJaaBloV1AwAArmhUQvPX", 2023-10-18T20:30:35.120 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.120 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.121 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.121 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000057", 2023-10-18T20:30:35.121 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.122 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.122 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.122 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10 2023-10-18T20:30:35.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.122 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.123 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.123 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.123 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.123 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:35.123 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.820867Z", 2023-10-18T20:30:35.124 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.124 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.124 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.124 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.124 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.125 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.125 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.125 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.125 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_9EukGzYl_MShFC4nQDkH3fXzTte_wHl", 2023-10-18T20:30:35.126 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.126 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.126 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.127 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000058", 2023-10-18T20:30:35.127 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.127 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.127 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.127 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:30:35.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.128 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.128 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.128 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.129 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.129 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:35.129 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:28.372095Z", 2023-10-18T20:30:35.129 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.129 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.130 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.130 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.130 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.130 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.130 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.131 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.131 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_fuMmfg1vzd3XP5PWV61484UCrgevCr6", 2023-10-18T20:30:35.131 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.131 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.132 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.132 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000059", 2023-10-18T20:30:35.132 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.133 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.133 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.133 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:35.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.134 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.134 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.134 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.134 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.134 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:35.135 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:28.881315Z", 2023-10-18T20:30:35.135 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.135 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.135 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.135 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.136 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.136 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.136 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.136 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.137 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_w1sjIr1W2Tp4p1QQHFAlJt_IQ7MDkVE", 2023-10-18T20:30:35.137 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.137 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.137 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.138 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000060", 2023-10-18T20:30:35.138 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:30:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:30:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.403090Z", 2023-10-18T20:30:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_XCyOk9zuJu7iBBmJvSCkeg2qSeZaMpA", 2023-10-18T20:30:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.144 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000061", 2023-10-18T20:30:35.144 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.144 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.144 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.144 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:30:35.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.145 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.145 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.145 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.145 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.146 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.146 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.714009Z", 2023-10-18T20:30:35.146 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.146 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.147 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.147 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.147 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.147 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.147 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.148 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.148 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_oohKCUsnB9oxl0ToeJzBVO2TnezCzgw", 2023-10-18T20:30:35.148 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.148 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.149 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.149 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000062", 2023-10-18T20:30:35.149 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.150 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.150 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.150 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:35.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.150 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.151 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.151 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.151 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.151 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.151 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.736585Z", 2023-10-18T20:30:35.152 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.152 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.152 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.152 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.152 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.153 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.153 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.153 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.153 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_kML65yYmM343MkRnSp9vLcM-kxQFvLW", 2023-10-18T20:30:35.154 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.154 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.154 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.155 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000065", 2023-10-18T20:30:35.155 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.155 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.155 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.155 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:30:35.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.156 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.156 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.156 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.156 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.157 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:35.157 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:32.637722Z", 2023-10-18T20:30:35.157 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.157 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.157 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.158 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.158 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.158 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.158 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.159 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.159 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_ewQOS4aS2m4Xg3-WSweNwmkzGLbRKSR", 2023-10-18T20:30:35.159 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.159 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.160 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.160 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000076", 2023-10-18T20:30:35.160 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.161 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.161 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.161 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:35.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.161 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.162 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.162 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.162 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.162 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.162 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:26.141311Z", 2023-10-18T20:30:35.163 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.163 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.163 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.163 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.163 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.164 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.164 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.164 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.164 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_6GzQLETlyMD2x-s4UN4hG0K_-YEv9n3", 2023-10-18T20:30:35.165 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.165 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.165 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.166 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000133", 2023-10-18T20:30:35.166 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.166 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.167 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 16 2023-10-18T20:30:35.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.167 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.167 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.167 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.168 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.168 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.168 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.850838Z", 2023-10-18T20:30:35.168 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.168 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.169 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.169 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.169 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.169 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.169 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.170 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.170 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_0mLxAgR73qXOhUJF_JRzlWN-5z5PkgD", 2023-10-18T20:30:35.170 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.170 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.171 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.171 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000162", 2023-10-18T20:30:35.171 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.172 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.172 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.172 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:30:35.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.172 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.173 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.173 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.173 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.173 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:35.173 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.108982Z", 2023-10-18T20:30:35.174 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.174 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.174 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.174 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.174 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.175 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.175 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.175 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.175 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_uqFkb3EAlysXdnkCWHzHgGGc0_0Ing3", 2023-10-18T20:30:35.176 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.176 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.176 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.178 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000271", 2023-10-18T20:30:35.178 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.178 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.178 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.179 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:30:35.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.179 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.179 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.180 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.180 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.180 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.180 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.234381Z", 2023-10-18T20:30:35.180 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.181 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.181 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.181 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.181 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.181 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.182 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.182 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.182 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_Z9zKhoCZCaBlkEX5SgsejHgli48Bl5D", 2023-10-18T20:30:35.182 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.183 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.183 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.183 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000283", 2023-10-18T20:30:35.184 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.184 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.184 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.184 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:35.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.185 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.185 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.185 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.185 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.185 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.186 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.148673Z", 2023-10-18T20:30:35.186 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.186 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.186 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.186 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.187 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.187 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.187 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.187 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.188 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_FWdHd_qIykdKeNhlKO0V5hExO08Vgrp", 2023-10-18T20:30:35.188 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.188 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.188 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.189 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000317", 2023-10-18T20:30:35.189 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.189 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.190 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.190 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:35.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.190 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.190 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.191 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.191 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.191 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:30:35.191 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.294926Z", 2023-10-18T20:30:35.191 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.192 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.192 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.192 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.192 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.192 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.193 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.193 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.193 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_M1AUifqWNl47jNnXom0vGA2YtVM3g1X", 2023-10-18T20:30:35.193 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.194 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.194 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.195 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000423", 2023-10-18T20:30:35.195 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.195 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.195 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.195 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:30:35.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.196 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.196 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.196 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.196 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.197 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:30:35.197 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:30.240139Z", 2023-10-18T20:30:35.197 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.197 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.197 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.198 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.198 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.198 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.198 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.198 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.199 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_D4JwRPrFOBfJeyx9gHD7KY8sVLuM1Ml", 2023-10-18T20:30:35.199 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.199 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.199 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:35.200 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj00000000000000000467", 2023-10-18T20:30:35.200 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:30:35.201 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:30:35.201 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:30:35.201 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:30:35.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.201 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:30:35.202 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:30:35.202 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:30:35.202 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:30:35.202 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:30:35.202 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:30:31.027459Z", 2023-10-18T20:30:35.203 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:30:35.203 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:30:35.203 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:30:35.203 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:30:35.204 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:30:35.204 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:30:35.204 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:30:35.204 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:30:35.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:35.205 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_yzArmkgXcrmcUEF4rBVPa0wQc18CpcF", 2023-10-18T20:30:35.205 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:30:35.205 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:30:35.205 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:30:35.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:35.206 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:30:35.206 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-18T20:30:35.207 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T20:30:35.210 INFO:tasks.rgw.c1.client.1:Started 2023-10-18T20:30:35.210 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T20:30:35.233 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:30:35.233 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:30:35.234 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T20:30:36.235 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:36.258 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:30:36.259 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:30:36.268 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 23777 0 --:--:-- --:--:-- --:--:-- 23777 2023-10-18T20:30:36.269 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T20:30:36.269 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-58 2023-10-18T20:30:36.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:30:36.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:30:36.270 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:30:36.365 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:36.365 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:36.406 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:36.403+0000 7fc1a1f64780 1 bucket sync caught up with source: 2023-10-18T20:30:36.406 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.55.6, 00000000002.68.6, 00000000002.52.6, 00000000002.50.6, 00000000002.76.6, 00000000001.68.6, 00000000002.42.6, 00000000002.66.6, 00000000001.49.6, , 00000000002.49.6, 00000000002.53.6, 00000000001.62.6, 00000000002.101.6, , 00000000002.66.6, 00000000002.57.6, 00000000002.68.6, 00000000002.83.6, 00000000002.85.6, 00000000002.64.6, 00000000002.49.6, 00000000002.49.6, 00000000002.77.6, 00000000002.50.6, 00000000002.47.6, 00000000002.56.6, 00000000002.85.6, 00000000002.51.6, 00000000002.83.6, 00000000001.54.6, 00000000002.99.6, 00000000002.74.6, 00000000002.44.6, 00000000001.53.6, 00000000002.48.6, ] 2023-10-18T20:30:36.406 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.55.6, 00000000002.68.6, 00000000002.52.6, 00000000002.50.6, 00000000002.76.6, 00000000001.68.6, 00000000002.42.6, 00000000002.66.6, 00000000001.49.6, , 00000000002.49.6, 00000000002.53.6, 00000000001.62.6, 00000000002.101.6, , 00000000002.66.6, 00000000002.57.6, 00000000002.68.6, 00000000002.83.6, 00000000002.85.6, 00000000002.64.6, 00000000002.49.6, 00000000002.49.6, 00000000002.77.6, 00000000002.50.6, 00000000002.47.6, 00000000002.56.6, 00000000002.85.6, 00000000002.51.6, 00000000002.83.6, 00000000001.54.6, 00000000002.99.6, 00000000002.74.6, 00000000002.44.6, 00000000001.53.6, 00000000002.48.6, ] 2023-10-18T20:30:36.407 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:36.403+0000 7fc1a1f64780 0 bucket checkpoint complete 2023-10-18T20:30:36.420 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-58 2023-10-18T20:30:36.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:30:36.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:30:36.421 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:30:36.478 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:36.478 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:36.515 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:36.513+0000 7fd33fa8c780 1 waiting for incremental sync to catch up: 2023-10-18T20:30:36.516 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.603.6, 00000000001.297.6, 00000000001.309.6, 00000000001.692.6, 00000000001.934.6, 00000000001.500.6, 00000000001.920.6, 00000000001.310.6, , 00000000001.1172.6, 00000000001.1173.6] 2023-10-18T20:30:36.516 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000005.611.6, 00000000006.307.6, 00000000006.326.6, 00000000010.710.6, 00000000005.942.6, 00000000008.514.6, 00000000008.934.6, 00000000006.330.6, 00000000006.181.6, 00000000007.1191.6, 00000000004.1189.6] 2023-10-18T20:30:41.517 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:41.515+0000 7fd33fa8c780 1 bucket sync caught up with source: 2023-10-18T20:30:41.517 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000005.611.6, 00000000006.307.6, 00000000006.326.6, 00000000010.710.6, 00000000005.942.6, 00000000008.514.6, 00000000008.934.6, 00000000006.330.6, 00000000006.181.6, 00000000007.1191.6, 00000000004.1189.6] 2023-10-18T20:30:41.517 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000005.611.6, 00000000006.307.6, 00000000006.326.6, 00000000010.710.6, 00000000005.942.6, 00000000008.514.6, 00000000008.934.6, 00000000006.330.6, 00000000006.181.6, 00000000007.1191.6, 00000000004.1189.6] 2023-10-18T20:30:41.517 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:41.515+0000 7fd33fa8c780 0 bucket checkpoint complete 2023-10-18T20:30:41.534 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-58 zones={a2, a1} 2023-10-18T20:30:41.534 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:41.535 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:41.535 DEBUG:boto:Method: HEAD 2023-10-18T20:30:41.536 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:41.536 DEBUG:boto:Data: 2023-10-18T20:30:41.537 DEBUG:boto:Headers: {} 2023-10-18T20:30:41.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:41.538 DEBUG:boto:Port: 8001 2023-10-18T20:30:41.538 DEBUG:boto:Params: {} 2023-10-18T20:30:41.539 DEBUG:boto:Token: None 2023-10-18T20:30:41.539 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:41 GMT /bofrnf-58/ 2023-10-18T20:30:41.540 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lSNb/7IdQGTOnyZqfPTTPYMhKrE= 2023-10-18T20:30:41.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lSNb/7IdQGTOnyZqfPTTPYMhKrE=', 'Content-Length': '0'} 2023-10-18T20:30:41.541 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-10-18T20:30:41.541 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:30:42.360 DEBUG:boto:Token: None 2023-10-18T20:30:42.361 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:41 GMT /bofrnf-58/ 2023-10-18T20:30:42.362 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lSNb/7IdQGTOnyZqfPTTPYMhKrE= 2023-10-18T20:30:42.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lSNb/7IdQGTOnyZqfPTTPYMhKrE=', 'Content-Length': '0'} 2023-10-18T20:30:42.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069eade7fcdc1443d-0065304072-1af4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.369 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.370 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.370 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.371 DEBUG:boto:Path: /bofrnf-58/ 2023-10-18T20:30:42.371 DEBUG:boto:Data: 2023-10-18T20:30:42.372 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.372 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.372 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.373 DEBUG:boto:Params: {} 2023-10-18T20:30:42.373 DEBUG:boto:Token: None 2023-10-18T20:30:42.374 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/ 2023-10-18T20:30:42.374 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LawoEnb2s3k1b4vriANEfwF6dUU= 2023-10-18T20:30:42.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LawoEnb2s3k1b4vriANEfwF6dUU=', 'Content-Length': '0'} 2023-10-18T20:30:42.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000600041ece1d0be93-0065304072-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.382 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:30:42.382 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.383 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.383 DEBUG:boto:path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.384 DEBUG:boto:auth_path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.384 DEBUG:boto:Method: GET 2023-10-18T20:30:42.385 DEBUG:boto:Path: /bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.385 DEBUG:boto:Data: 2023-10-18T20:30:42.385 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.386 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.387 DEBUG:boto:Params: {} 2023-10-18T20:30:42.387 DEBUG:boto:Token: None 2023-10-18T20:30:42.387 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/?versions 2023-10-18T20:30:42.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TV7SKdVAAkEKt7cSWxBt+jEkxoM= 2023-10-18T20:30:42.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TV7SKdVAAkEKt7cSWxBt+jEkxoM=', 'Content-Length': '0'} 2023-10-18T20:30:42.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039f7f752a53db5c0-0065304072-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.392 DEBUG:boto:b'bofrnf-58999falseobj00000000000000000000nulltrue2023-10-18T20:30:32.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-18T20:30:31.840Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-18T20:30:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-18T20:30:32.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-18T20:30:32.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-18T20:30:31.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-18T20:30:31.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-18T20:30:32.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-18T20:30:29.743Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-18T20:30:32.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-18T20:30:31.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-18T20:30:32.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-18T20:30:32.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-18T20:30:32.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-18T20:30:31.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-18T20:30:29.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-18T20:30:32.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-18T20:30:31.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-18T20:30:32.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-18T20:30:32.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-18T20:30:32.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-18T20:30:32.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-18T20:30:30.350Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-18T20:30:30.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-18T20:30:32.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-18T20:30:29.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-18T20:30:32.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-18T20:30:32.295Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-18T20:30:26.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-18T20:30:31.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-18T20:30:32.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-18T20:30:29.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-18T20:30:27.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-18T20:30:32.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-18T20:30:31.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-18T20:30:29.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-18T20:30:32.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-18T20:30:28.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-18T20:30:29.255Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-18T20:30:32.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-18T20:30:28.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-18T20:30:32.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-18T20:30:29.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-18T20:30:30.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-18T20:30:32.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-18T20:30:27.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-18T20:30:31.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-18T20:30:28.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-18T20:30:28.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-18T20:30:32.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-18T20:30:30.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-18T20:30:31.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-18T20:30:32.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-18T20:30:26.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-18T20:30:30.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-18T20:30:31.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-18T20:30:31.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-18T20:30:31.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-18T20:30:31.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-18T20:30:30.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-18T20:30:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-18T20:30:42.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-18T20:30:42.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-18T20:30:42.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-18T20:30:42.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-18T20:30:42.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-18T20:30:42.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-18T20:30:42.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-10-18T20:30:42.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-10-18T20:30:42.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-10-18T20:30:42.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-18T20:30:42.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-18T20:30:42.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-18T20:30:42.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-18T20:30:42.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-10-18T20:30:42.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-10-18T20:30:42.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-10-18T20:30:42.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-10-18T20:30:42.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-10-18T20:30:42.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-10-18T20:30:42.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-10-18T20:30:42.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-10-18T20:30:42.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-10-18T20:30:42.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-10-18T20:30:42.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-10-18T20:30:42.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-10-18T20:30:42.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-10-18T20:30:42.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-10-18T20:30:42.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-10-18T20:30:42.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-10-18T20:30:42.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-10-18T20:30:42.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-10-18T20:30:42.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-10-18T20:30:42.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-10-18T20:30:42.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-10-18T20:30:42.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-10-18T20:30:42.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-10-18T20:30:42.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-10-18T20:30:42.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-10-18T20:30:42.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-10-18T20:30:42.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-10-18T20:30:42.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-10-18T20:30:42.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-10-18T20:30:42.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-10-18T20:30:42.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-10-18T20:30:42.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-10-18T20:30:42.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-10-18T20:30:42.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-10-18T20:30:42.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-10-18T20:30:42.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-10-18T20:30:42.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-10-18T20:30:42.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-10-18T20:30:42.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-10-18T20:30:42.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-10-18T20:30:42.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-10-18T20:30:42.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-10-18T20:30:42.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-10-18T20:30:42.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-10-18T20:30:42.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-10-18T20:30:42.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-10-18T20:30:42.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-10-18T20:30:42.418 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:30:42.419 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.419 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.420 DEBUG:boto:path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.420 DEBUG:boto:auth_path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.420 DEBUG:boto:Method: GET 2023-10-18T20:30:42.421 DEBUG:boto:Path: /bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.421 DEBUG:boto:Data: 2023-10-18T20:30:42.422 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.422 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.422 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.423 DEBUG:boto:Params: {} 2023-10-18T20:30:42.423 DEBUG:boto:Token: None 2023-10-18T20:30:42.424 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/?versions 2023-10-18T20:30:42.424 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TV7SKdVAAkEKt7cSWxBt+jEkxoM= 2023-10-18T20:30:42.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TV7SKdVAAkEKt7cSWxBt+jEkxoM=', 'Content-Length': '0'} 2023-10-18T20:30:42.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faf030cbf26a5280-0065304072-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.430 DEBUG:boto:b'bofrnf-58999falseobj00000000000000000000nulltrue2023-10-18T20:30:32.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-18T20:30:31.840Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-18T20:30:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-18T20:30:32.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-18T20:30:32.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-18T20:30:31.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-18T20:30:31.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-18T20:30:32.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-18T20:30:29.743Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-18T20:30:32.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-18T20:30:31.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-18T20:30:32.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-18T20:30:32.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-18T20:30:32.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-18T20:30:31.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-18T20:30:29.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-18T20:30:32.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-18T20:30:31.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-18T20:30:32.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-18T20:30:32.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-18T20:30:32.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-18T20:30:32.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-18T20:30:30.350Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-18T20:30:30.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-18T20:30:32.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-18T20:30:29.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-18T20:30:32.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-18T20:30:32.295Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-18T20:30:26.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-18T20:30:31.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-18T20:30:32.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-18T20:30:29.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-18T20:30:27.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-18T20:30:32.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-18T20:30:31.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-18T20:30:29.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-18T20:30:32.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-18T20:30:28.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-18T20:30:29.255Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-18T20:30:32.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-18T20:30:28.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-18T20:30:32.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-18T20:30:29.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-18T20:30:30.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-18T20:30:32.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-18T20:30:27.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-18T20:30:31.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-18T20:30:28.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-18T20:30:28.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-18T20:30:32.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-18T20:30:30.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-18T20:30:31.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-18T20:30:32.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-18T20:30:26.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-18T20:30:30.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-18T20:30:31.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-18T20:30:31.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-18T20:30:31.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-18T20:30:31.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-18T20:30:30.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-18T20:30:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-18T20:30:42.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-18T20:30:42.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-18T20:30:42.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-18T20:30:42.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-18T20:30:42.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-18T20:30:42.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-18T20:30:42.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-10-18T20:30:42.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-10-18T20:30:42.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-10-18T20:30:42.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-18T20:30:42.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-18T20:30:42.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-18T20:30:42.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-18T20:30:42.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-10-18T20:30:42.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-10-18T20:30:42.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-10-18T20:30:42.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-10-18T20:30:42.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-10-18T20:30:42.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-10-18T20:30:42.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-10-18T20:30:42.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-10-18T20:30:42.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-10-18T20:30:42.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-10-18T20:30:42.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-10-18T20:30:42.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-10-18T20:30:42.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-10-18T20:30:42.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-10-18T20:30:42.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-10-18T20:30:42.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-10-18T20:30:42.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-10-18T20:30:42.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-10-18T20:30:42.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-10-18T20:30:42.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-10-18T20:30:42.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-10-18T20:30:42.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-10-18T20:30:42.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-10-18T20:30:42.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-10-18T20:30:42.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-10-18T20:30:42.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-10-18T20:30:42.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-10-18T20:30:42.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-10-18T20:30:42.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-10-18T20:30:42.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-10-18T20:30:42.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-10-18T20:30:42.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-10-18T20:30:42.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-10-18T20:30:42.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-10-18T20:30:42.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-10-18T20:30:42.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-10-18T20:30:42.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-10-18T20:30:42.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-10-18T20:30:42.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-10-18T20:30:42.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-10-18T20:30:42.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-10-18T20:30:42.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-10-18T20:30:42.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-10-18T20:30:42.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-10-18T20:30:42.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-10-18T20:30:42.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-10-18T20:30:42.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-10-18T20:30:42.458 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.459 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.459 DEBUG:boto:path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.460 DEBUG:boto:auth_path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.460 DEBUG:boto:Method: GET 2023-10-18T20:30:42.460 DEBUG:boto:Path: /bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.461 DEBUG:boto:Data: 2023-10-18T20:30:42.461 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.462 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.462 DEBUG:boto:Params: {} 2023-10-18T20:30:42.463 DEBUG:boto:Token: None 2023-10-18T20:30:42.463 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/?versions 2023-10-18T20:30:42.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TV7SKdVAAkEKt7cSWxBt+jEkxoM= 2023-10-18T20:30:42.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TV7SKdVAAkEKt7cSWxBt+jEkxoM=', 'Content-Length': '0'} 2023-10-18T20:30:42.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000031730dac2b4337a-0065304072-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.468 DEBUG:boto:b'bofrnf-58999falseobj00000000000000000000nulltrue2023-10-18T20:30:32.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-18T20:30:31.840Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-18T20:30:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-18T20:30:32.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-18T20:30:32.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-18T20:30:31.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-18T20:30:31.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-18T20:30:32.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-18T20:30:29.743Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-18T20:30:32.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-18T20:30:31.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-18T20:30:32.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-18T20:30:32.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-18T20:30:32.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-18T20:30:31.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-18T20:30:29.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-18T20:30:32.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-18T20:30:31.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-18T20:30:32.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-18T20:30:32.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-18T20:30:32.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-18T20:30:32.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-18T20:30:30.350Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-18T20:30:30.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-18T20:30:32.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-18T20:30:29.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-18T20:30:32.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-18T20:30:32.295Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-18T20:30:26.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-18T20:30:31.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-18T20:30:32.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-18T20:30:29.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-18T20:30:27.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-18T20:30:32.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-18T20:30:31.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-18T20:30:29.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-18T20:30:32.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-18T20:30:28.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-18T20:30:29.255Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-18T20:30:32.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-18T20:30:28.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-18T20:30:32.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-18T20:30:29.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-18T20:30:30.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-18T20:30:32.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-18T20:30:27.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-18T20:30:31.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-18T20:30:28.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-18T20:30:28.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-18T20:30:32.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-18T20:30:30.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-18T20:30:31.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-18T20:30:32.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-18T20:30:26.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-18T20:30:30.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-18T20:30:31.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-18T20:30:31.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-18T20:30:31.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-18T20:30:31.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-18T20:30:30.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-18T20:30:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.471 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.473 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.474 DEBUG:boto:path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.474 DEBUG:boto:auth_path=/bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.474 DEBUG:boto:Method: GET 2023-10-18T20:30:42.475 DEBUG:boto:Path: /bofrnf-58/?versions&max-keys=999 2023-10-18T20:30:42.475 DEBUG:boto:Data: 2023-10-18T20:30:42.476 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.476 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.476 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.477 DEBUG:boto:Params: {} 2023-10-18T20:30:42.477 DEBUG:boto:Token: None 2023-10-18T20:30:42.478 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/?versions 2023-10-18T20:30:42.478 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TV7SKdVAAkEKt7cSWxBt+jEkxoM= 2023-10-18T20:30:42.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TV7SKdVAAkEKt7cSWxBt+jEkxoM=', 'Content-Length': '0'} 2023-10-18T20:30:42.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d35a48464b186129-0065304072-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.484 DEBUG:boto:b'bofrnf-58999falseobj00000000000000000000nulltrue2023-10-18T20:30:32.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-18T20:30:31.840Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-18T20:30:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-18T20:30:32.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-18T20:30:32.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-18T20:30:31.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-18T20:30:31.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-18T20:30:32.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-18T20:30:29.743Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-18T20:30:32.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-18T20:30:31.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-18T20:30:32.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-18T20:30:32.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-18T20:30:32.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-18T20:30:31.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-18T20:30:29.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-18T20:30:32.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-18T20:30:31.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-18T20:30:32.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-18T20:30:32.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-18T20:30:32.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-18T20:30:32.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-18T20:30:30.350Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-18T20:30:30.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-18T20:30:32.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-18T20:30:29.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-18T20:30:32.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-18T20:30:32.295Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-18T20:30:26.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-18T20:30:31.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-18T20:30:32.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-18T20:30:29.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-18T20:30:27.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-18T20:30:32.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-18T20:30:31.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-18T20:30:29.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-18T20:30:32.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-18T20:30:28.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-18T20:30:29.255Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-18T20:30:32.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-18T20:30:28.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-18T20:30:32.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-18T20:30:29.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-18T20:30:30.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-18T20:30:32.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-10-18T20:30:27.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-18T20:30:31.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-18T20:30:28.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-18T20:30:28.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-18T20:30:32.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-18T20:30:30.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-18T20:30:31.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-18T20:30:32.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-18T20:30:26.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-18T20:30:30.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-18T20:30:31.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-18T20:30:31.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-18T20:30:31.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-18T20:30:31.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-18T20:30:30.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-10-18T20:30:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-18T20:30:42.488 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.488 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.488 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.489 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.489 DEBUG:boto:Method: GET 2023-10-18T20:30:42.490 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.490 DEBUG:boto:Data: 2023-10-18T20:30:42.491 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.491 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.491 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.492 DEBUG:boto:Params: {} 2023-10-18T20:30:42.492 DEBUG:boto:Token: None 2023-10-18T20:30:42.493 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.493 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/fYyzXOGFCdTeyCUdrVuVLVgwpA= 2023-10-18T20:30:42.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/fYyzXOGFCdTeyCUdrVuVLVgwpA=', 'Content-Length': '0'} 2023-10-18T20:30:42.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813053ee92ed9e7f-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.498 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.498 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.499 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.499 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.500 DEBUG:boto:Method: GET 2023-10-18T20:30:42.500 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.500 DEBUG:boto:Data: 2023-10-18T20:30:42.501 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.501 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.502 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.502 DEBUG:boto:Params: {} 2023-10-18T20:30:42.503 DEBUG:boto:Token: None 2023-10-18T20:30:42.503 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.504 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/fYyzXOGFCdTeyCUdrVuVLVgwpA= 2023-10-18T20:30:42.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/fYyzXOGFCdTeyCUdrVuVLVgwpA=', 'Content-Length': '0'} 2023-10-18T20:30:42.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399b7344b2ec5f55-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.507 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.507 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.508 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.508 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.509 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.509 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.509 DEBUG:boto:Data: 2023-10-18T20:30:42.510 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.510 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.511 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.511 DEBUG:boto:Params: {} 2023-10-18T20:30:42.511 DEBUG:boto:Token: None 2023-10-18T20:30:42.512 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.512 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:k/CDdMjUGT7DRfux3XP6HruDEJE= 2023-10-18T20:30:42.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:k/CDdMjUGT7DRfux3XP6HruDEJE=', 'Content-Length': '0'} 2023-10-18T20:30:42.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faed99d91e6c98e5-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.515 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.516 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.516 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.517 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.517 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.517 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.518 DEBUG:boto:Data: 2023-10-18T20:30:42.518 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.519 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.519 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.519 DEBUG:boto:Params: {} 2023-10-18T20:30:42.520 DEBUG:boto:Token: None 2023-10-18T20:30:42.520 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.521 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:k/CDdMjUGT7DRfux3XP6HruDEJE= 2023-10-18T20:30:42.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:k/CDdMjUGT7DRfux3XP6HruDEJE=', 'Content-Length': '0'} 2023-10-18T20:30:42.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002671509618feb816-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-18T20:30:42.524 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.524 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.525 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.525 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.526 DEBUG:boto:Method: GET 2023-10-18T20:30:42.526 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.527 DEBUG:boto:Data: 2023-10-18T20:30:42.527 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.528 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.528 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.528 DEBUG:boto:Params: {} 2023-10-18T20:30:42.529 DEBUG:boto:Token: None 2023-10-18T20:30:42.529 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.530 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/fYyzXOGFCdTeyCUdrVuVLVgwpA= 2023-10-18T20:30:42.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/fYyzXOGFCdTeyCUdrVuVLVgwpA=', 'Content-Length': '0'} 2023-10-18T20:30:42.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000661aa1b924761b73-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.574 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.575 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.575 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.576 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.576 DEBUG:boto:Method: GET 2023-10-18T20:30:42.577 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.577 DEBUG:boto:Data: 2023-10-18T20:30:42.578 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.578 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.579 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.579 DEBUG:boto:Params: {} 2023-10-18T20:30:42.580 DEBUG:boto:Token: None 2023-10-18T20:30:42.580 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000?versionId=null 2023-10-18T20:30:42.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/fYyzXOGFCdTeyCUdrVuVLVgwpA= 2023-10-18T20:30:42.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/fYyzXOGFCdTeyCUdrVuVLVgwpA=', 'Content-Length': '0'} 2023-10-18T20:30:42.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a06cfcbe24ec77f-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.584 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.584 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.585 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.585 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.586 DEBUG:boto:Method: GET 2023-10-18T20:30:42.586 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.587 DEBUG:boto:Data: 2023-10-18T20:30:42.587 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.587 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.588 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.588 DEBUG:boto:Params: {} 2023-10-18T20:30:42.589 DEBUG:boto:Token: None 2023-10-18T20:30:42.589 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/ 2023-10-18T20:30:42.590 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY= 2023-10-18T20:30:42.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY=', 'Content-Length': '0'} 2023-10-18T20:30:42.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091cb8dab4e58172a-0065304072-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.593 DEBUG:boto:b'bofrnf-58obj000000000000000000001000falseobj000000000000000000002023-10-18T20:30:32.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.594 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.594 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.595 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.595 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.596 DEBUG:boto:Method: GET 2023-10-18T20:30:42.596 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.596 DEBUG:boto:Data: 2023-10-18T20:30:42.597 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.598 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.598 DEBUG:boto:Params: {} 2023-10-18T20:30:42.599 DEBUG:boto:Token: None 2023-10-18T20:30:42.599 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/ 2023-10-18T20:30:42.600 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY= 2023-10-18T20:30:42.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY=', 'Content-Length': '0'} 2023-10-18T20:30:42.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1e1767c2ad16dac-0065304072-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.646 DEBUG:boto:b'bofrnf-58obj000000000000000000001000falseobj000000000000000000002023-10-18T20:30:32.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.649 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.650 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.651 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.651 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.651 DEBUG:boto:Data: 2023-10-18T20:30:42.652 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.652 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.653 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.653 DEBUG:boto:Params: {} 2023-10-18T20:30:42.654 DEBUG:boto:Token: None 2023-10-18T20:30:42.654 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.655 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i3Ft+XYtdNuKjV233Fa/qIfefDc= 2023-10-18T20:30:42.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i3Ft+XYtdNuKjV233Fa/qIfefDc=', 'Content-Length': '0'} 2023-10-18T20:30:42.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099e2db0b91ad208-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.658 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.658 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.659 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.659 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.659 DEBUG:boto:Data: 2023-10-18T20:30:42.660 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.660 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.661 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.661 DEBUG:boto:Params: {} 2023-10-18T20:30:42.661 DEBUG:boto:Token: None 2023-10-18T20:30:42.662 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i3Ft+XYtdNuKjV233Fa/qIfefDc= 2023-10-18T20:30:42.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i3Ft+XYtdNuKjV233Fa/qIfefDc=', 'Content-Length': '0'} 2023-10-18T20:30:42.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d67d44f81aa6a11-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-18T20:30:42.666 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.666 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.667 DEBUG:boto:Method: GET 2023-10-18T20:30:42.667 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.668 DEBUG:boto:Data: 2023-10-18T20:30:42.668 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.669 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.669 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.670 DEBUG:boto:Params: {} 2023-10-18T20:30:42.671 DEBUG:boto:Token: None 2023-10-18T20:30:42.672 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.672 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:djigVqcfy7wsDDrCYtVINKrI8Og= 2023-10-18T20:30:42.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:djigVqcfy7wsDDrCYtVINKrI8Og=', 'Content-Length': '0'} 2023-10-18T20:30:42.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d25f4f53952810e9-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.676 DEBUG:boto:path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.677 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.677 DEBUG:boto:Method: GET 2023-10-18T20:30:42.678 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.678 DEBUG:boto:Data: 2023-10-18T20:30:42.679 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.679 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.680 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.680 DEBUG:boto:Params: {} 2023-10-18T20:30:42.680 DEBUG:boto:Token: None 2023-10-18T20:30:42.681 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000000 2023-10-18T20:30:42.681 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:djigVqcfy7wsDDrCYtVINKrI8Og= 2023-10-18T20:30:42.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:djigVqcfy7wsDDrCYtVINKrI8Og=', 'Content-Length': '0'} 2023-10-18T20:30:42.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e099978388a725-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.685 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.685 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.686 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.686 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.687 DEBUG:boto:Method: GET 2023-10-18T20:30:42.687 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.687 DEBUG:boto:Data: 2023-10-18T20:30:42.688 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.688 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.689 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.689 DEBUG:boto:Params: {} 2023-10-18T20:30:42.689 DEBUG:boto:Token: None 2023-10-18T20:30:42.690 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/ 2023-10-18T20:30:42.690 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY= 2023-10-18T20:30:42.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY=', 'Content-Length': '0'} 2023-10-18T20:30:42.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b352d79d3a659e3-0065304072-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.694 DEBUG:boto:b'bofrnf-58obj000000000000000000001000falseobj000000000000000000002023-10-18T20:30:32.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.695 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.695 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.696 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.697 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.697 DEBUG:boto:Method: GET 2023-10-18T20:30:42.698 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000000 2023-10-18T20:30:42.699 DEBUG:boto:Data: 2023-10-18T20:30:42.700 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.700 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.701 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.702 DEBUG:boto:Params: {} 2023-10-18T20:30:42.703 DEBUG:boto:Token: None 2023-10-18T20:30:42.704 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/ 2023-10-18T20:30:42.705 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY= 2023-10-18T20:30:42.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY=', 'Content-Length': '0'} 2023-10-18T20:30:42.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea75316fbe537ed-0065304072-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.710 DEBUG:boto:b'bofrnf-58obj000000000000000000001000falseobj000000000000000000002023-10-18T20:30:32.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-18T20:30:42.711 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.712 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.712 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.713 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.713 DEBUG:boto:Method: GET 2023-10-18T20:30:42.713 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.714 DEBUG:boto:Data: 2023-10-18T20:30:42.714 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.715 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.716 DEBUG:boto:Params: {} 2023-10-18T20:30:42.716 DEBUG:boto:Token: None 2023-10-18T20:30:42.717 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pONJEGfSYZHY/4DwT2tQmgdWZFc= 2023-10-18T20:30:42.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pONJEGfSYZHY/4DwT2tQmgdWZFc=', 'Content-Length': '0'} 2023-10-18T20:30:42.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf7d6d8e89897d8-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.721 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.721 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.721 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.722 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.722 DEBUG:boto:Method: GET 2023-10-18T20:30:42.723 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.723 DEBUG:boto:Data: 2023-10-18T20:30:42.724 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.724 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.724 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.725 DEBUG:boto:Params: {} 2023-10-18T20:30:42.725 DEBUG:boto:Token: None 2023-10-18T20:30:42.726 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.726 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pONJEGfSYZHY/4DwT2tQmgdWZFc= 2023-10-18T20:30:42.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pONJEGfSYZHY/4DwT2tQmgdWZFc=', 'Content-Length': '0'} 2023-10-18T20:30:42.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bd75d540104bd8d-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.730 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.730 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.730 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.731 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.731 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.732 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.732 DEBUG:boto:Data: 2023-10-18T20:30:42.732 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.733 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.733 DEBUG:boto:Params: {} 2023-10-18T20:30:42.734 DEBUG:boto:Token: None 2023-10-18T20:30:42.734 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cMi74yh+rd7ixO5mPVN/ahYhZ5I= 2023-10-18T20:30:42.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cMi74yh+rd7ixO5mPVN/ahYhZ5I=', 'Content-Length': '0'} 2023-10-18T20:30:42.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8a53ff5af82cdd2-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.739 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.740 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.740 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.741 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.742 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.742 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.743 DEBUG:boto:Data: 2023-10-18T20:30:42.744 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.745 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.745 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.746 DEBUG:boto:Params: {} 2023-10-18T20:30:42.747 DEBUG:boto:Token: None 2023-10-18T20:30:42.748 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cMi74yh+rd7ixO5mPVN/ahYhZ5I= 2023-10-18T20:30:42.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cMi74yh+rd7ixO5mPVN/ahYhZ5I=', 'Content-Length': '0'} 2023-10-18T20:30:42.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdcbb397b9cbd3df-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-18T20:30:42.752 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.752 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.753 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.753 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.754 DEBUG:boto:Method: GET 2023-10-18T20:30:42.754 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.755 DEBUG:boto:Data: 2023-10-18T20:30:42.755 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.755 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.756 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.756 DEBUG:boto:Params: {} 2023-10-18T20:30:42.757 DEBUG:boto:Token: None 2023-10-18T20:30:42.757 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.757 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pONJEGfSYZHY/4DwT2tQmgdWZFc= 2023-10-18T20:30:42.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pONJEGfSYZHY/4DwT2tQmgdWZFc=', 'Content-Length': '0'} 2023-10-18T20:30:42.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e50d0627dfffae6-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.761 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.761 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.762 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.762 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.762 DEBUG:boto:Method: GET 2023-10-18T20:30:42.763 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.763 DEBUG:boto:Data: 2023-10-18T20:30:42.764 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.764 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.764 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.765 DEBUG:boto:Params: {} 2023-10-18T20:30:42.765 DEBUG:boto:Token: None 2023-10-18T20:30:42.766 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001?versionId=null 2023-10-18T20:30:42.766 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pONJEGfSYZHY/4DwT2tQmgdWZFc= 2023-10-18T20:30:42.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pONJEGfSYZHY/4DwT2tQmgdWZFc=', 'Content-Length': '0'} 2023-10-18T20:30:42.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a66afef2db66817c-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.814 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.815 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.816 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.816 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.816 DEBUG:boto:Method: GET 2023-10-18T20:30:42.817 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.817 DEBUG:boto:Data: 2023-10-18T20:30:42.818 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.818 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.819 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.819 DEBUG:boto:Params: {} 2023-10-18T20:30:42.820 DEBUG:boto:Token: None 2023-10-18T20:30:42.820 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/ 2023-10-18T20:30:42.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY= 2023-10-18T20:30:42.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY=', 'Content-Length': '0'} 2023-10-18T20:30:42.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e2809dc94a00600-0065304072-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.825 DEBUG:boto:b'bofrnf-58obj000000000000000000011000falseobj000000000000000000012023-10-18T20:30:31.840Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.825 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.826 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.826 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.827 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.827 DEBUG:boto:Method: GET 2023-10-18T20:30:42.827 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.828 DEBUG:boto:Data: 2023-10-18T20:30:42.828 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.829 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.829 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.830 DEBUG:boto:Params: {} 2023-10-18T20:30:42.830 DEBUG:boto:Token: None 2023-10-18T20:30:42.831 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/ 2023-10-18T20:30:42.831 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY= 2023-10-18T20:30:42.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY=', 'Content-Length': '0'} 2023-10-18T20:30:42.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da978ae0bbe2b8bf-0065304072-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.836 DEBUG:boto:b'bofrnf-58obj000000000000000000011000falseobj000000000000000000012023-10-18T20:30:31.840Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.837 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.837 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.838 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.838 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.839 DEBUG:boto:Data: 2023-10-18T20:30:42.839 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.840 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.840 DEBUG:boto:Params: {} 2023-10-18T20:30:42.841 DEBUG:boto:Token: None 2023-10-18T20:30:42.841 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.843 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RLdMyRtmIQ8xJhl/LTaZiiJAV08= 2023-10-18T20:30:42.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RLdMyRtmIQ8xJhl/LTaZiiJAV08=', 'Content-Length': '0'} 2023-10-18T20:30:42.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c60c49578931b89-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.846 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.847 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.847 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.848 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.848 DEBUG:boto:Data: 2023-10-18T20:30:42.849 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.849 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.849 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.850 DEBUG:boto:Params: {} 2023-10-18T20:30:42.850 DEBUG:boto:Token: None 2023-10-18T20:30:42.851 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.853 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RLdMyRtmIQ8xJhl/LTaZiiJAV08= 2023-10-18T20:30:42.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RLdMyRtmIQ8xJhl/LTaZiiJAV08=', 'Content-Length': '0'} 2023-10-18T20:30:42.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d50b7463f1e23bec-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-18T20:30:42.857 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.857 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.858 DEBUG:boto:Method: GET 2023-10-18T20:30:42.858 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.859 DEBUG:boto:Data: 2023-10-18T20:30:42.859 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.860 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.860 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.861 DEBUG:boto:Params: {} 2023-10-18T20:30:42.861 DEBUG:boto:Token: None 2023-10-18T20:30:42.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ShanyobUKcSmItYgOsoR0shJqkQ= 2023-10-18T20:30:42.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ShanyobUKcSmItYgOsoR0shJqkQ=', 'Content-Length': '0'} 2023-10-18T20:30:42.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebfaa78f7a3932a-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.865 DEBUG:boto:path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.866 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.866 DEBUG:boto:Method: GET 2023-10-18T20:30:42.867 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.867 DEBUG:boto:Data: 2023-10-18T20:30:42.867 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.868 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.869 DEBUG:boto:Params: {} 2023-10-18T20:30:42.869 DEBUG:boto:Token: None 2023-10-18T20:30:42.870 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000001 2023-10-18T20:30:42.870 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ShanyobUKcSmItYgOsoR0shJqkQ= 2023-10-18T20:30:42.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ShanyobUKcSmItYgOsoR0shJqkQ=', 'Content-Length': '0'} 2023-10-18T20:30:42.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c986b36a2ec2fd2-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.874 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.874 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.875 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.875 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.876 DEBUG:boto:Method: GET 2023-10-18T20:30:42.876 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.877 DEBUG:boto:Data: 2023-10-18T20:30:42.877 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.878 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.878 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.878 DEBUG:boto:Params: {} 2023-10-18T20:30:42.879 DEBUG:boto:Token: None 2023-10-18T20:30:42.879 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/ 2023-10-18T20:30:42.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY= 2023-10-18T20:30:42.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY=', 'Content-Length': '0'} 2023-10-18T20:30:42.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b39b523765faeff6-0065304072-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.884 DEBUG:boto:b'bofrnf-58obj000000000000000000011000falseobj000000000000000000012023-10-18T20:30:31.840Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.885 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:42.885 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:42.885 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.886 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.886 DEBUG:boto:Method: GET 2023-10-18T20:30:42.887 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000001 2023-10-18T20:30:42.887 DEBUG:boto:Data: 2023-10-18T20:30:42.888 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.888 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.888 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.889 DEBUG:boto:Params: {} 2023-10-18T20:30:42.889 DEBUG:boto:Token: None 2023-10-18T20:30:42.890 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/ 2023-10-18T20:30:42.890 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY= 2023-10-18T20:30:42.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MjhJu2u/K7GOaB3zpSv0JhlP1mY=', 'Content-Length': '0'} 2023-10-18T20:30:42.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa67462ebac82992-0065304072-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.896 DEBUG:boto:b'bofrnf-58obj000000000000000000011000falseobj000000000000000000012023-10-18T20:30:31.840Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:42.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-18T20:30:42.897 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.897 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.898 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.898 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.899 DEBUG:boto:Method: GET 2023-10-18T20:30:42.899 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.900 DEBUG:boto:Data: 2023-10-18T20:30:42.900 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.901 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.901 DEBUG:boto:Params: {} 2023-10-18T20:30:42.902 DEBUG:boto:Token: None 2023-10-18T20:30:42.902 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.903 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B4HFy1uz1jeHQs8+P4h/zmLlSyw= 2023-10-18T20:30:42.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B4HFy1uz1jeHQs8+P4h/zmLlSyw=', 'Content-Length': '0'} 2023-10-18T20:30:42.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310a1ad4d8a7762b-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.950 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.951 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.952 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.952 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.952 DEBUG:boto:Method: GET 2023-10-18T20:30:42.953 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.953 DEBUG:boto:Data: 2023-10-18T20:30:42.954 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.954 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.955 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.955 DEBUG:boto:Params: {} 2023-10-18T20:30:42.956 DEBUG:boto:Token: None 2023-10-18T20:30:42.956 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.957 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B4HFy1uz1jeHQs8+P4h/zmLlSyw= 2023-10-18T20:30:42.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B4HFy1uz1jeHQs8+P4h/zmLlSyw=', 'Content-Length': '0'} 2023-10-18T20:30:42.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df45e828d0058c5c-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.960 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.961 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.961 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.961 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.962 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.962 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.963 DEBUG:boto:Data: 2023-10-18T20:30:42.963 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.964 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.964 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.964 DEBUG:boto:Params: {} 2023-10-18T20:30:42.965 DEBUG:boto:Token: None 2023-10-18T20:30:42.965 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.966 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pjIRi7JjILcaz5exRb9dHub0Miw= 2023-10-18T20:30:42.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pjIRi7JjILcaz5exRb9dHub0Miw=', 'Content-Length': '0'} 2023-10-18T20:30:42.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dee4b628f75717f8-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.969 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.969 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.970 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.970 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.971 DEBUG:boto:Method: HEAD 2023-10-18T20:30:42.971 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.971 DEBUG:boto:Data: 2023-10-18T20:30:42.972 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.972 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.973 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.973 DEBUG:boto:Params: {} 2023-10-18T20:30:42.973 DEBUG:boto:Token: None 2023-10-18T20:30:42.974 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.974 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pjIRi7JjILcaz5exRb9dHub0Miw= 2023-10-18T20:30:42.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pjIRi7JjILcaz5exRb9dHub0Miw=', 'Content-Length': '0'} 2023-10-18T20:30:42.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009395134c71fa1c89-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-18T20:30:42.978 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.978 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.979 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.979 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.980 DEBUG:boto:Method: GET 2023-10-18T20:30:42.980 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.981 DEBUG:boto:Data: 2023-10-18T20:30:42.981 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:42.982 DEBUG:boto:Port: 8001 2023-10-18T20:30:42.982 DEBUG:boto:Params: {} 2023-10-18T20:30:42.983 DEBUG:boto:Token: None 2023-10-18T20:30:42.983 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.984 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B4HFy1uz1jeHQs8+P4h/zmLlSyw= 2023-10-18T20:30:42.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B4HFy1uz1jeHQs8+P4h/zmLlSyw=', 'Content-Length': '0'} 2023-10-18T20:30:42.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83535aa19dadefd-0065304072-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:42.987 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.988 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:42.988 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.988 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.989 DEBUG:boto:Method: GET 2023-10-18T20:30:42.989 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.990 DEBUG:boto:Data: 2023-10-18T20:30:42.990 DEBUG:boto:Headers: {} 2023-10-18T20:30:42.991 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:42.991 DEBUG:boto:Port: 8000 2023-10-18T20:30:42.991 DEBUG:boto:Params: {} 2023-10-18T20:30:42.992 DEBUG:boto:Token: None 2023-10-18T20:30:42.992 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:42 GMT /bofrnf-58/obj00000000000000000002?versionId=null 2023-10-18T20:30:42.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B4HFy1uz1jeHQs8+P4h/zmLlSyw= 2023-10-18T20:30:42.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B4HFy1uz1jeHQs8+P4h/zmLlSyw=', 'Content-Length': '0'} 2023-10-18T20:30:42.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dd29f7f6681c030-0065304072-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.039 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.039 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.039 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.040 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.040 DEBUG:boto:Method: GET 2023-10-18T20:30:43.041 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.041 DEBUG:boto:Data: 2023-10-18T20:30:43.041 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.042 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.043 DEBUG:boto:Params: {} 2023-10-18T20:30:43.043 DEBUG:boto:Token: None 2023-10-18T20:30:43.044 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0b9d5b02e139756-0065304073-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.049 DEBUG:boto:b'bofrnf-58obj000000000000000000021000falseobj000000000000000000022023-10-18T20:30:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.050 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.051 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.051 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.052 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.053 DEBUG:boto:Method: GET 2023-10-18T20:30:43.054 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.054 DEBUG:boto:Data: 2023-10-18T20:30:43.058 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.059 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.059 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.060 DEBUG:boto:Params: {} 2023-10-18T20:30:43.060 DEBUG:boto:Token: None 2023-10-18T20:30:43.061 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.061 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d5520f50a7faec-0065304073-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.067 DEBUG:boto:b'bofrnf-58obj000000000000000000021000falseobj000000000000000000022023-10-18T20:30:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.068 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.068 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.069 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.069 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.070 DEBUG:boto:Data: 2023-10-18T20:30:43.070 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.071 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.071 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.071 DEBUG:boto:Params: {} 2023-10-18T20:30:43.072 DEBUG:boto:Token: None 2023-10-18T20:30:43.072 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.073 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HZytcHxK+InDGdbIYCTS+NHNyy0= 2023-10-18T20:30:43.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HZytcHxK+InDGdbIYCTS+NHNyy0=', 'Content-Length': '0'} 2023-10-18T20:30:43.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cdd0d90f5d77e2b-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.076 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.076 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.077 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.077 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.078 DEBUG:boto:Data: 2023-10-18T20:30:43.078 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.079 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.079 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.079 DEBUG:boto:Params: {} 2023-10-18T20:30:43.080 DEBUG:boto:Token: None 2023-10-18T20:30:43.080 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.081 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HZytcHxK+InDGdbIYCTS+NHNyy0= 2023-10-18T20:30:43.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HZytcHxK+InDGdbIYCTS+NHNyy0=', 'Content-Length': '0'} 2023-10-18T20:30:43.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e749dadda7d44c-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-18T20:30:43.084 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.085 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.085 DEBUG:boto:Method: GET 2023-10-18T20:30:43.086 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.086 DEBUG:boto:Data: 2023-10-18T20:30:43.086 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.087 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.087 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.088 DEBUG:boto:Params: {} 2023-10-18T20:30:43.088 DEBUG:boto:Token: None 2023-10-18T20:30:43.089 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qWfXvt2Ydcbc8mORooXdGGjnrpE= 2023-10-18T20:30:43.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qWfXvt2Ydcbc8mORooXdGGjnrpE=', 'Content-Length': '0'} 2023-10-18T20:30:43.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036406511a255e680-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.138 DEBUG:boto:path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.139 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.139 DEBUG:boto:Method: GET 2023-10-18T20:30:43.140 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.140 DEBUG:boto:Data: 2023-10-18T20:30:43.141 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.141 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.142 DEBUG:boto:Params: {} 2023-10-18T20:30:43.142 DEBUG:boto:Token: None 2023-10-18T20:30:43.143 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000002 2023-10-18T20:30:43.143 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qWfXvt2Ydcbc8mORooXdGGjnrpE= 2023-10-18T20:30:43.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qWfXvt2Ydcbc8mORooXdGGjnrpE=', 'Content-Length': '0'} 2023-10-18T20:30:43.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a938ade9eb63be68-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.147 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.147 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.148 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.148 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.149 DEBUG:boto:Method: GET 2023-10-18T20:30:43.149 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.149 DEBUG:boto:Data: 2023-10-18T20:30:43.150 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.151 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.151 DEBUG:boto:Params: {} 2023-10-18T20:30:43.152 DEBUG:boto:Token: None 2023-10-18T20:30:43.152 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081284391f0394c94-0065304073-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.156 DEBUG:boto:b'bofrnf-58obj000000000000000000021000falseobj000000000000000000022023-10-18T20:30:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.157 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.157 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.158 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.158 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.159 DEBUG:boto:Method: GET 2023-10-18T20:30:43.159 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000002 2023-10-18T20:30:43.159 DEBUG:boto:Data: 2023-10-18T20:30:43.160 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.160 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.161 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.161 DEBUG:boto:Params: {} 2023-10-18T20:30:43.162 DEBUG:boto:Token: None 2023-10-18T20:30:43.162 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a030034a5f9282e3-0065304073-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.214 DEBUG:boto:b'bofrnf-58obj000000000000000000021000falseobj000000000000000000022023-10-18T20:30:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-18T20:30:43.216 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.216 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.217 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.217 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.217 DEBUG:boto:Method: GET 2023-10-18T20:30:43.218 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.218 DEBUG:boto:Data: 2023-10-18T20:30:43.219 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.220 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.220 DEBUG:boto:Params: {} 2023-10-18T20:30:43.221 DEBUG:boto:Token: None 2023-10-18T20:30:43.222 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.223 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r85cnvuBNRCwo0hi5EOswJiMlXU= 2023-10-18T20:30:43.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r85cnvuBNRCwo0hi5EOswJiMlXU=', 'Content-Length': '0'} 2023-10-18T20:30:43.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000315a7b761155c807-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.226 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.227 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.227 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.228 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.228 DEBUG:boto:Method: GET 2023-10-18T20:30:43.229 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.229 DEBUG:boto:Data: 2023-10-18T20:30:43.229 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.230 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.231 DEBUG:boto:Params: {} 2023-10-18T20:30:43.232 DEBUG:boto:Token: None 2023-10-18T20:30:43.233 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.234 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r85cnvuBNRCwo0hi5EOswJiMlXU= 2023-10-18T20:30:43.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r85cnvuBNRCwo0hi5EOswJiMlXU=', 'Content-Length': '0'} 2023-10-18T20:30:43.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017275ebe56dd37f7-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.237 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.238 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.238 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.239 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.239 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.239 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.240 DEBUG:boto:Data: 2023-10-18T20:30:43.240 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.241 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.241 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.241 DEBUG:boto:Params: {} 2023-10-18T20:30:43.242 DEBUG:boto:Token: None 2023-10-18T20:30:43.243 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.243 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DBazByS3xoShmkruiFx4Pryb8oc= 2023-10-18T20:30:43.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DBazByS3xoShmkruiFx4Pryb8oc=', 'Content-Length': '0'} 2023-10-18T20:30:43.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8f06642d9d45132-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.246 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.247 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.247 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.247 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.248 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.248 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.248 DEBUG:boto:Data: 2023-10-18T20:30:43.249 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.249 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.250 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.250 DEBUG:boto:Params: {} 2023-10-18T20:30:43.250 DEBUG:boto:Token: None 2023-10-18T20:30:43.251 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.251 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DBazByS3xoShmkruiFx4Pryb8oc= 2023-10-18T20:30:43.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DBazByS3xoShmkruiFx4Pryb8oc=', 'Content-Length': '0'} 2023-10-18T20:30:43.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103f79db2db49c9e-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-18T20:30:43.255 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.256 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.256 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.257 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.257 DEBUG:boto:Method: GET 2023-10-18T20:30:43.258 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.258 DEBUG:boto:Data: 2023-10-18T20:30:43.259 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.259 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.260 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.260 DEBUG:boto:Params: {} 2023-10-18T20:30:43.261 DEBUG:boto:Token: None 2023-10-18T20:30:43.262 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.262 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r85cnvuBNRCwo0hi5EOswJiMlXU= 2023-10-18T20:30:43.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r85cnvuBNRCwo0hi5EOswJiMlXU=', 'Content-Length': '0'} 2023-10-18T20:30:43.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f04767cffee01a9-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.266 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.267 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.267 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.268 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.268 DEBUG:boto:Method: GET 2023-10-18T20:30:43.269 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.269 DEBUG:boto:Data: 2023-10-18T20:30:43.269 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.270 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.270 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.271 DEBUG:boto:Params: {} 2023-10-18T20:30:43.271 DEBUG:boto:Token: None 2023-10-18T20:30:43.272 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003?versionId=null 2023-10-18T20:30:43.273 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r85cnvuBNRCwo0hi5EOswJiMlXU= 2023-10-18T20:30:43.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r85cnvuBNRCwo0hi5EOswJiMlXU=', 'Content-Length': '0'} 2023-10-18T20:30:43.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e767fb5a99b1b728-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.276 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.277 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.277 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.278 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.278 DEBUG:boto:Method: GET 2023-10-18T20:30:43.279 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.279 DEBUG:boto:Data: 2023-10-18T20:30:43.280 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.280 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.281 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.281 DEBUG:boto:Params: {} 2023-10-18T20:30:43.281 DEBUG:boto:Token: None 2023-10-18T20:30:43.282 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.283 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8965579a94c750-0065304073-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.286 DEBUG:boto:b'bofrnf-58obj000000000000000000031000falseobj000000000000000000032023-10-18T20:30:32.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.287 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.288 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.288 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.288 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.289 DEBUG:boto:Method: GET 2023-10-18T20:30:43.289 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.290 DEBUG:boto:Data: 2023-10-18T20:30:43.290 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.291 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.291 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.291 DEBUG:boto:Params: {} 2023-10-18T20:30:43.292 DEBUG:boto:Token: None 2023-10-18T20:30:43.292 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.293 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064fece4015340e56-0065304073-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.298 DEBUG:boto:b'bofrnf-58obj000000000000000000031000falseobj000000000000000000032023-10-18T20:30:32.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.299 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.299 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.300 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.300 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.301 DEBUG:boto:Data: 2023-10-18T20:30:43.301 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.301 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.302 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.302 DEBUG:boto:Params: {} 2023-10-18T20:30:43.303 DEBUG:boto:Token: None 2023-10-18T20:30:43.303 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mhDd5tsLSKfhBuKtRYPPqQ4v5qA= 2023-10-18T20:30:43.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mhDd5tsLSKfhBuKtRYPPqQ4v5qA=', 'Content-Length': '0'} 2023-10-18T20:30:43.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084d04820f9201050-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.307 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.307 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.308 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.308 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.308 DEBUG:boto:Data: 2023-10-18T20:30:43.309 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.309 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.310 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.310 DEBUG:boto:Params: {} 2023-10-18T20:30:43.311 DEBUG:boto:Token: None 2023-10-18T20:30:43.311 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mhDd5tsLSKfhBuKtRYPPqQ4v5qA= 2023-10-18T20:30:43.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mhDd5tsLSKfhBuKtRYPPqQ4v5qA=', 'Content-Length': '0'} 2023-10-18T20:30:43.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052040199c1fb9f3d-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-18T20:30:43.315 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.315 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.316 DEBUG:boto:Method: GET 2023-10-18T20:30:43.316 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.317 DEBUG:boto:Data: 2023-10-18T20:30:43.317 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.318 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.319 DEBUG:boto:Params: {} 2023-10-18T20:30:43.319 DEBUG:boto:Token: None 2023-10-18T20:30:43.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JLD8UADWh6Lb0T2dD9Kz5kXrNuA= 2023-10-18T20:30:43.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JLD8UADWh6Lb0T2dD9Kz5kXrNuA=', 'Content-Length': '0'} 2023-10-18T20:30:43.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf39b1efccbe10b6-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.324 DEBUG:boto:path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.324 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.324 DEBUG:boto:Method: GET 2023-10-18T20:30:43.325 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.325 DEBUG:boto:Data: 2023-10-18T20:30:43.326 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.326 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.327 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.327 DEBUG:boto:Params: {} 2023-10-18T20:30:43.328 DEBUG:boto:Token: None 2023-10-18T20:30:43.328 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000003 2023-10-18T20:30:43.329 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JLD8UADWh6Lb0T2dD9Kz5kXrNuA= 2023-10-18T20:30:43.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JLD8UADWh6Lb0T2dD9Kz5kXrNuA=', 'Content-Length': '0'} 2023-10-18T20:30:43.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebce7040cd37c93-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.332 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.332 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.333 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.333 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.333 DEBUG:boto:Method: GET 2023-10-18T20:30:43.334 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.334 DEBUG:boto:Data: 2023-10-18T20:30:43.335 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.335 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.336 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.336 DEBUG:boto:Params: {} 2023-10-18T20:30:43.336 DEBUG:boto:Token: None 2023-10-18T20:30:43.337 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbbd5d182b634058-0065304073-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.341 DEBUG:boto:b'bofrnf-58obj000000000000000000031000falseobj000000000000000000032023-10-18T20:30:32.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.342 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.343 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.343 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.344 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.344 DEBUG:boto:Method: GET 2023-10-18T20:30:43.344 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000003 2023-10-18T20:30:43.345 DEBUG:boto:Data: 2023-10-18T20:30:43.345 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.346 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.347 DEBUG:boto:Params: {} 2023-10-18T20:30:43.347 DEBUG:boto:Token: None 2023-10-18T20:30:43.348 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dba0f5a4da52665-0065304073-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.398 DEBUG:boto:b'bofrnf-58obj000000000000000000031000falseobj000000000000000000032023-10-18T20:30:32.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-18T20:30:43.400 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.400 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.401 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.401 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.402 DEBUG:boto:Method: GET 2023-10-18T20:30:43.402 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.403 DEBUG:boto:Data: 2023-10-18T20:30:43.403 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.404 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.404 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.404 DEBUG:boto:Params: {} 2023-10-18T20:30:43.405 DEBUG:boto:Token: None 2023-10-18T20:30:43.405 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.406 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:owjQIBnv1655K+wnz3EgkbpSVAk= 2023-10-18T20:30:43.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:owjQIBnv1655K+wnz3EgkbpSVAk=', 'Content-Length': '0'} 2023-10-18T20:30:43.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f712688cd6d62f53-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.411 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.411 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.412 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.412 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.412 DEBUG:boto:Method: GET 2023-10-18T20:30:43.413 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.413 DEBUG:boto:Data: 2023-10-18T20:30:43.414 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.414 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.415 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.415 DEBUG:boto:Params: {} 2023-10-18T20:30:43.416 DEBUG:boto:Token: None 2023-10-18T20:30:43.416 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.417 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:owjQIBnv1655K+wnz3EgkbpSVAk= 2023-10-18T20:30:43.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:owjQIBnv1655K+wnz3EgkbpSVAk=', 'Content-Length': '0'} 2023-10-18T20:30:43.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b580d42363ce4420-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.420 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.421 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.421 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.422 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.422 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.423 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.423 DEBUG:boto:Data: 2023-10-18T20:30:43.424 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.424 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.425 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.425 DEBUG:boto:Params: {} 2023-10-18T20:30:43.426 DEBUG:boto:Token: None 2023-10-18T20:30:43.426 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.427 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kioSkgrAOQcNhP5PTw8xH1tE/RU= 2023-10-18T20:30:43.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kioSkgrAOQcNhP5PTw8xH1tE/RU=', 'Content-Length': '0'} 2023-10-18T20:30:43.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e9d15033b2a4528-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.430 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.431 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.431 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.432 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.432 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.433 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.433 DEBUG:boto:Data: 2023-10-18T20:30:43.433 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.434 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.434 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.435 DEBUG:boto:Params: {} 2023-10-18T20:30:43.435 DEBUG:boto:Token: None 2023-10-18T20:30:43.436 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.436 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kioSkgrAOQcNhP5PTw8xH1tE/RU= 2023-10-18T20:30:43.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kioSkgrAOQcNhP5PTw8xH1tE/RU=', 'Content-Length': '0'} 2023-10-18T20:30:43.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cf5487d02ff9c8b-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-18T20:30:43.440 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.441 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.441 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.442 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.442 DEBUG:boto:Method: GET 2023-10-18T20:30:43.443 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.443 DEBUG:boto:Data: 2023-10-18T20:30:43.444 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.444 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.445 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.445 DEBUG:boto:Params: {} 2023-10-18T20:30:43.446 DEBUG:boto:Token: None 2023-10-18T20:30:43.446 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.447 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:owjQIBnv1655K+wnz3EgkbpSVAk= 2023-10-18T20:30:43.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:owjQIBnv1655K+wnz3EgkbpSVAk=', 'Content-Length': '0'} 2023-10-18T20:30:43.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000027a2aa3bb2241f3-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.450 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.451 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.451 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.452 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.452 DEBUG:boto:Method: GET 2023-10-18T20:30:43.453 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.453 DEBUG:boto:Data: 2023-10-18T20:30:43.453 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.454 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.455 DEBUG:boto:Params: {} 2023-10-18T20:30:43.455 DEBUG:boto:Token: None 2023-10-18T20:30:43.456 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004?versionId=null 2023-10-18T20:30:43.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:owjQIBnv1655K+wnz3EgkbpSVAk= 2023-10-18T20:30:43.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:owjQIBnv1655K+wnz3EgkbpSVAk=', 'Content-Length': '0'} 2023-10-18T20:30:43.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1531ab728b51728-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.502 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.503 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.504 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.504 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.505 DEBUG:boto:Method: GET 2023-10-18T20:30:43.505 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.505 DEBUG:boto:Data: 2023-10-18T20:30:43.506 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.506 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.507 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.507 DEBUG:boto:Params: {} 2023-10-18T20:30:43.508 DEBUG:boto:Token: None 2023-10-18T20:30:43.509 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d016ca70d031bd99-0065304073-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.514 DEBUG:boto:b'bofrnf-58obj000000000000000000041000falseobj000000000000000000042023-10-18T20:30:32.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.514 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.515 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.515 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.516 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.516 DEBUG:boto:Method: GET 2023-10-18T20:30:43.517 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.517 DEBUG:boto:Data: 2023-10-18T20:30:43.517 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.518 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.518 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.519 DEBUG:boto:Params: {} 2023-10-18T20:30:43.519 DEBUG:boto:Token: None 2023-10-18T20:30:43.520 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099bb56c04d8dfe60-0065304073-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.526 DEBUG:boto:b'bofrnf-58obj000000000000000000041000falseobj000000000000000000042023-10-18T20:30:32.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.527 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.527 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.527 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.528 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.528 DEBUG:boto:Data: 2023-10-18T20:30:43.529 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.529 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.530 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.530 DEBUG:boto:Params: {} 2023-10-18T20:30:43.531 DEBUG:boto:Token: None 2023-10-18T20:30:43.531 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.531 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r5VJkJzXxtlsUZ13YREap+lOWMY= 2023-10-18T20:30:43.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r5VJkJzXxtlsUZ13YREap+lOWMY=', 'Content-Length': '0'} 2023-10-18T20:30:43.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b35105124b06b3f8-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.535 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.535 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.536 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.536 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.536 DEBUG:boto:Data: 2023-10-18T20:30:43.537 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.538 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.538 DEBUG:boto:Params: {} 2023-10-18T20:30:43.539 DEBUG:boto:Token: None 2023-10-18T20:30:43.539 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.540 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r5VJkJzXxtlsUZ13YREap+lOWMY= 2023-10-18T20:30:43.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r5VJkJzXxtlsUZ13YREap+lOWMY=', 'Content-Length': '0'} 2023-10-18T20:30:43.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4833c4eb5f1f54-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-18T20:30:43.543 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.544 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.544 DEBUG:boto:Method: GET 2023-10-18T20:30:43.545 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.545 DEBUG:boto:Data: 2023-10-18T20:30:43.546 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.546 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.547 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.547 DEBUG:boto:Params: {} 2023-10-18T20:30:43.548 DEBUG:boto:Token: None 2023-10-18T20:30:43.548 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.549 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xZmUCUaEX2uzIn22jbCvJKAlNSE= 2023-10-18T20:30:43.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xZmUCUaEX2uzIn22jbCvJKAlNSE=', 'Content-Length': '0'} 2023-10-18T20:30:43.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000222712d7ddf05f1e-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.552 DEBUG:boto:path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.552 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.553 DEBUG:boto:Method: GET 2023-10-18T20:30:43.553 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.554 DEBUG:boto:Data: 2023-10-18T20:30:43.554 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.555 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.555 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.556 DEBUG:boto:Params: {} 2023-10-18T20:30:43.556 DEBUG:boto:Token: None 2023-10-18T20:30:43.556 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000004 2023-10-18T20:30:43.557 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xZmUCUaEX2uzIn22jbCvJKAlNSE= 2023-10-18T20:30:43.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xZmUCUaEX2uzIn22jbCvJKAlNSE=', 'Content-Length': '0'} 2023-10-18T20:30:43.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b7dadf5225b0271-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.603 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.603 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.604 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.604 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.605 DEBUG:boto:Method: GET 2023-10-18T20:30:43.605 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.606 DEBUG:boto:Data: 2023-10-18T20:30:43.606 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.607 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.607 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.607 DEBUG:boto:Params: {} 2023-10-18T20:30:43.608 DEBUG:boto:Token: None 2023-10-18T20:30:43.608 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.609 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046d990fcd0166dc1-0065304073-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.613 DEBUG:boto:b'bofrnf-58obj000000000000000000041000falseobj000000000000000000042023-10-18T20:30:32.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.614 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.614 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.615 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.615 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.615 DEBUG:boto:Method: GET 2023-10-18T20:30:43.616 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000004 2023-10-18T20:30:43.616 DEBUG:boto:Data: 2023-10-18T20:30:43.617 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.617 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.618 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.618 DEBUG:boto:Params: {} 2023-10-18T20:30:43.619 DEBUG:boto:Token: None 2023-10-18T20:30:43.619 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.620 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000850f7e36e412b347-0065304073-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.625 DEBUG:boto:b'bofrnf-58obj000000000000000000041000falseobj000000000000000000042023-10-18T20:30:32.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-18T20:30:43.626 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.627 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.627 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.628 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.628 DEBUG:boto:Method: GET 2023-10-18T20:30:43.629 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.629 DEBUG:boto:Data: 2023-10-18T20:30:43.629 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.630 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.631 DEBUG:boto:Params: {} 2023-10-18T20:30:43.631 DEBUG:boto:Token: None 2023-10-18T20:30:43.632 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.632 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G53SIl+exctZfTIndtJtusZudS4= 2023-10-18T20:30:43.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G53SIl+exctZfTIndtJtusZudS4=', 'Content-Length': '0'} 2023-10-18T20:30:43.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eee0e7a5a3906d80-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.637 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.637 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.638 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.638 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.638 DEBUG:boto:Method: GET 2023-10-18T20:30:43.639 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.639 DEBUG:boto:Data: 2023-10-18T20:30:43.640 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.641 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.641 DEBUG:boto:Params: {} 2023-10-18T20:30:43.642 DEBUG:boto:Token: None 2023-10-18T20:30:43.642 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G53SIl+exctZfTIndtJtusZudS4= 2023-10-18T20:30:43.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G53SIl+exctZfTIndtJtusZudS4=', 'Content-Length': '0'} 2023-10-18T20:30:43.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4685881cb7785f1-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.646 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.647 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.647 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.647 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.648 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.648 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.649 DEBUG:boto:Data: 2023-10-18T20:30:43.649 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.650 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.650 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.650 DEBUG:boto:Params: {} 2023-10-18T20:30:43.651 DEBUG:boto:Token: None 2023-10-18T20:30:43.651 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a/6lCZoX2WabV+EmCsV9Q1/cgec= 2023-10-18T20:30:43.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a/6lCZoX2WabV+EmCsV9Q1/cgec=', 'Content-Length': '0'} 2023-10-18T20:30:43.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000015dce5b65b67a5a-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.655 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.655 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.656 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.656 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.657 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.657 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.657 DEBUG:boto:Data: 2023-10-18T20:30:43.658 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.658 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.659 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.659 DEBUG:boto:Params: {} 2023-10-18T20:30:43.660 DEBUG:boto:Token: None 2023-10-18T20:30:43.660 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.660 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a/6lCZoX2WabV+EmCsV9Q1/cgec= 2023-10-18T20:30:43.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a/6lCZoX2WabV+EmCsV9Q1/cgec=', 'Content-Length': '0'} 2023-10-18T20:30:43.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7e320fee1a73a4-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-18T20:30:43.664 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.665 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.665 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.665 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.666 DEBUG:boto:Method: GET 2023-10-18T20:30:43.666 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.667 DEBUG:boto:Data: 2023-10-18T20:30:43.667 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.667 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.668 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.668 DEBUG:boto:Params: {} 2023-10-18T20:30:43.669 DEBUG:boto:Token: None 2023-10-18T20:30:43.669 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.670 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G53SIl+exctZfTIndtJtusZudS4= 2023-10-18T20:30:43.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G53SIl+exctZfTIndtJtusZudS4=', 'Content-Length': '0'} 2023-10-18T20:30:43.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f014a5d176d5eb74-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.718 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.719 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.719 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.720 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.720 DEBUG:boto:Method: GET 2023-10-18T20:30:43.721 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.721 DEBUG:boto:Data: 2023-10-18T20:30:43.721 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.722 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.722 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.723 DEBUG:boto:Params: {} 2023-10-18T20:30:43.723 DEBUG:boto:Token: None 2023-10-18T20:30:43.724 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005?versionId=null 2023-10-18T20:30:43.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G53SIl+exctZfTIndtJtusZudS4= 2023-10-18T20:30:43.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G53SIl+exctZfTIndtJtusZudS4=', 'Content-Length': '0'} 2023-10-18T20:30:43.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000660da0a6792aeba-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.728 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.728 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.729 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.729 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.730 DEBUG:boto:Method: GET 2023-10-18T20:30:43.730 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.731 DEBUG:boto:Data: 2023-10-18T20:30:43.731 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.732 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.732 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.732 DEBUG:boto:Params: {} 2023-10-18T20:30:43.733 DEBUG:boto:Token: None 2023-10-18T20:30:43.733 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.734 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b47a936364dd4b93-0065304073-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.738 DEBUG:boto:b'bofrnf-58obj000000000000000000051000falseobj000000000000000000052023-10-18T20:30:31.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.738 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.739 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.739 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.740 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.740 DEBUG:boto:Method: GET 2023-10-18T20:30:43.741 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.741 DEBUG:boto:Data: 2023-10-18T20:30:43.742 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.742 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.743 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.743 DEBUG:boto:Params: {} 2023-10-18T20:30:43.744 DEBUG:boto:Token: None 2023-10-18T20:30:43.744 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.745 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fd039769f999837-0065304073-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.790 DEBUG:boto:b'bofrnf-58obj000000000000000000051000falseobj000000000000000000052023-10-18T20:30:31.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.791 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.792 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.792 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.793 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.793 DEBUG:boto:Data: 2023-10-18T20:30:43.794 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.794 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.794 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.795 DEBUG:boto:Params: {} 2023-10-18T20:30:43.795 DEBUG:boto:Token: None 2023-10-18T20:30:43.796 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.796 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9T2NR5YQKb7LByhA4EcEPTRezV4= 2023-10-18T20:30:43.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9T2NR5YQKb7LByhA4EcEPTRezV4=', 'Content-Length': '0'} 2023-10-18T20:30:43.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006daf8d788a254715-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.800 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.800 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.801 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.801 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.801 DEBUG:boto:Data: 2023-10-18T20:30:43.802 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.802 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.803 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.803 DEBUG:boto:Params: {} 2023-10-18T20:30:43.804 DEBUG:boto:Token: None 2023-10-18T20:30:43.804 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.805 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9T2NR5YQKb7LByhA4EcEPTRezV4= 2023-10-18T20:30:43.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9T2NR5YQKb7LByhA4EcEPTRezV4=', 'Content-Length': '0'} 2023-10-18T20:30:43.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030e0a29d94264f75-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-18T20:30:43.809 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.809 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.809 DEBUG:boto:Method: GET 2023-10-18T20:30:43.810 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.810 DEBUG:boto:Data: 2023-10-18T20:30:43.811 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.811 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.812 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.812 DEBUG:boto:Params: {} 2023-10-18T20:30:43.812 DEBUG:boto:Token: None 2023-10-18T20:30:43.813 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.813 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xOWJfpSaqHxdvV3rejpRo8AIxpg= 2023-10-18T20:30:43.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xOWJfpSaqHxdvV3rejpRo8AIxpg=', 'Content-Length': '0'} 2023-10-18T20:30:43.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b65ad52031540522-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.817 DEBUG:boto:path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.817 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.817 DEBUG:boto:Method: GET 2023-10-18T20:30:43.818 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.818 DEBUG:boto:Data: 2023-10-18T20:30:43.819 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.819 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.820 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.820 DEBUG:boto:Params: {} 2023-10-18T20:30:43.821 DEBUG:boto:Token: None 2023-10-18T20:30:43.821 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000005 2023-10-18T20:30:43.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xOWJfpSaqHxdvV3rejpRo8AIxpg= 2023-10-18T20:30:43.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xOWJfpSaqHxdvV3rejpRo8AIxpg=', 'Content-Length': '0'} 2023-10-18T20:30:43.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f57a61a4955e48c-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.825 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.825 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.825 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.826 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.826 DEBUG:boto:Method: GET 2023-10-18T20:30:43.827 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.827 DEBUG:boto:Data: 2023-10-18T20:30:43.828 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.828 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.828 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.829 DEBUG:boto:Params: {} 2023-10-18T20:30:43.829 DEBUG:boto:Token: None 2023-10-18T20:30:43.830 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.830 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000755fc869713faba2-0065304073-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.834 DEBUG:boto:b'bofrnf-58obj000000000000000000051000falseobj000000000000000000052023-10-18T20:30:31.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.835 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.866 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.867 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.867 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.867 DEBUG:boto:Method: GET 2023-10-18T20:30:43.868 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000005 2023-10-18T20:30:43.868 DEBUG:boto:Data: 2023-10-18T20:30:43.869 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.870 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.870 DEBUG:boto:Params: {} 2023-10-18T20:30:43.871 DEBUG:boto:Token: None 2023-10-18T20:30:43.871 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007616b3c8c0919479-0065304073-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.879 DEBUG:boto:b'bofrnf-58obj000000000000000000051000falseobj000000000000000000052023-10-18T20:30:31.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-18T20:30:43.880 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.880 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.881 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.881 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.882 DEBUG:boto:Method: GET 2023-10-18T20:30:43.882 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.883 DEBUG:boto:Data: 2023-10-18T20:30:43.883 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.883 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.884 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.884 DEBUG:boto:Params: {} 2023-10-18T20:30:43.885 DEBUG:boto:Token: None 2023-10-18T20:30:43.885 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.886 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r5humVfAB/zjnjgTMnOpGd6erEE= 2023-10-18T20:30:43.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r5humVfAB/zjnjgTMnOpGd6erEE=', 'Content-Length': '0'} 2023-10-18T20:30:43.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8cec7ac5bf61a2e-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.889 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.890 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.890 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.891 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.891 DEBUG:boto:Method: GET 2023-10-18T20:30:43.892 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.892 DEBUG:boto:Data: 2023-10-18T20:30:43.892 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.893 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.893 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.894 DEBUG:boto:Params: {} 2023-10-18T20:30:43.894 DEBUG:boto:Token: None 2023-10-18T20:30:43.895 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.895 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r5humVfAB/zjnjgTMnOpGd6erEE= 2023-10-18T20:30:43.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r5humVfAB/zjnjgTMnOpGd6erEE=', 'Content-Length': '0'} 2023-10-18T20:30:43.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d624a773a84c2fad-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.898 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.899 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.899 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.900 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.900 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.900 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.901 DEBUG:boto:Data: 2023-10-18T20:30:43.901 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.901 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.902 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.902 DEBUG:boto:Params: {} 2023-10-18T20:30:43.903 DEBUG:boto:Token: None 2023-10-18T20:30:43.903 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.904 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Oy+Y2oKaDjvsZaSRzeyU3xB5kdI= 2023-10-18T20:30:43.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Oy+Y2oKaDjvsZaSRzeyU3xB5kdI=', 'Content-Length': '0'} 2023-10-18T20:30:43.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0b6335210f13246-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.907 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.907 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.907 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.908 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.908 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.909 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.909 DEBUG:boto:Data: 2023-10-18T20:30:43.909 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.910 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.910 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.911 DEBUG:boto:Params: {} 2023-10-18T20:30:43.911 DEBUG:boto:Token: None 2023-10-18T20:30:43.912 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.912 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Oy+Y2oKaDjvsZaSRzeyU3xB5kdI= 2023-10-18T20:30:43.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Oy+Y2oKaDjvsZaSRzeyU3xB5kdI=', 'Content-Length': '0'} 2023-10-18T20:30:43.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee57d35fd691005d-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-18T20:30:43.916 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.916 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.917 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.917 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.918 DEBUG:boto:Method: GET 2023-10-18T20:30:43.918 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.919 DEBUG:boto:Data: 2023-10-18T20:30:43.919 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.920 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.920 DEBUG:boto:Params: {} 2023-10-18T20:30:43.921 DEBUG:boto:Token: None 2023-10-18T20:30:43.921 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r5humVfAB/zjnjgTMnOpGd6erEE= 2023-10-18T20:30:43.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r5humVfAB/zjnjgTMnOpGd6erEE=', 'Content-Length': '0'} 2023-10-18T20:30:43.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000781950d52bb061a6-0065304073-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.925 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.926 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.926 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.927 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.927 DEBUG:boto:Method: GET 2023-10-18T20:30:43.928 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.928 DEBUG:boto:Data: 2023-10-18T20:30:43.928 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.929 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.930 DEBUG:boto:Params: {} 2023-10-18T20:30:43.930 DEBUG:boto:Token: None 2023-10-18T20:30:43.931 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/obj00000000000000000006?versionId=null 2023-10-18T20:30:43.931 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r5humVfAB/zjnjgTMnOpGd6erEE= 2023-10-18T20:30:43.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r5humVfAB/zjnjgTMnOpGd6erEE=', 'Content-Length': '0'} 2023-10-18T20:30:43.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cfbcba7ba0035f1-0065304073-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.974 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.975 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.976 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:43.976 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:43.976 DEBUG:boto:Method: GET 2023-10-18T20:30:43.977 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:43.977 DEBUG:boto:Data: 2023-10-18T20:30:43.978 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.978 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:43.979 DEBUG:boto:Port: 8001 2023-10-18T20:30:43.979 DEBUG:boto:Params: {} 2023-10-18T20:30:43.979 DEBUG:boto:Token: None 2023-10-18T20:30:43.980 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.980 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006701a47c9da6f363-0065304073-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.984 DEBUG:boto:b'bofrnf-58obj000000000000000000061000falseobj000000000000000000062023-10-18T20:30:31.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.985 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:43.985 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:43.986 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:43.986 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:43.987 DEBUG:boto:Method: GET 2023-10-18T20:30:43.987 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:43.988 DEBUG:boto:Data: 2023-10-18T20:30:43.988 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.989 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:43.989 DEBUG:boto:Port: 8000 2023-10-18T20:30:43.989 DEBUG:boto:Params: {} 2023-10-18T20:30:43.990 DEBUG:boto:Token: None 2023-10-18T20:30:43.991 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:43 GMT /bofrnf-58/ 2023-10-18T20:30:43.991 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE= 2023-10-18T20:30:43.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6l1K2Qm4NE2feCnWofjiBI6PWE=', 'Content-Length': '0'} 2023-10-18T20:30:43.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef07a54a01d4d5e-0065304073-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:43.996 DEBUG:boto:b'bofrnf-58obj000000000000000000061000falseobj000000000000000000062023-10-18T20:30:31.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:43.997 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.997 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.998 DEBUG:boto:Method: HEAD 2023-10-18T20:30:43.998 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:43.998 DEBUG:boto:Data: 2023-10-18T20:30:43.999 DEBUG:boto:Headers: {} 2023-10-18T20:30:43.999 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.000 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.000 DEBUG:boto:Params: {} 2023-10-18T20:30:44.000 DEBUG:boto:Token: None 2023-10-18T20:30:44.001 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.001 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nYTenOfGAwLo4E/VZpYmHSd+1Y0= 2023-10-18T20:30:44.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nYTenOfGAwLo4E/VZpYmHSd+1Y0=', 'Content-Length': '0'} 2023-10-18T20:30:44.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd8ccdb87815240-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.005 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.005 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.005 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.006 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.006 DEBUG:boto:Data: 2023-10-18T20:30:44.007 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.007 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.008 DEBUG:boto:Params: {} 2023-10-18T20:30:44.008 DEBUG:boto:Token: None 2023-10-18T20:30:44.009 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nYTenOfGAwLo4E/VZpYmHSd+1Y0= 2023-10-18T20:30:44.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nYTenOfGAwLo4E/VZpYmHSd+1Y0=', 'Content-Length': '0'} 2023-10-18T20:30:44.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d40c401391f81d-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-18T20:30:44.013 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.013 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.014 DEBUG:boto:Method: GET 2023-10-18T20:30:44.014 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.015 DEBUG:boto:Data: 2023-10-18T20:30:44.015 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.015 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.016 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.016 DEBUG:boto:Params: {} 2023-10-18T20:30:44.017 DEBUG:boto:Token: None 2023-10-18T20:30:44.017 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.018 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pqaw0J4zQbVVfLPisHelnEfkB+Q= 2023-10-18T20:30:44.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pqaw0J4zQbVVfLPisHelnEfkB+Q=', 'Content-Length': '0'} 2023-10-18T20:30:44.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48a932d79629036-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.062 DEBUG:boto:path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.063 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.064 DEBUG:boto:Method: GET 2023-10-18T20:30:44.064 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.064 DEBUG:boto:Data: 2023-10-18T20:30:44.065 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.065 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.066 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.066 DEBUG:boto:Params: {} 2023-10-18T20:30:44.067 DEBUG:boto:Token: None 2023-10-18T20:30:44.067 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000006 2023-10-18T20:30:44.068 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pqaw0J4zQbVVfLPisHelnEfkB+Q= 2023-10-18T20:30:44.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pqaw0J4zQbVVfLPisHelnEfkB+Q=', 'Content-Length': '0'} 2023-10-18T20:30:44.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feb7fb78c6e17363-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.071 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.071 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.072 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:44.072 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:44.073 DEBUG:boto:Method: GET 2023-10-18T20:30:44.073 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:44.074 DEBUG:boto:Data: 2023-10-18T20:30:44.074 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.075 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.075 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.075 DEBUG:boto:Params: {} 2023-10-18T20:30:44.076 DEBUG:boto:Token: None 2023-10-18T20:30:44.076 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.077 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2457bf51686f7de-0065304074-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.082 DEBUG:boto:b'bofrnf-58obj000000000000000000061000falseobj000000000000000000062023-10-18T20:30:31.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.083 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.084 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.084 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:44.084 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:44.085 DEBUG:boto:Method: GET 2023-10-18T20:30:44.085 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000006 2023-10-18T20:30:44.086 DEBUG:boto:Data: 2023-10-18T20:30:44.086 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.087 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.087 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.088 DEBUG:boto:Params: {} 2023-10-18T20:30:44.088 DEBUG:boto:Token: None 2023-10-18T20:30:44.089 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023f449dde66a5ab3-0065304074-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.138 DEBUG:boto:b'bofrnf-58obj000000000000000000061000falseobj000000000000000000062023-10-18T20:30:31.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-18T20:30:44.140 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.140 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.141 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.141 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.142 DEBUG:boto:Method: GET 2023-10-18T20:30:44.142 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.142 DEBUG:boto:Data: 2023-10-18T20:30:44.143 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.144 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.144 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.144 DEBUG:boto:Params: {} 2023-10-18T20:30:44.145 DEBUG:boto:Token: None 2023-10-18T20:30:44.145 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.146 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:92TLA2qES8NnZh9CQt1kdqvYJ+M= 2023-10-18T20:30:44.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:92TLA2qES8NnZh9CQt1kdqvYJ+M=', 'Content-Length': '0'} 2023-10-18T20:30:44.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba09dccd52d9c7eb-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.150 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.150 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.151 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.151 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.152 DEBUG:boto:Method: GET 2023-10-18T20:30:44.152 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.153 DEBUG:boto:Data: 2023-10-18T20:30:44.153 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.154 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.154 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.154 DEBUG:boto:Params: {} 2023-10-18T20:30:44.155 DEBUG:boto:Token: None 2023-10-18T20:30:44.155 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.156 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:92TLA2qES8NnZh9CQt1kdqvYJ+M= 2023-10-18T20:30:44.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:92TLA2qES8NnZh9CQt1kdqvYJ+M=', 'Content-Length': '0'} 2023-10-18T20:30:44.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ba5bf578f105b4-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.159 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.160 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.160 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.161 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.161 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.161 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.162 DEBUG:boto:Data: 2023-10-18T20:30:44.162 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.163 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.163 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.163 DEBUG:boto:Params: {} 2023-10-18T20:30:44.164 DEBUG:boto:Token: None 2023-10-18T20:30:44.164 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.165 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5OP/aSFW6ShPH+o2fFFhpW9NL0U= 2023-10-18T20:30:44.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5OP/aSFW6ShPH+o2fFFhpW9NL0U=', 'Content-Length': '0'} 2023-10-18T20:30:44.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dacbb3ad7bfaceb-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.168 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.168 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.169 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.169 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.169 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.170 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.170 DEBUG:boto:Data: 2023-10-18T20:30:44.171 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.171 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.171 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.172 DEBUG:boto:Params: {} 2023-10-18T20:30:44.172 DEBUG:boto:Token: None 2023-10-18T20:30:44.173 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.173 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5OP/aSFW6ShPH+o2fFFhpW9NL0U= 2023-10-18T20:30:44.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5OP/aSFW6ShPH+o2fFFhpW9NL0U=', 'Content-Length': '0'} 2023-10-18T20:30:44.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31a3a595173c156-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-18T20:30:44.177 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.178 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.178 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.178 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.179 DEBUG:boto:Method: GET 2023-10-18T20:30:44.179 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.180 DEBUG:boto:Data: 2023-10-18T20:30:44.180 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.181 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.181 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.181 DEBUG:boto:Params: {} 2023-10-18T20:30:44.182 DEBUG:boto:Token: None 2023-10-18T20:30:44.182 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:92TLA2qES8NnZh9CQt1kdqvYJ+M= 2023-10-18T20:30:44.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:92TLA2qES8NnZh9CQt1kdqvYJ+M=', 'Content-Length': '0'} 2023-10-18T20:30:44.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000570af137c29ac211-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.186 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.187 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.187 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.187 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.188 DEBUG:boto:Method: GET 2023-10-18T20:30:44.188 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.189 DEBUG:boto:Data: 2023-10-18T20:30:44.189 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.190 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.190 DEBUG:boto:Params: {} 2023-10-18T20:30:44.191 DEBUG:boto:Token: None 2023-10-18T20:30:44.191 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007?versionId=null 2023-10-18T20:30:44.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:92TLA2qES8NnZh9CQt1kdqvYJ+M= 2023-10-18T20:30:44.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:92TLA2qES8NnZh9CQt1kdqvYJ+M=', 'Content-Length': '0'} 2023-10-18T20:30:44.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061736c85bbe75b4a-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.195 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.196 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.196 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.196 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.197 DEBUG:boto:Method: GET 2023-10-18T20:30:44.197 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.198 DEBUG:boto:Data: 2023-10-18T20:30:44.198 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.199 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.199 DEBUG:boto:Params: {} 2023-10-18T20:30:44.200 DEBUG:boto:Token: None 2023-10-18T20:30:44.200 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794573c9185517bc-0065304074-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.204 DEBUG:boto:b'bofrnf-58obj000000000000000000071000falseobj000000000000000000072023-10-18T20:30:32.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.205 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.205 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.206 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.206 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.207 DEBUG:boto:Method: GET 2023-10-18T20:30:44.207 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.208 DEBUG:boto:Data: 2023-10-18T20:30:44.208 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.208 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.209 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.209 DEBUG:boto:Params: {} 2023-10-18T20:30:44.210 DEBUG:boto:Token: None 2023-10-18T20:30:44.210 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e51d270da56cef8c-0065304074-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.262 DEBUG:boto:b'bofrnf-58obj000000000000000000071000falseobj000000000000000000072023-10-18T20:30:32.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.263 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.264 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.264 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.265 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.265 DEBUG:boto:Data: 2023-10-18T20:30:44.265 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.266 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.266 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.267 DEBUG:boto:Params: {} 2023-10-18T20:30:44.267 DEBUG:boto:Token: None 2023-10-18T20:30:44.268 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.268 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DQw6nEzLS6z1okxVBLbfpO8gCko= 2023-10-18T20:30:44.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DQw6nEzLS6z1okxVBLbfpO8gCko=', 'Content-Length': '0'} 2023-10-18T20:30:44.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9975b68b992bb5-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.271 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.272 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.272 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.273 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.273 DEBUG:boto:Data: 2023-10-18T20:30:44.274 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.274 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.274 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.275 DEBUG:boto:Params: {} 2023-10-18T20:30:44.275 DEBUG:boto:Token: None 2023-10-18T20:30:44.276 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DQw6nEzLS6z1okxVBLbfpO8gCko= 2023-10-18T20:30:44.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DQw6nEzLS6z1okxVBLbfpO8gCko=', 'Content-Length': '0'} 2023-10-18T20:30:44.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a55e042ae9d3ae-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-18T20:30:44.280 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.280 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.281 DEBUG:boto:Method: GET 2023-10-18T20:30:44.281 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.281 DEBUG:boto:Data: 2023-10-18T20:30:44.282 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.283 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.286 DEBUG:boto:Params: {} 2023-10-18T20:30:44.286 DEBUG:boto:Token: None 2023-10-18T20:30:44.287 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.287 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FJyEnFFrXBxG8/DTASnmqvIAUpc= 2023-10-18T20:30:44.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FJyEnFFrXBxG8/DTASnmqvIAUpc=', 'Content-Length': '0'} 2023-10-18T20:30:44.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005236203676228e2d-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.330 DEBUG:boto:path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.331 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.332 DEBUG:boto:Method: GET 2023-10-18T20:30:44.332 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.332 DEBUG:boto:Data: 2023-10-18T20:30:44.333 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.334 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.334 DEBUG:boto:Params: {} 2023-10-18T20:30:44.335 DEBUG:boto:Token: None 2023-10-18T20:30:44.335 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000007 2023-10-18T20:30:44.336 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FJyEnFFrXBxG8/DTASnmqvIAUpc= 2023-10-18T20:30:44.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FJyEnFFrXBxG8/DTASnmqvIAUpc=', 'Content-Length': '0'} 2023-10-18T20:30:44.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c00ab3b2fd0f9e91-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.339 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.340 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.340 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.341 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.341 DEBUG:boto:Method: GET 2023-10-18T20:30:44.342 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.342 DEBUG:boto:Data: 2023-10-18T20:30:44.343 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.343 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.343 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.344 DEBUG:boto:Params: {} 2023-10-18T20:30:44.344 DEBUG:boto:Token: None 2023-10-18T20:30:44.345 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.345 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb511218a1b5da26-0065304074-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.350 DEBUG:boto:b'bofrnf-58obj000000000000000000071000falseobj000000000000000000072023-10-18T20:30:32.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.350 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.351 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.351 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.352 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.352 DEBUG:boto:Method: GET 2023-10-18T20:30:44.352 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000007 2023-10-18T20:30:44.353 DEBUG:boto:Data: 2023-10-18T20:30:44.353 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.354 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.354 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.355 DEBUG:boto:Params: {} 2023-10-18T20:30:44.355 DEBUG:boto:Token: None 2023-10-18T20:30:44.356 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007fbc2e0237f3e96-0065304074-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.361 DEBUG:boto:b'bofrnf-58obj000000000000000000071000falseobj000000000000000000072023-10-18T20:30:32.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-18T20:30:44.363 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.363 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.364 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.364 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.364 DEBUG:boto:Method: GET 2023-10-18T20:30:44.365 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.365 DEBUG:boto:Data: 2023-10-18T20:30:44.366 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.366 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.366 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.367 DEBUG:boto:Params: {} 2023-10-18T20:30:44.367 DEBUG:boto:Token: None 2023-10-18T20:30:44.368 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.368 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6/UM9n1tya9piV3X1J57D+MBxiI= 2023-10-18T20:30:44.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6/UM9n1tya9piV3X1J57D+MBxiI=', 'Content-Length': '0'} 2023-10-18T20:30:44.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009884846fe6184f3-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.373 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.373 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.374 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.374 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.374 DEBUG:boto:Method: GET 2023-10-18T20:30:44.375 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.375 DEBUG:boto:Data: 2023-10-18T20:30:44.376 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.376 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.377 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.377 DEBUG:boto:Params: {} 2023-10-18T20:30:44.378 DEBUG:boto:Token: None 2023-10-18T20:30:44.378 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.379 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6/UM9n1tya9piV3X1J57D+MBxiI= 2023-10-18T20:30:44.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6/UM9n1tya9piV3X1J57D+MBxiI=', 'Content-Length': '0'} 2023-10-18T20:30:44.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a4afa2517aef2f7-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.422 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.423 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.423 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.424 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.424 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.425 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.425 DEBUG:boto:Data: 2023-10-18T20:30:44.425 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.426 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.427 DEBUG:boto:Params: {} 2023-10-18T20:30:44.427 DEBUG:boto:Token: None 2023-10-18T20:30:44.428 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pjwHwzQOvvj/at1pMkmSK8RcQgQ= 2023-10-18T20:30:44.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pjwHwzQOvvj/at1pMkmSK8RcQgQ=', 'Content-Length': '0'} 2023-10-18T20:30:44.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009269206a1dfbfe5c-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.432 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.432 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.432 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.433 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.433 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.434 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.434 DEBUG:boto:Data: 2023-10-18T20:30:44.434 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.435 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.435 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.436 DEBUG:boto:Params: {} 2023-10-18T20:30:44.436 DEBUG:boto:Token: None 2023-10-18T20:30:44.436 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.437 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pjwHwzQOvvj/at1pMkmSK8RcQgQ= 2023-10-18T20:30:44.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pjwHwzQOvvj/at1pMkmSK8RcQgQ=', 'Content-Length': '0'} 2023-10-18T20:30:44.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd12455580c59a57-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-18T20:30:44.440 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.441 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.441 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.442 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.442 DEBUG:boto:Method: GET 2023-10-18T20:30:44.442 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.443 DEBUG:boto:Data: 2023-10-18T20:30:44.443 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.444 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.444 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.445 DEBUG:boto:Params: {} 2023-10-18T20:30:44.445 DEBUG:boto:Token: None 2023-10-18T20:30:44.446 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.446 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6/UM9n1tya9piV3X1J57D+MBxiI= 2023-10-18T20:30:44.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6/UM9n1tya9piV3X1J57D+MBxiI=', 'Content-Length': '0'} 2023-10-18T20:30:44.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb46f18b101f542d-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.450 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.450 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.451 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.451 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.451 DEBUG:boto:Method: GET 2023-10-18T20:30:44.452 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.452 DEBUG:boto:Data: 2023-10-18T20:30:44.453 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.453 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.454 DEBUG:boto:Params: {} 2023-10-18T20:30:44.454 DEBUG:boto:Token: None 2023-10-18T20:30:44.455 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008?versionId=null 2023-10-18T20:30:44.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6/UM9n1tya9piV3X1J57D+MBxiI= 2023-10-18T20:30:44.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6/UM9n1tya9piV3X1J57D+MBxiI=', 'Content-Length': '0'} 2023-10-18T20:30:44.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f184f282588aab18-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.459 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.459 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.460 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.460 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.460 DEBUG:boto:Method: GET 2023-10-18T20:30:44.461 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.461 DEBUG:boto:Data: 2023-10-18T20:30:44.462 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.463 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.463 DEBUG:boto:Params: {} 2023-10-18T20:30:44.463 DEBUG:boto:Token: None 2023-10-18T20:30:44.464 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.465 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5b9efc74d757fd3-0065304074-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.468 DEBUG:boto:b'bofrnf-58obj000000000000000000081000falseobj000000000000000000082023-10-18T20:30:29.743Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.469 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.470 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.470 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.470 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.471 DEBUG:boto:Method: GET 2023-10-18T20:30:44.471 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.472 DEBUG:boto:Data: 2023-10-18T20:30:44.472 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.473 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.473 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.473 DEBUG:boto:Params: {} 2023-10-18T20:30:44.474 DEBUG:boto:Token: None 2023-10-18T20:30:44.474 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.475 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e78f3ef73cbdbb3-0065304074-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.480 DEBUG:boto:b'bofrnf-58obj000000000000000000081000falseobj000000000000000000082023-10-18T20:30:29.743Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.481 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.481 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.482 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.482 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.483 DEBUG:boto:Data: 2023-10-18T20:30:44.483 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.484 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.484 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.484 DEBUG:boto:Params: {} 2023-10-18T20:30:44.485 DEBUG:boto:Token: None 2023-10-18T20:30:44.485 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.486 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xhbqjI6Rg/D0U7oweLOxJuKCAn4= 2023-10-18T20:30:44.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xhbqjI6Rg/D0U7oweLOxJuKCAn4=', 'Content-Length': '0'} 2023-10-18T20:30:44.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbf4f52fae4a4474-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.489 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.489 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.490 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.490 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.491 DEBUG:boto:Data: 2023-10-18T20:30:44.491 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.491 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.492 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.492 DEBUG:boto:Params: {} 2023-10-18T20:30:44.493 DEBUG:boto:Token: None 2023-10-18T20:30:44.493 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.494 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xhbqjI6Rg/D0U7oweLOxJuKCAn4= 2023-10-18T20:30:44.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xhbqjI6Rg/D0U7oweLOxJuKCAn4=', 'Content-Length': '0'} 2023-10-18T20:30:44.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002887f36248d82d32-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-18T20:30:44.497 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.497 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.498 DEBUG:boto:Method: GET 2023-10-18T20:30:44.498 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.499 DEBUG:boto:Data: 2023-10-18T20:30:44.499 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.500 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.500 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.500 DEBUG:boto:Params: {} 2023-10-18T20:30:44.501 DEBUG:boto:Token: None 2023-10-18T20:30:44.501 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.502 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jpoEd9/U7hN0MGTy9L7DZaxlRQU= 2023-10-18T20:30:44.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jpoEd9/U7hN0MGTy9L7DZaxlRQU=', 'Content-Length': '0'} 2023-10-18T20:30:44.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3cc8cc7a52d8dc8-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.550 DEBUG:boto:path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.551 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.552 DEBUG:boto:Method: GET 2023-10-18T20:30:44.552 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.553 DEBUG:boto:Data: 2023-10-18T20:30:44.553 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.554 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.554 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.554 DEBUG:boto:Params: {} 2023-10-18T20:30:44.555 DEBUG:boto:Token: None 2023-10-18T20:30:44.555 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000008 2023-10-18T20:30:44.556 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jpoEd9/U7hN0MGTy9L7DZaxlRQU= 2023-10-18T20:30:44.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jpoEd9/U7hN0MGTy9L7DZaxlRQU=', 'Content-Length': '0'} 2023-10-18T20:30:44.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003195ccbefd13a25e-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.559 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.560 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.560 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.561 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.561 DEBUG:boto:Method: GET 2023-10-18T20:30:44.562 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.562 DEBUG:boto:Data: 2023-10-18T20:30:44.563 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.563 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.563 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.564 DEBUG:boto:Params: {} 2023-10-18T20:30:44.564 DEBUG:boto:Token: None 2023-10-18T20:30:44.565 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.565 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009124ebc7d955e03c-0065304074-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.569 DEBUG:boto:b'bofrnf-58obj000000000000000000081000falseobj000000000000000000082023-10-18T20:30:29.743Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.570 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.570 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.571 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.571 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.571 DEBUG:boto:Method: GET 2023-10-18T20:30:44.572 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000008 2023-10-18T20:30:44.572 DEBUG:boto:Data: 2023-10-18T20:30:44.573 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.573 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.574 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.574 DEBUG:boto:Params: {} 2023-10-18T20:30:44.574 DEBUG:boto:Token: None 2023-10-18T20:30:44.575 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.575 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aab7fb557ee1e27-0065304074-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.581 DEBUG:boto:b'bofrnf-58obj000000000000000000081000falseobj000000000000000000082023-10-18T20:30:29.743Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-18T20:30:44.582 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.582 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.583 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.583 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.584 DEBUG:boto:Method: GET 2023-10-18T20:30:44.584 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.585 DEBUG:boto:Data: 2023-10-18T20:30:44.585 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.586 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.586 DEBUG:boto:Params: {} 2023-10-18T20:30:44.587 DEBUG:boto:Token: None 2023-10-18T20:30:44.587 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.588 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dgL+Z77vvfHRFhxh8KTmLS+Zz08= 2023-10-18T20:30:44.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dgL+Z77vvfHRFhxh8KTmLS+Zz08=', 'Content-Length': '0'} 2023-10-18T20:30:44.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a60ea003fc1f04ad-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.591 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.592 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.592 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.593 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.593 DEBUG:boto:Method: GET 2023-10-18T20:30:44.594 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.594 DEBUG:boto:Data: 2023-10-18T20:30:44.595 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.595 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.596 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.596 DEBUG:boto:Params: {} 2023-10-18T20:30:44.597 DEBUG:boto:Token: None 2023-10-18T20:30:44.597 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.598 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dgL+Z77vvfHRFhxh8KTmLS+Zz08= 2023-10-18T20:30:44.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dgL+Z77vvfHRFhxh8KTmLS+Zz08=', 'Content-Length': '0'} 2023-10-18T20:30:44.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032cfbb06fcc15c70-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.601 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.601 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.602 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.602 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.602 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.603 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.603 DEBUG:boto:Data: 2023-10-18T20:30:44.604 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.604 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.604 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.605 DEBUG:boto:Params: {} 2023-10-18T20:30:44.605 DEBUG:boto:Token: None 2023-10-18T20:30:44.606 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.606 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DSP6XQJHGZwKC1MtZZmiOHcZCVg= 2023-10-18T20:30:44.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DSP6XQJHGZwKC1MtZZmiOHcZCVg=', 'Content-Length': '0'} 2023-10-18T20:30:44.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b10b97ca1c25b9c-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.609 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.609 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.610 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.610 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.611 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.611 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.611 DEBUG:boto:Data: 2023-10-18T20:30:44.612 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.612 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.613 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.613 DEBUG:boto:Params: {} 2023-10-18T20:30:44.613 DEBUG:boto:Token: None 2023-10-18T20:30:44.614 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.614 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DSP6XQJHGZwKC1MtZZmiOHcZCVg= 2023-10-18T20:30:44.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DSP6XQJHGZwKC1MtZZmiOHcZCVg=', 'Content-Length': '0'} 2023-10-18T20:30:44.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003da4193f4a5b8d61-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-18T20:30:44.618 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.618 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.619 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.619 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.619 DEBUG:boto:Method: GET 2023-10-18T20:30:44.620 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.620 DEBUG:boto:Data: 2023-10-18T20:30:44.621 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.621 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.622 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.622 DEBUG:boto:Params: {} 2023-10-18T20:30:44.623 DEBUG:boto:Token: None 2023-10-18T20:30:44.623 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dgL+Z77vvfHRFhxh8KTmLS+Zz08= 2023-10-18T20:30:44.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dgL+Z77vvfHRFhxh8KTmLS+Zz08=', 'Content-Length': '0'} 2023-10-18T20:30:44.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf7f849fbbbd59ee-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.627 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.627 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.628 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.628 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.629 DEBUG:boto:Method: GET 2023-10-18T20:30:44.629 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.629 DEBUG:boto:Data: 2023-10-18T20:30:44.630 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.631 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.631 DEBUG:boto:Params: {} 2023-10-18T20:30:44.632 DEBUG:boto:Token: None 2023-10-18T20:30:44.632 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009?versionId=null 2023-10-18T20:30:44.633 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dgL+Z77vvfHRFhxh8KTmLS+Zz08= 2023-10-18T20:30:44.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dgL+Z77vvfHRFhxh8KTmLS+Zz08=', 'Content-Length': '0'} 2023-10-18T20:30:44.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095398c9195937b2a-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.637 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.637 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.638 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.638 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.639 DEBUG:boto:Method: GET 2023-10-18T20:30:44.639 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.640 DEBUG:boto:Data: 2023-10-18T20:30:44.640 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.641 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.642 DEBUG:boto:Params: {} 2023-10-18T20:30:44.642 DEBUG:boto:Token: None 2023-10-18T20:30:44.643 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f6d359297fc27ca-0065304074-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.698 DEBUG:boto:b'bofrnf-58obj000000000000000000091000falseobj000000000000000000092023-10-18T20:30:32.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.699 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.699 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.700 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.700 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.701 DEBUG:boto:Method: GET 2023-10-18T20:30:44.701 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.702 DEBUG:boto:Data: 2023-10-18T20:30:44.702 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.702 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.703 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.707 DEBUG:boto:Params: {} 2023-10-18T20:30:44.708 DEBUG:boto:Token: None 2023-10-18T20:30:44.708 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000942a713b0c384f93-0065304074-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.715 DEBUG:boto:b'bofrnf-58obj000000000000000000091000falseobj000000000000000000092023-10-18T20:30:32.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.716 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.716 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.717 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.717 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.717 DEBUG:boto:Data: 2023-10-18T20:30:44.718 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.719 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.720 DEBUG:boto:Params: {} 2023-10-18T20:30:44.720 DEBUG:boto:Token: None 2023-10-18T20:30:44.721 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b/9GXqILj5+u20LOEtfDL/2B/As= 2023-10-18T20:30:44.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b/9GXqILj5+u20LOEtfDL/2B/As=', 'Content-Length': '0'} 2023-10-18T20:30:44.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1b0a8ebc3845bac-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.725 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.726 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.726 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.726 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.727 DEBUG:boto:Data: 2023-10-18T20:30:44.728 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.729 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.729 DEBUG:boto:Params: {} 2023-10-18T20:30:44.729 DEBUG:boto:Token: None 2023-10-18T20:30:44.730 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.731 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b/9GXqILj5+u20LOEtfDL/2B/As= 2023-10-18T20:30:44.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b/9GXqILj5+u20LOEtfDL/2B/As=', 'Content-Length': '0'} 2023-10-18T20:30:44.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e73ae9cf7cb083cc-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-18T20:30:44.735 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.735 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.736 DEBUG:boto:Method: GET 2023-10-18T20:30:44.736 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.737 DEBUG:boto:Data: 2023-10-18T20:30:44.738 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.738 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.739 DEBUG:boto:Params: {} 2023-10-18T20:30:44.739 DEBUG:boto:Token: None 2023-10-18T20:30:44.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:k8cDslX3zjpZjnP5k8Opw7IaK2Q= 2023-10-18T20:30:44.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:k8cDslX3zjpZjnP5k8Opw7IaK2Q=', 'Content-Length': '0'} 2023-10-18T20:30:44.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000010b488c936320d5-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.743 DEBUG:boto:path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.743 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.744 DEBUG:boto:Method: GET 2023-10-18T20:30:44.745 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.745 DEBUG:boto:Data: 2023-10-18T20:30:44.746 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.747 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.747 DEBUG:boto:Params: {} 2023-10-18T20:30:44.748 DEBUG:boto:Token: None 2023-10-18T20:30:44.748 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000009 2023-10-18T20:30:44.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:k8cDslX3zjpZjnP5k8Opw7IaK2Q= 2023-10-18T20:30:44.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:k8cDslX3zjpZjnP5k8Opw7IaK2Q=', 'Content-Length': '0'} 2023-10-18T20:30:44.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cdbcd14117c8dbe-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.752 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.753 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.753 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.754 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.754 DEBUG:boto:Method: GET 2023-10-18T20:30:44.754 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.755 DEBUG:boto:Data: 2023-10-18T20:30:44.756 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.756 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.757 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.757 DEBUG:boto:Params: {} 2023-10-18T20:30:44.758 DEBUG:boto:Token: None 2023-10-18T20:30:44.758 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.759 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003550babdb8e5eee8-0065304074-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.762 DEBUG:boto:b'bofrnf-58obj000000000000000000091000falseobj000000000000000000092023-10-18T20:30:32.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.763 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.764 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.764 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.764 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.765 DEBUG:boto:Method: GET 2023-10-18T20:30:44.765 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000009 2023-10-18T20:30:44.766 DEBUG:boto:Data: 2023-10-18T20:30:44.767 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.768 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.768 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.768 DEBUG:boto:Params: {} 2023-10-18T20:30:44.769 DEBUG:boto:Token: None 2023-10-18T20:30:44.769 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.770 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c507d3a27592af10-0065304074-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.818 DEBUG:boto:b'bofrnf-58obj000000000000000000091000falseobj000000000000000000092023-10-18T20:30:32.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-18T20:30:44.820 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.820 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.821 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.821 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.821 DEBUG:boto:Method: GET 2023-10-18T20:30:44.822 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.822 DEBUG:boto:Data: 2023-10-18T20:30:44.823 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.823 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.824 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.824 DEBUG:boto:Params: {} 2023-10-18T20:30:44.825 DEBUG:boto:Token: None 2023-10-18T20:30:44.825 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.826 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zfg5AXBJwEVNG6vIEDEia0mCJJE= 2023-10-18T20:30:44.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zfg5AXBJwEVNG6vIEDEia0mCJJE=', 'Content-Length': '0'} 2023-10-18T20:30:44.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e928bb8cbdc2dd68-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.829 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.830 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.830 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.830 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.831 DEBUG:boto:Method: GET 2023-10-18T20:30:44.831 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.832 DEBUG:boto:Data: 2023-10-18T20:30:44.832 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.833 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.833 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.833 DEBUG:boto:Params: {} 2023-10-18T20:30:44.834 DEBUG:boto:Token: None 2023-10-18T20:30:44.834 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.835 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zfg5AXBJwEVNG6vIEDEia0mCJJE= 2023-10-18T20:30:44.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zfg5AXBJwEVNG6vIEDEia0mCJJE=', 'Content-Length': '0'} 2023-10-18T20:30:44.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061dd0723648ca5bd-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.838 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.839 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.839 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.839 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.840 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.840 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.841 DEBUG:boto:Data: 2023-10-18T20:30:44.841 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.841 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.842 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.842 DEBUG:boto:Params: {} 2023-10-18T20:30:44.843 DEBUG:boto:Token: None 2023-10-18T20:30:44.843 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.844 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Lkehuv1p7129tAlbYj7Y5GiLww4= 2023-10-18T20:30:44.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Lkehuv1p7129tAlbYj7Y5GiLww4=', 'Content-Length': '0'} 2023-10-18T20:30:44.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108fab11ce11fbd7-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.847 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.847 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.848 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.848 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.849 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.849 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.849 DEBUG:boto:Data: 2023-10-18T20:30:44.850 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.850 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.851 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.851 DEBUG:boto:Params: {} 2023-10-18T20:30:44.852 DEBUG:boto:Token: None 2023-10-18T20:30:44.852 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.853 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Lkehuv1p7129tAlbYj7Y5GiLww4= 2023-10-18T20:30:44.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Lkehuv1p7129tAlbYj7Y5GiLww4=', 'Content-Length': '0'} 2023-10-18T20:30:44.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089ad4792f7b94241-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-18T20:30:44.856 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.856 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.857 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.857 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.858 DEBUG:boto:Method: GET 2023-10-18T20:30:44.858 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.859 DEBUG:boto:Data: 2023-10-18T20:30:44.859 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.860 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.861 DEBUG:boto:Params: {} 2023-10-18T20:30:44.861 DEBUG:boto:Token: None 2023-10-18T20:30:44.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zfg5AXBJwEVNG6vIEDEia0mCJJE= 2023-10-18T20:30:44.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zfg5AXBJwEVNG6vIEDEia0mCJJE=', 'Content-Length': '0'} 2023-10-18T20:30:44.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddec9d66baf4a8e6-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.866 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.866 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.867 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.867 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.868 DEBUG:boto:Method: GET 2023-10-18T20:30:44.868 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.869 DEBUG:boto:Data: 2023-10-18T20:30:44.869 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.870 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.871 DEBUG:boto:Params: {} 2023-10-18T20:30:44.871 DEBUG:boto:Token: None 2023-10-18T20:30:44.872 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010?versionId=null 2023-10-18T20:30:44.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zfg5AXBJwEVNG6vIEDEia0mCJJE= 2023-10-18T20:30:44.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zfg5AXBJwEVNG6vIEDEia0mCJJE=', 'Content-Length': '0'} 2023-10-18T20:30:44.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5aaf7da6043f0ba-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.875 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.876 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.876 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.877 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.877 DEBUG:boto:Method: GET 2023-10-18T20:30:44.878 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.878 DEBUG:boto:Data: 2023-10-18T20:30:44.878 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.879 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.880 DEBUG:boto:Params: {} 2023-10-18T20:30:44.880 DEBUG:boto:Token: None 2023-10-18T20:30:44.881 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.881 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a239089793e4b5e1-0065304074-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.885 DEBUG:boto:b'bofrnf-58obj000000000000000000101000falseobj000000000000000000102023-10-18T20:30:31.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.886 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.886 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.887 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.887 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.888 DEBUG:boto:Method: GET 2023-10-18T20:30:44.888 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.889 DEBUG:boto:Data: 2023-10-18T20:30:44.889 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.890 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.890 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.891 DEBUG:boto:Params: {} 2023-10-18T20:30:44.891 DEBUG:boto:Token: None 2023-10-18T20:30:44.892 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.892 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b987ac328e586997-0065304074-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.938 DEBUG:boto:b'bofrnf-58obj000000000000000000101000falseobj000000000000000000102023-10-18T20:30:31.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.940 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.941 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.942 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.942 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.943 DEBUG:boto:Data: 2023-10-18T20:30:44.943 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.943 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.944 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.944 DEBUG:boto:Params: {} 2023-10-18T20:30:44.945 DEBUG:boto:Token: None 2023-10-18T20:30:44.945 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.946 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IIBhC6LJk2iIJSQmyNV0vY4Ujnk= 2023-10-18T20:30:44.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IIBhC6LJk2iIJSQmyNV0vY4Ujnk=', 'Content-Length': '0'} 2023-10-18T20:30:44.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c46139b804a7311c-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.949 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.949 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.950 DEBUG:boto:Method: HEAD 2023-10-18T20:30:44.950 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.951 DEBUG:boto:Data: 2023-10-18T20:30:44.951 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.952 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.952 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.952 DEBUG:boto:Params: {} 2023-10-18T20:30:44.953 DEBUG:boto:Token: None 2023-10-18T20:30:44.953 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.954 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IIBhC6LJk2iIJSQmyNV0vY4Ujnk= 2023-10-18T20:30:44.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IIBhC6LJk2iIJSQmyNV0vY4Ujnk=', 'Content-Length': '0'} 2023-10-18T20:30:44.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4cbb86882438e58-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-18T20:30:44.957 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.958 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.958 DEBUG:boto:Method: GET 2023-10-18T20:30:44.959 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.959 DEBUG:boto:Data: 2023-10-18T20:30:44.960 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.960 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.960 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.961 DEBUG:boto:Params: {} 2023-10-18T20:30:44.961 DEBUG:boto:Token: None 2023-10-18T20:30:44.962 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.962 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aNE9Zw91/28R8hYn+hrnsnfyQi0= 2023-10-18T20:30:44.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aNE9Zw91/28R8hYn+hrnsnfyQi0=', 'Content-Length': '0'} 2023-10-18T20:30:44.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2a149580b336c6-0065304074-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.966 DEBUG:boto:path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.966 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.967 DEBUG:boto:Method: GET 2023-10-18T20:30:44.967 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.968 DEBUG:boto:Data: 2023-10-18T20:30:44.968 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.969 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.969 DEBUG:boto:Params: {} 2023-10-18T20:30:44.970 DEBUG:boto:Token: None 2023-10-18T20:30:44.970 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/obj00000000000000000010 2023-10-18T20:30:44.971 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aNE9Zw91/28R8hYn+hrnsnfyQi0= 2023-10-18T20:30:44.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aNE9Zw91/28R8hYn+hrnsnfyQi0=', 'Content-Length': '0'} 2023-10-18T20:30:44.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fba1ebfe989588f5-0065304074-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.974 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.975 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.975 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.976 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.976 DEBUG:boto:Method: GET 2023-10-18T20:30:44.977 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.977 DEBUG:boto:Data: 2023-10-18T20:30:44.978 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.978 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:44.979 DEBUG:boto:Port: 8001 2023-10-18T20:30:44.979 DEBUG:boto:Params: {} 2023-10-18T20:30:44.980 DEBUG:boto:Token: None 2023-10-18T20:30:44.980 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.981 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecea7ce1f9b3847a-0065304074-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.984 DEBUG:boto:b'bofrnf-58obj000000000000000000101000falseobj000000000000000000102023-10-18T20:30:31.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.985 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:44.985 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:44.986 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.986 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.987 DEBUG:boto:Method: GET 2023-10-18T20:30:44.987 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000010 2023-10-18T20:30:44.988 DEBUG:boto:Data: 2023-10-18T20:30:44.988 DEBUG:boto:Headers: {} 2023-10-18T20:30:44.988 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:44.989 DEBUG:boto:Port: 8000 2023-10-18T20:30:44.989 DEBUG:boto:Params: {} 2023-10-18T20:30:44.990 DEBUG:boto:Token: None 2023-10-18T20:30:44.990 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:44 GMT /bofrnf-58/ 2023-10-18T20:30:44.991 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY= 2023-10-18T20:30:44.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx2avbvK3cdIhrLuLhZAD8QR1aY=', 'Content-Length': '0'} 2023-10-18T20:30:44.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f58bc792018b3434-0065304074-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:44.996 DEBUG:boto:b'bofrnf-58obj000000000000000000101000falseobj000000000000000000102023-10-18T20:30:31.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:44.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-18T20:30:44.997 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:44.998 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:44.998 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:44.999 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:44.999 DEBUG:boto:Method: GET 2023-10-18T20:30:44.999 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.000 DEBUG:boto:Data: 2023-10-18T20:30:45.000 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.001 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.002 DEBUG:boto:Params: {} 2023-10-18T20:30:45.002 DEBUG:boto:Token: None 2023-10-18T20:30:45.003 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/gQAErR4ZyS3zC7OiEFMYo9GN8k= 2023-10-18T20:30:45.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/gQAErR4ZyS3zC7OiEFMYo9GN8k=', 'Content-Length': '0'} 2023-10-18T20:30:45.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091df74e6036bdda7-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.046 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.047 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.048 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.048 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.049 DEBUG:boto:Method: GET 2023-10-18T20:30:45.049 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.050 DEBUG:boto:Data: 2023-10-18T20:30:45.050 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.051 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.051 DEBUG:boto:Params: {} 2023-10-18T20:30:45.052 DEBUG:boto:Token: None 2023-10-18T20:30:45.052 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.053 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/gQAErR4ZyS3zC7OiEFMYo9GN8k= 2023-10-18T20:30:45.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/gQAErR4ZyS3zC7OiEFMYo9GN8k=', 'Content-Length': '0'} 2023-10-18T20:30:45.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076d3b629af6be845-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.056 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.057 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.057 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.058 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.058 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.059 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.059 DEBUG:boto:Data: 2023-10-18T20:30:45.060 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.060 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.061 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.061 DEBUG:boto:Params: {} 2023-10-18T20:30:45.062 DEBUG:boto:Token: None 2023-10-18T20:30:45.062 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.062 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2fEuKvGCRypyQqXzWqVBlmhERYw= 2023-10-18T20:30:45.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2fEuKvGCRypyQqXzWqVBlmhERYw=', 'Content-Length': '0'} 2023-10-18T20:30:45.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024352c4fad0c25ea-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.066 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.066 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.067 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.067 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.068 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.068 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.068 DEBUG:boto:Data: 2023-10-18T20:30:45.069 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.069 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.070 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.070 DEBUG:boto:Params: {} 2023-10-18T20:30:45.071 DEBUG:boto:Token: None 2023-10-18T20:30:45.071 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.072 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2fEuKvGCRypyQqXzWqVBlmhERYw= 2023-10-18T20:30:45.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2fEuKvGCRypyQqXzWqVBlmhERYw=', 'Content-Length': '0'} 2023-10-18T20:30:45.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d677e0d556cae65-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-18T20:30:45.075 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.076 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.076 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.077 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.077 DEBUG:boto:Method: GET 2023-10-18T20:30:45.078 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.078 DEBUG:boto:Data: 2023-10-18T20:30:45.079 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.079 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.080 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.080 DEBUG:boto:Params: {} 2023-10-18T20:30:45.081 DEBUG:boto:Token: None 2023-10-18T20:30:45.081 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.082 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/gQAErR4ZyS3zC7OiEFMYo9GN8k= 2023-10-18T20:30:45.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/gQAErR4ZyS3zC7OiEFMYo9GN8k=', 'Content-Length': '0'} 2023-10-18T20:30:45.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33ef57687d15ccd-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.085 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.085 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.086 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.086 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.087 DEBUG:boto:Method: GET 2023-10-18T20:30:45.087 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.088 DEBUG:boto:Data: 2023-10-18T20:30:45.088 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.089 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.089 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.090 DEBUG:boto:Params: {} 2023-10-18T20:30:45.090 DEBUG:boto:Token: None 2023-10-18T20:30:45.091 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011?versionId=null 2023-10-18T20:30:45.091 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/gQAErR4ZyS3zC7OiEFMYo9GN8k= 2023-10-18T20:30:45.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/gQAErR4ZyS3zC7OiEFMYo9GN8k=', 'Content-Length': '0'} 2023-10-18T20:30:45.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c09224b07321226-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.095 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.095 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.096 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.096 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.097 DEBUG:boto:Method: GET 2023-10-18T20:30:45.097 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.098 DEBUG:boto:Data: 2023-10-18T20:30:45.098 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.099 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.099 DEBUG:boto:Params: {} 2023-10-18T20:30:45.100 DEBUG:boto:Token: None 2023-10-18T20:30:45.100 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.101 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b5c5a9a2b10a478-0065304075-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.105 DEBUG:boto:b'bofrnf-58obj000000000000000000111000falseobj000000000000000000112023-10-18T20:30:32.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.106 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.110 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.110 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.111 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.111 DEBUG:boto:Method: GET 2023-10-18T20:30:45.112 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.112 DEBUG:boto:Data: 2023-10-18T20:30:45.112 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.113 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.113 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.114 DEBUG:boto:Params: {} 2023-10-18T20:30:45.114 DEBUG:boto:Token: None 2023-10-18T20:30:45.115 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d477cf3cb9cc347-0065304075-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.121 DEBUG:boto:b'bofrnf-58obj000000000000000000111000falseobj000000000000000000112023-10-18T20:30:32.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.122 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.122 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.123 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.123 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.124 DEBUG:boto:Data: 2023-10-18T20:30:45.124 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.125 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.125 DEBUG:boto:Params: {} 2023-10-18T20:30:45.126 DEBUG:boto:Token: None 2023-10-18T20:30:45.126 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.127 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0UXw1hHKJ51uC9DM8NaC1k5ZuB4= 2023-10-18T20:30:45.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0UXw1hHKJ51uC9DM8NaC1k5ZuB4=', 'Content-Length': '0'} 2023-10-18T20:30:45.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d019085ef7f6a0-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.130 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.130 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.131 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.131 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.132 DEBUG:boto:Data: 2023-10-18T20:30:45.132 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.132 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.133 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.133 DEBUG:boto:Params: {} 2023-10-18T20:30:45.134 DEBUG:boto:Token: None 2023-10-18T20:30:45.134 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0UXw1hHKJ51uC9DM8NaC1k5ZuB4= 2023-10-18T20:30:45.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0UXw1hHKJ51uC9DM8NaC1k5ZuB4=', 'Content-Length': '0'} 2023-10-18T20:30:45.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007978ca5fec7b6497-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-18T20:30:45.139 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.139 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.140 DEBUG:boto:Method: GET 2023-10-18T20:30:45.140 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.140 DEBUG:boto:Data: 2023-10-18T20:30:45.141 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.142 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.142 DEBUG:boto:Params: {} 2023-10-18T20:30:45.143 DEBUG:boto:Token: None 2023-10-18T20:30:45.143 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CUMX4Q6jzV+9A6O28KzBv2eAAfE= 2023-10-18T20:30:45.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CUMX4Q6jzV+9A6O28KzBv2eAAfE=', 'Content-Length': '0'} 2023-10-18T20:30:45.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f7fe742ce85371-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.147 DEBUG:boto:path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.147 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.148 DEBUG:boto:Method: GET 2023-10-18T20:30:45.148 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.148 DEBUG:boto:Data: 2023-10-18T20:30:45.149 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.149 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.150 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.150 DEBUG:boto:Params: {} 2023-10-18T20:30:45.151 DEBUG:boto:Token: None 2023-10-18T20:30:45.151 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000011 2023-10-18T20:30:45.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CUMX4Q6jzV+9A6O28KzBv2eAAfE= 2023-10-18T20:30:45.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CUMX4Q6jzV+9A6O28KzBv2eAAfE=', 'Content-Length': '0'} 2023-10-18T20:30:45.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a78859493440b7-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.155 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.155 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.156 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.156 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.157 DEBUG:boto:Method: GET 2023-10-18T20:30:45.157 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.157 DEBUG:boto:Data: 2023-10-18T20:30:45.158 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.159 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.159 DEBUG:boto:Params: {} 2023-10-18T20:30:45.160 DEBUG:boto:Token: None 2023-10-18T20:30:45.160 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000599454d44c4aa568-0065304075-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.164 DEBUG:boto:b'bofrnf-58obj000000000000000000111000falseobj000000000000000000112023-10-18T20:30:32.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.165 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.166 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.166 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.166 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.167 DEBUG:boto:Method: GET 2023-10-18T20:30:45.167 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000011 2023-10-18T20:30:45.168 DEBUG:boto:Data: 2023-10-18T20:30:45.168 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.169 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.169 DEBUG:boto:Params: {} 2023-10-18T20:30:45.170 DEBUG:boto:Token: None 2023-10-18T20:30:45.170 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1d886f9b5ab743b-0065304075-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.218 DEBUG:boto:b'bofrnf-58obj000000000000000000111000falseobj000000000000000000112023-10-18T20:30:32.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-18T20:30:45.219 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.220 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.220 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.221 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.221 DEBUG:boto:Method: GET 2023-10-18T20:30:45.221 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.222 DEBUG:boto:Data: 2023-10-18T20:30:45.222 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.223 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.223 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.224 DEBUG:boto:Params: {} 2023-10-18T20:30:45.224 DEBUG:boto:Token: None 2023-10-18T20:30:45.224 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.225 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bqgrjkc09loGpE9meHDancqr5cc= 2023-10-18T20:30:45.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bqgrjkc09loGpE9meHDancqr5cc=', 'Content-Length': '0'} 2023-10-18T20:30:45.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d4e9f5954458e24-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.228 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.229 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.229 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.230 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.230 DEBUG:boto:Method: GET 2023-10-18T20:30:45.231 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.231 DEBUG:boto:Data: 2023-10-18T20:30:45.231 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.232 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.232 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.232 DEBUG:boto:Params: {} 2023-10-18T20:30:45.233 DEBUG:boto:Token: None 2023-10-18T20:30:45.233 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.234 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bqgrjkc09loGpE9meHDancqr5cc= 2023-10-18T20:30:45.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bqgrjkc09loGpE9meHDancqr5cc=', 'Content-Length': '0'} 2023-10-18T20:30:45.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc1fa3a691e6916e-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.237 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.237 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.238 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.238 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.238 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.239 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.239 DEBUG:boto:Data: 2023-10-18T20:30:45.240 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.240 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.240 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.241 DEBUG:boto:Params: {} 2023-10-18T20:30:45.241 DEBUG:boto:Token: None 2023-10-18T20:30:45.242 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.242 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ET7oG9lq0xUPJiSKAOfgYa3H694= 2023-10-18T20:30:45.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ET7oG9lq0xUPJiSKAOfgYa3H694=', 'Content-Length': '0'} 2023-10-18T20:30:45.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a382ade922c73542-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.245 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.245 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.246 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.246 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.247 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.247 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.248 DEBUG:boto:Data: 2023-10-18T20:30:45.248 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.249 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.249 DEBUG:boto:Params: {} 2023-10-18T20:30:45.250 DEBUG:boto:Token: None 2023-10-18T20:30:45.250 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.251 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ET7oG9lq0xUPJiSKAOfgYa3H694= 2023-10-18T20:30:45.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ET7oG9lq0xUPJiSKAOfgYa3H694=', 'Content-Length': '0'} 2023-10-18T20:30:45.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2acf771a8a15fc5-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-18T20:30:45.254 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.254 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.255 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.255 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.256 DEBUG:boto:Method: GET 2023-10-18T20:30:45.256 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.257 DEBUG:boto:Data: 2023-10-18T20:30:45.257 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.257 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.258 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.258 DEBUG:boto:Params: {} 2023-10-18T20:30:45.259 DEBUG:boto:Token: None 2023-10-18T20:30:45.259 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bqgrjkc09loGpE9meHDancqr5cc= 2023-10-18T20:30:45.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bqgrjkc09loGpE9meHDancqr5cc=', 'Content-Length': '0'} 2023-10-18T20:30:45.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc055181b24235b1-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.263 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.263 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.264 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.264 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.264 DEBUG:boto:Method: GET 2023-10-18T20:30:45.265 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.265 DEBUG:boto:Data: 2023-10-18T20:30:45.265 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.266 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.266 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.267 DEBUG:boto:Params: {} 2023-10-18T20:30:45.267 DEBUG:boto:Token: None 2023-10-18T20:30:45.268 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012?versionId=null 2023-10-18T20:30:45.268 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bqgrjkc09loGpE9meHDancqr5cc= 2023-10-18T20:30:45.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bqgrjkc09loGpE9meHDancqr5cc=', 'Content-Length': '0'} 2023-10-18T20:30:45.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfafc236f933ae6b-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.271 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.271 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.272 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.272 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.273 DEBUG:boto:Method: GET 2023-10-18T20:30:45.273 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.273 DEBUG:boto:Data: 2023-10-18T20:30:45.274 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.274 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.275 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.275 DEBUG:boto:Params: {} 2023-10-18T20:30:45.276 DEBUG:boto:Token: None 2023-10-18T20:30:45.276 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000485850f31af5100b-0065304075-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.280 DEBUG:boto:b'bofrnf-58obj000000000000000000121000falseobj000000000000000000122023-10-18T20:30:32.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.281 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.281 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.281 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.282 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.282 DEBUG:boto:Method: GET 2023-10-18T20:30:45.283 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.283 DEBUG:boto:Data: 2023-10-18T20:30:45.284 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.284 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.284 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.285 DEBUG:boto:Params: {} 2023-10-18T20:30:45.285 DEBUG:boto:Token: None 2023-10-18T20:30:45.286 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.286 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c316c2d30a54eb1-0065304075-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.292 DEBUG:boto:b'bofrnf-58obj000000000000000000121000falseobj000000000000000000122023-10-18T20:30:32.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.292 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.293 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.293 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.294 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.294 DEBUG:boto:Data: 2023-10-18T20:30:45.294 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.295 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.295 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.296 DEBUG:boto:Params: {} 2023-10-18T20:30:45.296 DEBUG:boto:Token: None 2023-10-18T20:30:45.297 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.297 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fJdil3GVKjCZrjADxloRXzIzepQ= 2023-10-18T20:30:45.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fJdil3GVKjCZrjADxloRXzIzepQ=', 'Content-Length': '0'} 2023-10-18T20:30:45.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5101b9974cb43f7-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.300 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.300 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.301 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.301 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.302 DEBUG:boto:Data: 2023-10-18T20:30:45.302 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.303 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.303 DEBUG:boto:Params: {} 2023-10-18T20:30:45.304 DEBUG:boto:Token: None 2023-10-18T20:30:45.304 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fJdil3GVKjCZrjADxloRXzIzepQ= 2023-10-18T20:30:45.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fJdil3GVKjCZrjADxloRXzIzepQ=', 'Content-Length': '0'} 2023-10-18T20:30:45.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003707af7b5705a7f1-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-18T20:30:45.308 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.309 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.309 DEBUG:boto:Method: GET 2023-10-18T20:30:45.310 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.310 DEBUG:boto:Data: 2023-10-18T20:30:45.310 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.311 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.311 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.312 DEBUG:boto:Params: {} 2023-10-18T20:30:45.312 DEBUG:boto:Token: None 2023-10-18T20:30:45.313 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8e3c7klWr3whnb1jmz84vmoLrLs= 2023-10-18T20:30:45.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8e3c7klWr3whnb1jmz84vmoLrLs=', 'Content-Length': '0'} 2023-10-18T20:30:45.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d708123949d2d71-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.319 DEBUG:boto:path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.319 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.320 DEBUG:boto:Method: GET 2023-10-18T20:30:45.320 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.320 DEBUG:boto:Data: 2023-10-18T20:30:45.321 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.321 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.322 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.322 DEBUG:boto:Params: {} 2023-10-18T20:30:45.323 DEBUG:boto:Token: None 2023-10-18T20:30:45.323 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000012 2023-10-18T20:30:45.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8e3c7klWr3whnb1jmz84vmoLrLs= 2023-10-18T20:30:45.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8e3c7klWr3whnb1jmz84vmoLrLs=', 'Content-Length': '0'} 2023-10-18T20:30:45.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083513209021563d7-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.370 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.371 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.372 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.372 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.372 DEBUG:boto:Method: GET 2023-10-18T20:30:45.373 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.373 DEBUG:boto:Data: 2023-10-18T20:30:45.374 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.375 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.375 DEBUG:boto:Params: {} 2023-10-18T20:30:45.376 DEBUG:boto:Token: None 2023-10-18T20:30:45.376 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.377 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000095d3786685c2ba2-0065304075-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.381 DEBUG:boto:b'bofrnf-58obj000000000000000000121000falseobj000000000000000000122023-10-18T20:30:32.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.382 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.382 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.382 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.383 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.383 DEBUG:boto:Method: GET 2023-10-18T20:30:45.384 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000012 2023-10-18T20:30:45.384 DEBUG:boto:Data: 2023-10-18T20:30:45.385 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.385 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.385 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.386 DEBUG:boto:Params: {} 2023-10-18T20:30:45.386 DEBUG:boto:Token: None 2023-10-18T20:30:45.387 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001697ca97b48cb0c3-0065304075-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.393 DEBUG:boto:b'bofrnf-58obj000000000000000000121000falseobj000000000000000000122023-10-18T20:30:32.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-18T20:30:45.394 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.395 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.395 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.396 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.396 DEBUG:boto:Method: GET 2023-10-18T20:30:45.397 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.397 DEBUG:boto:Data: 2023-10-18T20:30:45.397 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.398 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.399 DEBUG:boto:Params: {} 2023-10-18T20:30:45.399 DEBUG:boto:Token: None 2023-10-18T20:30:45.400 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.400 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PPA2sJiazCP66xAQ7/11sWv1O6Q= 2023-10-18T20:30:45.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PPA2sJiazCP66xAQ7/11sWv1O6Q=', 'Content-Length': '0'} 2023-10-18T20:30:45.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d8628898f876a5c-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.404 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.405 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.405 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.406 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.406 DEBUG:boto:Method: GET 2023-10-18T20:30:45.407 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.407 DEBUG:boto:Data: 2023-10-18T20:30:45.408 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.409 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.409 DEBUG:boto:Params: {} 2023-10-18T20:30:45.410 DEBUG:boto:Token: None 2023-10-18T20:30:45.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PPA2sJiazCP66xAQ7/11sWv1O6Q= 2023-10-18T20:30:45.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PPA2sJiazCP66xAQ7/11sWv1O6Q=', 'Content-Length': '0'} 2023-10-18T20:30:45.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98c8eea0fc5d927-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.414 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.415 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.415 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.415 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.416 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.416 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.417 DEBUG:boto:Data: 2023-10-18T20:30:45.417 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.418 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.419 DEBUG:boto:Params: {} 2023-10-18T20:30:45.419 DEBUG:boto:Token: None 2023-10-18T20:30:45.420 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lSU1RK95Lvh72gCZVTV/hGjA2BE= 2023-10-18T20:30:45.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lSU1RK95Lvh72gCZVTV/hGjA2BE=', 'Content-Length': '0'} 2023-10-18T20:30:45.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e768916f66a9d83b-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.424 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.425 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.425 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.426 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.427 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.427 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.428 DEBUG:boto:Data: 2023-10-18T20:30:45.428 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.429 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.430 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.430 DEBUG:boto:Params: {} 2023-10-18T20:30:45.431 DEBUG:boto:Token: None 2023-10-18T20:30:45.432 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.432 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lSU1RK95Lvh72gCZVTV/hGjA2BE= 2023-10-18T20:30:45.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lSU1RK95Lvh72gCZVTV/hGjA2BE=', 'Content-Length': '0'} 2023-10-18T20:30:45.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2fec4dc7c64719e-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-18T20:30:45.435 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.436 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.436 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.437 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.437 DEBUG:boto:Method: GET 2023-10-18T20:30:45.438 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.438 DEBUG:boto:Data: 2023-10-18T20:30:45.438 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.439 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.439 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.440 DEBUG:boto:Params: {} 2023-10-18T20:30:45.440 DEBUG:boto:Token: None 2023-10-18T20:30:45.441 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.441 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PPA2sJiazCP66xAQ7/11sWv1O6Q= 2023-10-18T20:30:45.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PPA2sJiazCP66xAQ7/11sWv1O6Q=', 'Content-Length': '0'} 2023-10-18T20:30:45.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099888ef542a35c7e-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.490 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.491 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.492 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.492 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.493 DEBUG:boto:Method: GET 2023-10-18T20:30:45.493 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.493 DEBUG:boto:Data: 2023-10-18T20:30:45.494 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.495 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.495 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.495 DEBUG:boto:Params: {} 2023-10-18T20:30:45.496 DEBUG:boto:Token: None 2023-10-18T20:30:45.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013?versionId=null 2023-10-18T20:30:45.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PPA2sJiazCP66xAQ7/11sWv1O6Q= 2023-10-18T20:30:45.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PPA2sJiazCP66xAQ7/11sWv1O6Q=', 'Content-Length': '0'} 2023-10-18T20:30:45.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ec5dbb724b6d88-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.501 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.501 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.502 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.502 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.503 DEBUG:boto:Method: GET 2023-10-18T20:30:45.503 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.504 DEBUG:boto:Data: 2023-10-18T20:30:45.504 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.504 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.505 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.505 DEBUG:boto:Params: {} 2023-10-18T20:30:45.506 DEBUG:boto:Token: None 2023-10-18T20:30:45.506 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.507 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e93f530b53d06390-0065304075-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.511 DEBUG:boto:b'bofrnf-58obj000000000000000000131000falseobj000000000000000000132023-10-18T20:30:32.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.511 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.512 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.512 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.513 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.513 DEBUG:boto:Method: GET 2023-10-18T20:30:45.514 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.514 DEBUG:boto:Data: 2023-10-18T20:30:45.514 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.515 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.515 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.516 DEBUG:boto:Params: {} 2023-10-18T20:30:45.516 DEBUG:boto:Token: None 2023-10-18T20:30:45.517 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.517 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e67e105f22d7e92-0065304075-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.522 DEBUG:boto:b'bofrnf-58obj000000000000000000131000falseobj000000000000000000132023-10-18T20:30:32.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.523 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.523 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.524 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.524 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.524 DEBUG:boto:Data: 2023-10-18T20:30:45.525 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.526 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.526 DEBUG:boto:Params: {} 2023-10-18T20:30:45.526 DEBUG:boto:Token: None 2023-10-18T20:30:45.527 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.528 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b3jvToyRe9MRqcbKVnPK87a5q8M= 2023-10-18T20:30:45.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b3jvToyRe9MRqcbKVnPK87a5q8M=', 'Content-Length': '0'} 2023-10-18T20:30:45.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d3f7bc4f0aff598-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.531 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.531 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.532 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.532 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.533 DEBUG:boto:Data: 2023-10-18T20:30:45.533 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.533 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.534 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.534 DEBUG:boto:Params: {} 2023-10-18T20:30:45.535 DEBUG:boto:Token: None 2023-10-18T20:30:45.535 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b3jvToyRe9MRqcbKVnPK87a5q8M= 2023-10-18T20:30:45.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b3jvToyRe9MRqcbKVnPK87a5q8M=', 'Content-Length': '0'} 2023-10-18T20:30:45.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001236197c72ae67d1-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-18T20:30:45.539 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.540 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.540 DEBUG:boto:Method: GET 2023-10-18T20:30:45.541 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.541 DEBUG:boto:Data: 2023-10-18T20:30:45.542 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.542 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.543 DEBUG:boto:Params: {} 2023-10-18T20:30:45.543 DEBUG:boto:Token: None 2023-10-18T20:30:45.544 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ppYz6utYsis9wWxog7WcvHhBvmk= 2023-10-18T20:30:45.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ppYz6utYsis9wWxog7WcvHhBvmk=', 'Content-Length': '0'} 2023-10-18T20:30:45.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba49b8c392de3f48-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.547 DEBUG:boto:path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.548 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.548 DEBUG:boto:Method: GET 2023-10-18T20:30:45.549 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.549 DEBUG:boto:Data: 2023-10-18T20:30:45.550 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.550 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.551 DEBUG:boto:Params: {} 2023-10-18T20:30:45.551 DEBUG:boto:Token: None 2023-10-18T20:30:45.552 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000013 2023-10-18T20:30:45.552 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ppYz6utYsis9wWxog7WcvHhBvmk= 2023-10-18T20:30:45.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ppYz6utYsis9wWxog7WcvHhBvmk=', 'Content-Length': '0'} 2023-10-18T20:30:45.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223900ef0ff5387c-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.598 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.599 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.599 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.600 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.600 DEBUG:boto:Method: GET 2023-10-18T20:30:45.601 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.601 DEBUG:boto:Data: 2023-10-18T20:30:45.602 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.602 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.603 DEBUG:boto:Params: {} 2023-10-18T20:30:45.603 DEBUG:boto:Token: None 2023-10-18T20:30:45.604 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019f30ead60c562eb-0065304075-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.608 DEBUG:boto:b'bofrnf-58obj000000000000000000131000falseobj000000000000000000132023-10-18T20:30:32.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.609 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.609 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.610 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.610 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.611 DEBUG:boto:Method: GET 2023-10-18T20:30:45.611 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000013 2023-10-18T20:30:45.612 DEBUG:boto:Data: 2023-10-18T20:30:45.612 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.612 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.613 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.613 DEBUG:boto:Params: {} 2023-10-18T20:30:45.614 DEBUG:boto:Token: None 2023-10-18T20:30:45.614 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a656d65804cbb53f-0065304075-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.620 DEBUG:boto:b'bofrnf-58obj000000000000000000131000falseobj000000000000000000132023-10-18T20:30:32.103Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-18T20:30:45.621 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.622 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.622 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.623 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.623 DEBUG:boto:Method: GET 2023-10-18T20:30:45.623 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.624 DEBUG:boto:Data: 2023-10-18T20:30:45.624 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.625 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.626 DEBUG:boto:Params: {} 2023-10-18T20:30:45.626 DEBUG:boto:Token: None 2023-10-18T20:30:45.627 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.627 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X6O6g1G6KdRlJoKEY9jnpLerX80= 2023-10-18T20:30:45.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X6O6g1G6KdRlJoKEY9jnpLerX80=', 'Content-Length': '0'} 2023-10-18T20:30:45.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160d101c8f059601-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.674 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.675 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.675 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.676 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.676 DEBUG:boto:Method: GET 2023-10-18T20:30:45.677 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.677 DEBUG:boto:Data: 2023-10-18T20:30:45.678 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.678 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.678 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.679 DEBUG:boto:Params: {} 2023-10-18T20:30:45.679 DEBUG:boto:Token: None 2023-10-18T20:30:45.680 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.680 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X6O6g1G6KdRlJoKEY9jnpLerX80= 2023-10-18T20:30:45.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X6O6g1G6KdRlJoKEY9jnpLerX80=', 'Content-Length': '0'} 2023-10-18T20:30:45.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2613b4cb17703f4-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.684 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.684 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.685 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.685 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.685 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.686 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.686 DEBUG:boto:Data: 2023-10-18T20:30:45.687 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.687 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.687 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.688 DEBUG:boto:Params: {} 2023-10-18T20:30:45.688 DEBUG:boto:Token: None 2023-10-18T20:30:45.689 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.689 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZmMc12dlFqeVvhxlLcs6rE8gZwM= 2023-10-18T20:30:45.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZmMc12dlFqeVvhxlLcs6rE8gZwM=', 'Content-Length': '0'} 2023-10-18T20:30:45.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2884d20bae1a764-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.692 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.693 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.693 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.694 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.694 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.694 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.695 DEBUG:boto:Data: 2023-10-18T20:30:45.695 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.696 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.696 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.696 DEBUG:boto:Params: {} 2023-10-18T20:30:45.697 DEBUG:boto:Token: None 2023-10-18T20:30:45.697 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.698 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZmMc12dlFqeVvhxlLcs6rE8gZwM= 2023-10-18T20:30:45.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZmMc12dlFqeVvhxlLcs6rE8gZwM=', 'Content-Length': '0'} 2023-10-18T20:30:45.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001af6deeb1f6f4fe2-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-18T20:30:45.702 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.702 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.703 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.703 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.703 DEBUG:boto:Method: GET 2023-10-18T20:30:45.704 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.704 DEBUG:boto:Data: 2023-10-18T20:30:45.705 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.705 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.705 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.706 DEBUG:boto:Params: {} 2023-10-18T20:30:45.706 DEBUG:boto:Token: None 2023-10-18T20:30:45.707 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.707 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X6O6g1G6KdRlJoKEY9jnpLerX80= 2023-10-18T20:30:45.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X6O6g1G6KdRlJoKEY9jnpLerX80=', 'Content-Length': '0'} 2023-10-18T20:30:45.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000312763f75d8b41c2-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.710 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.711 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.711 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.712 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.712 DEBUG:boto:Method: GET 2023-10-18T20:30:45.713 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.713 DEBUG:boto:Data: 2023-10-18T20:30:45.713 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.714 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.714 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.715 DEBUG:boto:Params: {} 2023-10-18T20:30:45.715 DEBUG:boto:Token: None 2023-10-18T20:30:45.716 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014?versionId=null 2023-10-18T20:30:45.716 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X6O6g1G6KdRlJoKEY9jnpLerX80= 2023-10-18T20:30:45.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X6O6g1G6KdRlJoKEY9jnpLerX80=', 'Content-Length': '0'} 2023-10-18T20:30:45.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2d238f423ec06bb-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.719 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.720 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.720 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.721 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.721 DEBUG:boto:Method: GET 2023-10-18T20:30:45.722 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.722 DEBUG:boto:Data: 2023-10-18T20:30:45.722 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.723 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.724 DEBUG:boto:Params: {} 2023-10-18T20:30:45.724 DEBUG:boto:Token: None 2023-10-18T20:30:45.725 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb45f85ed07ff89-0065304075-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.729 DEBUG:boto:b'bofrnf-58obj000000000000000000141000falseobj000000000000000000142023-10-18T20:30:31.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.729 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.730 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.730 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.731 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.731 DEBUG:boto:Method: GET 2023-10-18T20:30:45.732 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.732 DEBUG:boto:Data: 2023-10-18T20:30:45.732 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.733 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.734 DEBUG:boto:Params: {} 2023-10-18T20:30:45.734 DEBUG:boto:Token: None 2023-10-18T20:30:45.735 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e24bf183c68dd4c4-0065304075-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.782 DEBUG:boto:b'bofrnf-58obj000000000000000000141000falseobj000000000000000000142023-10-18T20:30:31.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.783 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.784 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.784 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.784 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.785 DEBUG:boto:Data: 2023-10-18T20:30:45.785 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.785 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.786 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.786 DEBUG:boto:Params: {} 2023-10-18T20:30:45.787 DEBUG:boto:Token: None 2023-10-18T20:30:45.787 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.788 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sivPxxNB5EnztTzMo5yKxwPBJPk= 2023-10-18T20:30:45.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sivPxxNB5EnztTzMo5yKxwPBJPk=', 'Content-Length': '0'} 2023-10-18T20:30:45.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e938d33d7eadb3b-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.791 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.791 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.791 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.792 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.792 DEBUG:boto:Data: 2023-10-18T20:30:45.793 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.793 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.793 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.794 DEBUG:boto:Params: {} 2023-10-18T20:30:45.794 DEBUG:boto:Token: None 2023-10-18T20:30:45.795 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.795 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sivPxxNB5EnztTzMo5yKxwPBJPk= 2023-10-18T20:30:45.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sivPxxNB5EnztTzMo5yKxwPBJPk=', 'Content-Length': '0'} 2023-10-18T20:30:45.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000635c09bed198fab9-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-18T20:30:45.799 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.799 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.799 DEBUG:boto:Method: GET 2023-10-18T20:30:45.800 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.800 DEBUG:boto:Data: 2023-10-18T20:30:45.801 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.801 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.802 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.802 DEBUG:boto:Params: {} 2023-10-18T20:30:45.802 DEBUG:boto:Token: None 2023-10-18T20:30:45.803 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.803 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cXbBWs4WS6TUBqyr95jsfJCl6EE= 2023-10-18T20:30:45.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cXbBWs4WS6TUBqyr95jsfJCl6EE=', 'Content-Length': '0'} 2023-10-18T20:30:45.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2cda8f8ccecb583-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.806 DEBUG:boto:path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.807 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.807 DEBUG:boto:Method: GET 2023-10-18T20:30:45.808 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.808 DEBUG:boto:Data: 2023-10-18T20:30:45.808 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.809 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.809 DEBUG:boto:Params: {} 2023-10-18T20:30:45.810 DEBUG:boto:Token: None 2023-10-18T20:30:45.810 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000014 2023-10-18T20:30:45.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cXbBWs4WS6TUBqyr95jsfJCl6EE= 2023-10-18T20:30:45.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cXbBWs4WS6TUBqyr95jsfJCl6EE=', 'Content-Length': '0'} 2023-10-18T20:30:45.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d00de926c257c02c-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.814 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.814 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.815 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.815 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.816 DEBUG:boto:Method: GET 2023-10-18T20:30:45.816 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.816 DEBUG:boto:Data: 2023-10-18T20:30:45.817 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.817 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.818 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.818 DEBUG:boto:Params: {} 2023-10-18T20:30:45.819 DEBUG:boto:Token: None 2023-10-18T20:30:45.819 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.820 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdcf05064d6d9ddb-0065304075-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.823 DEBUG:boto:b'bofrnf-58obj000000000000000000141000falseobj000000000000000000142023-10-18T20:30:31.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.824 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.824 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.825 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.825 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.825 DEBUG:boto:Method: GET 2023-10-18T20:30:45.826 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000014 2023-10-18T20:30:45.826 DEBUG:boto:Data: 2023-10-18T20:30:45.827 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.827 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.827 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.828 DEBUG:boto:Params: {} 2023-10-18T20:30:45.828 DEBUG:boto:Token: None 2023-10-18T20:30:45.829 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.829 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061294ed7b94abcb2-0065304075-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.834 DEBUG:boto:b'bofrnf-58obj000000000000000000141000falseobj000000000000000000142023-10-18T20:30:31.758Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-18T20:30:45.835 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.836 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.836 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.837 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.837 DEBUG:boto:Method: GET 2023-10-18T20:30:45.837 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.838 DEBUG:boto:Data: 2023-10-18T20:30:45.838 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.839 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.839 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.839 DEBUG:boto:Params: {} 2023-10-18T20:30:45.840 DEBUG:boto:Token: None 2023-10-18T20:30:45.840 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.841 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BGcK7GzqCOFeY6sq+x3FvsXOAQo= 2023-10-18T20:30:45.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BGcK7GzqCOFeY6sq+x3FvsXOAQo=', 'Content-Length': '0'} 2023-10-18T20:30:45.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005350929bd0519126-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.844 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.845 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.845 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.846 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.846 DEBUG:boto:Method: GET 2023-10-18T20:30:45.847 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.847 DEBUG:boto:Data: 2023-10-18T20:30:45.847 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.848 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.849 DEBUG:boto:Params: {} 2023-10-18T20:30:45.849 DEBUG:boto:Token: None 2023-10-18T20:30:45.850 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.850 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BGcK7GzqCOFeY6sq+x3FvsXOAQo= 2023-10-18T20:30:45.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BGcK7GzqCOFeY6sq+x3FvsXOAQo=', 'Content-Length': '0'} 2023-10-18T20:30:45.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc6978e68f5e484a-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.853 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.854 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.854 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.855 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.855 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.855 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.856 DEBUG:boto:Data: 2023-10-18T20:30:45.856 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.857 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.857 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.857 DEBUG:boto:Params: {} 2023-10-18T20:30:45.858 DEBUG:boto:Token: None 2023-10-18T20:30:45.858 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/DMxU/I07NcdXzDJ1aWEAn4Z920= 2023-10-18T20:30:45.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/DMxU/I07NcdXzDJ1aWEAn4Z920=', 'Content-Length': '0'} 2023-10-18T20:30:45.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ccb9ecfa520c7b9-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.862 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.862 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.863 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.863 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.864 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.864 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.864 DEBUG:boto:Data: 2023-10-18T20:30:45.865 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.865 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.866 DEBUG:boto:Params: {} 2023-10-18T20:30:45.866 DEBUG:boto:Token: None 2023-10-18T20:30:45.867 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/DMxU/I07NcdXzDJ1aWEAn4Z920= 2023-10-18T20:30:45.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/DMxU/I07NcdXzDJ1aWEAn4Z920=', 'Content-Length': '0'} 2023-10-18T20:30:45.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030f621f19a8519d3-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-18T20:30:45.871 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.871 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.871 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.872 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.872 DEBUG:boto:Method: GET 2023-10-18T20:30:45.873 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.873 DEBUG:boto:Data: 2023-10-18T20:30:45.873 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.874 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.874 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.875 DEBUG:boto:Params: {} 2023-10-18T20:30:45.875 DEBUG:boto:Token: None 2023-10-18T20:30:45.876 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.876 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BGcK7GzqCOFeY6sq+x3FvsXOAQo= 2023-10-18T20:30:45.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BGcK7GzqCOFeY6sq+x3FvsXOAQo=', 'Content-Length': '0'} 2023-10-18T20:30:45.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007407811ae13de210-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.922 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.923 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.923 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.924 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.924 DEBUG:boto:Method: GET 2023-10-18T20:30:45.925 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.925 DEBUG:boto:Data: 2023-10-18T20:30:45.925 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.926 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.926 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.927 DEBUG:boto:Params: {} 2023-10-18T20:30:45.927 DEBUG:boto:Token: None 2023-10-18T20:30:45.928 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015?versionId=null 2023-10-18T20:30:45.928 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BGcK7GzqCOFeY6sq+x3FvsXOAQo= 2023-10-18T20:30:45.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BGcK7GzqCOFeY6sq+x3FvsXOAQo=', 'Content-Length': '0'} 2023-10-18T20:30:45.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017d29179a34fd55a-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.934 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.934 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.934 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:45.935 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:45.935 DEBUG:boto:Method: GET 2023-10-18T20:30:45.936 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:45.936 DEBUG:boto:Data: 2023-10-18T20:30:45.936 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.937 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.938 DEBUG:boto:Params: {} 2023-10-18T20:30:45.938 DEBUG:boto:Token: None 2023-10-18T20:30:45.938 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d7fb9f283138990-0065304075-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.942 DEBUG:boto:b'bofrnf-58obj000000000000000000151000falseobj000000000000000000152023-10-18T20:30:29.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.943 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:45.944 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:45.944 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:45.944 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:45.945 DEBUG:boto:Method: GET 2023-10-18T20:30:45.945 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:45.946 DEBUG:boto:Data: 2023-10-18T20:30:45.946 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.947 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.947 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.947 DEBUG:boto:Params: {} 2023-10-18T20:30:45.948 DEBUG:boto:Token: None 2023-10-18T20:30:45.948 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/ 2023-10-18T20:30:45.949 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE= 2023-10-18T20:30:45.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vEpZ3wdzSfXjXS9gtmXh0haB+YE=', 'Content-Length': '0'} 2023-10-18T20:30:45.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b32a54d373a7849a-0065304075-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.954 DEBUG:boto:b'bofrnf-58obj000000000000000000151000falseobj000000000000000000152023-10-18T20:30:29.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:45.954 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.955 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.955 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.956 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.956 DEBUG:boto:Data: 2023-10-18T20:30:45.957 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.957 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.958 DEBUG:boto:Params: {} 2023-10-18T20:30:45.958 DEBUG:boto:Token: None 2023-10-18T20:30:45.959 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G3yTJ908N+aKcvTQZ3TylDsuS38= 2023-10-18T20:30:45.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G3yTJ908N+aKcvTQZ3TylDsuS38=', 'Content-Length': '0'} 2023-10-18T20:30:45.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c810daf974f05653-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.962 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.963 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.963 DEBUG:boto:Method: HEAD 2023-10-18T20:30:45.963 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.964 DEBUG:boto:Data: 2023-10-18T20:30:45.964 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.965 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.965 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.965 DEBUG:boto:Params: {} 2023-10-18T20:30:45.966 DEBUG:boto:Token: None 2023-10-18T20:30:45.966 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.967 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G3yTJ908N+aKcvTQZ3TylDsuS38= 2023-10-18T20:30:45.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G3yTJ908N+aKcvTQZ3TylDsuS38=', 'Content-Length': '0'} 2023-10-18T20:30:45.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec26aad40c9adcb-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-18T20:30:45.970 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.971 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.971 DEBUG:boto:Method: GET 2023-10-18T20:30:45.971 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.972 DEBUG:boto:Data: 2023-10-18T20:30:45.972 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.973 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:45.973 DEBUG:boto:Port: 8001 2023-10-18T20:30:45.973 DEBUG:boto:Params: {} 2023-10-18T20:30:45.974 DEBUG:boto:Token: None 2023-10-18T20:30:45.974 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.975 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PqZnQxswdhrRzaEwdcixpJpyHhg= 2023-10-18T20:30:45.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PqZnQxswdhrRzaEwdcixpJpyHhg=', 'Content-Length': '0'} 2023-10-18T20:30:45.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073455032ca150639-0065304075-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:45.978 DEBUG:boto:path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.979 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.979 DEBUG:boto:Method: GET 2023-10-18T20:30:45.979 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.980 DEBUG:boto:Data: 2023-10-18T20:30:45.980 DEBUG:boto:Headers: {} 2023-10-18T20:30:45.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:45.981 DEBUG:boto:Port: 8000 2023-10-18T20:30:45.981 DEBUG:boto:Params: {} 2023-10-18T20:30:45.982 DEBUG:boto:Token: None 2023-10-18T20:30:45.982 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:45 GMT /bofrnf-58/obj00000000000000000015 2023-10-18T20:30:45.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PqZnQxswdhrRzaEwdcixpJpyHhg= 2023-10-18T20:30:45.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PqZnQxswdhrRzaEwdcixpJpyHhg=', 'Content-Length': '0'} 2023-10-18T20:30:45.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001de5512a7cd29619-0065304075-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.030 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.031 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.031 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:46.032 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:46.032 DEBUG:boto:Method: GET 2023-10-18T20:30:46.032 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:46.033 DEBUG:boto:Data: 2023-10-18T20:30:46.033 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.034 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.034 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.035 DEBUG:boto:Params: {} 2023-10-18T20:30:46.035 DEBUG:boto:Token: None 2023-10-18T20:30:46.035 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d388a94a94257880-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.040 DEBUG:boto:b'bofrnf-58obj000000000000000000151000falseobj000000000000000000152023-10-18T20:30:29.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.040 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.041 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.041 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:46.041 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:46.042 DEBUG:boto:Method: GET 2023-10-18T20:30:46.042 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000015 2023-10-18T20:30:46.043 DEBUG:boto:Data: 2023-10-18T20:30:46.043 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.044 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.044 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.044 DEBUG:boto:Params: {} 2023-10-18T20:30:46.045 DEBUG:boto:Token: None 2023-10-18T20:30:46.045 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.046 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e497d1e3c44dd2df-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.051 DEBUG:boto:b'bofrnf-58obj000000000000000000151000falseobj000000000000000000152023-10-18T20:30:29.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-18T20:30:46.052 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.052 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.053 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.053 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.054 DEBUG:boto:Method: GET 2023-10-18T20:30:46.054 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.055 DEBUG:boto:Data: 2023-10-18T20:30:46.055 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.055 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.056 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.056 DEBUG:boto:Params: {} 2023-10-18T20:30:46.057 DEBUG:boto:Token: None 2023-10-18T20:30:46.057 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.058 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OIYgJIgu34UbW4UDfI7ywrFt/Mc= 2023-10-18T20:30:46.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OIYgJIgu34UbW4UDfI7ywrFt/Mc=', 'Content-Length': '0'} 2023-10-18T20:30:46.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000934a7fd132f389af-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.061 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.062 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.062 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.063 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.063 DEBUG:boto:Method: GET 2023-10-18T20:30:46.064 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.064 DEBUG:boto:Data: 2023-10-18T20:30:46.064 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.065 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.065 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.066 DEBUG:boto:Params: {} 2023-10-18T20:30:46.066 DEBUG:boto:Token: None 2023-10-18T20:30:46.067 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.067 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OIYgJIgu34UbW4UDfI7ywrFt/Mc= 2023-10-18T20:30:46.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OIYgJIgu34UbW4UDfI7ywrFt/Mc=', 'Content-Length': '0'} 2023-10-18T20:30:46.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b110fc9d603060d7-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.070 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.070 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.071 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.071 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.072 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.072 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.072 DEBUG:boto:Data: 2023-10-18T20:30:46.073 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.073 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.074 DEBUG:boto:Params: {} 2023-10-18T20:30:46.074 DEBUG:boto:Token: None 2023-10-18T20:30:46.075 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.075 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Y7zWIAxfvlff8T1UOhKNdbxh3kA= 2023-10-18T20:30:46.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Y7zWIAxfvlff8T1UOhKNdbxh3kA=', 'Content-Length': '0'} 2023-10-18T20:30:46.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000328e5d30f6d7ba51-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.078 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.079 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.079 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.079 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.080 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.080 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.080 DEBUG:boto:Data: 2023-10-18T20:30:46.081 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.081 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.082 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.082 DEBUG:boto:Params: {} 2023-10-18T20:30:46.082 DEBUG:boto:Token: None 2023-10-18T20:30:46.083 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.083 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Y7zWIAxfvlff8T1UOhKNdbxh3kA= 2023-10-18T20:30:46.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Y7zWIAxfvlff8T1UOhKNdbxh3kA=', 'Content-Length': '0'} 2023-10-18T20:30:46.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113b323b45d14b06-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-18T20:30:46.087 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.087 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.087 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.088 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.088 DEBUG:boto:Method: GET 2023-10-18T20:30:46.089 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.089 DEBUG:boto:Data: 2023-10-18T20:30:46.089 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.090 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.090 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.091 DEBUG:boto:Params: {} 2023-10-18T20:30:46.091 DEBUG:boto:Token: None 2023-10-18T20:30:46.091 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.092 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OIYgJIgu34UbW4UDfI7ywrFt/Mc= 2023-10-18T20:30:46.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OIYgJIgu34UbW4UDfI7ywrFt/Mc=', 'Content-Length': '0'} 2023-10-18T20:30:46.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001702eea0a6f5b6cb-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.095 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.095 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.096 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.096 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.097 DEBUG:boto:Method: GET 2023-10-18T20:30:46.097 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.097 DEBUG:boto:Data: 2023-10-18T20:30:46.098 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.099 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.099 DEBUG:boto:Params: {} 2023-10-18T20:30:46.100 DEBUG:boto:Token: None 2023-10-18T20:30:46.100 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016?versionId=null 2023-10-18T20:30:46.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OIYgJIgu34UbW4UDfI7ywrFt/Mc= 2023-10-18T20:30:46.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OIYgJIgu34UbW4UDfI7ywrFt/Mc=', 'Content-Length': '0'} 2023-10-18T20:30:46.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f4bde08bec91ca-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.104 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.104 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.104 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.105 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.105 DEBUG:boto:Method: GET 2023-10-18T20:30:46.106 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.106 DEBUG:boto:Data: 2023-10-18T20:30:46.106 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.107 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.107 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.108 DEBUG:boto:Params: {} 2023-10-18T20:30:46.108 DEBUG:boto:Token: None 2023-10-18T20:30:46.109 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.109 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023266b25d1efd0b0-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.154 DEBUG:boto:b'bofrnf-58obj000000000000000000161000falseobj000000000000000000162023-10-18T20:30:32.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.155 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.156 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.156 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.156 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.157 DEBUG:boto:Method: GET 2023-10-18T20:30:46.157 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.158 DEBUG:boto:Data: 2023-10-18T20:30:46.158 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.159 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.159 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.159 DEBUG:boto:Params: {} 2023-10-18T20:30:46.160 DEBUG:boto:Token: None 2023-10-18T20:30:46.160 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082db13d96e7e197d-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.166 DEBUG:boto:b'bofrnf-58obj000000000000000000161000falseobj000000000000000000162023-10-18T20:30:32.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.167 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.167 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.168 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.168 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.169 DEBUG:boto:Data: 2023-10-18T20:30:46.169 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.170 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.170 DEBUG:boto:Params: {} 2023-10-18T20:30:46.171 DEBUG:boto:Token: None 2023-10-18T20:30:46.171 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kiDwY5/GkKvR34Cx/bJXj/Mk8h4= 2023-10-18T20:30:46.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kiDwY5/GkKvR34Cx/bJXj/Mk8h4=', 'Content-Length': '0'} 2023-10-18T20:30:46.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d6bd1e62d95068d-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.175 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.175 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.175 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.176 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.176 DEBUG:boto:Data: 2023-10-18T20:30:46.177 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.177 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.177 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.178 DEBUG:boto:Params: {} 2023-10-18T20:30:46.178 DEBUG:boto:Token: None 2023-10-18T20:30:46.179 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.179 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kiDwY5/GkKvR34Cx/bJXj/Mk8h4= 2023-10-18T20:30:46.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kiDwY5/GkKvR34Cx/bJXj/Mk8h4=', 'Content-Length': '0'} 2023-10-18T20:30:46.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c2de0d1df4f33ba-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-18T20:30:46.183 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.183 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.184 DEBUG:boto:Method: GET 2023-10-18T20:30:46.184 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.184 DEBUG:boto:Data: 2023-10-18T20:30:46.185 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.186 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.186 DEBUG:boto:Params: {} 2023-10-18T20:30:46.186 DEBUG:boto:Token: None 2023-10-18T20:30:46.187 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.187 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e7XZ/yRcIILHLMjP0osBbIgC36w= 2023-10-18T20:30:46.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e7XZ/yRcIILHLMjP0osBbIgC36w=', 'Content-Length': '0'} 2023-10-18T20:30:46.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfebbf0e1cec77be-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.191 DEBUG:boto:path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.191 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.191 DEBUG:boto:Method: GET 2023-10-18T20:30:46.192 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.192 DEBUG:boto:Data: 2023-10-18T20:30:46.193 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.194 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.194 DEBUG:boto:Params: {} 2023-10-18T20:30:46.194 DEBUG:boto:Token: None 2023-10-18T20:30:46.195 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000016 2023-10-18T20:30:46.195 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e7XZ/yRcIILHLMjP0osBbIgC36w= 2023-10-18T20:30:46.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e7XZ/yRcIILHLMjP0osBbIgC36w=', 'Content-Length': '0'} 2023-10-18T20:30:46.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18ea85bf366d3f9-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.238 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.239 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.239 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.240 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.242 DEBUG:boto:Method: GET 2023-10-18T20:30:46.242 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.243 DEBUG:boto:Data: 2023-10-18T20:30:46.243 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.244 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.244 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.244 DEBUG:boto:Params: {} 2023-10-18T20:30:46.245 DEBUG:boto:Token: None 2023-10-18T20:30:46.245 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.246 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014d103c47958fb4d-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.250 DEBUG:boto:b'bofrnf-58obj000000000000000000161000falseobj000000000000000000162023-10-18T20:30:32.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.250 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.251 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.251 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.251 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.252 DEBUG:boto:Method: GET 2023-10-18T20:30:46.252 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000016 2023-10-18T20:30:46.253 DEBUG:boto:Data: 2023-10-18T20:30:46.253 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.254 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.254 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.254 DEBUG:boto:Params: {} 2023-10-18T20:30:46.255 DEBUG:boto:Token: None 2023-10-18T20:30:46.255 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.256 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f68f7e4ab31e701-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.261 DEBUG:boto:b'bofrnf-58obj000000000000000000161000falseobj000000000000000000162023-10-18T20:30:32.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-18T20:30:46.262 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.263 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.263 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.263 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.264 DEBUG:boto:Method: GET 2023-10-18T20:30:46.264 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.265 DEBUG:boto:Data: 2023-10-18T20:30:46.265 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.265 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.266 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.266 DEBUG:boto:Params: {} 2023-10-18T20:30:46.267 DEBUG:boto:Token: None 2023-10-18T20:30:46.267 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.267 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EomUOkKIAfc83LO+1yH4KBMYSA4= 2023-10-18T20:30:46.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EomUOkKIAfc83LO+1yH4KBMYSA4=', 'Content-Length': '0'} 2023-10-18T20:30:46.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab4dff396e28f006-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.271 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.271 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.272 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.272 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.272 DEBUG:boto:Method: GET 2023-10-18T20:30:46.273 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.273 DEBUG:boto:Data: 2023-10-18T20:30:46.274 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.274 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.274 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.275 DEBUG:boto:Params: {} 2023-10-18T20:30:46.275 DEBUG:boto:Token: None 2023-10-18T20:30:46.276 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EomUOkKIAfc83LO+1yH4KBMYSA4= 2023-10-18T20:30:46.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EomUOkKIAfc83LO+1yH4KBMYSA4=', 'Content-Length': '0'} 2023-10-18T20:30:46.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e4965f25066876-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.279 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.280 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.280 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.280 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.281 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.281 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.281 DEBUG:boto:Data: 2023-10-18T20:30:46.282 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.283 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.283 DEBUG:boto:Params: {} 2023-10-18T20:30:46.284 DEBUG:boto:Token: None 2023-10-18T20:30:46.284 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QSuER+Dzk7jTN2BbejXmdpvxXhg= 2023-10-18T20:30:46.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QSuER+Dzk7jTN2BbejXmdpvxXhg=', 'Content-Length': '0'} 2023-10-18T20:30:46.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000375805e80a782920-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.287 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.288 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.288 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.288 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.289 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.289 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.289 DEBUG:boto:Data: 2023-10-18T20:30:46.290 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.290 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.291 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.291 DEBUG:boto:Params: {} 2023-10-18T20:30:46.291 DEBUG:boto:Token: None 2023-10-18T20:30:46.292 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.292 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QSuER+Dzk7jTN2BbejXmdpvxXhg= 2023-10-18T20:30:46.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QSuER+Dzk7jTN2BbejXmdpvxXhg=', 'Content-Length': '0'} 2023-10-18T20:30:46.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e0f0d7fe444d4ab-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-18T20:30:46.296 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.296 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.297 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.297 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.298 DEBUG:boto:Method: GET 2023-10-18T20:30:46.298 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.299 DEBUG:boto:Data: 2023-10-18T20:30:46.299 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.300 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.300 DEBUG:boto:Params: {} 2023-10-18T20:30:46.301 DEBUG:boto:Token: None 2023-10-18T20:30:46.301 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.302 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EomUOkKIAfc83LO+1yH4KBMYSA4= 2023-10-18T20:30:46.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EomUOkKIAfc83LO+1yH4KBMYSA4=', 'Content-Length': '0'} 2023-10-18T20:30:46.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1f99488692a30d3-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.350 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.351 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.351 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.352 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.352 DEBUG:boto:Method: GET 2023-10-18T20:30:46.353 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.353 DEBUG:boto:Data: 2023-10-18T20:30:46.353 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.354 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.354 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.355 DEBUG:boto:Params: {} 2023-10-18T20:30:46.355 DEBUG:boto:Token: None 2023-10-18T20:30:46.356 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017?versionId=null 2023-10-18T20:30:46.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EomUOkKIAfc83LO+1yH4KBMYSA4= 2023-10-18T20:30:46.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EomUOkKIAfc83LO+1yH4KBMYSA4=', 'Content-Length': '0'} 2023-10-18T20:30:46.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f4c56099c19653d-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.360 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.360 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.361 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.361 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.362 DEBUG:boto:Method: GET 2023-10-18T20:30:46.362 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.362 DEBUG:boto:Data: 2023-10-18T20:30:46.363 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.363 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.364 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.364 DEBUG:boto:Params: {} 2023-10-18T20:30:46.365 DEBUG:boto:Token: None 2023-10-18T20:30:46.365 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.365 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c53731174016506d-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.369 DEBUG:boto:b'bofrnf-58obj000000000000000000171000falseobj000000000000000000172023-10-18T20:30:31.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.370 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.370 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.371 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.371 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.371 DEBUG:boto:Method: GET 2023-10-18T20:30:46.372 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.372 DEBUG:boto:Data: 2023-10-18T20:30:46.373 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.373 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.374 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.374 DEBUG:boto:Params: {} 2023-10-18T20:30:46.374 DEBUG:boto:Token: None 2023-10-18T20:30:46.375 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.375 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000420dc17f21ff360f-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.381 DEBUG:boto:b'bofrnf-58obj000000000000000000171000falseobj000000000000000000172023-10-18T20:30:31.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.381 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.382 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.382 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.383 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.383 DEBUG:boto:Data: 2023-10-18T20:30:46.383 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.384 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.384 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.385 DEBUG:boto:Params: {} 2023-10-18T20:30:46.385 DEBUG:boto:Token: None 2023-10-18T20:30:46.386 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.386 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HqY8Oc6DuMR0UWx5oJUw4pmLvpA= 2023-10-18T20:30:46.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HqY8Oc6DuMR0UWx5oJUw4pmLvpA=', 'Content-Length': '0'} 2023-10-18T20:30:46.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f0fc2812b16efdb-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.389 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.390 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.390 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.391 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.391 DEBUG:boto:Data: 2023-10-18T20:30:46.391 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.392 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.392 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.392 DEBUG:boto:Params: {} 2023-10-18T20:30:46.393 DEBUG:boto:Token: None 2023-10-18T20:30:46.393 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.394 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HqY8Oc6DuMR0UWx5oJUw4pmLvpA= 2023-10-18T20:30:46.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HqY8Oc6DuMR0UWx5oJUw4pmLvpA=', 'Content-Length': '0'} 2023-10-18T20:30:46.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6be256c802115ae-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-18T20:30:46.397 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.397 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.398 DEBUG:boto:Method: GET 2023-10-18T20:30:46.398 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.399 DEBUG:boto:Data: 2023-10-18T20:30:46.399 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.400 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.400 DEBUG:boto:Params: {} 2023-10-18T20:30:46.401 DEBUG:boto:Token: None 2023-10-18T20:30:46.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TNdvZDaDBzn2X/8vdfH526M0B38= 2023-10-18T20:30:46.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TNdvZDaDBzn2X/8vdfH526M0B38=', 'Content-Length': '0'} 2023-10-18T20:30:46.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000765a3f9ee4a6829a-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.405 DEBUG:boto:path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.405 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.406 DEBUG:boto:Method: GET 2023-10-18T20:30:46.406 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.406 DEBUG:boto:Data: 2023-10-18T20:30:46.407 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.407 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.408 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.408 DEBUG:boto:Params: {} 2023-10-18T20:30:46.409 DEBUG:boto:Token: None 2023-10-18T20:30:46.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000017 2023-10-18T20:30:46.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TNdvZDaDBzn2X/8vdfH526M0B38= 2023-10-18T20:30:46.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TNdvZDaDBzn2X/8vdfH526M0B38=', 'Content-Length': '0'} 2023-10-18T20:30:46.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62b4df4b16b5dcf-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.413 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.413 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.414 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.414 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.415 DEBUG:boto:Method: GET 2023-10-18T20:30:46.415 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.415 DEBUG:boto:Data: 2023-10-18T20:30:46.416 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.416 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.417 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.417 DEBUG:boto:Params: {} 2023-10-18T20:30:46.417 DEBUG:boto:Token: None 2023-10-18T20:30:46.418 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005660dc2d08ac0b34-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.422 DEBUG:boto:b'bofrnf-58obj000000000000000000171000falseobj000000000000000000172023-10-18T20:30:31.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.423 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.423 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.424 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.424 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.424 DEBUG:boto:Method: GET 2023-10-18T20:30:46.425 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000017 2023-10-18T20:30:46.425 DEBUG:boto:Data: 2023-10-18T20:30:46.426 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.427 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.427 DEBUG:boto:Params: {} 2023-10-18T20:30:46.427 DEBUG:boto:Token: None 2023-10-18T20:30:46.428 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000810aa51ebf94a638-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.433 DEBUG:boto:b'bofrnf-58obj000000000000000000171000falseobj000000000000000000172023-10-18T20:30:31.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-18T20:30:46.434 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.435 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.435 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.436 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.436 DEBUG:boto:Method: GET 2023-10-18T20:30:46.437 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.437 DEBUG:boto:Data: 2023-10-18T20:30:46.437 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.438 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.438 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.439 DEBUG:boto:Params: {} 2023-10-18T20:30:46.439 DEBUG:boto:Token: None 2023-10-18T20:30:46.440 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obl4XIq7G/TNKUkP59erUuYklaU= 2023-10-18T20:30:46.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obl4XIq7G/TNKUkP59erUuYklaU=', 'Content-Length': '0'} 2023-10-18T20:30:46.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ec4acc4a9433191-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.443 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.444 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.444 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.445 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.445 DEBUG:boto:Method: GET 2023-10-18T20:30:46.446 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.446 DEBUG:boto:Data: 2023-10-18T20:30:46.446 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.447 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.447 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.448 DEBUG:boto:Params: {} 2023-10-18T20:30:46.448 DEBUG:boto:Token: None 2023-10-18T20:30:46.449 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.449 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obl4XIq7G/TNKUkP59erUuYklaU= 2023-10-18T20:30:46.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obl4XIq7G/TNKUkP59erUuYklaU=', 'Content-Length': '0'} 2023-10-18T20:30:46.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f23ec03da286d4e4-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.495 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.495 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.496 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.496 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.497 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.497 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.497 DEBUG:boto:Data: 2023-10-18T20:30:46.498 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.499 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.499 DEBUG:boto:Params: {} 2023-10-18T20:30:46.500 DEBUG:boto:Token: None 2023-10-18T20:30:46.500 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.501 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FmNNSB4t6FheqXuSES3TfEStARc= 2023-10-18T20:30:46.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FmNNSB4t6FheqXuSES3TfEStARc=', 'Content-Length': '0'} 2023-10-18T20:30:46.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c59689f12635257-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.504 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.504 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.505 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.505 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.505 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.506 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.506 DEBUG:boto:Data: 2023-10-18T20:30:46.507 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.507 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.508 DEBUG:boto:Params: {} 2023-10-18T20:30:46.508 DEBUG:boto:Token: None 2023-10-18T20:30:46.509 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FmNNSB4t6FheqXuSES3TfEStARc= 2023-10-18T20:30:46.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FmNNSB4t6FheqXuSES3TfEStARc=', 'Content-Length': '0'} 2023-10-18T20:30:46.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b2fd22dd466d06-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-18T20:30:46.513 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.513 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.514 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.514 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.514 DEBUG:boto:Method: GET 2023-10-18T20:30:46.515 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.515 DEBUG:boto:Data: 2023-10-18T20:30:46.516 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.517 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.517 DEBUG:boto:Params: {} 2023-10-18T20:30:46.517 DEBUG:boto:Token: None 2023-10-18T20:30:46.518 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.518 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obl4XIq7G/TNKUkP59erUuYklaU= 2023-10-18T20:30:46.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obl4XIq7G/TNKUkP59erUuYklaU=', 'Content-Length': '0'} 2023-10-18T20:30:46.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d92a31e7608f25-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.522 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.522 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.523 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.523 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.524 DEBUG:boto:Method: GET 2023-10-18T20:30:46.524 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.524 DEBUG:boto:Data: 2023-10-18T20:30:46.525 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.526 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.526 DEBUG:boto:Params: {} 2023-10-18T20:30:46.526 DEBUG:boto:Token: None 2023-10-18T20:30:46.527 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018?versionId=null 2023-10-18T20:30:46.527 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:obl4XIq7G/TNKUkP59erUuYklaU= 2023-10-18T20:30:46.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:obl4XIq7G/TNKUkP59erUuYklaU=', 'Content-Length': '0'} 2023-10-18T20:30:46.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a48c1979418afdf2-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.531 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.531 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.531 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.532 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.532 DEBUG:boto:Method: GET 2023-10-18T20:30:46.532 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.533 DEBUG:boto:Data: 2023-10-18T20:30:46.533 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.534 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.534 DEBUG:boto:Params: {} 2023-10-18T20:30:46.535 DEBUG:boto:Token: None 2023-10-18T20:30:46.535 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083060554f47a5b82-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.540 DEBUG:boto:b'bofrnf-58obj000000000000000000181000falseobj000000000000000000182023-10-18T20:30:32.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.540 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.541 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.541 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.541 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.542 DEBUG:boto:Method: GET 2023-10-18T20:30:46.542 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.543 DEBUG:boto:Data: 2023-10-18T20:30:46.543 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.544 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.544 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.545 DEBUG:boto:Params: {} 2023-10-18T20:30:46.545 DEBUG:boto:Token: None 2023-10-18T20:30:46.546 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.546 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000172fcbf0292d5e74-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.552 DEBUG:boto:b'bofrnf-58obj000000000000000000181000falseobj000000000000000000182023-10-18T20:30:32.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.552 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.553 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.553 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.553 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.554 DEBUG:boto:Data: 2023-10-18T20:30:46.554 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.555 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.555 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.555 DEBUG:boto:Params: {} 2023-10-18T20:30:46.556 DEBUG:boto:Token: None 2023-10-18T20:30:46.556 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.556 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TBwMyVDpKFPUwdV09WYJKbaDtsU= 2023-10-18T20:30:46.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TBwMyVDpKFPUwdV09WYJKbaDtsU=', 'Content-Length': '0'} 2023-10-18T20:30:46.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e0633aada4f0c8-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.559 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.560 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.560 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.561 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.561 DEBUG:boto:Data: 2023-10-18T20:30:46.561 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.562 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.562 DEBUG:boto:Params: {} 2023-10-18T20:30:46.563 DEBUG:boto:Token: None 2023-10-18T20:30:46.563 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TBwMyVDpKFPUwdV09WYJKbaDtsU= 2023-10-18T20:30:46.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TBwMyVDpKFPUwdV09WYJKbaDtsU=', 'Content-Length': '0'} 2023-10-18T20:30:46.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7dd645bf078d40-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-18T20:30:46.567 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.568 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.568 DEBUG:boto:Method: GET 2023-10-18T20:30:46.568 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.569 DEBUG:boto:Data: 2023-10-18T20:30:46.569 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.570 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.570 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.570 DEBUG:boto:Params: {} 2023-10-18T20:30:46.571 DEBUG:boto:Token: None 2023-10-18T20:30:46.571 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.572 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DHq6OAYiPhpRBgpHhJij49nFYvg= 2023-10-18T20:30:46.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DHq6OAYiPhpRBgpHhJij49nFYvg=', 'Content-Length': '0'} 2023-10-18T20:30:46.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe5e5dd5d173a15-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.618 DEBUG:boto:path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.619 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.619 DEBUG:boto:Method: GET 2023-10-18T20:30:46.620 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.620 DEBUG:boto:Data: 2023-10-18T20:30:46.620 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.621 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.621 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.622 DEBUG:boto:Params: {} 2023-10-18T20:30:46.622 DEBUG:boto:Token: None 2023-10-18T20:30:46.623 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000018 2023-10-18T20:30:46.623 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DHq6OAYiPhpRBgpHhJij49nFYvg= 2023-10-18T20:30:46.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DHq6OAYiPhpRBgpHhJij49nFYvg=', 'Content-Length': '0'} 2023-10-18T20:30:46.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003004938838d588f-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.626 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.627 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.627 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.628 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.628 DEBUG:boto:Method: GET 2023-10-18T20:30:46.628 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.629 DEBUG:boto:Data: 2023-10-18T20:30:46.629 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.630 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.631 DEBUG:boto:Params: {} 2023-10-18T20:30:46.631 DEBUG:boto:Token: None 2023-10-18T20:30:46.631 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.632 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007c35e79b7d1c53b-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.636 DEBUG:boto:b'bofrnf-58obj000000000000000000181000falseobj000000000000000000182023-10-18T20:30:32.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.636 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.637 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.637 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.638 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.638 DEBUG:boto:Method: GET 2023-10-18T20:30:46.638 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000018 2023-10-18T20:30:46.639 DEBUG:boto:Data: 2023-10-18T20:30:46.639 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.640 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.640 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.640 DEBUG:boto:Params: {} 2023-10-18T20:30:46.641 DEBUG:boto:Token: None 2023-10-18T20:30:46.641 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.642 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa79c61eb7ac30a-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.647 DEBUG:boto:b'bofrnf-58obj000000000000000000181000falseobj000000000000000000182023-10-18T20:30:32.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-18T20:30:46.648 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.648 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.648 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.649 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.649 DEBUG:boto:Method: GET 2023-10-18T20:30:46.650 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.650 DEBUG:boto:Data: 2023-10-18T20:30:46.650 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.651 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.652 DEBUG:boto:Params: {} 2023-10-18T20:30:46.652 DEBUG:boto:Token: None 2023-10-18T20:30:46.652 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9HVYzt8SeWCFB2ssxFFQhRKj6Iw= 2023-10-18T20:30:46.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9HVYzt8SeWCFB2ssxFFQhRKj6Iw=', 'Content-Length': '0'} 2023-10-18T20:30:46.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a201d05cc228439a-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.656 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.657 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.657 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.658 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.658 DEBUG:boto:Method: GET 2023-10-18T20:30:46.658 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.659 DEBUG:boto:Data: 2023-10-18T20:30:46.659 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.660 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.660 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.660 DEBUG:boto:Params: {} 2023-10-18T20:30:46.661 DEBUG:boto:Token: None 2023-10-18T20:30:46.661 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9HVYzt8SeWCFB2ssxFFQhRKj6Iw= 2023-10-18T20:30:46.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9HVYzt8SeWCFB2ssxFFQhRKj6Iw=', 'Content-Length': '0'} 2023-10-18T20:30:46.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f68a8aac290f65b-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.710 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.711 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.711 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.712 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.712 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.712 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.713 DEBUG:boto:Data: 2023-10-18T20:30:46.713 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.714 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.714 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.714 DEBUG:boto:Params: {} 2023-10-18T20:30:46.715 DEBUG:boto:Token: None 2023-10-18T20:30:46.715 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.716 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a/MHDG+iNkmkf13rDbWiJaxc9EQ= 2023-10-18T20:30:46.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a/MHDG+iNkmkf13rDbWiJaxc9EQ=', 'Content-Length': '0'} 2023-10-18T20:30:46.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a0620a068b349d-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.719 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.719 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.720 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.720 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.720 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.721 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.721 DEBUG:boto:Data: 2023-10-18T20:30:46.721 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.722 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.722 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.723 DEBUG:boto:Params: {} 2023-10-18T20:30:46.723 DEBUG:boto:Token: None 2023-10-18T20:30:46.724 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.724 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a/MHDG+iNkmkf13rDbWiJaxc9EQ= 2023-10-18T20:30:46.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a/MHDG+iNkmkf13rDbWiJaxc9EQ=', 'Content-Length': '0'} 2023-10-18T20:30:46.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d569105073c3ca8-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-18T20:30:46.728 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.728 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.728 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.729 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.729 DEBUG:boto:Method: GET 2023-10-18T20:30:46.730 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.730 DEBUG:boto:Data: 2023-10-18T20:30:46.731 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.731 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.731 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.732 DEBUG:boto:Params: {} 2023-10-18T20:30:46.732 DEBUG:boto:Token: None 2023-10-18T20:30:46.733 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.733 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9HVYzt8SeWCFB2ssxFFQhRKj6Iw= 2023-10-18T20:30:46.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9HVYzt8SeWCFB2ssxFFQhRKj6Iw=', 'Content-Length': '0'} 2023-10-18T20:30:46.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033552ea2ed7d9c11-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.737 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.737 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.737 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.738 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.738 DEBUG:boto:Method: GET 2023-10-18T20:30:46.739 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.739 DEBUG:boto:Data: 2023-10-18T20:30:46.739 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.740 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.740 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.741 DEBUG:boto:Params: {} 2023-10-18T20:30:46.741 DEBUG:boto:Token: None 2023-10-18T20:30:46.741 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019?versionId=null 2023-10-18T20:30:46.742 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9HVYzt8SeWCFB2ssxFFQhRKj6Iw= 2023-10-18T20:30:46.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9HVYzt8SeWCFB2ssxFFQhRKj6Iw=', 'Content-Length': '0'} 2023-10-18T20:30:46.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e63de921bc35467c-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.745 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.745 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.746 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.746 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.747 DEBUG:boto:Method: GET 2023-10-18T20:30:46.747 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.748 DEBUG:boto:Data: 2023-10-18T20:30:46.748 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.749 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.749 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.749 DEBUG:boto:Params: {} 2023-10-18T20:30:46.750 DEBUG:boto:Token: None 2023-10-18T20:30:46.750 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.751 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093008d054654f66b-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.754 DEBUG:boto:b'bofrnf-58obj000000000000000000191000falseobj000000000000000000192023-10-18T20:30:32.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.755 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.756 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.756 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.756 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.757 DEBUG:boto:Method: GET 2023-10-18T20:30:46.757 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.758 DEBUG:boto:Data: 2023-10-18T20:30:46.758 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.759 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.759 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.760 DEBUG:boto:Params: {} 2023-10-18T20:30:46.760 DEBUG:boto:Token: None 2023-10-18T20:30:46.761 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.761 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9c6d254311d79cc-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.766 DEBUG:boto:b'bofrnf-58obj000000000000000000191000falseobj000000000000000000192023-10-18T20:30:32.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.767 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.768 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.768 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.768 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.769 DEBUG:boto:Data: 2023-10-18T20:30:46.769 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.770 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.770 DEBUG:boto:Params: {} 2023-10-18T20:30:46.771 DEBUG:boto:Token: None 2023-10-18T20:30:46.771 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6+z73XEhN56hBnAGiwxIUTcjDTk= 2023-10-18T20:30:46.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6+z73XEhN56hBnAGiwxIUTcjDTk=', 'Content-Length': '0'} 2023-10-18T20:30:46.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd4e284c574ef43-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.775 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.775 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.776 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.776 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.777 DEBUG:boto:Data: 2023-10-18T20:30:46.777 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.777 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.778 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.778 DEBUG:boto:Params: {} 2023-10-18T20:30:46.779 DEBUG:boto:Token: None 2023-10-18T20:30:46.779 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6+z73XEhN56hBnAGiwxIUTcjDTk= 2023-10-18T20:30:46.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6+z73XEhN56hBnAGiwxIUTcjDTk=', 'Content-Length': '0'} 2023-10-18T20:30:46.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d00c1dd9d6ca94e8-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-18T20:30:46.783 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.783 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.784 DEBUG:boto:Method: GET 2023-10-18T20:30:46.784 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.785 DEBUG:boto:Data: 2023-10-18T20:30:46.785 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.786 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.786 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.786 DEBUG:boto:Params: {} 2023-10-18T20:30:46.787 DEBUG:boto:Token: None 2023-10-18T20:30:46.787 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.788 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PDVB8t4tODdeSHm4V50W8oqcY+A= 2023-10-18T20:30:46.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PDVB8t4tODdeSHm4V50W8oqcY+A=', 'Content-Length': '0'} 2023-10-18T20:30:46.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da81eb57a4ff800f-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.791 DEBUG:boto:path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.792 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.792 DEBUG:boto:Method: GET 2023-10-18T20:30:46.792 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.793 DEBUG:boto:Data: 2023-10-18T20:30:46.793 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.794 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.794 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.795 DEBUG:boto:Params: {} 2023-10-18T20:30:46.795 DEBUG:boto:Token: None 2023-10-18T20:30:46.796 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000019 2023-10-18T20:30:46.796 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PDVB8t4tODdeSHm4V50W8oqcY+A= 2023-10-18T20:30:46.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PDVB8t4tODdeSHm4V50W8oqcY+A=', 'Content-Length': '0'} 2023-10-18T20:30:46.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000156b4fda1ae1faa2-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.799 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.800 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.800 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.801 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.801 DEBUG:boto:Method: GET 2023-10-18T20:30:46.802 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.802 DEBUG:boto:Data: 2023-10-18T20:30:46.803 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.803 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.803 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.804 DEBUG:boto:Params: {} 2023-10-18T20:30:46.804 DEBUG:boto:Token: None 2023-10-18T20:30:46.805 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.805 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000925a741d1792c90b-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.850 DEBUG:boto:b'bofrnf-58obj000000000000000000191000falseobj000000000000000000192023-10-18T20:30:32.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.851 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.852 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.852 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.853 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.853 DEBUG:boto:Method: GET 2023-10-18T20:30:46.854 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000019 2023-10-18T20:30:46.854 DEBUG:boto:Data: 2023-10-18T20:30:46.855 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.855 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.856 DEBUG:boto:Params: {} 2023-10-18T20:30:46.856 DEBUG:boto:Token: None 2023-10-18T20:30:46.857 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.858 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ccae99e0834b709-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.864 DEBUG:boto:b'bofrnf-58obj000000000000000000191000falseobj000000000000000000192023-10-18T20:30:32.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-18T20:30:46.865 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.865 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.865 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.866 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.866 DEBUG:boto:Method: GET 2023-10-18T20:30:46.867 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.867 DEBUG:boto:Data: 2023-10-18T20:30:46.868 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.869 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.869 DEBUG:boto:Params: {} 2023-10-18T20:30:46.869 DEBUG:boto:Token: None 2023-10-18T20:30:46.870 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.870 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zSwpNr3i48u2uFQZPiOMLzFLwKM= 2023-10-18T20:30:46.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zSwpNr3i48u2uFQZPiOMLzFLwKM=', 'Content-Length': '0'} 2023-10-18T20:30:46.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e94ee06ed11946-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.874 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.874 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.875 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.875 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.876 DEBUG:boto:Method: GET 2023-10-18T20:30:46.876 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.877 DEBUG:boto:Data: 2023-10-18T20:30:46.877 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.878 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.878 DEBUG:boto:Params: {} 2023-10-18T20:30:46.879 DEBUG:boto:Token: None 2023-10-18T20:30:46.879 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zSwpNr3i48u2uFQZPiOMLzFLwKM= 2023-10-18T20:30:46.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zSwpNr3i48u2uFQZPiOMLzFLwKM=', 'Content-Length': '0'} 2023-10-18T20:30:46.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001167beb99e96619f-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.926 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.927 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.927 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.928 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.928 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.928 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.929 DEBUG:boto:Data: 2023-10-18T20:30:46.929 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.930 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.930 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.930 DEBUG:boto:Params: {} 2023-10-18T20:30:46.931 DEBUG:boto:Token: None 2023-10-18T20:30:46.931 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.932 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:py3kTtchpGOqwfy8P6la/xFOGo8= 2023-10-18T20:30:46.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:py3kTtchpGOqwfy8P6la/xFOGo8=', 'Content-Length': '0'} 2023-10-18T20:30:46.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff69de3a81560488-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.935 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.935 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.936 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.936 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.937 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.937 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.938 DEBUG:boto:Data: 2023-10-18T20:30:46.938 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.938 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.939 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.939 DEBUG:boto:Params: {} 2023-10-18T20:30:46.940 DEBUG:boto:Token: None 2023-10-18T20:30:46.940 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.941 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:py3kTtchpGOqwfy8P6la/xFOGo8= 2023-10-18T20:30:46.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:py3kTtchpGOqwfy8P6la/xFOGo8=', 'Content-Length': '0'} 2023-10-18T20:30:46.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d334d2891505665-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-18T20:30:46.944 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.945 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.945 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.945 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.946 DEBUG:boto:Method: GET 2023-10-18T20:30:46.946 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.947 DEBUG:boto:Data: 2023-10-18T20:30:46.947 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.947 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.948 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.948 DEBUG:boto:Params: {} 2023-10-18T20:30:46.949 DEBUG:boto:Token: None 2023-10-18T20:30:46.949 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.950 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zSwpNr3i48u2uFQZPiOMLzFLwKM= 2023-10-18T20:30:46.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zSwpNr3i48u2uFQZPiOMLzFLwKM=', 'Content-Length': '0'} 2023-10-18T20:30:46.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022636f48d2b0eb06-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.953 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.953 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.954 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.954 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.955 DEBUG:boto:Method: GET 2023-10-18T20:30:46.955 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.955 DEBUG:boto:Data: 2023-10-18T20:30:46.957 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.958 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.958 DEBUG:boto:Params: {} 2023-10-18T20:30:46.959 DEBUG:boto:Token: None 2023-10-18T20:30:46.959 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000020?versionId=null 2023-10-18T20:30:46.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zSwpNr3i48u2uFQZPiOMLzFLwKM= 2023-10-18T20:30:46.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zSwpNr3i48u2uFQZPiOMLzFLwKM=', 'Content-Length': '0'} 2023-10-18T20:30:46.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002938a217e551d084-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.963 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.963 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.964 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:46.964 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:46.964 DEBUG:boto:Method: GET 2023-10-18T20:30:46.965 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:46.965 DEBUG:boto:Data: 2023-10-18T20:30:46.965 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.966 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.967 DEBUG:boto:Params: {} 2023-10-18T20:30:46.967 DEBUG:boto:Token: None 2023-10-18T20:30:46.968 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000739ac5cb1df5cc80-0065304076-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.971 DEBUG:boto:b'bofrnf-58obj000000000000000000201000falseobj000000000000000000202023-10-18T20:30:32.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.972 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:46.973 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:46.973 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:46.973 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:46.974 DEBUG:boto:Method: GET 2023-10-18T20:30:46.974 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:46.975 DEBUG:boto:Data: 2023-10-18T20:30:46.975 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.976 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.976 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.976 DEBUG:boto:Params: {} 2023-10-18T20:30:46.977 DEBUG:boto:Token: None 2023-10-18T20:30:46.977 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/ 2023-10-18T20:30:46.978 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48= 2023-10-18T20:30:46.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mx9/Ii51Fljluiu4zhz2Ql0KU48=', 'Content-Length': '0'} 2023-10-18T20:30:46.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009de434f16b12d9b-0065304076-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.983 DEBUG:boto:b'bofrnf-58obj000000000000000000201000falseobj000000000000000000202023-10-18T20:30:32.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:46.984 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.984 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.985 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.985 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.985 DEBUG:boto:Data: 2023-10-18T20:30:46.986 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.986 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:46.987 DEBUG:boto:Port: 8001 2023-10-18T20:30:46.987 DEBUG:boto:Params: {} 2023-10-18T20:30:46.987 DEBUG:boto:Token: None 2023-10-18T20:30:46.988 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.988 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AJssD+MsVxoVocLTukJp4DRugKY= 2023-10-18T20:30:46.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AJssD+MsVxoVocLTukJp4DRugKY=', 'Content-Length': '0'} 2023-10-18T20:30:46.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daf962136949c53c-0065304076-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.991 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.992 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.992 DEBUG:boto:Method: HEAD 2023-10-18T20:30:46.993 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.993 DEBUG:boto:Data: 2023-10-18T20:30:46.994 DEBUG:boto:Headers: {} 2023-10-18T20:30:46.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:46.994 DEBUG:boto:Port: 8000 2023-10-18T20:30:46.995 DEBUG:boto:Params: {} 2023-10-18T20:30:46.995 DEBUG:boto:Token: None 2023-10-18T20:30:46.996 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:46 GMT /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:46.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AJssD+MsVxoVocLTukJp4DRugKY= 2023-10-18T20:30:46.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AJssD+MsVxoVocLTukJp4DRugKY=', 'Content-Length': '0'} 2023-10-18T20:30:46.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f780917f55a21d5d-0065304076-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:46.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-18T20:30:47.000 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:47.000 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:47.000 DEBUG:boto:Method: GET 2023-10-18T20:30:47.001 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:47.001 DEBUG:boto:Data: 2023-10-18T20:30:47.002 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.002 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.003 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.003 DEBUG:boto:Params: {} 2023-10-18T20:30:47.004 DEBUG:boto:Token: None 2023-10-18T20:30:47.004 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:47.005 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hjM5cgJB+Vf3u4LbaGb9U3BCWcY= 2023-10-18T20:30:47.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hjM5cgJB+Vf3u4LbaGb9U3BCWcY=', 'Content-Length': '0'} 2023-10-18T20:30:47.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c409566ef1f122a8-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.054 DEBUG:boto:path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:47.055 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000020 2023-10-18T20:30:47.055 DEBUG:boto:Method: GET 2023-10-18T20:30:47.056 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:47.056 DEBUG:boto:Data: 2023-10-18T20:30:47.057 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.057 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.058 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.058 DEBUG:boto:Params: {} 2023-10-18T20:30:47.059 DEBUG:boto:Token: None 2023-10-18T20:30:47.059 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000020 2023-10-18T20:30:47.059 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hjM5cgJB+Vf3u4LbaGb9U3BCWcY= 2023-10-18T20:30:47.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hjM5cgJB+Vf3u4LbaGb9U3BCWcY=', 'Content-Length': '0'} 2023-10-18T20:30:47.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fafbcb0365c55ad-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.063 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.063 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.064 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:47.064 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:47.065 DEBUG:boto:Method: GET 2023-10-18T20:30:47.065 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:47.066 DEBUG:boto:Data: 2023-10-18T20:30:47.066 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.067 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.067 DEBUG:boto:Params: {} 2023-10-18T20:30:47.068 DEBUG:boto:Token: None 2023-10-18T20:30:47.068 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.069 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ecadc86957ce333-0065304077-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.072 DEBUG:boto:b'bofrnf-58obj000000000000000000201000falseobj000000000000000000202023-10-18T20:30:32.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.073 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.074 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.074 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:47.074 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:47.075 DEBUG:boto:Method: GET 2023-10-18T20:30:47.075 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000020 2023-10-18T20:30:47.076 DEBUG:boto:Data: 2023-10-18T20:30:47.076 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.077 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.077 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.077 DEBUG:boto:Params: {} 2023-10-18T20:30:47.078 DEBUG:boto:Token: None 2023-10-18T20:30:47.078 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.079 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045bc86dca1831f5d-0065304077-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.084 DEBUG:boto:b'bofrnf-58obj000000000000000000201000falseobj000000000000000000202023-10-18T20:30:32.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-18T20:30:47.085 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.086 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.086 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.087 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.087 DEBUG:boto:Method: GET 2023-10-18T20:30:47.088 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.088 DEBUG:boto:Data: 2023-10-18T20:30:47.088 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.089 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.089 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.090 DEBUG:boto:Params: {} 2023-10-18T20:30:47.090 DEBUG:boto:Token: None 2023-10-18T20:30:47.091 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.091 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ze9bwXTWnXAveVsol3Qy0Z+tTDU= 2023-10-18T20:30:47.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ze9bwXTWnXAveVsol3Qy0Z+tTDU=', 'Content-Length': '0'} 2023-10-18T20:30:47.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd261a662c89eaee-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.095 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.095 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.096 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.096 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.096 DEBUG:boto:Method: GET 2023-10-18T20:30:47.097 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.097 DEBUG:boto:Data: 2023-10-18T20:30:47.098 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.098 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.099 DEBUG:boto:Params: {} 2023-10-18T20:30:47.099 DEBUG:boto:Token: None 2023-10-18T20:30:47.100 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ze9bwXTWnXAveVsol3Qy0Z+tTDU= 2023-10-18T20:30:47.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ze9bwXTWnXAveVsol3Qy0Z+tTDU=', 'Content-Length': '0'} 2023-10-18T20:30:47.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f16080c927cfded3-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.104 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.104 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.104 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.105 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.105 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.106 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.106 DEBUG:boto:Data: 2023-10-18T20:30:47.106 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.107 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.107 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.108 DEBUG:boto:Params: {} 2023-10-18T20:30:47.108 DEBUG:boto:Token: None 2023-10-18T20:30:47.109 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.109 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OvsaHmckf1OiSEn0axGHCk6qcXA= 2023-10-18T20:30:47.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OvsaHmckf1OiSEn0axGHCk6qcXA=', 'Content-Length': '0'} 2023-10-18T20:30:47.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d81fe986cdbd0433-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.112 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.113 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.113 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.113 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.114 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.114 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.115 DEBUG:boto:Data: 2023-10-18T20:30:47.115 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.116 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.116 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.116 DEBUG:boto:Params: {} 2023-10-18T20:30:47.117 DEBUG:boto:Token: None 2023-10-18T20:30:47.117 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.118 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OvsaHmckf1OiSEn0axGHCk6qcXA= 2023-10-18T20:30:47.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OvsaHmckf1OiSEn0axGHCk6qcXA=', 'Content-Length': '0'} 2023-10-18T20:30:47.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087ea791fc5c4788f-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-18T20:30:47.122 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.122 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.123 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.123 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.123 DEBUG:boto:Method: GET 2023-10-18T20:30:47.124 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.124 DEBUG:boto:Data: 2023-10-18T20:30:47.125 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.125 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.126 DEBUG:boto:Params: {} 2023-10-18T20:30:47.126 DEBUG:boto:Token: None 2023-10-18T20:30:47.127 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.127 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ze9bwXTWnXAveVsol3Qy0Z+tTDU= 2023-10-18T20:30:47.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ze9bwXTWnXAveVsol3Qy0Z+tTDU=', 'Content-Length': '0'} 2023-10-18T20:30:47.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c72a733b4b3fdcd5-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.131 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.131 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.132 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.132 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.132 DEBUG:boto:Method: GET 2023-10-18T20:30:47.133 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.133 DEBUG:boto:Data: 2023-10-18T20:30:47.134 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.134 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.135 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.135 DEBUG:boto:Params: {} 2023-10-18T20:30:47.135 DEBUG:boto:Token: None 2023-10-18T20:30:47.136 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021?versionId=null 2023-10-18T20:30:47.136 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ze9bwXTWnXAveVsol3Qy0Z+tTDU= 2023-10-18T20:30:47.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ze9bwXTWnXAveVsol3Qy0Z+tTDU=', 'Content-Length': '0'} 2023-10-18T20:30:47.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067a11400e1cdfdd4-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.186 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.187 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.188 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.188 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.188 DEBUG:boto:Method: GET 2023-10-18T20:30:47.189 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.189 DEBUG:boto:Data: 2023-10-18T20:30:47.190 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.190 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.191 DEBUG:boto:Params: {} 2023-10-18T20:30:47.191 DEBUG:boto:Token: None 2023-10-18T20:30:47.192 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620ef6ac808f4030-0065304077-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.196 DEBUG:boto:b'bofrnf-58obj000000000000000000211000falseobj000000000000000000212023-10-18T20:30:32.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.197 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.197 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.198 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.198 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.199 DEBUG:boto:Method: GET 2023-10-18T20:30:47.199 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.200 DEBUG:boto:Data: 2023-10-18T20:30:47.200 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.200 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.201 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.201 DEBUG:boto:Params: {} 2023-10-18T20:30:47.202 DEBUG:boto:Token: None 2023-10-18T20:30:47.202 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.203 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042cf5719e98284fe-0065304077-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.208 DEBUG:boto:b'bofrnf-58obj000000000000000000211000falseobj000000000000000000212023-10-18T20:30:32.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.209 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.209 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.210 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.210 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.210 DEBUG:boto:Data: 2023-10-18T20:30:47.211 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.211 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.212 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.212 DEBUG:boto:Params: {} 2023-10-18T20:30:47.212 DEBUG:boto:Token: None 2023-10-18T20:30:47.213 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uIZHutRZ34IK1dYNN6C70D5c5Gg= 2023-10-18T20:30:47.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uIZHutRZ34IK1dYNN6C70D5c5Gg=', 'Content-Length': '0'} 2023-10-18T20:30:47.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842622f536ebcce9-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.217 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.217 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.218 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.218 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.218 DEBUG:boto:Data: 2023-10-18T20:30:47.219 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.220 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.220 DEBUG:boto:Params: {} 2023-10-18T20:30:47.221 DEBUG:boto:Token: None 2023-10-18T20:30:47.221 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uIZHutRZ34IK1dYNN6C70D5c5Gg= 2023-10-18T20:30:47.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uIZHutRZ34IK1dYNN6C70D5c5Gg=', 'Content-Length': '0'} 2023-10-18T20:30:47.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014895484b5f2d429-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-18T20:30:47.225 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.225 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.226 DEBUG:boto:Method: GET 2023-10-18T20:30:47.226 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.227 DEBUG:boto:Data: 2023-10-18T20:30:47.227 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.227 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.229 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.229 DEBUG:boto:Params: {} 2023-10-18T20:30:47.230 DEBUG:boto:Token: None 2023-10-18T20:30:47.230 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0VVX7AwgG8TAk738nHTSXEMqlFw= 2023-10-18T20:30:47.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0VVX7AwgG8TAk738nHTSXEMqlFw=', 'Content-Length': '0'} 2023-10-18T20:30:47.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd92b593df570f04-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.274 DEBUG:boto:path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.275 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.275 DEBUG:boto:Method: GET 2023-10-18T20:30:47.276 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.276 DEBUG:boto:Data: 2023-10-18T20:30:47.277 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.277 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.278 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.278 DEBUG:boto:Params: {} 2023-10-18T20:30:47.279 DEBUG:boto:Token: None 2023-10-18T20:30:47.279 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000021 2023-10-18T20:30:47.280 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0VVX7AwgG8TAk738nHTSXEMqlFw= 2023-10-18T20:30:47.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0VVX7AwgG8TAk738nHTSXEMqlFw=', 'Content-Length': '0'} 2023-10-18T20:30:47.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d7046b216c6e31-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.283 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.284 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.284 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.285 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.285 DEBUG:boto:Method: GET 2023-10-18T20:30:47.286 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.286 DEBUG:boto:Data: 2023-10-18T20:30:47.287 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.287 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.288 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.288 DEBUG:boto:Params: {} 2023-10-18T20:30:47.289 DEBUG:boto:Token: None 2023-10-18T20:30:47.290 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.290 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6c664d4faf580f3-0065304077-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.294 DEBUG:boto:b'bofrnf-58obj000000000000000000211000falseobj000000000000000000212023-10-18T20:30:32.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.295 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.295 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.296 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.296 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.297 DEBUG:boto:Method: GET 2023-10-18T20:30:47.297 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000021 2023-10-18T20:30:47.297 DEBUG:boto:Data: 2023-10-18T20:30:47.298 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.298 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.299 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.299 DEBUG:boto:Params: {} 2023-10-18T20:30:47.300 DEBUG:boto:Token: None 2023-10-18T20:30:47.300 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aec3f3ab4a57f287-0065304077-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.350 DEBUG:boto:b'bofrnf-58obj000000000000000000211000falseobj000000000000000000212023-10-18T20:30:32.151Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-18T20:30:47.352 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.352 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.353 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.353 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.354 DEBUG:boto:Method: GET 2023-10-18T20:30:47.354 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.355 DEBUG:boto:Data: 2023-10-18T20:30:47.355 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.355 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.356 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.356 DEBUG:boto:Params: {} 2023-10-18T20:30:47.357 DEBUG:boto:Token: None 2023-10-18T20:30:47.357 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LdmSsww6DGbJ4vVSiAPq7kHltps= 2023-10-18T20:30:47.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LdmSsww6DGbJ4vVSiAPq7kHltps=', 'Content-Length': '0'} 2023-10-18T20:30:47.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000263d453de8256871-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.362 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.362 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.362 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.363 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.363 DEBUG:boto:Method: GET 2023-10-18T20:30:47.364 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.364 DEBUG:boto:Data: 2023-10-18T20:30:47.364 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.365 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.365 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.366 DEBUG:boto:Params: {} 2023-10-18T20:30:47.366 DEBUG:boto:Token: None 2023-10-18T20:30:47.367 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.367 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LdmSsww6DGbJ4vVSiAPq7kHltps= 2023-10-18T20:30:47.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LdmSsww6DGbJ4vVSiAPq7kHltps=', 'Content-Length': '0'} 2023-10-18T20:30:47.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000322307aaeb0d6f27-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.370 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.371 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.371 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.371 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.372 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.372 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.373 DEBUG:boto:Data: 2023-10-18T20:30:47.373 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.373 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.374 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.374 DEBUG:boto:Params: {} 2023-10-18T20:30:47.374 DEBUG:boto:Token: None 2023-10-18T20:30:47.375 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.375 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Kr+RY65QFFL00SajR0a30puoMWo= 2023-10-18T20:30:47.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Kr+RY65QFFL00SajR0a30puoMWo=', 'Content-Length': '0'} 2023-10-18T20:30:47.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af7244342ff1142e-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.379 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.379 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.379 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.380 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.380 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.381 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.381 DEBUG:boto:Data: 2023-10-18T20:30:47.381 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.382 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.382 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.382 DEBUG:boto:Params: {} 2023-10-18T20:30:47.383 DEBUG:boto:Token: None 2023-10-18T20:30:47.383 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.384 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Kr+RY65QFFL00SajR0a30puoMWo= 2023-10-18T20:30:47.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Kr+RY65QFFL00SajR0a30puoMWo=', 'Content-Length': '0'} 2023-10-18T20:30:47.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e0846ffeabaed0c-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-18T20:30:47.387 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.388 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.388 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.388 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.389 DEBUG:boto:Method: GET 2023-10-18T20:30:47.389 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.390 DEBUG:boto:Data: 2023-10-18T20:30:47.390 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.391 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.391 DEBUG:boto:Params: {} 2023-10-18T20:30:47.392 DEBUG:boto:Token: None 2023-10-18T20:30:47.392 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.393 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LdmSsww6DGbJ4vVSiAPq7kHltps= 2023-10-18T20:30:47.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LdmSsww6DGbJ4vVSiAPq7kHltps=', 'Content-Length': '0'} 2023-10-18T20:30:47.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c76f186d810737-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.396 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.396 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.397 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.397 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.398 DEBUG:boto:Method: GET 2023-10-18T20:30:47.398 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.399 DEBUG:boto:Data: 2023-10-18T20:30:47.399 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.400 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.400 DEBUG:boto:Params: {} 2023-10-18T20:30:47.401 DEBUG:boto:Token: None 2023-10-18T20:30:47.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022?versionId=null 2023-10-18T20:30:47.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LdmSsww6DGbJ4vVSiAPq7kHltps= 2023-10-18T20:30:47.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LdmSsww6DGbJ4vVSiAPq7kHltps=', 'Content-Length': '0'} 2023-10-18T20:30:47.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf8fdfb21f1224d8-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.405 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.405 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.406 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.406 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.407 DEBUG:boto:Method: GET 2023-10-18T20:30:47.407 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.408 DEBUG:boto:Data: 2023-10-18T20:30:47.408 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.409 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.409 DEBUG:boto:Params: {} 2023-10-18T20:30:47.410 DEBUG:boto:Token: None 2023-10-18T20:30:47.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8024a753cc61e57-0065304077-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.415 DEBUG:boto:b'bofrnf-58obj000000000000000000221000falseobj000000000000000000222023-10-18T20:30:30.350Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.415 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.416 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.416 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.417 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.417 DEBUG:boto:Method: GET 2023-10-18T20:30:47.418 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.418 DEBUG:boto:Data: 2023-10-18T20:30:47.418 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.419 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.419 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.420 DEBUG:boto:Params: {} 2023-10-18T20:30:47.420 DEBUG:boto:Token: None 2023-10-18T20:30:47.421 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.421 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c1c30f8ad370877-0065304077-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.426 DEBUG:boto:b'bofrnf-58obj000000000000000000221000falseobj000000000000000000222023-10-18T20:30:30.350Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.427 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.427 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.428 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.428 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.428 DEBUG:boto:Data: 2023-10-18T20:30:47.429 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.429 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.430 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.430 DEBUG:boto:Params: {} 2023-10-18T20:30:47.431 DEBUG:boto:Token: None 2023-10-18T20:30:47.431 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.431 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aG0sFPsXAktez2wSRQscSITZ6Vs= 2023-10-18T20:30:47.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aG0sFPsXAktez2wSRQscSITZ6Vs=', 'Content-Length': '0'} 2023-10-18T20:30:47.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000328c85bf85105f00-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.434 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.435 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.435 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.435 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.436 DEBUG:boto:Data: 2023-10-18T20:30:47.436 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.436 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.437 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.437 DEBUG:boto:Params: {} 2023-10-18T20:30:47.438 DEBUG:boto:Token: None 2023-10-18T20:30:47.438 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.439 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aG0sFPsXAktez2wSRQscSITZ6Vs= 2023-10-18T20:30:47.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aG0sFPsXAktez2wSRQscSITZ6Vs=', 'Content-Length': '0'} 2023-10-18T20:30:47.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4cb18f0907c03d6-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-18T20:30:47.442 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.442 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.443 DEBUG:boto:Method: GET 2023-10-18T20:30:47.443 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.444 DEBUG:boto:Data: 2023-10-18T20:30:47.444 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.445 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.445 DEBUG:boto:Params: {} 2023-10-18T20:30:47.446 DEBUG:boto:Token: None 2023-10-18T20:30:47.446 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.447 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1pHoJruXuERDdh+eA/CIVOdbBiM= 2023-10-18T20:30:47.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1pHoJruXuERDdh+eA/CIVOdbBiM=', 'Content-Length': '0'} 2023-10-18T20:30:47.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000823e499b85aeade-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.450 DEBUG:boto:path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.451 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.451 DEBUG:boto:Method: GET 2023-10-18T20:30:47.451 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.452 DEBUG:boto:Data: 2023-10-18T20:30:47.452 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.453 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.453 DEBUG:boto:Params: {} 2023-10-18T20:30:47.454 DEBUG:boto:Token: None 2023-10-18T20:30:47.454 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000022 2023-10-18T20:30:47.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1pHoJruXuERDdh+eA/CIVOdbBiM= 2023-10-18T20:30:47.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1pHoJruXuERDdh+eA/CIVOdbBiM=', 'Content-Length': '0'} 2023-10-18T20:30:47.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b910d81f1d93491-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.458 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.458 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.459 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.459 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.460 DEBUG:boto:Method: GET 2023-10-18T20:30:47.460 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.460 DEBUG:boto:Data: 2023-10-18T20:30:47.461 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.461 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.462 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.462 DEBUG:boto:Params: {} 2023-10-18T20:30:47.463 DEBUG:boto:Token: None 2023-10-18T20:30:47.463 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b37e59f68a91ea0e-0065304077-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.467 DEBUG:boto:b'bofrnf-58obj000000000000000000221000falseobj000000000000000000222023-10-18T20:30:30.350Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.468 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.468 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.469 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.469 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.469 DEBUG:boto:Method: GET 2023-10-18T20:30:47.470 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000022 2023-10-18T20:30:47.470 DEBUG:boto:Data: 2023-10-18T20:30:47.471 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.472 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.472 DEBUG:boto:Params: {} 2023-10-18T20:30:47.472 DEBUG:boto:Token: None 2023-10-18T20:30:47.473 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.473 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000615ac9fa999b3e03-0065304077-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.478 DEBUG:boto:b'bofrnf-58obj000000000000000000221000falseobj000000000000000000222023-10-18T20:30:30.350Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-18T20:30:47.480 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.480 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.480 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.481 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.481 DEBUG:boto:Method: GET 2023-10-18T20:30:47.482 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.482 DEBUG:boto:Data: 2023-10-18T20:30:47.483 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.483 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.483 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.484 DEBUG:boto:Params: {} 2023-10-18T20:30:47.484 DEBUG:boto:Token: None 2023-10-18T20:30:47.485 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.485 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+50jcPAbtC6Okp2tVKZINBfNKHQ= 2023-10-18T20:30:47.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+50jcPAbtC6Okp2tVKZINBfNKHQ=', 'Content-Length': '0'} 2023-10-18T20:30:47.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff121b6f7f49d480-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.534 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.535 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.535 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.536 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.536 DEBUG:boto:Method: GET 2023-10-18T20:30:47.537 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.537 DEBUG:boto:Data: 2023-10-18T20:30:47.538 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.538 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.538 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.539 DEBUG:boto:Params: {} 2023-10-18T20:30:47.540 DEBUG:boto:Token: None 2023-10-18T20:30:47.540 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.541 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+50jcPAbtC6Okp2tVKZINBfNKHQ= 2023-10-18T20:30:47.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+50jcPAbtC6Okp2tVKZINBfNKHQ=', 'Content-Length': '0'} 2023-10-18T20:30:47.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3f9289685ce8b64-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.544 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.544 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.545 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.545 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.546 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.546 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.546 DEBUG:boto:Data: 2023-10-18T20:30:47.547 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.547 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.548 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.548 DEBUG:boto:Params: {} 2023-10-18T20:30:47.549 DEBUG:boto:Token: None 2023-10-18T20:30:47.549 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.550 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e6Pcqy3PrWwbUmOepcLzSP8SFeo= 2023-10-18T20:30:47.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e6Pcqy3PrWwbUmOepcLzSP8SFeo=', 'Content-Length': '0'} 2023-10-18T20:30:47.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027e47d29741916b6-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.553 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.553 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.553 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.554 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.554 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.555 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.555 DEBUG:boto:Data: 2023-10-18T20:30:47.556 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.556 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.556 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.557 DEBUG:boto:Params: {} 2023-10-18T20:30:47.557 DEBUG:boto:Token: None 2023-10-18T20:30:47.558 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.558 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e6Pcqy3PrWwbUmOepcLzSP8SFeo= 2023-10-18T20:30:47.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e6Pcqy3PrWwbUmOepcLzSP8SFeo=', 'Content-Length': '0'} 2023-10-18T20:30:47.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffb4b823e47b7ed4-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-18T20:30:47.562 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.562 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.563 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.563 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.563 DEBUG:boto:Method: GET 2023-10-18T20:30:47.564 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.564 DEBUG:boto:Data: 2023-10-18T20:30:47.565 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.565 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.566 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.566 DEBUG:boto:Params: {} 2023-10-18T20:30:47.567 DEBUG:boto:Token: None 2023-10-18T20:30:47.567 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+50jcPAbtC6Okp2tVKZINBfNKHQ= 2023-10-18T20:30:47.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+50jcPAbtC6Okp2tVKZINBfNKHQ=', 'Content-Length': '0'} 2023-10-18T20:30:47.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbbba7724af0cc5a-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.571 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.571 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.572 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.572 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.573 DEBUG:boto:Method: GET 2023-10-18T20:30:47.573 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.574 DEBUG:boto:Data: 2023-10-18T20:30:47.574 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.574 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.575 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.575 DEBUG:boto:Params: {} 2023-10-18T20:30:47.576 DEBUG:boto:Token: None 2023-10-18T20:30:47.576 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023?versionId=null 2023-10-18T20:30:47.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+50jcPAbtC6Okp2tVKZINBfNKHQ= 2023-10-18T20:30:47.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+50jcPAbtC6Okp2tVKZINBfNKHQ=', 'Content-Length': '0'} 2023-10-18T20:30:47.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1035b2f02a4bf0e-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.626 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.627 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.627 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.628 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.628 DEBUG:boto:Method: GET 2023-10-18T20:30:47.629 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.629 DEBUG:boto:Data: 2023-10-18T20:30:47.630 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.631 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.631 DEBUG:boto:Params: {} 2023-10-18T20:30:47.632 DEBUG:boto:Token: None 2023-10-18T20:30:47.632 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.633 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678fa3427011eb31-0065304077-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.636 DEBUG:boto:b'bofrnf-58obj000000000000000000231000falseobj000000000000000000232023-10-18T20:30:30.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.637 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.638 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.638 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.639 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.639 DEBUG:boto:Method: GET 2023-10-18T20:30:47.640 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.640 DEBUG:boto:Data: 2023-10-18T20:30:47.641 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.641 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.642 DEBUG:boto:Params: {} 2023-10-18T20:30:47.642 DEBUG:boto:Token: None 2023-10-18T20:30:47.643 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000025d02abc740cc25-0065304077-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.648 DEBUG:boto:b'bofrnf-58obj000000000000000000231000falseobj000000000000000000232023-10-18T20:30:30.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.649 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.650 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.650 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.650 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.651 DEBUG:boto:Data: 2023-10-18T20:30:47.651 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.652 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.652 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.653 DEBUG:boto:Params: {} 2023-10-18T20:30:47.653 DEBUG:boto:Token: None 2023-10-18T20:30:47.654 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.654 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NKouxTKTpUkrHd74qYPoM/4h5UI= 2023-10-18T20:30:47.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NKouxTKTpUkrHd74qYPoM/4h5UI=', 'Content-Length': '0'} 2023-10-18T20:30:47.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbf0cd2a8c9be258-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.657 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.658 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.658 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.659 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.659 DEBUG:boto:Data: 2023-10-18T20:30:47.660 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.660 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.660 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.661 DEBUG:boto:Params: {} 2023-10-18T20:30:47.661 DEBUG:boto:Token: None 2023-10-18T20:30:47.662 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NKouxTKTpUkrHd74qYPoM/4h5UI= 2023-10-18T20:30:47.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NKouxTKTpUkrHd74qYPoM/4h5UI=', 'Content-Length': '0'} 2023-10-18T20:30:47.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006905d34656cd9833-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-18T20:30:47.666 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.666 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.666 DEBUG:boto:Method: GET 2023-10-18T20:30:47.667 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.667 DEBUG:boto:Data: 2023-10-18T20:30:47.668 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.668 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.669 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.669 DEBUG:boto:Params: {} 2023-10-18T20:30:47.669 DEBUG:boto:Token: None 2023-10-18T20:30:47.670 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.671 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P59qqjHGAQ2gqU1u5knP86ijIrM= 2023-10-18T20:30:47.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P59qqjHGAQ2gqU1u5knP86ijIrM=', 'Content-Length': '0'} 2023-10-18T20:30:47.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b08552ed3ffc4368-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.674 DEBUG:boto:path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.674 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.675 DEBUG:boto:Method: GET 2023-10-18T20:30:47.675 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.675 DEBUG:boto:Data: 2023-10-18T20:30:47.676 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.676 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.677 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.677 DEBUG:boto:Params: {} 2023-10-18T20:30:47.678 DEBUG:boto:Token: None 2023-10-18T20:30:47.678 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000023 2023-10-18T20:30:47.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P59qqjHGAQ2gqU1u5knP86ijIrM= 2023-10-18T20:30:47.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P59qqjHGAQ2gqU1u5knP86ijIrM=', 'Content-Length': '0'} 2023-10-18T20:30:47.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079aa925874335c1f-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.682 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.682 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.683 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.683 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.684 DEBUG:boto:Method: GET 2023-10-18T20:30:47.684 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.685 DEBUG:boto:Data: 2023-10-18T20:30:47.685 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.685 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.686 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.686 DEBUG:boto:Params: {} 2023-10-18T20:30:47.687 DEBUG:boto:Token: None 2023-10-18T20:30:47.687 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2fbf0e6a956911-0065304077-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.692 DEBUG:boto:b'bofrnf-58obj000000000000000000231000falseobj000000000000000000232023-10-18T20:30:30.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.692 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.693 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.693 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.694 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.694 DEBUG:boto:Method: GET 2023-10-18T20:30:47.695 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000023 2023-10-18T20:30:47.695 DEBUG:boto:Data: 2023-10-18T20:30:47.696 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.696 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.696 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.697 DEBUG:boto:Params: {} 2023-10-18T20:30:47.697 DEBUG:boto:Token: None 2023-10-18T20:30:47.698 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.698 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e60dbb2386df509-0065304077-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.704 DEBUG:boto:b'bofrnf-58obj000000000000000000231000falseobj000000000000000000232023-10-18T20:30:30.261Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-18T20:30:47.705 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.705 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.706 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.706 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.707 DEBUG:boto:Method: GET 2023-10-18T20:30:47.707 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.707 DEBUG:boto:Data: 2023-10-18T20:30:47.708 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.708 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.709 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.709 DEBUG:boto:Params: {} 2023-10-18T20:30:47.710 DEBUG:boto:Token: None 2023-10-18T20:30:47.710 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.711 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kJOV0x9F2kmkTFWTeU4X2aV7WCU= 2023-10-18T20:30:47.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kJOV0x9F2kmkTFWTeU4X2aV7WCU=', 'Content-Length': '0'} 2023-10-18T20:30:47.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025ba6e7d4e6e9120-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.754 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.755 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.755 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.756 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.756 DEBUG:boto:Method: GET 2023-10-18T20:30:47.756 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.757 DEBUG:boto:Data: 2023-10-18T20:30:47.757 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.758 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.758 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.759 DEBUG:boto:Params: {} 2023-10-18T20:30:47.759 DEBUG:boto:Token: None 2023-10-18T20:30:47.760 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.760 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kJOV0x9F2kmkTFWTeU4X2aV7WCU= 2023-10-18T20:30:47.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kJOV0x9F2kmkTFWTeU4X2aV7WCU=', 'Content-Length': '0'} 2023-10-18T20:30:47.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041d97816d1518424-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.763 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.764 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.764 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.764 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.765 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.765 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.766 DEBUG:boto:Data: 2023-10-18T20:30:47.766 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.767 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.767 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.767 DEBUG:boto:Params: {} 2023-10-18T20:30:47.768 DEBUG:boto:Token: None 2023-10-18T20:30:47.768 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.769 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPAbT7ogQGHGDm1RVJpU5KP8SXU= 2023-10-18T20:30:47.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPAbT7ogQGHGDm1RVJpU5KP8SXU=', 'Content-Length': '0'} 2023-10-18T20:30:47.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0482c8363a7f11d-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.771 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.772 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.772 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.773 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.773 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.773 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.774 DEBUG:boto:Data: 2023-10-18T20:30:47.774 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.775 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.775 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.775 DEBUG:boto:Params: {} 2023-10-18T20:30:47.776 DEBUG:boto:Token: None 2023-10-18T20:30:47.776 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.777 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fPAbT7ogQGHGDm1RVJpU5KP8SXU= 2023-10-18T20:30:47.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fPAbT7ogQGHGDm1RVJpU5KP8SXU=', 'Content-Length': '0'} 2023-10-18T20:30:47.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c601210bd078fb-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-18T20:30:47.780 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.781 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.781 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.781 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.782 DEBUG:boto:Method: GET 2023-10-18T20:30:47.782 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.783 DEBUG:boto:Data: 2023-10-18T20:30:47.783 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.784 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.784 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.785 DEBUG:boto:Params: {} 2023-10-18T20:30:47.785 DEBUG:boto:Token: None 2023-10-18T20:30:47.785 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.786 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kJOV0x9F2kmkTFWTeU4X2aV7WCU= 2023-10-18T20:30:47.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kJOV0x9F2kmkTFWTeU4X2aV7WCU=', 'Content-Length': '0'} 2023-10-18T20:30:47.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b133a7df27d86669-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.789 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.790 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.790 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.791 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.791 DEBUG:boto:Method: GET 2023-10-18T20:30:47.791 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.792 DEBUG:boto:Data: 2023-10-18T20:30:47.792 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.793 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.793 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.794 DEBUG:boto:Params: {} 2023-10-18T20:30:47.794 DEBUG:boto:Token: None 2023-10-18T20:30:47.795 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024?versionId=null 2023-10-18T20:30:47.795 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kJOV0x9F2kmkTFWTeU4X2aV7WCU= 2023-10-18T20:30:47.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kJOV0x9F2kmkTFWTeU4X2aV7WCU=', 'Content-Length': '0'} 2023-10-18T20:30:47.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b791a3c297051e2-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.838 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.839 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.839 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.840 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.840 DEBUG:boto:Method: GET 2023-10-18T20:30:47.840 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.841 DEBUG:boto:Data: 2023-10-18T20:30:47.841 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.842 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.842 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.842 DEBUG:boto:Params: {} 2023-10-18T20:30:47.843 DEBUG:boto:Token: None 2023-10-18T20:30:47.843 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.844 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f117e9b2ed9f608-0065304077-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.847 DEBUG:boto:b'bofrnf-58obj000000000000000000241000falseobj000000000000000000242023-10-18T20:30:32.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.848 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.849 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.849 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.849 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.850 DEBUG:boto:Method: GET 2023-10-18T20:30:47.850 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.851 DEBUG:boto:Data: 2023-10-18T20:30:47.851 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.851 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.852 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.852 DEBUG:boto:Params: {} 2023-10-18T20:30:47.853 DEBUG:boto:Token: None 2023-10-18T20:30:47.853 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.854 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecb536eca780aacc-0065304077-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.859 DEBUG:boto:b'bofrnf-58obj000000000000000000241000falseobj000000000000000000242023-10-18T20:30:32.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.859 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.860 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.860 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.861 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.861 DEBUG:boto:Data: 2023-10-18T20:30:47.862 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.862 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.862 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.863 DEBUG:boto:Params: {} 2023-10-18T20:30:47.863 DEBUG:boto:Token: None 2023-10-18T20:30:47.864 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.864 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/ORImpXFxHoOMMOJREmC7xKnW6U= 2023-10-18T20:30:47.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/ORImpXFxHoOMMOJREmC7xKnW6U=', 'Content-Length': '0'} 2023-10-18T20:30:47.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b30126f95551253-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.867 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.867 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.868 DEBUG:boto:Method: HEAD 2023-10-18T20:30:47.868 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.869 DEBUG:boto:Data: 2023-10-18T20:30:47.869 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.870 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.870 DEBUG:boto:Params: {} 2023-10-18T20:30:47.871 DEBUG:boto:Token: None 2023-10-18T20:30:47.871 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/ORImpXFxHoOMMOJREmC7xKnW6U= 2023-10-18T20:30:47.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/ORImpXFxHoOMMOJREmC7xKnW6U=', 'Content-Length': '0'} 2023-10-18T20:30:47.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079bc06e067e0a381-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-18T20:30:47.875 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.876 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.876 DEBUG:boto:Method: GET 2023-10-18T20:30:47.876 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.877 DEBUG:boto:Data: 2023-10-18T20:30:47.877 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.878 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.878 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.878 DEBUG:boto:Params: {} 2023-10-18T20:30:47.879 DEBUG:boto:Token: None 2023-10-18T20:30:47.879 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5M3B1EggFu8YgT3MdBC5JbJw+yI= 2023-10-18T20:30:47.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5M3B1EggFu8YgT3MdBC5JbJw+yI=', 'Content-Length': '0'} 2023-10-18T20:30:47.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2fda2c0a79ce5a4-0065304077-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.926 DEBUG:boto:path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.927 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.927 DEBUG:boto:Method: GET 2023-10-18T20:30:47.928 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.928 DEBUG:boto:Data: 2023-10-18T20:30:47.929 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.930 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.930 DEBUG:boto:Params: {} 2023-10-18T20:30:47.931 DEBUG:boto:Token: None 2023-10-18T20:30:47.931 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/obj00000000000000000024 2023-10-18T20:30:47.932 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5M3B1EggFu8YgT3MdBC5JbJw+yI= 2023-10-18T20:30:47.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5M3B1EggFu8YgT3MdBC5JbJw+yI=', 'Content-Length': '0'} 2023-10-18T20:30:47.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c839594b3c290c-0065304077-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.935 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.935 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.936 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.936 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.937 DEBUG:boto:Method: GET 2023-10-18T20:30:47.937 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.938 DEBUG:boto:Data: 2023-10-18T20:30:47.938 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.939 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:47.939 DEBUG:boto:Port: 8001 2023-10-18T20:30:47.940 DEBUG:boto:Params: {} 2023-10-18T20:30:47.940 DEBUG:boto:Token: None 2023-10-18T20:30:47.940 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.941 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b438de1e321c965e-0065304077-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.945 DEBUG:boto:b'bofrnf-58obj000000000000000000241000falseobj000000000000000000242023-10-18T20:30:32.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.946 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:47.946 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:47.947 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.947 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.947 DEBUG:boto:Method: GET 2023-10-18T20:30:47.948 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000024 2023-10-18T20:30:47.948 DEBUG:boto:Data: 2023-10-18T20:30:47.949 DEBUG:boto:Headers: {} 2023-10-18T20:30:47.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:47.950 DEBUG:boto:Port: 8000 2023-10-18T20:30:47.950 DEBUG:boto:Params: {} 2023-10-18T20:30:47.951 DEBUG:boto:Token: None 2023-10-18T20:30:47.951 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:47 GMT /bofrnf-58/ 2023-10-18T20:30:47.952 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4= 2023-10-18T20:30:47.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kgRj6UbRTKgmyqge3yN3zt/9nP4=', 'Content-Length': '0'} 2023-10-18T20:30:47.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c746b530c92b8e69-0065304077-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:47.998 DEBUG:boto:b'bofrnf-58obj000000000000000000241000falseobj000000000000000000242023-10-18T20:30:32.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:47.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-18T20:30:48.000 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.000 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.001 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.001 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.001 DEBUG:boto:Method: GET 2023-10-18T20:30:48.002 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.002 DEBUG:boto:Data: 2023-10-18T20:30:48.003 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.003 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.004 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.004 DEBUG:boto:Params: {} 2023-10-18T20:30:48.005 DEBUG:boto:Token: None 2023-10-18T20:30:48.005 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.006 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e6nEu2CnVzDUcSEhA1L+aQ5KmHw= 2023-10-18T20:30:48.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e6nEu2CnVzDUcSEhA1L+aQ5KmHw=', 'Content-Length': '0'} 2023-10-18T20:30:48.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005766c6cb5b70f992-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.009 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.010 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.010 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.011 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.011 DEBUG:boto:Method: GET 2023-10-18T20:30:48.012 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.012 DEBUG:boto:Data: 2023-10-18T20:30:48.013 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.013 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.013 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.014 DEBUG:boto:Params: {} 2023-10-18T20:30:48.014 DEBUG:boto:Token: None 2023-10-18T20:30:48.015 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.015 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e6nEu2CnVzDUcSEhA1L+aQ5KmHw= 2023-10-18T20:30:48.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e6nEu2CnVzDUcSEhA1L+aQ5KmHw=', 'Content-Length': '0'} 2023-10-18T20:30:48.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093482e528293fdf6-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.019 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.019 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.020 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.020 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.020 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.021 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.021 DEBUG:boto:Data: 2023-10-18T20:30:48.022 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.022 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.022 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.023 DEBUG:boto:Params: {} 2023-10-18T20:30:48.023 DEBUG:boto:Token: None 2023-10-18T20:30:48.024 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.024 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EiIDjyocZKDSd0PCSOgvLyl8b44= 2023-10-18T20:30:48.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EiIDjyocZKDSd0PCSOgvLyl8b44=', 'Content-Length': '0'} 2023-10-18T20:30:48.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005203518ae7b10de6-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.027 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.028 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.028 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.029 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.029 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.029 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.030 DEBUG:boto:Data: 2023-10-18T20:30:48.030 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.031 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.031 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.032 DEBUG:boto:Params: {} 2023-10-18T20:30:48.032 DEBUG:boto:Token: None 2023-10-18T20:30:48.032 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.033 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EiIDjyocZKDSd0PCSOgvLyl8b44= 2023-10-18T20:30:48.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EiIDjyocZKDSd0PCSOgvLyl8b44=', 'Content-Length': '0'} 2023-10-18T20:30:48.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a85de9a039b52223-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-18T20:30:48.037 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.037 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.038 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.038 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.039 DEBUG:boto:Method: GET 2023-10-18T20:30:48.039 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.039 DEBUG:boto:Data: 2023-10-18T20:30:48.040 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.040 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.041 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.041 DEBUG:boto:Params: {} 2023-10-18T20:30:48.042 DEBUG:boto:Token: None 2023-10-18T20:30:48.042 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.043 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e6nEu2CnVzDUcSEhA1L+aQ5KmHw= 2023-10-18T20:30:48.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e6nEu2CnVzDUcSEhA1L+aQ5KmHw=', 'Content-Length': '0'} 2023-10-18T20:30:48.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecfbe6b5e7e397d4-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.046 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.047 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.047 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.048 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.048 DEBUG:boto:Method: GET 2023-10-18T20:30:48.049 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.049 DEBUG:boto:Data: 2023-10-18T20:30:48.050 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.050 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.051 DEBUG:boto:Params: {} 2023-10-18T20:30:48.052 DEBUG:boto:Token: None 2023-10-18T20:30:48.052 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025?versionId=null 2023-10-18T20:30:48.053 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e6nEu2CnVzDUcSEhA1L+aQ5KmHw= 2023-10-18T20:30:48.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e6nEu2CnVzDUcSEhA1L+aQ5KmHw=', 'Content-Length': '0'} 2023-10-18T20:30:48.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1d26b4b53de9f6-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.056 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.057 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.057 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.057 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.058 DEBUG:boto:Method: GET 2023-10-18T20:30:48.058 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.059 DEBUG:boto:Data: 2023-10-18T20:30:48.059 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.060 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.060 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.061 DEBUG:boto:Params: {} 2023-10-18T20:30:48.061 DEBUG:boto:Token: None 2023-10-18T20:30:48.062 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.062 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b87079e2defc435-0065304078-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.066 DEBUG:boto:b'bofrnf-58obj000000000000000000251000falseobj000000000000000000252023-10-18T20:30:29.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.067 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.067 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.068 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.068 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.068 DEBUG:boto:Method: GET 2023-10-18T20:30:48.069 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.069 DEBUG:boto:Data: 2023-10-18T20:30:48.070 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.070 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.071 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.071 DEBUG:boto:Params: {} 2023-10-18T20:30:48.071 DEBUG:boto:Token: None 2023-10-18T20:30:48.072 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.072 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c60dc7b94d32c21d-0065304078-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.077 DEBUG:boto:b'bofrnf-58obj000000000000000000251000falseobj000000000000000000252023-10-18T20:30:29.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.078 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.078 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.079 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.079 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.080 DEBUG:boto:Data: 2023-10-18T20:30:48.080 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.080 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.081 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.081 DEBUG:boto:Params: {} 2023-10-18T20:30:48.082 DEBUG:boto:Token: None 2023-10-18T20:30:48.082 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.083 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TR+Ld9Td9ZK4mGB2WwoqMukHeKk= 2023-10-18T20:30:48.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TR+Ld9Td9ZK4mGB2WwoqMukHeKk=', 'Content-Length': '0'} 2023-10-18T20:30:48.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f0d850ce4fe7e42-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.086 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.086 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.087 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.087 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.087 DEBUG:boto:Data: 2023-10-18T20:30:48.088 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.088 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.089 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.089 DEBUG:boto:Params: {} 2023-10-18T20:30:48.090 DEBUG:boto:Token: None 2023-10-18T20:30:48.090 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.091 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TR+Ld9Td9ZK4mGB2WwoqMukHeKk= 2023-10-18T20:30:48.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TR+Ld9Td9ZK4mGB2WwoqMukHeKk=', 'Content-Length': '0'} 2023-10-18T20:30:48.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001588d0063bffcd1e-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-18T20:30:48.094 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.095 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.095 DEBUG:boto:Method: GET 2023-10-18T20:30:48.096 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.096 DEBUG:boto:Data: 2023-10-18T20:30:48.097 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.097 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.098 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.098 DEBUG:boto:Params: {} 2023-10-18T20:30:48.099 DEBUG:boto:Token: None 2023-10-18T20:30:48.099 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1IQD4anFc2u+KvtQ4xA47BDHIAc= 2023-10-18T20:30:48.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1IQD4anFc2u+KvtQ4xA47BDHIAc=', 'Content-Length': '0'} 2023-10-18T20:30:48.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd738d3884ae1f0f-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.103 DEBUG:boto:path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.103 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.104 DEBUG:boto:Method: GET 2023-10-18T20:30:48.104 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.105 DEBUG:boto:Data: 2023-10-18T20:30:48.105 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.106 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.106 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.107 DEBUG:boto:Params: {} 2023-10-18T20:30:48.107 DEBUG:boto:Token: None 2023-10-18T20:30:48.108 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000025 2023-10-18T20:30:48.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1IQD4anFc2u+KvtQ4xA47BDHIAc= 2023-10-18T20:30:48.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1IQD4anFc2u+KvtQ4xA47BDHIAc=', 'Content-Length': '0'} 2023-10-18T20:30:48.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fd6fd6397d18858-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.158 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.159 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.159 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.160 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.160 DEBUG:boto:Method: GET 2023-10-18T20:30:48.161 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.161 DEBUG:boto:Data: 2023-10-18T20:30:48.161 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.162 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.162 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.163 DEBUG:boto:Params: {} 2023-10-18T20:30:48.163 DEBUG:boto:Token: None 2023-10-18T20:30:48.164 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.164 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c25e6194fb3acbee-0065304078-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.168 DEBUG:boto:b'bofrnf-58obj000000000000000000251000falseobj000000000000000000252023-10-18T20:30:29.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.169 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.169 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.170 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.170 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.170 DEBUG:boto:Method: GET 2023-10-18T20:30:48.171 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000025 2023-10-18T20:30:48.171 DEBUG:boto:Data: 2023-10-18T20:30:48.172 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.172 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.173 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.173 DEBUG:boto:Params: {} 2023-10-18T20:30:48.173 DEBUG:boto:Token: None 2023-10-18T20:30:48.174 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.174 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc064f613474f6cd-0065304078-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.180 DEBUG:boto:b'bofrnf-58obj000000000000000000251000falseobj000000000000000000252023-10-18T20:30:29.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-18T20:30:48.181 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.181 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.182 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.182 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.183 DEBUG:boto:Method: GET 2023-10-18T20:30:48.185 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.186 DEBUG:boto:Data: 2023-10-18T20:30:48.186 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.186 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.187 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.187 DEBUG:boto:Params: {} 2023-10-18T20:30:48.188 DEBUG:boto:Token: None 2023-10-18T20:30:48.188 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.189 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ja9CdxAS7CLmvTolYDdu6ci1tsw= 2023-10-18T20:30:48.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ja9CdxAS7CLmvTolYDdu6ci1tsw=', 'Content-Length': '0'} 2023-10-18T20:30:48.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f71e1da8d7e5dd77-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.234 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.235 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.235 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.236 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.236 DEBUG:boto:Method: GET 2023-10-18T20:30:48.236 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.237 DEBUG:boto:Data: 2023-10-18T20:30:48.237 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.238 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.238 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.239 DEBUG:boto:Params: {} 2023-10-18T20:30:48.239 DEBUG:boto:Token: None 2023-10-18T20:30:48.240 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ja9CdxAS7CLmvTolYDdu6ci1tsw= 2023-10-18T20:30:48.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ja9CdxAS7CLmvTolYDdu6ci1tsw=', 'Content-Length': '0'} 2023-10-18T20:30:48.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e68aac689a530e51-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.243 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.244 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.244 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.245 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.245 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.245 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.246 DEBUG:boto:Data: 2023-10-18T20:30:48.246 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.247 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.247 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.248 DEBUG:boto:Params: {} 2023-10-18T20:30:48.248 DEBUG:boto:Token: None 2023-10-18T20:30:48.249 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.249 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:u7Ov1V6nhTFTWrWc8jr475U3nRM= 2023-10-18T20:30:48.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:u7Ov1V6nhTFTWrWc8jr475U3nRM=', 'Content-Length': '0'} 2023-10-18T20:30:48.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002042752555287014-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.252 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.253 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.253 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.253 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.254 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.254 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.255 DEBUG:boto:Data: 2023-10-18T20:30:48.255 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.256 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.256 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.257 DEBUG:boto:Params: {} 2023-10-18T20:30:48.257 DEBUG:boto:Token: None 2023-10-18T20:30:48.258 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.258 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:u7Ov1V6nhTFTWrWc8jr475U3nRM= 2023-10-18T20:30:48.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:u7Ov1V6nhTFTWrWc8jr475U3nRM=', 'Content-Length': '0'} 2023-10-18T20:30:48.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ddc1645c105a91b-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-18T20:30:48.262 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.262 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.263 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.263 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.264 DEBUG:boto:Method: GET 2023-10-18T20:30:48.264 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.265 DEBUG:boto:Data: 2023-10-18T20:30:48.265 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.265 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.266 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.266 DEBUG:boto:Params: {} 2023-10-18T20:30:48.267 DEBUG:boto:Token: None 2023-10-18T20:30:48.267 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.268 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ja9CdxAS7CLmvTolYDdu6ci1tsw= 2023-10-18T20:30:48.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ja9CdxAS7CLmvTolYDdu6ci1tsw=', 'Content-Length': '0'} 2023-10-18T20:30:48.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d8fb29033fe41e-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.271 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.272 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.272 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.272 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.273 DEBUG:boto:Method: GET 2023-10-18T20:30:48.273 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.274 DEBUG:boto:Data: 2023-10-18T20:30:48.274 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.275 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.275 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.276 DEBUG:boto:Params: {} 2023-10-18T20:30:48.276 DEBUG:boto:Token: None 2023-10-18T20:30:48.277 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026?versionId=null 2023-10-18T20:30:48.277 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ja9CdxAS7CLmvTolYDdu6ci1tsw= 2023-10-18T20:30:48.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ja9CdxAS7CLmvTolYDdu6ci1tsw=', 'Content-Length': '0'} 2023-10-18T20:30:48.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9c4de6f90beba2e-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.281 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.281 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.281 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.282 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.282 DEBUG:boto:Method: GET 2023-10-18T20:30:48.283 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.283 DEBUG:boto:Data: 2023-10-18T20:30:48.284 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.284 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.285 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.285 DEBUG:boto:Params: {} 2023-10-18T20:30:48.286 DEBUG:boto:Token: None 2023-10-18T20:30:48.286 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.287 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e43ab2e6029ae3a-0065304078-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.291 DEBUG:boto:b'bofrnf-58obj000000000000000000261000falseobj000000000000000000262023-10-18T20:30:32.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.291 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.292 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.292 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.293 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.293 DEBUG:boto:Method: GET 2023-10-18T20:30:48.294 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.294 DEBUG:boto:Data: 2023-10-18T20:30:48.295 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.295 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.296 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.296 DEBUG:boto:Params: {} 2023-10-18T20:30:48.297 DEBUG:boto:Token: None 2023-10-18T20:30:48.297 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.298 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037adc2f702ff7cce-0065304078-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.303 DEBUG:boto:b'bofrnf-58obj000000000000000000261000falseobj000000000000000000262023-10-18T20:30:32.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.304 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.304 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.305 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.305 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.306 DEBUG:boto:Data: 2023-10-18T20:30:48.306 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.307 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.307 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.307 DEBUG:boto:Params: {} 2023-10-18T20:30:48.308 DEBUG:boto:Token: None 2023-10-18T20:30:48.308 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.309 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Il9r5seF3/SKKJN5mmdtOawSado= 2023-10-18T20:30:48.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Il9r5seF3/SKKJN5mmdtOawSado=', 'Content-Length': '0'} 2023-10-18T20:30:48.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a35293daf78a8305-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.312 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.312 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.313 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.313 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.314 DEBUG:boto:Data: 2023-10-18T20:30:48.314 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.315 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.315 DEBUG:boto:Params: {} 2023-10-18T20:30:48.316 DEBUG:boto:Token: None 2023-10-18T20:30:48.316 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.317 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Il9r5seF3/SKKJN5mmdtOawSado= 2023-10-18T20:30:48.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Il9r5seF3/SKKJN5mmdtOawSado=', 'Content-Length': '0'} 2023-10-18T20:30:48.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee5ff9bc848d6ca2-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-18T20:30:48.320 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.321 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.321 DEBUG:boto:Method: GET 2023-10-18T20:30:48.322 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.322 DEBUG:boto:Data: 2023-10-18T20:30:48.323 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.323 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.324 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.324 DEBUG:boto:Params: {} 2023-10-18T20:30:48.325 DEBUG:boto:Token: None 2023-10-18T20:30:48.325 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.326 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aOZKqEncYHj4jDlguIECneecaao= 2023-10-18T20:30:48.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aOZKqEncYHj4jDlguIECneecaao=', 'Content-Length': '0'} 2023-10-18T20:30:48.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b46091618f13d0-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.370 DEBUG:boto:path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.371 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.371 DEBUG:boto:Method: GET 2023-10-18T20:30:48.372 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.372 DEBUG:boto:Data: 2023-10-18T20:30:48.373 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.373 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.373 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.374 DEBUG:boto:Params: {} 2023-10-18T20:30:48.374 DEBUG:boto:Token: None 2023-10-18T20:30:48.375 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000026 2023-10-18T20:30:48.375 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aOZKqEncYHj4jDlguIECneecaao= 2023-10-18T20:30:48.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aOZKqEncYHj4jDlguIECneecaao=', 'Content-Length': '0'} 2023-10-18T20:30:48.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012961522ac33182f-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.378 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.379 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.379 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.380 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.380 DEBUG:boto:Method: GET 2023-10-18T20:30:48.380 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.381 DEBUG:boto:Data: 2023-10-18T20:30:48.381 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.382 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.382 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.383 DEBUG:boto:Params: {} 2023-10-18T20:30:48.384 DEBUG:boto:Token: None 2023-10-18T20:30:48.384 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.385 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000def90a3c643d6030-0065304078-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.388 DEBUG:boto:b'bofrnf-58obj000000000000000000261000falseobj000000000000000000262023-10-18T20:30:32.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.389 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.390 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.390 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.391 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.391 DEBUG:boto:Method: GET 2023-10-18T20:30:48.391 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000026 2023-10-18T20:30:48.392 DEBUG:boto:Data: 2023-10-18T20:30:48.392 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.393 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.393 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.394 DEBUG:boto:Params: {} 2023-10-18T20:30:48.394 DEBUG:boto:Token: None 2023-10-18T20:30:48.395 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.395 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075efcdf7cdb932e7-0065304078-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.401 DEBUG:boto:b'bofrnf-58obj000000000000000000261000falseobj000000000000000000262023-10-18T20:30:32.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-18T20:30:48.402 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.402 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.403 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.403 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.404 DEBUG:boto:Method: GET 2023-10-18T20:30:48.404 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.405 DEBUG:boto:Data: 2023-10-18T20:30:48.405 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.406 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.406 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.407 DEBUG:boto:Params: {} 2023-10-18T20:30:48.407 DEBUG:boto:Token: None 2023-10-18T20:30:48.408 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.408 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q308DYfyrApoK2+rKL+/MVMjceM= 2023-10-18T20:30:48.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q308DYfyrApoK2+rKL+/MVMjceM=', 'Content-Length': '0'} 2023-10-18T20:30:48.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f7c83d7d1760662-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.412 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.412 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.413 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.413 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.414 DEBUG:boto:Method: GET 2023-10-18T20:30:48.414 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.415 DEBUG:boto:Data: 2023-10-18T20:30:48.415 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.416 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.416 DEBUG:boto:Params: {} 2023-10-18T20:30:48.417 DEBUG:boto:Token: None 2023-10-18T20:30:48.417 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q308DYfyrApoK2+rKL+/MVMjceM= 2023-10-18T20:30:48.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q308DYfyrApoK2+rKL+/MVMjceM=', 'Content-Length': '0'} 2023-10-18T20:30:48.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df82a26874e1e175-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.421 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.422 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.422 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.423 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.423 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.423 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.424 DEBUG:boto:Data: 2023-10-18T20:30:48.424 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.424 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.425 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.425 DEBUG:boto:Params: {} 2023-10-18T20:30:48.426 DEBUG:boto:Token: None 2023-10-18T20:30:48.426 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.427 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SpiovwO8OrPZ3W+15eB0+BRQdOg= 2023-10-18T20:30:48.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SpiovwO8OrPZ3W+15eB0+BRQdOg=', 'Content-Length': '0'} 2023-10-18T20:30:48.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cc51d2c8eea904c-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.430 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.430 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.431 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.431 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.432 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.432 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.433 DEBUG:boto:Data: 2023-10-18T20:30:48.433 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.434 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.434 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.435 DEBUG:boto:Params: {} 2023-10-18T20:30:48.435 DEBUG:boto:Token: None 2023-10-18T20:30:48.436 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.436 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SpiovwO8OrPZ3W+15eB0+BRQdOg= 2023-10-18T20:30:48.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SpiovwO8OrPZ3W+15eB0+BRQdOg=', 'Content-Length': '0'} 2023-10-18T20:30:48.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008446b110690cb1d8-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-18T20:30:48.439 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.440 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.441 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.441 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.442 DEBUG:boto:Method: GET 2023-10-18T20:30:48.442 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.443 DEBUG:boto:Data: 2023-10-18T20:30:48.443 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.444 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.444 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.444 DEBUG:boto:Params: {} 2023-10-18T20:30:48.445 DEBUG:boto:Token: None 2023-10-18T20:30:48.445 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.446 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q308DYfyrApoK2+rKL+/MVMjceM= 2023-10-18T20:30:48.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q308DYfyrApoK2+rKL+/MVMjceM=', 'Content-Length': '0'} 2023-10-18T20:30:48.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caddc2e01ed65553-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.490 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.491 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.491 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.492 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.492 DEBUG:boto:Method: GET 2023-10-18T20:30:48.493 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.493 DEBUG:boto:Data: 2023-10-18T20:30:48.493 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.494 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.495 DEBUG:boto:Params: {} 2023-10-18T20:30:48.495 DEBUG:boto:Token: None 2023-10-18T20:30:48.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027?versionId=null 2023-10-18T20:30:48.496 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q308DYfyrApoK2+rKL+/MVMjceM= 2023-10-18T20:30:48.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q308DYfyrApoK2+rKL+/MVMjceM=', 'Content-Length': '0'} 2023-10-18T20:30:48.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008288252f66641e04-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.500 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.500 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.500 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.501 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.501 DEBUG:boto:Method: GET 2023-10-18T20:30:48.502 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.502 DEBUG:boto:Data: 2023-10-18T20:30:48.503 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.504 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.504 DEBUG:boto:Params: {} 2023-10-18T20:30:48.504 DEBUG:boto:Token: None 2023-10-18T20:30:48.505 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd485eadff8645d-0065304078-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.509 DEBUG:boto:b'bofrnf-58obj000000000000000000271000falseobj000000000000000000272023-10-18T20:30:32.295Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.510 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.511 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.511 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.512 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.512 DEBUG:boto:Method: GET 2023-10-18T20:30:48.513 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.513 DEBUG:boto:Data: 2023-10-18T20:30:48.513 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.514 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.514 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.515 DEBUG:boto:Params: {} 2023-10-18T20:30:48.515 DEBUG:boto:Token: None 2023-10-18T20:30:48.516 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024fc30043d789663-0065304078-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.522 DEBUG:boto:b'bofrnf-58obj000000000000000000271000falseobj000000000000000000272023-10-18T20:30:32.295Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.523 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.523 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.524 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.524 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.524 DEBUG:boto:Data: 2023-10-18T20:30:48.525 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.525 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.526 DEBUG:boto:Params: {} 2023-10-18T20:30:48.526 DEBUG:boto:Token: None 2023-10-18T20:30:48.527 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.527 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D6fQYZzFkMiBDkJgE5/KBNrN0zA= 2023-10-18T20:30:48.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D6fQYZzFkMiBDkJgE5/KBNrN0zA=', 'Content-Length': '0'} 2023-10-18T20:30:48.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004872b50aa3ce0571-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.530 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.531 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.531 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.532 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.532 DEBUG:boto:Data: 2023-10-18T20:30:48.533 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.533 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.534 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.534 DEBUG:boto:Params: {} 2023-10-18T20:30:48.535 DEBUG:boto:Token: None 2023-10-18T20:30:48.535 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D6fQYZzFkMiBDkJgE5/KBNrN0zA= 2023-10-18T20:30:48.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D6fQYZzFkMiBDkJgE5/KBNrN0zA=', 'Content-Length': '0'} 2023-10-18T20:30:48.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076dfd3e6e62437f3-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-18T20:30:48.539 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.540 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.540 DEBUG:boto:Method: GET 2023-10-18T20:30:48.541 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.541 DEBUG:boto:Data: 2023-10-18T20:30:48.541 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.542 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.543 DEBUG:boto:Params: {} 2023-10-18T20:30:48.543 DEBUG:boto:Token: None 2023-10-18T20:30:48.544 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ol3kU/qFIaK//yvwOl8rj8UlYHw= 2023-10-18T20:30:48.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ol3kU/qFIaK//yvwOl8rj8UlYHw=', 'Content-Length': '0'} 2023-10-18T20:30:48.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005496fe8e37e9de0e-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.547 DEBUG:boto:path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.548 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.548 DEBUG:boto:Method: GET 2023-10-18T20:30:48.549 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.549 DEBUG:boto:Data: 2023-10-18T20:30:48.550 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.550 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.551 DEBUG:boto:Params: {} 2023-10-18T20:30:48.551 DEBUG:boto:Token: None 2023-10-18T20:30:48.552 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000027 2023-10-18T20:30:48.552 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ol3kU/qFIaK//yvwOl8rj8UlYHw= 2023-10-18T20:30:48.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ol3kU/qFIaK//yvwOl8rj8UlYHw=', 'Content-Length': '0'} 2023-10-18T20:30:48.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d157d7a0f6bf1ec2-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.602 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.603 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.603 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.604 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.604 DEBUG:boto:Method: GET 2023-10-18T20:30:48.605 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.605 DEBUG:boto:Data: 2023-10-18T20:30:48.606 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.606 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.606 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.607 DEBUG:boto:Params: {} 2023-10-18T20:30:48.607 DEBUG:boto:Token: None 2023-10-18T20:30:48.608 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf0390b00a14f5a4-0065304078-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.612 DEBUG:boto:b'bofrnf-58obj000000000000000000271000falseobj000000000000000000272023-10-18T20:30:32.295Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.613 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.613 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.614 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.614 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.615 DEBUG:boto:Method: GET 2023-10-18T20:30:48.615 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000027 2023-10-18T20:30:48.616 DEBUG:boto:Data: 2023-10-18T20:30:48.616 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.617 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.617 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.617 DEBUG:boto:Params: {} 2023-10-18T20:30:48.618 DEBUG:boto:Token: None 2023-10-18T20:30:48.618 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.619 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005240b4c36335a5fa-0065304078-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.624 DEBUG:boto:b'bofrnf-58obj000000000000000000271000falseobj000000000000000000272023-10-18T20:30:32.295Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-18T20:30:48.626 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.626 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.626 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.627 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.627 DEBUG:boto:Method: GET 2023-10-18T20:30:48.628 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.628 DEBUG:boto:Data: 2023-10-18T20:30:48.629 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.629 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.629 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.630 DEBUG:boto:Params: {} 2023-10-18T20:30:48.630 DEBUG:boto:Token: None 2023-10-18T20:30:48.631 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.631 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zoL7Ybrnr/WqhicQJHZkStMfqFc= 2023-10-18T20:30:48.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zoL7Ybrnr/WqhicQJHZkStMfqFc=', 'Content-Length': '0'} 2023-10-18T20:30:48.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a03d7ca1c8a7360-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.635 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.635 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.636 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.637 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.637 DEBUG:boto:Method: GET 2023-10-18T20:30:48.637 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.638 DEBUG:boto:Data: 2023-10-18T20:30:48.638 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.639 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.639 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.640 DEBUG:boto:Params: {} 2023-10-18T20:30:48.640 DEBUG:boto:Token: None 2023-10-18T20:30:48.641 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.641 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zoL7Ybrnr/WqhicQJHZkStMfqFc= 2023-10-18T20:30:48.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zoL7Ybrnr/WqhicQJHZkStMfqFc=', 'Content-Length': '0'} 2023-10-18T20:30:48.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a73668c9ad5c1f73-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.644 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.645 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.645 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.646 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.646 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.647 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.647 DEBUG:boto:Data: 2023-10-18T20:30:48.647 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.648 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.648 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.649 DEBUG:boto:Params: {} 2023-10-18T20:30:48.649 DEBUG:boto:Token: None 2023-10-18T20:30:48.649 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.650 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X3aAdhb2uJPSShHQKv/x/Xva5v0= 2023-10-18T20:30:48.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X3aAdhb2uJPSShHQKv/x/Xva5v0=', 'Content-Length': '0'} 2023-10-18T20:30:48.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050bc9f625c16d362-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.653 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.653 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.654 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.654 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.655 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.655 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.655 DEBUG:boto:Data: 2023-10-18T20:30:48.656 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.656 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.657 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.657 DEBUG:boto:Params: {} 2023-10-18T20:30:48.658 DEBUG:boto:Token: None 2023-10-18T20:30:48.658 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.659 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X3aAdhb2uJPSShHQKv/x/Xva5v0= 2023-10-18T20:30:48.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X3aAdhb2uJPSShHQKv/x/Xva5v0=', 'Content-Length': '0'} 2023-10-18T20:30:48.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b44ee6007e854f5-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-18T20:30:48.662 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.662 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.663 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.663 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.664 DEBUG:boto:Method: GET 2023-10-18T20:30:48.664 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.664 DEBUG:boto:Data: 2023-10-18T20:30:48.665 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.665 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.666 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.666 DEBUG:boto:Params: {} 2023-10-18T20:30:48.667 DEBUG:boto:Token: None 2023-10-18T20:30:48.667 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zoL7Ybrnr/WqhicQJHZkStMfqFc= 2023-10-18T20:30:48.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zoL7Ybrnr/WqhicQJHZkStMfqFc=', 'Content-Length': '0'} 2023-10-18T20:30:48.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b97605d1018497-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.714 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.715 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.715 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.716 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.716 DEBUG:boto:Method: GET 2023-10-18T20:30:48.717 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.717 DEBUG:boto:Data: 2023-10-18T20:30:48.718 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.718 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.719 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.719 DEBUG:boto:Params: {} 2023-10-18T20:30:48.719 DEBUG:boto:Token: None 2023-10-18T20:30:48.720 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028?versionId=null 2023-10-18T20:30:48.720 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zoL7Ybrnr/WqhicQJHZkStMfqFc= 2023-10-18T20:30:48.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zoL7Ybrnr/WqhicQJHZkStMfqFc=', 'Content-Length': '0'} 2023-10-18T20:30:48.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009041cfab604d2996-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.724 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.724 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.725 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.725 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.726 DEBUG:boto:Method: GET 2023-10-18T20:30:48.726 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.726 DEBUG:boto:Data: 2023-10-18T20:30:48.727 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.727 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.728 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.728 DEBUG:boto:Params: {} 2023-10-18T20:30:48.729 DEBUG:boto:Token: None 2023-10-18T20:30:48.729 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.730 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b4f07f4f5ec6222-0065304078-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.734 DEBUG:boto:b'bofrnf-58obj000000000000000000281000falseobj000000000000000000282023-10-18T20:30:26.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.735 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.735 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.736 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.736 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.737 DEBUG:boto:Method: GET 2023-10-18T20:30:48.737 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.738 DEBUG:boto:Data: 2023-10-18T20:30:48.738 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.739 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.739 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.739 DEBUG:boto:Params: {} 2023-10-18T20:30:48.740 DEBUG:boto:Token: None 2023-10-18T20:30:48.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.741 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4ec6acac6ff8673-0065304078-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.786 DEBUG:boto:b'bofrnf-58obj000000000000000000281000falseobj000000000000000000282023-10-18T20:30:26.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.787 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.788 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.788 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.789 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.789 DEBUG:boto:Data: 2023-10-18T20:30:48.789 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.790 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.791 DEBUG:boto:Params: {} 2023-10-18T20:30:48.791 DEBUG:boto:Token: None 2023-10-18T20:30:48.792 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4PHFVxUWXqUvybDIe1FeOD6HJqo= 2023-10-18T20:30:48.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4PHFVxUWXqUvybDIe1FeOD6HJqo=', 'Content-Length': '0'} 2023-10-18T20:30:48.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094860c5f8716d5fe-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.796 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.796 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.797 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.797 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.798 DEBUG:boto:Data: 2023-10-18T20:30:48.798 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.799 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.799 DEBUG:boto:Params: {} 2023-10-18T20:30:48.801 DEBUG:boto:Token: None 2023-10-18T20:30:48.802 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.802 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4PHFVxUWXqUvybDIe1FeOD6HJqo= 2023-10-18T20:30:48.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4PHFVxUWXqUvybDIe1FeOD6HJqo=', 'Content-Length': '0'} 2023-10-18T20:30:48.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005affa16f9d825e50-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-18T20:30:48.806 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.806 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.807 DEBUG:boto:Method: GET 2023-10-18T20:30:48.807 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.808 DEBUG:boto:Data: 2023-10-18T20:30:48.808 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.809 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.809 DEBUG:boto:Params: {} 2023-10-18T20:30:48.810 DEBUG:boto:Token: None 2023-10-18T20:30:48.810 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ab3cdn0nMgc5ZoCpfGwtXAHJzNc= 2023-10-18T20:30:48.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ab3cdn0nMgc5ZoCpfGwtXAHJzNc=', 'Content-Length': '0'} 2023-10-18T20:30:48.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aa272bbd3adc880-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.814 DEBUG:boto:path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.815 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.815 DEBUG:boto:Method: GET 2023-10-18T20:30:48.816 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.816 DEBUG:boto:Data: 2023-10-18T20:30:48.817 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.817 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.818 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.818 DEBUG:boto:Params: {} 2023-10-18T20:30:48.819 DEBUG:boto:Token: None 2023-10-18T20:30:48.819 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000028 2023-10-18T20:30:48.820 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ab3cdn0nMgc5ZoCpfGwtXAHJzNc= 2023-10-18T20:30:48.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ab3cdn0nMgc5ZoCpfGwtXAHJzNc=', 'Content-Length': '0'} 2023-10-18T20:30:48.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cebda96a5087dcd6-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.823 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.823 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.824 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.824 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.825 DEBUG:boto:Method: GET 2023-10-18T20:30:48.825 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.826 DEBUG:boto:Data: 2023-10-18T20:30:48.826 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.827 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.827 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.827 DEBUG:boto:Params: {} 2023-10-18T20:30:48.828 DEBUG:boto:Token: None 2023-10-18T20:30:48.829 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.829 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084f8047059f8ec4e-0065304078-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.833 DEBUG:boto:b'bofrnf-58obj000000000000000000281000falseobj000000000000000000282023-10-18T20:30:26.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.833 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.834 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.834 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.835 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.835 DEBUG:boto:Method: GET 2023-10-18T20:30:48.836 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000028 2023-10-18T20:30:48.836 DEBUG:boto:Data: 2023-10-18T20:30:48.837 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.838 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.838 DEBUG:boto:Params: {} 2023-10-18T20:30:48.839 DEBUG:boto:Token: None 2023-10-18T20:30:48.839 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006af61c30db92ef1f-0065304078-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.845 DEBUG:boto:b'bofrnf-58obj000000000000000000281000falseobj000000000000000000282023-10-18T20:30:26.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-18T20:30:48.846 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.846 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.847 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.847 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.848 DEBUG:boto:Method: GET 2023-10-18T20:30:48.848 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.848 DEBUG:boto:Data: 2023-10-18T20:30:48.849 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.849 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.850 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.850 DEBUG:boto:Params: {} 2023-10-18T20:30:48.851 DEBUG:boto:Token: None 2023-10-18T20:30:48.851 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.851 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cDWpViUI7CsMPdPEA0jzGbZiMwo= 2023-10-18T20:30:48.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cDWpViUI7CsMPdPEA0jzGbZiMwo=', 'Content-Length': '0'} 2023-10-18T20:30:48.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001af42c853d5d8c63-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.855 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.855 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.856 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.856 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.856 DEBUG:boto:Method: GET 2023-10-18T20:30:48.857 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.857 DEBUG:boto:Data: 2023-10-18T20:30:48.858 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.859 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.859 DEBUG:boto:Params: {} 2023-10-18T20:30:48.860 DEBUG:boto:Token: None 2023-10-18T20:30:48.861 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.861 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cDWpViUI7CsMPdPEA0jzGbZiMwo= 2023-10-18T20:30:48.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cDWpViUI7CsMPdPEA0jzGbZiMwo=', 'Content-Length': '0'} 2023-10-18T20:30:48.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048954c956159f4c3-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.907 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.907 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.907 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.908 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.908 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.909 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.909 DEBUG:boto:Data: 2023-10-18T20:30:48.909 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.910 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.910 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.911 DEBUG:boto:Params: {} 2023-10-18T20:30:48.911 DEBUG:boto:Token: None 2023-10-18T20:30:48.911 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.912 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ov1g5jDV1vJTiJ1dDgmi936Nj7w= 2023-10-18T20:30:48.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ov1g5jDV1vJTiJ1dDgmi936Nj7w=', 'Content-Length': '0'} 2023-10-18T20:30:48.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b7f2c036649d78c-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.915 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.915 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.916 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.916 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.917 DEBUG:boto:Method: HEAD 2023-10-18T20:30:48.917 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.917 DEBUG:boto:Data: 2023-10-18T20:30:48.918 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.918 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.919 DEBUG:boto:Params: {} 2023-10-18T20:30:48.919 DEBUG:boto:Token: None 2023-10-18T20:30:48.920 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ov1g5jDV1vJTiJ1dDgmi936Nj7w= 2023-10-18T20:30:48.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ov1g5jDV1vJTiJ1dDgmi936Nj7w=', 'Content-Length': '0'} 2023-10-18T20:30:48.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064f4bfe7ecf4e0dc-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-18T20:30:48.923 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.924 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.924 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.924 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.925 DEBUG:boto:Method: GET 2023-10-18T20:30:48.925 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.926 DEBUG:boto:Data: 2023-10-18T20:30:48.926 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.927 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.927 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.927 DEBUG:boto:Params: {} 2023-10-18T20:30:48.928 DEBUG:boto:Token: None 2023-10-18T20:30:48.928 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cDWpViUI7CsMPdPEA0jzGbZiMwo= 2023-10-18T20:30:48.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cDWpViUI7CsMPdPEA0jzGbZiMwo=', 'Content-Length': '0'} 2023-10-18T20:30:48.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b4c9917567c076a-0065304078-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.974 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.975 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:48.975 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.976 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.976 DEBUG:boto:Method: GET 2023-10-18T20:30:48.976 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.977 DEBUG:boto:Data: 2023-10-18T20:30:48.977 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.978 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.978 DEBUG:boto:Params: {} 2023-10-18T20:30:48.979 DEBUG:boto:Token: None 2023-10-18T20:30:48.979 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/obj00000000000000000029?versionId=null 2023-10-18T20:30:48.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cDWpViUI7CsMPdPEA0jzGbZiMwo= 2023-10-18T20:30:48.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cDWpViUI7CsMPdPEA0jzGbZiMwo=', 'Content-Length': '0'} 2023-10-18T20:30:48.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000609bdf63305fe6f4-0065304078-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.983 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.983 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.984 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:48.984 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:48.984 DEBUG:boto:Method: GET 2023-10-18T20:30:48.985 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:48.985 DEBUG:boto:Data: 2023-10-18T20:30:48.986 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.986 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:48.986 DEBUG:boto:Port: 8001 2023-10-18T20:30:48.987 DEBUG:boto:Params: {} 2023-10-18T20:30:48.987 DEBUG:boto:Token: None 2023-10-18T20:30:48.988 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.988 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:48.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e16173fb0488b0df-0065304078-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:48.992 DEBUG:boto:b'bofrnf-58obj000000000000000000291000falseobj000000000000000000292023-10-18T20:30:31.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:48.992 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:48.993 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:48.993 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:48.994 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:48.994 DEBUG:boto:Method: GET 2023-10-18T20:30:48.994 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:48.995 DEBUG:boto:Data: 2023-10-18T20:30:48.995 DEBUG:boto:Headers: {} 2023-10-18T20:30:48.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:48.996 DEBUG:boto:Port: 8000 2023-10-18T20:30:48.997 DEBUG:boto:Params: {} 2023-10-18T20:30:48.997 DEBUG:boto:Token: None 2023-10-18T20:30:48.998 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:48 GMT /bofrnf-58/ 2023-10-18T20:30:48.998 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM= 2023-10-18T20:30:48.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EX/RvTM0JbE0sM5TY8cDkzaZWMM=', 'Content-Length': '0'} 2023-10-18T20:30:49.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9914048d7e46993-0065304078-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.003 DEBUG:boto:b'bofrnf-58obj000000000000000000291000falseobj000000000000000000292023-10-18T20:30:31.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.005 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.006 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.006 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.007 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.007 DEBUG:boto:Data: 2023-10-18T20:30:49.008 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.008 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.009 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.009 DEBUG:boto:Params: {} 2023-10-18T20:30:49.010 DEBUG:boto:Token: None 2023-10-18T20:30:49.010 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.011 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5IArrhWZsGgxsDmfWhJfETSS0zA= 2023-10-18T20:30:49.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5IArrhWZsGgxsDmfWhJfETSS0zA=', 'Content-Length': '0'} 2023-10-18T20:30:49.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f2737adf2f12c1-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.014 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.014 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.015 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.015 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.015 DEBUG:boto:Data: 2023-10-18T20:30:49.016 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.016 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.017 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.017 DEBUG:boto:Params: {} 2023-10-18T20:30:49.018 DEBUG:boto:Token: None 2023-10-18T20:30:49.018 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.019 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5IArrhWZsGgxsDmfWhJfETSS0zA= 2023-10-18T20:30:49.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5IArrhWZsGgxsDmfWhJfETSS0zA=', 'Content-Length': '0'} 2023-10-18T20:30:49.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c290909a3875b5-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-18T20:30:49.022 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.023 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.023 DEBUG:boto:Method: GET 2023-10-18T20:30:49.024 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.024 DEBUG:boto:Data: 2023-10-18T20:30:49.025 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.025 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.025 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.026 DEBUG:boto:Params: {} 2023-10-18T20:30:49.026 DEBUG:boto:Token: None 2023-10-18T20:30:49.027 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.027 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QuVWzBvfxONEo2e+OuzE6PJtr+I= 2023-10-18T20:30:49.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QuVWzBvfxONEo2e+OuzE6PJtr+I=', 'Content-Length': '0'} 2023-10-18T20:30:49.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ce7c21ed36014a-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.030 DEBUG:boto:path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.031 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.031 DEBUG:boto:Method: GET 2023-10-18T20:30:49.032 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.032 DEBUG:boto:Data: 2023-10-18T20:30:49.033 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.033 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.034 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.034 DEBUG:boto:Params: {} 2023-10-18T20:30:49.035 DEBUG:boto:Token: None 2023-10-18T20:30:49.035 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000029 2023-10-18T20:30:49.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QuVWzBvfxONEo2e+OuzE6PJtr+I= 2023-10-18T20:30:49.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QuVWzBvfxONEo2e+OuzE6PJtr+I=', 'Content-Length': '0'} 2023-10-18T20:30:49.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31fedaf3dde9a38-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.039 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.039 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.040 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:49.040 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:49.040 DEBUG:boto:Method: GET 2023-10-18T20:30:49.041 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:49.041 DEBUG:boto:Data: 2023-10-18T20:30:49.042 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.042 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.043 DEBUG:boto:Params: {} 2023-10-18T20:30:49.043 DEBUG:boto:Token: None 2023-10-18T20:30:49.044 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007830add8a70a0f41-0065304079-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.048 DEBUG:boto:b'bofrnf-58obj000000000000000000291000falseobj000000000000000000292023-10-18T20:30:31.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.048 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.049 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.049 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:49.050 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:49.050 DEBUG:boto:Method: GET 2023-10-18T20:30:49.051 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000029 2023-10-18T20:30:49.051 DEBUG:boto:Data: 2023-10-18T20:30:49.052 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.052 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.052 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.053 DEBUG:boto:Params: {} 2023-10-18T20:30:49.053 DEBUG:boto:Token: None 2023-10-18T20:30:49.054 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.054 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d304ac496386d2af-0065304079-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.059 DEBUG:boto:b'bofrnf-58obj000000000000000000291000falseobj000000000000000000292023-10-18T20:30:31.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-18T20:30:49.061 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.061 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.062 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.062 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.062 DEBUG:boto:Method: GET 2023-10-18T20:30:49.063 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.063 DEBUG:boto:Data: 2023-10-18T20:30:49.064 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.065 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.065 DEBUG:boto:Params: {} 2023-10-18T20:30:49.066 DEBUG:boto:Token: None 2023-10-18T20:30:49.066 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.067 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:anz5W2OuCCtSqrVyMBBUrGwQf8A= 2023-10-18T20:30:49.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:anz5W2OuCCtSqrVyMBBUrGwQf8A=', 'Content-Length': '0'} 2023-10-18T20:30:49.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad65fdc5d05a6c49-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.110 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.111 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.112 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.112 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.112 DEBUG:boto:Method: GET 2023-10-18T20:30:49.113 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.113 DEBUG:boto:Data: 2023-10-18T20:30:49.114 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.114 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.115 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.115 DEBUG:boto:Params: {} 2023-10-18T20:30:49.116 DEBUG:boto:Token: None 2023-10-18T20:30:49.116 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.117 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:anz5W2OuCCtSqrVyMBBUrGwQf8A= 2023-10-18T20:30:49.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:anz5W2OuCCtSqrVyMBBUrGwQf8A=', 'Content-Length': '0'} 2023-10-18T20:30:49.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095f3c6ae97251f4b-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.120 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.120 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.121 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.121 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.122 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.122 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.123 DEBUG:boto:Data: 2023-10-18T20:30:49.123 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.123 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.124 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.124 DEBUG:boto:Params: {} 2023-10-18T20:30:49.125 DEBUG:boto:Token: None 2023-10-18T20:30:49.125 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.126 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jiq6ZaAVDdOWRXHi4jmjK6RbqdM= 2023-10-18T20:30:49.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jiq6ZaAVDdOWRXHi4jmjK6RbqdM=', 'Content-Length': '0'} 2023-10-18T20:30:49.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041348b73bc8db35d-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.129 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.129 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.129 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.130 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.130 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.131 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.131 DEBUG:boto:Data: 2023-10-18T20:30:49.131 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.132 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.132 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.133 DEBUG:boto:Params: {} 2023-10-18T20:30:49.133 DEBUG:boto:Token: None 2023-10-18T20:30:49.133 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.134 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jiq6ZaAVDdOWRXHi4jmjK6RbqdM= 2023-10-18T20:30:49.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jiq6ZaAVDdOWRXHi4jmjK6RbqdM=', 'Content-Length': '0'} 2023-10-18T20:30:49.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005987b006d04dd974-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-18T20:30:49.137 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.138 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.138 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.139 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.139 DEBUG:boto:Method: GET 2023-10-18T20:30:49.139 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.140 DEBUG:boto:Data: 2023-10-18T20:30:49.140 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.141 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.142 DEBUG:boto:Params: {} 2023-10-18T20:30:49.142 DEBUG:boto:Token: None 2023-10-18T20:30:49.143 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.143 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:anz5W2OuCCtSqrVyMBBUrGwQf8A= 2023-10-18T20:30:49.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:anz5W2OuCCtSqrVyMBBUrGwQf8A=', 'Content-Length': '0'} 2023-10-18T20:30:49.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042e1e318ec8279c7-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.146 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.147 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.147 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.148 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.148 DEBUG:boto:Method: GET 2023-10-18T20:30:49.149 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.149 DEBUG:boto:Data: 2023-10-18T20:30:49.149 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.150 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.151 DEBUG:boto:Params: {} 2023-10-18T20:30:49.151 DEBUG:boto:Token: None 2023-10-18T20:30:49.152 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030?versionId=null 2023-10-18T20:30:49.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:anz5W2OuCCtSqrVyMBBUrGwQf8A= 2023-10-18T20:30:49.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:anz5W2OuCCtSqrVyMBBUrGwQf8A=', 'Content-Length': '0'} 2023-10-18T20:30:49.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a9b335b7c2704aa-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.198 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.199 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.200 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.200 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.200 DEBUG:boto:Method: GET 2023-10-18T20:30:49.201 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.201 DEBUG:boto:Data: 2023-10-18T20:30:49.202 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.202 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.202 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.203 DEBUG:boto:Params: {} 2023-10-18T20:30:49.203 DEBUG:boto:Token: None 2023-10-18T20:30:49.204 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.204 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f141872a45406e6-0065304079-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.208 DEBUG:boto:b'bofrnf-58obj000000000000000000301000falseobj000000000000000000302023-10-18T20:30:32.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.209 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.209 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.210 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.210 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.211 DEBUG:boto:Method: GET 2023-10-18T20:30:49.211 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.212 DEBUG:boto:Data: 2023-10-18T20:30:49.212 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.212 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.213 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.213 DEBUG:boto:Params: {} 2023-10-18T20:30:49.214 DEBUG:boto:Token: None 2023-10-18T20:30:49.214 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc0ee1c899da3f41-0065304079-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.220 DEBUG:boto:b'bofrnf-58obj000000000000000000301000falseobj000000000000000000302023-10-18T20:30:32.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.221 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.221 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.222 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.222 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.223 DEBUG:boto:Data: 2023-10-18T20:30:49.223 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.223 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.224 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.224 DEBUG:boto:Params: {} 2023-10-18T20:30:49.225 DEBUG:boto:Token: None 2023-10-18T20:30:49.225 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.225 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JRQRUx6i7f/y671cT0g+Qfig5YQ= 2023-10-18T20:30:49.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JRQRUx6i7f/y671cT0g+Qfig5YQ=', 'Content-Length': '0'} 2023-10-18T20:30:49.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029f676b6bffe6297-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.229 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.229 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.230 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.230 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.230 DEBUG:boto:Data: 2023-10-18T20:30:49.231 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.231 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.231 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.232 DEBUG:boto:Params: {} 2023-10-18T20:30:49.232 DEBUG:boto:Token: None 2023-10-18T20:30:49.233 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.233 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JRQRUx6i7f/y671cT0g+Qfig5YQ= 2023-10-18T20:30:49.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JRQRUx6i7f/y671cT0g+Qfig5YQ=', 'Content-Length': '0'} 2023-10-18T20:30:49.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d280da98c92611c-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-18T20:30:49.237 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.237 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.238 DEBUG:boto:Method: GET 2023-10-18T20:30:49.238 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.238 DEBUG:boto:Data: 2023-10-18T20:30:49.239 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.239 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.240 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.240 DEBUG:boto:Params: {} 2023-10-18T20:30:49.241 DEBUG:boto:Token: None 2023-10-18T20:30:49.241 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.242 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BN7CvpLaQ0Rqsp5d4ggpRhI+h1o= 2023-10-18T20:30:49.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BN7CvpLaQ0Rqsp5d4ggpRhI+h1o=', 'Content-Length': '0'} 2023-10-18T20:30:49.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f27f28112a5d68c-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.245 DEBUG:boto:path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.245 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.246 DEBUG:boto:Method: GET 2023-10-18T20:30:49.246 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.247 DEBUG:boto:Data: 2023-10-18T20:30:49.247 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.247 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.248 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.248 DEBUG:boto:Params: {} 2023-10-18T20:30:49.249 DEBUG:boto:Token: None 2023-10-18T20:30:49.249 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000030 2023-10-18T20:30:49.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BN7CvpLaQ0Rqsp5d4ggpRhI+h1o= 2023-10-18T20:30:49.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BN7CvpLaQ0Rqsp5d4ggpRhI+h1o=', 'Content-Length': '0'} 2023-10-18T20:30:49.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050af168ceb28051d-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.253 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.253 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.254 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.254 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.255 DEBUG:boto:Method: GET 2023-10-18T20:30:49.255 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.255 DEBUG:boto:Data: 2023-10-18T20:30:49.256 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.256 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.257 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.257 DEBUG:boto:Params: {} 2023-10-18T20:30:49.258 DEBUG:boto:Token: None 2023-10-18T20:30:49.258 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.259 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9d6fb691ace22c0-0065304079-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.262 DEBUG:boto:b'bofrnf-58obj000000000000000000301000falseobj000000000000000000302023-10-18T20:30:32.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.263 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.263 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.264 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.264 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.265 DEBUG:boto:Method: GET 2023-10-18T20:30:49.265 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000030 2023-10-18T20:30:49.265 DEBUG:boto:Data: 2023-10-18T20:30:49.266 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.266 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.267 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.267 DEBUG:boto:Params: {} 2023-10-18T20:30:49.268 DEBUG:boto:Token: None 2023-10-18T20:30:49.268 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.269 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba9c5ec5f5b583d6-0065304079-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.273 DEBUG:boto:b'bofrnf-58obj000000000000000000301000falseobj000000000000000000302023-10-18T20:30:32.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-18T20:30:49.274 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.275 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.275 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.276 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.276 DEBUG:boto:Method: GET 2023-10-18T20:30:49.277 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.277 DEBUG:boto:Data: 2023-10-18T20:30:49.278 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.278 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.278 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.279 DEBUG:boto:Params: {} 2023-10-18T20:30:49.279 DEBUG:boto:Token: None 2023-10-18T20:30:49.280 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.280 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OxDKDjbjYCNMHlve4quS0467GbI= 2023-10-18T20:30:49.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OxDKDjbjYCNMHlve4quS0467GbI=', 'Content-Length': '0'} 2023-10-18T20:30:49.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000782120522d1f52d0-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.284 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.284 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.285 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.285 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.286 DEBUG:boto:Method: GET 2023-10-18T20:30:49.286 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.287 DEBUG:boto:Data: 2023-10-18T20:30:49.287 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.288 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.289 DEBUG:boto:Params: {} 2023-10-18T20:30:49.289 DEBUG:boto:Token: None 2023-10-18T20:30:49.290 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.290 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OxDKDjbjYCNMHlve4quS0467GbI= 2023-10-18T20:30:49.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OxDKDjbjYCNMHlve4quS0467GbI=', 'Content-Length': '0'} 2023-10-18T20:30:49.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7c2105d4ca0bd7-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.334 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.335 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.335 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.336 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.336 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.337 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.337 DEBUG:boto:Data: 2023-10-18T20:30:49.338 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.339 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.339 DEBUG:boto:Params: {} 2023-10-18T20:30:49.340 DEBUG:boto:Token: None 2023-10-18T20:30:49.340 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.341 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W+uCkfEyzHgi/FKjrgpx88V2rXU= 2023-10-18T20:30:49.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W+uCkfEyzHgi/FKjrgpx88V2rXU=', 'Content-Length': '0'} 2023-10-18T20:30:49.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa4a2d25ce8f51da-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.344 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.344 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.345 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.345 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.346 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.346 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.346 DEBUG:boto:Data: 2023-10-18T20:30:49.347 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.347 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.348 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.348 DEBUG:boto:Params: {} 2023-10-18T20:30:49.349 DEBUG:boto:Token: None 2023-10-18T20:30:49.349 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.349 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W+uCkfEyzHgi/FKjrgpx88V2rXU= 2023-10-18T20:30:49.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W+uCkfEyzHgi/FKjrgpx88V2rXU=', 'Content-Length': '0'} 2023-10-18T20:30:49.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049a167e3e0d30607-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-18T20:30:49.353 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.353 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.354 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.354 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.355 DEBUG:boto:Method: GET 2023-10-18T20:30:49.355 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.356 DEBUG:boto:Data: 2023-10-18T20:30:49.356 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.356 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.357 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.357 DEBUG:boto:Params: {} 2023-10-18T20:30:49.358 DEBUG:boto:Token: None 2023-10-18T20:30:49.358 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.359 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OxDKDjbjYCNMHlve4quS0467GbI= 2023-10-18T20:30:49.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OxDKDjbjYCNMHlve4quS0467GbI=', 'Content-Length': '0'} 2023-10-18T20:30:49.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a703076cd3cad3b-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.402 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.403 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.403 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.404 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.404 DEBUG:boto:Method: GET 2023-10-18T20:30:49.405 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.405 DEBUG:boto:Data: 2023-10-18T20:30:49.406 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.406 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.406 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.407 DEBUG:boto:Params: {} 2023-10-18T20:30:49.407 DEBUG:boto:Token: None 2023-10-18T20:30:49.408 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031?versionId=null 2023-10-18T20:30:49.408 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OxDKDjbjYCNMHlve4quS0467GbI= 2023-10-18T20:30:49.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OxDKDjbjYCNMHlve4quS0467GbI=', 'Content-Length': '0'} 2023-10-18T20:30:49.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3aa646a80a741f2-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.412 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.412 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.413 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.413 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.414 DEBUG:boto:Method: GET 2023-10-18T20:30:49.414 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.414 DEBUG:boto:Data: 2023-10-18T20:30:49.415 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.417 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.417 DEBUG:boto:Params: {} 2023-10-18T20:30:49.418 DEBUG:boto:Token: None 2023-10-18T20:30:49.418 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002aba07998bad402-0065304079-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.423 DEBUG:boto:b'bofrnf-58obj000000000000000000311000falseobj000000000000000000312023-10-18T20:30:29.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.423 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.424 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.424 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.425 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.425 DEBUG:boto:Method: GET 2023-10-18T20:30:49.425 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.426 DEBUG:boto:Data: 2023-10-18T20:30:49.426 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.427 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.427 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.428 DEBUG:boto:Params: {} 2023-10-18T20:30:49.428 DEBUG:boto:Token: None 2023-10-18T20:30:49.429 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.429 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000511227f4b9b303ed-0065304079-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.434 DEBUG:boto:b'bofrnf-58obj000000000000000000311000falseobj000000000000000000312023-10-18T20:30:29.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.435 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.436 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.436 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.436 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.437 DEBUG:boto:Data: 2023-10-18T20:30:49.437 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.437 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.438 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.438 DEBUG:boto:Params: {} 2023-10-18T20:30:49.439 DEBUG:boto:Token: None 2023-10-18T20:30:49.439 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+YGmSFNx11VYPap7OlPW23FydIc= 2023-10-18T20:30:49.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+YGmSFNx11VYPap7OlPW23FydIc=', 'Content-Length': '0'} 2023-10-18T20:30:49.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078cc48bc74474023-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.443 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.443 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.444 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.444 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.444 DEBUG:boto:Data: 2023-10-18T20:30:49.445 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.446 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.446 DEBUG:boto:Params: {} 2023-10-18T20:30:49.447 DEBUG:boto:Token: None 2023-10-18T20:30:49.447 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.447 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+YGmSFNx11VYPap7OlPW23FydIc= 2023-10-18T20:30:49.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+YGmSFNx11VYPap7OlPW23FydIc=', 'Content-Length': '0'} 2023-10-18T20:30:49.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082bb9fcc4d03e69b-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-18T20:30:49.451 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.451 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.452 DEBUG:boto:Method: GET 2023-10-18T20:30:49.452 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.453 DEBUG:boto:Data: 2023-10-18T20:30:49.453 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.454 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.454 DEBUG:boto:Params: {} 2023-10-18T20:30:49.455 DEBUG:boto:Token: None 2023-10-18T20:30:49.455 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:roeijuvt03ck5M3TTsdC89B+Rls= 2023-10-18T20:30:49.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:roeijuvt03ck5M3TTsdC89B+Rls=', 'Content-Length': '0'} 2023-10-18T20:30:49.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff050f0b4e421ab-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.498 DEBUG:boto:path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.499 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.499 DEBUG:boto:Method: GET 2023-10-18T20:30:49.500 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.500 DEBUG:boto:Data: 2023-10-18T20:30:49.501 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.501 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.502 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.502 DEBUG:boto:Params: {} 2023-10-18T20:30:49.503 DEBUG:boto:Token: None 2023-10-18T20:30:49.503 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000031 2023-10-18T20:30:49.504 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:roeijuvt03ck5M3TTsdC89B+Rls= 2023-10-18T20:30:49.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:roeijuvt03ck5M3TTsdC89B+Rls=', 'Content-Length': '0'} 2023-10-18T20:30:49.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f363ab0ddb5b67d6-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.507 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.508 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.508 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.508 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.509 DEBUG:boto:Method: GET 2023-10-18T20:30:49.509 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.510 DEBUG:boto:Data: 2023-10-18T20:30:49.510 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.510 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.511 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.511 DEBUG:boto:Params: {} 2023-10-18T20:30:49.512 DEBUG:boto:Token: None 2023-10-18T20:30:49.512 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.513 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dd1cf0c7b9aa5f1-0065304079-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.516 DEBUG:boto:b'bofrnf-58obj000000000000000000311000falseobj000000000000000000312023-10-18T20:30:29.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.517 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.518 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.518 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.519 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.520 DEBUG:boto:Method: GET 2023-10-18T20:30:49.520 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000031 2023-10-18T20:30:49.520 DEBUG:boto:Data: 2023-10-18T20:30:49.521 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.521 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.522 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.522 DEBUG:boto:Params: {} 2023-10-18T20:30:49.523 DEBUG:boto:Token: None 2023-10-18T20:30:49.523 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064755a289710787e-0065304079-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.570 DEBUG:boto:b'bofrnf-58obj000000000000000000311000falseobj000000000000000000312023-10-18T20:30:29.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-18T20:30:49.572 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.572 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.573 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.573 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.573 DEBUG:boto:Method: GET 2023-10-18T20:30:49.574 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.574 DEBUG:boto:Data: 2023-10-18T20:30:49.575 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.575 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.576 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.576 DEBUG:boto:Params: {} 2023-10-18T20:30:49.576 DEBUG:boto:Token: None 2023-10-18T20:30:49.577 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7V+x5GtSZ9oBd9u49SRVRa1xlw= 2023-10-18T20:30:49.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7V+x5GtSZ9oBd9u49SRVRa1xlw=', 'Content-Length': '0'} 2023-10-18T20:30:49.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4edfa1293c58e37-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.581 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.582 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.582 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.582 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.583 DEBUG:boto:Method: GET 2023-10-18T20:30:49.583 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.584 DEBUG:boto:Data: 2023-10-18T20:30:49.584 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.585 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.585 DEBUG:boto:Params: {} 2023-10-18T20:30:49.586 DEBUG:boto:Token: None 2023-10-18T20:30:49.587 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.587 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7V+x5GtSZ9oBd9u49SRVRa1xlw= 2023-10-18T20:30:49.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7V+x5GtSZ9oBd9u49SRVRa1xlw=', 'Content-Length': '0'} 2023-10-18T20:30:49.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca49f16bcfed2b8e-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.590 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.591 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.591 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.592 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.592 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.593 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.593 DEBUG:boto:Data: 2023-10-18T20:30:49.593 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.594 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.594 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.595 DEBUG:boto:Params: {} 2023-10-18T20:30:49.595 DEBUG:boto:Token: None 2023-10-18T20:30:49.596 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.596 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Sfg2eV6UarAx42pgvh44JY9VV18= 2023-10-18T20:30:49.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Sfg2eV6UarAx42pgvh44JY9VV18=', 'Content-Length': '0'} 2023-10-18T20:30:49.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000347afa8019bca220-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.599 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.599 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.600 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.600 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.601 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.601 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.601 DEBUG:boto:Data: 2023-10-18T20:30:49.602 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.603 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.603 DEBUG:boto:Params: {} 2023-10-18T20:30:49.604 DEBUG:boto:Token: None 2023-10-18T20:30:49.604 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Sfg2eV6UarAx42pgvh44JY9VV18= 2023-10-18T20:30:49.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Sfg2eV6UarAx42pgvh44JY9VV18=', 'Content-Length': '0'} 2023-10-18T20:30:49.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072cbe4f6051bcf1a-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-18T20:30:49.608 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.609 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.609 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.610 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.610 DEBUG:boto:Method: GET 2023-10-18T20:30:49.611 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.611 DEBUG:boto:Data: 2023-10-18T20:30:49.612 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.612 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.613 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.614 DEBUG:boto:Params: {} 2023-10-18T20:30:49.614 DEBUG:boto:Token: None 2023-10-18T20:30:49.615 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7V+x5GtSZ9oBd9u49SRVRa1xlw= 2023-10-18T20:30:49.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7V+x5GtSZ9oBd9u49SRVRa1xlw=', 'Content-Length': '0'} 2023-10-18T20:30:49.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001abbf3b36809a975-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.618 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.621 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.621 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.622 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.622 DEBUG:boto:Method: GET 2023-10-18T20:30:49.623 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.623 DEBUG:boto:Data: 2023-10-18T20:30:49.623 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.624 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.624 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.625 DEBUG:boto:Params: {} 2023-10-18T20:30:49.625 DEBUG:boto:Token: None 2023-10-18T20:30:49.626 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032?versionId=null 2023-10-18T20:30:49.626 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z7V+x5GtSZ9oBd9u49SRVRa1xlw= 2023-10-18T20:30:49.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z7V+x5GtSZ9oBd9u49SRVRa1xlw=', 'Content-Length': '0'} 2023-10-18T20:30:49.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c2e1c1f7db900a1-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.629 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.630 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.630 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.631 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.631 DEBUG:boto:Method: GET 2023-10-18T20:30:49.631 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.632 DEBUG:boto:Data: 2023-10-18T20:30:49.632 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.633 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.633 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.634 DEBUG:boto:Params: {} 2023-10-18T20:30:49.634 DEBUG:boto:Token: None 2023-10-18T20:30:49.635 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.635 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb55b07600143174-0065304079-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.639 DEBUG:boto:b'bofrnf-58obj000000000000000000321000falseobj000000000000000000322023-10-18T20:30:27.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.640 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.640 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.640 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.641 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.641 DEBUG:boto:Method: GET 2023-10-18T20:30:49.642 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.642 DEBUG:boto:Data: 2023-10-18T20:30:49.643 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.643 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.644 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.644 DEBUG:boto:Params: {} 2023-10-18T20:30:49.645 DEBUG:boto:Token: None 2023-10-18T20:30:49.645 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.646 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071232c08b3b4d03f-0065304079-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.651 DEBUG:boto:b'bofrnf-58obj000000000000000000321000falseobj000000000000000000322023-10-18T20:30:27.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.651 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.652 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.652 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.653 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.653 DEBUG:boto:Data: 2023-10-18T20:30:49.654 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.655 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.655 DEBUG:boto:Params: {} 2023-10-18T20:30:49.655 DEBUG:boto:Token: None 2023-10-18T20:30:49.656 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F+o7MzdrmAX0c991vmGTy7GdmXE= 2023-10-18T20:30:49.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F+o7MzdrmAX0c991vmGTy7GdmXE=', 'Content-Length': '0'} 2023-10-18T20:30:49.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a915182cab95ed5-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.659 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.660 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.660 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.660 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.661 DEBUG:boto:Data: 2023-10-18T20:30:49.661 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.662 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.663 DEBUG:boto:Params: {} 2023-10-18T20:30:49.663 DEBUG:boto:Token: None 2023-10-18T20:30:49.664 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F+o7MzdrmAX0c991vmGTy7GdmXE= 2023-10-18T20:30:49.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F+o7MzdrmAX0c991vmGTy7GdmXE=', 'Content-Length': '0'} 2023-10-18T20:30:49.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe3f357fd6d210b-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-18T20:30:49.668 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.669 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.669 DEBUG:boto:Method: GET 2023-10-18T20:30:49.670 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.670 DEBUG:boto:Data: 2023-10-18T20:30:49.670 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.671 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.672 DEBUG:boto:Params: {} 2023-10-18T20:30:49.672 DEBUG:boto:Token: None 2023-10-18T20:30:49.672 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:u+CezXCXxuwzQd+cukCGatxTdUo= 2023-10-18T20:30:49.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:u+CezXCXxuwzQd+cukCGatxTdUo=', 'Content-Length': '0'} 2023-10-18T20:30:49.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acd238d787dd7340-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.722 DEBUG:boto:path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.723 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.724 DEBUG:boto:Method: GET 2023-10-18T20:30:49.724 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.724 DEBUG:boto:Data: 2023-10-18T20:30:49.725 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.726 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.726 DEBUG:boto:Params: {} 2023-10-18T20:30:49.727 DEBUG:boto:Token: None 2023-10-18T20:30:49.727 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000032 2023-10-18T20:30:49.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:u+CezXCXxuwzQd+cukCGatxTdUo= 2023-10-18T20:30:49.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:u+CezXCXxuwzQd+cukCGatxTdUo=', 'Content-Length': '0'} 2023-10-18T20:30:49.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ea146f521eee701-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.731 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.732 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.732 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.733 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.733 DEBUG:boto:Method: GET 2023-10-18T20:30:49.734 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.734 DEBUG:boto:Data: 2023-10-18T20:30:49.735 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.735 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.735 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.736 DEBUG:boto:Params: {} 2023-10-18T20:30:49.737 DEBUG:boto:Token: None 2023-10-18T20:30:49.737 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.738 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000160a1891897de6d0-0065304079-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.741 DEBUG:boto:b'bofrnf-58obj000000000000000000321000falseobj000000000000000000322023-10-18T20:30:27.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.742 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.743 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.743 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.744 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.744 DEBUG:boto:Method: GET 2023-10-18T20:30:49.744 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000032 2023-10-18T20:30:49.745 DEBUG:boto:Data: 2023-10-18T20:30:49.745 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.746 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.746 DEBUG:boto:Params: {} 2023-10-18T20:30:49.747 DEBUG:boto:Token: None 2023-10-18T20:30:49.747 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fe873794c49420f-0065304079-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.753 DEBUG:boto:b'bofrnf-58obj000000000000000000321000falseobj000000000000000000322023-10-18T20:30:27.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-18T20:30:49.754 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.755 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.755 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.756 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.756 DEBUG:boto:Method: GET 2023-10-18T20:30:49.757 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.757 DEBUG:boto:Data: 2023-10-18T20:30:49.758 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.758 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.759 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.759 DEBUG:boto:Params: {} 2023-10-18T20:30:49.760 DEBUG:boto:Token: None 2023-10-18T20:30:49.760 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.761 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZRmv68C7tXRw0baQfCXdvtZg2Q0= 2023-10-18T20:30:49.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZRmv68C7tXRw0baQfCXdvtZg2Q0=', 'Content-Length': '0'} 2023-10-18T20:30:49.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ee5cd8ae6a3cf3-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.765 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.765 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.766 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.766 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.767 DEBUG:boto:Method: GET 2023-10-18T20:30:49.767 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.768 DEBUG:boto:Data: 2023-10-18T20:30:49.768 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.768 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.769 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.769 DEBUG:boto:Params: {} 2023-10-18T20:30:49.770 DEBUG:boto:Token: None 2023-10-18T20:30:49.770 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.771 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZRmv68C7tXRw0baQfCXdvtZg2Q0= 2023-10-18T20:30:49.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZRmv68C7tXRw0baQfCXdvtZg2Q0=', 'Content-Length': '0'} 2023-10-18T20:30:49.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000820609deacd11a8-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.774 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.775 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.775 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.776 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.776 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.776 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.777 DEBUG:boto:Data: 2023-10-18T20:30:49.777 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.778 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.778 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.779 DEBUG:boto:Params: {} 2023-10-18T20:30:49.779 DEBUG:boto:Token: None 2023-10-18T20:30:49.780 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yst1VBSZz1H8JKcZNF2Ly4UJ9Bo= 2023-10-18T20:30:49.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yst1VBSZz1H8JKcZNF2Ly4UJ9Bo=', 'Content-Length': '0'} 2023-10-18T20:30:49.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007983630d5bac4f49-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.783 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.783 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.784 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.784 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.785 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.785 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.786 DEBUG:boto:Data: 2023-10-18T20:30:49.786 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.787 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.787 DEBUG:boto:Params: {} 2023-10-18T20:30:49.788 DEBUG:boto:Token: None 2023-10-18T20:30:49.789 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yst1VBSZz1H8JKcZNF2Ly4UJ9Bo= 2023-10-18T20:30:49.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yst1VBSZz1H8JKcZNF2Ly4UJ9Bo=', 'Content-Length': '0'} 2023-10-18T20:30:49.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b2e0d26cc0bd456-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-18T20:30:49.792 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.793 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.793 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.794 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.794 DEBUG:boto:Method: GET 2023-10-18T20:30:49.795 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.795 DEBUG:boto:Data: 2023-10-18T20:30:49.795 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.796 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.796 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.797 DEBUG:boto:Params: {} 2023-10-18T20:30:49.797 DEBUG:boto:Token: None 2023-10-18T20:30:49.798 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.798 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZRmv68C7tXRw0baQfCXdvtZg2Q0= 2023-10-18T20:30:49.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZRmv68C7tXRw0baQfCXdvtZg2Q0=', 'Content-Length': '0'} 2023-10-18T20:30:49.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe7cc8fd6b247c8-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.802 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.802 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.803 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.803 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.804 DEBUG:boto:Method: GET 2023-10-18T20:30:49.804 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.804 DEBUG:boto:Data: 2023-10-18T20:30:49.805 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.806 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.806 DEBUG:boto:Params: {} 2023-10-18T20:30:49.806 DEBUG:boto:Token: None 2023-10-18T20:30:49.807 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033?versionId=null 2023-10-18T20:30:49.807 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZRmv68C7tXRw0baQfCXdvtZg2Q0= 2023-10-18T20:30:49.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZRmv68C7tXRw0baQfCXdvtZg2Q0=', 'Content-Length': '0'} 2023-10-18T20:30:49.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000687c3085c0bc0a81-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.811 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.811 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.812 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.812 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.812 DEBUG:boto:Method: GET 2023-10-18T20:30:49.813 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.813 DEBUG:boto:Data: 2023-10-18T20:30:49.814 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.815 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.815 DEBUG:boto:Params: {} 2023-10-18T20:30:49.815 DEBUG:boto:Token: None 2023-10-18T20:30:49.816 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab437d410ac5b91-0065304079-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.820 DEBUG:boto:b'bofrnf-58obj000000000000000000331000falseobj000000000000000000332023-10-18T20:30:32.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.821 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.821 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.822 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.822 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.823 DEBUG:boto:Method: GET 2023-10-18T20:30:49.823 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.823 DEBUG:boto:Data: 2023-10-18T20:30:49.824 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.824 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.825 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.825 DEBUG:boto:Params: {} 2023-10-18T20:30:49.826 DEBUG:boto:Token: None 2023-10-18T20:30:49.826 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.827 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d825aa86c129ce39-0065304079-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.878 DEBUG:boto:b'bofrnf-58obj000000000000000000331000falseobj000000000000000000332023-10-18T20:30:32.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.879 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.879 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.880 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.880 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.881 DEBUG:boto:Data: 2023-10-18T20:30:49.881 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.881 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.882 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.882 DEBUG:boto:Params: {} 2023-10-18T20:30:49.883 DEBUG:boto:Token: None 2023-10-18T20:30:49.883 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2PCJ6OQodJmTGMZA7DGZYKm+E70= 2023-10-18T20:30:49.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2PCJ6OQodJmTGMZA7DGZYKm+E70=', 'Content-Length': '0'} 2023-10-18T20:30:49.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d56474d416ebba6-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.887 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.887 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.887 DEBUG:boto:Method: HEAD 2023-10-18T20:30:49.888 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.888 DEBUG:boto:Data: 2023-10-18T20:30:49.889 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.889 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.890 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.890 DEBUG:boto:Params: {} 2023-10-18T20:30:49.891 DEBUG:boto:Token: None 2023-10-18T20:30:49.891 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.892 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2PCJ6OQodJmTGMZA7DGZYKm+E70= 2023-10-18T20:30:49.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2PCJ6OQodJmTGMZA7DGZYKm+E70=', 'Content-Length': '0'} 2023-10-18T20:30:49.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000938c4c4c719b9680-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-18T20:30:49.895 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.896 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.896 DEBUG:boto:Method: GET 2023-10-18T20:30:49.897 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.897 DEBUG:boto:Data: 2023-10-18T20:30:49.898 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.898 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.899 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.899 DEBUG:boto:Params: {} 2023-10-18T20:30:49.900 DEBUG:boto:Token: None 2023-10-18T20:30:49.900 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.901 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YIAP0WQjCcvs63UOqsD3MgtSyM0= 2023-10-18T20:30:49.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YIAP0WQjCcvs63UOqsD3MgtSyM0=', 'Content-Length': '0'} 2023-10-18T20:30:49.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3520fe75419cfa-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.904 DEBUG:boto:path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.904 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.904 DEBUG:boto:Method: GET 2023-10-18T20:30:49.905 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.905 DEBUG:boto:Data: 2023-10-18T20:30:49.906 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.906 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.907 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.907 DEBUG:boto:Params: {} 2023-10-18T20:30:49.907 DEBUG:boto:Token: None 2023-10-18T20:30:49.908 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000033 2023-10-18T20:30:49.908 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YIAP0WQjCcvs63UOqsD3MgtSyM0= 2023-10-18T20:30:49.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YIAP0WQjCcvs63UOqsD3MgtSyM0=', 'Content-Length': '0'} 2023-10-18T20:30:49.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051859ff7f17d7fe7-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.912 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.912 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.913 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.913 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.913 DEBUG:boto:Method: GET 2023-10-18T20:30:49.914 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.914 DEBUG:boto:Data: 2023-10-18T20:30:49.915 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.915 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.916 DEBUG:boto:Params: {} 2023-10-18T20:30:49.916 DEBUG:boto:Token: None 2023-10-18T20:30:49.917 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712b68b9ed253017-0065304079-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.962 DEBUG:boto:b'bofrnf-58obj000000000000000000331000falseobj000000000000000000332023-10-18T20:30:32.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.963 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:49.963 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:49.964 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.964 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.965 DEBUG:boto:Method: GET 2023-10-18T20:30:49.965 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000033 2023-10-18T20:30:49.966 DEBUG:boto:Data: 2023-10-18T20:30:49.966 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.967 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.967 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.967 DEBUG:boto:Params: {} 2023-10-18T20:30:49.968 DEBUG:boto:Token: None 2023-10-18T20:30:49.968 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/ 2023-10-18T20:30:49.969 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw= 2023-10-18T20:30:49.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PXZPSCe59nlAkDdyTKuq4uOvGFw=', 'Content-Length': '0'} 2023-10-18T20:30:49.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047a2d0e305f5dab1-0065304079-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.975 DEBUG:boto:b'bofrnf-58obj000000000000000000331000falseobj000000000000000000332023-10-18T20:30:32.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:49.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-18T20:30:49.976 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:49.977 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:49.977 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:49.977 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:49.978 DEBUG:boto:Method: GET 2023-10-18T20:30:49.978 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:49.979 DEBUG:boto:Data: 2023-10-18T20:30:49.979 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.979 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:49.980 DEBUG:boto:Port: 8001 2023-10-18T20:30:49.980 DEBUG:boto:Params: {} 2023-10-18T20:30:49.981 DEBUG:boto:Token: None 2023-10-18T20:30:49.981 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:49.982 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RUiyfLvpS1W2TM1OyAUBeTFdfOU= 2023-10-18T20:30:49.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RUiyfLvpS1W2TM1OyAUBeTFdfOU=', 'Content-Length': '0'} 2023-10-18T20:30:49.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b6d9c7a04c2070c-0065304079-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:49.985 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:49.986 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:49.986 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:49.987 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:49.987 DEBUG:boto:Method: GET 2023-10-18T20:30:49.988 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:49.988 DEBUG:boto:Data: 2023-10-18T20:30:49.989 DEBUG:boto:Headers: {} 2023-10-18T20:30:49.989 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:49.989 DEBUG:boto:Port: 8000 2023-10-18T20:30:49.990 DEBUG:boto:Params: {} 2023-10-18T20:30:49.990 DEBUG:boto:Token: None 2023-10-18T20:30:49.991 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:49 GMT /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:49.991 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RUiyfLvpS1W2TM1OyAUBeTFdfOU= 2023-10-18T20:30:49.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RUiyfLvpS1W2TM1OyAUBeTFdfOU=', 'Content-Length': '0'} 2023-10-18T20:30:49.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3dd26d328cefe44-0065304079-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.034 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.035 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.035 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.036 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.036 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.036 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.037 DEBUG:boto:Data: 2023-10-18T20:30:50.037 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.038 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.038 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.039 DEBUG:boto:Params: {} 2023-10-18T20:30:50.039 DEBUG:boto:Token: None 2023-10-18T20:30:50.040 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ljdh8a444CiOqJMOTWak3HBrie8= 2023-10-18T20:30:50.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ljdh8a444CiOqJMOTWak3HBrie8=', 'Content-Length': '0'} 2023-10-18T20:30:50.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007310f3756f7d3ebc-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.043 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.044 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.044 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.045 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.045 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.045 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.046 DEBUG:boto:Data: 2023-10-18T20:30:50.046 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.047 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.047 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.048 DEBUG:boto:Params: {} 2023-10-18T20:30:50.048 DEBUG:boto:Token: None 2023-10-18T20:30:50.049 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ljdh8a444CiOqJMOTWak3HBrie8= 2023-10-18T20:30:50.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ljdh8a444CiOqJMOTWak3HBrie8=', 'Content-Length': '0'} 2023-10-18T20:30:50.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d2291359c73f5b-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-18T20:30:50.053 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.053 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.054 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.054 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.055 DEBUG:boto:Method: GET 2023-10-18T20:30:50.055 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.056 DEBUG:boto:Data: 2023-10-18T20:30:50.056 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.057 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.057 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.057 DEBUG:boto:Params: {} 2023-10-18T20:30:50.058 DEBUG:boto:Token: None 2023-10-18T20:30:50.058 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.059 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qzd+DRjhR627jLWOR4kkmrpyxSM= 2023-10-18T20:30:50.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qzd+DRjhR627jLWOR4kkmrpyxSM=', 'Content-Length': '0'} 2023-10-18T20:30:50.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6cd2a4113f0e69b-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.062 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.063 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.063 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.064 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.064 DEBUG:boto:Method: GET 2023-10-18T20:30:50.065 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.065 DEBUG:boto:Data: 2023-10-18T20:30:50.065 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.066 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.067 DEBUG:boto:Params: {} 2023-10-18T20:30:50.067 DEBUG:boto:Token: None 2023-10-18T20:30:50.068 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000036?versionId=null 2023-10-18T20:30:50.068 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qzd+DRjhR627jLWOR4kkmrpyxSM= 2023-10-18T20:30:50.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qzd+DRjhR627jLWOR4kkmrpyxSM=', 'Content-Length': '0'} 2023-10-18T20:30:50.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f36b7e46602fe7-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.072 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.072 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.073 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.073 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.074 DEBUG:boto:Method: GET 2023-10-18T20:30:50.074 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.075 DEBUG:boto:Data: 2023-10-18T20:30:50.075 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.075 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.076 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.076 DEBUG:boto:Params: {} 2023-10-18T20:30:50.077 DEBUG:boto:Token: None 2023-10-18T20:30:50.077 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.078 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b5fb0db01628c6-006530407a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.082 DEBUG:boto:b'bofrnf-58obj000000000000000000361000falseobj000000000000000000362023-10-18T20:30:31.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.082 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.083 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.083 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.084 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.084 DEBUG:boto:Method: GET 2023-10-18T20:30:50.085 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.085 DEBUG:boto:Data: 2023-10-18T20:30:50.086 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.086 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.086 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.087 DEBUG:boto:Params: {} 2023-10-18T20:30:50.088 DEBUG:boto:Token: None 2023-10-18T20:30:50.088 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000111555865aeedf8d-006530407a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.094 DEBUG:boto:b'bofrnf-58obj000000000000000000361000falseobj000000000000000000362023-10-18T20:30:31.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.095 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.095 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.096 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.096 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.096 DEBUG:boto:Data: 2023-10-18T20:30:50.097 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.097 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.098 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.098 DEBUG:boto:Params: {} 2023-10-18T20:30:50.099 DEBUG:boto:Token: None 2023-10-18T20:30:50.099 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bLjMEvSUahfZeaKBfT7LBiKxnSQ= 2023-10-18T20:30:50.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bLjMEvSUahfZeaKBfT7LBiKxnSQ=', 'Content-Length': '0'} 2023-10-18T20:30:50.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332ec5f1bad0c5c0-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.103 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.103 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.104 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.104 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.105 DEBUG:boto:Data: 2023-10-18T20:30:50.105 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.106 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.106 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.106 DEBUG:boto:Params: {} 2023-10-18T20:30:50.107 DEBUG:boto:Token: None 2023-10-18T20:30:50.107 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bLjMEvSUahfZeaKBfT7LBiKxnSQ= 2023-10-18T20:30:50.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bLjMEvSUahfZeaKBfT7LBiKxnSQ=', 'Content-Length': '0'} 2023-10-18T20:30:50.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000800f647fe4e29a40-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-18T20:30:50.111 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.112 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.112 DEBUG:boto:Method: GET 2023-10-18T20:30:50.113 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.113 DEBUG:boto:Data: 2023-10-18T20:30:50.113 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.114 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.114 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.115 DEBUG:boto:Params: {} 2023-10-18T20:30:50.115 DEBUG:boto:Token: None 2023-10-18T20:30:50.116 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.116 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SltFXvfoCG8oaO2vGK2aZ5dvA9M= 2023-10-18T20:30:50.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SltFXvfoCG8oaO2vGK2aZ5dvA9M=', 'Content-Length': '0'} 2023-10-18T20:30:50.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2dd111ab3ab3bff-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.163 DEBUG:boto:path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.163 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.164 DEBUG:boto:Method: GET 2023-10-18T20:30:50.164 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.165 DEBUG:boto:Data: 2023-10-18T20:30:50.165 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.165 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.166 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.166 DEBUG:boto:Params: {} 2023-10-18T20:30:50.167 DEBUG:boto:Token: None 2023-10-18T20:30:50.167 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000036 2023-10-18T20:30:50.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SltFXvfoCG8oaO2vGK2aZ5dvA9M= 2023-10-18T20:30:50.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SltFXvfoCG8oaO2vGK2aZ5dvA9M=', 'Content-Length': '0'} 2023-10-18T20:30:50.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1a626d0dfe7f56b-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.172 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.172 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.173 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.173 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.174 DEBUG:boto:Method: GET 2023-10-18T20:30:50.174 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.174 DEBUG:boto:Data: 2023-10-18T20:30:50.175 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.175 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.176 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.176 DEBUG:boto:Params: {} 2023-10-18T20:30:50.177 DEBUG:boto:Token: None 2023-10-18T20:30:50.177 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.178 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f3215a1858d3d03-006530407a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.182 DEBUG:boto:b'bofrnf-58obj000000000000000000361000falseobj000000000000000000362023-10-18T20:30:31.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.182 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.183 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.183 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.184 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.184 DEBUG:boto:Method: GET 2023-10-18T20:30:50.185 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000036 2023-10-18T20:30:50.185 DEBUG:boto:Data: 2023-10-18T20:30:50.186 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.186 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.187 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.188 DEBUG:boto:Params: {} 2023-10-18T20:30:50.188 DEBUG:boto:Token: None 2023-10-18T20:30:50.190 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.191 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d2925c99c3cc0f5-006530407a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.197 DEBUG:boto:b'bofrnf-58obj000000000000000000361000falseobj000000000000000000362023-10-18T20:30:31.654Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-10-18T20:30:50.198 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.199 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.199 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.199 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.200 DEBUG:boto:Method: GET 2023-10-18T20:30:50.200 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.201 DEBUG:boto:Data: 2023-10-18T20:30:50.201 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.202 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.202 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.203 DEBUG:boto:Params: {} 2023-10-18T20:30:50.203 DEBUG:boto:Token: None 2023-10-18T20:30:50.204 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.204 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eIXFPpa5/4Tnh8I/VK5NIi02pfM= 2023-10-18T20:30:50.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eIXFPpa5/4Tnh8I/VK5NIi02pfM=', 'Content-Length': '0'} 2023-10-18T20:30:50.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6a2850a1511ef20-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.208 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.208 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.209 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.209 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.210 DEBUG:boto:Method: GET 2023-10-18T20:30:50.210 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.210 DEBUG:boto:Data: 2023-10-18T20:30:50.211 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.211 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.212 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.212 DEBUG:boto:Params: {} 2023-10-18T20:30:50.213 DEBUG:boto:Token: None 2023-10-18T20:30:50.213 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.214 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eIXFPpa5/4Tnh8I/VK5NIi02pfM= 2023-10-18T20:30:50.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eIXFPpa5/4Tnh8I/VK5NIi02pfM=', 'Content-Length': '0'} 2023-10-18T20:30:50.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f01394616ac1085-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.217 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.218 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.218 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.218 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.219 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.219 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.220 DEBUG:boto:Data: 2023-10-18T20:30:50.220 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.220 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.221 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.221 DEBUG:boto:Params: {} 2023-10-18T20:30:50.222 DEBUG:boto:Token: None 2023-10-18T20:30:50.222 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.223 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HGHzyhK6VN9Sx6caev/x3Dbujc4= 2023-10-18T20:30:50.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HGHzyhK6VN9Sx6caev/x3Dbujc4=', 'Content-Length': '0'} 2023-10-18T20:30:50.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098237da9a392b8bd-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.226 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.226 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.227 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.227 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.228 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.228 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.229 DEBUG:boto:Data: 2023-10-18T20:30:50.229 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.229 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.230 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.230 DEBUG:boto:Params: {} 2023-10-18T20:30:50.231 DEBUG:boto:Token: None 2023-10-18T20:30:50.231 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HGHzyhK6VN9Sx6caev/x3Dbujc4= 2023-10-18T20:30:50.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HGHzyhK6VN9Sx6caev/x3Dbujc4=', 'Content-Length': '0'} 2023-10-18T20:30:50.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096fe366e3b58addd-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-10-18T20:30:50.235 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.237 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.238 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.238 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.239 DEBUG:boto:Method: GET 2023-10-18T20:30:50.239 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.240 DEBUG:boto:Data: 2023-10-18T20:30:50.240 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.241 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.241 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.242 DEBUG:boto:Params: {} 2023-10-18T20:30:50.242 DEBUG:boto:Token: None 2023-10-18T20:30:50.243 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.243 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eIXFPpa5/4Tnh8I/VK5NIi02pfM= 2023-10-18T20:30:50.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eIXFPpa5/4Tnh8I/VK5NIi02pfM=', 'Content-Length': '0'} 2023-10-18T20:30:50.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e959605e309fee-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.246 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.247 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.247 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.248 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.248 DEBUG:boto:Method: GET 2023-10-18T20:30:50.249 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.249 DEBUG:boto:Data: 2023-10-18T20:30:50.250 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.250 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.251 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.251 DEBUG:boto:Params: {} 2023-10-18T20:30:50.252 DEBUG:boto:Token: None 2023-10-18T20:30:50.252 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037?versionId=null 2023-10-18T20:30:50.253 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eIXFPpa5/4Tnh8I/VK5NIi02pfM= 2023-10-18T20:30:50.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eIXFPpa5/4Tnh8I/VK5NIi02pfM=', 'Content-Length': '0'} 2023-10-18T20:30:50.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d49d7e860ef3c9-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.298 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.299 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.300 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.300 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.300 DEBUG:boto:Method: GET 2023-10-18T20:30:50.301 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.301 DEBUG:boto:Data: 2023-10-18T20:30:50.301 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.302 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.303 DEBUG:boto:Params: {} 2023-10-18T20:30:50.303 DEBUG:boto:Token: None 2023-10-18T20:30:50.304 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034e431780f476e3c-006530407a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.308 DEBUG:boto:b'bofrnf-58obj000000000000000000371000falseobj000000000000000000372023-10-18T20:30:29.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.309 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.310 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.310 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.311 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.311 DEBUG:boto:Method: GET 2023-10-18T20:30:50.311 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.312 DEBUG:boto:Data: 2023-10-18T20:30:50.312 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.313 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.314 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.315 DEBUG:boto:Params: {} 2023-10-18T20:30:50.316 DEBUG:boto:Token: None 2023-10-18T20:30:50.316 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.317 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005236665b1a1b823a-006530407a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.322 DEBUG:boto:b'bofrnf-58obj000000000000000000371000falseobj000000000000000000372023-10-18T20:30:29.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.323 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.323 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.324 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.324 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.325 DEBUG:boto:Data: 2023-10-18T20:30:50.325 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.325 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.326 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.326 DEBUG:boto:Params: {} 2023-10-18T20:30:50.327 DEBUG:boto:Token: None 2023-10-18T20:30:50.327 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g/kDB+wxQa3gmbVWE5hOvi9EbJw= 2023-10-18T20:30:50.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g/kDB+wxQa3gmbVWE5hOvi9EbJw=', 'Content-Length': '0'} 2023-10-18T20:30:50.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf8438d425504ea0-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.331 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.331 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.332 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.332 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.332 DEBUG:boto:Data: 2023-10-18T20:30:50.333 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.334 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.334 DEBUG:boto:Params: {} 2023-10-18T20:30:50.335 DEBUG:boto:Token: None 2023-10-18T20:30:50.335 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.335 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g/kDB+wxQa3gmbVWE5hOvi9EbJw= 2023-10-18T20:30:50.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g/kDB+wxQa3gmbVWE5hOvi9EbJw=', 'Content-Length': '0'} 2023-10-18T20:30:50.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000faffeea98520106-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-10-18T20:30:50.339 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.339 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.340 DEBUG:boto:Method: GET 2023-10-18T20:30:50.340 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.340 DEBUG:boto:Data: 2023-10-18T20:30:50.341 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.341 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.342 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.342 DEBUG:boto:Params: {} 2023-10-18T20:30:50.343 DEBUG:boto:Token: None 2023-10-18T20:30:50.343 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.344 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KoVjy2J0ezpIfd7iSTCwe4ZRxI0= 2023-10-18T20:30:50.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KoVjy2J0ezpIfd7iSTCwe4ZRxI0=', 'Content-Length': '0'} 2023-10-18T20:30:50.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f07e706a381ceea-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.390 DEBUG:boto:path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.391 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.391 DEBUG:boto:Method: GET 2023-10-18T20:30:50.392 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.392 DEBUG:boto:Data: 2023-10-18T20:30:50.393 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.393 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.394 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.394 DEBUG:boto:Params: {} 2023-10-18T20:30:50.395 DEBUG:boto:Token: None 2023-10-18T20:30:50.395 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000037 2023-10-18T20:30:50.396 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KoVjy2J0ezpIfd7iSTCwe4ZRxI0= 2023-10-18T20:30:50.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KoVjy2J0ezpIfd7iSTCwe4ZRxI0=', 'Content-Length': '0'} 2023-10-18T20:30:50.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39c8ba506b998b6-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.399 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.399 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.400 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.400 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.401 DEBUG:boto:Method: GET 2023-10-18T20:30:50.401 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.401 DEBUG:boto:Data: 2023-10-18T20:30:50.402 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.402 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.403 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.403 DEBUG:boto:Params: {} 2023-10-18T20:30:50.404 DEBUG:boto:Token: None 2023-10-18T20:30:50.404 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.405 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000714207ba870c67d3-006530407a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.408 DEBUG:boto:b'bofrnf-58obj000000000000000000371000falseobj000000000000000000372023-10-18T20:30:29.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.409 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.409 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.410 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.410 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.411 DEBUG:boto:Method: GET 2023-10-18T20:30:50.411 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000037 2023-10-18T20:30:50.412 DEBUG:boto:Data: 2023-10-18T20:30:50.412 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.413 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.413 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.413 DEBUG:boto:Params: {} 2023-10-18T20:30:50.414 DEBUG:boto:Token: None 2023-10-18T20:30:50.414 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.415 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1730a06c965904b-006530407a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.466 DEBUG:boto:b'bofrnf-58obj000000000000000000371000falseobj000000000000000000372023-10-18T20:30:29.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-18T20:30:50.468 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.468 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.469 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.469 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.469 DEBUG:boto:Method: GET 2023-10-18T20:30:50.470 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.470 DEBUG:boto:Data: 2023-10-18T20:30:50.471 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.472 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.472 DEBUG:boto:Params: {} 2023-10-18T20:30:50.472 DEBUG:boto:Token: None 2023-10-18T20:30:50.473 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.473 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:husgwQ8IhIjTmkHxjPj9SPmUdj4= 2023-10-18T20:30:50.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:husgwQ8IhIjTmkHxjPj9SPmUdj4=', 'Content-Length': '0'} 2023-10-18T20:30:50.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000665a5bbf5312f370-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.477 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.478 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.478 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.479 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.479 DEBUG:boto:Method: GET 2023-10-18T20:30:50.479 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.480 DEBUG:boto:Data: 2023-10-18T20:30:50.480 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.481 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.482 DEBUG:boto:Params: {} 2023-10-18T20:30:50.482 DEBUG:boto:Token: None 2023-10-18T20:30:50.483 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.483 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:husgwQ8IhIjTmkHxjPj9SPmUdj4= 2023-10-18T20:30:50.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:husgwQ8IhIjTmkHxjPj9SPmUdj4=', 'Content-Length': '0'} 2023-10-18T20:30:50.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000142cb56eb5c4956e-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.487 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.487 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.488 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.488 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.489 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.489 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.489 DEBUG:boto:Data: 2023-10-18T20:30:50.490 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.490 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.491 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.491 DEBUG:boto:Params: {} 2023-10-18T20:30:50.492 DEBUG:boto:Token: None 2023-10-18T20:30:50.492 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.493 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bwAiIdh6ufKg06wW+ByhNcLHz0M= 2023-10-18T20:30:50.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bwAiIdh6ufKg06wW+ByhNcLHz0M=', 'Content-Length': '0'} 2023-10-18T20:30:50.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af07172038e4cff8-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.496 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.496 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.496 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.497 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.497 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.498 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.498 DEBUG:boto:Data: 2023-10-18T20:30:50.499 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.499 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.499 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.500 DEBUG:boto:Params: {} 2023-10-18T20:30:50.500 DEBUG:boto:Token: None 2023-10-18T20:30:50.500 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.501 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bwAiIdh6ufKg06wW+ByhNcLHz0M= 2023-10-18T20:30:50.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bwAiIdh6ufKg06wW+ByhNcLHz0M=', 'Content-Length': '0'} 2023-10-18T20:30:50.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f5674b2ecdf105-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-18T20:30:50.504 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.505 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.505 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.505 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.506 DEBUG:boto:Method: GET 2023-10-18T20:30:50.506 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.507 DEBUG:boto:Data: 2023-10-18T20:30:50.507 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.508 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.508 DEBUG:boto:Params: {} 2023-10-18T20:30:50.509 DEBUG:boto:Token: None 2023-10-18T20:30:50.509 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.510 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:husgwQ8IhIjTmkHxjPj9SPmUdj4= 2023-10-18T20:30:50.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:husgwQ8IhIjTmkHxjPj9SPmUdj4=', 'Content-Length': '0'} 2023-10-18T20:30:50.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b8f303d1d137623-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.513 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.513 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.514 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.514 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.515 DEBUG:boto:Method: GET 2023-10-18T20:30:50.515 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.515 DEBUG:boto:Data: 2023-10-18T20:30:50.516 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.517 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.517 DEBUG:boto:Params: {} 2023-10-18T20:30:50.517 DEBUG:boto:Token: None 2023-10-18T20:30:50.518 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039?versionId=null 2023-10-18T20:30:50.518 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:husgwQ8IhIjTmkHxjPj9SPmUdj4= 2023-10-18T20:30:50.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:husgwQ8IhIjTmkHxjPj9SPmUdj4=', 'Content-Length': '0'} 2023-10-18T20:30:50.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b76a9ac092040678-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.522 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.523 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.523 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.523 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.524 DEBUG:boto:Method: GET 2023-10-18T20:30:50.524 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.525 DEBUG:boto:Data: 2023-10-18T20:30:50.525 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.526 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.526 DEBUG:boto:Params: {} 2023-10-18T20:30:50.527 DEBUG:boto:Token: None 2023-10-18T20:30:50.527 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.528 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7cf77740c1c5162-006530407a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.531 DEBUG:boto:b'bofrnf-58obj000000000000000000391000falseobj000000000000000000392023-10-18T20:30:32.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.532 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.532 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.533 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.533 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.534 DEBUG:boto:Method: GET 2023-10-18T20:30:50.534 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.534 DEBUG:boto:Data: 2023-10-18T20:30:50.535 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.535 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.536 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.536 DEBUG:boto:Params: {} 2023-10-18T20:30:50.537 DEBUG:boto:Token: None 2023-10-18T20:30:50.537 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.537 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000025a8102fe2e17ee-006530407a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.543 DEBUG:boto:b'bofrnf-58obj000000000000000000391000falseobj000000000000000000392023-10-18T20:30:32.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.543 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.544 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.544 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.544 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.545 DEBUG:boto:Data: 2023-10-18T20:30:50.545 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.545 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.546 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.546 DEBUG:boto:Params: {} 2023-10-18T20:30:50.547 DEBUG:boto:Token: None 2023-10-18T20:30:50.547 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.547 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YzaS1pVxjlYPV25s5vHofY/1v1Y= 2023-10-18T20:30:50.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YzaS1pVxjlYPV25s5vHofY/1v1Y=', 'Content-Length': '0'} 2023-10-18T20:30:50.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008cc535a19b770b0-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.550 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.551 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.551 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.552 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.552 DEBUG:boto:Data: 2023-10-18T20:30:50.552 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.553 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.553 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.553 DEBUG:boto:Params: {} 2023-10-18T20:30:50.554 DEBUG:boto:Token: None 2023-10-18T20:30:50.554 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YzaS1pVxjlYPV25s5vHofY/1v1Y= 2023-10-18T20:30:50.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YzaS1pVxjlYPV25s5vHofY/1v1Y=', 'Content-Length': '0'} 2023-10-18T20:30:50.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b388f546fb25b8e-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-18T20:30:50.558 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.558 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.559 DEBUG:boto:Method: GET 2023-10-18T20:30:50.559 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.560 DEBUG:boto:Data: 2023-10-18T20:30:50.560 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.561 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.561 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.561 DEBUG:boto:Params: {} 2023-10-18T20:30:50.562 DEBUG:boto:Token: None 2023-10-18T20:30:50.562 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.563 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9U2VhfZ2kx6YwazA54uC+SVZ7tc= 2023-10-18T20:30:50.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9U2VhfZ2kx6YwazA54uC+SVZ7tc=', 'Content-Length': '0'} 2023-10-18T20:30:50.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a1a660d8ddcd203-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.606 DEBUG:boto:path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.607 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.607 DEBUG:boto:Method: GET 2023-10-18T20:30:50.608 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.608 DEBUG:boto:Data: 2023-10-18T20:30:50.608 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.609 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.609 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.610 DEBUG:boto:Params: {} 2023-10-18T20:30:50.610 DEBUG:boto:Token: None 2023-10-18T20:30:50.611 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000039 2023-10-18T20:30:50.611 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9U2VhfZ2kx6YwazA54uC+SVZ7tc= 2023-10-18T20:30:50.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9U2VhfZ2kx6YwazA54uC+SVZ7tc=', 'Content-Length': '0'} 2023-10-18T20:30:50.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2eb03205a8afb8c-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.614 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.615 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.615 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.616 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.616 DEBUG:boto:Method: GET 2023-10-18T20:30:50.617 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.617 DEBUG:boto:Data: 2023-10-18T20:30:50.617 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.618 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.618 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.619 DEBUG:boto:Params: {} 2023-10-18T20:30:50.619 DEBUG:boto:Token: None 2023-10-18T20:30:50.620 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.620 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095c66768d2470486-006530407a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.624 DEBUG:boto:b'bofrnf-58obj000000000000000000391000falseobj000000000000000000392023-10-18T20:30:32.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.624 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.625 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.625 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.625 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.626 DEBUG:boto:Method: GET 2023-10-18T20:30:50.626 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000039 2023-10-18T20:30:50.627 DEBUG:boto:Data: 2023-10-18T20:30:50.627 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.627 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.628 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.628 DEBUG:boto:Params: {} 2023-10-18T20:30:50.629 DEBUG:boto:Token: None 2023-10-18T20:30:50.629 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.629 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000260a6e4d5575f392-006530407a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.634 DEBUG:boto:b'bofrnf-58obj000000000000000000391000falseobj000000000000000000392023-10-18T20:30:32.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-18T20:30:50.636 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.636 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.636 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.637 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.637 DEBUG:boto:Method: GET 2023-10-18T20:30:50.638 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.638 DEBUG:boto:Data: 2023-10-18T20:30:50.639 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.639 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.640 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.640 DEBUG:boto:Params: {} 2023-10-18T20:30:50.640 DEBUG:boto:Token: None 2023-10-18T20:30:50.641 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.641 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7pL9sigbPUKCEUlrve9YDknLGJk= 2023-10-18T20:30:50.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7pL9sigbPUKCEUlrve9YDknLGJk=', 'Content-Length': '0'} 2023-10-18T20:30:50.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000874d3d2f22a512e9-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.645 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.645 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.646 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.646 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.647 DEBUG:boto:Method: GET 2023-10-18T20:30:50.647 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.647 DEBUG:boto:Data: 2023-10-18T20:30:50.648 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.648 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.649 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.649 DEBUG:boto:Params: {} 2023-10-18T20:30:50.650 DEBUG:boto:Token: None 2023-10-18T20:30:50.650 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.650 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7pL9sigbPUKCEUlrve9YDknLGJk= 2023-10-18T20:30:50.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7pL9sigbPUKCEUlrve9YDknLGJk=', 'Content-Length': '0'} 2023-10-18T20:30:50.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d6822fb2a00dcf4-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.654 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.654 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.655 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.655 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.655 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.656 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.656 DEBUG:boto:Data: 2023-10-18T20:30:50.657 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.657 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.657 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.658 DEBUG:boto:Params: {} 2023-10-18T20:30:50.658 DEBUG:boto:Token: None 2023-10-18T20:30:50.659 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.659 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8GOhM1J2g+XJr+v3gBJ4685sSy0= 2023-10-18T20:30:50.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8GOhM1J2g+XJr+v3gBJ4685sSy0=', 'Content-Length': '0'} 2023-10-18T20:30:50.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c94d98bee59e56f-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.662 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.663 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.663 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.663 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.664 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.664 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.665 DEBUG:boto:Data: 2023-10-18T20:30:50.665 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.665 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.666 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.666 DEBUG:boto:Params: {} 2023-10-18T20:30:50.667 DEBUG:boto:Token: None 2023-10-18T20:30:50.667 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8GOhM1J2g+XJr+v3gBJ4685sSy0= 2023-10-18T20:30:50.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8GOhM1J2g+XJr+v3gBJ4685sSy0=', 'Content-Length': '0'} 2023-10-18T20:30:50.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000074a318e12ce462c-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-18T20:30:50.671 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.672 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.672 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.673 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.673 DEBUG:boto:Method: GET 2023-10-18T20:30:50.673 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.674 DEBUG:boto:Data: 2023-10-18T20:30:50.674 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.675 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.675 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.676 DEBUG:boto:Params: {} 2023-10-18T20:30:50.676 DEBUG:boto:Token: None 2023-10-18T20:30:50.676 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.677 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7pL9sigbPUKCEUlrve9YDknLGJk= 2023-10-18T20:30:50.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7pL9sigbPUKCEUlrve9YDknLGJk=', 'Content-Length': '0'} 2023-10-18T20:30:50.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007870cb0c08839940-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.681 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.681 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.682 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.682 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.683 DEBUG:boto:Method: GET 2023-10-18T20:30:50.683 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.684 DEBUG:boto:Data: 2023-10-18T20:30:50.684 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.685 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.685 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.686 DEBUG:boto:Params: {} 2023-10-18T20:30:50.686 DEBUG:boto:Token: None 2023-10-18T20:30:50.687 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040?versionId=null 2023-10-18T20:30:50.687 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7pL9sigbPUKCEUlrve9YDknLGJk= 2023-10-18T20:30:50.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7pL9sigbPUKCEUlrve9YDknLGJk=', 'Content-Length': '0'} 2023-10-18T20:30:50.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72bf7caf2865159-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.691 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.691 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.692 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.692 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.693 DEBUG:boto:Method: GET 2023-10-18T20:30:50.693 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.693 DEBUG:boto:Data: 2023-10-18T20:30:50.694 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.694 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.695 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.695 DEBUG:boto:Params: {} 2023-10-18T20:30:50.696 DEBUG:boto:Token: None 2023-10-18T20:30:50.696 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.697 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3684ea5dc68c543-006530407a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.700 DEBUG:boto:b'bofrnf-58obj000000000000000000401000falseobj000000000000000000402023-10-18T20:30:28.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.701 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.701 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.702 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.702 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.703 DEBUG:boto:Method: GET 2023-10-18T20:30:50.703 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.703 DEBUG:boto:Data: 2023-10-18T20:30:50.704 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.704 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.705 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.705 DEBUG:boto:Params: {} 2023-10-18T20:30:50.706 DEBUG:boto:Token: None 2023-10-18T20:30:50.706 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.707 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047789953df535c8a-006530407a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.711 DEBUG:boto:b'bofrnf-58obj000000000000000000401000falseobj000000000000000000402023-10-18T20:30:28.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.712 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.712 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.713 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.713 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.714 DEBUG:boto:Data: 2023-10-18T20:30:50.714 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.715 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.715 DEBUG:boto:Params: {} 2023-10-18T20:30:50.716 DEBUG:boto:Token: None 2023-10-18T20:30:50.716 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:24mLLpoStxqj+Y8vIrsG9MNuwz8= 2023-10-18T20:30:50.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:24mLLpoStxqj+Y8vIrsG9MNuwz8=', 'Content-Length': '0'} 2023-10-18T20:30:50.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf745343de3eaf3a-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.720 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.720 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.721 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.721 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.722 DEBUG:boto:Data: 2023-10-18T20:30:50.722 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.723 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.724 DEBUG:boto:Params: {} 2023-10-18T20:30:50.724 DEBUG:boto:Token: None 2023-10-18T20:30:50.725 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:24mLLpoStxqj+Y8vIrsG9MNuwz8= 2023-10-18T20:30:50.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:24mLLpoStxqj+Y8vIrsG9MNuwz8=', 'Content-Length': '0'} 2023-10-18T20:30:50.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd4701e80c845ca4-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-18T20:30:50.729 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.729 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.729 DEBUG:boto:Method: GET 2023-10-18T20:30:50.730 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.730 DEBUG:boto:Data: 2023-10-18T20:30:50.731 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.731 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.732 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.732 DEBUG:boto:Params: {} 2023-10-18T20:30:50.732 DEBUG:boto:Token: None 2023-10-18T20:30:50.733 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.733 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XKaDuN4cpMmIoZ6y892VSwEE2vo= 2023-10-18T20:30:50.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XKaDuN4cpMmIoZ6y892VSwEE2vo=', 'Content-Length': '0'} 2023-10-18T20:30:50.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e163f3794ca5be3-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.778 DEBUG:boto:path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.779 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.779 DEBUG:boto:Method: GET 2023-10-18T20:30:50.780 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.780 DEBUG:boto:Data: 2023-10-18T20:30:50.781 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.782 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.782 DEBUG:boto:Params: {} 2023-10-18T20:30:50.783 DEBUG:boto:Token: None 2023-10-18T20:30:50.783 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000040 2023-10-18T20:30:50.784 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XKaDuN4cpMmIoZ6y892VSwEE2vo= 2023-10-18T20:30:50.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XKaDuN4cpMmIoZ6y892VSwEE2vo=', 'Content-Length': '0'} 2023-10-18T20:30:50.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2ec1fb48c822bb-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.787 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.788 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.788 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.789 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.789 DEBUG:boto:Method: GET 2023-10-18T20:30:50.790 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.790 DEBUG:boto:Data: 2023-10-18T20:30:50.791 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.791 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.792 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.792 DEBUG:boto:Params: {} 2023-10-18T20:30:50.793 DEBUG:boto:Token: None 2023-10-18T20:30:50.793 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.794 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac85d6ffdbb7e427-006530407a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.797 DEBUG:boto:b'bofrnf-58obj000000000000000000401000falseobj000000000000000000402023-10-18T20:30:28.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.798 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.799 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.799 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.799 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.800 DEBUG:boto:Method: GET 2023-10-18T20:30:50.800 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000040 2023-10-18T20:30:50.801 DEBUG:boto:Data: 2023-10-18T20:30:50.801 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.802 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.802 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.803 DEBUG:boto:Params: {} 2023-10-18T20:30:50.804 DEBUG:boto:Token: None 2023-10-18T20:30:50.804 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.805 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b6674ea9972cd28-006530407a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.850 DEBUG:boto:b'bofrnf-58obj000000000000000000401000falseobj000000000000000000402023-10-18T20:30:28.049Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-18T20:30:50.852 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.852 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.853 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.853 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.853 DEBUG:boto:Method: GET 2023-10-18T20:30:50.854 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.854 DEBUG:boto:Data: 2023-10-18T20:30:50.855 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.856 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.856 DEBUG:boto:Params: {} 2023-10-18T20:30:50.857 DEBUG:boto:Token: None 2023-10-18T20:30:50.857 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.858 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M/qLpeAlKUwq3C6Gp41rp9CDN+4= 2023-10-18T20:30:50.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M/qLpeAlKUwq3C6Gp41rp9CDN+4=', 'Content-Length': '0'} 2023-10-18T20:30:50.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba93024b28d5551-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.861 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.862 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.862 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.862 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.863 DEBUG:boto:Method: GET 2023-10-18T20:30:50.863 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.864 DEBUG:boto:Data: 2023-10-18T20:30:50.864 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.864 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.865 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.865 DEBUG:boto:Params: {} 2023-10-18T20:30:50.866 DEBUG:boto:Token: None 2023-10-18T20:30:50.866 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M/qLpeAlKUwq3C6Gp41rp9CDN+4= 2023-10-18T20:30:50.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M/qLpeAlKUwq3C6Gp41rp9CDN+4=', 'Content-Length': '0'} 2023-10-18T20:30:50.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000191da4896dc75b7-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.870 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.871 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.871 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.871 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.872 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.872 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.873 DEBUG:boto:Data: 2023-10-18T20:30:50.873 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.874 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.874 DEBUG:boto:Params: {} 2023-10-18T20:30:50.875 DEBUG:boto:Token: None 2023-10-18T20:30:50.875 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.876 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OgwnbuflwNb4slLuQLnxbAMLnP8= 2023-10-18T20:30:50.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OgwnbuflwNb4slLuQLnxbAMLnP8=', 'Content-Length': '0'} 2023-10-18T20:30:50.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f404d275047af6-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.879 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.879 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.879 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.880 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.880 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.881 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.881 DEBUG:boto:Data: 2023-10-18T20:30:50.881 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.882 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.883 DEBUG:boto:Params: {} 2023-10-18T20:30:50.883 DEBUG:boto:Token: None 2023-10-18T20:30:50.884 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OgwnbuflwNb4slLuQLnxbAMLnP8= 2023-10-18T20:30:50.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OgwnbuflwNb4slLuQLnxbAMLnP8=', 'Content-Length': '0'} 2023-10-18T20:30:50.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f04027a646ebe8e-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-18T20:30:50.888 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.888 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.889 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.889 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.890 DEBUG:boto:Method: GET 2023-10-18T20:30:50.890 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.890 DEBUG:boto:Data: 2023-10-18T20:30:50.891 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.891 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.892 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.892 DEBUG:boto:Params: {} 2023-10-18T20:30:50.893 DEBUG:boto:Token: None 2023-10-18T20:30:50.893 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.894 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M/qLpeAlKUwq3C6Gp41rp9CDN+4= 2023-10-18T20:30:50.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M/qLpeAlKUwq3C6Gp41rp9CDN+4=', 'Content-Length': '0'} 2023-10-18T20:30:50.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dc8c42b986bfb64-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.897 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.897 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.898 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.898 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.899 DEBUG:boto:Method: GET 2023-10-18T20:30:50.899 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.900 DEBUG:boto:Data: 2023-10-18T20:30:50.900 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.901 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.901 DEBUG:boto:Params: {} 2023-10-18T20:30:50.902 DEBUG:boto:Token: None 2023-10-18T20:30:50.902 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042?versionId=null 2023-10-18T20:30:50.903 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M/qLpeAlKUwq3C6Gp41rp9CDN+4= 2023-10-18T20:30:50.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M/qLpeAlKUwq3C6Gp41rp9CDN+4=', 'Content-Length': '0'} 2023-10-18T20:30:50.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d98d46b82e0a4a79-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.906 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.906 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.907 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:50.907 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:50.908 DEBUG:boto:Method: GET 2023-10-18T20:30:50.908 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:50.909 DEBUG:boto:Data: 2023-10-18T20:30:50.909 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.909 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.910 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.910 DEBUG:boto:Params: {} 2023-10-18T20:30:50.911 DEBUG:boto:Token: None 2023-10-18T20:30:50.911 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.912 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c3efbe4a51bc04a-006530407a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.915 DEBUG:boto:b'bofrnf-58obj000000000000000000421000falseobj000000000000000000422023-10-18T20:30:29.255Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.916 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:50.917 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:50.917 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:50.918 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:50.918 DEBUG:boto:Method: GET 2023-10-18T20:30:50.918 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:50.919 DEBUG:boto:Data: 2023-10-18T20:30:50.919 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.920 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.920 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.920 DEBUG:boto:Params: {} 2023-10-18T20:30:50.921 DEBUG:boto:Token: None 2023-10-18T20:30:50.921 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/ 2023-10-18T20:30:50.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4= 2023-10-18T20:30:50.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g9l9AAYHsfFom7ZtRhkS07mnQw4=', 'Content-Length': '0'} 2023-10-18T20:30:50.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fc198eb5f387173-006530407a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.927 DEBUG:boto:b'bofrnf-58obj000000000000000000421000falseobj000000000000000000422023-10-18T20:30:29.255Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:50.928 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.928 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.929 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.929 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.929 DEBUG:boto:Data: 2023-10-18T20:30:50.930 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.930 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.931 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.931 DEBUG:boto:Params: {} 2023-10-18T20:30:50.931 DEBUG:boto:Token: None 2023-10-18T20:30:50.932 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.932 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a/f5UkxalfUmmVIuxkwMWWTSEQM= 2023-10-18T20:30:50.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a/f5UkxalfUmmVIuxkwMWWTSEQM=', 'Content-Length': '0'} 2023-10-18T20:30:50.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191d8b01e0ac8ffd-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.935 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.936 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.936 DEBUG:boto:Method: HEAD 2023-10-18T20:30:50.937 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.937 DEBUG:boto:Data: 2023-10-18T20:30:50.938 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.938 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.938 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.939 DEBUG:boto:Params: {} 2023-10-18T20:30:50.939 DEBUG:boto:Token: None 2023-10-18T20:30:50.940 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.940 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a/f5UkxalfUmmVIuxkwMWWTSEQM= 2023-10-18T20:30:50.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a/f5UkxalfUmmVIuxkwMWWTSEQM=', 'Content-Length': '0'} 2023-10-18T20:30:50.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0d95c7539274aff-006530407a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-18T20:30:50.944 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.944 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.944 DEBUG:boto:Method: GET 2023-10-18T20:30:50.945 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.945 DEBUG:boto:Data: 2023-10-18T20:30:50.946 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.946 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:50.947 DEBUG:boto:Port: 8001 2023-10-18T20:30:50.947 DEBUG:boto:Params: {} 2023-10-18T20:30:50.947 DEBUG:boto:Token: None 2023-10-18T20:30:50.948 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xJOghNZ10X7nANHN+nfdRKegmeE= 2023-10-18T20:30:50.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xJOghNZ10X7nANHN+nfdRKegmeE=', 'Content-Length': '0'} 2023-10-18T20:30:50.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e72905e4cff6de1-006530407a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:50.994 DEBUG:boto:path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.995 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.996 DEBUG:boto:Method: GET 2023-10-18T20:30:50.996 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:50.996 DEBUG:boto:Data: 2023-10-18T20:30:50.997 DEBUG:boto:Headers: {} 2023-10-18T20:30:50.997 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:50.998 DEBUG:boto:Port: 8000 2023-10-18T20:30:50.998 DEBUG:boto:Params: {} 2023-10-18T20:30:50.999 DEBUG:boto:Token: None 2023-10-18T20:30:50.999 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:50 GMT /bofrnf-58/obj00000000000000000042 2023-10-18T20:30:51.000 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xJOghNZ10X7nANHN+nfdRKegmeE= 2023-10-18T20:30:51.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xJOghNZ10X7nANHN+nfdRKegmeE=', 'Content-Length': '0'} 2023-10-18T20:30:51.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e6b14be3691ebd4-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.003 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.004 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.004 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:51.005 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:51.005 DEBUG:boto:Method: GET 2023-10-18T20:30:51.005 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:51.006 DEBUG:boto:Data: 2023-10-18T20:30:51.006 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.007 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.008 DEBUG:boto:Params: {} 2023-10-18T20:30:51.008 DEBUG:boto:Token: None 2023-10-18T20:30:51.009 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ef17edfa5d1490-006530407b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.013 DEBUG:boto:b'bofrnf-58obj000000000000000000421000falseobj000000000000000000422023-10-18T20:30:29.255Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.014 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.014 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.015 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:51.015 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:51.016 DEBUG:boto:Method: GET 2023-10-18T20:30:51.016 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000042 2023-10-18T20:30:51.017 DEBUG:boto:Data: 2023-10-18T20:30:51.017 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.017 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.018 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.018 DEBUG:boto:Params: {} 2023-10-18T20:30:51.019 DEBUG:boto:Token: None 2023-10-18T20:30:51.019 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.020 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042fd13615e138858-006530407b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.025 DEBUG:boto:b'bofrnf-58obj000000000000000000421000falseobj000000000000000000422023-10-18T20:30:29.255Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-18T20:30:51.026 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.027 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.027 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.028 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.028 DEBUG:boto:Method: GET 2023-10-18T20:30:51.029 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.029 DEBUG:boto:Data: 2023-10-18T20:30:51.029 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.030 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.030 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.031 DEBUG:boto:Params: {} 2023-10-18T20:30:51.031 DEBUG:boto:Token: None 2023-10-18T20:30:51.032 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nEQOuw+1VF6gDdPjwQMDxrn3zM4= 2023-10-18T20:30:51.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nEQOuw+1VF6gDdPjwQMDxrn3zM4=', 'Content-Length': '0'} 2023-10-18T20:30:51.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c21cd867df16dc03-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.036 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.036 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.037 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.037 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.038 DEBUG:boto:Method: GET 2023-10-18T20:30:51.038 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.039 DEBUG:boto:Data: 2023-10-18T20:30:51.039 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.040 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.040 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.040 DEBUG:boto:Params: {} 2023-10-18T20:30:51.041 DEBUG:boto:Token: None 2023-10-18T20:30:51.041 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.042 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nEQOuw+1VF6gDdPjwQMDxrn3zM4= 2023-10-18T20:30:51.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nEQOuw+1VF6gDdPjwQMDxrn3zM4=', 'Content-Length': '0'} 2023-10-18T20:30:51.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd28574d925999e3-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.045 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.046 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.046 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.047 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.047 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.047 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.048 DEBUG:boto:Data: 2023-10-18T20:30:51.048 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.048 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.049 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.049 DEBUG:boto:Params: {} 2023-10-18T20:30:51.050 DEBUG:boto:Token: None 2023-10-18T20:30:51.050 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.051 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m8TJ51GBPYQUa2aaZTbrNODRGF4= 2023-10-18T20:30:51.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m8TJ51GBPYQUa2aaZTbrNODRGF4=', 'Content-Length': '0'} 2023-10-18T20:30:51.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072199ec497a82f94-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.053 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.054 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.054 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.055 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.057 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.058 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.058 DEBUG:boto:Data: 2023-10-18T20:30:51.059 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.059 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.060 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.060 DEBUG:boto:Params: {} 2023-10-18T20:30:51.060 DEBUG:boto:Token: None 2023-10-18T20:30:51.061 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.061 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m8TJ51GBPYQUa2aaZTbrNODRGF4= 2023-10-18T20:30:51.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m8TJ51GBPYQUa2aaZTbrNODRGF4=', 'Content-Length': '0'} 2023-10-18T20:30:51.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017fc1b1d70c04947-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-18T20:30:51.065 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.065 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.066 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.066 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.066 DEBUG:boto:Method: GET 2023-10-18T20:30:51.067 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.067 DEBUG:boto:Data: 2023-10-18T20:30:51.068 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.068 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.068 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.069 DEBUG:boto:Params: {} 2023-10-18T20:30:51.069 DEBUG:boto:Token: None 2023-10-18T20:30:51.070 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.070 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nEQOuw+1VF6gDdPjwQMDxrn3zM4= 2023-10-18T20:30:51.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nEQOuw+1VF6gDdPjwQMDxrn3zM4=', 'Content-Length': '0'} 2023-10-18T20:30:51.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a402c6a25475063f-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.073 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.074 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.074 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.075 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.075 DEBUG:boto:Method: GET 2023-10-18T20:30:51.076 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.076 DEBUG:boto:Data: 2023-10-18T20:30:51.077 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.077 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.077 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.078 DEBUG:boto:Params: {} 2023-10-18T20:30:51.078 DEBUG:boto:Token: None 2023-10-18T20:30:51.079 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045?versionId=null 2023-10-18T20:30:51.080 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nEQOuw+1VF6gDdPjwQMDxrn3zM4= 2023-10-18T20:30:51.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nEQOuw+1VF6gDdPjwQMDxrn3zM4=', 'Content-Length': '0'} 2023-10-18T20:30:51.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003044d5a92d3bf2a8-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.083 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.083 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.084 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.084 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.085 DEBUG:boto:Method: GET 2023-10-18T20:30:51.085 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.086 DEBUG:boto:Data: 2023-10-18T20:30:51.086 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.087 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.087 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.088 DEBUG:boto:Params: {} 2023-10-18T20:30:51.088 DEBUG:boto:Token: None 2023-10-18T20:30:51.089 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fd9910bea1f7090-006530407b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.093 DEBUG:boto:b'bofrnf-58obj000000000000000000451000falseobj000000000000000000452023-10-18T20:30:32.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.093 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.094 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.094 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.095 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.095 DEBUG:boto:Method: GET 2023-10-18T20:30:51.096 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.096 DEBUG:boto:Data: 2023-10-18T20:30:51.097 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.097 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.098 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.098 DEBUG:boto:Params: {} 2023-10-18T20:30:51.099 DEBUG:boto:Token: None 2023-10-18T20:30:51.099 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.099 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1395a9f93f69394-006530407b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.146 DEBUG:boto:b'bofrnf-58obj000000000000000000451000falseobj000000000000000000452023-10-18T20:30:32.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.147 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.148 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.148 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.148 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.149 DEBUG:boto:Data: 2023-10-18T20:30:51.149 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.150 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.150 DEBUG:boto:Params: {} 2023-10-18T20:30:51.151 DEBUG:boto:Token: None 2023-10-18T20:30:51.151 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gu7RAaj+XuHoicWFRM7ae6sqDDc= 2023-10-18T20:30:51.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gu7RAaj+XuHoicWFRM7ae6sqDDc=', 'Content-Length': '0'} 2023-10-18T20:30:51.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007859d0b13fc2eb80-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.155 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.155 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.156 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.156 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.157 DEBUG:boto:Data: 2023-10-18T20:30:51.157 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.158 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.159 DEBUG:boto:Params: {} 2023-10-18T20:30:51.159 DEBUG:boto:Token: None 2023-10-18T20:30:51.160 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.160 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gu7RAaj+XuHoicWFRM7ae6sqDDc= 2023-10-18T20:30:51.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gu7RAaj+XuHoicWFRM7ae6sqDDc=', 'Content-Length': '0'} 2023-10-18T20:30:51.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c709597d53e171-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-18T20:30:51.164 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.164 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.165 DEBUG:boto:Method: GET 2023-10-18T20:30:51.165 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.166 DEBUG:boto:Data: 2023-10-18T20:30:51.166 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.167 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.167 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.168 DEBUG:boto:Params: {} 2023-10-18T20:30:51.168 DEBUG:boto:Token: None 2023-10-18T20:30:51.168 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.169 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kTXGXZ1ngjs+1nPAcX1Z5b4u9rs= 2023-10-18T20:30:51.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kTXGXZ1ngjs+1nPAcX1Z5b4u9rs=', 'Content-Length': '0'} 2023-10-18T20:30:51.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f275d4c961f66995-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.218 DEBUG:boto:path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.219 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.219 DEBUG:boto:Method: GET 2023-10-18T20:30:51.220 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.220 DEBUG:boto:Data: 2023-10-18T20:30:51.220 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.221 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.221 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.222 DEBUG:boto:Params: {} 2023-10-18T20:30:51.222 DEBUG:boto:Token: None 2023-10-18T20:30:51.223 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000045 2023-10-18T20:30:51.223 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kTXGXZ1ngjs+1nPAcX1Z5b4u9rs= 2023-10-18T20:30:51.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kTXGXZ1ngjs+1nPAcX1Z5b4u9rs=', 'Content-Length': '0'} 2023-10-18T20:30:51.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7d0600e037955e-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.227 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.227 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.227 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.228 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.228 DEBUG:boto:Method: GET 2023-10-18T20:30:51.229 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.229 DEBUG:boto:Data: 2023-10-18T20:30:51.230 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.231 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.231 DEBUG:boto:Params: {} 2023-10-18T20:30:51.231 DEBUG:boto:Token: None 2023-10-18T20:30:51.232 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097d4b03339a7b0ab-006530407b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.236 DEBUG:boto:b'bofrnf-58obj000000000000000000451000falseobj000000000000000000452023-10-18T20:30:32.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.237 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.237 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.238 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.238 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.238 DEBUG:boto:Method: GET 2023-10-18T20:30:51.239 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000045 2023-10-18T20:30:51.239 DEBUG:boto:Data: 2023-10-18T20:30:51.240 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.240 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.240 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.241 DEBUG:boto:Params: {} 2023-10-18T20:30:51.241 DEBUG:boto:Token: None 2023-10-18T20:30:51.242 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.242 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006ebe76ab79cad08-006530407b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.247 DEBUG:boto:b'bofrnf-58obj000000000000000000451000falseobj000000000000000000452023-10-18T20:30:32.375Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-18T20:30:51.248 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.249 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.249 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.250 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.250 DEBUG:boto:Method: GET 2023-10-18T20:30:51.251 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.251 DEBUG:boto:Data: 2023-10-18T20:30:51.251 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.252 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.252 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.253 DEBUG:boto:Params: {} 2023-10-18T20:30:51.253 DEBUG:boto:Token: None 2023-10-18T20:30:51.254 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.254 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/cSxV60vIfTg1wVBycyr1H6dzg= 2023-10-18T20:30:51.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/cSxV60vIfTg1wVBycyr1H6dzg=', 'Content-Length': '0'} 2023-10-18T20:30:51.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a6baa6e7b72c3f-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.258 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.258 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.259 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.259 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.259 DEBUG:boto:Method: GET 2023-10-18T20:30:51.260 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.260 DEBUG:boto:Data: 2023-10-18T20:30:51.261 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.261 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.261 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.262 DEBUG:boto:Params: {} 2023-10-18T20:30:51.262 DEBUG:boto:Token: None 2023-10-18T20:30:51.263 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/cSxV60vIfTg1wVBycyr1H6dzg= 2023-10-18T20:30:51.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/cSxV60vIfTg1wVBycyr1H6dzg=', 'Content-Length': '0'} 2023-10-18T20:30:51.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f5b72143bf84d4-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.268 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.268 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.269 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.269 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.270 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.270 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.271 DEBUG:boto:Data: 2023-10-18T20:30:51.271 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.271 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.272 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.272 DEBUG:boto:Params: {} 2023-10-18T20:30:51.273 DEBUG:boto:Token: None 2023-10-18T20:30:51.273 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.274 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UUsc+qjXg1hLAz51V7DO/vhD+WQ= 2023-10-18T20:30:51.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UUsc+qjXg1hLAz51V7DO/vhD+WQ=', 'Content-Length': '0'} 2023-10-18T20:30:51.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fa07a3062ead7ab-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.277 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.277 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.278 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.278 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.279 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.279 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.279 DEBUG:boto:Data: 2023-10-18T20:30:51.280 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.280 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.281 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.281 DEBUG:boto:Params: {} 2023-10-18T20:30:51.281 DEBUG:boto:Token: None 2023-10-18T20:30:51.282 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.282 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UUsc+qjXg1hLAz51V7DO/vhD+WQ= 2023-10-18T20:30:51.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UUsc+qjXg1hLAz51V7DO/vhD+WQ=', 'Content-Length': '0'} 2023-10-18T20:30:51.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7f5bcba4c9f7bd4-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-18T20:30:51.286 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.286 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.287 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.287 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.288 DEBUG:boto:Method: GET 2023-10-18T20:30:51.288 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.289 DEBUG:boto:Data: 2023-10-18T20:30:51.289 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.290 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.290 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.290 DEBUG:boto:Params: {} 2023-10-18T20:30:51.291 DEBUG:boto:Token: None 2023-10-18T20:30:51.291 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.292 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/cSxV60vIfTg1wVBycyr1H6dzg= 2023-10-18T20:30:51.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/cSxV60vIfTg1wVBycyr1H6dzg=', 'Content-Length': '0'} 2023-10-18T20:30:51.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c6ce136aeb9002d-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.296 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.296 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.297 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.297 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.297 DEBUG:boto:Method: GET 2023-10-18T20:30:51.298 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.298 DEBUG:boto:Data: 2023-10-18T20:30:51.299 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.300 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.300 DEBUG:boto:Params: {} 2023-10-18T20:30:51.301 DEBUG:boto:Token: None 2023-10-18T20:30:51.301 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046?versionId=null 2023-10-18T20:30:51.302 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/cSxV60vIfTg1wVBycyr1H6dzg= 2023-10-18T20:30:51.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/cSxV60vIfTg1wVBycyr1H6dzg=', 'Content-Length': '0'} 2023-10-18T20:30:51.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049660399eb562a66-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.346 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.347 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.347 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.348 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.348 DEBUG:boto:Method: GET 2023-10-18T20:30:51.349 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.349 DEBUG:boto:Data: 2023-10-18T20:30:51.350 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.350 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.350 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.351 DEBUG:boto:Params: {} 2023-10-18T20:30:51.351 DEBUG:boto:Token: None 2023-10-18T20:30:51.352 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.352 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a85351bd4c660f-006530407b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.356 DEBUG:boto:b'bofrnf-58obj000000000000000000461000falseobj000000000000000000462023-10-18T20:30:28.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.357 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.357 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.357 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.358 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.358 DEBUG:boto:Method: GET 2023-10-18T20:30:51.359 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.359 DEBUG:boto:Data: 2023-10-18T20:30:51.360 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.360 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.361 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.361 DEBUG:boto:Params: {} 2023-10-18T20:30:51.361 DEBUG:boto:Token: None 2023-10-18T20:30:51.362 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.362 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a70e5aeea2c54f3-006530407b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.368 DEBUG:boto:b'bofrnf-58obj000000000000000000461000falseobj000000000000000000462023-10-18T20:30:28.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.369 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.369 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.370 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.370 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.371 DEBUG:boto:Data: 2023-10-18T20:30:51.371 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.372 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.372 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.372 DEBUG:boto:Params: {} 2023-10-18T20:30:51.373 DEBUG:boto:Token: None 2023-10-18T20:30:51.373 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.374 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jy6AM8I9mc2NQiNBAPdv5zYdfOI= 2023-10-18T20:30:51.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jy6AM8I9mc2NQiNBAPdv5zYdfOI=', 'Content-Length': '0'} 2023-10-18T20:30:51.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f46434acebef8dd3-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.377 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.377 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.378 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.378 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.378 DEBUG:boto:Data: 2023-10-18T20:30:51.379 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.379 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.380 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.380 DEBUG:boto:Params: {} 2023-10-18T20:30:51.380 DEBUG:boto:Token: None 2023-10-18T20:30:51.381 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.381 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jy6AM8I9mc2NQiNBAPdv5zYdfOI= 2023-10-18T20:30:51.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jy6AM8I9mc2NQiNBAPdv5zYdfOI=', 'Content-Length': '0'} 2023-10-18T20:30:51.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046925d274bfb3141-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-18T20:30:51.385 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.386 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.386 DEBUG:boto:Method: GET 2023-10-18T20:30:51.387 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.387 DEBUG:boto:Data: 2023-10-18T20:30:51.387 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.388 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.388 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.389 DEBUG:boto:Params: {} 2023-10-18T20:30:51.389 DEBUG:boto:Token: None 2023-10-18T20:30:51.389 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.390 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ipCuOpGzB0A9GNq1FfuoHdeKT+k= 2023-10-18T20:30:51.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ipCuOpGzB0A9GNq1FfuoHdeKT+k=', 'Content-Length': '0'} 2023-10-18T20:30:51.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b75055d39926d2db-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.434 DEBUG:boto:path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.435 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.435 DEBUG:boto:Method: GET 2023-10-18T20:30:51.436 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.436 DEBUG:boto:Data: 2023-10-18T20:30:51.437 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.437 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.438 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.438 DEBUG:boto:Params: {} 2023-10-18T20:30:51.439 DEBUG:boto:Token: None 2023-10-18T20:30:51.439 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000046 2023-10-18T20:30:51.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ipCuOpGzB0A9GNq1FfuoHdeKT+k= 2023-10-18T20:30:51.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ipCuOpGzB0A9GNq1FfuoHdeKT+k=', 'Content-Length': '0'} 2023-10-18T20:30:51.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d831810e535d27f0-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.443 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.443 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.444 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.444 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.445 DEBUG:boto:Method: GET 2023-10-18T20:30:51.445 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.446 DEBUG:boto:Data: 2023-10-18T20:30:51.446 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.447 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.447 DEBUG:boto:Params: {} 2023-10-18T20:30:51.448 DEBUG:boto:Token: None 2023-10-18T20:30:51.448 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.449 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000944ffd514dc251bb-006530407b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.453 DEBUG:boto:b'bofrnf-58obj000000000000000000461000falseobj000000000000000000462023-10-18T20:30:28.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.453 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.454 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.454 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.455 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.455 DEBUG:boto:Method: GET 2023-10-18T20:30:51.456 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000046 2023-10-18T20:30:51.456 DEBUG:boto:Data: 2023-10-18T20:30:51.456 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.457 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.458 DEBUG:boto:Params: {} 2023-10-18T20:30:51.458 DEBUG:boto:Token: None 2023-10-18T20:30:51.459 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.459 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000085f808bca62d50f-006530407b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.506 DEBUG:boto:b'bofrnf-58obj000000000000000000461000falseobj000000000000000000462023-10-18T20:30:28.030Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-18T20:30:51.508 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.508 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.508 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.509 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.509 DEBUG:boto:Method: GET 2023-10-18T20:30:51.510 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.510 DEBUG:boto:Data: 2023-10-18T20:30:51.511 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.511 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.511 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.512 DEBUG:boto:Params: {} 2023-10-18T20:30:51.512 DEBUG:boto:Token: None 2023-10-18T20:30:51.513 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.513 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JF+H3teYaw0NU3EcTCDWiU0LC2A= 2023-10-18T20:30:51.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JF+H3teYaw0NU3EcTCDWiU0LC2A=', 'Content-Length': '0'} 2023-10-18T20:30:51.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000512c27710ef6240-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.517 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.518 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.518 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.519 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.519 DEBUG:boto:Method: GET 2023-10-18T20:30:51.519 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.520 DEBUG:boto:Data: 2023-10-18T20:30:51.520 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.521 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.521 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.522 DEBUG:boto:Params: {} 2023-10-18T20:30:51.522 DEBUG:boto:Token: None 2023-10-18T20:30:51.523 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.523 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JF+H3teYaw0NU3EcTCDWiU0LC2A= 2023-10-18T20:30:51.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JF+H3teYaw0NU3EcTCDWiU0LC2A=', 'Content-Length': '0'} 2023-10-18T20:30:51.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e3e66910c88365c-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.527 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.527 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.527 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.528 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.528 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.529 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.529 DEBUG:boto:Data: 2023-10-18T20:30:51.529 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.530 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.531 DEBUG:boto:Params: {} 2023-10-18T20:30:51.531 DEBUG:boto:Token: None 2023-10-18T20:30:51.531 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KvXv24ODCNLmJ2WugxEA3OQbXrA= 2023-10-18T20:30:51.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KvXv24ODCNLmJ2WugxEA3OQbXrA=', 'Content-Length': '0'} 2023-10-18T20:30:51.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d3cbea90933f81-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.535 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.535 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.536 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.536 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.537 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.537 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.538 DEBUG:boto:Data: 2023-10-18T20:30:51.538 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.538 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.539 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.539 DEBUG:boto:Params: {} 2023-10-18T20:30:51.540 DEBUG:boto:Token: None 2023-10-18T20:30:51.540 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.540 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KvXv24ODCNLmJ2WugxEA3OQbXrA= 2023-10-18T20:30:51.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KvXv24ODCNLmJ2WugxEA3OQbXrA=', 'Content-Length': '0'} 2023-10-18T20:30:51.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a4b957f4759d1fa-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-18T20:30:51.544 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.544 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.545 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.545 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.546 DEBUG:boto:Method: GET 2023-10-18T20:30:51.546 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.546 DEBUG:boto:Data: 2023-10-18T20:30:51.547 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.547 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.548 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.548 DEBUG:boto:Params: {} 2023-10-18T20:30:51.549 DEBUG:boto:Token: None 2023-10-18T20:30:51.549 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.550 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JF+H3teYaw0NU3EcTCDWiU0LC2A= 2023-10-18T20:30:51.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JF+H3teYaw0NU3EcTCDWiU0LC2A=', 'Content-Length': '0'} 2023-10-18T20:30:51.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c64ae8e526d7dd14-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.553 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.553 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.554 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.554 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.555 DEBUG:boto:Method: GET 2023-10-18T20:30:51.555 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.555 DEBUG:boto:Data: 2023-10-18T20:30:51.556 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.556 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.557 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.557 DEBUG:boto:Params: {} 2023-10-18T20:30:51.558 DEBUG:boto:Token: None 2023-10-18T20:30:51.558 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048?versionId=null 2023-10-18T20:30:51.559 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JF+H3teYaw0NU3EcTCDWiU0LC2A= 2023-10-18T20:30:51.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JF+H3teYaw0NU3EcTCDWiU0LC2A=', 'Content-Length': '0'} 2023-10-18T20:30:51.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acbfc47110f75a0c-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.562 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.562 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.563 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.563 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.563 DEBUG:boto:Method: GET 2023-10-18T20:30:51.564 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.564 DEBUG:boto:Data: 2023-10-18T20:30:51.565 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.565 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.565 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.566 DEBUG:boto:Params: {} 2023-10-18T20:30:51.566 DEBUG:boto:Token: None 2023-10-18T20:30:51.567 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.567 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006655420ab57227be-006530407b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.571 DEBUG:boto:b'bofrnf-58obj000000000000000000481000falseobj000000000000000000482023-10-18T20:30:32.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.572 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.572 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.572 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.573 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.573 DEBUG:boto:Method: GET 2023-10-18T20:30:51.574 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.574 DEBUG:boto:Data: 2023-10-18T20:30:51.575 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.575 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.575 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.576 DEBUG:boto:Params: {} 2023-10-18T20:30:51.576 DEBUG:boto:Token: None 2023-10-18T20:30:51.577 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c72279fe34679a2b-006530407b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.582 DEBUG:boto:b'bofrnf-58obj000000000000000000481000falseobj000000000000000000482023-10-18T20:30:32.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.583 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.583 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.584 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.584 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.585 DEBUG:boto:Data: 2023-10-18T20:30:51.585 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.586 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.586 DEBUG:boto:Params: {} 2023-10-18T20:30:51.587 DEBUG:boto:Token: None 2023-10-18T20:30:51.587 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.588 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kg7jJTh4GNjIWkC1t9Ds/+qMcuA= 2023-10-18T20:30:51.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kg7jJTh4GNjIWkC1t9Ds/+qMcuA=', 'Content-Length': '0'} 2023-10-18T20:30:51.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e977ca51c94826f-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.591 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.591 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.592 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.592 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.592 DEBUG:boto:Data: 2023-10-18T20:30:51.593 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.593 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.594 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.594 DEBUG:boto:Params: {} 2023-10-18T20:30:51.594 DEBUG:boto:Token: None 2023-10-18T20:30:51.595 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kg7jJTh4GNjIWkC1t9Ds/+qMcuA= 2023-10-18T20:30:51.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kg7jJTh4GNjIWkC1t9Ds/+qMcuA=', 'Content-Length': '0'} 2023-10-18T20:30:51.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000358eb8c183cd26aa-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-18T20:30:51.599 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.599 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.600 DEBUG:boto:Method: GET 2023-10-18T20:30:51.600 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.601 DEBUG:boto:Data: 2023-10-18T20:30:51.601 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.601 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.602 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.602 DEBUG:boto:Params: {} 2023-10-18T20:30:51.603 DEBUG:boto:Token: None 2023-10-18T20:30:51.603 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dkja72OfqJ+Cuj26KWp3k/SAQPs= 2023-10-18T20:30:51.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dkja72OfqJ+Cuj26KWp3k/SAQPs=', 'Content-Length': '0'} 2023-10-18T20:30:51.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c823a57618b8cd-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.650 DEBUG:boto:path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.651 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.651 DEBUG:boto:Method: GET 2023-10-18T20:30:51.652 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.652 DEBUG:boto:Data: 2023-10-18T20:30:51.653 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.653 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.653 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.654 DEBUG:boto:Params: {} 2023-10-18T20:30:51.654 DEBUG:boto:Token: None 2023-10-18T20:30:51.655 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000048 2023-10-18T20:30:51.655 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dkja72OfqJ+Cuj26KWp3k/SAQPs= 2023-10-18T20:30:51.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dkja72OfqJ+Cuj26KWp3k/SAQPs=', 'Content-Length': '0'} 2023-10-18T20:30:51.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a544de5a83b34c5f-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.659 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.659 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.660 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.660 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.660 DEBUG:boto:Method: GET 2023-10-18T20:30:51.661 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.661 DEBUG:boto:Data: 2023-10-18T20:30:51.662 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.663 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.663 DEBUG:boto:Params: {} 2023-10-18T20:30:51.664 DEBUG:boto:Token: None 2023-10-18T20:30:51.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000718368670e6df4e-006530407b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.668 DEBUG:boto:b'bofrnf-58obj000000000000000000481000falseobj000000000000000000482023-10-18T20:30:32.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.669 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.669 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.670 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.670 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.670 DEBUG:boto:Method: GET 2023-10-18T20:30:51.671 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000048 2023-10-18T20:30:51.671 DEBUG:boto:Data: 2023-10-18T20:30:51.672 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.672 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.673 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.673 DEBUG:boto:Params: {} 2023-10-18T20:30:51.673 DEBUG:boto:Token: None 2023-10-18T20:30:51.674 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.674 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006611861df8b3cf9c-006530407b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.681 DEBUG:boto:b'bofrnf-58obj000000000000000000481000falseobj000000000000000000482023-10-18T20:30:32.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-18T20:30:51.682 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.683 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.683 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.684 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.684 DEBUG:boto:Method: GET 2023-10-18T20:30:51.684 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.685 DEBUG:boto:Data: 2023-10-18T20:30:51.685 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.686 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.687 DEBUG:boto:Params: {} 2023-10-18T20:30:51.687 DEBUG:boto:Token: None 2023-10-18T20:30:51.688 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OG4nK53eDztL44s3mMZgQNL8wzc= 2023-10-18T20:30:51.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OG4nK53eDztL44s3mMZgQNL8wzc=', 'Content-Length': '0'} 2023-10-18T20:30:51.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000743a36d8e2e6a530-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.692 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.693 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.693 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.693 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.694 DEBUG:boto:Method: GET 2023-10-18T20:30:51.694 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.695 DEBUG:boto:Data: 2023-10-18T20:30:51.695 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.695 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.696 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.696 DEBUG:boto:Params: {} 2023-10-18T20:30:51.697 DEBUG:boto:Token: None 2023-10-18T20:30:51.697 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.698 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OG4nK53eDztL44s3mMZgQNL8wzc= 2023-10-18T20:30:51.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OG4nK53eDztL44s3mMZgQNL8wzc=', 'Content-Length': '0'} 2023-10-18T20:30:51.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdc9d3c323b81778-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.742 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.743 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.743 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.744 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.744 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.745 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.745 DEBUG:boto:Data: 2023-10-18T20:30:51.745 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.746 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.747 DEBUG:boto:Params: {} 2023-10-18T20:30:51.747 DEBUG:boto:Token: None 2023-10-18T20:30:51.748 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9qCPwY0E1exhLASc3bRBq+YBS9w= 2023-10-18T20:30:51.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9qCPwY0E1exhLASc3bRBq+YBS9w=', 'Content-Length': '0'} 2023-10-18T20:30:51.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc42b15e8c0d6f1c-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.751 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.752 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.752 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.753 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.753 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.753 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.754 DEBUG:boto:Data: 2023-10-18T20:30:51.754 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.755 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.755 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.755 DEBUG:boto:Params: {} 2023-10-18T20:30:51.756 DEBUG:boto:Token: None 2023-10-18T20:30:51.756 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.757 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9qCPwY0E1exhLASc3bRBq+YBS9w= 2023-10-18T20:30:51.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9qCPwY0E1exhLASc3bRBq+YBS9w=', 'Content-Length': '0'} 2023-10-18T20:30:51.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e76b0b8aac2cf7-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-18T20:30:51.760 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.761 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.761 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.761 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.762 DEBUG:boto:Method: GET 2023-10-18T20:30:51.762 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.763 DEBUG:boto:Data: 2023-10-18T20:30:51.763 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.764 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.764 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.764 DEBUG:boto:Params: {} 2023-10-18T20:30:51.765 DEBUG:boto:Token: None 2023-10-18T20:30:51.765 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.766 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OG4nK53eDztL44s3mMZgQNL8wzc= 2023-10-18T20:30:51.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OG4nK53eDztL44s3mMZgQNL8wzc=', 'Content-Length': '0'} 2023-10-18T20:30:51.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b3db986f604155-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.814 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.815 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.815 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.816 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.816 DEBUG:boto:Method: GET 2023-10-18T20:30:51.817 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.817 DEBUG:boto:Data: 2023-10-18T20:30:51.817 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.818 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.818 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.819 DEBUG:boto:Params: {} 2023-10-18T20:30:51.819 DEBUG:boto:Token: None 2023-10-18T20:30:51.820 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049?versionId=null 2023-10-18T20:30:51.820 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OG4nK53eDztL44s3mMZgQNL8wzc= 2023-10-18T20:30:51.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OG4nK53eDztL44s3mMZgQNL8wzc=', 'Content-Length': '0'} 2023-10-18T20:30:51.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d74a397d3a3977-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.824 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.824 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.824 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.825 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.825 DEBUG:boto:Method: GET 2023-10-18T20:30:51.826 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.826 DEBUG:boto:Data: 2023-10-18T20:30:51.827 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.827 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.827 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.828 DEBUG:boto:Params: {} 2023-10-18T20:30:51.828 DEBUG:boto:Token: None 2023-10-18T20:30:51.829 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.829 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5422fb8c391442f-006530407b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.833 DEBUG:boto:b'bofrnf-58obj000000000000000000491000falseobj000000000000000000492023-10-18T20:30:29.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.834 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.834 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.834 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.835 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.835 DEBUG:boto:Method: GET 2023-10-18T20:30:51.836 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.836 DEBUG:boto:Data: 2023-10-18T20:30:51.836 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.837 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.838 DEBUG:boto:Params: {} 2023-10-18T20:30:51.838 DEBUG:boto:Token: None 2023-10-18T20:30:51.839 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.839 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce3e0d9d5eeaec11-006530407b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.844 DEBUG:boto:b'bofrnf-58obj000000000000000000491000falseobj000000000000000000492023-10-18T20:30:29.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.845 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.846 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.846 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.847 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.847 DEBUG:boto:Data: 2023-10-18T20:30:51.847 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.848 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.848 DEBUG:boto:Params: {} 2023-10-18T20:30:51.849 DEBUG:boto:Token: None 2023-10-18T20:30:51.849 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.851 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SlJI+rghx3skXZdcwJLvo0LUEzo= 2023-10-18T20:30:51.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SlJI+rghx3skXZdcwJLvo0LUEzo=', 'Content-Length': '0'} 2023-10-18T20:30:51.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006231973fc2bce3d1-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.854 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.854 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.855 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.855 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.856 DEBUG:boto:Data: 2023-10-18T20:30:51.856 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.856 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.857 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.857 DEBUG:boto:Params: {} 2023-10-18T20:30:51.858 DEBUG:boto:Token: None 2023-10-18T20:30:51.858 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SlJI+rghx3skXZdcwJLvo0LUEzo= 2023-10-18T20:30:51.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SlJI+rghx3skXZdcwJLvo0LUEzo=', 'Content-Length': '0'} 2023-10-18T20:30:51.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d98ea08cdf6a19c-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-18T20:30:51.862 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.863 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.863 DEBUG:boto:Method: GET 2023-10-18T20:30:51.864 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.864 DEBUG:boto:Data: 2023-10-18T20:30:51.865 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.865 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.866 DEBUG:boto:Params: {} 2023-10-18T20:30:51.866 DEBUG:boto:Token: None 2023-10-18T20:30:51.867 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1d8WitKwCN1BbKMVzKXRvShI0zI= 2023-10-18T20:30:51.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1d8WitKwCN1BbKMVzKXRvShI0zI=', 'Content-Length': '0'} 2023-10-18T20:30:51.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a50e1757ec1afb3-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.870 DEBUG:boto:path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.871 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.871 DEBUG:boto:Method: GET 2023-10-18T20:30:51.872 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.872 DEBUG:boto:Data: 2023-10-18T20:30:51.873 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.873 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.874 DEBUG:boto:Params: {} 2023-10-18T20:30:51.874 DEBUG:boto:Token: None 2023-10-18T20:30:51.875 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000049 2023-10-18T20:30:51.875 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1d8WitKwCN1BbKMVzKXRvShI0zI= 2023-10-18T20:30:51.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1d8WitKwCN1BbKMVzKXRvShI0zI=', 'Content-Length': '0'} 2023-10-18T20:30:51.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aa31c0d6a81c3d8-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.918 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.919 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.919 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.920 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.920 DEBUG:boto:Method: GET 2023-10-18T20:30:51.920 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.921 DEBUG:boto:Data: 2023-10-18T20:30:51.921 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.922 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.922 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.922 DEBUG:boto:Params: {} 2023-10-18T20:30:51.923 DEBUG:boto:Token: None 2023-10-18T20:30:51.923 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.924 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c2cfbb809d13feb-006530407b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.928 DEBUG:boto:b'bofrnf-58obj000000000000000000491000falseobj000000000000000000492023-10-18T20:30:29.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.928 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:51.929 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:51.929 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.930 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.930 DEBUG:boto:Method: GET 2023-10-18T20:30:51.930 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000049 2023-10-18T20:30:51.931 DEBUG:boto:Data: 2023-10-18T20:30:51.931 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.932 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.932 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.932 DEBUG:boto:Params: {} 2023-10-18T20:30:51.933 DEBUG:boto:Token: None 2023-10-18T20:30:51.933 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/ 2023-10-18T20:30:51.934 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE= 2023-10-18T20:30:51.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zt5etDSSQ+5FrANU4n7bQplFIPE=', 'Content-Length': '0'} 2023-10-18T20:30:51.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047a6c9330f2c91d5-006530407b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.939 DEBUG:boto:b'bofrnf-58obj000000000000000000491000falseobj000000000000000000492023-10-18T20:30:29.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:51.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-18T20:30:51.940 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.940 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.941 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.941 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.941 DEBUG:boto:Method: GET 2023-10-18T20:30:51.942 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.942 DEBUG:boto:Data: 2023-10-18T20:30:51.943 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.943 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.943 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.944 DEBUG:boto:Params: {} 2023-10-18T20:30:51.944 DEBUG:boto:Token: None 2023-10-18T20:30:51.945 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.945 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nYshScrfuuZaM1Di2RTNSoYkQg8= 2023-10-18T20:30:51.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nYshScrfuuZaM1Di2RTNSoYkQg8=', 'Content-Length': '0'} 2023-10-18T20:30:51.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000363632e667b76508-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.949 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.949 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.950 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.950 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.950 DEBUG:boto:Method: GET 2023-10-18T20:30:51.951 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.951 DEBUG:boto:Data: 2023-10-18T20:30:51.952 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.952 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.952 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.953 DEBUG:boto:Params: {} 2023-10-18T20:30:51.953 DEBUG:boto:Token: None 2023-10-18T20:30:51.954 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.954 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nYshScrfuuZaM1Di2RTNSoYkQg8= 2023-10-18T20:30:51.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nYshScrfuuZaM1Di2RTNSoYkQg8=', 'Content-Length': '0'} 2023-10-18T20:30:51.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000298f5adedef10f63-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.957 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.958 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.958 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.959 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.959 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.960 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.960 DEBUG:boto:Data: 2023-10-18T20:30:51.960 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.961 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.961 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.962 DEBUG:boto:Params: {} 2023-10-18T20:30:51.962 DEBUG:boto:Token: None 2023-10-18T20:30:51.963 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.963 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xu42Qsialbdig8ieqo/lC6Fpwtc= 2023-10-18T20:30:51.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xu42Qsialbdig8ieqo/lC6Fpwtc=', 'Content-Length': '0'} 2023-10-18T20:30:51.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae10f0a67c0d835d-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.967 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.967 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.968 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.968 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.968 DEBUG:boto:Method: HEAD 2023-10-18T20:30:51.969 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.969 DEBUG:boto:Data: 2023-10-18T20:30:51.970 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.970 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.970 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.971 DEBUG:boto:Params: {} 2023-10-18T20:30:51.971 DEBUG:boto:Token: None 2023-10-18T20:30:51.972 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.972 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xu42Qsialbdig8ieqo/lC6Fpwtc= 2023-10-18T20:30:51.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xu42Qsialbdig8ieqo/lC6Fpwtc=', 'Content-Length': '0'} 2023-10-18T20:30:51.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ec38959d3d11fff-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-18T20:30:51.976 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.976 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.977 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.977 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.978 DEBUG:boto:Method: GET 2023-10-18T20:30:51.978 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.978 DEBUG:boto:Data: 2023-10-18T20:30:51.979 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.979 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:51.980 DEBUG:boto:Port: 8001 2023-10-18T20:30:51.980 DEBUG:boto:Params: {} 2023-10-18T20:30:51.981 DEBUG:boto:Token: None 2023-10-18T20:30:51.981 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.982 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nYshScrfuuZaM1Di2RTNSoYkQg8= 2023-10-18T20:30:51.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nYshScrfuuZaM1Di2RTNSoYkQg8=', 'Content-Length': '0'} 2023-10-18T20:30:51.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046822f49ffd5d84f-006530407b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:51.985 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.985 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:51.986 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.986 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.986 DEBUG:boto:Method: GET 2023-10-18T20:30:51.987 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.987 DEBUG:boto:Data: 2023-10-18T20:30:51.988 DEBUG:boto:Headers: {} 2023-10-18T20:30:51.988 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:51.989 DEBUG:boto:Port: 8000 2023-10-18T20:30:51.989 DEBUG:boto:Params: {} 2023-10-18T20:30:51.989 DEBUG:boto:Token: None 2023-10-18T20:30:51.990 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:51 GMT /bofrnf-58/obj00000000000000000050?versionId=null 2023-10-18T20:30:51.990 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nYshScrfuuZaM1Di2RTNSoYkQg8= 2023-10-18T20:30:51.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nYshScrfuuZaM1Di2RTNSoYkQg8=', 'Content-Length': '0'} 2023-10-18T20:30:51.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071dfa3eadb885ffa-006530407b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.034 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.035 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.036 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.036 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.036 DEBUG:boto:Method: GET 2023-10-18T20:30:52.037 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.037 DEBUG:boto:Data: 2023-10-18T20:30:52.038 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.038 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.038 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.039 DEBUG:boto:Params: {} 2023-10-18T20:30:52.039 DEBUG:boto:Token: None 2023-10-18T20:30:52.040 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004807b9d0c652b83f-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.044 DEBUG:boto:b'bofrnf-58obj000000000000000000501000falseobj000000000000000000502023-10-18T20:30:30.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.045 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.046 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.046 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.046 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.047 DEBUG:boto:Method: GET 2023-10-18T20:30:52.047 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.048 DEBUG:boto:Data: 2023-10-18T20:30:52.048 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.048 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.049 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.049 DEBUG:boto:Params: {} 2023-10-18T20:30:52.050 DEBUG:boto:Token: None 2023-10-18T20:30:52.050 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.051 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c434a106a5a0391f-006530407c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.057 DEBUG:boto:b'bofrnf-58obj000000000000000000501000falseobj000000000000000000502023-10-18T20:30:30.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.058 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.058 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.059 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.059 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.059 DEBUG:boto:Data: 2023-10-18T20:30:52.060 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.060 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.061 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.061 DEBUG:boto:Params: {} 2023-10-18T20:30:52.062 DEBUG:boto:Token: None 2023-10-18T20:30:52.062 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.063 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F1CNKB6meGSzsINKJwlyU4ZSYjc= 2023-10-18T20:30:52.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F1CNKB6meGSzsINKJwlyU4ZSYjc=', 'Content-Length': '0'} 2023-10-18T20:30:52.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002023a6408861b468-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.066 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.066 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.067 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.067 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.068 DEBUG:boto:Data: 2023-10-18T20:30:52.068 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.068 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.069 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.069 DEBUG:boto:Params: {} 2023-10-18T20:30:52.070 DEBUG:boto:Token: None 2023-10-18T20:30:52.070 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.071 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F1CNKB6meGSzsINKJwlyU4ZSYjc= 2023-10-18T20:30:52.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F1CNKB6meGSzsINKJwlyU4ZSYjc=', 'Content-Length': '0'} 2023-10-18T20:30:52.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000156f31f398b46285-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-18T20:30:52.074 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.075 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.075 DEBUG:boto:Method: GET 2023-10-18T20:30:52.076 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.076 DEBUG:boto:Data: 2023-10-18T20:30:52.077 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.077 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.078 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.078 DEBUG:boto:Params: {} 2023-10-18T20:30:52.079 DEBUG:boto:Token: None 2023-10-18T20:30:52.079 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.079 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rfIAc2YmO4AVIp59Zm2LptYxoCU= 2023-10-18T20:30:52.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rfIAc2YmO4AVIp59Zm2LptYxoCU=', 'Content-Length': '0'} 2023-10-18T20:30:52.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a948f06b4235088-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.122 DEBUG:boto:path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.123 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.123 DEBUG:boto:Method: GET 2023-10-18T20:30:52.124 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.124 DEBUG:boto:Data: 2023-10-18T20:30:52.125 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.125 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.126 DEBUG:boto:Params: {} 2023-10-18T20:30:52.126 DEBUG:boto:Token: None 2023-10-18T20:30:52.127 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000050 2023-10-18T20:30:52.127 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rfIAc2YmO4AVIp59Zm2LptYxoCU= 2023-10-18T20:30:52.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rfIAc2YmO4AVIp59Zm2LptYxoCU=', 'Content-Length': '0'} 2023-10-18T20:30:52.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074cfbcdd11b36ef1-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.131 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.131 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.132 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.132 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.132 DEBUG:boto:Method: GET 2023-10-18T20:30:52.133 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.133 DEBUG:boto:Data: 2023-10-18T20:30:52.134 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.134 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.135 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.135 DEBUG:boto:Params: {} 2023-10-18T20:30:52.135 DEBUG:boto:Token: None 2023-10-18T20:30:52.136 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.136 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031748cce6fbf1537-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.140 DEBUG:boto:b'bofrnf-58obj000000000000000000501000falseobj000000000000000000502023-10-18T20:30:30.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.141 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.141 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.142 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.142 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.143 DEBUG:boto:Method: GET 2023-10-18T20:30:52.143 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000050 2023-10-18T20:30:52.143 DEBUG:boto:Data: 2023-10-18T20:30:52.144 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.144 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.145 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.145 DEBUG:boto:Params: {} 2023-10-18T20:30:52.145 DEBUG:boto:Token: None 2023-10-18T20:30:52.146 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.146 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b36d0a1129ae6125-006530407c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.195 DEBUG:boto:b'bofrnf-58obj000000000000000000501000falseobj000000000000000000502023-10-18T20:30:30.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-10-18T20:30:52.196 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.196 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.197 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.197 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.198 DEBUG:boto:Method: GET 2023-10-18T20:30:52.198 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.199 DEBUG:boto:Data: 2023-10-18T20:30:52.199 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.200 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.200 DEBUG:boto:Params: {} 2023-10-18T20:30:52.201 DEBUG:boto:Token: None 2023-10-18T20:30:52.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.202 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uOVP483EZ73gRMYwbgobL5CCD6Y= 2023-10-18T20:30:52.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uOVP483EZ73gRMYwbgobL5CCD6Y=', 'Content-Length': '0'} 2023-10-18T20:30:52.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031147de7ac5a910e-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.206 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.206 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.206 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.207 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.207 DEBUG:boto:Method: GET 2023-10-18T20:30:52.208 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.208 DEBUG:boto:Data: 2023-10-18T20:30:52.209 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.209 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.209 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.210 DEBUG:boto:Params: {} 2023-10-18T20:30:52.210 DEBUG:boto:Token: None 2023-10-18T20:30:52.211 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uOVP483EZ73gRMYwbgobL5CCD6Y= 2023-10-18T20:30:52.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uOVP483EZ73gRMYwbgobL5CCD6Y=', 'Content-Length': '0'} 2023-10-18T20:30:52.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646009f928defc2d-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.214 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.215 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.215 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.216 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.216 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.216 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.217 DEBUG:boto:Data: 2023-10-18T20:30:52.217 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.217 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.218 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.218 DEBUG:boto:Params: {} 2023-10-18T20:30:52.219 DEBUG:boto:Token: None 2023-10-18T20:30:52.219 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F3JD8BD0Eop/+uSDeS3G6up8YXY= 2023-10-18T20:30:52.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F3JD8BD0Eop/+uSDeS3G6up8YXY=', 'Content-Length': '0'} 2023-10-18T20:30:52.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a9d2c217f28198-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.222 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.223 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.223 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.224 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.224 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.224 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.225 DEBUG:boto:Data: 2023-10-18T20:30:52.225 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.226 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.226 DEBUG:boto:Params: {} 2023-10-18T20:30:52.227 DEBUG:boto:Token: None 2023-10-18T20:30:52.227 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.228 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F3JD8BD0Eop/+uSDeS3G6up8YXY= 2023-10-18T20:30:52.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F3JD8BD0Eop/+uSDeS3G6up8YXY=', 'Content-Length': '0'} 2023-10-18T20:30:52.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e61a18415df803f2-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-10-18T20:30:52.231 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.232 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.232 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.232 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.233 DEBUG:boto:Method: GET 2023-10-18T20:30:52.233 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.234 DEBUG:boto:Data: 2023-10-18T20:30:52.234 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.235 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.235 DEBUG:boto:Params: {} 2023-10-18T20:30:52.236 DEBUG:boto:Token: None 2023-10-18T20:30:52.236 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uOVP483EZ73gRMYwbgobL5CCD6Y= 2023-10-18T20:30:52.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uOVP483EZ73gRMYwbgobL5CCD6Y=', 'Content-Length': '0'} 2023-10-18T20:30:52.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000319dc80394f651b3-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.240 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.241 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.241 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.241 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.242 DEBUG:boto:Method: GET 2023-10-18T20:30:52.242 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.243 DEBUG:boto:Data: 2023-10-18T20:30:52.243 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.243 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.244 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.244 DEBUG:boto:Params: {} 2023-10-18T20:30:52.245 DEBUG:boto:Token: None 2023-10-18T20:30:52.245 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053?versionId=null 2023-10-18T20:30:52.246 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uOVP483EZ73gRMYwbgobL5CCD6Y= 2023-10-18T20:30:52.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uOVP483EZ73gRMYwbgobL5CCD6Y=', 'Content-Length': '0'} 2023-10-18T20:30:52.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060ffdee8f9091471-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.249 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.249 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.250 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.250 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.250 DEBUG:boto:Method: GET 2023-10-18T20:30:52.251 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.251 DEBUG:boto:Data: 2023-10-18T20:30:52.252 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.252 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.253 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.253 DEBUG:boto:Params: {} 2023-10-18T20:30:52.253 DEBUG:boto:Token: None 2023-10-18T20:30:52.254 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.254 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b48c17dfe0b94d22-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.258 DEBUG:boto:b'bofrnf-58obj000000000000000000531000falseobj000000000000000000532023-10-18T20:30:32.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.259 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.259 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.259 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.260 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.260 DEBUG:boto:Method: GET 2023-10-18T20:30:52.261 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.261 DEBUG:boto:Data: 2023-10-18T20:30:52.262 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.262 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.262 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.263 DEBUG:boto:Params: {} 2023-10-18T20:30:52.263 DEBUG:boto:Token: None 2023-10-18T20:30:52.264 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.264 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d10446699b6955d-006530407c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.269 DEBUG:boto:b'bofrnf-58obj000000000000000000531000falseobj000000000000000000532023-10-18T20:30:32.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.270 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.271 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.271 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.271 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.272 DEBUG:boto:Data: 2023-10-18T20:30:52.272 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.273 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.273 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.273 DEBUG:boto:Params: {} 2023-10-18T20:30:52.274 DEBUG:boto:Token: None 2023-10-18T20:30:52.274 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.275 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pmiK2qekVteCowwZdsKEVkqb/m8= 2023-10-18T20:30:52.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pmiK2qekVteCowwZdsKEVkqb/m8=', 'Content-Length': '0'} 2023-10-18T20:30:52.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4ac9ccb0e55fba-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.278 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.278 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.279 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.279 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.279 DEBUG:boto:Data: 2023-10-18T20:30:52.280 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.280 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.281 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.281 DEBUG:boto:Params: {} 2023-10-18T20:30:52.281 DEBUG:boto:Token: None 2023-10-18T20:30:52.282 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.282 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pmiK2qekVteCowwZdsKEVkqb/m8= 2023-10-18T20:30:52.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pmiK2qekVteCowwZdsKEVkqb/m8=', 'Content-Length': '0'} 2023-10-18T20:30:52.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000474387cc2804fb4-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-10-18T20:30:52.286 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.286 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.287 DEBUG:boto:Method: GET 2023-10-18T20:30:52.287 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.288 DEBUG:boto:Data: 2023-10-18T20:30:52.288 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.289 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.289 DEBUG:boto:Params: {} 2023-10-18T20:30:52.290 DEBUG:boto:Token: None 2023-10-18T20:30:52.290 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IZAgcTHLI5MRXvgK7u7d+kbmGf0= 2023-10-18T20:30:52.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IZAgcTHLI5MRXvgK7u7d+kbmGf0=', 'Content-Length': '0'} 2023-10-18T20:30:52.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a49e22de81947e9-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.296 DEBUG:boto:path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.297 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.297 DEBUG:boto:Method: GET 2023-10-18T20:30:52.298 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.298 DEBUG:boto:Data: 2023-10-18T20:30:52.298 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.299 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.300 DEBUG:boto:Params: {} 2023-10-18T20:30:52.300 DEBUG:boto:Token: None 2023-10-18T20:30:52.301 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000053 2023-10-18T20:30:52.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IZAgcTHLI5MRXvgK7u7d+kbmGf0= 2023-10-18T20:30:52.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IZAgcTHLI5MRXvgK7u7d+kbmGf0=', 'Content-Length': '0'} 2023-10-18T20:30:52.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a91f7acbff1370a-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.304 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.305 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.305 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.306 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.306 DEBUG:boto:Method: GET 2023-10-18T20:30:52.307 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.307 DEBUG:boto:Data: 2023-10-18T20:30:52.307 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.308 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.308 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.309 DEBUG:boto:Params: {} 2023-10-18T20:30:52.309 DEBUG:boto:Token: None 2023-10-18T20:30:52.310 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.310 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1fb10df8c389691-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.314 DEBUG:boto:b'bofrnf-58obj000000000000000000531000falseobj000000000000000000532023-10-18T20:30:32.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.314 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.315 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.315 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.316 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.316 DEBUG:boto:Method: GET 2023-10-18T20:30:52.316 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000053 2023-10-18T20:30:52.317 DEBUG:boto:Data: 2023-10-18T20:30:52.317 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.318 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.319 DEBUG:boto:Params: {} 2023-10-18T20:30:52.319 DEBUG:boto:Token: None 2023-10-18T20:30:52.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017c3373543aeaffd-006530407c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.325 DEBUG:boto:b'bofrnf-58obj000000000000000000531000falseobj000000000000000000532023-10-18T20:30:32.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-10-18T20:30:52.327 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.327 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.327 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.328 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.328 DEBUG:boto:Method: GET 2023-10-18T20:30:52.329 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.329 DEBUG:boto:Data: 2023-10-18T20:30:52.330 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.330 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.330 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.331 DEBUG:boto:Params: {} 2023-10-18T20:30:52.331 DEBUG:boto:Token: None 2023-10-18T20:30:52.332 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.332 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5aof5SXqEPj3LfGMl51Ixjk43Hw= 2023-10-18T20:30:52.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5aof5SXqEPj3LfGMl51Ixjk43Hw=', 'Content-Length': '0'} 2023-10-18T20:30:52.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab6c0f2b59026219-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.336 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.336 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.336 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.337 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.337 DEBUG:boto:Method: GET 2023-10-18T20:30:52.338 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.338 DEBUG:boto:Data: 2023-10-18T20:30:52.339 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.339 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.339 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.340 DEBUG:boto:Params: {} 2023-10-18T20:30:52.340 DEBUG:boto:Token: None 2023-10-18T20:30:52.341 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.341 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5aof5SXqEPj3LfGMl51Ixjk43Hw= 2023-10-18T20:30:52.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5aof5SXqEPj3LfGMl51Ixjk43Hw=', 'Content-Length': '0'} 2023-10-18T20:30:52.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e9b8da4f284b52-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.386 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.387 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.387 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.388 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.388 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.389 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.389 DEBUG:boto:Data: 2023-10-18T20:30:52.389 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.390 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.391 DEBUG:boto:Params: {} 2023-10-18T20:30:52.391 DEBUG:boto:Token: None 2023-10-18T20:30:52.392 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wDiWWK/5JFHgp+3uW9pbgGz+eJM= 2023-10-18T20:30:52.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wDiWWK/5JFHgp+3uW9pbgGz+eJM=', 'Content-Length': '0'} 2023-10-18T20:30:52.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6029eaa742827c-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.395 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.396 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.396 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.396 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.397 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.397 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.398 DEBUG:boto:Data: 2023-10-18T20:30:52.398 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.399 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.399 DEBUG:boto:Params: {} 2023-10-18T20:30:52.400 DEBUG:boto:Token: None 2023-10-18T20:30:52.400 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wDiWWK/5JFHgp+3uW9pbgGz+eJM= 2023-10-18T20:30:52.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wDiWWK/5JFHgp+3uW9pbgGz+eJM=', 'Content-Length': '0'} 2023-10-18T20:30:52.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038733580c5af377d-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-10-18T20:30:52.404 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.404 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.405 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.405 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.406 DEBUG:boto:Method: GET 2023-10-18T20:30:52.406 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.406 DEBUG:boto:Data: 2023-10-18T20:30:52.407 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.407 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.408 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.408 DEBUG:boto:Params: {} 2023-10-18T20:30:52.409 DEBUG:boto:Token: None 2023-10-18T20:30:52.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5aof5SXqEPj3LfGMl51Ixjk43Hw= 2023-10-18T20:30:52.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5aof5SXqEPj3LfGMl51Ixjk43Hw=', 'Content-Length': '0'} 2023-10-18T20:30:52.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c90dca7fac9f2612-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.413 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.413 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.414 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.414 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.415 DEBUG:boto:Method: GET 2023-10-18T20:30:52.415 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.415 DEBUG:boto:Data: 2023-10-18T20:30:52.416 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.416 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.417 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.417 DEBUG:boto:Params: {} 2023-10-18T20:30:52.417 DEBUG:boto:Token: None 2023-10-18T20:30:52.418 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055?versionId=null 2023-10-18T20:30:52.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5aof5SXqEPj3LfGMl51Ixjk43Hw= 2023-10-18T20:30:52.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5aof5SXqEPj3LfGMl51Ixjk43Hw=', 'Content-Length': '0'} 2023-10-18T20:30:52.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064018df8b41ae227-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.422 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.422 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.423 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.423 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.423 DEBUG:boto:Method: GET 2023-10-18T20:30:52.424 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.424 DEBUG:boto:Data: 2023-10-18T20:30:52.425 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.425 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.426 DEBUG:boto:Params: {} 2023-10-18T20:30:52.426 DEBUG:boto:Token: None 2023-10-18T20:30:52.427 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.427 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000395d0bd331f363c6-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.431 DEBUG:boto:b'bofrnf-58obj000000000000000000551000falseobj000000000000000000552023-10-18T20:30:27.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.431 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.432 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.432 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.433 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.433 DEBUG:boto:Method: GET 2023-10-18T20:30:52.433 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.434 DEBUG:boto:Data: 2023-10-18T20:30:52.434 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.435 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.435 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.435 DEBUG:boto:Params: {} 2023-10-18T20:30:52.436 DEBUG:boto:Token: None 2023-10-18T20:30:52.436 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.437 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45ee7d37413a7f6-006530407c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.442 DEBUG:boto:b'bofrnf-58obj000000000000000000551000falseobj000000000000000000552023-10-18T20:30:27.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.442 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.443 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.443 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.444 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.444 DEBUG:boto:Data: 2023-10-18T20:30:52.444 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.445 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.446 DEBUG:boto:Params: {} 2023-10-18T20:30:52.446 DEBUG:boto:Token: None 2023-10-18T20:30:52.446 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.447 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:21zCA3wS1Kusdit1GHS5Ht9B/5c= 2023-10-18T20:30:52.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:21zCA3wS1Kusdit1GHS5Ht9B/5c=', 'Content-Length': '0'} 2023-10-18T20:30:52.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000974ef63ea4d7d1f5-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.450 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.450 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.451 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.451 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.452 DEBUG:boto:Data: 2023-10-18T20:30:52.452 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.453 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.453 DEBUG:boto:Params: {} 2023-10-18T20:30:52.454 DEBUG:boto:Token: None 2023-10-18T20:30:52.454 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:21zCA3wS1Kusdit1GHS5Ht9B/5c= 2023-10-18T20:30:52.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:21zCA3wS1Kusdit1GHS5Ht9B/5c=', 'Content-Length': '0'} 2023-10-18T20:30:52.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009424198c4b84ab9c-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-10-18T20:30:52.458 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.458 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.459 DEBUG:boto:Method: GET 2023-10-18T20:30:52.459 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.460 DEBUG:boto:Data: 2023-10-18T20:30:52.460 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.461 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.461 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.461 DEBUG:boto:Params: {} 2023-10-18T20:30:52.462 DEBUG:boto:Token: None 2023-10-18T20:30:52.462 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.463 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n0cp3DrRjsYQILRbT1WxpMLKGL0= 2023-10-18T20:30:52.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n0cp3DrRjsYQILRbT1WxpMLKGL0=', 'Content-Length': '0'} 2023-10-18T20:30:52.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029a6957748a3c76a-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.506 DEBUG:boto:path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.507 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.508 DEBUG:boto:Method: GET 2023-10-18T20:30:52.508 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.508 DEBUG:boto:Data: 2023-10-18T20:30:52.509 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.509 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.510 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.510 DEBUG:boto:Params: {} 2023-10-18T20:30:52.511 DEBUG:boto:Token: None 2023-10-18T20:30:52.511 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000055 2023-10-18T20:30:52.512 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n0cp3DrRjsYQILRbT1WxpMLKGL0= 2023-10-18T20:30:52.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n0cp3DrRjsYQILRbT1WxpMLKGL0=', 'Content-Length': '0'} 2023-10-18T20:30:52.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbafde7f7fb5012f-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.515 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.516 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.516 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.517 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.517 DEBUG:boto:Method: GET 2023-10-18T20:30:52.518 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.518 DEBUG:boto:Data: 2023-10-18T20:30:52.518 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.519 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.519 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.520 DEBUG:boto:Params: {} 2023-10-18T20:30:52.520 DEBUG:boto:Token: None 2023-10-18T20:30:52.521 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.521 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2fb087e32262b0-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.525 DEBUG:boto:b'bofrnf-58obj000000000000000000551000falseobj000000000000000000552023-10-18T20:30:27.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.526 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.527 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.527 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.527 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.528 DEBUG:boto:Method: GET 2023-10-18T20:30:52.528 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000055 2023-10-18T20:30:52.529 DEBUG:boto:Data: 2023-10-18T20:30:52.529 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.530 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.531 DEBUG:boto:Params: {} 2023-10-18T20:30:52.531 DEBUG:boto:Token: None 2023-10-18T20:30:52.532 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070430f6cc448f7e7-006530407c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.538 DEBUG:boto:b'bofrnf-58obj000000000000000000551000falseobj000000000000000000552023-10-18T20:30:27.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-18T20:30:52.539 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.539 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.540 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.540 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.541 DEBUG:boto:Method: GET 2023-10-18T20:30:52.541 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.541 DEBUG:boto:Data: 2023-10-18T20:30:52.542 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.543 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.543 DEBUG:boto:Params: {} 2023-10-18T20:30:52.544 DEBUG:boto:Token: None 2023-10-18T20:30:52.544 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iUHSBROhnCta83kDfEkvbkhJaaE= 2023-10-18T20:30:52.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iUHSBROhnCta83kDfEkvbkhJaaE=', 'Content-Length': '0'} 2023-10-18T20:30:52.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4469d505eecc2f0-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.548 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.549 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.549 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.550 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.550 DEBUG:boto:Method: GET 2023-10-18T20:30:52.551 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.551 DEBUG:boto:Data: 2023-10-18T20:30:52.552 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.552 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.553 DEBUG:boto:Params: {} 2023-10-18T20:30:52.553 DEBUG:boto:Token: None 2023-10-18T20:30:52.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.554 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iUHSBROhnCta83kDfEkvbkhJaaE= 2023-10-18T20:30:52.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iUHSBROhnCta83kDfEkvbkhJaaE=', 'Content-Length': '0'} 2023-10-18T20:30:52.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fac56f158501f0a-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.558 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.558 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.559 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.559 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.559 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.560 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.560 DEBUG:boto:Data: 2023-10-18T20:30:52.561 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.561 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.561 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.562 DEBUG:boto:Params: {} 2023-10-18T20:30:52.562 DEBUG:boto:Token: None 2023-10-18T20:30:52.563 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.563 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JFO6SAE5MN0kar81hPGbkPOWlMw= 2023-10-18T20:30:52.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JFO6SAE5MN0kar81hPGbkPOWlMw=', 'Content-Length': '0'} 2023-10-18T20:30:52.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001531771d7040de6a-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.566 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.567 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.567 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.567 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.568 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.568 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.569 DEBUG:boto:Data: 2023-10-18T20:30:52.569 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.569 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.570 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.570 DEBUG:boto:Params: {} 2023-10-18T20:30:52.571 DEBUG:boto:Token: None 2023-10-18T20:30:52.571 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.572 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JFO6SAE5MN0kar81hPGbkPOWlMw= 2023-10-18T20:30:52.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JFO6SAE5MN0kar81hPGbkPOWlMw=', 'Content-Length': '0'} 2023-10-18T20:30:52.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000564a8e2b184e7889-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-18T20:30:52.575 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.575 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.576 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.576 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.577 DEBUG:boto:Method: GET 2023-10-18T20:30:52.577 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.578 DEBUG:boto:Data: 2023-10-18T20:30:52.578 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.579 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.579 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.579 DEBUG:boto:Params: {} 2023-10-18T20:30:52.580 DEBUG:boto:Token: None 2023-10-18T20:30:52.580 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iUHSBROhnCta83kDfEkvbkhJaaE= 2023-10-18T20:30:52.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iUHSBROhnCta83kDfEkvbkhJaaE=', 'Content-Length': '0'} 2023-10-18T20:30:52.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7fd66a43ac850e8-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.584 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.585 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.585 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.585 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.586 DEBUG:boto:Method: GET 2023-10-18T20:30:52.586 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.587 DEBUG:boto:Data: 2023-10-18T20:30:52.587 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.588 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.588 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.588 DEBUG:boto:Params: {} 2023-10-18T20:30:52.589 DEBUG:boto:Token: None 2023-10-18T20:30:52.589 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057?versionId=null 2023-10-18T20:30:52.590 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iUHSBROhnCta83kDfEkvbkhJaaE= 2023-10-18T20:30:52.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iUHSBROhnCta83kDfEkvbkhJaaE=', 'Content-Length': '0'} 2023-10-18T20:30:52.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000693dd559100deaeb-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.593 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.593 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.594 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.594 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.595 DEBUG:boto:Method: GET 2023-10-18T20:30:52.595 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.596 DEBUG:boto:Data: 2023-10-18T20:30:52.596 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.597 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.598 DEBUG:boto:Params: {} 2023-10-18T20:30:52.598 DEBUG:boto:Token: None 2023-10-18T20:30:52.599 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.599 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0815d7b983dea06-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.646 DEBUG:boto:b'bofrnf-58obj000000000000000000571000falseobj000000000000000000572023-10-18T20:30:31.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.647 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.648 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.648 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.649 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.649 DEBUG:boto:Method: GET 2023-10-18T20:30:52.650 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.650 DEBUG:boto:Data: 2023-10-18T20:30:52.650 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.651 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.652 DEBUG:boto:Params: {} 2023-10-18T20:30:52.652 DEBUG:boto:Token: None 2023-10-18T20:30:52.653 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eaf2f3d63d17795-006530407c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.659 DEBUG:boto:b'bofrnf-58obj000000000000000000571000falseobj000000000000000000572023-10-18T20:30:31.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.659 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.660 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.660 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.661 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.661 DEBUG:boto:Data: 2023-10-18T20:30:52.661 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.662 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.663 DEBUG:boto:Params: {} 2023-10-18T20:30:52.663 DEBUG:boto:Token: None 2023-10-18T20:30:52.663 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kjVnY82PvxC7byJLfBkUTkf4Ge4= 2023-10-18T20:30:52.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kjVnY82PvxC7byJLfBkUTkf4Ge4=', 'Content-Length': '0'} 2023-10-18T20:30:52.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068daa96233e3d01f-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.667 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.668 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.668 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.668 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.669 DEBUG:boto:Data: 2023-10-18T20:30:52.669 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.670 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.670 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.671 DEBUG:boto:Params: {} 2023-10-18T20:30:52.671 DEBUG:boto:Token: None 2023-10-18T20:30:52.671 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.672 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kjVnY82PvxC7byJLfBkUTkf4Ge4= 2023-10-18T20:30:52.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kjVnY82PvxC7byJLfBkUTkf4Ge4=', 'Content-Length': '0'} 2023-10-18T20:30:52.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011c7b759d68e61e7-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-18T20:30:52.676 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.676 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.676 DEBUG:boto:Method: GET 2023-10-18T20:30:52.677 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.677 DEBUG:boto:Data: 2023-10-18T20:30:52.678 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.678 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.679 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.679 DEBUG:boto:Params: {} 2023-10-18T20:30:52.680 DEBUG:boto:Token: None 2023-10-18T20:30:52.680 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.681 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0ChSSZV2R2hhu6reW2Ti4Knql3k= 2023-10-18T20:30:52.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0ChSSZV2R2hhu6reW2Ti4Knql3k=', 'Content-Length': '0'} 2023-10-18T20:30:52.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8d5b31c0890d8b-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.684 DEBUG:boto:path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.684 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.685 DEBUG:boto:Method: GET 2023-10-18T20:30:52.685 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.686 DEBUG:boto:Data: 2023-10-18T20:30:52.686 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.687 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.687 DEBUG:boto:Params: {} 2023-10-18T20:30:52.688 DEBUG:boto:Token: None 2023-10-18T20:30:52.688 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000057 2023-10-18T20:30:52.689 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0ChSSZV2R2hhu6reW2Ti4Knql3k= 2023-10-18T20:30:52.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0ChSSZV2R2hhu6reW2Ti4Knql3k=', 'Content-Length': '0'} 2023-10-18T20:30:52.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57e6c75880f3800-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.734 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.735 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.735 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.735 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.736 DEBUG:boto:Method: GET 2023-10-18T20:30:52.736 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.737 DEBUG:boto:Data: 2023-10-18T20:30:52.737 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.738 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.738 DEBUG:boto:Params: {} 2023-10-18T20:30:52.739 DEBUG:boto:Token: None 2023-10-18T20:30:52.739 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5b88b7656b86948-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.743 DEBUG:boto:b'bofrnf-58obj000000000000000000571000falseobj000000000000000000572023-10-18T20:30:31.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.744 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.745 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.745 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.745 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.746 DEBUG:boto:Method: GET 2023-10-18T20:30:52.746 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000057 2023-10-18T20:30:52.747 DEBUG:boto:Data: 2023-10-18T20:30:52.747 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.748 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.748 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.749 DEBUG:boto:Params: {} 2023-10-18T20:30:52.749 DEBUG:boto:Token: None 2023-10-18T20:30:52.750 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.751 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c43eb62ad15bc191-006530407c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.757 DEBUG:boto:b'bofrnf-58obj000000000000000000571000falseobj000000000000000000572023-10-18T20:30:31.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-18T20:30:52.758 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.758 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.759 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.759 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.760 DEBUG:boto:Method: GET 2023-10-18T20:30:52.760 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.761 DEBUG:boto:Data: 2023-10-18T20:30:52.761 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.762 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.762 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.762 DEBUG:boto:Params: {} 2023-10-18T20:30:52.763 DEBUG:boto:Token: None 2023-10-18T20:30:52.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z1arsFB7iqN9wbJCCku8TWQ3Jfk= 2023-10-18T20:30:52.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z1arsFB7iqN9wbJCCku8TWQ3Jfk=', 'Content-Length': '0'} 2023-10-18T20:30:52.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb227c813db10f13-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.810 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.811 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.811 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.812 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.812 DEBUG:boto:Method: GET 2023-10-18T20:30:52.813 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.813 DEBUG:boto:Data: 2023-10-18T20:30:52.814 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.814 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.815 DEBUG:boto:Params: {} 2023-10-18T20:30:52.815 DEBUG:boto:Token: None 2023-10-18T20:30:52.816 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z1arsFB7iqN9wbJCCku8TWQ3Jfk= 2023-10-18T20:30:52.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z1arsFB7iqN9wbJCCku8TWQ3Jfk=', 'Content-Length': '0'} 2023-10-18T20:30:52.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a84c647fbd87540-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.820 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.820 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.820 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.821 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.821 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.822 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.822 DEBUG:boto:Data: 2023-10-18T20:30:52.822 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.823 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.823 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.824 DEBUG:boto:Params: {} 2023-10-18T20:30:52.824 DEBUG:boto:Token: None 2023-10-18T20:30:52.825 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.825 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fQbCZjPSvUCrM1RBVnRk5OHQejE= 2023-10-18T20:30:52.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fQbCZjPSvUCrM1RBVnRk5OHQejE=', 'Content-Length': '0'} 2023-10-18T20:30:52.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018dee440bf3b98f3-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.828 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.828 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.829 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.829 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.830 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.830 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.830 DEBUG:boto:Data: 2023-10-18T20:30:52.831 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.831 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.832 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.832 DEBUG:boto:Params: {} 2023-10-18T20:30:52.833 DEBUG:boto:Token: None 2023-10-18T20:30:52.833 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fQbCZjPSvUCrM1RBVnRk5OHQejE= 2023-10-18T20:30:52.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fQbCZjPSvUCrM1RBVnRk5OHQejE=', 'Content-Length': '0'} 2023-10-18T20:30:52.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088320c077aa1a7e9-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-18T20:30:52.837 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.838 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.838 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.838 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.839 DEBUG:boto:Method: GET 2023-10-18T20:30:52.839 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.840 DEBUG:boto:Data: 2023-10-18T20:30:52.840 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.841 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.841 DEBUG:boto:Params: {} 2023-10-18T20:30:52.842 DEBUG:boto:Token: None 2023-10-18T20:30:52.842 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.843 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z1arsFB7iqN9wbJCCku8TWQ3Jfk= 2023-10-18T20:30:52.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z1arsFB7iqN9wbJCCku8TWQ3Jfk=', 'Content-Length': '0'} 2023-10-18T20:30:52.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abbd8f6c25327691-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.846 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.846 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.847 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.847 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.848 DEBUG:boto:Method: GET 2023-10-18T20:30:52.848 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.848 DEBUG:boto:Data: 2023-10-18T20:30:52.849 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.849 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.850 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.850 DEBUG:boto:Params: {} 2023-10-18T20:30:52.851 DEBUG:boto:Token: None 2023-10-18T20:30:52.851 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058?versionId=null 2023-10-18T20:30:52.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z1arsFB7iqN9wbJCCku8TWQ3Jfk= 2023-10-18T20:30:52.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z1arsFB7iqN9wbJCCku8TWQ3Jfk=', 'Content-Length': '0'} 2023-10-18T20:30:52.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000156ca73fcafe7020-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.855 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.855 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.856 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:52.856 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:52.857 DEBUG:boto:Method: GET 2023-10-18T20:30:52.857 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:52.857 DEBUG:boto:Data: 2023-10-18T20:30:52.858 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.859 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.859 DEBUG:boto:Params: {} 2023-10-18T20:30:52.860 DEBUG:boto:Token: None 2023-10-18T20:30:52.860 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f13ebeb4d7c6829-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.864 DEBUG:boto:b'bofrnf-58obj000000000000000000581000falseobj000000000000000000582023-10-18T20:30:28.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.865 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.865 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.866 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:52.866 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:52.867 DEBUG:boto:Method: GET 2023-10-18T20:30:52.867 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:52.868 DEBUG:boto:Data: 2023-10-18T20:30:52.868 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.869 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.869 DEBUG:boto:Params: {} 2023-10-18T20:30:52.870 DEBUG:boto:Token: None 2023-10-18T20:30:52.870 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.871 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8ac500c4ec8f7da-006530407c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.918 DEBUG:boto:b'bofrnf-58obj000000000000000000581000falseobj000000000000000000582023-10-18T20:30:28.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:52.919 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.920 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.920 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.921 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.921 DEBUG:boto:Data: 2023-10-18T20:30:52.921 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.922 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.922 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.923 DEBUG:boto:Params: {} 2023-10-18T20:30:52.923 DEBUG:boto:Token: None 2023-10-18T20:30:52.924 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.924 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s+Se9RlOUiHF1yI9U6Z1Dq1/wPU= 2023-10-18T20:30:52.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s+Se9RlOUiHF1yI9U6Z1Dq1/wPU=', 'Content-Length': '0'} 2023-10-18T20:30:52.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed0facd18c1723a-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.927 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.928 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.928 DEBUG:boto:Method: HEAD 2023-10-18T20:30:52.928 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.929 DEBUG:boto:Data: 2023-10-18T20:30:52.929 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.930 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.930 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.931 DEBUG:boto:Params: {} 2023-10-18T20:30:52.931 DEBUG:boto:Token: None 2023-10-18T20:30:52.931 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.932 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s+Se9RlOUiHF1yI9U6Z1Dq1/wPU= 2023-10-18T20:30:52.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s+Se9RlOUiHF1yI9U6Z1Dq1/wPU=', 'Content-Length': '0'} 2023-10-18T20:30:52.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de7b6ee1796fab5-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-18T20:30:52.935 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.936 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.936 DEBUG:boto:Method: GET 2023-10-18T20:30:52.937 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.937 DEBUG:boto:Data: 2023-10-18T20:30:52.938 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.938 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.938 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.939 DEBUG:boto:Params: {} 2023-10-18T20:30:52.939 DEBUG:boto:Token: None 2023-10-18T20:30:52.940 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.940 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/XPRvd7Cm7lYiqWKbpU1gcw6Sw= 2023-10-18T20:30:52.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/XPRvd7Cm7lYiqWKbpU1gcw6Sw=', 'Content-Length': '0'} 2023-10-18T20:30:52.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca9bc9c6c94c4730-006530407c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.944 DEBUG:boto:path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.944 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.945 DEBUG:boto:Method: GET 2023-10-18T20:30:52.945 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.945 DEBUG:boto:Data: 2023-10-18T20:30:52.946 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.946 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:52.947 DEBUG:boto:Port: 8000 2023-10-18T20:30:52.947 DEBUG:boto:Params: {} 2023-10-18T20:30:52.948 DEBUG:boto:Token: None 2023-10-18T20:30:52.948 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/obj00000000000000000058 2023-10-18T20:30:52.949 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d/XPRvd7Cm7lYiqWKbpU1gcw6Sw= 2023-10-18T20:30:52.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d/XPRvd7Cm7lYiqWKbpU1gcw6Sw=', 'Content-Length': '0'} 2023-10-18T20:30:52.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2b7527bf85c45f-006530407c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:52.952 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:52.952 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:52.953 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:52.953 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:52.954 DEBUG:boto:Method: GET 2023-10-18T20:30:52.954 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:52.955 DEBUG:boto:Data: 2023-10-18T20:30:52.955 DEBUG:boto:Headers: {} 2023-10-18T20:30:52.956 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:52.956 DEBUG:boto:Port: 8001 2023-10-18T20:30:52.956 DEBUG:boto:Params: {} 2023-10-18T20:30:52.957 DEBUG:boto:Token: None 2023-10-18T20:30:52.957 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:52 GMT /bofrnf-58/ 2023-10-18T20:30:52.958 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY= 2023-10-18T20:30:52.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:52 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tpkVQz4KnEYgnZrR8obs+TlFzuY=', 'Content-Length': '0'} 2023-10-18T20:30:52.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bd6590d1b89bfbc-006530407c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.002 DEBUG:boto:b'bofrnf-58obj000000000000000000581000falseobj000000000000000000582023-10-18T20:30:28.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.003 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.004 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.004 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:53.005 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:53.006 DEBUG:boto:Method: GET 2023-10-18T20:30:53.006 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000058 2023-10-18T20:30:53.007 DEBUG:boto:Data: 2023-10-18T20:30:53.007 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.008 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.009 DEBUG:boto:Params: {} 2023-10-18T20:30:53.009 DEBUG:boto:Token: None 2023-10-18T20:30:53.010 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.010 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000020eb6dc9fd3f4ff-006530407d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.015 DEBUG:boto:b'bofrnf-58obj000000000000000000581000falseobj000000000000000000582023-10-18T20:30:28.372Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-18T20:30:53.017 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.017 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.018 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.018 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.019 DEBUG:boto:Method: GET 2023-10-18T20:30:53.019 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.020 DEBUG:boto:Data: 2023-10-18T20:30:53.020 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.021 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.021 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.021 DEBUG:boto:Params: {} 2023-10-18T20:30:53.022 DEBUG:boto:Token: None 2023-10-18T20:30:53.022 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.023 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0V8teR7aovYfnxszqZJgAsFEauM= 2023-10-18T20:30:53.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0V8teR7aovYfnxszqZJgAsFEauM=', 'Content-Length': '0'} 2023-10-18T20:30:53.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241afef5c045fbcb-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.027 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.027 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.027 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.028 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.028 DEBUG:boto:Method: GET 2023-10-18T20:30:53.029 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.029 DEBUG:boto:Data: 2023-10-18T20:30:53.029 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.030 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.030 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.031 DEBUG:boto:Params: {} 2023-10-18T20:30:53.031 DEBUG:boto:Token: None 2023-10-18T20:30:53.032 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0V8teR7aovYfnxszqZJgAsFEauM= 2023-10-18T20:30:53.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0V8teR7aovYfnxszqZJgAsFEauM=', 'Content-Length': '0'} 2023-10-18T20:30:53.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003e43441e6b4e512-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.082 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.083 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.083 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.084 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.084 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.085 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.085 DEBUG:boto:Data: 2023-10-18T20:30:53.085 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.086 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.086 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.087 DEBUG:boto:Params: {} 2023-10-18T20:30:53.087 DEBUG:boto:Token: None 2023-10-18T20:30:53.088 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.088 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:89c04kDyvSRK9U8/0KaN/IwovHQ= 2023-10-18T20:30:53.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:89c04kDyvSRK9U8/0KaN/IwovHQ=', 'Content-Length': '0'} 2023-10-18T20:30:53.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062654d9747fe5fd1-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.091 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.091 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.092 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.092 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.093 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.093 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.093 DEBUG:boto:Data: 2023-10-18T20:30:53.094 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.094 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.095 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.095 DEBUG:boto:Params: {} 2023-10-18T20:30:53.096 DEBUG:boto:Token: None 2023-10-18T20:30:53.096 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.097 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:89c04kDyvSRK9U8/0KaN/IwovHQ= 2023-10-18T20:30:53.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:89c04kDyvSRK9U8/0KaN/IwovHQ=', 'Content-Length': '0'} 2023-10-18T20:30:53.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b37d81cd1c30c5-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-18T20:30:53.100 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.100 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.101 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.101 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.102 DEBUG:boto:Method: GET 2023-10-18T20:30:53.102 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.103 DEBUG:boto:Data: 2023-10-18T20:30:53.103 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.103 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.104 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.104 DEBUG:boto:Params: {} 2023-10-18T20:30:53.105 DEBUG:boto:Token: None 2023-10-18T20:30:53.105 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.106 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0V8teR7aovYfnxszqZJgAsFEauM= 2023-10-18T20:30:53.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0V8teR7aovYfnxszqZJgAsFEauM=', 'Content-Length': '0'} 2023-10-18T20:30:53.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000544067d74fc2a7cf-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.109 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.110 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.110 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.111 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.111 DEBUG:boto:Method: GET 2023-10-18T20:30:53.112 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.112 DEBUG:boto:Data: 2023-10-18T20:30:53.113 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.113 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.113 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.114 DEBUG:boto:Params: {} 2023-10-18T20:30:53.114 DEBUG:boto:Token: None 2023-10-18T20:30:53.115 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059?versionId=null 2023-10-18T20:30:53.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0V8teR7aovYfnxszqZJgAsFEauM= 2023-10-18T20:30:53.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0V8teR7aovYfnxszqZJgAsFEauM=', 'Content-Length': '0'} 2023-10-18T20:30:53.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f75bf68b8fe2c66-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.118 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.119 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.119 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.120 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.120 DEBUG:boto:Method: GET 2023-10-18T20:30:53.121 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.121 DEBUG:boto:Data: 2023-10-18T20:30:53.121 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.122 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.123 DEBUG:boto:Params: {} 2023-10-18T20:30:53.123 DEBUG:boto:Token: None 2023-10-18T20:30:53.124 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.124 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6d1c969d2b8916b-006530407d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.128 DEBUG:boto:b'bofrnf-58obj000000000000000000591000falseobj000000000000000000592023-10-18T20:30:28.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.129 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.129 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.129 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.130 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.130 DEBUG:boto:Method: GET 2023-10-18T20:30:53.131 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.131 DEBUG:boto:Data: 2023-10-18T20:30:53.132 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.132 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.132 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.133 DEBUG:boto:Params: {} 2023-10-18T20:30:53.133 DEBUG:boto:Token: None 2023-10-18T20:30:53.134 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.134 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086ff3db7225ed1e9-006530407d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.139 DEBUG:boto:b'bofrnf-58obj000000000000000000591000falseobj000000000000000000592023-10-18T20:30:28.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.140 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.141 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.141 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.141 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.142 DEBUG:boto:Data: 2023-10-18T20:30:53.142 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.143 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.143 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.143 DEBUG:boto:Params: {} 2023-10-18T20:30:53.144 DEBUG:boto:Token: None 2023-10-18T20:30:53.144 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.145 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a5UXTqnQxvPHJxqJr56Yu1BmQv0= 2023-10-18T20:30:53.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a5UXTqnQxvPHJxqJr56Yu1BmQv0=', 'Content-Length': '0'} 2023-10-18T20:30:53.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307cc8b5dec7dba4-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.148 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.148 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.149 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.149 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.150 DEBUG:boto:Data: 2023-10-18T20:30:53.150 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.151 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.151 DEBUG:boto:Params: {} 2023-10-18T20:30:53.152 DEBUG:boto:Token: None 2023-10-18T20:30:53.152 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a5UXTqnQxvPHJxqJr56Yu1BmQv0= 2023-10-18T20:30:53.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a5UXTqnQxvPHJxqJr56Yu1BmQv0=', 'Content-Length': '0'} 2023-10-18T20:30:53.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d12fea547cc660fc-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-18T20:30:53.156 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.157 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.157 DEBUG:boto:Method: GET 2023-10-18T20:30:53.157 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.158 DEBUG:boto:Data: 2023-10-18T20:30:53.158 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.159 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.159 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.160 DEBUG:boto:Params: {} 2023-10-18T20:30:53.160 DEBUG:boto:Token: None 2023-10-18T20:30:53.161 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vXyduFvCztYy+dgBCi7qZcza73Q= 2023-10-18T20:30:53.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vXyduFvCztYy+dgBCi7qZcza73Q=', 'Content-Length': '0'} 2023-10-18T20:30:53.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007582fa7b7e0b5afa-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.206 DEBUG:boto:path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.207 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.207 DEBUG:boto:Method: GET 2023-10-18T20:30:53.208 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.208 DEBUG:boto:Data: 2023-10-18T20:30:53.209 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.209 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.210 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.210 DEBUG:boto:Params: {} 2023-10-18T20:30:53.211 DEBUG:boto:Token: None 2023-10-18T20:30:53.211 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000059 2023-10-18T20:30:53.212 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vXyduFvCztYy+dgBCi7qZcza73Q= 2023-10-18T20:30:53.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vXyduFvCztYy+dgBCi7qZcza73Q=', 'Content-Length': '0'} 2023-10-18T20:30:53.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dac977fe56bb1ae-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.215 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.216 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.216 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.217 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.217 DEBUG:boto:Method: GET 2023-10-18T20:30:53.218 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.218 DEBUG:boto:Data: 2023-10-18T20:30:53.219 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.220 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.220 DEBUG:boto:Params: {} 2023-10-18T20:30:53.221 DEBUG:boto:Token: None 2023-10-18T20:30:53.221 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.222 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ec478dfcea5e18c-006530407d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.226 DEBUG:boto:b'bofrnf-58obj000000000000000000591000falseobj000000000000000000592023-10-18T20:30:28.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.226 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.227 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.227 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.228 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.228 DEBUG:boto:Method: GET 2023-10-18T20:30:53.229 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000059 2023-10-18T20:30:53.229 DEBUG:boto:Data: 2023-10-18T20:30:53.230 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.230 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.231 DEBUG:boto:Params: {} 2023-10-18T20:30:53.231 DEBUG:boto:Token: None 2023-10-18T20:30:53.232 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031118e312ad6e2ca-006530407d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.237 DEBUG:boto:b'bofrnf-58obj000000000000000000591000falseobj000000000000000000592023-10-18T20:30:28.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-18T20:30:53.239 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.239 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.239 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.240 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.240 DEBUG:boto:Method: GET 2023-10-18T20:30:53.241 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.241 DEBUG:boto:Data: 2023-10-18T20:30:53.242 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.243 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.243 DEBUG:boto:Params: {} 2023-10-18T20:30:53.244 DEBUG:boto:Token: None 2023-10-18T20:30:53.244 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:64L88b0K4R4HM+gndCYFKsDAlkM= 2023-10-18T20:30:53.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:64L88b0K4R4HM+gndCYFKsDAlkM=', 'Content-Length': '0'} 2023-10-18T20:30:53.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d22084b7e153ed36-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.248 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.248 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.249 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.250 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.250 DEBUG:boto:Method: GET 2023-10-18T20:30:53.250 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.251 DEBUG:boto:Data: 2023-10-18T20:30:53.251 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.252 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.252 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.253 DEBUG:boto:Params: {} 2023-10-18T20:30:53.253 DEBUG:boto:Token: None 2023-10-18T20:30:53.254 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.254 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:64L88b0K4R4HM+gndCYFKsDAlkM= 2023-10-18T20:30:53.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:64L88b0K4R4HM+gndCYFKsDAlkM=', 'Content-Length': '0'} 2023-10-18T20:30:53.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dec5a5ce08fe0e30-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.258 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.258 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.259 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.259 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.260 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.260 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.260 DEBUG:boto:Data: 2023-10-18T20:30:53.261 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.261 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.262 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.262 DEBUG:boto:Params: {} 2023-10-18T20:30:53.262 DEBUG:boto:Token: None 2023-10-18T20:30:53.263 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iS5gpiUxL6o3V2opzknMCqjHmXw= 2023-10-18T20:30:53.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iS5gpiUxL6o3V2opzknMCqjHmXw=', 'Content-Length': '0'} 2023-10-18T20:30:53.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009447c9b398f98c06-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.266 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.267 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.267 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.267 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.268 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.268 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.269 DEBUG:boto:Data: 2023-10-18T20:30:53.269 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.270 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.270 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.270 DEBUG:boto:Params: {} 2023-10-18T20:30:53.271 DEBUG:boto:Token: None 2023-10-18T20:30:53.271 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.272 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iS5gpiUxL6o3V2opzknMCqjHmXw= 2023-10-18T20:30:53.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iS5gpiUxL6o3V2opzknMCqjHmXw=', 'Content-Length': '0'} 2023-10-18T20:30:53.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a59ec0f61df6e0-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-18T20:30:53.275 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.277 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.278 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.278 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.278 DEBUG:boto:Method: GET 2023-10-18T20:30:53.279 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.279 DEBUG:boto:Data: 2023-10-18T20:30:53.280 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.280 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.281 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.281 DEBUG:boto:Params: {} 2023-10-18T20:30:53.281 DEBUG:boto:Token: None 2023-10-18T20:30:53.282 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.282 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:64L88b0K4R4HM+gndCYFKsDAlkM= 2023-10-18T20:30:53.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:64L88b0K4R4HM+gndCYFKsDAlkM=', 'Content-Length': '0'} 2023-10-18T20:30:53.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094af4d15af20ee6a-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.286 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.286 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.287 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.287 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.287 DEBUG:boto:Method: GET 2023-10-18T20:30:53.288 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.288 DEBUG:boto:Data: 2023-10-18T20:30:53.289 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.289 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.289 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.290 DEBUG:boto:Params: {} 2023-10-18T20:30:53.290 DEBUG:boto:Token: None 2023-10-18T20:30:53.291 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060?versionId=null 2023-10-18T20:30:53.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:64L88b0K4R4HM+gndCYFKsDAlkM= 2023-10-18T20:30:53.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:64L88b0K4R4HM+gndCYFKsDAlkM=', 'Content-Length': '0'} 2023-10-18T20:30:53.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ece59677f662faa-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.334 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.335 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.336 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.336 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.336 DEBUG:boto:Method: GET 2023-10-18T20:30:53.337 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.337 DEBUG:boto:Data: 2023-10-18T20:30:53.338 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.339 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.339 DEBUG:boto:Params: {} 2023-10-18T20:30:53.339 DEBUG:boto:Token: None 2023-10-18T20:30:53.340 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9889223644523d9-006530407d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.344 DEBUG:boto:b'bofrnf-58obj000000000000000000601000falseobj000000000000000000602023-10-18T20:30:32.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.345 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.345 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.346 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.346 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.346 DEBUG:boto:Method: GET 2023-10-18T20:30:53.347 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.347 DEBUG:boto:Data: 2023-10-18T20:30:53.348 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.348 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.349 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.349 DEBUG:boto:Params: {} 2023-10-18T20:30:53.350 DEBUG:boto:Token: None 2023-10-18T20:30:53.350 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e76f56f85fb3cb36-006530407d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.356 DEBUG:boto:b'bofrnf-58obj000000000000000000601000falseobj000000000000000000602023-10-18T20:30:32.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.356 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.357 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.357 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.358 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.358 DEBUG:boto:Data: 2023-10-18T20:30:53.359 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.359 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.359 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.360 DEBUG:boto:Params: {} 2023-10-18T20:30:53.360 DEBUG:boto:Token: None 2023-10-18T20:30:53.361 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.361 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2+3nJKRWrcuIRZtvwdBV+FyyNdI= 2023-10-18T20:30:53.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2+3nJKRWrcuIRZtvwdBV+FyyNdI=', 'Content-Length': '0'} 2023-10-18T20:30:53.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ba1296c9501164-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.364 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.365 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.365 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.365 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.366 DEBUG:boto:Data: 2023-10-18T20:30:53.366 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.367 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.368 DEBUG:boto:Params: {} 2023-10-18T20:30:53.368 DEBUG:boto:Token: None 2023-10-18T20:30:53.368 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2+3nJKRWrcuIRZtvwdBV+FyyNdI= 2023-10-18T20:30:53.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2+3nJKRWrcuIRZtvwdBV+FyyNdI=', 'Content-Length': '0'} 2023-10-18T20:30:53.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000255bf46d5aa9181f-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-18T20:30:53.373 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.373 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.373 DEBUG:boto:Method: GET 2023-10-18T20:30:53.374 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.374 DEBUG:boto:Data: 2023-10-18T20:30:53.375 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.375 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.376 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.376 DEBUG:boto:Params: {} 2023-10-18T20:30:53.377 DEBUG:boto:Token: None 2023-10-18T20:30:53.377 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.378 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tOxKUsuIlkJq7ljytpPWwUnR/L4= 2023-10-18T20:30:53.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tOxKUsuIlkJq7ljytpPWwUnR/L4=', 'Content-Length': '0'} 2023-10-18T20:30:53.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e7d017967f88ed-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.426 DEBUG:boto:path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.427 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.427 DEBUG:boto:Method: GET 2023-10-18T20:30:53.428 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.428 DEBUG:boto:Data: 2023-10-18T20:30:53.429 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.429 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.430 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.430 DEBUG:boto:Params: {} 2023-10-18T20:30:53.431 DEBUG:boto:Token: None 2023-10-18T20:30:53.431 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000060 2023-10-18T20:30:53.432 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tOxKUsuIlkJq7ljytpPWwUnR/L4= 2023-10-18T20:30:53.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tOxKUsuIlkJq7ljytpPWwUnR/L4=', 'Content-Length': '0'} 2023-10-18T20:30:53.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f3b879f1d9d4b4c-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.435 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.436 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.436 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.437 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.437 DEBUG:boto:Method: GET 2023-10-18T20:30:53.437 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.438 DEBUG:boto:Data: 2023-10-18T20:30:53.438 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.439 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.439 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.440 DEBUG:boto:Params: {} 2023-10-18T20:30:53.440 DEBUG:boto:Token: None 2023-10-18T20:30:53.441 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.441 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b4c9919567e1496-006530407d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.445 DEBUG:boto:b'bofrnf-58obj000000000000000000601000falseobj000000000000000000602023-10-18T20:30:32.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.446 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.446 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.446 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.447 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.447 DEBUG:boto:Method: GET 2023-10-18T20:30:53.448 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000060 2023-10-18T20:30:53.448 DEBUG:boto:Data: 2023-10-18T20:30:53.449 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.449 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.449 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.450 DEBUG:boto:Params: {} 2023-10-18T20:30:53.450 DEBUG:boto:Token: None 2023-10-18T20:30:53.451 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.451 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b66a0358fa65fc7-006530407d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.502 DEBUG:boto:b'bofrnf-58obj000000000000000000601000falseobj000000000000000000602023-10-18T20:30:32.403Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-18T20:30:53.504 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.504 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.504 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.505 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.505 DEBUG:boto:Method: GET 2023-10-18T20:30:53.506 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.506 DEBUG:boto:Data: 2023-10-18T20:30:53.507 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.507 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.508 DEBUG:boto:Params: {} 2023-10-18T20:30:53.508 DEBUG:boto:Token: None 2023-10-18T20:30:53.509 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:90dy/kyzUsQSJ8rdJv1y4Io0cCw= 2023-10-18T20:30:53.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:90dy/kyzUsQSJ8rdJv1y4Io0cCw=', 'Content-Length': '0'} 2023-10-18T20:30:53.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f5095b409e767eb-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.513 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.513 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.514 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.514 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.515 DEBUG:boto:Method: GET 2023-10-18T20:30:53.515 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.516 DEBUG:boto:Data: 2023-10-18T20:30:53.516 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.517 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.517 DEBUG:boto:Params: {} 2023-10-18T20:30:53.518 DEBUG:boto:Token: None 2023-10-18T20:30:53.518 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.519 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:90dy/kyzUsQSJ8rdJv1y4Io0cCw= 2023-10-18T20:30:53.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:90dy/kyzUsQSJ8rdJv1y4Io0cCw=', 'Content-Length': '0'} 2023-10-18T20:30:53.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000426bfbfcf52a5736-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.522 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.523 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.523 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.523 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.524 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.524 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.525 DEBUG:boto:Data: 2023-10-18T20:30:53.525 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.526 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.526 DEBUG:boto:Params: {} 2023-10-18T20:30:53.527 DEBUG:boto:Token: None 2023-10-18T20:30:53.527 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.528 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f6pkWQRl0dEhU6e/aKjZzSGqWuI= 2023-10-18T20:30:53.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f6pkWQRl0dEhU6e/aKjZzSGqWuI=', 'Content-Length': '0'} 2023-10-18T20:30:53.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000578132f9e8700a9e-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.531 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.531 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.532 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.532 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.533 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.533 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.533 DEBUG:boto:Data: 2023-10-18T20:30:53.534 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.535 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.535 DEBUG:boto:Params: {} 2023-10-18T20:30:53.535 DEBUG:boto:Token: None 2023-10-18T20:30:53.536 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f6pkWQRl0dEhU6e/aKjZzSGqWuI= 2023-10-18T20:30:53.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f6pkWQRl0dEhU6e/aKjZzSGqWuI=', 'Content-Length': '0'} 2023-10-18T20:30:53.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffacfe2c6e1616c1-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-18T20:30:53.540 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.540 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.541 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.541 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.541 DEBUG:boto:Method: GET 2023-10-18T20:30:53.542 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.542 DEBUG:boto:Data: 2023-10-18T20:30:53.543 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.543 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.544 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.544 DEBUG:boto:Params: {} 2023-10-18T20:30:53.545 DEBUG:boto:Token: None 2023-10-18T20:30:53.545 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.546 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:90dy/kyzUsQSJ8rdJv1y4Io0cCw= 2023-10-18T20:30:53.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:90dy/kyzUsQSJ8rdJv1y4Io0cCw=', 'Content-Length': '0'} 2023-10-18T20:30:53.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f7aa8601dd46a5-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.549 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.549 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.550 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.550 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.551 DEBUG:boto:Method: GET 2023-10-18T20:30:53.551 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.552 DEBUG:boto:Data: 2023-10-18T20:30:53.552 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.553 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.553 DEBUG:boto:Params: {} 2023-10-18T20:30:53.554 DEBUG:boto:Token: None 2023-10-18T20:30:53.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061?versionId=null 2023-10-18T20:30:53.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:90dy/kyzUsQSJ8rdJv1y4Io0cCw= 2023-10-18T20:30:53.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:90dy/kyzUsQSJ8rdJv1y4Io0cCw=', 'Content-Length': '0'} 2023-10-18T20:30:53.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee7360f6f7fc331f-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.558 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.559 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.559 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.559 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.560 DEBUG:boto:Method: GET 2023-10-18T20:30:53.560 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.561 DEBUG:boto:Data: 2023-10-18T20:30:53.561 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.562 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.562 DEBUG:boto:Params: {} 2023-10-18T20:30:53.563 DEBUG:boto:Token: None 2023-10-18T20:30:53.563 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ab0f154a2a55322-006530407d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.568 DEBUG:boto:b'bofrnf-58obj000000000000000000611000falseobj000000000000000000612023-10-18T20:30:30.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.569 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.569 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.569 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.570 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.570 DEBUG:boto:Method: GET 2023-10-18T20:30:53.571 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.571 DEBUG:boto:Data: 2023-10-18T20:30:53.572 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.572 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.572 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.573 DEBUG:boto:Params: {} 2023-10-18T20:30:53.573 DEBUG:boto:Token: None 2023-10-18T20:30:53.574 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.574 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f331318878a666a-006530407d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.579 DEBUG:boto:b'bofrnf-58obj000000000000000000611000falseobj000000000000000000612023-10-18T20:30:30.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.580 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.581 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.581 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.581 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.582 DEBUG:boto:Data: 2023-10-18T20:30:53.582 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.583 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.583 DEBUG:boto:Params: {} 2023-10-18T20:30:53.584 DEBUG:boto:Token: None 2023-10-18T20:30:53.584 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XpsEXIBjrwOAoVC0rI6Zy/wzIvg= 2023-10-18T20:30:53.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XpsEXIBjrwOAoVC0rI6Zy/wzIvg=', 'Content-Length': '0'} 2023-10-18T20:30:53.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d793049b0bde136a-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.588 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.589 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.589 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.589 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.590 DEBUG:boto:Data: 2023-10-18T20:30:53.590 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.591 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.591 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.591 DEBUG:boto:Params: {} 2023-10-18T20:30:53.592 DEBUG:boto:Token: None 2023-10-18T20:30:53.592 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.593 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XpsEXIBjrwOAoVC0rI6Zy/wzIvg= 2023-10-18T20:30:53.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XpsEXIBjrwOAoVC0rI6Zy/wzIvg=', 'Content-Length': '0'} 2023-10-18T20:30:53.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007aa857b799911b6-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-18T20:30:53.596 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.597 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.597 DEBUG:boto:Method: GET 2023-10-18T20:30:53.598 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.598 DEBUG:boto:Data: 2023-10-18T20:30:53.598 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.599 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.599 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.600 DEBUG:boto:Params: {} 2023-10-18T20:30:53.600 DEBUG:boto:Token: None 2023-10-18T20:30:53.601 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.601 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PNl6QLujE5y4mVAfqAVlEFwolh0= 2023-10-18T20:30:53.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PNl6QLujE5y4mVAfqAVlEFwolh0=', 'Content-Length': '0'} 2023-10-18T20:30:53.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e41e839d8e425a2-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.650 DEBUG:boto:path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.651 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.652 DEBUG:boto:Method: GET 2023-10-18T20:30:53.652 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.653 DEBUG:boto:Data: 2023-10-18T20:30:53.653 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.653 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.654 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.654 DEBUG:boto:Params: {} 2023-10-18T20:30:53.655 DEBUG:boto:Token: None 2023-10-18T20:30:53.655 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000061 2023-10-18T20:30:53.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PNl6QLujE5y4mVAfqAVlEFwolh0= 2023-10-18T20:30:53.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PNl6QLujE5y4mVAfqAVlEFwolh0=', 'Content-Length': '0'} 2023-10-18T20:30:53.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae638f0f5b0157e-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.659 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.660 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.660 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.661 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.661 DEBUG:boto:Method: GET 2023-10-18T20:30:53.662 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.662 DEBUG:boto:Data: 2023-10-18T20:30:53.662 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.663 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.663 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.664 DEBUG:boto:Params: {} 2023-10-18T20:30:53.664 DEBUG:boto:Token: None 2023-10-18T20:30:53.665 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abf80d79a49bcf34-006530407d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.669 DEBUG:boto:b'bofrnf-58obj000000000000000000611000falseobj000000000000000000612023-10-18T20:30:30.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.670 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.670 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.671 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.671 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.671 DEBUG:boto:Method: GET 2023-10-18T20:30:53.672 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000061 2023-10-18T20:30:53.672 DEBUG:boto:Data: 2023-10-18T20:30:53.673 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.673 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.674 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.674 DEBUG:boto:Params: {} 2023-10-18T20:30:53.674 DEBUG:boto:Token: None 2023-10-18T20:30:53.675 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.675 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6d0c821eec6aca9-006530407d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.680 DEBUG:boto:b'bofrnf-58obj000000000000000000611000falseobj000000000000000000612023-10-18T20:30:30.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-18T20:30:53.682 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.682 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.683 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.683 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.683 DEBUG:boto:Method: GET 2023-10-18T20:30:53.684 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.684 DEBUG:boto:Data: 2023-10-18T20:30:53.685 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.685 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.686 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.686 DEBUG:boto:Params: {} 2023-10-18T20:30:53.687 DEBUG:boto:Token: None 2023-10-18T20:30:53.687 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZohklCefMbWn+qf4aX32Qtifd/g= 2023-10-18T20:30:53.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZohklCefMbWn+qf4aX32Qtifd/g=', 'Content-Length': '0'} 2023-10-18T20:30:53.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013905f05885b9dce-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.693 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.693 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.694 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.694 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.695 DEBUG:boto:Method: GET 2023-10-18T20:30:53.695 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.696 DEBUG:boto:Data: 2023-10-18T20:30:53.696 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.696 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.697 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.698 DEBUG:boto:Params: {} 2023-10-18T20:30:53.698 DEBUG:boto:Token: None 2023-10-18T20:30:53.699 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.700 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZohklCefMbWn+qf4aX32Qtifd/g= 2023-10-18T20:30:53.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZohklCefMbWn+qf4aX32Qtifd/g=', 'Content-Length': '0'} 2023-10-18T20:30:53.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004830af0fc7a8d99f-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.705 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.706 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.706 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.707 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.708 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.708 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.708 DEBUG:boto:Data: 2023-10-18T20:30:53.709 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.709 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.710 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.710 DEBUG:boto:Params: {} 2023-10-18T20:30:53.710 DEBUG:boto:Token: None 2023-10-18T20:30:53.711 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.711 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwZ3lWgQniWGx0QUcRz0KMkAISg= 2023-10-18T20:30:53.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwZ3lWgQniWGx0QUcRz0KMkAISg=', 'Content-Length': '0'} 2023-10-18T20:30:53.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f87ce75bbe9891b-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.715 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.715 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.715 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.716 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.716 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.717 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.717 DEBUG:boto:Data: 2023-10-18T20:30:53.718 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.718 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.719 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.719 DEBUG:boto:Params: {} 2023-10-18T20:30:53.719 DEBUG:boto:Token: None 2023-10-18T20:30:53.720 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.720 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwZ3lWgQniWGx0QUcRz0KMkAISg= 2023-10-18T20:30:53.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwZ3lWgQniWGx0QUcRz0KMkAISg=', 'Content-Length': '0'} 2023-10-18T20:30:53.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b714892470bca4-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-18T20:30:53.724 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.724 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.725 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.725 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.726 DEBUG:boto:Method: GET 2023-10-18T20:30:53.726 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.729 DEBUG:boto:Data: 2023-10-18T20:30:53.729 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.730 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.730 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.731 DEBUG:boto:Params: {} 2023-10-18T20:30:53.731 DEBUG:boto:Token: None 2023-10-18T20:30:53.732 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.732 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZohklCefMbWn+qf4aX32Qtifd/g= 2023-10-18T20:30:53.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZohklCefMbWn+qf4aX32Qtifd/g=', 'Content-Length': '0'} 2023-10-18T20:30:53.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4f41707965b3ba6-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.736 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.736 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.737 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.737 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.737 DEBUG:boto:Method: GET 2023-10-18T20:30:53.738 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.738 DEBUG:boto:Data: 2023-10-18T20:30:53.739 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.739 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.740 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.740 DEBUG:boto:Params: {} 2023-10-18T20:30:53.741 DEBUG:boto:Token: None 2023-10-18T20:30:53.741 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062?versionId=null 2023-10-18T20:30:53.742 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZohklCefMbWn+qf4aX32Qtifd/g= 2023-10-18T20:30:53.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZohklCefMbWn+qf4aX32Qtifd/g=', 'Content-Length': '0'} 2023-10-18T20:30:53.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002365e4175f2c5b40-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.786 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.787 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.788 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.788 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.789 DEBUG:boto:Method: GET 2023-10-18T20:30:53.789 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.789 DEBUG:boto:Data: 2023-10-18T20:30:53.790 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.791 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.791 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.791 DEBUG:boto:Params: {} 2023-10-18T20:30:53.792 DEBUG:boto:Token: None 2023-10-18T20:30:53.792 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.793 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f23b53e7e119c3aa-006530407d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.797 DEBUG:boto:b'bofrnf-58obj000000000000000000621000falseobj000000000000000000622023-10-18T20:30:31.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.798 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.798 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.799 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.800 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.800 DEBUG:boto:Method: GET 2023-10-18T20:30:53.800 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.801 DEBUG:boto:Data: 2023-10-18T20:30:53.801 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.802 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.802 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.803 DEBUG:boto:Params: {} 2023-10-18T20:30:53.803 DEBUG:boto:Token: None 2023-10-18T20:30:53.804 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.804 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002de1119d6f136262-006530407d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.811 DEBUG:boto:b'bofrnf-58obj000000000000000000621000falseobj000000000000000000622023-10-18T20:30:31.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.812 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.812 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.813 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.813 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.814 DEBUG:boto:Data: 2023-10-18T20:30:53.814 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.815 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.815 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.815 DEBUG:boto:Params: {} 2023-10-18T20:30:53.816 DEBUG:boto:Token: None 2023-10-18T20:30:53.816 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.817 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:20HeKGuj5Ld1LV4c9q+pj1dJskc= 2023-10-18T20:30:53.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:20HeKGuj5Ld1LV4c9q+pj1dJskc=', 'Content-Length': '0'} 2023-10-18T20:30:53.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfa724b30e1b9a3a-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.820 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.821 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.821 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.821 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.822 DEBUG:boto:Data: 2023-10-18T20:30:53.822 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.823 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.823 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.823 DEBUG:boto:Params: {} 2023-10-18T20:30:53.824 DEBUG:boto:Token: None 2023-10-18T20:30:53.824 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.825 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:20HeKGuj5Ld1LV4c9q+pj1dJskc= 2023-10-18T20:30:53.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:20HeKGuj5Ld1LV4c9q+pj1dJskc=', 'Content-Length': '0'} 2023-10-18T20:30:53.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad28f28981f5370d-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-18T20:30:53.828 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.829 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.829 DEBUG:boto:Method: GET 2023-10-18T20:30:53.830 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.830 DEBUG:boto:Data: 2023-10-18T20:30:53.831 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.831 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.831 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.832 DEBUG:boto:Params: {} 2023-10-18T20:30:53.832 DEBUG:boto:Token: None 2023-10-18T20:30:53.833 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.833 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J+5N6ni47e09L4ak8ebmRVkjDAo= 2023-10-18T20:30:53.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J+5N6ni47e09L4ak8ebmRVkjDAo=', 'Content-Length': '0'} 2023-10-18T20:30:53.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef96dd26f4caaa2a-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.837 DEBUG:boto:path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.837 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.838 DEBUG:boto:Method: GET 2023-10-18T20:30:53.839 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.840 DEBUG:boto:Data: 2023-10-18T20:30:53.841 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.841 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.842 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.843 DEBUG:boto:Params: {} 2023-10-18T20:30:53.843 DEBUG:boto:Token: None 2023-10-18T20:30:53.844 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000062 2023-10-18T20:30:53.844 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:J+5N6ni47e09L4ak8ebmRVkjDAo= 2023-10-18T20:30:53.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J+5N6ni47e09L4ak8ebmRVkjDAo=', 'Content-Length': '0'} 2023-10-18T20:30:53.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc47b5cf1ec79164-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.894 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.895 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.896 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.896 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.897 DEBUG:boto:Method: GET 2023-10-18T20:30:53.897 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.898 DEBUG:boto:Data: 2023-10-18T20:30:53.898 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.899 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.899 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.899 DEBUG:boto:Params: {} 2023-10-18T20:30:53.900 DEBUG:boto:Token: None 2023-10-18T20:30:53.900 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.901 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd66e75f085b2b5-006530407d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.905 DEBUG:boto:b'bofrnf-58obj000000000000000000621000falseobj000000000000000000622023-10-18T20:30:31.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.906 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:53.906 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:53.907 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.907 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.908 DEBUG:boto:Method: GET 2023-10-18T20:30:53.908 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000062 2023-10-18T20:30:53.909 DEBUG:boto:Data: 2023-10-18T20:30:53.909 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.910 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.911 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.912 DEBUG:boto:Params: {} 2023-10-18T20:30:53.913 DEBUG:boto:Token: None 2023-10-18T20:30:53.913 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/ 2023-10-18T20:30:53.914 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso= 2023-10-18T20:30:53.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vH0m3ME0BwQgGqWz3qUdnMAGjso=', 'Content-Length': '0'} 2023-10-18T20:30:53.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da4e541d2fb2d971-006530407d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.921 DEBUG:boto:b'bofrnf-58obj000000000000000000621000falseobj000000000000000000622023-10-18T20:30:31.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:53.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-18T20:30:53.923 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.923 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.924 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.925 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.926 DEBUG:boto:Method: GET 2023-10-18T20:30:53.926 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.927 DEBUG:boto:Data: 2023-10-18T20:30:53.928 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.930 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.930 DEBUG:boto:Params: {} 2023-10-18T20:30:53.931 DEBUG:boto:Token: None 2023-10-18T20:30:53.933 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.934 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m9HZ9EbD/x+w7pdEbmJt3XABvdY= 2023-10-18T20:30:53.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m9HZ9EbD/x+w7pdEbmJt3XABvdY=', 'Content-Length': '0'} 2023-10-18T20:30:53.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003411667bf479cc8c-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.938 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.938 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.939 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.939 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.940 DEBUG:boto:Method: GET 2023-10-18T20:30:53.940 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.941 DEBUG:boto:Data: 2023-10-18T20:30:53.941 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.942 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.942 DEBUG:boto:Params: {} 2023-10-18T20:30:53.943 DEBUG:boto:Token: None 2023-10-18T20:30:53.943 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.944 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m9HZ9EbD/x+w7pdEbmJt3XABvdY= 2023-10-18T20:30:53.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m9HZ9EbD/x+w7pdEbmJt3XABvdY=', 'Content-Length': '0'} 2023-10-18T20:30:53.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000569351c5e96f54a2-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.947 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.948 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.948 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.948 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.949 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.949 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.950 DEBUG:boto:Data: 2023-10-18T20:30:53.950 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.950 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.951 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.951 DEBUG:boto:Params: {} 2023-10-18T20:30:53.952 DEBUG:boto:Token: None 2023-10-18T20:30:53.952 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.953 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ApAANt+DuqNL/f4TysyLNE25zS4= 2023-10-18T20:30:53.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ApAANt+DuqNL/f4TysyLNE25zS4=', 'Content-Length': '0'} 2023-10-18T20:30:53.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f3fc5b78a9e6c05-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.956 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.956 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.956 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.957 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.957 DEBUG:boto:Method: HEAD 2023-10-18T20:30:53.958 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.958 DEBUG:boto:Data: 2023-10-18T20:30:53.959 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.959 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.959 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.960 DEBUG:boto:Params: {} 2023-10-18T20:30:53.960 DEBUG:boto:Token: None 2023-10-18T20:30:53.961 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.961 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ApAANt+DuqNL/f4TysyLNE25zS4= 2023-10-18T20:30:53.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ApAANt+DuqNL/f4TysyLNE25zS4=', 'Content-Length': '0'} 2023-10-18T20:30:53.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825241dbc37cad2a-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-18T20:30:53.965 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.965 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.966 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.966 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.966 DEBUG:boto:Method: GET 2023-10-18T20:30:53.967 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.967 DEBUG:boto:Data: 2023-10-18T20:30:53.968 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.968 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:53.969 DEBUG:boto:Port: 8001 2023-10-18T20:30:53.969 DEBUG:boto:Params: {} 2023-10-18T20:30:53.970 DEBUG:boto:Token: None 2023-10-18T20:30:53.970 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.971 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m9HZ9EbD/x+w7pdEbmJt3XABvdY= 2023-10-18T20:30:53.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m9HZ9EbD/x+w7pdEbmJt3XABvdY=', 'Content-Length': '0'} 2023-10-18T20:30:53.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d9d46126a520876-006530407d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:53.974 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.974 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:53.975 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.975 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.976 DEBUG:boto:Method: GET 2023-10-18T20:30:53.976 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.976 DEBUG:boto:Data: 2023-10-18T20:30:53.977 DEBUG:boto:Headers: {} 2023-10-18T20:30:53.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:53.978 DEBUG:boto:Port: 8000 2023-10-18T20:30:53.978 DEBUG:boto:Params: {} 2023-10-18T20:30:53.979 DEBUG:boto:Token: None 2023-10-18T20:30:53.979 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:53 GMT /bofrnf-58/obj00000000000000000065?versionId=null 2023-10-18T20:30:53.980 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m9HZ9EbD/x+w7pdEbmJt3XABvdY= 2023-10-18T20:30:53.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m9HZ9EbD/x+w7pdEbmJt3XABvdY=', 'Content-Length': '0'} 2023-10-18T20:30:53.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026e82d1c0e34933-006530407d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.022 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.023 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.024 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.025 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.025 DEBUG:boto:Method: GET 2023-10-18T20:30:54.026 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.026 DEBUG:boto:Data: 2023-10-18T20:30:54.027 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.027 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.027 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.028 DEBUG:boto:Params: {} 2023-10-18T20:30:54.029 DEBUG:boto:Token: None 2023-10-18T20:30:54.029 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a197c8b4e197149d-006530407e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.034 DEBUG:boto:b'bofrnf-58obj000000000000000000651000falseobj000000000000000000652023-10-18T20:30:32.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.034 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.035 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.036 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.036 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.037 DEBUG:boto:Method: GET 2023-10-18T20:30:54.038 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.039 DEBUG:boto:Data: 2023-10-18T20:30:54.040 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.040 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.041 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.042 DEBUG:boto:Params: {} 2023-10-18T20:30:54.043 DEBUG:boto:Token: None 2023-10-18T20:30:54.043 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a044293334cbb352-006530407e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.051 DEBUG:boto:b'bofrnf-58obj000000000000000000651000falseobj000000000000000000652023-10-18T20:30:32.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.052 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.053 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.053 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.054 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.055 DEBUG:boto:Data: 2023-10-18T20:30:54.056 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.057 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.057 DEBUG:boto:Params: {} 2023-10-18T20:30:54.057 DEBUG:boto:Token: None 2023-10-18T20:30:54.058 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.058 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hmiS2tp43TQE3sCxthw+t33TL9I= 2023-10-18T20:30:54.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hmiS2tp43TQE3sCxthw+t33TL9I=', 'Content-Length': '0'} 2023-10-18T20:30:54.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000802a68283f90e291-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.062 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.062 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.063 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.063 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.063 DEBUG:boto:Data: 2023-10-18T20:30:54.064 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.064 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.065 DEBUG:boto:Params: {} 2023-10-18T20:30:54.065 DEBUG:boto:Token: None 2023-10-18T20:30:54.066 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.066 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hmiS2tp43TQE3sCxthw+t33TL9I= 2023-10-18T20:30:54.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hmiS2tp43TQE3sCxthw+t33TL9I=', 'Content-Length': '0'} 2023-10-18T20:30:54.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8715ea234360163-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-18T20:30:54.070 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.071 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.071 DEBUG:boto:Method: GET 2023-10-18T20:30:54.071 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.072 DEBUG:boto:Data: 2023-10-18T20:30:54.072 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.073 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.073 DEBUG:boto:Params: {} 2023-10-18T20:30:54.074 DEBUG:boto:Token: None 2023-10-18T20:30:54.074 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.075 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fS8Chk6+U69v7knHO/L7SdlldM0= 2023-10-18T20:30:54.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fS8Chk6+U69v7knHO/L7SdlldM0=', 'Content-Length': '0'} 2023-10-18T20:30:54.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9cd7f1f3d775dea-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.078 DEBUG:boto:path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.078 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.079 DEBUG:boto:Method: GET 2023-10-18T20:30:54.176 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.177 DEBUG:boto:Data: 2023-10-18T20:30:54.177 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.177 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.178 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.178 DEBUG:boto:Params: {} 2023-10-18T20:30:54.179 DEBUG:boto:Token: None 2023-10-18T20:30:54.179 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000065 2023-10-18T20:30:54.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fS8Chk6+U69v7knHO/L7SdlldM0= 2023-10-18T20:30:54.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fS8Chk6+U69v7knHO/L7SdlldM0=', 'Content-Length': '0'} 2023-10-18T20:30:54.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000665fd7fe233d1aac-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.183 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.184 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.184 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.185 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.185 DEBUG:boto:Method: GET 2023-10-18T20:30:54.186 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.186 DEBUG:boto:Data: 2023-10-18T20:30:54.187 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.187 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.188 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.188 DEBUG:boto:Params: {} 2023-10-18T20:30:54.189 DEBUG:boto:Token: None 2023-10-18T20:30:54.189 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.190 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b37d48337e2bfeaf-006530407e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.193 DEBUG:boto:b'bofrnf-58obj000000000000000000651000falseobj000000000000000000652023-10-18T20:30:32.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.194 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.195 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.195 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.195 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.196 DEBUG:boto:Method: GET 2023-10-18T20:30:54.196 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000065 2023-10-18T20:30:54.197 DEBUG:boto:Data: 2023-10-18T20:30:54.197 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.198 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.198 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.199 DEBUG:boto:Params: {} 2023-10-18T20:30:54.199 DEBUG:boto:Token: None 2023-10-18T20:30:54.200 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.200 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b4b9cc15513148d-006530407e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.205 DEBUG:boto:b'bofrnf-58obj000000000000000000651000falseobj000000000000000000652023-10-18T20:30:32.637Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-10-18T20:30:54.207 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.207 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.208 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.208 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.209 DEBUG:boto:Method: GET 2023-10-18T20:30:54.209 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.210 DEBUG:boto:Data: 2023-10-18T20:30:54.210 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.211 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.211 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.211 DEBUG:boto:Params: {} 2023-10-18T20:30:54.212 DEBUG:boto:Token: None 2023-10-18T20:30:54.213 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FIu3E7ozXeU8gXK2DEONSqsVCAw= 2023-10-18T20:30:54.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FIu3E7ozXeU8gXK2DEONSqsVCAw=', 'Content-Length': '0'} 2023-10-18T20:30:54.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b51599a6660509b-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.217 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.217 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.218 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.218 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.219 DEBUG:boto:Method: GET 2023-10-18T20:30:54.219 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.220 DEBUG:boto:Data: 2023-10-18T20:30:54.220 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.221 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.221 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.222 DEBUG:boto:Params: {} 2023-10-18T20:30:54.223 DEBUG:boto:Token: None 2023-10-18T20:30:54.223 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.224 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FIu3E7ozXeU8gXK2DEONSqsVCAw= 2023-10-18T20:30:54.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FIu3E7ozXeU8gXK2DEONSqsVCAw=', 'Content-Length': '0'} 2023-10-18T20:30:54.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483ae42303c52bcd-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.274 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.275 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.276 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.277 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.277 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.278 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.279 DEBUG:boto:Data: 2023-10-18T20:30:54.279 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.280 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.281 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.281 DEBUG:boto:Params: {} 2023-10-18T20:30:54.282 DEBUG:boto:Token: None 2023-10-18T20:30:54.282 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.283 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G0U4olWHTGzjVVyKV50Kyc9ydP0= 2023-10-18T20:30:54.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G0U4olWHTGzjVVyKV50Kyc9ydP0=', 'Content-Length': '0'} 2023-10-18T20:30:54.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047f482996b839d46-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.286 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.287 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.287 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.287 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.288 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.288 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.289 DEBUG:boto:Data: 2023-10-18T20:30:54.289 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.289 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.290 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.290 DEBUG:boto:Params: {} 2023-10-18T20:30:54.291 DEBUG:boto:Token: None 2023-10-18T20:30:54.291 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.292 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G0U4olWHTGzjVVyKV50Kyc9ydP0= 2023-10-18T20:30:54.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G0U4olWHTGzjVVyKV50Kyc9ydP0=', 'Content-Length': '0'} 2023-10-18T20:30:54.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000071f5c681b5a3704-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-10-18T20:30:54.295 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.296 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.296 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.297 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.297 DEBUG:boto:Method: GET 2023-10-18T20:30:54.298 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.298 DEBUG:boto:Data: 2023-10-18T20:30:54.298 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.299 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.300 DEBUG:boto:Params: {} 2023-10-18T20:30:54.300 DEBUG:boto:Token: None 2023-10-18T20:30:54.301 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FIu3E7ozXeU8gXK2DEONSqsVCAw= 2023-10-18T20:30:54.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FIu3E7ozXeU8gXK2DEONSqsVCAw=', 'Content-Length': '0'} 2023-10-18T20:30:54.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000141bc7bb873d04f9-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.304 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.305 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.305 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.306 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.306 DEBUG:boto:Method: GET 2023-10-18T20:30:54.307 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.307 DEBUG:boto:Data: 2023-10-18T20:30:54.307 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.308 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.308 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.309 DEBUG:boto:Params: {} 2023-10-18T20:30:54.309 DEBUG:boto:Token: None 2023-10-18T20:30:54.310 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076?versionId=null 2023-10-18T20:30:54.310 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FIu3E7ozXeU8gXK2DEONSqsVCAw= 2023-10-18T20:30:54.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FIu3E7ozXeU8gXK2DEONSqsVCAw=', 'Content-Length': '0'} 2023-10-18T20:30:54.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dca5431e593b5b9-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.313 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.314 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.314 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.315 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.315 DEBUG:boto:Method: GET 2023-10-18T20:30:54.315 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.316 DEBUG:boto:Data: 2023-10-18T20:30:54.316 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.317 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.317 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.318 DEBUG:boto:Params: {} 2023-10-18T20:30:54.318 DEBUG:boto:Token: None 2023-10-18T20:30:54.319 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.319 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554d5f8c4043e235-006530407e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.323 DEBUG:boto:b'bofrnf-58obj000000000000000000761000falseobj000000000000000000762023-10-18T20:30:26.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.323 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.324 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.324 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.325 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.325 DEBUG:boto:Method: GET 2023-10-18T20:30:54.326 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.326 DEBUG:boto:Data: 2023-10-18T20:30:54.326 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.327 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.327 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.328 DEBUG:boto:Params: {} 2023-10-18T20:30:54.328 DEBUG:boto:Token: None 2023-10-18T20:30:54.329 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.329 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb60360ae57ad28-006530407e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.335 DEBUG:boto:b'bofrnf-58obj000000000000000000761000falseobj000000000000000000762023-10-18T20:30:26.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.335 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.336 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.336 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.336 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.337 DEBUG:boto:Data: 2023-10-18T20:30:54.337 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.338 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.338 DEBUG:boto:Params: {} 2023-10-18T20:30:54.339 DEBUG:boto:Token: None 2023-10-18T20:30:54.340 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YHSeWPMaO5bCjHnaEzODqo6rR5U= 2023-10-18T20:30:54.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YHSeWPMaO5bCjHnaEzODqo6rR5U=', 'Content-Length': '0'} 2023-10-18T20:30:54.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004208c189119256ee-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.343 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.344 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.344 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.344 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.345 DEBUG:boto:Data: 2023-10-18T20:30:54.345 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.346 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.346 DEBUG:boto:Params: {} 2023-10-18T20:30:54.347 DEBUG:boto:Token: None 2023-10-18T20:30:54.347 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.349 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YHSeWPMaO5bCjHnaEzODqo6rR5U= 2023-10-18T20:30:54.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YHSeWPMaO5bCjHnaEzODqo6rR5U=', 'Content-Length': '0'} 2023-10-18T20:30:54.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000470d3417cb7a64c4-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-10-18T20:30:54.353 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.353 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.354 DEBUG:boto:Method: GET 2023-10-18T20:30:54.354 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.355 DEBUG:boto:Data: 2023-10-18T20:30:54.355 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.356 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.356 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.356 DEBUG:boto:Params: {} 2023-10-18T20:30:54.357 DEBUG:boto:Token: None 2023-10-18T20:30:54.358 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F+NFCDRvAAF+ccP7RAAfb6l5Rtg= 2023-10-18T20:30:54.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F+NFCDRvAAF+ccP7RAAfb6l5Rtg=', 'Content-Length': '0'} 2023-10-18T20:30:54.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072605a72de24bbce-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.402 DEBUG:boto:path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.403 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.403 DEBUG:boto:Method: GET 2023-10-18T20:30:54.404 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.404 DEBUG:boto:Data: 2023-10-18T20:30:54.405 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.405 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.406 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.406 DEBUG:boto:Params: {} 2023-10-18T20:30:54.407 DEBUG:boto:Token: None 2023-10-18T20:30:54.407 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000076 2023-10-18T20:30:54.408 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F+NFCDRvAAF+ccP7RAAfb6l5Rtg= 2023-10-18T20:30:54.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F+NFCDRvAAF+ccP7RAAfb6l5Rtg=', 'Content-Length': '0'} 2023-10-18T20:30:54.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a17baa2bf5d625e6-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.411 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.412 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.412 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.412 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.413 DEBUG:boto:Method: GET 2023-10-18T20:30:54.413 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.414 DEBUG:boto:Data: 2023-10-18T20:30:54.414 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.415 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.415 DEBUG:boto:Params: {} 2023-10-18T20:30:54.416 DEBUG:boto:Token: None 2023-10-18T20:30:54.416 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.417 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d533481fdebea46d-006530407e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.421 DEBUG:boto:b'bofrnf-58obj000000000000000000761000falseobj000000000000000000762023-10-18T20:30:26.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.421 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.422 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.422 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.423 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.423 DEBUG:boto:Method: GET 2023-10-18T20:30:54.424 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000076 2023-10-18T20:30:54.424 DEBUG:boto:Data: 2023-10-18T20:30:54.424 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.425 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.426 DEBUG:boto:Params: {} 2023-10-18T20:30:54.426 DEBUG:boto:Token: None 2023-10-18T20:30:54.427 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.427 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca49cd643d72eae3-006530407e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.474 DEBUG:boto:b'bofrnf-58obj000000000000000000761000falseobj000000000000000000762023-10-18T20:30:26.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-18T20:30:54.476 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.476 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.476 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.477 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.477 DEBUG:boto:Method: GET 2023-10-18T20:30:54.478 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.478 DEBUG:boto:Data: 2023-10-18T20:30:54.479 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.479 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.479 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.480 DEBUG:boto:Params: {} 2023-10-18T20:30:54.480 DEBUG:boto:Token: None 2023-10-18T20:30:54.481 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.481 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QuNcGWsq9iVfNXBlMeqRgaNOEM0= 2023-10-18T20:30:54.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QuNcGWsq9iVfNXBlMeqRgaNOEM0=', 'Content-Length': '0'} 2023-10-18T20:30:54.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092023adb58c5b742-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.485 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.485 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.485 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.486 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.486 DEBUG:boto:Method: GET 2023-10-18T20:30:54.487 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.487 DEBUG:boto:Data: 2023-10-18T20:30:54.488 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.488 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.489 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.489 DEBUG:boto:Params: {} 2023-10-18T20:30:54.489 DEBUG:boto:Token: None 2023-10-18T20:30:54.490 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.490 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QuNcGWsq9iVfNXBlMeqRgaNOEM0= 2023-10-18T20:30:54.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QuNcGWsq9iVfNXBlMeqRgaNOEM0=', 'Content-Length': '0'} 2023-10-18T20:30:54.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f123f4dd1e402b-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.493 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.494 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.494 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.494 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.495 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.495 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.496 DEBUG:boto:Data: 2023-10-18T20:30:54.496 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.496 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.497 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.497 DEBUG:boto:Params: {} 2023-10-18T20:30:54.497 DEBUG:boto:Token: None 2023-10-18T20:30:54.498 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.498 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r4X9eALpN4NC6SLASTGqSh5G/K8= 2023-10-18T20:30:54.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r4X9eALpN4NC6SLASTGqSh5G/K8=', 'Content-Length': '0'} 2023-10-18T20:30:54.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000021b290481d8f68-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.501 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.501 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.502 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.502 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.503 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.503 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.503 DEBUG:boto:Data: 2023-10-18T20:30:54.504 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.504 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.504 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.505 DEBUG:boto:Params: {} 2023-10-18T20:30:54.505 DEBUG:boto:Token: None 2023-10-18T20:30:54.506 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.506 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r4X9eALpN4NC6SLASTGqSh5G/K8= 2023-10-18T20:30:54.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r4X9eALpN4NC6SLASTGqSh5G/K8=', 'Content-Length': '0'} 2023-10-18T20:30:54.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dd87dece9050dc5-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-18T20:30:54.509 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.510 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.510 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.511 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.511 DEBUG:boto:Method: GET 2023-10-18T20:30:54.512 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.512 DEBUG:boto:Data: 2023-10-18T20:30:54.513 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.513 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.513 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.514 DEBUG:boto:Params: {} 2023-10-18T20:30:54.514 DEBUG:boto:Token: None 2023-10-18T20:30:54.515 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.515 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QuNcGWsq9iVfNXBlMeqRgaNOEM0= 2023-10-18T20:30:54.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QuNcGWsq9iVfNXBlMeqRgaNOEM0=', 'Content-Length': '0'} 2023-10-18T20:30:54.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d8923a916e41962-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.518 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.519 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.519 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.520 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.520 DEBUG:boto:Method: GET 2023-10-18T20:30:54.521 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.521 DEBUG:boto:Data: 2023-10-18T20:30:54.522 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.522 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.523 DEBUG:boto:Params: {} 2023-10-18T20:30:54.523 DEBUG:boto:Token: None 2023-10-18T20:30:54.524 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133?versionId=null 2023-10-18T20:30:54.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QuNcGWsq9iVfNXBlMeqRgaNOEM0= 2023-10-18T20:30:54.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QuNcGWsq9iVfNXBlMeqRgaNOEM0=', 'Content-Length': '0'} 2023-10-18T20:30:54.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042958f62479be338-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.527 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.528 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.528 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.528 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.529 DEBUG:boto:Method: GET 2023-10-18T20:30:54.529 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.530 DEBUG:boto:Data: 2023-10-18T20:30:54.530 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.531 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.531 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.532 DEBUG:boto:Params: {} 2023-10-18T20:30:54.532 DEBUG:boto:Token: None 2023-10-18T20:30:54.533 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003440d84ac96d9f93-006530407e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.536 DEBUG:boto:b'bofrnf-58obj000000000000000001331000falseobj000000000000000001332023-10-18T20:30:30.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.537 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.538 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.538 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.538 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.539 DEBUG:boto:Method: GET 2023-10-18T20:30:54.539 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.540 DEBUG:boto:Data: 2023-10-18T20:30:54.540 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.541 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.542 DEBUG:boto:Params: {} 2023-10-18T20:30:54.542 DEBUG:boto:Token: None 2023-10-18T20:30:54.543 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b953eb0d83a16b3-006530407e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.548 DEBUG:boto:b'bofrnf-58obj000000000000000001331000falseobj000000000000000001332023-10-18T20:30:30.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.549 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.549 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.549 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.550 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.550 DEBUG:boto:Data: 2023-10-18T20:30:54.553 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.553 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.554 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.554 DEBUG:boto:Params: {} 2023-10-18T20:30:54.554 DEBUG:boto:Token: None 2023-10-18T20:30:54.555 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qVH8ZQEDTUEIkmHp0HwGoeY/JVQ= 2023-10-18T20:30:54.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qVH8ZQEDTUEIkmHp0HwGoeY/JVQ=', 'Content-Length': '0'} 2023-10-18T20:30:54.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006351d06e9350798f-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.558 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.559 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.559 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.560 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.560 DEBUG:boto:Data: 2023-10-18T20:30:54.561 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.561 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.561 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.562 DEBUG:boto:Params: {} 2023-10-18T20:30:54.562 DEBUG:boto:Token: None 2023-10-18T20:30:54.563 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.563 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qVH8ZQEDTUEIkmHp0HwGoeY/JVQ= 2023-10-18T20:30:54.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qVH8ZQEDTUEIkmHp0HwGoeY/JVQ=', 'Content-Length': '0'} 2023-10-18T20:30:54.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ea9b838f65635a4-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-18T20:30:54.567 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.568 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.568 DEBUG:boto:Method: GET 2023-10-18T20:30:54.568 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.569 DEBUG:boto:Data: 2023-10-18T20:30:54.569 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.570 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.570 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.570 DEBUG:boto:Params: {} 2023-10-18T20:30:54.571 DEBUG:boto:Token: None 2023-10-18T20:30:54.571 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.572 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5skgJz//VTtGol+hiK2wBinco5I= 2023-10-18T20:30:54.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5skgJz//VTtGol+hiK2wBinco5I=', 'Content-Length': '0'} 2023-10-18T20:30:54.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c33ab16777e535b-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.575 DEBUG:boto:path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.576 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.576 DEBUG:boto:Method: GET 2023-10-18T20:30:54.576 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.577 DEBUG:boto:Data: 2023-10-18T20:30:54.577 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.578 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.578 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.579 DEBUG:boto:Params: {} 2023-10-18T20:30:54.579 DEBUG:boto:Token: None 2023-10-18T20:30:54.579 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000133 2023-10-18T20:30:54.580 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5skgJz//VTtGol+hiK2wBinco5I= 2023-10-18T20:30:54.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5skgJz//VTtGol+hiK2wBinco5I=', 'Content-Length': '0'} 2023-10-18T20:30:54.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b274830535ffd3-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.626 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.627 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.627 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.628 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.628 DEBUG:boto:Method: GET 2023-10-18T20:30:54.629 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.629 DEBUG:boto:Data: 2023-10-18T20:30:54.629 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.630 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.631 DEBUG:boto:Params: {} 2023-10-18T20:30:54.631 DEBUG:boto:Token: None 2023-10-18T20:30:54.631 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.632 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a731a2fabad95a9-006530407e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.636 DEBUG:boto:b'bofrnf-58obj000000000000000001331000falseobj000000000000000001332023-10-18T20:30:30.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.637 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.637 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.638 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.638 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.639 DEBUG:boto:Method: GET 2023-10-18T20:30:54.639 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000133 2023-10-18T20:30:54.640 DEBUG:boto:Data: 2023-10-18T20:30:54.640 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.640 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.641 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.641 DEBUG:boto:Params: {} 2023-10-18T20:30:54.642 DEBUG:boto:Token: None 2023-10-18T20:30:54.642 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009182060998aefc0a-006530407e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.648 DEBUG:boto:b'bofrnf-58obj000000000000000001331000falseobj000000000000000001332023-10-18T20:30:30.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-10-18T20:30:54.649 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.649 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.650 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.650 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.651 DEBUG:boto:Method: GET 2023-10-18T20:30:54.651 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.652 DEBUG:boto:Data: 2023-10-18T20:30:54.652 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.653 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.653 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.653 DEBUG:boto:Params: {} 2023-10-18T20:30:54.654 DEBUG:boto:Token: None 2023-10-18T20:30:54.654 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.655 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yf+HEoCxiBO+NWq+6j5OAKuueZk= 2023-10-18T20:30:54.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yf+HEoCxiBO+NWq+6j5OAKuueZk=', 'Content-Length': '0'} 2023-10-18T20:30:54.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006301160cac9a72e8-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.658 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.659 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.659 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.660 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.660 DEBUG:boto:Method: GET 2023-10-18T20:30:54.660 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.661 DEBUG:boto:Data: 2023-10-18T20:30:54.661 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.662 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.663 DEBUG:boto:Params: {} 2023-10-18T20:30:54.663 DEBUG:boto:Token: None 2023-10-18T20:30:54.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yf+HEoCxiBO+NWq+6j5OAKuueZk= 2023-10-18T20:30:54.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yf+HEoCxiBO+NWq+6j5OAKuueZk=', 'Content-Length': '0'} 2023-10-18T20:30:54.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025ca0bb55065ab60-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.667 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.668 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.668 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.669 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.669 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.670 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.670 DEBUG:boto:Data: 2023-10-18T20:30:54.671 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.671 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.672 DEBUG:boto:Params: {} 2023-10-18T20:30:54.672 DEBUG:boto:Token: None 2023-10-18T20:30:54.673 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KQPsfsLgH8J6tpZucPHxC9LSqBI= 2023-10-18T20:30:54.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KQPsfsLgH8J6tpZucPHxC9LSqBI=', 'Content-Length': '0'} 2023-10-18T20:30:54.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4ff246f8168f149-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.676 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.677 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.677 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.678 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.678 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.679 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.679 DEBUG:boto:Data: 2023-10-18T20:30:54.679 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.680 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.681 DEBUG:boto:Params: {} 2023-10-18T20:30:54.681 DEBUG:boto:Token: None 2023-10-18T20:30:54.682 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.682 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KQPsfsLgH8J6tpZucPHxC9LSqBI= 2023-10-18T20:30:54.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KQPsfsLgH8J6tpZucPHxC9LSqBI=', 'Content-Length': '0'} 2023-10-18T20:30:54.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a66f6f31221eef12-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-10-18T20:30:54.686 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.686 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.687 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.687 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.688 DEBUG:boto:Method: GET 2023-10-18T20:30:54.688 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.688 DEBUG:boto:Data: 2023-10-18T20:30:54.689 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.690 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.690 DEBUG:boto:Params: {} 2023-10-18T20:30:54.691 DEBUG:boto:Token: None 2023-10-18T20:30:54.691 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yf+HEoCxiBO+NWq+6j5OAKuueZk= 2023-10-18T20:30:54.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yf+HEoCxiBO+NWq+6j5OAKuueZk=', 'Content-Length': '0'} 2023-10-18T20:30:54.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4c15bdff1eb9978-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.695 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.695 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.696 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.696 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.697 DEBUG:boto:Method: GET 2023-10-18T20:30:54.697 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.698 DEBUG:boto:Data: 2023-10-18T20:30:54.698 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.699 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.699 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.699 DEBUG:boto:Params: {} 2023-10-18T20:30:54.700 DEBUG:boto:Token: None 2023-10-18T20:30:54.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162?versionId=null 2023-10-18T20:30:54.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yf+HEoCxiBO+NWq+6j5OAKuueZk= 2023-10-18T20:30:54.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yf+HEoCxiBO+NWq+6j5OAKuueZk=', 'Content-Length': '0'} 2023-10-18T20:30:54.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058b1561be25499aa-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.750 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.751 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.751 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.752 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.752 DEBUG:boto:Method: GET 2023-10-18T20:30:54.753 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.753 DEBUG:boto:Data: 2023-10-18T20:30:54.754 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.755 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.755 DEBUG:boto:Params: {} 2023-10-18T20:30:54.757 DEBUG:boto:Token: None 2023-10-18T20:30:54.757 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.758 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dae534b6103dcfa0-006530407e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.762 DEBUG:boto:b'bofrnf-58obj000000000000000001621000falseobj000000000000000001622023-10-18T20:30:31.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.763 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.763 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.763 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.764 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.764 DEBUG:boto:Method: GET 2023-10-18T20:30:54.765 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.765 DEBUG:boto:Data: 2023-10-18T20:30:54.766 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.766 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.767 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.767 DEBUG:boto:Params: {} 2023-10-18T20:30:54.768 DEBUG:boto:Token: None 2023-10-18T20:30:54.768 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.769 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2ca33c986f5c8ea-006530407e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.774 DEBUG:boto:b'bofrnf-58obj000000000000000001621000falseobj000000000000000001622023-10-18T20:30:31.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.775 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.775 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.776 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.776 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.777 DEBUG:boto:Data: 2023-10-18T20:30:54.777 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.777 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.778 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.778 DEBUG:boto:Params: {} 2023-10-18T20:30:54.779 DEBUG:boto:Token: None 2023-10-18T20:30:54.779 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t5Qy/51oawCj0afS6g5mNm9gwZ4= 2023-10-18T20:30:54.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t5Qy/51oawCj0afS6g5mNm9gwZ4=', 'Content-Length': '0'} 2023-10-18T20:30:54.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078dbb31adc2a5b04-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.783 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.783 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.783 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.784 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.784 DEBUG:boto:Data: 2023-10-18T20:30:54.785 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.785 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.785 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.786 DEBUG:boto:Params: {} 2023-10-18T20:30:54.786 DEBUG:boto:Token: None 2023-10-18T20:30:54.787 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.787 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t5Qy/51oawCj0afS6g5mNm9gwZ4= 2023-10-18T20:30:54.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t5Qy/51oawCj0afS6g5mNm9gwZ4=', 'Content-Length': '0'} 2023-10-18T20:30:54.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c9e78527da0c5e-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-10-18T20:30:54.791 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.791 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.792 DEBUG:boto:Method: GET 2023-10-18T20:30:54.792 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.793 DEBUG:boto:Data: 2023-10-18T20:30:54.793 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.793 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.794 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.794 DEBUG:boto:Params: {} 2023-10-18T20:30:54.795 DEBUG:boto:Token: None 2023-10-18T20:30:54.795 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.796 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WLvay7O+tlYdhDx4oI/O1Q0rmrI= 2023-10-18T20:30:54.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WLvay7O+tlYdhDx4oI/O1Q0rmrI=', 'Content-Length': '0'} 2023-10-18T20:30:54.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa86747cb87d763e-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.799 DEBUG:boto:path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.800 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.800 DEBUG:boto:Method: GET 2023-10-18T20:30:54.801 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.801 DEBUG:boto:Data: 2023-10-18T20:30:54.802 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.802 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.802 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.803 DEBUG:boto:Params: {} 2023-10-18T20:30:54.803 DEBUG:boto:Token: None 2023-10-18T20:30:54.804 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000162 2023-10-18T20:30:54.804 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WLvay7O+tlYdhDx4oI/O1Q0rmrI= 2023-10-18T20:30:54.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WLvay7O+tlYdhDx4oI/O1Q0rmrI=', 'Content-Length': '0'} 2023-10-18T20:30:54.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8a3d3fc94f8c80-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.808 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.808 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.809 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.809 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.810 DEBUG:boto:Method: GET 2023-10-18T20:30:54.810 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.810 DEBUG:boto:Data: 2023-10-18T20:30:54.811 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.811 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.812 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.812 DEBUG:boto:Params: {} 2023-10-18T20:30:54.813 DEBUG:boto:Token: None 2023-10-18T20:30:54.813 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.814 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000400d382cc7924aa8-006530407e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.862 DEBUG:boto:b'bofrnf-58obj000000000000000001621000falseobj000000000000000001622023-10-18T20:30:31.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.863 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.864 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.864 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.865 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.865 DEBUG:boto:Method: GET 2023-10-18T20:30:54.865 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000162 2023-10-18T20:30:54.866 DEBUG:boto:Data: 2023-10-18T20:30:54.866 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.867 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.867 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.867 DEBUG:boto:Params: {} 2023-10-18T20:30:54.868 DEBUG:boto:Token: None 2023-10-18T20:30:54.868 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.869 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab5bb198f4655e2f-006530407e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.874 DEBUG:boto:b'bofrnf-58obj000000000000000001621000falseobj000000000000000001622023-10-18T20:30:31.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-18T20:30:54.876 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.876 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.876 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.877 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.877 DEBUG:boto:Method: GET 2023-10-18T20:30:54.878 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.878 DEBUG:boto:Data: 2023-10-18T20:30:54.878 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.879 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.880 DEBUG:boto:Params: {} 2023-10-18T20:30:54.880 DEBUG:boto:Token: None 2023-10-18T20:30:54.881 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.881 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DIsl+/elF1FMOAWwNBHm4nB8kMU= 2023-10-18T20:30:54.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DIsl+/elF1FMOAWwNBHm4nB8kMU=', 'Content-Length': '0'} 2023-10-18T20:30:54.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ad78b06ae9f8fc-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.884 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.885 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.885 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.886 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.886 DEBUG:boto:Method: GET 2023-10-18T20:30:54.887 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.887 DEBUG:boto:Data: 2023-10-18T20:30:54.887 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.888 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.888 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.889 DEBUG:boto:Params: {} 2023-10-18T20:30:54.889 DEBUG:boto:Token: None 2023-10-18T20:30:54.889 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.890 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DIsl+/elF1FMOAWwNBHm4nB8kMU= 2023-10-18T20:30:54.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DIsl+/elF1FMOAWwNBHm4nB8kMU=', 'Content-Length': '0'} 2023-10-18T20:30:54.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055cb0ddfb4d0152-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.893 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.894 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.894 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.895 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.895 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.896 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.896 DEBUG:boto:Data: 2023-10-18T20:30:54.896 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.897 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.897 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.897 DEBUG:boto:Params: {} 2023-10-18T20:30:54.898 DEBUG:boto:Token: None 2023-10-18T20:30:54.898 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.899 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qJoWp7PNZd2Ny9/9A9KxfHj3ViY= 2023-10-18T20:30:54.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qJoWp7PNZd2Ny9/9A9KxfHj3ViY=', 'Content-Length': '0'} 2023-10-18T20:30:54.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c36f6155fd047dd0-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.902 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.902 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.903 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.903 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.903 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.904 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.904 DEBUG:boto:Data: 2023-10-18T20:30:54.904 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.905 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.905 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.906 DEBUG:boto:Params: {} 2023-10-18T20:30:54.906 DEBUG:boto:Token: None 2023-10-18T20:30:54.906 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.907 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qJoWp7PNZd2Ny9/9A9KxfHj3ViY= 2023-10-18T20:30:54.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qJoWp7PNZd2Ny9/9A9KxfHj3ViY=', 'Content-Length': '0'} 2023-10-18T20:30:54.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e0ba25d83741e2d-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-18T20:30:54.910 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.911 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.911 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.911 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.912 DEBUG:boto:Method: GET 2023-10-18T20:30:54.912 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.913 DEBUG:boto:Data: 2023-10-18T20:30:54.913 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.913 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.914 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.914 DEBUG:boto:Params: {} 2023-10-18T20:30:54.915 DEBUG:boto:Token: None 2023-10-18T20:30:54.915 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.916 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DIsl+/elF1FMOAWwNBHm4nB8kMU= 2023-10-18T20:30:54.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DIsl+/elF1FMOAWwNBHm4nB8kMU=', 'Content-Length': '0'} 2023-10-18T20:30:54.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed2434a41385d5d-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.919 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.919 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.920 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.920 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.920 DEBUG:boto:Method: GET 2023-10-18T20:30:54.921 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.921 DEBUG:boto:Data: 2023-10-18T20:30:54.922 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.922 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.922 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.923 DEBUG:boto:Params: {} 2023-10-18T20:30:54.923 DEBUG:boto:Token: None 2023-10-18T20:30:54.924 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000271?versionId=null 2023-10-18T20:30:54.924 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DIsl+/elF1FMOAWwNBHm4nB8kMU= 2023-10-18T20:30:54.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DIsl+/elF1FMOAWwNBHm4nB8kMU=', 'Content-Length': '0'} 2023-10-18T20:30:54.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dfbc1d57ef7af36-006530407e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.927 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.928 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.928 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:54.929 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:54.929 DEBUG:boto:Method: GET 2023-10-18T20:30:54.930 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:54.930 DEBUG:boto:Data: 2023-10-18T20:30:54.930 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.931 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.932 DEBUG:boto:Params: {} 2023-10-18T20:30:54.932 DEBUG:boto:Token: None 2023-10-18T20:30:54.933 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000947aa3d32d4ab7d5-006530407e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.937 DEBUG:boto:b'bofrnf-58obj000000000000000002711000falseobj000000000000000002712023-10-18T20:30:31.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.938 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:54.938 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:54.939 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:54.939 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:54.940 DEBUG:boto:Method: GET 2023-10-18T20:30:54.940 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:54.940 DEBUG:boto:Data: 2023-10-18T20:30:54.941 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:54.942 DEBUG:boto:Port: 8000 2023-10-18T20:30:54.942 DEBUG:boto:Params: {} 2023-10-18T20:30:54.943 DEBUG:boto:Token: None 2023-10-18T20:30:54.943 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/ 2023-10-18T20:30:54.944 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E= 2023-10-18T20:30:54.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:376VBUOmgEUEgIWWrEVCTqaRZ+E=', 'Content-Length': '0'} 2023-10-18T20:30:54.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0349133158277cb-006530407e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.990 DEBUG:boto:b'bofrnf-58obj000000000000000002711000falseobj000000000000000002712023-10-18T20:30:31.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:54.991 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.992 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.992 DEBUG:boto:Method: HEAD 2023-10-18T20:30:54.992 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.993 DEBUG:boto:Data: 2023-10-18T20:30:54.993 DEBUG:boto:Headers: {} 2023-10-18T20:30:54.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:54.994 DEBUG:boto:Port: 8001 2023-10-18T20:30:54.995 DEBUG:boto:Params: {} 2023-10-18T20:30:54.995 DEBUG:boto:Token: None 2023-10-18T20:30:54.995 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:54 GMT /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:54.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:57iBPqh/FNunGAQzD5hBEhhYrJY= 2023-10-18T20:30:54.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:57iBPqh/FNunGAQzD5hBEhhYrJY=', 'Content-Length': '0'} 2023-10-18T20:30:54.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d496b93d945f6ed3-006530407e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:54.999 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.000 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.000 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.001 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.001 DEBUG:boto:Data: 2023-10-18T20:30:55.001 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.002 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.002 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.003 DEBUG:boto:Params: {} 2023-10-18T20:30:55.003 DEBUG:boto:Token: None 2023-10-18T20:30:55.004 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RWJ3nJfmnA4ppE5M5rHEH3ix5mY= 2023-10-18T20:30:55.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RWJ3nJfmnA4ppE5M5rHEH3ix5mY=', 'Content-Length': '0'} 2023-10-18T20:30:55.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099eaddc62d63184d-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-18T20:30:55.008 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.008 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.009 DEBUG:boto:Method: GET 2023-10-18T20:30:55.009 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.010 DEBUG:boto:Data: 2023-10-18T20:30:55.010 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.011 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.011 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.012 DEBUG:boto:Params: {} 2023-10-18T20:30:55.012 DEBUG:boto:Token: None 2023-10-18T20:30:55.012 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.013 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JsYQW6UNrWv+FdY5L22OVaX2HnA= 2023-10-18T20:30:55.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JsYQW6UNrWv+FdY5L22OVaX2HnA=', 'Content-Length': '0'} 2023-10-18T20:30:55.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd090f0f47561812-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.016 DEBUG:boto:path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.017 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.017 DEBUG:boto:Method: GET 2023-10-18T20:30:55.018 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.018 DEBUG:boto:Data: 2023-10-18T20:30:55.018 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.019 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.019 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.020 DEBUG:boto:Params: {} 2023-10-18T20:30:55.020 DEBUG:boto:Token: None 2023-10-18T20:30:55.021 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000271 2023-10-18T20:30:55.021 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JsYQW6UNrWv+FdY5L22OVaX2HnA= 2023-10-18T20:30:55.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JsYQW6UNrWv+FdY5L22OVaX2HnA=', 'Content-Length': '0'} 2023-10-18T20:30:55.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d71f3fe72cd7b92-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.024 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.025 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.025 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:55.026 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:55.026 DEBUG:boto:Method: GET 2023-10-18T20:30:55.026 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:55.027 DEBUG:boto:Data: 2023-10-18T20:30:55.027 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.028 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.028 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.029 DEBUG:boto:Params: {} 2023-10-18T20:30:55.029 DEBUG:boto:Token: None 2023-10-18T20:30:55.029 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9f2913f60754d5f-006530407f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.034 DEBUG:boto:b'bofrnf-58obj000000000000000002711000falseobj000000000000000002712023-10-18T20:30:31.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.034 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.035 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.035 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:55.036 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:55.036 DEBUG:boto:Method: GET 2023-10-18T20:30:55.037 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000271 2023-10-18T20:30:55.037 DEBUG:boto:Data: 2023-10-18T20:30:55.037 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.038 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.038 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.039 DEBUG:boto:Params: {} 2023-10-18T20:30:55.039 DEBUG:boto:Token: None 2023-10-18T20:30:55.040 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003934721fdf901b39-006530407f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.046 DEBUG:boto:b'bofrnf-58obj000000000000000002711000falseobj000000000000000002712023-10-18T20:30:31.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-18T20:30:55.047 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.047 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.048 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.048 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.049 DEBUG:boto:Method: GET 2023-10-18T20:30:55.049 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.049 DEBUG:boto:Data: 2023-10-18T20:30:55.050 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.051 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.051 DEBUG:boto:Params: {} 2023-10-18T20:30:55.052 DEBUG:boto:Token: None 2023-10-18T20:30:55.052 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.053 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X+iLKwganHYZfEjufCwhSpmkLOM= 2023-10-18T20:30:55.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X+iLKwganHYZfEjufCwhSpmkLOM=', 'Content-Length': '0'} 2023-10-18T20:30:55.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1f213cf5020b2ca-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.056 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.057 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.057 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.057 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.058 DEBUG:boto:Method: GET 2023-10-18T20:30:55.058 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.059 DEBUG:boto:Data: 2023-10-18T20:30:55.059 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.060 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.060 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.060 DEBUG:boto:Params: {} 2023-10-18T20:30:55.061 DEBUG:boto:Token: None 2023-10-18T20:30:55.061 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.062 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X+iLKwganHYZfEjufCwhSpmkLOM= 2023-10-18T20:30:55.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X+iLKwganHYZfEjufCwhSpmkLOM=', 'Content-Length': '0'} 2023-10-18T20:30:55.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fc05f399e58a42c-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.107 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.108 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.108 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.109 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.110 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.110 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.111 DEBUG:boto:Data: 2023-10-18T20:30:55.112 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.113 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.113 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.114 DEBUG:boto:Params: {} 2023-10-18T20:30:55.114 DEBUG:boto:Token: None 2023-10-18T20:30:55.115 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CeP9sq+eyFbxXFD18NuBpcRHGHI= 2023-10-18T20:30:55.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CeP9sq+eyFbxXFD18NuBpcRHGHI=', 'Content-Length': '0'} 2023-10-18T20:30:55.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045b7717e861bd82c-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.119 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.119 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.120 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.120 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.120 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.121 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.121 DEBUG:boto:Data: 2023-10-18T20:30:55.122 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.123 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.123 DEBUG:boto:Params: {} 2023-10-18T20:30:55.123 DEBUG:boto:Token: None 2023-10-18T20:30:55.124 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.125 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CeP9sq+eyFbxXFD18NuBpcRHGHI= 2023-10-18T20:30:55.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CeP9sq+eyFbxXFD18NuBpcRHGHI=', 'Content-Length': '0'} 2023-10-18T20:30:55.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eaa17ed476aca70-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-18T20:30:55.129 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.129 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.129 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.130 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.130 DEBUG:boto:Method: GET 2023-10-18T20:30:55.131 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.131 DEBUG:boto:Data: 2023-10-18T20:30:55.132 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.132 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.133 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.133 DEBUG:boto:Params: {} 2023-10-18T20:30:55.134 DEBUG:boto:Token: None 2023-10-18T20:30:55.134 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X+iLKwganHYZfEjufCwhSpmkLOM= 2023-10-18T20:30:55.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X+iLKwganHYZfEjufCwhSpmkLOM=', 'Content-Length': '0'} 2023-10-18T20:30:55.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066a285d1557652ee-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.138 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.139 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.139 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.139 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.140 DEBUG:boto:Method: GET 2023-10-18T20:30:55.140 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.141 DEBUG:boto:Data: 2023-10-18T20:30:55.141 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.142 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.142 DEBUG:boto:Params: {} 2023-10-18T20:30:55.143 DEBUG:boto:Token: None 2023-10-18T20:30:55.143 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283?versionId=null 2023-10-18T20:30:55.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X+iLKwganHYZfEjufCwhSpmkLOM= 2023-10-18T20:30:55.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X+iLKwganHYZfEjufCwhSpmkLOM=', 'Content-Length': '0'} 2023-10-18T20:30:55.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4520efc1d671596-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.147 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.147 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.148 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.148 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.149 DEBUG:boto:Method: GET 2023-10-18T20:30:55.149 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.150 DEBUG:boto:Data: 2023-10-18T20:30:55.150 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.151 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.151 DEBUG:boto:Params: {} 2023-10-18T20:30:55.152 DEBUG:boto:Token: None 2023-10-18T20:30:55.153 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd2c31136be68d14-006530407f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.157 DEBUG:boto:b'bofrnf-58obj000000000000000002831000falseobj000000000000000002832023-10-18T20:30:31.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.158 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.159 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.159 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.160 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.160 DEBUG:boto:Method: GET 2023-10-18T20:30:55.160 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.161 DEBUG:boto:Data: 2023-10-18T20:30:55.162 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.162 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.162 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.166 DEBUG:boto:Params: {} 2023-10-18T20:30:55.166 DEBUG:boto:Token: None 2023-10-18T20:30:55.167 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5c2499dd5d7973d-006530407f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.214 DEBUG:boto:b'bofrnf-58obj000000000000000002831000falseobj000000000000000002832023-10-18T20:30:31.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.215 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.216 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.216 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.217 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.217 DEBUG:boto:Data: 2023-10-18T20:30:55.217 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.218 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.219 DEBUG:boto:Params: {} 2023-10-18T20:30:55.219 DEBUG:boto:Token: None 2023-10-18T20:30:55.220 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NKMo9r7CrX+yYNGBKvmc6gLralw= 2023-10-18T20:30:55.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NKMo9r7CrX+yYNGBKvmc6gLralw=', 'Content-Length': '0'} 2023-10-18T20:30:55.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009548509d282a090c-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.223 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.224 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.224 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.225 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.225 DEBUG:boto:Data: 2023-10-18T20:30:55.226 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.227 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.227 DEBUG:boto:Params: {} 2023-10-18T20:30:55.227 DEBUG:boto:Token: None 2023-10-18T20:30:55.228 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.229 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NKMo9r7CrX+yYNGBKvmc6gLralw= 2023-10-18T20:30:55.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NKMo9r7CrX+yYNGBKvmc6gLralw=', 'Content-Length': '0'} 2023-10-18T20:30:55.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ec2ba2d699177a1-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-18T20:30:55.233 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.233 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.234 DEBUG:boto:Method: GET 2023-10-18T20:30:55.234 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.235 DEBUG:boto:Data: 2023-10-18T20:30:55.235 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.236 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.236 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.236 DEBUG:boto:Params: {} 2023-10-18T20:30:55.237 DEBUG:boto:Token: None 2023-10-18T20:30:55.238 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.238 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uev8huaFjADM1mBlSyq0o7qlHSo= 2023-10-18T20:30:55.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uev8huaFjADM1mBlSyq0o7qlHSo=', 'Content-Length': '0'} 2023-10-18T20:30:55.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037f331b51ebcba01-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.282 DEBUG:boto:path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.283 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.284 DEBUG:boto:Method: GET 2023-10-18T20:30:55.284 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.285 DEBUG:boto:Data: 2023-10-18T20:30:55.285 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.286 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.286 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.287 DEBUG:boto:Params: {} 2023-10-18T20:30:55.287 DEBUG:boto:Token: None 2023-10-18T20:30:55.288 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000283 2023-10-18T20:30:55.288 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uev8huaFjADM1mBlSyq0o7qlHSo= 2023-10-18T20:30:55.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uev8huaFjADM1mBlSyq0o7qlHSo=', 'Content-Length': '0'} 2023-10-18T20:30:55.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1920bb9303a6584-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.293 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.294 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.294 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.295 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.295 DEBUG:boto:Method: GET 2023-10-18T20:30:55.296 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.296 DEBUG:boto:Data: 2023-10-18T20:30:55.297 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.297 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.298 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.298 DEBUG:boto:Params: {} 2023-10-18T20:30:55.299 DEBUG:boto:Token: None 2023-10-18T20:30:55.299 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.300 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000641807a53add539b-006530407f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.305 DEBUG:boto:b'bofrnf-58obj000000000000000002831000falseobj000000000000000002832023-10-18T20:30:31.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.305 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.306 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.307 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.307 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.308 DEBUG:boto:Method: GET 2023-10-18T20:30:55.308 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000283 2023-10-18T20:30:55.309 DEBUG:boto:Data: 2023-10-18T20:30:55.309 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.310 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.311 DEBUG:boto:Params: {} 2023-10-18T20:30:55.311 DEBUG:boto:Token: None 2023-10-18T20:30:55.312 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018782e05124a3731-006530407f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.317 DEBUG:boto:b'bofrnf-58obj000000000000000002831000falseobj000000000000000002832023-10-18T20:30:31.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-10-18T20:30:55.319 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.319 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.320 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.320 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.320 DEBUG:boto:Method: GET 2023-10-18T20:30:55.321 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.321 DEBUG:boto:Data: 2023-10-18T20:30:55.322 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.323 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.323 DEBUG:boto:Params: {} 2023-10-18T20:30:55.323 DEBUG:boto:Token: None 2023-10-18T20:30:55.324 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M+9XYWfdpJVG0rVDFliwH3I4pnk= 2023-10-18T20:30:55.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M+9XYWfdpJVG0rVDFliwH3I4pnk=', 'Content-Length': '0'} 2023-10-18T20:30:55.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d6409686fbcfa91-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.328 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.328 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.329 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.329 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.329 DEBUG:boto:Method: GET 2023-10-18T20:30:55.330 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.330 DEBUG:boto:Data: 2023-10-18T20:30:55.331 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.331 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.332 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.332 DEBUG:boto:Params: {} 2023-10-18T20:30:55.332 DEBUG:boto:Token: None 2023-10-18T20:30:55.333 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.333 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M+9XYWfdpJVG0rVDFliwH3I4pnk= 2023-10-18T20:30:55.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M+9XYWfdpJVG0rVDFliwH3I4pnk=', 'Content-Length': '0'} 2023-10-18T20:30:55.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e750b7e32b721005-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.382 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.383 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.383 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.384 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.384 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.385 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.385 DEBUG:boto:Data: 2023-10-18T20:30:55.385 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.386 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.387 DEBUG:boto:Params: {} 2023-10-18T20:30:55.387 DEBUG:boto:Token: None 2023-10-18T20:30:55.388 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gf1GE/55xC5V18v4b9YlSONtiSE= 2023-10-18T20:30:55.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gf1GE/55xC5V18v4b9YlSONtiSE=', 'Content-Length': '0'} 2023-10-18T20:30:55.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee3bef5d88d3d007-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.391 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.392 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.392 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.393 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.393 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.394 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.394 DEBUG:boto:Data: 2023-10-18T20:30:55.394 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.395 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.395 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.396 DEBUG:boto:Params: {} 2023-10-18T20:30:55.396 DEBUG:boto:Token: None 2023-10-18T20:30:55.397 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.397 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gf1GE/55xC5V18v4b9YlSONtiSE= 2023-10-18T20:30:55.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gf1GE/55xC5V18v4b9YlSONtiSE=', 'Content-Length': '0'} 2023-10-18T20:30:55.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd55775276c78b56-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-10-18T20:30:55.401 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.401 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.401 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.402 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.402 DEBUG:boto:Method: GET 2023-10-18T20:30:55.403 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.403 DEBUG:boto:Data: 2023-10-18T20:30:55.404 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.404 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.404 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.405 DEBUG:boto:Params: {} 2023-10-18T20:30:55.405 DEBUG:boto:Token: None 2023-10-18T20:30:55.406 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.406 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M+9XYWfdpJVG0rVDFliwH3I4pnk= 2023-10-18T20:30:55.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M+9XYWfdpJVG0rVDFliwH3I4pnk=', 'Content-Length': '0'} 2023-10-18T20:30:55.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4bbd7e53dca53f3-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.450 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.451 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.452 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.453 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.454 DEBUG:boto:Method: GET 2023-10-18T20:30:55.455 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.455 DEBUG:boto:Data: 2023-10-18T20:30:55.456 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.458 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.458 DEBUG:boto:Params: {} 2023-10-18T20:30:55.459 DEBUG:boto:Token: None 2023-10-18T20:30:55.459 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317?versionId=null 2023-10-18T20:30:55.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M+9XYWfdpJVG0rVDFliwH3I4pnk= 2023-10-18T20:30:55.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M+9XYWfdpJVG0rVDFliwH3I4pnk=', 'Content-Length': '0'} 2023-10-18T20:30:55.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bd5de7e7caf7774-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.464 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.464 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.464 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.465 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.465 DEBUG:boto:Method: GET 2023-10-18T20:30:55.466 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.466 DEBUG:boto:Data: 2023-10-18T20:30:55.467 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.467 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.467 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.468 DEBUG:boto:Params: {} 2023-10-18T20:30:55.468 DEBUG:boto:Token: None 2023-10-18T20:30:55.469 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba1a92e164daa5c4-006530407f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.473 DEBUG:boto:b'bofrnf-58obj000000000000000003171000falseobj000000000000000003172023-10-18T20:30:31.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.474 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.474 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.475 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.475 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.475 DEBUG:boto:Method: GET 2023-10-18T20:30:55.476 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.477 DEBUG:boto:Data: 2023-10-18T20:30:55.477 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.478 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.478 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.479 DEBUG:boto:Params: {} 2023-10-18T20:30:55.480 DEBUG:boto:Token: None 2023-10-18T20:30:55.481 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.481 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009795e32804938897-006530407f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.488 DEBUG:boto:b'bofrnf-58obj000000000000000003171000falseobj000000000000000003172023-10-18T20:30:31.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.489 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.489 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.490 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.491 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.491 DEBUG:boto:Data: 2023-10-18T20:30:55.492 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.492 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.493 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.493 DEBUG:boto:Params: {} 2023-10-18T20:30:55.494 DEBUG:boto:Token: None 2023-10-18T20:30:55.494 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.495 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7zoDlVV0na4HhFCC1EVaCgyOaEk= 2023-10-18T20:30:55.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7zoDlVV0na4HhFCC1EVaCgyOaEk=', 'Content-Length': '0'} 2023-10-18T20:30:55.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce77a214e68d17b-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.498 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.498 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.499 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.499 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.500 DEBUG:boto:Data: 2023-10-18T20:30:55.500 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.500 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.501 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.501 DEBUG:boto:Params: {} 2023-10-18T20:30:55.502 DEBUG:boto:Token: None 2023-10-18T20:30:55.502 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.503 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7zoDlVV0na4HhFCC1EVaCgyOaEk= 2023-10-18T20:30:55.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7zoDlVV0na4HhFCC1EVaCgyOaEk=', 'Content-Length': '0'} 2023-10-18T20:30:55.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0cbcc504479b7d7-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-10-18T20:30:55.506 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.507 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.507 DEBUG:boto:Method: GET 2023-10-18T20:30:55.508 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.508 DEBUG:boto:Data: 2023-10-18T20:30:55.508 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.509 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.509 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.510 DEBUG:boto:Params: {} 2023-10-18T20:30:55.510 DEBUG:boto:Token: None 2023-10-18T20:30:55.511 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.511 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QHQ2UnqgRVD2K0VxNWL8uccz++w= 2023-10-18T20:30:55.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QHQ2UnqgRVD2K0VxNWL8uccz++w=', 'Content-Length': '0'} 2023-10-18T20:30:55.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000827df6daadbd7c0d-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.515 DEBUG:boto:path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.515 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.515 DEBUG:boto:Method: GET 2023-10-18T20:30:55.516 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.516 DEBUG:boto:Data: 2023-10-18T20:30:55.517 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.517 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.517 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.518 DEBUG:boto:Params: {} 2023-10-18T20:30:55.518 DEBUG:boto:Token: None 2023-10-18T20:30:55.519 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000317 2023-10-18T20:30:55.519 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QHQ2UnqgRVD2K0VxNWL8uccz++w= 2023-10-18T20:30:55.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QHQ2UnqgRVD2K0VxNWL8uccz++w=', 'Content-Length': '0'} 2023-10-18T20:30:55.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069fa85f954fad9ec-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.562 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.563 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.563 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.564 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.564 DEBUG:boto:Method: GET 2023-10-18T20:30:55.565 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.565 DEBUG:boto:Data: 2023-10-18T20:30:55.566 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.566 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.567 DEBUG:boto:Params: {} 2023-10-18T20:30:55.567 DEBUG:boto:Token: None 2023-10-18T20:30:55.568 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07782622a479f1e-006530407f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.572 DEBUG:boto:b'bofrnf-58obj000000000000000003171000falseobj000000000000000003172023-10-18T20:30:31.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.573 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.573 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.574 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.574 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.575 DEBUG:boto:Method: GET 2023-10-18T20:30:55.577 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000317 2023-10-18T20:30:55.577 DEBUG:boto:Data: 2023-10-18T20:30:55.578 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.578 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.579 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.579 DEBUG:boto:Params: {} 2023-10-18T20:30:55.580 DEBUG:boto:Token: None 2023-10-18T20:30:55.580 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b99b9fd749ca465e-006530407f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.587 DEBUG:boto:b'bofrnf-58obj000000000000000003171000falseobj000000000000000003172023-10-18T20:30:31.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-18T20:30:55.588 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.588 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.589 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.589 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.589 DEBUG:boto:Method: GET 2023-10-18T20:30:55.590 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.590 DEBUG:boto:Data: 2023-10-18T20:30:55.591 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.591 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.591 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.592 DEBUG:boto:Params: {} 2023-10-18T20:30:55.592 DEBUG:boto:Token: None 2023-10-18T20:30:55.593 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.593 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dJhyX6W12PkvLsti7i7dkcVFdQ0= 2023-10-18T20:30:55.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dJhyX6W12PkvLsti7i7dkcVFdQ0=', 'Content-Length': '0'} 2023-10-18T20:30:55.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fad89c775ac52e7-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.597 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.598 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.598 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.598 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.599 DEBUG:boto:Method: GET 2023-10-18T20:30:55.599 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.600 DEBUG:boto:Data: 2023-10-18T20:30:55.600 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.601 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.601 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.602 DEBUG:boto:Params: {} 2023-10-18T20:30:55.602 DEBUG:boto:Token: None 2023-10-18T20:30:55.603 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.603 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dJhyX6W12PkvLsti7i7dkcVFdQ0= 2023-10-18T20:30:55.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dJhyX6W12PkvLsti7i7dkcVFdQ0=', 'Content-Length': '0'} 2023-10-18T20:30:55.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999ec7aea816aeb4-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.607 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.607 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.608 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.608 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.608 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.609 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.609 DEBUG:boto:Data: 2023-10-18T20:30:55.610 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.610 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.611 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.611 DEBUG:boto:Params: {} 2023-10-18T20:30:55.611 DEBUG:boto:Token: None 2023-10-18T20:30:55.612 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.612 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NOIVOqWM4Aw4twu/0uFPWMJxgDw= 2023-10-18T20:30:55.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NOIVOqWM4Aw4twu/0uFPWMJxgDw=', 'Content-Length': '0'} 2023-10-18T20:30:55.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3f20f8c61f13079-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.615 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.616 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.616 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.617 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.617 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.618 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.618 DEBUG:boto:Data: 2023-10-18T20:30:55.618 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.619 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.619 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.620 DEBUG:boto:Params: {} 2023-10-18T20:30:55.620 DEBUG:boto:Token: None 2023-10-18T20:30:55.620 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.621 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NOIVOqWM4Aw4twu/0uFPWMJxgDw= 2023-10-18T20:30:55.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NOIVOqWM4Aw4twu/0uFPWMJxgDw=', 'Content-Length': '0'} 2023-10-18T20:30:55.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b32ba57f205098ae-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-18T20:30:55.625 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.625 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.625 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.626 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.626 DEBUG:boto:Method: GET 2023-10-18T20:30:55.627 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.627 DEBUG:boto:Data: 2023-10-18T20:30:55.627 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.628 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.628 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.629 DEBUG:boto:Params: {} 2023-10-18T20:30:55.629 DEBUG:boto:Token: None 2023-10-18T20:30:55.629 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.630 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dJhyX6W12PkvLsti7i7dkcVFdQ0= 2023-10-18T20:30:55.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dJhyX6W12PkvLsti7i7dkcVFdQ0=', 'Content-Length': '0'} 2023-10-18T20:30:55.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c022805131a6e69-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.678 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.679 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.679 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.680 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.680 DEBUG:boto:Method: GET 2023-10-18T20:30:55.680 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.681 DEBUG:boto:Data: 2023-10-18T20:30:55.681 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.682 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.682 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.682 DEBUG:boto:Params: {} 2023-10-18T20:30:55.683 DEBUG:boto:Token: None 2023-10-18T20:30:55.683 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423?versionId=null 2023-10-18T20:30:55.684 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dJhyX6W12PkvLsti7i7dkcVFdQ0= 2023-10-18T20:30:55.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dJhyX6W12PkvLsti7i7dkcVFdQ0=', 'Content-Length': '0'} 2023-10-18T20:30:55.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746e1a1fff33aceb-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.687 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.687 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.688 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.688 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.689 DEBUG:boto:Method: GET 2023-10-18T20:30:55.689 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.690 DEBUG:boto:Data: 2023-10-18T20:30:55.690 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.691 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.691 DEBUG:boto:Params: {} 2023-10-18T20:30:55.692 DEBUG:boto:Token: None 2023-10-18T20:30:55.692 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.693 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9620c8d42c7db57-006530407f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.696 DEBUG:boto:b'bofrnf-58obj000000000000000004231000falseobj000000000000000004232023-10-18T20:30:30.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.697 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.697 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.698 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.698 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.699 DEBUG:boto:Method: GET 2023-10-18T20:30:55.699 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.700 DEBUG:boto:Data: 2023-10-18T20:30:55.700 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.701 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.701 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.701 DEBUG:boto:Params: {} 2023-10-18T20:30:55.702 DEBUG:boto:Token: None 2023-10-18T20:30:55.702 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.703 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000254cf5a5a9f35be5-006530407f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.708 DEBUG:boto:b'bofrnf-58obj000000000000000004231000falseobj000000000000000004232023-10-18T20:30:30.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.709 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.709 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.710 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.710 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.711 DEBUG:boto:Data: 2023-10-18T20:30:55.711 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.711 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.712 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.712 DEBUG:boto:Params: {} 2023-10-18T20:30:55.712 DEBUG:boto:Token: None 2023-10-18T20:30:55.713 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.713 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AabEFdN7lbvG373eMInFTgRFs7s= 2023-10-18T20:30:55.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AabEFdN7lbvG373eMInFTgRFs7s=', 'Content-Length': '0'} 2023-10-18T20:30:55.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000698a527eca344890-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.716 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.717 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.717 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.717 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.718 DEBUG:boto:Data: 2023-10-18T20:30:55.718 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.718 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.719 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.719 DEBUG:boto:Params: {} 2023-10-18T20:30:55.720 DEBUG:boto:Token: None 2023-10-18T20:30:55.720 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AabEFdN7lbvG373eMInFTgRFs7s= 2023-10-18T20:30:55.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AabEFdN7lbvG373eMInFTgRFs7s=', 'Content-Length': '0'} 2023-10-18T20:30:55.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d1644578d698b65-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-18T20:30:55.724 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.724 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.725 DEBUG:boto:Method: GET 2023-10-18T20:30:55.725 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.726 DEBUG:boto:Data: 2023-10-18T20:30:55.726 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.726 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.727 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.727 DEBUG:boto:Params: {} 2023-10-18T20:30:55.728 DEBUG:boto:Token: None 2023-10-18T20:30:55.728 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C/P6rjZhVfrxdcXNqcbGsnfOnBU= 2023-10-18T20:30:55.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C/P6rjZhVfrxdcXNqcbGsnfOnBU=', 'Content-Length': '0'} 2023-10-18T20:30:55.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f5fc790d4fafdc-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.732 DEBUG:boto:path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.732 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.732 DEBUG:boto:Method: GET 2023-10-18T20:30:55.733 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.733 DEBUG:boto:Data: 2023-10-18T20:30:55.734 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.734 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.734 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.735 DEBUG:boto:Params: {} 2023-10-18T20:30:55.735 DEBUG:boto:Token: None 2023-10-18T20:30:55.736 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000423 2023-10-18T20:30:55.736 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C/P6rjZhVfrxdcXNqcbGsnfOnBU= 2023-10-18T20:30:55.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C/P6rjZhVfrxdcXNqcbGsnfOnBU=', 'Content-Length': '0'} 2023-10-18T20:30:55.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d11210feb0a53d-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.786 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.787 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.787 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.788 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.788 DEBUG:boto:Method: GET 2023-10-18T20:30:55.789 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.789 DEBUG:boto:Data: 2023-10-18T20:30:55.790 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.790 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.791 DEBUG:boto:Params: {} 2023-10-18T20:30:55.791 DEBUG:boto:Token: None 2023-10-18T20:30:55.792 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d657ed45f3d8d495-006530407f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.796 DEBUG:boto:b'bofrnf-58obj000000000000000004231000falseobj000000000000000004232023-10-18T20:30:30.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.797 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.797 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.798 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.798 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.799 DEBUG:boto:Method: GET 2023-10-18T20:30:55.799 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000423 2023-10-18T20:30:55.799 DEBUG:boto:Data: 2023-10-18T20:30:55.800 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.800 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.801 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.801 DEBUG:boto:Params: {} 2023-10-18T20:30:55.802 DEBUG:boto:Token: None 2023-10-18T20:30:55.802 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.803 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca2b6678a611608-006530407f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.808 DEBUG:boto:b'bofrnf-58obj000000000000000004231000falseobj000000000000000004232023-10-18T20:30:30.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-10-18T20:30:55.809 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.810 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.810 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.811 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.811 DEBUG:boto:Method: GET 2023-10-18T20:30:55.811 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.812 DEBUG:boto:Data: 2023-10-18T20:30:55.812 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.813 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.813 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.813 DEBUG:boto:Params: {} 2023-10-18T20:30:55.814 DEBUG:boto:Token: None 2023-10-18T20:30:55.815 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.815 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ai2KouaCKNbUKicOu7In9Y64R4Q= 2023-10-18T20:30:55.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ai2KouaCKNbUKicOu7In9Y64R4Q=', 'Content-Length': '0'} 2023-10-18T20:30:55.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d6c73210068b96-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.862 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.863 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.863 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.864 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.864 DEBUG:boto:Method: GET 2023-10-18T20:30:55.865 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.865 DEBUG:boto:Data: 2023-10-18T20:30:55.866 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.866 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.867 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.867 DEBUG:boto:Params: {} 2023-10-18T20:30:55.868 DEBUG:boto:Token: None 2023-10-18T20:30:55.868 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.869 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ai2KouaCKNbUKicOu7In9Y64R4Q= 2023-10-18T20:30:55.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ai2KouaCKNbUKicOu7In9Y64R4Q=', 'Content-Length': '0'} 2023-10-18T20:30:55.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d72596606ce6fee-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.872 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.872 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.873 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.873 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.874 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.874 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.874 DEBUG:boto:Data: 2023-10-18T20:30:55.875 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.876 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.876 DEBUG:boto:Params: {} 2023-10-18T20:30:55.876 DEBUG:boto:Token: None 2023-10-18T20:30:55.877 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.877 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2G35670GU+Ewhu81fHsGY5GOwhs= 2023-10-18T20:30:55.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2G35670GU+Ewhu81fHsGY5GOwhs=', 'Content-Length': '0'} 2023-10-18T20:30:55.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044882971fc929dd8-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.881 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.881 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.881 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.882 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.882 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.883 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.883 DEBUG:boto:Data: 2023-10-18T20:30:55.883 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.884 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.884 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.885 DEBUG:boto:Params: {} 2023-10-18T20:30:55.885 DEBUG:boto:Token: None 2023-10-18T20:30:55.886 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.886 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2G35670GU+Ewhu81fHsGY5GOwhs= 2023-10-18T20:30:55.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2G35670GU+Ewhu81fHsGY5GOwhs=', 'Content-Length': '0'} 2023-10-18T20:30:55.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e315cdf5ccb5e487-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-10-18T20:30:55.890 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.890 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.891 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.891 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.891 DEBUG:boto:Method: GET 2023-10-18T20:30:55.892 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.892 DEBUG:boto:Data: 2023-10-18T20:30:55.893 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.893 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.893 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.894 DEBUG:boto:Params: {} 2023-10-18T20:30:55.894 DEBUG:boto:Token: None 2023-10-18T20:30:55.895 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.895 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ai2KouaCKNbUKicOu7In9Y64R4Q= 2023-10-18T20:30:55.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ai2KouaCKNbUKicOu7In9Y64R4Q=', 'Content-Length': '0'} 2023-10-18T20:30:55.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d7fa5d418fa150-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.899 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.899 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.900 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.900 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.900 DEBUG:boto:Method: GET 2023-10-18T20:30:55.901 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.901 DEBUG:boto:Data: 2023-10-18T20:30:55.902 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.902 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.903 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.903 DEBUG:boto:Params: {} 2023-10-18T20:30:55.903 DEBUG:boto:Token: None 2023-10-18T20:30:55.904 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000467?versionId=null 2023-10-18T20:30:55.904 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ai2KouaCKNbUKicOu7In9Y64R4Q= 2023-10-18T20:30:55.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ai2KouaCKNbUKicOu7In9Y64R4Q=', 'Content-Length': '0'} 2023-10-18T20:30:55.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000909c75afdd7574c0-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.908 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.908 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.909 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:55.909 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:55.910 DEBUG:boto:Method: GET 2023-10-18T20:30:55.910 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:55.911 DEBUG:boto:Data: 2023-10-18T20:30:55.911 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.911 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.912 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.912 DEBUG:boto:Params: {} 2023-10-18T20:30:55.913 DEBUG:boto:Token: None 2023-10-18T20:30:55.913 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.914 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e72deb82d2cd9a7c-006530407f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.917 DEBUG:boto:b'bofrnf-58obj000000000000000004671000falseobj000000000000000004672023-10-18T20:30:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.918 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:55.918 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:55.919 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:55.919 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:55.920 DEBUG:boto:Method: GET 2023-10-18T20:30:55.920 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:55.921 DEBUG:boto:Data: 2023-10-18T20:30:55.921 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.921 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.922 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.922 DEBUG:boto:Params: {} 2023-10-18T20:30:55.923 DEBUG:boto:Token: None 2023-10-18T20:30:55.923 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/ 2023-10-18T20:30:55.924 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4= 2023-10-18T20:30:55.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q4wTzG67LYHN5sQXEBa9A7jAI+4=', 'Content-Length': '0'} 2023-10-18T20:30:55.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f3f9ce235acb9ae-006530407f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.970 DEBUG:boto:b'bofrnf-58obj000000000000000004671000falseobj000000000000000004672023-10-18T20:30:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:55.971 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.972 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.972 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.972 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.973 DEBUG:boto:Data: 2023-10-18T20:30:55.973 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.974 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.975 DEBUG:boto:Params: {} 2023-10-18T20:30:55.975 DEBUG:boto:Token: None 2023-10-18T20:30:55.975 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.976 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i+yhVadPBpvHbzf5gnUwMYvWO8s= 2023-10-18T20:30:55.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i+yhVadPBpvHbzf5gnUwMYvWO8s=', 'Content-Length': '0'} 2023-10-18T20:30:55.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ee68c0d6866987-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.980 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.980 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.980 DEBUG:boto:Method: HEAD 2023-10-18T20:30:55.981 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.981 DEBUG:boto:Data: 2023-10-18T20:30:55.982 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.982 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:55.983 DEBUG:boto:Port: 8000 2023-10-18T20:30:55.983 DEBUG:boto:Params: {} 2023-10-18T20:30:55.985 DEBUG:boto:Token: None 2023-10-18T20:30:55.985 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.986 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i+yhVadPBpvHbzf5gnUwMYvWO8s= 2023-10-18T20:30:55.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i+yhVadPBpvHbzf5gnUwMYvWO8s=', 'Content-Length': '0'} 2023-10-18T20:30:55.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3136a15f2f313f-006530407f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-10-18T20:30:55.990 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.990 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.991 DEBUG:boto:Method: GET 2023-10-18T20:30:55.991 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.992 DEBUG:boto:Data: 2023-10-18T20:30:55.992 DEBUG:boto:Headers: {} 2023-10-18T20:30:55.992 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:55.993 DEBUG:boto:Port: 8001 2023-10-18T20:30:55.993 DEBUG:boto:Params: {} 2023-10-18T20:30:55.994 DEBUG:boto:Token: None 2023-10-18T20:30:55.994 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:55 GMT /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.995 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vcXPSKUoBxPSAQOG7C7EQlbtwhc= 2023-10-18T20:30:55.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vcXPSKUoBxPSAQOG7C7EQlbtwhc=', 'Content-Length': '0'} 2023-10-18T20:30:55.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-58:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab77f82774acaefd-006530407f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:55.998 DEBUG:boto:path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.999 DEBUG:boto:auth_path=/bofrnf-58/obj00000000000000000467 2023-10-18T20:30:55.999 DEBUG:boto:Method: GET 2023-10-18T20:30:56.000 DEBUG:boto:Path: /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:56.000 DEBUG:boto:Data: 2023-10-18T20:30:56.000 DEBUG:boto:Headers: {} 2023-10-18T20:30:56.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:56.001 DEBUG:boto:Port: 8000 2023-10-18T20:30:56.002 DEBUG:boto:Params: {} 2023-10-18T20:30:56.002 DEBUG:boto:Token: None 2023-10-18T20:30:56.003 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:56 GMT /bofrnf-58/obj00000000000000000467 2023-10-18T20:30:56.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a2tkZ6vMeoPkkEncHeBG6I1dRLo= 2023-10-18T20:30:56.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a2tkZ6vMeoPkkEncHeBG6I1dRLo=', 'Content-Length': '0'} 2023-10-18T20:30:56.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:30:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a916bd1383565cc4-0065304080-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:30:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:56.006 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:56.007 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:56.007 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:56.008 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:56.008 DEBUG:boto:Method: GET 2023-10-18T20:30:56.009 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:56.009 DEBUG:boto:Data: 2023-10-18T20:30:56.009 DEBUG:boto:Headers: {} 2023-10-18T20:30:56.010 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:30:56.010 DEBUG:boto:Port: 8001 2023-10-18T20:30:56.011 DEBUG:boto:Params: {} 2023-10-18T20:30:56.011 DEBUG:boto:Token: None 2023-10-18T20:30:56.012 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:56 GMT /bofrnf-58/ 2023-10-18T20:30:56.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M2W4RnDeGIb19bMHX2zPtWOeGFc= 2023-10-18T20:30:56.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M2W4RnDeGIb19bMHX2zPtWOeGFc=', 'Content-Length': '0'} 2023-10-18T20:30:56.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098d16dff57d5f2a6-0065304080-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:56.016 DEBUG:boto:b'bofrnf-58obj000000000000000004671000falseobj000000000000000004672023-10-18T20:30:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:56.017 DEBUG:boto:path=/bofrnf-58/ 2023-10-18T20:30:56.017 DEBUG:boto:auth_path=/bofrnf-58/ 2023-10-18T20:30:56.018 DEBUG:boto:path=/bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:56.018 DEBUG:boto:auth_path=/bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:56.019 DEBUG:boto:Method: GET 2023-10-18T20:30:56.019 DEBUG:boto:Path: /bofrnf-58/?prefix=obj00000000000000000467 2023-10-18T20:30:56.019 DEBUG:boto:Data: 2023-10-18T20:30:56.020 DEBUG:boto:Headers: {} 2023-10-18T20:30:56.020 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:56.021 DEBUG:boto:Port: 8000 2023-10-18T20:30:56.021 DEBUG:boto:Params: {} 2023-10-18T20:30:56.022 DEBUG:boto:Token: None 2023-10-18T20:30:56.022 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:30:56 GMT /bofrnf-58/ 2023-10-18T20:30:56.023 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M2W4RnDeGIb19bMHX2zPtWOeGFc= 2023-10-18T20:30:56.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M2W4RnDeGIb19bMHX2zPtWOeGFc=', 'Content-Length': '0'} 2023-10-18T20:30:56.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007996b020c1b19180-0065304080-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:30:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:56.027 DEBUG:boto:b'bofrnf-58obj000000000000000004671000falseobj000000000000000004672023-10-18T20:30:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:30:56.028 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-58 zones={a2, a1} 2023-10-18T20:30:56.029 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2023-10-18T20:30:56.032 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:30:56.032 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:30:56.033 DEBUG:boto:Method: PUT 2023-10-18T20:30:56.033 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:30:56.033 DEBUG:boto:Data: 2023-10-18T20:30:56.034 DEBUG:boto:Headers: {} 2023-10-18T20:30:56.034 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:30:56.034 DEBUG:boto:Port: 8000 2023-10-18T20:30:56.035 DEBUG:boto:Params: {} 2023-10-18T20:30:56.035 DEBUG:boto:Token: None 2023-10-18T20:30:56.036 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:30:56 GMT /bofrnf-59/ 2023-10-18T20:30:56.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UViEqlSRU/qKhPwE+hBfVLyr0Gk= 2023-10-18T20:30:56.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:30:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UViEqlSRU/qKhPwE+hBfVLyr0Gk=', 'Content-Length': '0'} 2023-10-18T20:30:56.055 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fab3a97ce7adc86c-0065304080-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:30:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:30:56.055 DEBUG:rgw_multi.tests:created bucket=bofrnf-59 2023-10-18T20:30:56.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:56.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:56.057 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:56.111 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:56.111 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:56.144 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:30:56.157 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:30:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661056.052606_1089.1", 2023-10-18T20:30:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:30:56.052606Z" 2023-10-18T20:30:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:30:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T20:30:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:30:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:30:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:56.161 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:30:56.162 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:56.163 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:30:56.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:56.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:30:56.164 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:30:56.213 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:30:56.214 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:30:56.241 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:56.240+0000 7f543c18a780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:30:56.242 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:30:56.241+0000 7f543c18a780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:30:56.254 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:30:56.255 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:30:56.255 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:30:56.255 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:30:56.256 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:30:56.256 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:30:56.256 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:56.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:56.256 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:30:56.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:56.257 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:30:56.257 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:56.257 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:56.258 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:30:56.258 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:56.258 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:30:56.258 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:56.258 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:30:56.259 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:56.259 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:56.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:56.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:56.260 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:30:56.260 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:56.260 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:56.260 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:30:56.260 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:56.261 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:56.261 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:56.261 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:30:56.261 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:56.261 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:56.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:56.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:56.262 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:30:56.262 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:56.263 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:56.263 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:56.263 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:56.263 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:56.263 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:56.264 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:30:56.264 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:56.264 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:56.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:56.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:30:56.265 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:30:56.265 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:30:56.265 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:30:56.265 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:30:56.266 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:30:56.266 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:30:56.266 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:30:56.266 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:30:56.266 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:30:56.267 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:56.267 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:30:56.267 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:30:56.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:56.268 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:30:56.268 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:30:56.268 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:30:56.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:30:56.268 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:30:56Z" 2023-10-18T20:30:56.268 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:30:56.270 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:30:56Z" } 2023-10-18T20:30:56.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:30:56.271 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:56.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:30:56.271 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:01.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:01.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:01.274 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:01.331 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:01.331 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:01.363 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:01.361+0000 7f073b141780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:01.363 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:01.362+0000 7f073b141780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:01.377 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:01.377 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:01.378 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:01.378 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:01.378 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:01.378 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:01.378 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:01.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:01.379 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:01.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:01.379 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:01.380 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:01.380 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:01.380 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:01.380 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:01.380 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:01.381 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:01.381 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:01.381 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:01.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:01.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:01.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:01.382 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:01.382 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:01.382 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:01.383 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:01.383 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:01.383 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:01.383 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:01.383 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:01.384 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:01.384 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:01.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:01.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:01.384 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:01.385 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:01.385 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:01.385 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:01.385 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:01.386 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:01.386 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:01.386 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:01.386 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:01.386 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:01.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:01.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:01.387 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:01.387 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:01.388 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:01.388 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:01.388 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:01.388 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:01.388 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:01.389 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:01.389 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:01.389 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:01.389 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:01.389 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:01.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:01.390 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:01.390 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:01.390 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:01.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:01.391 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:01Z" 2023-10-18T20:31:01.391 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:01.392 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:01Z" } 2023-10-18T20:31:01.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:01.393 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:01.393 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:01.394 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:06.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:06.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:06.397 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:06.453 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:06.454 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:06.486 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:06.484+0000 7f79ff0de780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:06.486 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:06.485+0000 7f79ff0de780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:06.500 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:06Z" 2023-10-18T20:31:06.514 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:06.515 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:06Z" } 2023-10-18T20:31:06.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:06.516 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:06.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:06.517 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:11.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:11.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:11.519 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:11.577 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:11.578 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:11.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:11.608+0000 7fa7a1ff8780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:11.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:11.609+0000 7fa7a1ff8780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:11.624 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:11.624 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:11.624 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:11.625 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:11.625 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:11.625 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:11.625 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:11.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:11.626 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:11.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:11.626 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:11.626 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:11.627 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:11.627 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:11.627 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:11.627 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:11.627 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:11.628 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:11.628 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:11.628 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:11.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:11.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:11.629 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:11.629 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:11.629 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:11.630 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:11.630 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:11.630 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:11.631 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:11.631 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:11.632 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:11.632 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:11.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:11.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:11.633 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:11.634 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:11.634 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:11.635 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:11.635 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:11.635 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:11.636 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:11.636 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:11.636 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:11.637 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:11.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:11.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:11.638 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:11.638 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:11.638 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:11.639 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:11.639 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:11.639 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:11.640 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:11.640 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:11.640 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:11.640 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:11.641 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:11.641 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:11.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:11.641 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:11.641 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:11.642 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:11.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:11.642 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:11Z" 2023-10-18T20:31:11.642 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:11.644 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:11Z" } 2023-10-18T20:31:11.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:11.645 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:11.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:11.646 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:16.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:16.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:16.649 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:16.703 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:16.703 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:16.733 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:16.732+0000 7fc6fdf1b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:16.734 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:16.733+0000 7fc6fdf1b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:16.747 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:16.747 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:16.748 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:16.748 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:16.748 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:16.748 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:16.749 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:16.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:16.749 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:16.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:16.749 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:16.750 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:16.750 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:16.750 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:16.750 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:16.750 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:16.751 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:16.751 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:16.751 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:16.751 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:16.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:16.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:16.752 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:16.752 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:16.752 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:16.753 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:16.753 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:16.753 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:16.753 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:16.753 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:16.754 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:16.754 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:16.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:16.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:16.754 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:16.755 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:16.755 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:16.755 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:16.755 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:16.756 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:16.756 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:16.756 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:16.756 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:16.756 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:16.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:16.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:16.757 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:16.757 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:16.757 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:16.758 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:16.758 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:16.758 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:16.758 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:16.759 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:16.759 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:16.759 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:16.759 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:16.759 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:16.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:16.760 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:16.760 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:16.760 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:16.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:16.760 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:16Z" 2023-10-18T20:31:16.761 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:16.762 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:16Z" } 2023-10-18T20:31:16.762 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:16.763 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:16.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:16.764 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:21.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:21.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:21.766 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:21.823 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:21.824 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:21.855 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:21.853+0000 7f7732dd4780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:21.856 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:21.854+0000 7f7732dd4780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:21.869 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:21.870 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:21.870 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:21.870 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:21.871 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:21.871 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:21.871 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:21.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:21.871 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:21.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:21.872 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:21.872 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:21.872 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:21.872 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:21.873 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:21.873 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:21.873 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:21.873 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:21.873 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:21.874 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:21.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:21.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:21.874 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:21.875 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:21.875 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:21.875 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:21.875 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:21.875 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:21.876 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:21.876 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:21.876 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:21.876 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:21.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:21.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:21.877 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:21.877 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:21.877 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:21.877 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:21.878 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:21.878 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:21.878 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:21.878 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:21.879 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:21.879 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:21.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:21.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:21.879 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:21.880 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:21.880 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:21.880 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:21.880 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:21.880 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:21.881 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:21.881 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:21.881 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:21.881 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:21.881 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:21.882 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:21.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:21.882 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:21.882 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:21.882 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:21.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:21.883 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:21Z" 2023-10-18T20:31:21.883 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:21.884 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:21Z" } 2023-10-18T20:31:21.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:21.885 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:21.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:21.886 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:26.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:26.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:26.888 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:26.945 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:26.946 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:26.978 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:26.975+0000 7f4de3742780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:26.978 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:26.976+0000 7f4de3742780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:26.991 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:26.992 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:26.992 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:26.992 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:26.992 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:26.993 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:26.993 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:26.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:26.993 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:26.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:26.994 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:26.994 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:26.994 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:26.994 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:26.994 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:26.995 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:26.995 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:26.995 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:26.995 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:26.995 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:26.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:26.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:26.996 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:26.996 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:26.996 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:26.997 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:26.997 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:26.997 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:26.997 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:26.997 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:26.997 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:26.998 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:26.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:26.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:26.998 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:26.999 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:26.999 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:26.999 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:26.999 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:26.999 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:27.000 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:27.000 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:27.000 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:27.000 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:27.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:27.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:27.001 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:27.001 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:27.001 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:27.001 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:27.001 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:27.002 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:27.002 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:27.002 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:27.002 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:27.003 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:27.003 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:27.003 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:27.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:27.003 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:27.004 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:27.004 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:27.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:27.004 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:26Z" 2023-10-18T20:31:27.004 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:27.005 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:26Z" } 2023-10-18T20:31:27.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:27.006 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:27.007 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:27.007 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:32.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:32.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:32.009 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:32.067 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:32.067 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:32.099 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:32.098+0000 7f5f32e34780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:32.100 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:32.099+0000 7f5f32e34780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:32.113 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:32Z" 2023-10-18T20:31:32.128 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:32.129 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:32Z" } 2023-10-18T20:31:32.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:32.130 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:32.130 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:32.131 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:37.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:37.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:37.133 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:37.190 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:37.190 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:37.221 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:37.219+0000 7f6526c73780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:37.221 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:37.220+0000 7f6526c73780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:37.235 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:37Z" 2023-10-18T20:31:37.249 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:37.250 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:37Z" } 2023-10-18T20:31:37.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:37.251 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:37.252 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:37.252 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:42.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:42.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:42.255 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:42.312 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:42.312 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:42.344 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:42.343+0000 7f83cea55780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:42.345 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:42.344+0000 7f83cea55780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:42.358 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:42.359 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:42.360 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:42.361 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:42.365 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:42.366 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:42.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:42.368 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:42.369 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:42.370 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:42Z" 2023-10-18T20:31:42.372 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:42.373 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:42Z" } 2023-10-18T20:31:42.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:42.374 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:42.374 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:42.375 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:47.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:47.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:47.376 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:47.435 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:47.435 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:47.465 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:47.464+0000 7f0dcecbb780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:47.466 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:47.465+0000 7f0dcecbb780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:47.481 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:47.481 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:47.481 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:47.481 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:47.482 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:47.482 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:47.482 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:47.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:47.483 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:47.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:47.483 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:47.483 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:47.483 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:47.484 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:47.484 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:47.484 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:47.484 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:47.484 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:47.485 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:47.485 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:47.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:47.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:47.485 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:47.486 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:47.486 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:47.486 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:47.486 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:47.487 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:47.487 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:47.487 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:47.487 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:47.487 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:47.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:47.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:47.488 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:47.488 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:47.488 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:47.489 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:47.489 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:47.489 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:47.489 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:47.490 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:47.490 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:47.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:47.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:47.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:47.491 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:47.491 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:47.491 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:47.491 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:47.492 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:47.492 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:47.492 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:47.492 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:47.492 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:47.493 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:47.493 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:47.493 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:47.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:47.494 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:47.494 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:47.494 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:47.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:47.494 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:47Z" 2023-10-18T20:31:47.495 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:47.496 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:47Z" } 2023-10-18T20:31:47.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:47.497 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:47.497 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:47.497 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:52.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:52.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:52.500 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:52.556 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:52.556 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:52.587 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:52.586+0000 7fe2648ff780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:52.588 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:52.587+0000 7fe2648ff780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:52.602 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:52.602 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:52.602 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:52.603 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:52.603 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:52.603 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:52.603 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:52.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:52.604 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:52.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:52.604 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:52.604 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:52.604 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:52.605 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:52.605 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:52.605 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:52.605 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:52.606 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:52.606 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:52.606 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:52.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:52.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:52.607 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:52.607 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:52.607 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:52.607 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:52.608 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:52.608 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:52.608 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:52.608 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:52.609 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:52.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:52.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:52.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:52.610 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:52.610 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:52.610 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:52.610 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:52.611 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:52.611 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:52.611 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:52.611 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:52.611 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:52.612 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:52.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:52.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:52.612 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:52.612 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:52.613 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:52.613 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:52.613 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:52.614 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:52.614 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:52.614 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:52.614 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:52.614 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:52.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:52.615 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:52.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:52.615 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:52.616 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:52.616 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:52.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:52.616 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:52Z" 2023-10-18T20:31:52.616 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:52.617 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:52Z" } 2023-10-18T20:31:52.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:52.618 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:52.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:52.619 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:31:57.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:57.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:31:57.622 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:31:57.680 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:31:57.680 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:31:57.711 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:57.710+0000 7f9f5683f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:57.711 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:31:57.711+0000 7f9f5683f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:31:57.725 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:31:57.726 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:31:57.726 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:31:57.726 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:31:57.726 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:31:57.727 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:31:57.727 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:57.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:57.727 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:31:57.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:57.728 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:31:57.728 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:57.728 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:57.728 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:31:57.729 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:57.729 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:31:57.729 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:57.729 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:31:57.729 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:57.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:57.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:57.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:57.731 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:31:57.731 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:57.731 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:57.731 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:31:57.731 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:57.732 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:57.732 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:57.732 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:31:57.732 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:57.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:57.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:57.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:57.733 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:31:57.733 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:57.734 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:57.734 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:57.734 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:57.734 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:57.734 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:57.735 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:57.735 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:57.735 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:57.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:57.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:31:57.736 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:31:57.736 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:31:57.736 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:31:57.736 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:31:57.736 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:31:57.737 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:31:57.737 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:31:57.737 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:31:57.737 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:31:57.737 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:57.738 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:31:57.738 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:31:57.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:57.738 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:31:57.738 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:31:57.739 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:31:57.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:31:57.739 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:31:57Z" 2023-10-18T20:31:57.739 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:31:57.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:31:57Z" } 2023-10-18T20:31:57.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:31:57.741 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:57.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:31:57.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:32:02.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:02.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:02.744 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:32:02.802 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:32:02.802 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:32:02.834 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:02.833+0000 7f65e8120780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:02.835 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:02.834+0000 7f65e8120780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:02.848 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:32:02.849 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:32:02.849 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:32:02.849 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:32:02.850 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:32:02.850 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:32:02.850 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:02.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:02.850 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:32:02.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:02.851 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:32:02.851 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:02.851 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:02.852 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:32:02.852 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:02.852 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:32:02.853 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:02.853 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:32:02.853 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:02.853 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:02.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:02.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:02.854 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:32:02.854 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:02.854 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:02.855 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:32:02.855 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:02.855 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:02.855 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:02.855 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:32:02.855 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:02.856 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:02.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:02.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:02.856 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:32:02.857 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:02.857 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:02.857 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:02.857 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:02.857 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:02.858 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:02.858 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:02.858 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:02.858 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:02.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:02.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:02.859 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:32:02.859 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:02.859 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:02.859 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:02.860 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:02.860 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:02.860 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:02.860 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:02.860 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:02.861 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:02.861 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:02.861 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:32:02.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:02.861 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:32:02.862 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:32:02.862 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:32:02.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:02.862 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:32:02Z" 2023-10-18T20:32:02.862 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:32:02.863 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:32:02Z" } 2023-10-18T20:32:02.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:32:02.864 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:02.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:02.865 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:32:07.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:07.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:07.867 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:32:07.925 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:32:07.925 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:32:07.956 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:07.955+0000 7f5fd1732780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:07.957 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:07.956+0000 7f5fd1732780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:07.971 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:32:07.971 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:32:07.971 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:32:07.971 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:32:07.972 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:32:07.972 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:32:07.972 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:07.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:07.972 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:32:07.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:07.973 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:32:07.973 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:07.973 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:07.973 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:32:07.973 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:07.974 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:32:07.974 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:07.974 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:32:07.974 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:07.975 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:07.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:07.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:07.975 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:32:07.975 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:07.976 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:07.976 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:32:07.976 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:07.976 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:07.976 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:07.976 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:32:07.977 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:07.977 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:07.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:07.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:07.978 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:32:07.978 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:07.978 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:07.978 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:07.978 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:07.979 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:07.979 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:07.979 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:07.979 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:07.979 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:07.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:07.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:07.980 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:32:07.980 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:07.980 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:07.981 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:07.981 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:07.981 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:07.981 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:07.981 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:07.982 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:07.982 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:07.982 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:07.982 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:32:07.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:07.983 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:32:07.983 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:32:07.983 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:32:07.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:07.983 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:32:07Z" 2023-10-18T20:32:07.984 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:32:07.985 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:32:07Z" } 2023-10-18T20:32:07.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:32:07.986 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:07.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:07.987 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:32:12.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:12.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:12.989 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:32:13.047 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:32:13.047 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:32:13.079 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:13.077+0000 7f1f1636d780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:13.080 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:13.078+0000 7f1f1636d780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:13.093 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:32:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:32:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:32:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:32:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:32:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:32:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:32:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:32:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:32:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:32:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:32:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:32:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:13.099 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:32:13.099 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:13.099 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:13.099 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:13.099 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:32:13.100 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:13.100 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:13.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:13.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:13.100 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:32:13.101 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:13.101 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:13.101 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:13.101 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:13.101 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:13.102 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:13.102 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:13.102 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:13.102 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:13.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:13.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:13.103 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:32:13.103 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:13.103 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:13.104 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:13.104 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:13.104 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:13.104 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:13.104 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:13.105 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:13.105 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:13.105 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:13.105 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:32:13.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:13.106 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:32:13.106 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:32:13.106 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:32:13.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:13.106 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:32:13Z" 2023-10-18T20:32:13.107 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:32:13.107 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:32:13Z" } 2023-10-18T20:32:13.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:32:13.108 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:13.109 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:13.109 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:32:18.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:18.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:18.112 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:32:18.169 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:32:18.170 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:32:18.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:18.199+0000 7fefd1840780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:18.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:18.201+0000 7fefd1840780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:18.217 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:32:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:32:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:32:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:32:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:32:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:32:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:32:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:32:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:32:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:32:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:32:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:32:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:32:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:32:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:32:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:32:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:32:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:32:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:32:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:32:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:32:18Z" 2023-10-18T20:32:18.233 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:32:18.234 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:32:18Z" } 2023-10-18T20:32:18.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:32:18.236 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:18.236 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:18.236 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:32:23.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:23.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:23.239 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:32:23.296 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:32:23.297 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:32:23.328 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:23.326+0000 7f2cba470780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:23.329 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:23.327+0000 7f2cba470780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:23.342 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:32:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:32:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:32:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:32:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:32:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:32:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:32:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:32:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661001.581996_1019.1", 2023-10-18T20:32:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:32:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:01.581996Z", 2023-10-18T20:32:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:32:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:32:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:32:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:32:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:32:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:32:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:32:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:32:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:32:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:32:23Z" 2023-10-18T20:32:23.356 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:32:23.357 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661001.581996_1019.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:01.581996Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:32:23Z" } 2023-10-18T20:32:23.357 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661001.581996_1019.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:01.581996Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:32:23.358 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:23.359 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661001.581996_1019.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:23.359 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661001.581996_1019.1 2023-10-18T20:32:28.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:28.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:28.362 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:32:28.419 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:32:28.419 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:32:28.450 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:28.449+0000 7fe229dd3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:28.451 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:28.450+0000 7fe229dd3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:28.465 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:32:28.465 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:32:28.465 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:32:28.465 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:32:28.466 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:32:28.466 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:32:28.466 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:28.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:28.466 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:32:28.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:28.467 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:32:28.467 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:28.467 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:28.468 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661025.883298_1028.1", 2023-10-18T20:32:28.468 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:28.468 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:32:28.468 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:28.468 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:25.883298Z", 2023-10-18T20:32:28.469 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:28.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:28.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:28.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:28.469 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:32:28.470 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:28.470 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:28.470 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:32:28.470 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:28.471 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:28.471 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:28.471 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:32:28.471 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:28.471 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:28.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:28.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:28.472 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:32:28.472 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:28.472 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:28.473 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:28.473 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:28.473 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:28.473 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:28.473 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:28.474 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:28.474 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:28.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:28.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:28.475 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:32:28.475 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:28.475 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:28.475 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:28.475 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:28.476 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:28.476 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:28.476 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:28.476 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:28.476 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:28.477 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:28.477 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:32:28.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:28.477 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:32:28.477 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:32:28.478 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:32:28.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:28.478 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:32:28Z" 2023-10-18T20:32:28.478 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:32:28.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661025.883298_1028.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:25.883298Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:32:28Z" } 2023-10-18T20:32:28.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661025.883298_1028.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:25.883298Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:32:28.480 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:28.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661025.883298_1028.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:28.481 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697661056.052606_1089.1 target=1_1697661025.883298_1028.1 2023-10-18T20:32:33.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:33.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:32:33.484 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:32:33.542 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:32:33.542 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:32:33.573 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:33.571+0000 7fc14f05b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:33.573 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:33.572+0000 7fc14f05b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:32:33.587 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:32:33.588 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:32:33.588 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:32:33.588 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:32:33.588 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:32:33.588 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:32:33.589 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:33.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:33.589 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:32:33.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:33.590 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:32:33.590 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:33.590 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:33.591 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661056.052606_1089.1", 2023-10-18T20:32:33.591 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:33.591 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:32:33.591 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:33.591 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:30:56.052606Z", 2023-10-18T20:32:33.592 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:33.592 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:33.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:33.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:33.592 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:32:33.593 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:33.593 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:33.593 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:32:33.593 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:33.594 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:33.594 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:33.594 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:32:33.594 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:33.595 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:33.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:33.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:33.595 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:32:33.596 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:33.596 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:33.596 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:33.596 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:33.596 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:33.597 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:33.597 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:33.597 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:33.597 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:33.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:33.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:32:33.598 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:32:33.598 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:32:33.598 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:32:33.599 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:32:33.599 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:32:33.599 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:32:33.600 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:32:33.600 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:32:33.600 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:32:33.600 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:33.601 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:32:33.601 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:32:33.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:33.601 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:32:33.601 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:32:33.602 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:32:33.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:32:33.602 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:32:33Z" 2023-10-18T20:32:33.602 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:32:33.603 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661056.052606_1089.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:30:56.052606Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:32:33Z" } 2023-10-18T20:32:33.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661056.052606_1089.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:30:56.052606Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:32:33.604 DEBUG:rgw_multi.tests:log_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:33.605 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661056.052606_1089.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:32:33.605 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:32:33.606 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:32:33.606 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:32:33.607 DEBUG:boto:Method: HEAD 2023-10-18T20:32:33.607 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:32:33.607 DEBUG:boto:Data: 2023-10-18T20:32:33.608 DEBUG:boto:Headers: {} 2023-10-18T20:32:33.608 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.609 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.609 DEBUG:boto:Params: {} 2023-10-18T20:32:33.610 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:32:33.611 DEBUG:boto:Token: None 2023-10-18T20:32:33.611 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/ 2023-10-18T20:32:33.612 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE= 2023-10-18T20:32:33.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE=', 'Content-Length': '0'} 2023-10-18T20:32:33.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098774fc7a052cfbf-00653040e1-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.617 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:32:33.618 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:32:33.618 DEBUG:boto:Method: PUT 2023-10-18T20:32:33.618 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:32:33.619 DEBUG:boto:Data: 2023-10-18T20:32:33.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:32:33.620 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.620 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.620 DEBUG:boto:Params: {} 2023-10-18T20:32:33.621 DEBUG:boto:Token: None 2023-10-18T20:32:33.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/obj11 2023-10-18T20:32:33.622 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dA43kxLcnLVqN3dvc9mLeWXXfNY= 2023-10-18T20:32:33.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dA43kxLcnLVqN3dvc9mLeWXXfNY='} 2023-10-18T20:32:33.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fd247662d52275c-00653040e1-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.630 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:32:33.630 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:32:33.630 DEBUG:boto:Method: HEAD 2023-10-18T20:32:33.631 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:32:33.631 DEBUG:boto:Data: 2023-10-18T20:32:33.632 DEBUG:boto:Headers: {} 2023-10-18T20:32:33.632 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.633 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.633 DEBUG:boto:Params: {} 2023-10-18T20:32:33.633 DEBUG:boto:Token: None 2023-10-18T20:32:33.634 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/ 2023-10-18T20:32:33.635 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE= 2023-10-18T20:32:33.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE=', 'Content-Length': '0'} 2023-10-18T20:32:33.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f13653fc06b20466-00653040e1-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.638 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:32:33.639 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:32:33.639 DEBUG:boto:Method: PUT 2023-10-18T20:32:33.639 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:32:33.640 DEBUG:boto:Data: 2023-10-18T20:32:33.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:32:33.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.641 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.642 DEBUG:boto:Params: {} 2023-10-18T20:32:33.642 DEBUG:boto:Token: None 2023-10-18T20:32:33.643 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/obj22 2023-10-18T20:32:33.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GjFXCOVYHml30JN3am/zX8H1i/g= 2023-10-18T20:32:33.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GjFXCOVYHml30JN3am/zX8H1i/g='} 2023-10-18T20:32:33.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b0b627f04b2c948-00653040e1-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.648 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:32:33.648 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:32:33.649 DEBUG:boto:Method: HEAD 2023-10-18T20:32:33.649 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:32:33.650 DEBUG:boto:Data: 2023-10-18T20:32:33.650 DEBUG:boto:Headers: {} 2023-10-18T20:32:33.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.651 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.651 DEBUG:boto:Params: {} 2023-10-18T20:32:33.652 DEBUG:boto:Token: None 2023-10-18T20:32:33.653 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/ 2023-10-18T20:32:33.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE= 2023-10-18T20:32:33.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE=', 'Content-Length': '0'} 2023-10-18T20:32:33.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000252762c064eb74e7-00653040e1-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.657 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:32:33.657 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:32:33.658 DEBUG:boto:Method: PUT 2023-10-18T20:32:33.658 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:32:33.659 DEBUG:boto:Data: 2023-10-18T20:32:33.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:32:33.660 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.660 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.661 DEBUG:boto:Params: {} 2023-10-18T20:32:33.661 DEBUG:boto:Token: None 2023-10-18T20:32:33.662 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/obj33 2023-10-18T20:32:33.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:62gE2eWCCBK7P7kXMFzxZxi9SaU= 2023-10-18T20:32:33.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:62gE2eWCCBK7P7kXMFzxZxi9SaU='} 2023-10-18T20:32:33.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d4263850e1f8bc2-00653040e1-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.676 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:32:33.676 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:32:33.676 DEBUG:boto:Method: HEAD 2023-10-18T20:32:33.677 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:32:33.677 DEBUG:boto:Data: 2023-10-18T20:32:33.678 DEBUG:boto:Headers: {} 2023-10-18T20:32:33.678 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.678 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.679 DEBUG:boto:Params: {} 2023-10-18T20:32:33.679 DEBUG:boto:Token: None 2023-10-18T20:32:33.680 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/ 2023-10-18T20:32:33.680 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE= 2023-10-18T20:32:33.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE=', 'Content-Length': '0'} 2023-10-18T20:32:33.684 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdeebd5d91eca72f-00653040e1-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.685 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:32:33.685 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:32:33.686 DEBUG:boto:Method: PUT 2023-10-18T20:32:33.686 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:32:33.687 DEBUG:boto:Data: 2023-10-18T20:32:33.687 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:32:33.688 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.688 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.689 DEBUG:boto:Params: {} 2023-10-18T20:32:33.689 DEBUG:boto:Token: None 2023-10-18T20:32:33.689 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/obj44 2023-10-18T20:32:33.690 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5zKCneb0GmgLHhB9MhZ5vNnbeo0= 2023-10-18T20:32:33.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5zKCneb0GmgLHhB9MhZ5vNnbeo0='} 2023-10-18T20:32:33.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b7cb0172026f914-00653040e1-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.736 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:32:33.736 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:32:33.737 DEBUG:boto:Method: HEAD 2023-10-18T20:32:33.737 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:32:33.737 DEBUG:boto:Data: 2023-10-18T20:32:33.738 DEBUG:boto:Headers: {} 2023-10-18T20:32:33.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.739 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.739 DEBUG:boto:Params: {} 2023-10-18T20:32:33.740 DEBUG:boto:Token: None 2023-10-18T20:32:33.740 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/ 2023-10-18T20:32:33.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE= 2023-10-18T20:32:33.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9+LlGbIKfKBxaFglpwQ5YO34PiE=', 'Content-Length': '0'} 2023-10-18T20:32:33.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000272b5037fc9283a9-00653040e1-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.744 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:32:33.744 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:32:33.745 DEBUG:boto:Method: PUT 2023-10-18T20:32:33.745 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:32:33.746 DEBUG:boto:Data: 2023-10-18T20:32:33.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:32:33.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:32:33.747 DEBUG:boto:Port: 8000 2023-10-18T20:32:33.747 DEBUG:boto:Params: {} 2023-10-18T20:32:33.748 DEBUG:boto:Token: None 2023-10-18T20:32:33.748 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:32:33 GMT /bofrnf-59/obj55 2023-10-18T20:32:33.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UqcfUF4SJC8zCH0vCHY644eSakc= 2023-10-18T20:32:33.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:32:33 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UqcfUF4SJC8zCH0vCHY644eSakc='} 2023-10-18T20:32:33.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087af75eb0211ada6-00653040e1-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:32:33.754 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-59 2023-10-18T20:32:33.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:32:33.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:32:33.755 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:32:33.812 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:32:33.813 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:32:33.846 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:33.845+0000 7f5a9c1a9780 1 waiting to reach incremental sync.. 2023-10-18T20:32:38.847 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:38.845+0000 7f5a9c1a9780 1 waiting to reach incremental sync.. 2023-10-18T20:32:43.848 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:43.846+0000 7f5a9c1a9780 1 waiting to reach incremental sync.. 2023-10-18T20:32:48.848 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:48.847+0000 7f5a9c1a9780 1 waiting to reach incremental sync.. 2023-10-18T20:32:53.848 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:53.847+0000 7f5a9c1a9780 1 waiting to reach incremental sync.. 2023-10-18T20:32:58.851 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:58.849+0000 7f5a9c1a9780 1 bucket sync caught up with source: 2023-10-18T20:32:58.851 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.65.6, 00000000002.67.6, 00000000002.63.6] 2023-10-18T20:32:58.851 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.65.6, 00000000002.67.6, 00000000002.63.6] 2023-10-18T20:32:58.852 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:58.849+0000 7f5a9c1a9780 0 bucket checkpoint complete 2023-10-18T20:32:58.868 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-59 2023-10-18T20:32:58.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:32:58.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:32:58.870 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:32:58.931 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:32:58.932 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:32:58.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:32:58.967+0000 7fb8d0072780 1 waiting for incremental sync to catch up: 2023-10-18T20:32:58.969 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , 00000000002.436.6, 00000000002.210.6] 2023-10-18T20:32:58.969 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.212.6, 00000000002.436.6, 00000000002.210.6] 2023-10-18T20:33:03.971 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:03.968+0000 7fb8d0072780 1 bucket sync caught up with source: 2023-10-18T20:33:03.971 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.212.6, 00000000002.436.6, 00000000002.210.6] 2023-10-18T20:33:03.971 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.212.6, 00000000002.436.6, 00000000002.210.6] 2023-10-18T20:33:03.972 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:03.968+0000 7fb8d0072780 0 bucket checkpoint complete 2023-10-18T20:33:03.987 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:33:03.988 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:03.988 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:03.989 DEBUG:boto:Method: HEAD 2023-10-18T20:33:03.989 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:33:03.989 DEBUG:boto:Data: 2023-10-18T20:33:03.990 DEBUG:boto:Headers: {} 2023-10-18T20:33:03.990 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:03.991 DEBUG:boto:Port: 8001 2023-10-18T20:33:03.991 DEBUG:boto:Params: {} 2023-10-18T20:33:03.992 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:33:03.992 DEBUG:boto:Token: None 2023-10-18T20:33:03.993 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:03 GMT /bofrnf-59/ 2023-10-18T20:33:03.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KOsFB9T6L0OpH1jidDGKsSspUak= 2023-10-18T20:33:03.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KOsFB9T6L0OpH1jidDGKsSspUak=', 'Content-Length': '0'} 2023-10-18T20:33:03.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f808f4d047e14653-00653040ff-1af4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:33:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:03.998 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:03.999 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:03.999 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.000 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:33:04.000 DEBUG:boto:Data: 2023-10-18T20:33:04.000 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.001 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.001 DEBUG:boto:Params: {} 2023-10-18T20:33:04.002 DEBUG:boto:Token: None 2023-10-18T20:33:04.002 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AHyEuAXQACqo1+BgaI3byVwgSA0= 2023-10-18T20:33:04.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AHyEuAXQACqo1+BgaI3byVwgSA0=', 'Content-Length': '0'} 2023-10-18T20:33:04.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000884b1e45d636a7ee-0065304100-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.006 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:33:04.007 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.007 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.008 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.008 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.008 DEBUG:boto:Method: GET 2023-10-18T20:33:04.009 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.009 DEBUG:boto:Data: 2023-10-18T20:33:04.009 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.010 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.010 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.011 DEBUG:boto:Params: {} 2023-10-18T20:33:04.011 DEBUG:boto:Token: None 2023-10-18T20:33:04.011 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/?versions 2023-10-18T20:33:04.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:23/TVbhmVXZWyGQdm+3602vYSZs= 2023-10-18T20:33:04.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:23/TVbhmVXZWyGQdm+3602vYSZs=', 'Content-Length': '0'} 2023-10-18T20:33:04.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f48dd4194497f4a-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.015 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:33:04.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:33:04.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:33:04.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:33:04.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:33:04.018 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:33:04.019 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.019 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.020 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.020 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.020 DEBUG:boto:Method: GET 2023-10-18T20:33:04.021 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.021 DEBUG:boto:Data: 2023-10-18T20:33:04.022 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.022 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.023 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.023 DEBUG:boto:Params: {} 2023-10-18T20:33:04.023 DEBUG:boto:Token: None 2023-10-18T20:33:04.024 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/?versions 2023-10-18T20:33:04.024 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:23/TVbhmVXZWyGQdm+3602vYSZs= 2023-10-18T20:33:04.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:23/TVbhmVXZWyGQdm+3602vYSZs=', 'Content-Length': '0'} 2023-10-18T20:33:04.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006178adcfedd518b3-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.028 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:33:04.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:33:04.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:33:04.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:33:04.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:33:04.031 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.031 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.031 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.032 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.032 DEBUG:boto:Method: GET 2023-10-18T20:33:04.033 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.033 DEBUG:boto:Data: 2023-10-18T20:33:04.034 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.034 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.034 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.035 DEBUG:boto:Params: {} 2023-10-18T20:33:04.035 DEBUG:boto:Token: None 2023-10-18T20:33:04.036 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/?versions 2023-10-18T20:33:04.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:23/TVbhmVXZWyGQdm+3602vYSZs= 2023-10-18T20:33:04.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:23/TVbhmVXZWyGQdm+3602vYSZs=', 'Content-Length': '0'} 2023-10-18T20:33:04.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d27f5bfc04e56ff-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.040 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.041 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.041 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.041 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.042 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.042 DEBUG:boto:Method: GET 2023-10-18T20:33:04.043 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:33:04.043 DEBUG:boto:Data: 2023-10-18T20:33:04.043 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.044 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.044 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.045 DEBUG:boto:Params: {} 2023-10-18T20:33:04.045 DEBUG:boto:Token: None 2023-10-18T20:33:04.046 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/?versions 2023-10-18T20:33:04.046 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:23/TVbhmVXZWyGQdm+3602vYSZs= 2023-10-18T20:33:04.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:23/TVbhmVXZWyGQdm+3602vYSZs=', 'Content-Length': '0'} 2023-10-18T20:33:04.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000133e93aa95a7442c-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.050 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:33:04.051 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.051 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.052 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.052 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.052 DEBUG:boto:Method: GET 2023-10-18T20:33:04.053 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.053 DEBUG:boto:Data: 2023-10-18T20:33:04.054 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.054 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.055 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.055 DEBUG:boto:Params: {} 2023-10-18T20:33:04.055 DEBUG:boto:Token: None 2023-10-18T20:33:04.056 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.056 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eZaC/tDU/qiiU0e2c3TybSEZMMU= 2023-10-18T20:33:04.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eZaC/tDU/qiiU0e2c3TybSEZMMU=', 'Content-Length': '0'} 2023-10-18T20:33:04.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009257a9bea935ee37-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.059 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.060 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.060 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.061 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.061 DEBUG:boto:Method: GET 2023-10-18T20:33:04.061 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.062 DEBUG:boto:Data: 2023-10-18T20:33:04.062 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.063 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.063 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.063 DEBUG:boto:Params: {} 2023-10-18T20:33:04.064 DEBUG:boto:Token: None 2023-10-18T20:33:04.064 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.065 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eZaC/tDU/qiiU0e2c3TybSEZMMU= 2023-10-18T20:33:04.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eZaC/tDU/qiiU0e2c3TybSEZMMU=', 'Content-Length': '0'} 2023-10-18T20:33:04.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bca557cf23bfae0-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.114 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.116 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.116 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.116 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.117 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.117 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.118 DEBUG:boto:Data: 2023-10-18T20:33:04.118 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.118 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.119 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.119 DEBUG:boto:Params: {} 2023-10-18T20:33:04.120 DEBUG:boto:Token: None 2023-10-18T20:33:04.120 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.121 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xbTNMVD8osmEkNOw2HHXZPml8Xo= 2023-10-18T20:33:04.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xbTNMVD8osmEkNOw2HHXZPml8Xo=', 'Content-Length': '0'} 2023-10-18T20:33:04.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c356da4368ceb938-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.124 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.125 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.125 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.125 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.126 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.126 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.127 DEBUG:boto:Data: 2023-10-18T20:33:04.127 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.127 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.128 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.128 DEBUG:boto:Params: {} 2023-10-18T20:33:04.129 DEBUG:boto:Token: None 2023-10-18T20:33:04.129 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.130 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xbTNMVD8osmEkNOw2HHXZPml8Xo= 2023-10-18T20:33:04.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xbTNMVD8osmEkNOw2HHXZPml8Xo=', 'Content-Length': '0'} 2023-10-18T20:33:04.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b64ab5986c884fc1-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:33:04.134 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.134 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.135 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.135 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.135 DEBUG:boto:Method: GET 2023-10-18T20:33:04.136 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.136 DEBUG:boto:Data: 2023-10-18T20:33:04.137 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.137 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.137 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.138 DEBUG:boto:Params: {} 2023-10-18T20:33:04.138 DEBUG:boto:Token: None 2023-10-18T20:33:04.139 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.139 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eZaC/tDU/qiiU0e2c3TybSEZMMU= 2023-10-18T20:33:04.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eZaC/tDU/qiiU0e2c3TybSEZMMU=', 'Content-Length': '0'} 2023-10-18T20:33:04.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d6d6cd7a0546eaf-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.143 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.143 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.144 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.144 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.144 DEBUG:boto:Method: GET 2023-10-18T20:33:04.145 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.145 DEBUG:boto:Data: 2023-10-18T20:33:04.146 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.146 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.147 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.147 DEBUG:boto:Params: {} 2023-10-18T20:33:04.148 DEBUG:boto:Token: None 2023-10-18T20:33:04.148 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:33:04.149 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eZaC/tDU/qiiU0e2c3TybSEZMMU= 2023-10-18T20:33:04.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eZaC/tDU/qiiU0e2c3TybSEZMMU=', 'Content-Length': '0'} 2023-10-18T20:33:04.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ceb8988a3779e63-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.152 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.152 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.153 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.153 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.154 DEBUG:boto:Method: GET 2023-10-18T20:33:04.154 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.154 DEBUG:boto:Data: 2023-10-18T20:33:04.155 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.155 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.156 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.156 DEBUG:boto:Params: {} 2023-10-18T20:33:04.157 DEBUG:boto:Token: None 2023-10-18T20:33:04.157 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.158 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d0b7bd6e5b2893e-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.202 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.203 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.204 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.204 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.205 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.205 DEBUG:boto:Method: GET 2023-10-18T20:33:04.206 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.206 DEBUG:boto:Data: 2023-10-18T20:33:04.207 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.208 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.208 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.209 DEBUG:boto:Params: {} 2023-10-18T20:33:04.209 DEBUG:boto:Token: None 2023-10-18T20:33:04.210 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.210 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6af7d50fbdd6494-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.214 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.215 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.215 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.215 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.216 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:33:04.216 DEBUG:boto:Data: 2023-10-18T20:33:04.217 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.217 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.217 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.218 DEBUG:boto:Params: {} 2023-10-18T20:33:04.218 DEBUG:boto:Token: None 2023-10-18T20:33:04.219 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11 2023-10-18T20:33:04.219 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1C+/mgs8lcOY4ayvmXM5krc7os0= 2023-10-18T20:33:04.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1C+/mgs8lcOY4ayvmXM5krc7os0=', 'Content-Length': '0'} 2023-10-18T20:33:04.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d70fa01d2976ce1d-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.222 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.223 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.223 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.223 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:33:04.224 DEBUG:boto:Data: 2023-10-18T20:33:04.224 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.225 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.225 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.225 DEBUG:boto:Params: {} 2023-10-18T20:33:04.226 DEBUG:boto:Token: None 2023-10-18T20:33:04.226 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11 2023-10-18T20:33:04.227 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1C+/mgs8lcOY4ayvmXM5krc7os0= 2023-10-18T20:33:04.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1C+/mgs8lcOY4ayvmXM5krc7os0=', 'Content-Length': '0'} 2023-10-18T20:33:04.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad63e315bc1663c1-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:33:04.230 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.231 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.231 DEBUG:boto:Method: GET 2023-10-18T20:33:04.231 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:33:04.232 DEBUG:boto:Data: 2023-10-18T20:33:04.232 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.233 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.233 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.234 DEBUG:boto:Params: {} 2023-10-18T20:33:04.234 DEBUG:boto:Token: None 2023-10-18T20:33:04.235 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11 2023-10-18T20:33:04.235 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:31oxeH6iW9ROphAkwZ3WN+msIRU= 2023-10-18T20:33:04.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:31oxeH6iW9ROphAkwZ3WN+msIRU=', 'Content-Length': '0'} 2023-10-18T20:33:04.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f62202388d18150-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.238 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:33:04.238 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:33:04.239 DEBUG:boto:Method: GET 2023-10-18T20:33:04.239 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:33:04.240 DEBUG:boto:Data: 2023-10-18T20:33:04.240 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.240 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.241 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.241 DEBUG:boto:Params: {} 2023-10-18T20:33:04.242 DEBUG:boto:Token: None 2023-10-18T20:33:04.242 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj11 2023-10-18T20:33:04.243 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:31oxeH6iW9ROphAkwZ3WN+msIRU= 2023-10-18T20:33:04.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:31oxeH6iW9ROphAkwZ3WN+msIRU=', 'Content-Length': '0'} 2023-10-18T20:33:04.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006836686077254088-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.246 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.246 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.247 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.247 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.248 DEBUG:boto:Method: GET 2023-10-18T20:33:04.248 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.248 DEBUG:boto:Data: 2023-10-18T20:33:04.249 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.249 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.250 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.250 DEBUG:boto:Params: {} 2023-10-18T20:33:04.251 DEBUG:boto:Token: None 2023-10-18T20:33:04.251 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000966d4374651c8304-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.255 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.256 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.256 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.257 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.257 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.258 DEBUG:boto:Method: GET 2023-10-18T20:33:04.258 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:33:04.259 DEBUG:boto:Data: 2023-10-18T20:33:04.259 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.259 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.260 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.260 DEBUG:boto:Params: {} 2023-10-18T20:33:04.261 DEBUG:boto:Token: None 2023-10-18T20:33:04.261 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.262 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c791925cadfd016e-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.266 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:33:04.267 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.267 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.268 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.268 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.268 DEBUG:boto:Method: GET 2023-10-18T20:33:04.269 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.269 DEBUG:boto:Data: 2023-10-18T20:33:04.270 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.270 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.271 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.271 DEBUG:boto:Params: {} 2023-10-18T20:33:04.271 DEBUG:boto:Token: None 2023-10-18T20:33:04.272 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.272 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gIx4PaND5rP0i41mVa3c6eJeCEE= 2023-10-18T20:33:04.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gIx4PaND5rP0i41mVa3c6eJeCEE=', 'Content-Length': '0'} 2023-10-18T20:33:04.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbf4a0d3030543f9-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.275 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.276 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.276 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.277 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.277 DEBUG:boto:Method: GET 2023-10-18T20:33:04.278 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.278 DEBUG:boto:Data: 2023-10-18T20:33:04.279 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.279 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.280 DEBUG:boto:Params: {} 2023-10-18T20:33:04.280 DEBUG:boto:Token: None 2023-10-18T20:33:04.281 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.281 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gIx4PaND5rP0i41mVa3c6eJeCEE= 2023-10-18T20:33:04.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gIx4PaND5rP0i41mVa3c6eJeCEE=', 'Content-Length': '0'} 2023-10-18T20:33:04.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb56a5aafdd6385f-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.285 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.285 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.285 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.286 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.286 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.287 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.287 DEBUG:boto:Data: 2023-10-18T20:33:04.288 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.288 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.289 DEBUG:boto:Params: {} 2023-10-18T20:33:04.289 DEBUG:boto:Token: None 2023-10-18T20:33:04.290 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.290 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5q2DQWoFMXG7AHOXDaYb14MvIWY= 2023-10-18T20:33:04.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5q2DQWoFMXG7AHOXDaYb14MvIWY=', 'Content-Length': '0'} 2023-10-18T20:33:04.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f5fd81c7d62738-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.293 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.293 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.294 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.294 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.295 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.295 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.296 DEBUG:boto:Data: 2023-10-18T20:33:04.296 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.296 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.297 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.297 DEBUG:boto:Params: {} 2023-10-18T20:33:04.298 DEBUG:boto:Token: None 2023-10-18T20:33:04.298 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.299 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5q2DQWoFMXG7AHOXDaYb14MvIWY= 2023-10-18T20:33:04.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5q2DQWoFMXG7AHOXDaYb14MvIWY=', 'Content-Length': '0'} 2023-10-18T20:33:04.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089ba696db3d7daf7-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:33:04.302 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.302 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.303 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.305 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.306 DEBUG:boto:Method: GET 2023-10-18T20:33:04.306 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.306 DEBUG:boto:Data: 2023-10-18T20:33:04.307 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.307 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.308 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.308 DEBUG:boto:Params: {} 2023-10-18T20:33:04.309 DEBUG:boto:Token: None 2023-10-18T20:33:04.309 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.310 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gIx4PaND5rP0i41mVa3c6eJeCEE= 2023-10-18T20:33:04.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gIx4PaND5rP0i41mVa3c6eJeCEE=', 'Content-Length': '0'} 2023-10-18T20:33:04.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbda9654682023ed-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.354 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.355 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.355 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.356 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.356 DEBUG:boto:Method: GET 2023-10-18T20:33:04.356 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.357 DEBUG:boto:Data: 2023-10-18T20:33:04.357 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.358 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.359 DEBUG:boto:Params: {} 2023-10-18T20:33:04.359 DEBUG:boto:Token: None 2023-10-18T20:33:04.360 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:33:04.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gIx4PaND5rP0i41mVa3c6eJeCEE= 2023-10-18T20:33:04.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gIx4PaND5rP0i41mVa3c6eJeCEE=', 'Content-Length': '0'} 2023-10-18T20:33:04.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279e94e69a09aa10-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.364 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.364 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.365 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.365 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.365 DEBUG:boto:Method: GET 2023-10-18T20:33:04.366 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.366 DEBUG:boto:Data: 2023-10-18T20:33:04.367 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.368 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.368 DEBUG:boto:Params: {} 2023-10-18T20:33:04.368 DEBUG:boto:Token: None 2023-10-18T20:33:04.369 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070cfd4c775f75420-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.373 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.374 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.374 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.375 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.375 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.376 DEBUG:boto:Method: GET 2023-10-18T20:33:04.376 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.376 DEBUG:boto:Data: 2023-10-18T20:33:04.377 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.378 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.378 DEBUG:boto:Params: {} 2023-10-18T20:33:04.379 DEBUG:boto:Token: None 2023-10-18T20:33:04.379 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.380 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000106c08a087a4f263-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.383 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.384 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.385 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.385 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.386 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:33:04.386 DEBUG:boto:Data: 2023-10-18T20:33:04.386 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.387 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.387 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.388 DEBUG:boto:Params: {} 2023-10-18T20:33:04.388 DEBUG:boto:Token: None 2023-10-18T20:33:04.389 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22 2023-10-18T20:33:04.389 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CTbJt1NZxf0oiaYIm/+DIpAUkx4= 2023-10-18T20:33:04.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CTbJt1NZxf0oiaYIm/+DIpAUkx4=', 'Content-Length': '0'} 2023-10-18T20:33:04.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbd7e408461c186b-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.392 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.393 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.393 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.394 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:33:04.394 DEBUG:boto:Data: 2023-10-18T20:33:04.394 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.395 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.395 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.395 DEBUG:boto:Params: {} 2023-10-18T20:33:04.396 DEBUG:boto:Token: None 2023-10-18T20:33:04.396 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22 2023-10-18T20:33:04.397 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CTbJt1NZxf0oiaYIm/+DIpAUkx4= 2023-10-18T20:33:04.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CTbJt1NZxf0oiaYIm/+DIpAUkx4=', 'Content-Length': '0'} 2023-10-18T20:33:04.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055f154a0d612384a-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:33:04.400 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.400 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.401 DEBUG:boto:Method: GET 2023-10-18T20:33:04.401 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:33:04.402 DEBUG:boto:Data: 2023-10-18T20:33:04.402 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.403 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.403 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.403 DEBUG:boto:Params: {} 2023-10-18T20:33:04.404 DEBUG:boto:Token: None 2023-10-18T20:33:04.404 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22 2023-10-18T20:33:04.405 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ydgXl0My0jL8RIE1S83ZNo/BYLo= 2023-10-18T20:33:04.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ydgXl0My0jL8RIE1S83ZNo/BYLo=', 'Content-Length': '0'} 2023-10-18T20:33:04.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001265ca726805711e-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.408 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:33:04.409 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:33:04.409 DEBUG:boto:Method: GET 2023-10-18T20:33:04.409 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:33:04.410 DEBUG:boto:Data: 2023-10-18T20:33:04.410 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.411 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.411 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.411 DEBUG:boto:Params: {} 2023-10-18T20:33:04.412 DEBUG:boto:Token: None 2023-10-18T20:33:04.412 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj22 2023-10-18T20:33:04.413 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ydgXl0My0jL8RIE1S83ZNo/BYLo= 2023-10-18T20:33:04.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ydgXl0My0jL8RIE1S83ZNo/BYLo=', 'Content-Length': '0'} 2023-10-18T20:33:04.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f6ac30bd6ced7f-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.462 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.463 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.464 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.464 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.464 DEBUG:boto:Method: GET 2023-10-18T20:33:04.465 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.465 DEBUG:boto:Data: 2023-10-18T20:33:04.465 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.466 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.466 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.467 DEBUG:boto:Params: {} 2023-10-18T20:33:04.467 DEBUG:boto:Token: None 2023-10-18T20:33:04.468 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.468 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003244594781721c06-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.472 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.472 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.473 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.473 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.474 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.474 DEBUG:boto:Method: GET 2023-10-18T20:33:04.475 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:33:04.475 DEBUG:boto:Data: 2023-10-18T20:33:04.475 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.476 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.476 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.477 DEBUG:boto:Params: {} 2023-10-18T20:33:04.477 DEBUG:boto:Token: None 2023-10-18T20:33:04.478 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.478 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006ad04fd77917246-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.482 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:33:04.483 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.483 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.484 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.484 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.484 DEBUG:boto:Method: GET 2023-10-18T20:33:04.485 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.485 DEBUG:boto:Data: 2023-10-18T20:33:04.486 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.486 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.487 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.487 DEBUG:boto:Params: {} 2023-10-18T20:33:04.487 DEBUG:boto:Token: None 2023-10-18T20:33:04.488 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.488 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JWeQ1xUNEscX/mPi+OFGaGVU4Aw= 2023-10-18T20:33:04.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JWeQ1xUNEscX/mPi+OFGaGVU4Aw=', 'Content-Length': '0'} 2023-10-18T20:33:04.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd50434ecba7b136-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.491 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.492 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.492 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.493 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.493 DEBUG:boto:Method: GET 2023-10-18T20:33:04.494 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.494 DEBUG:boto:Data: 2023-10-18T20:33:04.494 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.495 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.495 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.495 DEBUG:boto:Params: {} 2023-10-18T20:33:04.496 DEBUG:boto:Token: None 2023-10-18T20:33:04.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JWeQ1xUNEscX/mPi+OFGaGVU4Aw= 2023-10-18T20:33:04.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JWeQ1xUNEscX/mPi+OFGaGVU4Aw=', 'Content-Length': '0'} 2023-10-18T20:33:04.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ee961be5c7d2b1c-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.500 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.500 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.500 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.501 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.501 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.501 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.502 DEBUG:boto:Data: 2023-10-18T20:33:04.502 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.503 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.503 DEBUG:boto:Params: {} 2023-10-18T20:33:04.504 DEBUG:boto:Token: None 2023-10-18T20:33:04.504 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fFOMmQEMpahD3vuac9whKp7TXkk= 2023-10-18T20:33:04.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fFOMmQEMpahD3vuac9whKp7TXkk=', 'Content-Length': '0'} 2023-10-18T20:33:04.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035639413638f8232-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.510 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.511 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.511 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.511 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.512 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.512 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.513 DEBUG:boto:Data: 2023-10-18T20:33:04.513 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.514 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.514 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.515 DEBUG:boto:Params: {} 2023-10-18T20:33:04.515 DEBUG:boto:Token: None 2023-10-18T20:33:04.516 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fFOMmQEMpahD3vuac9whKp7TXkk= 2023-10-18T20:33:04.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fFOMmQEMpahD3vuac9whKp7TXkk=', 'Content-Length': '0'} 2023-10-18T20:33:04.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef73241522c2f4d1-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:33:04.520 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.520 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.520 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.521 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.521 DEBUG:boto:Method: GET 2023-10-18T20:33:04.522 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.522 DEBUG:boto:Data: 2023-10-18T20:33:04.523 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.523 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.524 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.524 DEBUG:boto:Params: {} 2023-10-18T20:33:04.525 DEBUG:boto:Token: None 2023-10-18T20:33:04.525 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.526 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JWeQ1xUNEscX/mPi+OFGaGVU4Aw= 2023-10-18T20:33:04.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JWeQ1xUNEscX/mPi+OFGaGVU4Aw=', 'Content-Length': '0'} 2023-10-18T20:33:04.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f0c67d3d2b8e75f-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.571 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.571 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.571 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.572 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.572 DEBUG:boto:Method: GET 2023-10-18T20:33:04.573 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.573 DEBUG:boto:Data: 2023-10-18T20:33:04.574 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.574 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.574 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.575 DEBUG:boto:Params: {} 2023-10-18T20:33:04.575 DEBUG:boto:Token: None 2023-10-18T20:33:04.576 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:33:04.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JWeQ1xUNEscX/mPi+OFGaGVU4Aw= 2023-10-18T20:33:04.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JWeQ1xUNEscX/mPi+OFGaGVU4Aw=', 'Content-Length': '0'} 2023-10-18T20:33:04.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d15519c10ed6eb1-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.580 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.580 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.581 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.581 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.581 DEBUG:boto:Method: GET 2023-10-18T20:33:04.582 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.582 DEBUG:boto:Data: 2023-10-18T20:33:04.583 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.584 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.584 DEBUG:boto:Params: {} 2023-10-18T20:33:04.585 DEBUG:boto:Token: None 2023-10-18T20:33:04.585 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092a1963e129989de-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.589 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.590 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.591 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.591 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.592 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.592 DEBUG:boto:Method: GET 2023-10-18T20:33:04.592 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.593 DEBUG:boto:Data: 2023-10-18T20:33:04.593 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.594 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.594 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.595 DEBUG:boto:Params: {} 2023-10-18T20:33:04.595 DEBUG:boto:Token: None 2023-10-18T20:33:04.596 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.596 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bdc206d123a7aea-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.642 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.643 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.644 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.644 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.644 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:33:04.645 DEBUG:boto:Data: 2023-10-18T20:33:04.645 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.646 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.646 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.647 DEBUG:boto:Params: {} 2023-10-18T20:33:04.647 DEBUG:boto:Token: None 2023-10-18T20:33:04.648 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33 2023-10-18T20:33:04.648 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GFhDEJK+dV7lLB8PO49ay7gnER8= 2023-10-18T20:33:04.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GFhDEJK+dV7lLB8PO49ay7gnER8=', 'Content-Length': '0'} 2023-10-18T20:33:04.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c305d4ea5bb26df-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.651 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.652 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.652 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.653 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:33:04.653 DEBUG:boto:Data: 2023-10-18T20:33:04.653 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.654 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.655 DEBUG:boto:Params: {} 2023-10-18T20:33:04.655 DEBUG:boto:Token: None 2023-10-18T20:33:04.656 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33 2023-10-18T20:33:04.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GFhDEJK+dV7lLB8PO49ay7gnER8= 2023-10-18T20:33:04.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GFhDEJK+dV7lLB8PO49ay7gnER8=', 'Content-Length': '0'} 2023-10-18T20:33:04.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d3725ce8a49e865-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:33:04.660 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.660 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.661 DEBUG:boto:Method: GET 2023-10-18T20:33:04.661 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:33:04.661 DEBUG:boto:Data: 2023-10-18T20:33:04.662 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.663 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.663 DEBUG:boto:Params: {} 2023-10-18T20:33:04.664 DEBUG:boto:Token: None 2023-10-18T20:33:04.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33 2023-10-18T20:33:04.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BP6vTsnzKGEN+vyGQv5J08j73b0= 2023-10-18T20:33:04.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BP6vTsnzKGEN+vyGQv5J08j73b0=', 'Content-Length': '0'} 2023-10-18T20:33:04.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d6dc0228b0c877d-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.668 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:33:04.669 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:33:04.669 DEBUG:boto:Method: GET 2023-10-18T20:33:04.669 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:33:04.670 DEBUG:boto:Data: 2023-10-18T20:33:04.670 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.671 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.672 DEBUG:boto:Params: {} 2023-10-18T20:33:04.672 DEBUG:boto:Token: None 2023-10-18T20:33:04.673 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj33 2023-10-18T20:33:04.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BP6vTsnzKGEN+vyGQv5J08j73b0= 2023-10-18T20:33:04.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BP6vTsnzKGEN+vyGQv5J08j73b0=', 'Content-Length': '0'} 2023-10-18T20:33:04.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbab73debbde0d46-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.677 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.677 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.678 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.678 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.678 DEBUG:boto:Method: GET 2023-10-18T20:33:04.679 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.679 DEBUG:boto:Data: 2023-10-18T20:33:04.680 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.680 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.681 DEBUG:boto:Params: {} 2023-10-18T20:33:04.681 DEBUG:boto:Token: None 2023-10-18T20:33:04.682 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.682 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000776ba15918aef176-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.726 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.727 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.728 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.728 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.729 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.729 DEBUG:boto:Method: GET 2023-10-18T20:33:04.730 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:33:04.730 DEBUG:boto:Data: 2023-10-18T20:33:04.730 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.731 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.731 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.732 DEBUG:boto:Params: {} 2023-10-18T20:33:04.732 DEBUG:boto:Token: None 2023-10-18T20:33:04.733 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.733 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dd580188045497c-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.737 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:33:04.738 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.739 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.739 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.740 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.740 DEBUG:boto:Method: GET 2023-10-18T20:33:04.741 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.741 DEBUG:boto:Data: 2023-10-18T20:33:04.742 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.742 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.743 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.743 DEBUG:boto:Params: {} 2023-10-18T20:33:04.743 DEBUG:boto:Token: None 2023-10-18T20:33:04.744 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.744 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QwCbANya08722z6XUNAMYG65Rzg= 2023-10-18T20:33:04.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QwCbANya08722z6XUNAMYG65Rzg=', 'Content-Length': '0'} 2023-10-18T20:33:04.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe75a23fc03e026-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.748 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.749 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.749 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.750 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.750 DEBUG:boto:Method: GET 2023-10-18T20:33:04.750 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.751 DEBUG:boto:Data: 2023-10-18T20:33:04.752 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.752 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.753 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.753 DEBUG:boto:Params: {} 2023-10-18T20:33:04.754 DEBUG:boto:Token: None 2023-10-18T20:33:04.754 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QwCbANya08722z6XUNAMYG65Rzg= 2023-10-18T20:33:04.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QwCbANya08722z6XUNAMYG65Rzg=', 'Content-Length': '0'} 2023-10-18T20:33:04.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052d5d13adbe1079e-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.758 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.759 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.759 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.759 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.760 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.760 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.761 DEBUG:boto:Data: 2023-10-18T20:33:04.761 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.762 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.762 DEBUG:boto:Params: {} 2023-10-18T20:33:04.763 DEBUG:boto:Token: None 2023-10-18T20:33:04.763 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XKAeJlRy1EcZBKY/z8jwtIlgmKk= 2023-10-18T20:33:04.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XKAeJlRy1EcZBKY/z8jwtIlgmKk=', 'Content-Length': '0'} 2023-10-18T20:33:04.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cae6e1c0cc58dc1-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.767 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.767 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.768 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.768 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.768 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.769 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.769 DEBUG:boto:Data: 2023-10-18T20:33:04.770 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.770 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.771 DEBUG:boto:Params: {} 2023-10-18T20:33:04.771 DEBUG:boto:Token: None 2023-10-18T20:33:04.772 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XKAeJlRy1EcZBKY/z8jwtIlgmKk= 2023-10-18T20:33:04.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XKAeJlRy1EcZBKY/z8jwtIlgmKk=', 'Content-Length': '0'} 2023-10-18T20:33:04.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba92340dbee9029c-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:33:04.776 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.776 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.776 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.777 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.777 DEBUG:boto:Method: GET 2023-10-18T20:33:04.778 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.778 DEBUG:boto:Data: 2023-10-18T20:33:04.779 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.780 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.780 DEBUG:boto:Params: {} 2023-10-18T20:33:04.781 DEBUG:boto:Token: None 2023-10-18T20:33:04.781 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QwCbANya08722z6XUNAMYG65Rzg= 2023-10-18T20:33:04.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QwCbANya08722z6XUNAMYG65Rzg=', 'Content-Length': '0'} 2023-10-18T20:33:04.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9e53ded6af83134-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.785 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.785 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.786 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.786 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.787 DEBUG:boto:Method: GET 2023-10-18T20:33:04.787 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.787 DEBUG:boto:Data: 2023-10-18T20:33:04.788 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.788 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.789 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.789 DEBUG:boto:Params: {} 2023-10-18T20:33:04.790 DEBUG:boto:Token: None 2023-10-18T20:33:04.790 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:33:04.791 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QwCbANya08722z6XUNAMYG65Rzg= 2023-10-18T20:33:04.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QwCbANya08722z6XUNAMYG65Rzg=', 'Content-Length': '0'} 2023-10-18T20:33:04.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb905f205895dbf-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.834 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.835 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.836 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.836 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.836 DEBUG:boto:Method: GET 2023-10-18T20:33:04.837 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.837 DEBUG:boto:Data: 2023-10-18T20:33:04.838 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.839 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.839 DEBUG:boto:Params: {} 2023-10-18T20:33:04.840 DEBUG:boto:Token: None 2023-10-18T20:33:04.840 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.841 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e4894fbc576e75-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.844 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.845 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.846 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.846 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.846 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.847 DEBUG:boto:Method: GET 2023-10-18T20:33:04.847 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.848 DEBUG:boto:Data: 2023-10-18T20:33:04.848 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.849 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.849 DEBUG:boto:Params: {} 2023-10-18T20:33:04.850 DEBUG:boto:Token: None 2023-10-18T20:33:04.850 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.851 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6946eb85b93855d-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.855 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.855 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.856 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.856 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.856 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:33:04.857 DEBUG:boto:Data: 2023-10-18T20:33:04.857 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.858 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.859 DEBUG:boto:Params: {} 2023-10-18T20:33:04.859 DEBUG:boto:Token: None 2023-10-18T20:33:04.859 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44 2023-10-18T20:33:04.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wkx6+8y0RxFzaDd70X/4YW21DTI= 2023-10-18T20:33:04.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wkx6+8y0RxFzaDd70X/4YW21DTI=', 'Content-Length': '0'} 2023-10-18T20:33:04.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072db653f783c80e9-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.863 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.863 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.863 DEBUG:boto:Method: HEAD 2023-10-18T20:33:04.864 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:33:04.864 DEBUG:boto:Data: 2023-10-18T20:33:04.865 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.865 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.866 DEBUG:boto:Params: {} 2023-10-18T20:33:04.866 DEBUG:boto:Token: None 2023-10-18T20:33:04.867 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44 2023-10-18T20:33:04.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wkx6+8y0RxFzaDd70X/4YW21DTI= 2023-10-18T20:33:04.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wkx6+8y0RxFzaDd70X/4YW21DTI=', 'Content-Length': '0'} 2023-10-18T20:33:04.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add4f9433b2015dc-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:33:04.871 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.871 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.872 DEBUG:boto:Method: GET 2023-10-18T20:33:04.872 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:33:04.872 DEBUG:boto:Data: 2023-10-18T20:33:04.873 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.874 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.874 DEBUG:boto:Params: {} 2023-10-18T20:33:04.875 DEBUG:boto:Token: None 2023-10-18T20:33:04.875 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44 2023-10-18T20:33:04.876 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ew6MZB6+5x0hBcdjQMwkH04Tvdc= 2023-10-18T20:33:04.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ew6MZB6+5x0hBcdjQMwkH04Tvdc=', 'Content-Length': '0'} 2023-10-18T20:33:04.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073009b9a0dce86b8-0065304100-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.922 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:33:04.923 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:33:04.923 DEBUG:boto:Method: GET 2023-10-18T20:33:04.924 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:33:04.924 DEBUG:boto:Data: 2023-10-18T20:33:04.925 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.925 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.926 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.926 DEBUG:boto:Params: {} 2023-10-18T20:33:04.927 DEBUG:boto:Token: None 2023-10-18T20:33:04.927 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/obj44 2023-10-18T20:33:04.928 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ew6MZB6+5x0hBcdjQMwkH04Tvdc= 2023-10-18T20:33:04.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ew6MZB6+5x0hBcdjQMwkH04Tvdc=', 'Content-Length': '0'} 2023-10-18T20:33:04.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b19b092eac86364e-0065304100-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.931 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.932 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.932 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.932 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.933 DEBUG:boto:Method: GET 2023-10-18T20:33:04.933 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.934 DEBUG:boto:Data: 2023-10-18T20:33:04.934 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:04.935 DEBUG:boto:Port: 8001 2023-10-18T20:33:04.936 DEBUG:boto:Params: {} 2023-10-18T20:33:04.936 DEBUG:boto:Token: None 2023-10-18T20:33:04.937 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.937 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c0a60ef0a9c0926-0065304100-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.941 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.941 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:04.942 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:04.942 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.943 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.943 DEBUG:boto:Method: GET 2023-10-18T20:33:04.944 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:33:04.944 DEBUG:boto:Data: 2023-10-18T20:33:04.945 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:04.946 DEBUG:boto:Port: 8000 2023-10-18T20:33:04.946 DEBUG:boto:Params: {} 2023-10-18T20:33:04.947 DEBUG:boto:Token: None 2023-10-18T20:33:04.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:04 GMT /bofrnf-59/ 2023-10-18T20:33:04.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY= 2023-10-18T20:33:04.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhATcF93tzu7lVTQrnzXagE+PAY=', 'Content-Length': '0'} 2023-10-18T20:33:04.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003638d75842a143ff-0065304100-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:04.994 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:04.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:33:04.996 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:04.996 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:04.997 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:04.997 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:04.998 DEBUG:boto:Method: GET 2023-10-18T20:33:04.998 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:04.998 DEBUG:boto:Data: 2023-10-18T20:33:04.999 DEBUG:boto:Headers: {} 2023-10-18T20:33:04.999 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:05.000 DEBUG:boto:Port: 8001 2023-10-18T20:33:05.000 DEBUG:boto:Params: {} 2023-10-18T20:33:05.001 DEBUG:boto:Token: None 2023-10-18T20:33:05.001 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.002 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GRze5olPlRdAMJVpN2nC7Vp7ra4= 2023-10-18T20:33:05.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GRze5olPlRdAMJVpN2nC7Vp7ra4=', 'Content-Length': '0'} 2023-10-18T20:33:05.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe9316685affcaf8-0065304101-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.006 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:05.006 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:05.006 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.007 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.007 DEBUG:boto:Method: GET 2023-10-18T20:33:05.008 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.008 DEBUG:boto:Data: 2023-10-18T20:33:05.009 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.009 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.010 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.010 DEBUG:boto:Params: {} 2023-10-18T20:33:05.011 DEBUG:boto:Token: None 2023-10-18T20:33:05.011 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GRze5olPlRdAMJVpN2nC7Vp7ra4= 2023-10-18T20:33:05.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GRze5olPlRdAMJVpN2nC7Vp7ra4=', 'Content-Length': '0'} 2023-10-18T20:33:05.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031eafbb4acbad62c-0065304101-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.015 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:05.016 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:05.016 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.016 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.017 DEBUG:boto:Method: HEAD 2023-10-18T20:33:05.017 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.018 DEBUG:boto:Data: 2023-10-18T20:33:05.018 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.018 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:05.019 DEBUG:boto:Port: 8001 2023-10-18T20:33:05.019 DEBUG:boto:Params: {} 2023-10-18T20:33:05.020 DEBUG:boto:Token: None 2023-10-18T20:33:05.020 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.021 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R4q5MZBMCp73F4UtL1yV4hDl1r4= 2023-10-18T20:33:05.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R4q5MZBMCp73F4UtL1yV4hDl1r4=', 'Content-Length': '0'} 2023-10-18T20:33:05.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bda9b2c4a98b4bae-0065304101-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.024 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:05.024 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:05.025 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.025 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.026 DEBUG:boto:Method: HEAD 2023-10-18T20:33:05.026 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.027 DEBUG:boto:Data: 2023-10-18T20:33:05.027 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.027 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.028 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.028 DEBUG:boto:Params: {} 2023-10-18T20:33:05.029 DEBUG:boto:Token: None 2023-10-18T20:33:05.029 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R4q5MZBMCp73F4UtL1yV4hDl1r4= 2023-10-18T20:33:05.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R4q5MZBMCp73F4UtL1yV4hDl1r4=', 'Content-Length': '0'} 2023-10-18T20:33:05.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050540cc0bad38a22-0065304101-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:33:05.034 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:05.034 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:05.035 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.035 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.035 DEBUG:boto:Method: GET 2023-10-18T20:33:05.036 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.036 DEBUG:boto:Data: 2023-10-18T20:33:05.037 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.037 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:05.037 DEBUG:boto:Port: 8001 2023-10-18T20:33:05.038 DEBUG:boto:Params: {} 2023-10-18T20:33:05.038 DEBUG:boto:Token: None 2023-10-18T20:33:05.039 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GRze5olPlRdAMJVpN2nC7Vp7ra4= 2023-10-18T20:33:05.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GRze5olPlRdAMJVpN2nC7Vp7ra4=', 'Content-Length': '0'} 2023-10-18T20:33:05.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14af05878a93411-0065304101-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.042 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:05.043 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:05.043 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.044 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.044 DEBUG:boto:Method: GET 2023-10-18T20:33:05.044 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.045 DEBUG:boto:Data: 2023-10-18T20:33:05.045 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.046 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.046 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.046 DEBUG:boto:Params: {} 2023-10-18T20:33:05.047 DEBUG:boto:Token: None 2023-10-18T20:33:05.047 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:33:05.048 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GRze5olPlRdAMJVpN2nC7Vp7ra4= 2023-10-18T20:33:05.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GRze5olPlRdAMJVpN2nC7Vp7ra4=', 'Content-Length': '0'} 2023-10-18T20:33:05.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000237c77b9733f2fd1-0065304101-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.051 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:05.051 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:05.052 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.052 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.053 DEBUG:boto:Method: GET 2023-10-18T20:33:05.053 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.053 DEBUG:boto:Data: 2023-10-18T20:33:05.054 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.054 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:05.055 DEBUG:boto:Port: 8001 2023-10-18T20:33:05.055 DEBUG:boto:Params: {} 2023-10-18T20:33:05.056 DEBUG:boto:Token: None 2023-10-18T20:33:05.056 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/ 2023-10-18T20:33:05.056 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tYah+DS4GatIa0RExcdLWV9Pa68= 2023-10-18T20:33:05.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tYah+DS4GatIa0RExcdLWV9Pa68=', 'Content-Length': '0'} 2023-10-18T20:33:05.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a90342bb8625816-0065304101-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.060 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:05.061 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:05.061 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:05.062 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.063 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.063 DEBUG:boto:Method: GET 2023-10-18T20:33:05.063 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.064 DEBUG:boto:Data: 2023-10-18T20:33:05.064 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.065 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.065 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.065 DEBUG:boto:Params: {} 2023-10-18T20:33:05.066 DEBUG:boto:Token: None 2023-10-18T20:33:05.066 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/ 2023-10-18T20:33:05.067 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tYah+DS4GatIa0RExcdLWV9Pa68= 2023-10-18T20:33:05.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tYah+DS4GatIa0RExcdLWV9Pa68=', 'Content-Length': '0'} 2023-10-18T20:33:05.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c6d6e0a36f293f0-0065304101-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.110 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:05.111 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:05.112 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:05.112 DEBUG:boto:Method: HEAD 2023-10-18T20:33:05.113 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:33:05.113 DEBUG:boto:Data: 2023-10-18T20:33:05.114 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.114 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:05.115 DEBUG:boto:Port: 8001 2023-10-18T20:33:05.115 DEBUG:boto:Params: {} 2023-10-18T20:33:05.115 DEBUG:boto:Token: None 2023-10-18T20:33:05.116 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55 2023-10-18T20:33:05.116 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TYYIg0+Bc+sFVRr/lg6wB69auVc= 2023-10-18T20:33:05.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TYYIg0+Bc+sFVRr/lg6wB69auVc=', 'Content-Length': '0'} 2023-10-18T20:33:05.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050cc6a5275804cd2-0065304101-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.120 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:05.120 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:05.121 DEBUG:boto:Method: HEAD 2023-10-18T20:33:05.121 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:33:05.121 DEBUG:boto:Data: 2023-10-18T20:33:05.122 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.123 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.123 DEBUG:boto:Params: {} 2023-10-18T20:33:05.124 DEBUG:boto:Token: None 2023-10-18T20:33:05.124 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55 2023-10-18T20:33:05.125 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TYYIg0+Bc+sFVRr/lg6wB69auVc= 2023-10-18T20:33:05.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TYYIg0+Bc+sFVRr/lg6wB69auVc=', 'Content-Length': '0'} 2023-10-18T20:33:05.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000323b4f93dede739a-0065304101-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:33:05.128 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:05.128 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:05.129 DEBUG:boto:Method: GET 2023-10-18T20:33:05.129 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:33:05.130 DEBUG:boto:Data: 2023-10-18T20:33:05.130 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.131 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:05.131 DEBUG:boto:Port: 8001 2023-10-18T20:33:05.131 DEBUG:boto:Params: {} 2023-10-18T20:33:05.132 DEBUG:boto:Token: None 2023-10-18T20:33:05.132 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55 2023-10-18T20:33:05.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eblkf71qMZL78zd0udDxDkC2QRU= 2023-10-18T20:33:05.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eblkf71qMZL78zd0udDxDkC2QRU=', 'Content-Length': '0'} 2023-10-18T20:33:05.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c128ffa3d0dcd4-0065304101-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.136 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:33:05.137 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:33:05.137 DEBUG:boto:Method: GET 2023-10-18T20:33:05.138 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:33:05.138 DEBUG:boto:Data: 2023-10-18T20:33:05.139 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.139 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.139 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.140 DEBUG:boto:Params: {} 2023-10-18T20:33:05.140 DEBUG:boto:Token: None 2023-10-18T20:33:05.141 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj55 2023-10-18T20:33:05.141 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eblkf71qMZL78zd0udDxDkC2QRU= 2023-10-18T20:33:05.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eblkf71qMZL78zd0udDxDkC2QRU=', 'Content-Length': '0'} 2023-10-18T20:33:05.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1daa7830cefdc9c-0065304101-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.145 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:05.145 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:05.146 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.146 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.146 DEBUG:boto:Method: GET 2023-10-18T20:33:05.147 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.147 DEBUG:boto:Data: 2023-10-18T20:33:05.148 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.148 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:33:05.149 DEBUG:boto:Port: 8001 2023-10-18T20:33:05.149 DEBUG:boto:Params: {} 2023-10-18T20:33:05.150 DEBUG:boto:Token: None 2023-10-18T20:33:05.150 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/ 2023-10-18T20:33:05.151 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tYah+DS4GatIa0RExcdLWV9Pa68= 2023-10-18T20:33:05.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tYah+DS4GatIa0RExcdLWV9Pa68=', 'Content-Length': '0'} 2023-10-18T20:33:05.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba36b0a1d03ae89-0065304101-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.154 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:05.155 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:05.155 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:05.156 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.156 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.157 DEBUG:boto:Method: GET 2023-10-18T20:33:05.157 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:33:05.157 DEBUG:boto:Data: 2023-10-18T20:33:05.158 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.159 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.159 DEBUG:boto:Params: {} 2023-10-18T20:33:05.160 DEBUG:boto:Token: None 2023-10-18T20:33:05.160 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/ 2023-10-18T20:33:05.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tYah+DS4GatIa0RExcdLWV9Pa68= 2023-10-18T20:33:05.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tYah+DS4GatIa0RExcdLWV9Pa68=', 'Content-Length': '0'} 2023-10-18T20:33:05.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b924e219647c2d3-0065304101-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.164 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:33:05.165 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:33:05.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:33:05.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:33:05.166 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:33:05.227 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:33:05.228 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:33:05.273 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:33:05.273 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-59 2023-10-18T20:33:05.277 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 5 2023-10-18T20:33:05.313 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:05.314 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:05.314 DEBUG:boto:Method: HEAD 2023-10-18T20:33:05.315 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:33:05.315 DEBUG:boto:Data: 2023-10-18T20:33:05.316 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.316 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.316 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.317 DEBUG:boto:Params: {} 2023-10-18T20:33:05.317 DEBUG:boto:Token: None 2023-10-18T20:33:05.318 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/ 2023-10-18T20:33:05.318 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI= 2023-10-18T20:33:05.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI=', 'Content-Length': '0'} 2023-10-18T20:33:05.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f68537b76e1bca8-0065304101-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.322 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:33:05.323 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:33:05.325 DEBUG:boto:Method: PUT 2023-10-18T20:33:05.325 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:33:05.326 DEBUG:boto:Data: 2023-10-18T20:33:05.326 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:33:05.327 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.327 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.327 DEBUG:boto:Params: {} 2023-10-18T20:33:05.328 DEBUG:boto:Token: None 2023-10-18T20:33:05.328 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj17 2023-10-18T20:33:05.329 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aa1zqzbBoM85/C+qbNKSUyvTmcs= 2023-10-18T20:33:05.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aa1zqzbBoM85/C+qbNKSUyvTmcs='} 2023-10-18T20:33:05.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000746c21cc7c8f2a46-0065304101-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.335 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:05.335 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:05.335 DEBUG:boto:Method: HEAD 2023-10-18T20:33:05.336 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:33:05.336 DEBUG:boto:Data: 2023-10-18T20:33:05.336 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.337 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.337 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.338 DEBUG:boto:Params: {} 2023-10-18T20:33:05.338 DEBUG:boto:Token: None 2023-10-18T20:33:05.339 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/ 2023-10-18T20:33:05.339 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI= 2023-10-18T20:33:05.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI=', 'Content-Length': '0'} 2023-10-18T20:33:05.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086cd108e66c20a81-0065304101-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.342 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:33:05.343 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:33:05.343 DEBUG:boto:Method: PUT 2023-10-18T20:33:05.344 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:33:05.344 DEBUG:boto:Data: 2023-10-18T20:33:05.345 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:33:05.345 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.345 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.346 DEBUG:boto:Params: {} 2023-10-18T20:33:05.346 DEBUG:boto:Token: None 2023-10-18T20:33:05.347 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj34 2023-10-18T20:33:05.347 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tmvS+WXemV4arUu+n+zSZtAZ9bM= 2023-10-18T20:33:05.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tmvS+WXemV4arUu+n+zSZtAZ9bM='} 2023-10-18T20:33:05.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d30a20872f9e19b-0065304101-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.395 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:05.396 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:05.396 DEBUG:boto:Method: HEAD 2023-10-18T20:33:05.397 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:33:05.397 DEBUG:boto:Data: 2023-10-18T20:33:05.397 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.398 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.399 DEBUG:boto:Params: {} 2023-10-18T20:33:05.399 DEBUG:boto:Token: None 2023-10-18T20:33:05.400 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/ 2023-10-18T20:33:05.400 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI= 2023-10-18T20:33:05.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI=', 'Content-Length': '0'} 2023-10-18T20:33:05.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd040ed156c4dfd7-0065304101-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.405 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:33:05.405 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:33:05.406 DEBUG:boto:Method: PUT 2023-10-18T20:33:05.406 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:33:05.407 DEBUG:boto:Data: 2023-10-18T20:33:05.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:33:05.407 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.408 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.408 DEBUG:boto:Params: {} 2023-10-18T20:33:05.409 DEBUG:boto:Token: None 2023-10-18T20:33:05.409 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj51 2023-10-18T20:33:05.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0SdZkrNOzxD4uwF8fTPa8vXDeaA= 2023-10-18T20:33:05.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0SdZkrNOzxD4uwF8fTPa8vXDeaA='} 2023-10-18T20:33:05.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000803773286ff7109e-0065304101-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.416 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:05.416 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:05.417 DEBUG:boto:Method: HEAD 2023-10-18T20:33:05.417 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:33:05.418 DEBUG:boto:Data: 2023-10-18T20:33:05.418 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.419 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.419 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.419 DEBUG:boto:Params: {} 2023-10-18T20:33:05.420 DEBUG:boto:Token: None 2023-10-18T20:33:05.420 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/ 2023-10-18T20:33:05.421 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI= 2023-10-18T20:33:05.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI=', 'Content-Length': '0'} 2023-10-18T20:33:05.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cacf6792c207f38-0065304101-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.424 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:33:05.424 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:33:05.425 DEBUG:boto:Method: PUT 2023-10-18T20:33:05.425 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:33:05.426 DEBUG:boto:Data: 2023-10-18T20:33:05.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:33:05.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.427 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.427 DEBUG:boto:Params: {} 2023-10-18T20:33:05.428 DEBUG:boto:Token: None 2023-10-18T20:33:05.428 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj68 2023-10-18T20:33:05.429 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zv6Ngl/iwhlBtAeIjvdZKvNoGlQ= 2023-10-18T20:33:05.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zv6Ngl/iwhlBtAeIjvdZKvNoGlQ='} 2023-10-18T20:33:05.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002591f4b08c67b65a-0065304101-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.475 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:33:05.476 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:33:05.476 DEBUG:boto:Method: HEAD 2023-10-18T20:33:05.477 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:33:05.477 DEBUG:boto:Data: 2023-10-18T20:33:05.477 DEBUG:boto:Headers: {} 2023-10-18T20:33:05.478 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.478 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.479 DEBUG:boto:Params: {} 2023-10-18T20:33:05.479 DEBUG:boto:Token: None 2023-10-18T20:33:05.480 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/ 2023-10-18T20:33:05.480 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI= 2023-10-18T20:33:05.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QL0d3Qil0S7Ai9CaBHsKjTl2gbI=', 'Content-Length': '0'} 2023-10-18T20:33:05.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bfc59b8f8dd61e5-0065304101-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.484 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:33:05.484 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:33:05.485 DEBUG:boto:Method: PUT 2023-10-18T20:33:05.485 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:33:05.486 DEBUG:boto:Data: 2023-10-18T20:33:05.486 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:33:05.487 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:33:05.487 DEBUG:boto:Port: 8000 2023-10-18T20:33:05.488 DEBUG:boto:Params: {} 2023-10-18T20:33:05.488 DEBUG:boto:Token: None 2023-10-18T20:33:05.489 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:33:05 GMT /bofrnf-59/obj85 2023-10-18T20:33:05.489 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AVgkDX6110oGnjnPWf0DfjV5Xeg= 2023-10-18T20:33:05.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:33:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AVgkDX6110oGnjnPWf0DfjV5Xeg='} 2023-10-18T20:33:05.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005861bbbb8f53e03-0065304101-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:33:05.495 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-59 2023-10-18T20:33:05.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:33:05.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:33:05.497 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:33:05.553 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:33:05.553 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:33:05.587 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:05.584+0000 7f036f2f0780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T20:33:10.588 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:10.585+0000 7f036f2f0780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T20:33:15.588 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:15.587+0000 7f036f2f0780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T20:33:20.590 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:20.589+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:33:20.591 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-18T20:33:20.591 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:33:25.592 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:25.590+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:33:25.592 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-18T20:33:25.593 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:33:30.593 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:30.592+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:33:30.594 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:33:30.594 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:33:35.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:35.594+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:33:35.596 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:33:35.596 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:33:40.598 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:40.596+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:33:40.598 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:33:40.599 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:33:45.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:45.597+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:33:45.600 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:33:45.600 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:33:50.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:50.599+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:33:50.600 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:33:50.601 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:33:55.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:33:55.599+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:33:55.602 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:33:55.602 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:34:00.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:00.601+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:34:00.604 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:34:00.604 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:34:05.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:05.602+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:34:05.605 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:34:05.605 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:34:10.606 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:10.605+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:34:10.607 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:34:10.607 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:34:15.608 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:15.606+0000 7f036f2f0780 1 waiting for incremental sync to catch up: 2023-10-18T20:34:15.608 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, , , , ] 2023-10-18T20:34:15.608 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:34:20.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:20.608+0000 7f036f2f0780 1 bucket sync caught up with source: 2023-10-18T20:34:20.610 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:34:20.610 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000001.53.6, , , , , , 00000000001.55.6, , 00000000001.52.6, , , , 00000000001.73.6, 00000000001.60.6, , , ] 2023-10-18T20:34:20.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:20.608+0000 7f036f2f0780 0 bucket checkpoint complete 2023-10-18T20:34:20.627 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-59 2023-10-18T20:34:20.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:34:20.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:34:20.629 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:34:20.690 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:34:20.690 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:34:20.728 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:20.727+0000 7f823f94f780 1 bucket sync caught up with source: 2023-10-18T20:34:20.729 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.750.6, 00000000001.1166.6, , , 00000000001.751.6, , , 00000000001.212.6, 00000000003.444.6, 00000000003.226.6] 2023-10-18T20:34:20.729 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000001.750.6, 00000000001.1166.6, , , 00000000001.751.6, , , 00000000001.212.6, 00000000003.444.6, 00000000003.226.6] 2023-10-18T20:34:20.729 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:20.727+0000 7f823f94f780 0 bucket checkpoint complete 2023-10-18T20:34:20.742 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:34:20.743 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:20.743 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:20.744 DEBUG:boto:Method: HEAD 2023-10-18T20:34:20.744 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:34:20.744 DEBUG:boto:Data: 2023-10-18T20:34:20.745 DEBUG:boto:Headers: {} 2023-10-18T20:34:20.745 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:20.746 DEBUG:boto:Port: 8001 2023-10-18T20:34:20.746 DEBUG:boto:Params: {} 2023-10-18T20:34:20.747 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:34:20.747 DEBUG:boto:Token: None 2023-10-18T20:34:20.748 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:20 GMT /bofrnf-59/ 2023-10-18T20:34:20.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B9SQ66WETEaqk8NAlehC5sN0R7o= 2023-10-18T20:34:20.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B9SQ66WETEaqk8NAlehC5sN0R7o=', 'Content-Length': '0'} 2023-10-18T20:34:20.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acfd350213541735-006530414c-1af4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:34:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:20.760 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:20.761 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:20.762 DEBUG:boto:Method: HEAD 2023-10-18T20:34:20.763 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:34:20.763 DEBUG:boto:Data: 2023-10-18T20:34:20.764 DEBUG:boto:Headers: {} 2023-10-18T20:34:20.764 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:20.764 DEBUG:boto:Port: 8000 2023-10-18T20:34:20.765 DEBUG:boto:Params: {} 2023-10-18T20:34:20.765 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:34:20.766 DEBUG:boto:Token: None 2023-10-18T20:34:20.766 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:20 GMT /bofrnf-59/ 2023-10-18T20:34:20.767 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B9SQ66WETEaqk8NAlehC5sN0R7o= 2023-10-18T20:34:20.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B9SQ66WETEaqk8NAlehC5sN0R7o=', 'Content-Length': '0'} 2023-10-18T20:34:20.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007c0b41f59e0378f-006530414c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:34:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:20.773 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:34:20.774 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:20.774 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:20.780 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.781 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.781 DEBUG:boto:Method: GET 2023-10-18T20:34:20.782 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.782 DEBUG:boto:Data: 2023-10-18T20:34:20.787 DEBUG:boto:Headers: {} 2023-10-18T20:34:20.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:20.788 DEBUG:boto:Port: 8001 2023-10-18T20:34:20.788 DEBUG:boto:Params: {} 2023-10-18T20:34:20.789 DEBUG:boto:Token: None 2023-10-18T20:34:20.789 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:20 GMT /bofrnf-59/?versions 2023-10-18T20:34:20.790 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JbGueb3p7CRy2D3PSWw7VRGuJys= 2023-10-18T20:34:20.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JbGueb3p7CRy2D3PSWw7VRGuJys=', 'Content-Length': '0'} 2023-10-18T20:34:20.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce0b1b943e56189d-006530414c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:20.799 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:20.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:34:20.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:34:20.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:34:20.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:34:20.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:34:20.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:34:20.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:34:20.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:34:20.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:34:20.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:34:20.815 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:34:20.815 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:20.816 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:20.816 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.817 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.817 DEBUG:boto:Method: GET 2023-10-18T20:34:20.817 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.818 DEBUG:boto:Data: 2023-10-18T20:34:20.818 DEBUG:boto:Headers: {} 2023-10-18T20:34:20.819 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:20.819 DEBUG:boto:Port: 8000 2023-10-18T20:34:20.820 DEBUG:boto:Params: {} 2023-10-18T20:34:20.820 DEBUG:boto:Token: None 2023-10-18T20:34:20.821 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:20 GMT /bofrnf-59/?versions 2023-10-18T20:34:20.828 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JbGueb3p7CRy2D3PSWw7VRGuJys= 2023-10-18T20:34:20.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JbGueb3p7CRy2D3PSWw7VRGuJys=', 'Content-Length': '0'} 2023-10-18T20:34:20.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077942f55e51073c2-006530414c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:20.834 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:20.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:34:20.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:34:20.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:34:20.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:34:20.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:34:20.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:34:20.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:34:20.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:34:20.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:34:20.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:34:20.852 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:20.852 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:20.853 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.853 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.854 DEBUG:boto:Method: GET 2023-10-18T20:34:20.855 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.855 DEBUG:boto:Data: 2023-10-18T20:34:20.855 DEBUG:boto:Headers: {} 2023-10-18T20:34:20.856 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:20.856 DEBUG:boto:Port: 8001 2023-10-18T20:34:20.857 DEBUG:boto:Params: {} 2023-10-18T20:34:20.857 DEBUG:boto:Token: None 2023-10-18T20:34:20.858 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:20 GMT /bofrnf-59/?versions 2023-10-18T20:34:20.858 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JbGueb3p7CRy2D3PSWw7VRGuJys= 2023-10-18T20:34:20.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JbGueb3p7CRy2D3PSWw7VRGuJys=', 'Content-Length': '0'} 2023-10-18T20:34:20.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94cc87ddacd39ae-006530414c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:20.863 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:20.864 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:20.872 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:20.875 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.876 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.876 DEBUG:boto:Method: GET 2023-10-18T20:34:20.877 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:34:20.877 DEBUG:boto:Data: 2023-10-18T20:34:20.877 DEBUG:boto:Headers: {} 2023-10-18T20:34:20.878 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:20.878 DEBUG:boto:Port: 8000 2023-10-18T20:34:20.879 DEBUG:boto:Params: {} 2023-10-18T20:34:20.879 DEBUG:boto:Token: None 2023-10-18T20:34:20.880 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:20 GMT /bofrnf-59/?versions 2023-10-18T20:34:20.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JbGueb3p7CRy2D3PSWw7VRGuJys= 2023-10-18T20:34:20.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JbGueb3p7CRy2D3PSWw7VRGuJys=', 'Content-Length': '0'} 2023-10-18T20:34:20.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046255537e1675716-006530414c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:20.898 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:20.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:34:20.915 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:20.916 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:20.916 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.917 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.917 DEBUG:boto:Method: GET 2023-10-18T20:34:20.918 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.918 DEBUG:boto:Data: 2023-10-18T20:34:20.919 DEBUG:boto:Headers: {} 2023-10-18T20:34:20.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:20.920 DEBUG:boto:Port: 8001 2023-10-18T20:34:20.920 DEBUG:boto:Params: {} 2023-10-18T20:34:20.921 DEBUG:boto:Token: None 2023-10-18T20:34:20.921 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:20 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MZ6GZ8Sv7ZDLOb2T8qPsQcneHmI= 2023-10-18T20:34:20.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MZ6GZ8Sv7ZDLOb2T8qPsQcneHmI=', 'Content-Length': '0'} 2023-10-18T20:34:20.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e539c242d124138-006530414c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:20.954 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:20.954 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:20.955 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.955 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.956 DEBUG:boto:Method: GET 2023-10-18T20:34:20.956 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.957 DEBUG:boto:Data: 2023-10-18T20:34:20.957 DEBUG:boto:Headers: {} 2023-10-18T20:34:20.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:20.958 DEBUG:boto:Port: 8000 2023-10-18T20:34:20.959 DEBUG:boto:Params: {} 2023-10-18T20:34:20.960 DEBUG:boto:Token: None 2023-10-18T20:34:20.960 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:20 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.961 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MZ6GZ8Sv7ZDLOb2T8qPsQcneHmI= 2023-10-18T20:34:20.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MZ6GZ8Sv7ZDLOb2T8qPsQcneHmI=', 'Content-Length': '0'} 2023-10-18T20:34:20.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b01dcdee2cd6b8-006530414c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:20.964 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:20.965 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:20.965 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.965 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.966 DEBUG:boto:Method: HEAD 2023-10-18T20:34:20.975 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.976 DEBUG:boto:Data: 2023-10-18T20:34:20.976 DEBUG:boto:Headers: {} 2023-10-18T20:34:20.976 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:20.977 DEBUG:boto:Port: 8001 2023-10-18T20:34:20.977 DEBUG:boto:Params: {} 2023-10-18T20:34:20.978 DEBUG:boto:Token: None 2023-10-18T20:34:20.978 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:20 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:20.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Xt85e8MU9IZ672gfcS3b+i3txU= 2023-10-18T20:34:21.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Xt85e8MU9IZ672gfcS3b+i3txU=', 'Content-Length': '0'} 2023-10-18T20:34:21.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b9507ca7c4b535-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.017 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:21.017 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:21.018 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.018 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.019 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.019 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.019 DEBUG:boto:Data: 2023-10-18T20:34:21.020 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.020 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.021 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.021 DEBUG:boto:Params: {} 2023-10-18T20:34:21.022 DEBUG:boto:Token: None 2023-10-18T20:34:21.022 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.023 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MsMVoMpmRPMLwK7ExeI2DauDb3I= 2023-10-18T20:34:21.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MsMVoMpmRPMLwK7ExeI2DauDb3I=', 'Content-Length': '0'} 2023-10-18T20:34:21.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3a5a0200ed35e4c-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:34:21.036 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:21.036 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:21.037 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.037 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.037 DEBUG:boto:Method: GET 2023-10-18T20:34:21.038 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.038 DEBUG:boto:Data: 2023-10-18T20:34:21.039 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.039 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.040 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.040 DEBUG:boto:Params: {} 2023-10-18T20:34:21.041 DEBUG:boto:Token: None 2023-10-18T20:34:21.041 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.042 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CfZ3gJV1hfZS3FX/7hdQ/KXJpoU= 2023-10-18T20:34:21.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CfZ3gJV1hfZS3FX/7hdQ/KXJpoU=', 'Content-Length': '0'} 2023-10-18T20:34:21.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c774ee097d9f0853-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.045 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:21.045 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:21.045 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.046 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.046 DEBUG:boto:Method: GET 2023-10-18T20:34:21.047 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.047 DEBUG:boto:Data: 2023-10-18T20:34:21.048 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.048 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.049 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.049 DEBUG:boto:Params: {} 2023-10-18T20:34:21.050 DEBUG:boto:Token: None 2023-10-18T20:34:21.051 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:34:21.051 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CfZ3gJV1hfZS3FX/7hdQ/KXJpoU= 2023-10-18T20:34:21.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CfZ3gJV1hfZS3FX/7hdQ/KXJpoU=', 'Content-Length': '0'} 2023-10-18T20:34:21.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f3c5eb1cbe8c6bb-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.075 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.076 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.076 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.077 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.077 DEBUG:boto:Method: GET 2023-10-18T20:34:21.077 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.078 DEBUG:boto:Data: 2023-10-18T20:34:21.078 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.079 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.079 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.079 DEBUG:boto:Params: {} 2023-10-18T20:34:21.080 DEBUG:boto:Token: None 2023-10-18T20:34:21.080 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.081 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d75febd4577afae-006530414d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.099 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.100 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.100 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.101 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.101 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.101 DEBUG:boto:Method: GET 2023-10-18T20:34:21.102 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.102 DEBUG:boto:Data: 2023-10-18T20:34:21.103 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.103 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.104 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.104 DEBUG:boto:Params: {} 2023-10-18T20:34:21.104 DEBUG:boto:Token: None 2023-10-18T20:34:21.105 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.105 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000651c482eb31e58e5-006530414d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.113 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.114 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:21.114 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:21.115 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.115 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:34:21.116 DEBUG:boto:Data: 2023-10-18T20:34:21.117 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.117 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.118 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.118 DEBUG:boto:Params: {} 2023-10-18T20:34:21.118 DEBUG:boto:Token: None 2023-10-18T20:34:21.119 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj11 2023-10-18T20:34:21.119 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZszdFxrXS8+j1YEmyB9UOOI9Gds= 2023-10-18T20:34:21.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZszdFxrXS8+j1YEmyB9UOOI9Gds=', 'Content-Length': '0'} 2023-10-18T20:34:21.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c52369129b04318c-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.122 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:21.123 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:21.123 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.123 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:34:21.124 DEBUG:boto:Data: 2023-10-18T20:34:21.124 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.125 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.125 DEBUG:boto:Params: {} 2023-10-18T20:34:21.126 DEBUG:boto:Token: None 2023-10-18T20:34:21.126 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj11 2023-10-18T20:34:21.127 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZszdFxrXS8+j1YEmyB9UOOI9Gds= 2023-10-18T20:34:21.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZszdFxrXS8+j1YEmyB9UOOI9Gds=', 'Content-Length': '0'} 2023-10-18T20:34:21.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f31a8b04dad389f1-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:34:21.131 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:21.131 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:21.131 DEBUG:boto:Method: GET 2023-10-18T20:34:21.132 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:34:21.132 DEBUG:boto:Data: 2023-10-18T20:34:21.133 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.133 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.134 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.134 DEBUG:boto:Params: {} 2023-10-18T20:34:21.134 DEBUG:boto:Token: None 2023-10-18T20:34:21.135 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj11 2023-10-18T20:34:21.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1rbscX8SEIWCRTej8rIchInF5BQ= 2023-10-18T20:34:21.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1rbscX8SEIWCRTej8rIchInF5BQ=', 'Content-Length': '0'} 2023-10-18T20:34:21.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c387e620100e3e4d-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.139 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:34:21.139 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:34:21.140 DEBUG:boto:Method: GET 2023-10-18T20:34:21.140 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:34:21.140 DEBUG:boto:Data: 2023-10-18T20:34:21.141 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.142 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.142 DEBUG:boto:Params: {} 2023-10-18T20:34:21.143 DEBUG:boto:Token: None 2023-10-18T20:34:21.143 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj11 2023-10-18T20:34:21.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1rbscX8SEIWCRTej8rIchInF5BQ= 2023-10-18T20:34:21.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1rbscX8SEIWCRTej8rIchInF5BQ=', 'Content-Length': '0'} 2023-10-18T20:34:21.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a37cf8b2b565effb-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.147 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.147 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.147 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.148 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.148 DEBUG:boto:Method: GET 2023-10-18T20:34:21.149 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.149 DEBUG:boto:Data: 2023-10-18T20:34:21.150 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.150 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.151 DEBUG:boto:Params: {} 2023-10-18T20:34:21.151 DEBUG:boto:Token: None 2023-10-18T20:34:21.152 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f7a540b5265ce5-006530414d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.156 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.156 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.157 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.157 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.157 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.158 DEBUG:boto:Method: GET 2023-10-18T20:34:21.159 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:34:21.159 DEBUG:boto:Data: 2023-10-18T20:34:21.160 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.160 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.161 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.161 DEBUG:boto:Params: {} 2023-10-18T20:34:21.162 DEBUG:boto:Token: None 2023-10-18T20:34:21.163 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2383146c5785785-006530414d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.167 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:34:21.169 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.170 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.171 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.171 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.172 DEBUG:boto:Method: GET 2023-10-18T20:34:21.173 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.174 DEBUG:boto:Data: 2023-10-18T20:34:21.174 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.175 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.175 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.176 DEBUG:boto:Params: {} 2023-10-18T20:34:21.176 DEBUG:boto:Token: None 2023-10-18T20:34:21.177 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.177 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T17wCGptxD3aXM4A36lLlFlz7pM= 2023-10-18T20:34:21.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T17wCGptxD3aXM4A36lLlFlz7pM=', 'Content-Length': '0'} 2023-10-18T20:34:21.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f0b700393a237b7-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.226 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.227 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.228 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.228 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.229 DEBUG:boto:Method: GET 2023-10-18T20:34:21.229 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.229 DEBUG:boto:Data: 2023-10-18T20:34:21.230 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.231 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.231 DEBUG:boto:Params: {} 2023-10-18T20:34:21.232 DEBUG:boto:Token: None 2023-10-18T20:34:21.232 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.233 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T17wCGptxD3aXM4A36lLlFlz7pM= 2023-10-18T20:34:21.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T17wCGptxD3aXM4A36lLlFlz7pM=', 'Content-Length': '0'} 2023-10-18T20:34:21.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c78367cb6a1c8076-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.237 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.237 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.237 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.238 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.238 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.239 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.239 DEBUG:boto:Data: 2023-10-18T20:34:21.239 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.240 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.240 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.241 DEBUG:boto:Params: {} 2023-10-18T20:34:21.241 DEBUG:boto:Token: None 2023-10-18T20:34:21.242 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.242 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i6wMnU7bk1G0O4ZIpsSTcXgSuWQ= 2023-10-18T20:34:21.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i6wMnU7bk1G0O4ZIpsSTcXgSuWQ=', 'Content-Length': '0'} 2023-10-18T20:34:21.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d608858a3c79d7-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.247 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.247 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.248 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.248 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.249 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.249 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.250 DEBUG:boto:Data: 2023-10-18T20:34:21.250 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.251 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.251 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.251 DEBUG:boto:Params: {} 2023-10-18T20:34:21.252 DEBUG:boto:Token: None 2023-10-18T20:34:21.252 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.253 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i6wMnU7bk1G0O4ZIpsSTcXgSuWQ= 2023-10-18T20:34:21.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i6wMnU7bk1G0O4ZIpsSTcXgSuWQ=', 'Content-Length': '0'} 2023-10-18T20:34:21.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000599b671aecb00150-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:34:21.256 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.257 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.257 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.258 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.258 DEBUG:boto:Method: GET 2023-10-18T20:34:21.259 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.259 DEBUG:boto:Data: 2023-10-18T20:34:21.260 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.260 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.261 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.261 DEBUG:boto:Params: {} 2023-10-18T20:34:21.261 DEBUG:boto:Token: None 2023-10-18T20:34:21.262 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.262 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T17wCGptxD3aXM4A36lLlFlz7pM= 2023-10-18T20:34:21.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T17wCGptxD3aXM4A36lLlFlz7pM=', 'Content-Length': '0'} 2023-10-18T20:34:21.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082ce4d3061243196-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.265 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.266 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.266 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.267 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.267 DEBUG:boto:Method: GET 2023-10-18T20:34:21.267 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.268 DEBUG:boto:Data: 2023-10-18T20:34:21.268 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.269 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.269 DEBUG:boto:Params: {} 2023-10-18T20:34:21.270 DEBUG:boto:Token: None 2023-10-18T20:34:21.270 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:34:21.271 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T17wCGptxD3aXM4A36lLlFlz7pM= 2023-10-18T20:34:21.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T17wCGptxD3aXM4A36lLlFlz7pM=', 'Content-Length': '0'} 2023-10-18T20:34:21.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbca4341c8e60b7c-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.314 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.315 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.315 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.316 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.316 DEBUG:boto:Method: GET 2023-10-18T20:34:21.316 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.317 DEBUG:boto:Data: 2023-10-18T20:34:21.317 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.318 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.320 DEBUG:boto:Params: {} 2023-10-18T20:34:21.320 DEBUG:boto:Token: None 2023-10-18T20:34:21.321 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000354636895434da30-006530414d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.325 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.326 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.326 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.327 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.327 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.327 DEBUG:boto:Method: GET 2023-10-18T20:34:21.328 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.328 DEBUG:boto:Data: 2023-10-18T20:34:21.329 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.329 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.330 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.330 DEBUG:boto:Params: {} 2023-10-18T20:34:21.331 DEBUG:boto:Token: None 2023-10-18T20:34:21.331 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.332 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1a16f9654c6b656-006530414d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.335 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.336 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.336 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.337 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.337 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:34:21.338 DEBUG:boto:Data: 2023-10-18T20:34:21.338 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.339 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.339 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.340 DEBUG:boto:Params: {} 2023-10-18T20:34:21.340 DEBUG:boto:Token: None 2023-10-18T20:34:21.340 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17 2023-10-18T20:34:21.341 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uOgFBfU7WLQ11lnYHLGhJVsWiKA= 2023-10-18T20:34:21.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uOgFBfU7WLQ11lnYHLGhJVsWiKA=', 'Content-Length': '0'} 2023-10-18T20:34:21.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d4a951dc1dde4c7-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.344 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.344 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.344 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.345 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:34:21.345 DEBUG:boto:Data: 2023-10-18T20:34:21.346 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.346 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.347 DEBUG:boto:Params: {} 2023-10-18T20:34:21.347 DEBUG:boto:Token: None 2023-10-18T20:34:21.348 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17 2023-10-18T20:34:21.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uOgFBfU7WLQ11lnYHLGhJVsWiKA= 2023-10-18T20:34:21.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uOgFBfU7WLQ11lnYHLGhJVsWiKA=', 'Content-Length': '0'} 2023-10-18T20:34:21.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ef7b51d154a8f92-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:34:21.351 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.352 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.352 DEBUG:boto:Method: GET 2023-10-18T20:34:21.352 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:34:21.353 DEBUG:boto:Data: 2023-10-18T20:34:21.353 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.354 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.354 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.354 DEBUG:boto:Params: {} 2023-10-18T20:34:21.355 DEBUG:boto:Token: None 2023-10-18T20:34:21.355 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17 2023-10-18T20:34:21.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RamjKbNDJc9U1jtCpB0Nv8dNXco= 2023-10-18T20:34:21.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RamjKbNDJc9U1jtCpB0Nv8dNXco=', 'Content-Length': '0'} 2023-10-18T20:34:21.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b20419f58e8faac-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.359 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:34:21.359 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:34:21.360 DEBUG:boto:Method: GET 2023-10-18T20:34:21.360 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:34:21.361 DEBUG:boto:Data: 2023-10-18T20:34:21.361 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.361 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.362 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.362 DEBUG:boto:Params: {} 2023-10-18T20:34:21.363 DEBUG:boto:Token: None 2023-10-18T20:34:21.363 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj17 2023-10-18T20:34:21.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RamjKbNDJc9U1jtCpB0Nv8dNXco= 2023-10-18T20:34:21.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RamjKbNDJc9U1jtCpB0Nv8dNXco=', 'Content-Length': '0'} 2023-10-18T20:34:21.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000984d1d83fc677beb-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.367 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.367 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.368 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.368 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.369 DEBUG:boto:Method: GET 2023-10-18T20:34:21.369 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.369 DEBUG:boto:Data: 2023-10-18T20:34:21.370 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.371 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.371 DEBUG:boto:Params: {} 2023-10-18T20:34:21.372 DEBUG:boto:Token: None 2023-10-18T20:34:21.372 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0af49eb258150f0-006530414d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.418 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.419 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.420 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.420 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.421 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.421 DEBUG:boto:Method: GET 2023-10-18T20:34:21.421 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:34:21.422 DEBUG:boto:Data: 2023-10-18T20:34:21.422 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.423 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.423 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.423 DEBUG:boto:Params: {} 2023-10-18T20:34:21.424 DEBUG:boto:Token: None 2023-10-18T20:34:21.424 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.425 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e95e7ba24e976456-006530414d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.429 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:34:21.430 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.430 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.431 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.431 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.431 DEBUG:boto:Method: GET 2023-10-18T20:34:21.432 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.432 DEBUG:boto:Data: 2023-10-18T20:34:21.433 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.434 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.434 DEBUG:boto:Params: {} 2023-10-18T20:34:21.435 DEBUG:boto:Token: None 2023-10-18T20:34:21.435 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.436 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kH83BmbwI1GYPAgN7Ad6OwShl3o= 2023-10-18T20:34:21.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kH83BmbwI1GYPAgN7Ad6OwShl3o=', 'Content-Length': '0'} 2023-10-18T20:34:21.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feadeca11e60e28e-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.439 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.439 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.440 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.440 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.441 DEBUG:boto:Method: GET 2023-10-18T20:34:21.441 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.441 DEBUG:boto:Data: 2023-10-18T20:34:21.442 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.443 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.443 DEBUG:boto:Params: {} 2023-10-18T20:34:21.444 DEBUG:boto:Token: None 2023-10-18T20:34:21.444 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.445 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kH83BmbwI1GYPAgN7Ad6OwShl3o= 2023-10-18T20:34:21.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kH83BmbwI1GYPAgN7Ad6OwShl3o=', 'Content-Length': '0'} 2023-10-18T20:34:21.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e8c101040db4a6-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.448 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.449 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.449 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.449 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.450 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.450 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.451 DEBUG:boto:Data: 2023-10-18T20:34:21.451 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.452 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.452 DEBUG:boto:Params: {} 2023-10-18T20:34:21.453 DEBUG:boto:Token: None 2023-10-18T20:34:21.454 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.454 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jOrHAh8eIhSCL+SDdP3qSajHKEQ= 2023-10-18T20:34:21.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jOrHAh8eIhSCL+SDdP3qSajHKEQ=', 'Content-Length': '0'} 2023-10-18T20:34:21.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d188cbaa11c6448e-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.457 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.457 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.458 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.458 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.458 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.459 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.459 DEBUG:boto:Data: 2023-10-18T20:34:21.460 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.460 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.460 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.461 DEBUG:boto:Params: {} 2023-10-18T20:34:21.461 DEBUG:boto:Token: None 2023-10-18T20:34:21.462 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.462 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jOrHAh8eIhSCL+SDdP3qSajHKEQ= 2023-10-18T20:34:21.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jOrHAh8eIhSCL+SDdP3qSajHKEQ=', 'Content-Length': '0'} 2023-10-18T20:34:21.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013160d6a4d3db583-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:34:21.466 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.466 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.467 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.467 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.467 DEBUG:boto:Method: GET 2023-10-18T20:34:21.468 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.468 DEBUG:boto:Data: 2023-10-18T20:34:21.469 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.469 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.470 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.470 DEBUG:boto:Params: {} 2023-10-18T20:34:21.471 DEBUG:boto:Token: None 2023-10-18T20:34:21.471 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.472 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kH83BmbwI1GYPAgN7Ad6OwShl3o= 2023-10-18T20:34:21.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kH83BmbwI1GYPAgN7Ad6OwShl3o=', 'Content-Length': '0'} 2023-10-18T20:34:21.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a8c3c1cef4ea173-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.475 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.475 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.476 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.476 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.477 DEBUG:boto:Method: GET 2023-10-18T20:34:21.477 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.477 DEBUG:boto:Data: 2023-10-18T20:34:21.478 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.478 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.479 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.479 DEBUG:boto:Params: {} 2023-10-18T20:34:21.480 DEBUG:boto:Token: None 2023-10-18T20:34:21.480 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:34:21.481 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kH83BmbwI1GYPAgN7Ad6OwShl3o= 2023-10-18T20:34:21.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kH83BmbwI1GYPAgN7Ad6OwShl3o=', 'Content-Length': '0'} 2023-10-18T20:34:21.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064536dbcb146dd23-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.484 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.485 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.485 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.485 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.486 DEBUG:boto:Method: GET 2023-10-18T20:34:21.486 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.487 DEBUG:boto:Data: 2023-10-18T20:34:21.487 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.488 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.488 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.489 DEBUG:boto:Params: {} 2023-10-18T20:34:21.489 DEBUG:boto:Token: None 2023-10-18T20:34:21.490 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.490 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079e309fce9bf1f66-006530414d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.494 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.494 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.495 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.495 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.496 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.496 DEBUG:boto:Method: GET 2023-10-18T20:34:21.497 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.497 DEBUG:boto:Data: 2023-10-18T20:34:21.498 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.499 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.499 DEBUG:boto:Params: {} 2023-10-18T20:34:21.499 DEBUG:boto:Token: None 2023-10-18T20:34:21.500 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.500 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081741cbfd6489e2e-006530414d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.505 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.505 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.506 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.506 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.506 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:34:21.507 DEBUG:boto:Data: 2023-10-18T20:34:21.507 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.508 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.508 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.509 DEBUG:boto:Params: {} 2023-10-18T20:34:21.509 DEBUG:boto:Token: None 2023-10-18T20:34:21.509 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22 2023-10-18T20:34:21.510 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HXMblsc7kXEignsCS48IwxKsteY= 2023-10-18T20:34:21.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HXMblsc7kXEignsCS48IwxKsteY=', 'Content-Length': '0'} 2023-10-18T20:34:21.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009ec35f9bde69272-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.513 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.514 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.515 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.515 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:34:21.515 DEBUG:boto:Data: 2023-10-18T20:34:21.516 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.517 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.517 DEBUG:boto:Params: {} 2023-10-18T20:34:21.518 DEBUG:boto:Token: None 2023-10-18T20:34:21.518 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22 2023-10-18T20:34:21.519 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HXMblsc7kXEignsCS48IwxKsteY= 2023-10-18T20:34:21.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HXMblsc7kXEignsCS48IwxKsteY=', 'Content-Length': '0'} 2023-10-18T20:34:21.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa31a89a3ba59eee-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:34:21.522 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.522 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.523 DEBUG:boto:Method: GET 2023-10-18T20:34:21.523 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:34:21.524 DEBUG:boto:Data: 2023-10-18T20:34:21.524 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.525 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.525 DEBUG:boto:Params: {} 2023-10-18T20:34:21.526 DEBUG:boto:Token: None 2023-10-18T20:34:21.526 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22 2023-10-18T20:34:21.527 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DJBv8G733GIlbPOHJiALxrWG2/s= 2023-10-18T20:34:21.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DJBv8G733GIlbPOHJiALxrWG2/s=', 'Content-Length': '0'} 2023-10-18T20:34:21.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da22c9cddb562fc9-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.530 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:34:21.531 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:34:21.531 DEBUG:boto:Method: GET 2023-10-18T20:34:21.532 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:34:21.532 DEBUG:boto:Data: 2023-10-18T20:34:21.533 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.533 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.533 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.534 DEBUG:boto:Params: {} 2023-10-18T20:34:21.534 DEBUG:boto:Token: None 2023-10-18T20:34:21.535 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj22 2023-10-18T20:34:21.535 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DJBv8G733GIlbPOHJiALxrWG2/s= 2023-10-18T20:34:21.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DJBv8G733GIlbPOHJiALxrWG2/s=', 'Content-Length': '0'} 2023-10-18T20:34:21.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f18e9c29cd26adb-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.582 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.583 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.583 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.584 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.584 DEBUG:boto:Method: GET 2023-10-18T20:34:21.585 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.585 DEBUG:boto:Data: 2023-10-18T20:34:21.586 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.586 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.587 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.587 DEBUG:boto:Params: {} 2023-10-18T20:34:21.588 DEBUG:boto:Token: None 2023-10-18T20:34:21.588 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.589 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000909bf21999f967be-006530414d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.592 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.593 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.593 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.594 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.594 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.595 DEBUG:boto:Method: GET 2023-10-18T20:34:21.595 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:34:21.595 DEBUG:boto:Data: 2023-10-18T20:34:21.596 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.596 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.597 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.597 DEBUG:boto:Params: {} 2023-10-18T20:34:21.598 DEBUG:boto:Token: None 2023-10-18T20:34:21.598 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.599 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f8b47a743efcbb3-006530414d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.605 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:34:21.606 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.606 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.607 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.607 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.607 DEBUG:boto:Method: GET 2023-10-18T20:34:21.608 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.608 DEBUG:boto:Data: 2023-10-18T20:34:21.609 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.609 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.609 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.610 DEBUG:boto:Params: {} 2023-10-18T20:34:21.610 DEBUG:boto:Token: None 2023-10-18T20:34:21.611 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.611 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dTQzRgjRMcbwKxtPE7YXBdC+wRw= 2023-10-18T20:34:21.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dTQzRgjRMcbwKxtPE7YXBdC+wRw=', 'Content-Length': '0'} 2023-10-18T20:34:21.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6377eaab1df3897-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.658 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.659 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.659 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.660 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.660 DEBUG:boto:Method: GET 2023-10-18T20:34:21.660 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.661 DEBUG:boto:Data: 2023-10-18T20:34:21.661 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.662 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.663 DEBUG:boto:Params: {} 2023-10-18T20:34:21.663 DEBUG:boto:Token: None 2023-10-18T20:34:21.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dTQzRgjRMcbwKxtPE7YXBdC+wRw= 2023-10-18T20:34:21.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dTQzRgjRMcbwKxtPE7YXBdC+wRw=', 'Content-Length': '0'} 2023-10-18T20:34:21.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000732ae2e93c84d25f-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.667 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.667 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.668 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.668 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.669 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.669 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.669 DEBUG:boto:Data: 2023-10-18T20:34:21.670 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.670 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.671 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.671 DEBUG:boto:Params: {} 2023-10-18T20:34:21.672 DEBUG:boto:Token: None 2023-10-18T20:34:21.672 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhH+UxgsnIFgNBKoeRbjLDrmjNY= 2023-10-18T20:34:21.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhH+UxgsnIFgNBKoeRbjLDrmjNY=', 'Content-Length': '0'} 2023-10-18T20:34:21.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f53225498b1a6a-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.676 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.676 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.676 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.677 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.677 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.678 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.678 DEBUG:boto:Data: 2023-10-18T20:34:21.679 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.679 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.679 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.680 DEBUG:boto:Params: {} 2023-10-18T20:34:21.680 DEBUG:boto:Token: None 2023-10-18T20:34:21.681 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.681 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PhH+UxgsnIFgNBKoeRbjLDrmjNY= 2023-10-18T20:34:21.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PhH+UxgsnIFgNBKoeRbjLDrmjNY=', 'Content-Length': '0'} 2023-10-18T20:34:21.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d399b57ed275c40-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:34:21.685 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.685 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.686 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.686 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.686 DEBUG:boto:Method: GET 2023-10-18T20:34:21.687 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.687 DEBUG:boto:Data: 2023-10-18T20:34:21.688 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.688 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.689 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.689 DEBUG:boto:Params: {} 2023-10-18T20:34:21.690 DEBUG:boto:Token: None 2023-10-18T20:34:21.690 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.691 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dTQzRgjRMcbwKxtPE7YXBdC+wRw= 2023-10-18T20:34:21.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dTQzRgjRMcbwKxtPE7YXBdC+wRw=', 'Content-Length': '0'} 2023-10-18T20:34:21.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc824273a5ab6015-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.694 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.694 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.695 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.695 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.696 DEBUG:boto:Method: GET 2023-10-18T20:34:21.696 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.696 DEBUG:boto:Data: 2023-10-18T20:34:21.697 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.697 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.698 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.698 DEBUG:boto:Params: {} 2023-10-18T20:34:21.698 DEBUG:boto:Token: None 2023-10-18T20:34:21.699 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:34:21.699 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dTQzRgjRMcbwKxtPE7YXBdC+wRw= 2023-10-18T20:34:21.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dTQzRgjRMcbwKxtPE7YXBdC+wRw=', 'Content-Length': '0'} 2023-10-18T20:34:21.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000606f5472d866abc1-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.703 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.703 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.703 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.704 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.704 DEBUG:boto:Method: GET 2023-10-18T20:34:21.705 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.705 DEBUG:boto:Data: 2023-10-18T20:34:21.705 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.706 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.707 DEBUG:boto:Params: {} 2023-10-18T20:34:21.707 DEBUG:boto:Token: None 2023-10-18T20:34:21.708 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.708 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1d4baf0a329ce34-006530414d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.712 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.713 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.713 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.713 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.714 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.714 DEBUG:boto:Method: GET 2023-10-18T20:34:21.715 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.715 DEBUG:boto:Data: 2023-10-18T20:34:21.716 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.716 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.717 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.717 DEBUG:boto:Params: {} 2023-10-18T20:34:21.718 DEBUG:boto:Token: None 2023-10-18T20:34:21.718 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.719 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bc1980736c33dcb-006530414d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.723 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.723 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.724 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.724 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.724 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:34:21.725 DEBUG:boto:Data: 2023-10-18T20:34:21.725 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.726 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.726 DEBUG:boto:Params: {} 2023-10-18T20:34:21.727 DEBUG:boto:Token: None 2023-10-18T20:34:21.727 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33 2023-10-18T20:34:21.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ai8CUQ58d9X3jm/R7jvrYxPrcwU= 2023-10-18T20:34:21.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ai8CUQ58d9X3jm/R7jvrYxPrcwU=', 'Content-Length': '0'} 2023-10-18T20:34:21.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f6c3a9057423755-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.731 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.731 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.731 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.732 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:34:21.732 DEBUG:boto:Data: 2023-10-18T20:34:21.733 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.734 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.734 DEBUG:boto:Params: {} 2023-10-18T20:34:21.734 DEBUG:boto:Token: None 2023-10-18T20:34:21.735 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33 2023-10-18T20:34:21.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ai8CUQ58d9X3jm/R7jvrYxPrcwU= 2023-10-18T20:34:21.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ai8CUQ58d9X3jm/R7jvrYxPrcwU=', 'Content-Length': '0'} 2023-10-18T20:34:21.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df4dc8b35a4ec44-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:34:21.739 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.739 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.739 DEBUG:boto:Method: GET 2023-10-18T20:34:21.740 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:34:21.740 DEBUG:boto:Data: 2023-10-18T20:34:21.741 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.741 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.741 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.742 DEBUG:boto:Params: {} 2023-10-18T20:34:21.742 DEBUG:boto:Token: None 2023-10-18T20:34:21.743 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33 2023-10-18T20:34:21.743 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:K28jYt1aH6Hog8nNhmvulT+lsOM= 2023-10-18T20:34:21.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:K28jYt1aH6Hog8nNhmvulT+lsOM=', 'Content-Length': '0'} 2023-10-18T20:34:21.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c58d8ae88924971-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.786 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:34:21.787 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:34:21.787 DEBUG:boto:Method: GET 2023-10-18T20:34:21.788 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:34:21.788 DEBUG:boto:Data: 2023-10-18T20:34:21.789 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.789 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.790 DEBUG:boto:Params: {} 2023-10-18T20:34:21.790 DEBUG:boto:Token: None 2023-10-18T20:34:21.791 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj33 2023-10-18T20:34:21.791 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:K28jYt1aH6Hog8nNhmvulT+lsOM= 2023-10-18T20:34:21.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:K28jYt1aH6Hog8nNhmvulT+lsOM=', 'Content-Length': '0'} 2023-10-18T20:34:21.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfda032444a40c27-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.794 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.795 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.795 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.796 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.796 DEBUG:boto:Method: GET 2023-10-18T20:34:21.797 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.797 DEBUG:boto:Data: 2023-10-18T20:34:21.797 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.798 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.798 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.799 DEBUG:boto:Params: {} 2023-10-18T20:34:21.799 DEBUG:boto:Token: None 2023-10-18T20:34:21.800 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.800 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac171e0c9311d2d2-006530414d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.803 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.804 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.805 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.805 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.805 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.806 DEBUG:boto:Method: GET 2023-10-18T20:34:21.806 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:34:21.807 DEBUG:boto:Data: 2023-10-18T20:34:21.807 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.808 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.809 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.809 DEBUG:boto:Params: {} 2023-10-18T20:34:21.810 DEBUG:boto:Token: None 2023-10-18T20:34:21.810 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000174bd49289b553e-006530414d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.814 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:34:21.815 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:21.816 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:21.816 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.817 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.817 DEBUG:boto:Method: GET 2023-10-18T20:34:21.817 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.818 DEBUG:boto:Data: 2023-10-18T20:34:21.818 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.819 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.819 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.819 DEBUG:boto:Params: {} 2023-10-18T20:34:21.820 DEBUG:boto:Token: None 2023-10-18T20:34:21.820 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3nVDsD4JQ02WC8otr2muecMu33o= 2023-10-18T20:34:21.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3nVDsD4JQ02WC8otr2muecMu33o=', 'Content-Length': '0'} 2023-10-18T20:34:21.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085dcd4dab2c3ab82-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.824 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:21.824 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:21.825 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.825 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.826 DEBUG:boto:Method: GET 2023-10-18T20:34:21.826 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.827 DEBUG:boto:Data: 2023-10-18T20:34:21.827 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.827 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.828 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.828 DEBUG:boto:Params: {} 2023-10-18T20:34:21.829 DEBUG:boto:Token: None 2023-10-18T20:34:21.829 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.830 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3nVDsD4JQ02WC8otr2muecMu33o= 2023-10-18T20:34:21.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3nVDsD4JQ02WC8otr2muecMu33o=', 'Content-Length': '0'} 2023-10-18T20:34:21.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e653b51f69f72b6-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.833 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:21.833 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:21.834 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.834 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.835 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.835 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.836 DEBUG:boto:Data: 2023-10-18T20:34:21.836 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.836 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.837 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.837 DEBUG:boto:Params: {} 2023-10-18T20:34:21.838 DEBUG:boto:Token: None 2023-10-18T20:34:21.838 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.839 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D98DQhCBVchwnO3OzziIDfdB5/8= 2023-10-18T20:34:21.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D98DQhCBVchwnO3OzziIDfdB5/8=', 'Content-Length': '0'} 2023-10-18T20:34:21.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094387d2fc0662374-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.842 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:21.842 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:21.842 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.843 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.843 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.844 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.844 DEBUG:boto:Data: 2023-10-18T20:34:21.844 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.845 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.845 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.846 DEBUG:boto:Params: {} 2023-10-18T20:34:21.846 DEBUG:boto:Token: None 2023-10-18T20:34:21.847 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.847 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:D98DQhCBVchwnO3OzziIDfdB5/8= 2023-10-18T20:34:21.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:D98DQhCBVchwnO3OzziIDfdB5/8=', 'Content-Length': '0'} 2023-10-18T20:34:21.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f645252c9219d06-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:34:21.851 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:21.851 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:21.852 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.852 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.853 DEBUG:boto:Method: GET 2023-10-18T20:34:21.853 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.854 DEBUG:boto:Data: 2023-10-18T20:34:21.854 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.854 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.855 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.855 DEBUG:boto:Params: {} 2023-10-18T20:34:21.856 DEBUG:boto:Token: None 2023-10-18T20:34:21.856 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.857 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3nVDsD4JQ02WC8otr2muecMu33o= 2023-10-18T20:34:21.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3nVDsD4JQ02WC8otr2muecMu33o=', 'Content-Length': '0'} 2023-10-18T20:34:21.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece6ed362626f8be-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.902 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:21.903 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:21.903 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.904 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.904 DEBUG:boto:Method: GET 2023-10-18T20:34:21.905 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.905 DEBUG:boto:Data: 2023-10-18T20:34:21.906 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.906 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.907 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.907 DEBUG:boto:Params: {} 2023-10-18T20:34:21.908 DEBUG:boto:Token: None 2023-10-18T20:34:21.908 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:34:21.909 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3nVDsD4JQ02WC8otr2muecMu33o= 2023-10-18T20:34:21.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3nVDsD4JQ02WC8otr2muecMu33o=', 'Content-Length': '0'} 2023-10-18T20:34:21.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9406df4a4e74518-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.912 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.912 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.912 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:34:21.913 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:34:21.913 DEBUG:boto:Method: GET 2023-10-18T20:34:21.914 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:34:21.914 DEBUG:boto:Data: 2023-10-18T20:34:21.914 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.915 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.916 DEBUG:boto:Params: {} 2023-10-18T20:34:21.916 DEBUG:boto:Token: None 2023-10-18T20:34:21.916 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094edbbda33db80f7-006530414d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.920 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.921 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:21.922 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:21.922 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:34:21.922 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:34:21.923 DEBUG:boto:Method: GET 2023-10-18T20:34:21.923 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:34:21.924 DEBUG:boto:Data: 2023-10-18T20:34:21.924 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.925 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.925 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.926 DEBUG:boto:Params: {} 2023-10-18T20:34:21.926 DEBUG:boto:Token: None 2023-10-18T20:34:21.927 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/ 2023-10-18T20:34:21.927 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI= 2023-10-18T20:34:21.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zqTm9G1osX27OCXTgINrBw38YqI=', 'Content-Length': '0'} 2023-10-18T20:34:21.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4346fe4eca3276-006530414d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.974 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:21.975 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:21.976 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:21.976 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.976 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:34:21.977 DEBUG:boto:Data: 2023-10-18T20:34:21.977 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.978 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.978 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.979 DEBUG:boto:Params: {} 2023-10-18T20:34:21.979 DEBUG:boto:Token: None 2023-10-18T20:34:21.980 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj34 2023-10-18T20:34:21.980 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9S7sG2ESujWM8y2AsSvKNwO69JU= 2023-10-18T20:34:21.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9S7sG2ESujWM8y2AsSvKNwO69JU=', 'Content-Length': '0'} 2023-10-18T20:34:21.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007885d2f551a45f55-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.983 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:21.984 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:21.984 DEBUG:boto:Method: HEAD 2023-10-18T20:34:21.984 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:34:21.985 DEBUG:boto:Data: 2023-10-18T20:34:21.985 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.986 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:21.986 DEBUG:boto:Port: 8000 2023-10-18T20:34:21.986 DEBUG:boto:Params: {} 2023-10-18T20:34:21.987 DEBUG:boto:Token: None 2023-10-18T20:34:21.987 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj34 2023-10-18T20:34:21.988 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9S7sG2ESujWM8y2AsSvKNwO69JU= 2023-10-18T20:34:21.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9S7sG2ESujWM8y2AsSvKNwO69JU=', 'Content-Length': '0'} 2023-10-18T20:34:21.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017dffd9e058783cf-006530414d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:34:21.991 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:21.992 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:21.992 DEBUG:boto:Method: GET 2023-10-18T20:34:21.993 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:34:21.993 DEBUG:boto:Data: 2023-10-18T20:34:21.994 DEBUG:boto:Headers: {} 2023-10-18T20:34:21.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:21.995 DEBUG:boto:Port: 8001 2023-10-18T20:34:21.995 DEBUG:boto:Params: {} 2023-10-18T20:34:21.995 DEBUG:boto:Token: None 2023-10-18T20:34:21.996 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:21 GMT /bofrnf-59/obj34 2023-10-18T20:34:21.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:whHeB93WRV7BWBdh6wc2Ei35UDw= 2023-10-18T20:34:21.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:whHeB93WRV7BWBdh6wc2Ei35UDw=', 'Content-Length': '0'} 2023-10-18T20:34:21.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a15ca57d8d79c17b-006530414d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:21.999 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:34:22.000 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:34:22.000 DEBUG:boto:Method: GET 2023-10-18T20:34:22.001 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:34:22.001 DEBUG:boto:Data: 2023-10-18T20:34:22.001 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.002 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.002 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.002 DEBUG:boto:Params: {} 2023-10-18T20:34:22.003 DEBUG:boto:Token: None 2023-10-18T20:34:22.003 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj34 2023-10-18T20:34:22.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dNz6uCGsjrbKBCDedOmHeHFHSGI= 2023-10-18T20:34:22.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dNz6uCGsjrbKBCDedOmHeHFHSGI=', 'Content-Length': '0'} 2023-10-18T20:34:22.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f68cbf07ca741f58-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.007 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.007 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.008 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:34:22.008 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:34:22.009 DEBUG:boto:Method: GET 2023-10-18T20:34:22.009 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:34:22.009 DEBUG:boto:Data: 2023-10-18T20:34:22.010 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.010 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.011 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.011 DEBUG:boto:Params: {} 2023-10-18T20:34:22.012 DEBUG:boto:Token: None 2023-10-18T20:34:22.012 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.013 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a069feb49d8237-006530414e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.058 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.059 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.060 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.060 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:34:22.060 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:34:22.061 DEBUG:boto:Method: GET 2023-10-18T20:34:22.061 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:34:22.062 DEBUG:boto:Data: 2023-10-18T20:34:22.062 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.062 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.063 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.063 DEBUG:boto:Params: {} 2023-10-18T20:34:22.064 DEBUG:boto:Token: None 2023-10-18T20:34:22.064 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.065 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d0ae21896d99cac-006530414e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.068 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:34:22.069 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.070 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.070 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.071 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.071 DEBUG:boto:Method: GET 2023-10-18T20:34:22.072 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.072 DEBUG:boto:Data: 2023-10-18T20:34:22.072 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.073 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.074 DEBUG:boto:Params: {} 2023-10-18T20:34:22.074 DEBUG:boto:Token: None 2023-10-18T20:34:22.075 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.075 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NaHyZjX3io4CtYYk/ZjD7GcNf2I= 2023-10-18T20:34:22.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NaHyZjX3io4CtYYk/ZjD7GcNf2I=', 'Content-Length': '0'} 2023-10-18T20:34:22.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c98ad434ccbb93e4-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.079 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.079 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.080 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.080 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.081 DEBUG:boto:Method: GET 2023-10-18T20:34:22.081 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.081 DEBUG:boto:Data: 2023-10-18T20:34:22.082 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.083 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.083 DEBUG:boto:Params: {} 2023-10-18T20:34:22.084 DEBUG:boto:Token: None 2023-10-18T20:34:22.084 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NaHyZjX3io4CtYYk/ZjD7GcNf2I= 2023-10-18T20:34:22.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NaHyZjX3io4CtYYk/ZjD7GcNf2I=', 'Content-Length': '0'} 2023-10-18T20:34:22.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e407fe1a016b28b-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.131 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.132 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.132 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.132 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.133 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.133 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.134 DEBUG:boto:Data: 2023-10-18T20:34:22.134 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.134 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.135 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.135 DEBUG:boto:Params: {} 2023-10-18T20:34:22.136 DEBUG:boto:Token: None 2023-10-18T20:34:22.136 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.137 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dn8FLInletB8gt8TgXSnwSFTxLM= 2023-10-18T20:34:22.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dn8FLInletB8gt8TgXSnwSFTxLM=', 'Content-Length': '0'} 2023-10-18T20:34:22.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ee664aa34473b5-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.140 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.140 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.141 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.142 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.142 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.143 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.143 DEBUG:boto:Data: 2023-10-18T20:34:22.143 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.144 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.144 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.145 DEBUG:boto:Params: {} 2023-10-18T20:34:22.145 DEBUG:boto:Token: None 2023-10-18T20:34:22.146 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.146 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dn8FLInletB8gt8TgXSnwSFTxLM= 2023-10-18T20:34:22.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dn8FLInletB8gt8TgXSnwSFTxLM=', 'Content-Length': '0'} 2023-10-18T20:34:22.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a21e6ec16f7b7a44-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:34:22.150 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.150 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.151 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.151 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.152 DEBUG:boto:Method: GET 2023-10-18T20:34:22.152 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.153 DEBUG:boto:Data: 2023-10-18T20:34:22.153 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.154 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.154 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.154 DEBUG:boto:Params: {} 2023-10-18T20:34:22.155 DEBUG:boto:Token: None 2023-10-18T20:34:22.155 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.156 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NaHyZjX3io4CtYYk/ZjD7GcNf2I= 2023-10-18T20:34:22.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NaHyZjX3io4CtYYk/ZjD7GcNf2I=', 'Content-Length': '0'} 2023-10-18T20:34:22.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079abd7b11b0df2db-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.159 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.160 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.160 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.161 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.161 DEBUG:boto:Method: GET 2023-10-18T20:34:22.161 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.162 DEBUG:boto:Data: 2023-10-18T20:34:22.162 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.163 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.163 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.164 DEBUG:boto:Params: {} 2023-10-18T20:34:22.164 DEBUG:boto:Token: None 2023-10-18T20:34:22.165 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:34:22.165 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NaHyZjX3io4CtYYk/ZjD7GcNf2I= 2023-10-18T20:34:22.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NaHyZjX3io4CtYYk/ZjD7GcNf2I=', 'Content-Length': '0'} 2023-10-18T20:34:22.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f8ba71634e04904-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.169 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.169 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.170 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.170 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.170 DEBUG:boto:Method: GET 2023-10-18T20:34:22.171 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.171 DEBUG:boto:Data: 2023-10-18T20:34:22.172 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.172 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.173 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.173 DEBUG:boto:Params: {} 2023-10-18T20:34:22.173 DEBUG:boto:Token: None 2023-10-18T20:34:22.174 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.175 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4a55851b400f8b9-006530414e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.226 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.227 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.228 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.228 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.228 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.229 DEBUG:boto:Method: GET 2023-10-18T20:34:22.229 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.230 DEBUG:boto:Data: 2023-10-18T20:34:22.230 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.231 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.231 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.232 DEBUG:boto:Params: {} 2023-10-18T20:34:22.233 DEBUG:boto:Token: None 2023-10-18T20:34:22.234 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.234 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000955edba537d39319-006530414e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.238 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.239 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.240 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.240 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.241 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:34:22.241 DEBUG:boto:Data: 2023-10-18T20:34:22.241 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.242 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.243 DEBUG:boto:Params: {} 2023-10-18T20:34:22.243 DEBUG:boto:Token: None 2023-10-18T20:34:22.244 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44 2023-10-18T20:34:22.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XoyVzQ/jbBrJWooD2bSDNnP/5yk= 2023-10-18T20:34:22.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XoyVzQ/jbBrJWooD2bSDNnP/5yk=', 'Content-Length': '0'} 2023-10-18T20:34:22.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cc016e07b3bcd1f-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.247 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.248 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.248 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.249 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:34:22.249 DEBUG:boto:Data: 2023-10-18T20:34:22.249 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.250 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.250 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.251 DEBUG:boto:Params: {} 2023-10-18T20:34:22.251 DEBUG:boto:Token: None 2023-10-18T20:34:22.251 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44 2023-10-18T20:34:22.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XoyVzQ/jbBrJWooD2bSDNnP/5yk= 2023-10-18T20:34:22.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XoyVzQ/jbBrJWooD2bSDNnP/5yk=', 'Content-Length': '0'} 2023-10-18T20:34:22.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e754de654c7e635-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:34:22.255 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.255 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.256 DEBUG:boto:Method: GET 2023-10-18T20:34:22.256 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:34:22.257 DEBUG:boto:Data: 2023-10-18T20:34:22.257 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.257 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.258 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.258 DEBUG:boto:Params: {} 2023-10-18T20:34:22.259 DEBUG:boto:Token: None 2023-10-18T20:34:22.259 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44 2023-10-18T20:34:22.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7PefOa3k63xa+r7bMn+c1Nb0UAs= 2023-10-18T20:34:22.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7PefOa3k63xa+r7bMn+c1Nb0UAs=', 'Content-Length': '0'} 2023-10-18T20:34:22.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8ba44a0171b50b-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.263 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:34:22.263 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:34:22.264 DEBUG:boto:Method: GET 2023-10-18T20:34:22.264 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:34:22.264 DEBUG:boto:Data: 2023-10-18T20:34:22.265 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.265 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.265 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.266 DEBUG:boto:Params: {} 2023-10-18T20:34:22.266 DEBUG:boto:Token: None 2023-10-18T20:34:22.267 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj44 2023-10-18T20:34:22.267 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7PefOa3k63xa+r7bMn+c1Nb0UAs= 2023-10-18T20:34:22.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7PefOa3k63xa+r7bMn+c1Nb0UAs=', 'Content-Length': '0'} 2023-10-18T20:34:22.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e05e399912cad7f-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.271 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.271 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.271 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.272 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.272 DEBUG:boto:Method: GET 2023-10-18T20:34:22.273 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.273 DEBUG:boto:Data: 2023-10-18T20:34:22.273 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.274 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.274 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.275 DEBUG:boto:Params: {} 2023-10-18T20:34:22.275 DEBUG:boto:Token: None 2023-10-18T20:34:22.276 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001670cdc935e9f16a-006530414e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.279 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.280 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.281 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.281 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.281 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.282 DEBUG:boto:Method: GET 2023-10-18T20:34:22.282 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:34:22.283 DEBUG:boto:Data: 2023-10-18T20:34:22.283 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.284 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.284 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.284 DEBUG:boto:Params: {} 2023-10-18T20:34:22.285 DEBUG:boto:Token: None 2023-10-18T20:34:22.285 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.286 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd10497d1c5397f6-006530414e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.289 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:34:22.290 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.291 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.291 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.292 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.292 DEBUG:boto:Method: GET 2023-10-18T20:34:22.293 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.293 DEBUG:boto:Data: 2023-10-18T20:34:22.293 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.294 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.295 DEBUG:boto:Params: {} 2023-10-18T20:34:22.295 DEBUG:boto:Token: None 2023-10-18T20:34:22.296 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.296 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uh8bRP0Epd5RB3Ieew9hN2DMRzc= 2023-10-18T20:34:22.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uh8bRP0Epd5RB3Ieew9hN2DMRzc=', 'Content-Length': '0'} 2023-10-18T20:34:22.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d26ef7c99cff9c0-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.342 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.343 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.343 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.344 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.344 DEBUG:boto:Method: GET 2023-10-18T20:34:22.345 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.345 DEBUG:boto:Data: 2023-10-18T20:34:22.345 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.346 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.347 DEBUG:boto:Params: {} 2023-10-18T20:34:22.347 DEBUG:boto:Token: None 2023-10-18T20:34:22.348 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uh8bRP0Epd5RB3Ieew9hN2DMRzc= 2023-10-18T20:34:22.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uh8bRP0Epd5RB3Ieew9hN2DMRzc=', 'Content-Length': '0'} 2023-10-18T20:34:22.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e730ee9ede43d497-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.351 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.352 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.352 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.353 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.353 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.353 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.354 DEBUG:boto:Data: 2023-10-18T20:34:22.354 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.355 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.355 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.355 DEBUG:boto:Params: {} 2023-10-18T20:34:22.356 DEBUG:boto:Token: None 2023-10-18T20:34:22.356 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.357 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MdPVyPaAuVvRk6eRsrNtRDo67W8= 2023-10-18T20:34:22.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MdPVyPaAuVvRk6eRsrNtRDo67W8=', 'Content-Length': '0'} 2023-10-18T20:34:22.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028247fde47fcbff-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.359 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.360 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.360 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.360 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.361 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.361 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.361 DEBUG:boto:Data: 2023-10-18T20:34:22.362 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.362 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.363 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.363 DEBUG:boto:Params: {} 2023-10-18T20:34:22.363 DEBUG:boto:Token: None 2023-10-18T20:34:22.364 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MdPVyPaAuVvRk6eRsrNtRDo67W8= 2023-10-18T20:34:22.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MdPVyPaAuVvRk6eRsrNtRDo67W8=', 'Content-Length': '0'} 2023-10-18T20:34:22.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6bf8ab4b75b2de7-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:34:22.368 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.368 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.368 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.369 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.369 DEBUG:boto:Method: GET 2023-10-18T20:34:22.370 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.370 DEBUG:boto:Data: 2023-10-18T20:34:22.370 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.371 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.371 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.372 DEBUG:boto:Params: {} 2023-10-18T20:34:22.372 DEBUG:boto:Token: None 2023-10-18T20:34:22.373 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uh8bRP0Epd5RB3Ieew9hN2DMRzc= 2023-10-18T20:34:22.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uh8bRP0Epd5RB3Ieew9hN2DMRzc=', 'Content-Length': '0'} 2023-10-18T20:34:22.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a60c212636c787-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.376 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.377 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.377 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.378 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.378 DEBUG:boto:Method: GET 2023-10-18T20:34:22.378 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.379 DEBUG:boto:Data: 2023-10-18T20:34:22.379 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.380 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.380 DEBUG:boto:Params: {} 2023-10-18T20:34:22.381 DEBUG:boto:Token: None 2023-10-18T20:34:22.381 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:34:22.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uh8bRP0Epd5RB3Ieew9hN2DMRzc= 2023-10-18T20:34:22.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uh8bRP0Epd5RB3Ieew9hN2DMRzc=', 'Content-Length': '0'} 2023-10-18T20:34:22.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de268109eb43f39-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.426 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.427 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.428 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.428 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.429 DEBUG:boto:Method: GET 2023-10-18T20:34:22.429 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.430 DEBUG:boto:Data: 2023-10-18T20:34:22.430 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.430 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.431 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.432 DEBUG:boto:Params: {} 2023-10-18T20:34:22.432 DEBUG:boto:Token: None 2023-10-18T20:34:22.433 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.433 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067fcc48667a35549-006530414e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.437 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.438 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.438 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.439 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.439 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.440 DEBUG:boto:Method: GET 2023-10-18T20:34:22.440 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.441 DEBUG:boto:Data: 2023-10-18T20:34:22.441 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.442 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.443 DEBUG:boto:Params: {} 2023-10-18T20:34:22.443 DEBUG:boto:Token: None 2023-10-18T20:34:22.444 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.444 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050d19cda9fdc8027-006530414e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.448 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.449 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.449 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.450 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.450 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:34:22.451 DEBUG:boto:Data: 2023-10-18T20:34:22.451 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.452 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.453 DEBUG:boto:Params: {} 2023-10-18T20:34:22.453 DEBUG:boto:Token: None 2023-10-18T20:34:22.454 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51 2023-10-18T20:34:22.454 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jx5GASB520486/mprj9W1gVjWM0= 2023-10-18T20:34:22.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jx5GASB520486/mprj9W1gVjWM0=', 'Content-Length': '0'} 2023-10-18T20:34:22.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e98bef641e6837-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.458 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.458 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.459 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.459 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:34:22.460 DEBUG:boto:Data: 2023-10-18T20:34:22.460 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.460 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.461 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.461 DEBUG:boto:Params: {} 2023-10-18T20:34:22.462 DEBUG:boto:Token: None 2023-10-18T20:34:22.462 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51 2023-10-18T20:34:22.463 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jx5GASB520486/mprj9W1gVjWM0= 2023-10-18T20:34:22.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jx5GASB520486/mprj9W1gVjWM0=', 'Content-Length': '0'} 2023-10-18T20:34:22.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece5107d31aa8042-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:34:22.467 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.467 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.467 DEBUG:boto:Method: GET 2023-10-18T20:34:22.468 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:34:22.469 DEBUG:boto:Data: 2023-10-18T20:34:22.469 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.470 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.471 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.471 DEBUG:boto:Params: {} 2023-10-18T20:34:22.472 DEBUG:boto:Token: None 2023-10-18T20:34:22.473 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51 2023-10-18T20:34:22.474 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:96+bGDpPcdEm1qq7dFtJwyQHofI= 2023-10-18T20:34:22.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:96+bGDpPcdEm1qq7dFtJwyQHofI=', 'Content-Length': '0'} 2023-10-18T20:34:22.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a04aab710b531345-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.477 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:34:22.478 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:34:22.479 DEBUG:boto:Method: GET 2023-10-18T20:34:22.480 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:34:22.480 DEBUG:boto:Data: 2023-10-18T20:34:22.481 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.482 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.482 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.483 DEBUG:boto:Params: {} 2023-10-18T20:34:22.484 DEBUG:boto:Token: None 2023-10-18T20:34:22.484 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj51 2023-10-18T20:34:22.485 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:96+bGDpPcdEm1qq7dFtJwyQHofI= 2023-10-18T20:34:22.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:96+bGDpPcdEm1qq7dFtJwyQHofI=', 'Content-Length': '0'} 2023-10-18T20:34:22.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cacb835c430f6d18-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.489 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.490 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.490 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.491 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.491 DEBUG:boto:Method: GET 2023-10-18T20:34:22.492 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.492 DEBUG:boto:Data: 2023-10-18T20:34:22.492 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.493 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.493 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.494 DEBUG:boto:Params: {} 2023-10-18T20:34:22.494 DEBUG:boto:Token: None 2023-10-18T20:34:22.495 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.495 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008acbe33e743e194-006530414e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.499 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.499 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.500 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.500 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.501 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.501 DEBUG:boto:Method: GET 2023-10-18T20:34:22.502 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:34:22.502 DEBUG:boto:Data: 2023-10-18T20:34:22.502 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.503 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.504 DEBUG:boto:Params: {} 2023-10-18T20:34:22.504 DEBUG:boto:Token: None 2023-10-18T20:34:22.505 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ece0649f4c662b2-006530414e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.509 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:34:22.511 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.511 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.511 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.512 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.512 DEBUG:boto:Method: GET 2023-10-18T20:34:22.513 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.513 DEBUG:boto:Data: 2023-10-18T20:34:22.513 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.514 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.514 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.515 DEBUG:boto:Params: {} 2023-10-18T20:34:22.515 DEBUG:boto:Token: None 2023-10-18T20:34:22.516 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/MNvzXJ+zji6DaRegy8akiolSwE= 2023-10-18T20:34:22.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/MNvzXJ+zji6DaRegy8akiolSwE=', 'Content-Length': '0'} 2023-10-18T20:34:22.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000796b3596f56b876f-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.562 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.563 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.563 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.564 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.564 DEBUG:boto:Method: GET 2023-10-18T20:34:22.564 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.565 DEBUG:boto:Data: 2023-10-18T20:34:22.565 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.566 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.567 DEBUG:boto:Params: {} 2023-10-18T20:34:22.567 DEBUG:boto:Token: None 2023-10-18T20:34:22.568 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/MNvzXJ+zji6DaRegy8akiolSwE= 2023-10-18T20:34:22.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/MNvzXJ+zji6DaRegy8akiolSwE=', 'Content-Length': '0'} 2023-10-18T20:34:22.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a705b899b520a161-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.571 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.571 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.572 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.572 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.573 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.573 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.573 DEBUG:boto:Data: 2023-10-18T20:34:22.574 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.574 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.574 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.575 DEBUG:boto:Params: {} 2023-10-18T20:34:22.575 DEBUG:boto:Token: None 2023-10-18T20:34:22.576 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RWi8V4zbwizu0QaazE6xfuiLjbA= 2023-10-18T20:34:22.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RWi8V4zbwizu0QaazE6xfuiLjbA=', 'Content-Length': '0'} 2023-10-18T20:34:22.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000273c4382adf7f6c3-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.579 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.579 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.580 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.580 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.580 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.581 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.581 DEBUG:boto:Data: 2023-10-18T20:34:22.582 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.582 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.583 DEBUG:boto:Params: {} 2023-10-18T20:34:22.583 DEBUG:boto:Token: None 2023-10-18T20:34:22.584 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.584 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RWi8V4zbwizu0QaazE6xfuiLjbA= 2023-10-18T20:34:22.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RWi8V4zbwizu0QaazE6xfuiLjbA=', 'Content-Length': '0'} 2023-10-18T20:34:22.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072970d2e6b6a58b9-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:34:22.587 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.588 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.588 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.589 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.589 DEBUG:boto:Method: GET 2023-10-18T20:34:22.590 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.590 DEBUG:boto:Data: 2023-10-18T20:34:22.591 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.591 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.591 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.592 DEBUG:boto:Params: {} 2023-10-18T20:34:22.592 DEBUG:boto:Token: None 2023-10-18T20:34:22.593 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.593 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/MNvzXJ+zji6DaRegy8akiolSwE= 2023-10-18T20:34:22.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/MNvzXJ+zji6DaRegy8akiolSwE=', 'Content-Length': '0'} 2023-10-18T20:34:22.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057272be6bccb45c4-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.597 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.597 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.598 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.598 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.598 DEBUG:boto:Method: GET 2023-10-18T20:34:22.599 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.599 DEBUG:boto:Data: 2023-10-18T20:34:22.600 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.600 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.601 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.601 DEBUG:boto:Params: {} 2023-10-18T20:34:22.601 DEBUG:boto:Token: None 2023-10-18T20:34:22.602 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:34:22.602 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/MNvzXJ+zji6DaRegy8akiolSwE= 2023-10-18T20:34:22.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/MNvzXJ+zji6DaRegy8akiolSwE=', 'Content-Length': '0'} 2023-10-18T20:34:22.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ad52c1fd7348504-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.646 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.647 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.647 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.648 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.648 DEBUG:boto:Method: GET 2023-10-18T20:34:22.649 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.649 DEBUG:boto:Data: 2023-10-18T20:34:22.649 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.650 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.650 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.651 DEBUG:boto:Params: {} 2023-10-18T20:34:22.651 DEBUG:boto:Token: None 2023-10-18T20:34:22.651 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4dd3c72dde5cd7c-006530414e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.656 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.656 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.657 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.657 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.658 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.658 DEBUG:boto:Method: GET 2023-10-18T20:34:22.659 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.659 DEBUG:boto:Data: 2023-10-18T20:34:22.659 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.660 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.660 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.661 DEBUG:boto:Params: {} 2023-10-18T20:34:22.661 DEBUG:boto:Token: None 2023-10-18T20:34:22.662 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003306d778c03deb9d-006530414e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.667 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.667 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.668 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.668 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.669 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:34:22.669 DEBUG:boto:Data: 2023-10-18T20:34:22.670 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.670 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.670 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.671 DEBUG:boto:Params: {} 2023-10-18T20:34:22.671 DEBUG:boto:Token: None 2023-10-18T20:34:22.672 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55 2023-10-18T20:34:22.672 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OUdiM3r44SEAJgWTbrnXBy/jgJE= 2023-10-18T20:34:22.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OUdiM3r44SEAJgWTbrnXBy/jgJE=', 'Content-Length': '0'} 2023-10-18T20:34:22.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004477fe4531ce9f1-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.675 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.675 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.676 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.676 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:34:22.676 DEBUG:boto:Data: 2023-10-18T20:34:22.677 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.677 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.678 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.678 DEBUG:boto:Params: {} 2023-10-18T20:34:22.678 DEBUG:boto:Token: None 2023-10-18T20:34:22.679 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55 2023-10-18T20:34:22.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OUdiM3r44SEAJgWTbrnXBy/jgJE= 2023-10-18T20:34:22.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OUdiM3r44SEAJgWTbrnXBy/jgJE=', 'Content-Length': '0'} 2023-10-18T20:34:22.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e230bab851b78d1-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:34:22.683 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.683 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.684 DEBUG:boto:Method: GET 2023-10-18T20:34:22.684 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:34:22.684 DEBUG:boto:Data: 2023-10-18T20:34:22.685 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.685 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.686 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.686 DEBUG:boto:Params: {} 2023-10-18T20:34:22.686 DEBUG:boto:Token: None 2023-10-18T20:34:22.687 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55 2023-10-18T20:34:22.687 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RFWH3qdLy6u7IxjpVwYe6zNXAoo= 2023-10-18T20:34:22.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RFWH3qdLy6u7IxjpVwYe6zNXAoo=', 'Content-Length': '0'} 2023-10-18T20:34:22.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000866be30a906055f5-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.730 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:34:22.731 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:34:22.731 DEBUG:boto:Method: GET 2023-10-18T20:34:22.732 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:34:22.732 DEBUG:boto:Data: 2023-10-18T20:34:22.732 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.733 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.734 DEBUG:boto:Params: {} 2023-10-18T20:34:22.734 DEBUG:boto:Token: None 2023-10-18T20:34:22.734 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj55 2023-10-18T20:34:22.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RFWH3qdLy6u7IxjpVwYe6zNXAoo= 2023-10-18T20:34:22.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RFWH3qdLy6u7IxjpVwYe6zNXAoo=', 'Content-Length': '0'} 2023-10-18T20:34:22.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a5c5acf412a791c-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.738 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.739 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.739 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.739 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.740 DEBUG:boto:Method: GET 2023-10-18T20:34:22.740 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.741 DEBUG:boto:Data: 2023-10-18T20:34:22.741 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.741 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.742 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.742 DEBUG:boto:Params: {} 2023-10-18T20:34:22.743 DEBUG:boto:Token: None 2023-10-18T20:34:22.743 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.744 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000274a4a2e5f702a3-006530414e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.747 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.748 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.748 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.749 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.749 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.749 DEBUG:boto:Method: GET 2023-10-18T20:34:22.750 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:34:22.750 DEBUG:boto:Data: 2023-10-18T20:34:22.750 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.751 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.751 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.752 DEBUG:boto:Params: {} 2023-10-18T20:34:22.752 DEBUG:boto:Token: None 2023-10-18T20:34:22.752 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.753 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004039baee9b595cea-006530414e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.757 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:34:22.758 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.758 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.759 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.759 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.760 DEBUG:boto:Method: GET 2023-10-18T20:34:22.760 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.760 DEBUG:boto:Data: 2023-10-18T20:34:22.761 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.762 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.762 DEBUG:boto:Params: {} 2023-10-18T20:34:22.763 DEBUG:boto:Token: None 2023-10-18T20:34:22.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mWQJWOn6mdF59iq3UF4woZVixT0= 2023-10-18T20:34:22.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mWQJWOn6mdF59iq3UF4woZVixT0=', 'Content-Length': '0'} 2023-10-18T20:34:22.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069bd984795bc4fdf-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.767 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.767 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.768 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.768 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.769 DEBUG:boto:Method: GET 2023-10-18T20:34:22.769 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.769 DEBUG:boto:Data: 2023-10-18T20:34:22.770 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.770 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.771 DEBUG:boto:Params: {} 2023-10-18T20:34:22.771 DEBUG:boto:Token: None 2023-10-18T20:34:22.772 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mWQJWOn6mdF59iq3UF4woZVixT0= 2023-10-18T20:34:22.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mWQJWOn6mdF59iq3UF4woZVixT0=', 'Content-Length': '0'} 2023-10-18T20:34:22.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bf3c703334f645e-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.818 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.819 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.819 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.819 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.820 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.820 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.821 DEBUG:boto:Data: 2023-10-18T20:34:22.821 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.821 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.822 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.822 DEBUG:boto:Params: {} 2023-10-18T20:34:22.823 DEBUG:boto:Token: None 2023-10-18T20:34:22.823 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.823 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HXe7S0gYkyiPer60NMr1Ql4BBLg= 2023-10-18T20:34:22.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HXe7S0gYkyiPer60NMr1Ql4BBLg=', 'Content-Length': '0'} 2023-10-18T20:34:22.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cda33805f6dca727-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.826 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.827 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.827 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.827 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.828 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.828 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.829 DEBUG:boto:Data: 2023-10-18T20:34:22.829 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.829 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.830 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.830 DEBUG:boto:Params: {} 2023-10-18T20:34:22.830 DEBUG:boto:Token: None 2023-10-18T20:34:22.834 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.835 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HXe7S0gYkyiPer60NMr1Ql4BBLg= 2023-10-18T20:34:22.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HXe7S0gYkyiPer60NMr1Ql4BBLg=', 'Content-Length': '0'} 2023-10-18T20:34:22.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a453c86adfc92407-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:34:22.838 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.839 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.839 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.840 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.840 DEBUG:boto:Method: GET 2023-10-18T20:34:22.840 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.841 DEBUG:boto:Data: 2023-10-18T20:34:22.841 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.842 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.842 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.843 DEBUG:boto:Params: {} 2023-10-18T20:34:22.843 DEBUG:boto:Token: None 2023-10-18T20:34:22.843 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.844 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mWQJWOn6mdF59iq3UF4woZVixT0= 2023-10-18T20:34:22.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mWQJWOn6mdF59iq3UF4woZVixT0=', 'Content-Length': '0'} 2023-10-18T20:34:22.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000920205a03d792334-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.847 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.848 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.848 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.848 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.849 DEBUG:boto:Method: GET 2023-10-18T20:34:22.849 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.850 DEBUG:boto:Data: 2023-10-18T20:34:22.850 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.850 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.851 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.851 DEBUG:boto:Params: {} 2023-10-18T20:34:22.851 DEBUG:boto:Token: None 2023-10-18T20:34:22.852 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:34:22.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mWQJWOn6mdF59iq3UF4woZVixT0= 2023-10-18T20:34:22.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mWQJWOn6mdF59iq3UF4woZVixT0=', 'Content-Length': '0'} 2023-10-18T20:34:22.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000965b5f21b81a4b99-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.855 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.856 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.856 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.857 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.857 DEBUG:boto:Method: GET 2023-10-18T20:34:22.857 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.858 DEBUG:boto:Data: 2023-10-18T20:34:22.858 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.859 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.860 DEBUG:boto:Params: {} 2023-10-18T20:34:22.860 DEBUG:boto:Token: None 2023-10-18T20:34:22.861 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.861 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020e9bbbc9ea25f0b-006530414e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.906 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.907 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.908 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.908 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.908 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.909 DEBUG:boto:Method: GET 2023-10-18T20:34:22.909 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.909 DEBUG:boto:Data: 2023-10-18T20:34:22.910 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.910 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.911 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.911 DEBUG:boto:Params: {} 2023-10-18T20:34:22.911 DEBUG:boto:Token: None 2023-10-18T20:34:22.912 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.912 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ced9ff21d8958a7e-006530414e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.917 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.917 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.918 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.918 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.919 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:34:22.919 DEBUG:boto:Data: 2023-10-18T20:34:22.919 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.920 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.920 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.920 DEBUG:boto:Params: {} 2023-10-18T20:34:22.921 DEBUG:boto:Token: None 2023-10-18T20:34:22.921 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68 2023-10-18T20:34:22.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9KU0VTOi1wToGGM/5ZbfiM/V/sc= 2023-10-18T20:34:22.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9KU0VTOi1wToGGM/5ZbfiM/V/sc=', 'Content-Length': '0'} 2023-10-18T20:34:22.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dd6f8a681626a11-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.925 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.926 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.926 DEBUG:boto:Method: HEAD 2023-10-18T20:34:22.927 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:34:22.927 DEBUG:boto:Data: 2023-10-18T20:34:22.928 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.930 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.930 DEBUG:boto:Params: {} 2023-10-18T20:34:22.931 DEBUG:boto:Token: None 2023-10-18T20:34:22.932 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68 2023-10-18T20:34:22.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9KU0VTOi1wToGGM/5ZbfiM/V/sc= 2023-10-18T20:34:22.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9KU0VTOi1wToGGM/5ZbfiM/V/sc=', 'Content-Length': '0'} 2023-10-18T20:34:22.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000821c073d73d09fa1-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:34:22.938 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.939 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.942 DEBUG:boto:Method: GET 2023-10-18T20:34:22.942 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:34:22.943 DEBUG:boto:Data: 2023-10-18T20:34:22.944 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.944 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.945 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.946 DEBUG:boto:Params: {} 2023-10-18T20:34:22.946 DEBUG:boto:Token: None 2023-10-18T20:34:22.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68 2023-10-18T20:34:22.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:roHpo/AJ5GWzblwhZvOTf8CPmek= 2023-10-18T20:34:22.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:roHpo/AJ5GWzblwhZvOTf8CPmek=', 'Content-Length': '0'} 2023-10-18T20:34:22.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a032336efed71af2-006530414e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.952 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:34:22.953 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:34:22.953 DEBUG:boto:Method: GET 2023-10-18T20:34:22.954 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:34:22.955 DEBUG:boto:Data: 2023-10-18T20:34:22.956 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.956 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.957 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.958 DEBUG:boto:Params: {} 2023-10-18T20:34:22.959 DEBUG:boto:Token: None 2023-10-18T20:34:22.959 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/obj68 2023-10-18T20:34:22.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:roHpo/AJ5GWzblwhZvOTf8CPmek= 2023-10-18T20:34:22.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:roHpo/AJ5GWzblwhZvOTf8CPmek=', 'Content-Length': '0'} 2023-10-18T20:34:22.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b036460ab5a1b67-006530414e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.963 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.964 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.964 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.965 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.965 DEBUG:boto:Method: GET 2023-10-18T20:34:22.966 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.967 DEBUG:boto:Data: 2023-10-18T20:34:22.967 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.968 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:22.969 DEBUG:boto:Port: 8001 2023-10-18T20:34:22.970 DEBUG:boto:Params: {} 2023-10-18T20:34:22.970 DEBUG:boto:Token: None 2023-10-18T20:34:22.971 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.972 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab8417ac2c59e15-006530414e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:22.977 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:22.978 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:22.978 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:22.979 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.979 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.979 DEBUG:boto:Method: GET 2023-10-18T20:34:22.980 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:34:22.980 DEBUG:boto:Data: 2023-10-18T20:34:22.981 DEBUG:boto:Headers: {} 2023-10-18T20:34:22.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:22.981 DEBUG:boto:Port: 8000 2023-10-18T20:34:22.982 DEBUG:boto:Params: {} 2023-10-18T20:34:22.982 DEBUG:boto:Token: None 2023-10-18T20:34:22.983 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:22 GMT /bofrnf-59/ 2023-10-18T20:34:22.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8= 2023-10-18T20:34:22.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8B65aatIedfbUMj/iNZKwhq5ZF8=', 'Content-Length': '0'} 2023-10-18T20:34:22.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7aa456038bb38fa-006530414e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.030 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:23.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:34:23.031 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.032 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.032 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.033 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.033 DEBUG:boto:Method: GET 2023-10-18T20:34:23.033 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.034 DEBUG:boto:Data: 2023-10-18T20:34:23.034 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.035 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:23.035 DEBUG:boto:Port: 8001 2023-10-18T20:34:23.036 DEBUG:boto:Params: {} 2023-10-18T20:34:23.036 DEBUG:boto:Token: None 2023-10-18T20:34:23.037 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.037 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DZPFnP3FK6k2swgxIPcN9emaK3Q= 2023-10-18T20:34:23.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DZPFnP3FK6k2swgxIPcN9emaK3Q=', 'Content-Length': '0'} 2023-10-18T20:34:23.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bab31e7b2062cee5-006530414f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.041 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.041 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.042 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.042 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.042 DEBUG:boto:Method: GET 2023-10-18T20:34:23.043 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.043 DEBUG:boto:Data: 2023-10-18T20:34:23.044 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.044 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.045 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.045 DEBUG:boto:Params: {} 2023-10-18T20:34:23.045 DEBUG:boto:Token: None 2023-10-18T20:34:23.046 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.046 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DZPFnP3FK6k2swgxIPcN9emaK3Q= 2023-10-18T20:34:23.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DZPFnP3FK6k2swgxIPcN9emaK3Q=', 'Content-Length': '0'} 2023-10-18T20:34:23.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044e3b703f07b406a-006530414f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.049 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.050 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.050 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.051 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.051 DEBUG:boto:Method: HEAD 2023-10-18T20:34:23.051 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.052 DEBUG:boto:Data: 2023-10-18T20:34:23.052 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.053 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:23.053 DEBUG:boto:Port: 8001 2023-10-18T20:34:23.053 DEBUG:boto:Params: {} 2023-10-18T20:34:23.054 DEBUG:boto:Token: None 2023-10-18T20:34:23.054 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.055 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oHOYpUklkPA1j+0UA1mjRCIbevI= 2023-10-18T20:34:23.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oHOYpUklkPA1j+0UA1mjRCIbevI=', 'Content-Length': '0'} 2023-10-18T20:34:23.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b95774ded418b0c3-006530414f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.058 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.059 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.059 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.059 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.060 DEBUG:boto:Method: HEAD 2023-10-18T20:34:23.060 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.060 DEBUG:boto:Data: 2023-10-18T20:34:23.061 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.061 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.062 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.062 DEBUG:boto:Params: {} 2023-10-18T20:34:23.062 DEBUG:boto:Token: None 2023-10-18T20:34:23.063 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.063 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oHOYpUklkPA1j+0UA1mjRCIbevI= 2023-10-18T20:34:23.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oHOYpUklkPA1j+0UA1mjRCIbevI=', 'Content-Length': '0'} 2023-10-18T20:34:23.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce7c454f478ac0e-006530414f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:34:23.066 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.067 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.067 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.068 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.068 DEBUG:boto:Method: GET 2023-10-18T20:34:23.068 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.069 DEBUG:boto:Data: 2023-10-18T20:34:23.069 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.070 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:23.070 DEBUG:boto:Port: 8001 2023-10-18T20:34:23.070 DEBUG:boto:Params: {} 2023-10-18T20:34:23.071 DEBUG:boto:Token: None 2023-10-18T20:34:23.071 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.072 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DZPFnP3FK6k2swgxIPcN9emaK3Q= 2023-10-18T20:34:23.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DZPFnP3FK6k2swgxIPcN9emaK3Q=', 'Content-Length': '0'} 2023-10-18T20:34:23.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c3c8a53044ad061-006530414f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.075 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.075 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.075 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.076 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.076 DEBUG:boto:Method: GET 2023-10-18T20:34:23.077 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.077 DEBUG:boto:Data: 2023-10-18T20:34:23.077 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.078 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.078 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.079 DEBUG:boto:Params: {} 2023-10-18T20:34:23.079 DEBUG:boto:Token: None 2023-10-18T20:34:23.080 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:34:23.080 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DZPFnP3FK6k2swgxIPcN9emaK3Q= 2023-10-18T20:34:23.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DZPFnP3FK6k2swgxIPcN9emaK3Q=', 'Content-Length': '0'} 2023-10-18T20:34:23.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5186141b2d45e8f-006530414f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.083 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:23.083 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:23.084 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.084 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.085 DEBUG:boto:Method: GET 2023-10-18T20:34:23.085 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.086 DEBUG:boto:Data: 2023-10-18T20:34:23.086 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.086 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:23.087 DEBUG:boto:Port: 8001 2023-10-18T20:34:23.087 DEBUG:boto:Params: {} 2023-10-18T20:34:23.088 DEBUG:boto:Token: None 2023-10-18T20:34:23.088 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/ 2023-10-18T20:34:23.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1OT5v0eysOZ3HjkXtQAs2hS88tA= 2023-10-18T20:34:23.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1OT5v0eysOZ3HjkXtQAs2hS88tA=', 'Content-Length': '0'} 2023-10-18T20:34:23.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4fcd43d51b3eb2d-006530414f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.092 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:23.093 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:23.093 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:23.094 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.094 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.095 DEBUG:boto:Method: GET 2023-10-18T20:34:23.095 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.095 DEBUG:boto:Data: 2023-10-18T20:34:23.096 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.096 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.097 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.097 DEBUG:boto:Params: {} 2023-10-18T20:34:23.098 DEBUG:boto:Token: None 2023-10-18T20:34:23.098 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/ 2023-10-18T20:34:23.099 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1OT5v0eysOZ3HjkXtQAs2hS88tA= 2023-10-18T20:34:23.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1OT5v0eysOZ3HjkXtQAs2hS88tA=', 'Content-Length': '0'} 2023-10-18T20:34:23.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5a95d824dd1e8d0-006530414f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.102 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:23.103 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.103 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.104 DEBUG:boto:Method: HEAD 2023-10-18T20:34:23.104 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:34:23.105 DEBUG:boto:Data: 2023-10-18T20:34:23.105 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.105 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:23.106 DEBUG:boto:Port: 8001 2023-10-18T20:34:23.106 DEBUG:boto:Params: {} 2023-10-18T20:34:23.107 DEBUG:boto:Token: None 2023-10-18T20:34:23.107 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85 2023-10-18T20:34:23.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o9kj1RzYVBuLlNCVhDD57x9FfvE= 2023-10-18T20:34:23.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o9kj1RzYVBuLlNCVhDD57x9FfvE=', 'Content-Length': '0'} 2023-10-18T20:34:23.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11d837b0cd3598a-006530414f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.110 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.111 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.111 DEBUG:boto:Method: HEAD 2023-10-18T20:34:23.112 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:34:23.112 DEBUG:boto:Data: 2023-10-18T20:34:23.112 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.113 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.113 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.114 DEBUG:boto:Params: {} 2023-10-18T20:34:23.114 DEBUG:boto:Token: None 2023-10-18T20:34:23.115 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85 2023-10-18T20:34:23.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o9kj1RzYVBuLlNCVhDD57x9FfvE= 2023-10-18T20:34:23.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o9kj1RzYVBuLlNCVhDD57x9FfvE=', 'Content-Length': '0'} 2023-10-18T20:34:23.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2579a1b2131c50c-006530414f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:34:23.118 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.119 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.119 DEBUG:boto:Method: GET 2023-10-18T20:34:23.120 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:34:23.120 DEBUG:boto:Data: 2023-10-18T20:34:23.121 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.121 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:23.121 DEBUG:boto:Port: 8001 2023-10-18T20:34:23.122 DEBUG:boto:Params: {} 2023-10-18T20:34:23.122 DEBUG:boto:Token: None 2023-10-18T20:34:23.123 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85 2023-10-18T20:34:23.123 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NaR/VwvNUz/f5a1TDOkz4b56e9w= 2023-10-18T20:34:23.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NaR/VwvNUz/f5a1TDOkz4b56e9w=', 'Content-Length': '0'} 2023-10-18T20:34:23.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000407c68af4d891853-006530414f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.166 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:34:23.167 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:34:23.167 DEBUG:boto:Method: GET 2023-10-18T20:34:23.168 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:34:23.168 DEBUG:boto:Data: 2023-10-18T20:34:23.168 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.169 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.169 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.170 DEBUG:boto:Params: {} 2023-10-18T20:34:23.170 DEBUG:boto:Token: None 2023-10-18T20:34:23.170 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj85 2023-10-18T20:34:23.171 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NaR/VwvNUz/f5a1TDOkz4b56e9w= 2023-10-18T20:34:23.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NaR/VwvNUz/f5a1TDOkz4b56e9w=', 'Content-Length': '0'} 2023-10-18T20:34:23.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002953a306be9a2110-006530414f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.174 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:23.174 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:23.175 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.175 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.176 DEBUG:boto:Method: GET 2023-10-18T20:34:23.176 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.176 DEBUG:boto:Data: 2023-10-18T20:34:23.177 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.177 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:34:23.178 DEBUG:boto:Port: 8001 2023-10-18T20:34:23.178 DEBUG:boto:Params: {} 2023-10-18T20:34:23.178 DEBUG:boto:Token: None 2023-10-18T20:34:23.179 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/ 2023-10-18T20:34:23.179 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1OT5v0eysOZ3HjkXtQAs2hS88tA= 2023-10-18T20:34:23.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1OT5v0eysOZ3HjkXtQAs2hS88tA=', 'Content-Length': '0'} 2023-10-18T20:34:23.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef74c0bdf687da8-006530414f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.183 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:23.183 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:23.184 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:23.184 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.184 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.185 DEBUG:boto:Method: GET 2023-10-18T20:34:23.185 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:34:23.186 DEBUG:boto:Data: 2023-10-18T20:34:23.186 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.186 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.187 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.187 DEBUG:boto:Params: {} 2023-10-18T20:34:23.188 DEBUG:boto:Token: None 2023-10-18T20:34:23.188 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/ 2023-10-18T20:34:23.189 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1OT5v0eysOZ3HjkXtQAs2hS88tA= 2023-10-18T20:34:23.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1OT5v0eysOZ3HjkXtQAs2hS88tA=', 'Content-Length': '0'} 2023-10-18T20:34:23.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008427fe9eaadd086a-006530414f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.193 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:34:23.193 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:34:23.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:34:23.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:34:23.195 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:34:23.254 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:34:23.255 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:34:23.304 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:34:23.304 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-59 2023-10-18T20:34:23.311 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 10 2023-10-18T20:34:23.353 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:23.354 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:23.354 DEBUG:boto:Method: HEAD 2023-10-18T20:34:23.355 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:34:23.355 DEBUG:boto:Data: 2023-10-18T20:34:23.355 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.356 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.356 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.356 DEBUG:boto:Params: {} 2023-10-18T20:34:23.357 DEBUG:boto:Token: None 2023-10-18T20:34:23.357 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/ 2023-10-18T20:34:23.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc= 2023-10-18T20:34:23.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc=', 'Content-Length': '0'} 2023-10-18T20:34:23.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030259a1fa8bd6049-006530414f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.363 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:34:23.364 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:34:23.364 DEBUG:boto:Method: PUT 2023-10-18T20:34:23.365 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:34:23.366 DEBUG:boto:Data: 2023-10-18T20:34:23.367 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:34:23.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.368 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.369 DEBUG:boto:Params: {} 2023-10-18T20:34:23.369 DEBUG:boto:Token: None 2023-10-18T20:34:23.370 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj19 2023-10-18T20:34:23.371 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dCHx9NepS1AQeIi9prhBdu+y2No= 2023-10-18T20:34:23.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dCHx9NepS1AQeIi9prhBdu+y2No='} 2023-10-18T20:34:23.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000948bc6947707c856-006530414f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.378 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:23.378 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:23.379 DEBUG:boto:Method: HEAD 2023-10-18T20:34:23.379 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:34:23.380 DEBUG:boto:Data: 2023-10-18T20:34:23.380 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.381 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.382 DEBUG:boto:Params: {} 2023-10-18T20:34:23.382 DEBUG:boto:Token: None 2023-10-18T20:34:23.383 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/ 2023-10-18T20:34:23.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc= 2023-10-18T20:34:23.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc=', 'Content-Length': '0'} 2023-10-18T20:34:23.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d5dab486bb9fc3a-006530414f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.387 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:34:23.388 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:34:23.388 DEBUG:boto:Method: PUT 2023-10-18T20:34:23.388 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:34:23.389 DEBUG:boto:Data: 2023-10-18T20:34:23.389 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:34:23.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.390 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.390 DEBUG:boto:Params: {} 2023-10-18T20:34:23.391 DEBUG:boto:Token: None 2023-10-18T20:34:23.391 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj38 2023-10-18T20:34:23.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mpGE1jvGXJw0NKVhikhXix3G3u8= 2023-10-18T20:34:23.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mpGE1jvGXJw0NKVhikhXix3G3u8='} 2023-10-18T20:34:23.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c316b00b80a81e6c-006530414f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.436 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:23.437 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:23.437 DEBUG:boto:Method: HEAD 2023-10-18T20:34:23.438 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:34:23.439 DEBUG:boto:Data: 2023-10-18T20:34:23.439 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.440 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.441 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.441 DEBUG:boto:Params: {} 2023-10-18T20:34:23.441 DEBUG:boto:Token: None 2023-10-18T20:34:23.442 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/ 2023-10-18T20:34:23.442 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc= 2023-10-18T20:34:23.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc=', 'Content-Length': '0'} 2023-10-18T20:34:23.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005039c8d6c27b6c09-006530414f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.446 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:34:23.447 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:34:23.447 DEBUG:boto:Method: PUT 2023-10-18T20:34:23.448 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:34:23.448 DEBUG:boto:Data: 2023-10-18T20:34:23.448 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:34:23.449 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.449 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.450 DEBUG:boto:Params: {} 2023-10-18T20:34:23.450 DEBUG:boto:Token: None 2023-10-18T20:34:23.451 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj57 2023-10-18T20:34:23.451 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SXwCaHQOCDnA55F2aArgp7y4riM= 2023-10-18T20:34:23.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SXwCaHQOCDnA55F2aArgp7y4riM='} 2023-10-18T20:34:23.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e15a6b9de9386d50-006530414f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.456 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:23.456 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:23.457 DEBUG:boto:Method: HEAD 2023-10-18T20:34:23.457 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:34:23.457 DEBUG:boto:Data: 2023-10-18T20:34:23.458 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.458 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.459 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.459 DEBUG:boto:Params: {} 2023-10-18T20:34:23.460 DEBUG:boto:Token: None 2023-10-18T20:34:23.460 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/ 2023-10-18T20:34:23.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc= 2023-10-18T20:34:23.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc=', 'Content-Length': '0'} 2023-10-18T20:34:23.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008163a8ece6422a1f-006530414f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.464 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:34:23.465 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:34:23.465 DEBUG:boto:Method: PUT 2023-10-18T20:34:23.465 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:34:23.466 DEBUG:boto:Data: 2023-10-18T20:34:23.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:34:23.467 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.467 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.467 DEBUG:boto:Params: {} 2023-10-18T20:34:23.468 DEBUG:boto:Token: None 2023-10-18T20:34:23.468 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj76 2023-10-18T20:34:23.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uw2Qyr2si9vCsXhYSgZC48Ci0mE= 2023-10-18T20:34:23.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uw2Qyr2si9vCsXhYSgZC48Ci0mE='} 2023-10-18T20:34:23.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d7f9890d1f4acd2-006530414f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.515 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:34:23.516 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:34:23.516 DEBUG:boto:Method: HEAD 2023-10-18T20:34:23.516 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:34:23.517 DEBUG:boto:Data: 2023-10-18T20:34:23.517 DEBUG:boto:Headers: {} 2023-10-18T20:34:23.517 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.518 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.518 DEBUG:boto:Params: {} 2023-10-18T20:34:23.519 DEBUG:boto:Token: None 2023-10-18T20:34:23.519 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/ 2023-10-18T20:34:23.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc= 2023-10-18T20:34:23.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NfH90Z/xoWx7aYNMUY5LcjZdODc=', 'Content-Length': '0'} 2023-10-18T20:34:23.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a49785251cf121b6-006530414f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.523 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:34:23.524 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:34:23.524 DEBUG:boto:Method: PUT 2023-10-18T20:34:23.524 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:34:23.525 DEBUG:boto:Data: 2023-10-18T20:34:23.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:34:23.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:34:23.526 DEBUG:boto:Port: 8000 2023-10-18T20:34:23.526 DEBUG:boto:Params: {} 2023-10-18T20:34:23.527 DEBUG:boto:Token: None 2023-10-18T20:34:23.527 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:34:23 GMT /bofrnf-59/obj95 2023-10-18T20:34:23.528 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lCLd/aPhsYoJMYL732evq3HZM64= 2023-10-18T20:34:23.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:34:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lCLd/aPhsYoJMYL732evq3HZM64='} 2023-10-18T20:34:23.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037d74f52d1d52391-006530414f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:34:23.533 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-59 2023-10-18T20:34:23.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:34:23.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:34:23.534 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:34:23.589 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:34:23.590 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:34:23.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:23.621+0000 7fb0ff9c7780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:34:28.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:28.622+0000 7fb0ff9c7780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:34:33.624 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:33.623+0000 7fb0ff9c7780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T20:34:38.626 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:38.625+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:34:38.627 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-18T20:34:38.627 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:34:43.628 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:43.626+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:34:43.628 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-18T20:34:43.629 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:34:48.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:48.627+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:34:48.630 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-18T20:34:48.630 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:34:53.631 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:53.629+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:34:53.632 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-18T20:34:53.632 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:34:58.633 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:34:58.632+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:34:58.634 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:34:58.634 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:03.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:03.633+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:03.635 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:03.635 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:08.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:08.635+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:08.637 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:08.637 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:13.638 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:13.637+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:13.639 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:13.639 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:18.640 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:18.638+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:18.640 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:18.640 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:23.641 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:23.640+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:23.642 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:23.642 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:28.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:28.641+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:28.644 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:28.645 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:33.645 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:33.643+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:33.645 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:33.645 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:38.646 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:38.644+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:38.647 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:38.647 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:43.648 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:43.645+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:43.648 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:43.649 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:48.649 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:48.647+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:48.650 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:48.650 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:53.651 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:53.650+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:53.652 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000001.52.6, , , , , , , , , , , ] 2023-10-18T20:35:53.652 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:35:58.653 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:35:58.651+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:35:58.653 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:35:58.653 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:03.654 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:03.653+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:03.655 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:03.655 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:08.656 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:08.655+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:08.656 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:08.657 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:13.657 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:13.656+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:13.658 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:13.658 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:18.659 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:18.658+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:18.660 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:18.660 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:23.661 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:23.659+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:23.661 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:23.662 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:28.662 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:28.660+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:28.663 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:28.663 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:33.664 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:33.662+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:33.665 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:33.665 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:38.666 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:38.663+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:38.666 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:38.666 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:43.667 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:43.666+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:43.668 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:43.668 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:48.669 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:48.668+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:48.669 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:48.670 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:53.671 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:53.670+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:53.671 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000002.54.6, , , , , , , , , , , ] 2023-10-18T20:36:53.672 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:36:58.673 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:36:58.671+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:36:58.673 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:36:58.674 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:03.674 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:03.673+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:03.675 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:03.675 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:08.676 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:08.674+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:08.676 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:08.676 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:13.677 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:13.676+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:13.678 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:13.678 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:18.679 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:18.677+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:18.679 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:18.680 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:23.681 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:23.678+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:23.682 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:23.682 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:28.682 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:28.680+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:28.683 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:28.683 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:33.684 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:33.682+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:33.684 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:33.685 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:38.685 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:38.684+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:38.686 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:38.686 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:43.687 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:43.686+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:43.687 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:43.688 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:48.689 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:48.687+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:48.689 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:48.689 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:53.690 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:53.689+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:53.691 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:53.691 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:37:58.692 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:37:58.690+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:37:58.692 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:37:58.693 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:38:03.693 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:03.692+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:38:03.694 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:38:03.694 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:38:08.695 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:08.693+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:38:08.695 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:38:08.695 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:38:13.696 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:13.694+0000 7fb0ff9c7780 1 waiting for incremental sync to catch up: 2023-10-18T20:38:13.697 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000003.56.6, , , , , , , , , , , ] 2023-10-18T20:38:13.697 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:38:18.698 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:18.696+0000 7fb0ff9c7780 1 bucket sync caught up with source: 2023-10-18T20:38:18.699 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:38:18.699 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.59.6, 00000000004.58.6, , , , , , , , , , , ] 2023-10-18T20:38:18.699 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:18.696+0000 7fb0ff9c7780 0 bucket checkpoint complete 2023-10-18T20:38:18.716 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-59 2023-10-18T20:38:18.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:38:18.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:38:18.718 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:38:18.780 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:38:18.780 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:38:18.816 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:18.814+0000 7f91d5860780 1 bucket sync caught up with source: 2023-10-18T20:38:18.817 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.757.6, 00000000001.1166.6, 00000000001.1248.6, 00000000001.692.6, 00000000001.751.6, , , 00000000002.232.6, 00000000004.448.6, 00000000003.226.6] 2023-10-18T20:38:18.817 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.757.6, 00000000001.1166.6, 00000000001.1248.6, 00000000001.692.6, 00000000001.751.6, , , 00000000002.232.6, 00000000004.448.6, 00000000003.226.6] 2023-10-18T20:38:18.817 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:18.814+0000 7f91d5860780 0 bucket checkpoint complete 2023-10-18T20:38:18.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:38:18.831 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:18.831 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:18.832 DEBUG:boto:Method: HEAD 2023-10-18T20:38:18.832 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:38:18.833 DEBUG:boto:Data: 2023-10-18T20:38:18.833 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.834 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:18.834 DEBUG:boto:Port: 8001 2023-10-18T20:38:18.834 DEBUG:boto:Params: {} 2023-10-18T20:38:18.835 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:38:18.836 DEBUG:boto:Token: None 2023-10-18T20:38:18.836 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/ 2023-10-18T20:38:18.837 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1azM1O2S0jcVqEjq+AdRrN28n6Y= 2023-10-18T20:38:18.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1azM1O2S0jcVqEjq+AdRrN28n6Y=', 'Content-Length': '0'} 2023-10-18T20:38:18.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc81dd1571aeb743-006530423a-1af4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.843 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:18.843 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:18.844 DEBUG:boto:Method: HEAD 2023-10-18T20:38:18.844 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:38:18.844 DEBUG:boto:Data: 2023-10-18T20:38:18.845 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.845 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:18.846 DEBUG:boto:Port: 8000 2023-10-18T20:38:18.846 DEBUG:boto:Params: {} 2023-10-18T20:38:18.847 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:38:18.847 DEBUG:boto:Token: None 2023-10-18T20:38:18.848 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/ 2023-10-18T20:38:18.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1azM1O2S0jcVqEjq+AdRrN28n6Y= 2023-10-18T20:38:18.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1azM1O2S0jcVqEjq+AdRrN28n6Y=', 'Content-Length': '0'} 2023-10-18T20:38:18.853 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000948a7c8f4dc7d3e3-006530423a-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.853 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:38:18.854 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:18.854 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:18.855 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.855 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.855 DEBUG:boto:Method: GET 2023-10-18T20:38:18.856 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.856 DEBUG:boto:Data: 2023-10-18T20:38:18.856 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.857 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:18.857 DEBUG:boto:Port: 8001 2023-10-18T20:38:18.858 DEBUG:boto:Params: {} 2023-10-18T20:38:18.858 DEBUG:boto:Token: None 2023-10-18T20:38:18.859 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/?versions 2023-10-18T20:38:18.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kzYgb0Jbit9vy97WTsX1pDzxU8s= 2023-10-18T20:38:18.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kzYgb0Jbit9vy97WTsX1pDzxU8s=', 'Content-Length': '0'} 2023-10-18T20:38:18.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a912af666f2ae1be-006530423a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.864 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:18.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:38:18.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:38:18.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:38:18.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:38:18.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:38:18.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:38:18.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:38:18.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:38:18.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:38:18.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:38:18.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:38:18.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:38:18.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:38:18.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:38:18.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:38:18.872 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:38:18.872 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:18.873 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:18.873 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.873 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.874 DEBUG:boto:Method: GET 2023-10-18T20:38:18.874 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.875 DEBUG:boto:Data: 2023-10-18T20:38:18.875 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.876 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:18.876 DEBUG:boto:Port: 8000 2023-10-18T20:38:18.876 DEBUG:boto:Params: {} 2023-10-18T20:38:18.877 DEBUG:boto:Token: None 2023-10-18T20:38:18.877 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/?versions 2023-10-18T20:38:18.878 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kzYgb0Jbit9vy97WTsX1pDzxU8s= 2023-10-18T20:38:18.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kzYgb0Jbit9vy97WTsX1pDzxU8s=', 'Content-Length': '0'} 2023-10-18T20:38:18.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e9292af976f3a5b-006530423a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.882 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:18.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:38:18.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:38:18.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:38:18.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:38:18.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:38:18.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:38:18.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:38:18.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:38:18.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:38:18.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:38:18.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:38:18.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:38:18.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:38:18.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:38:18.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:38:18.889 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:18.889 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:18.890 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.890 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.891 DEBUG:boto:Method: GET 2023-10-18T20:38:18.891 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.892 DEBUG:boto:Data: 2023-10-18T20:38:18.892 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.892 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:18.893 DEBUG:boto:Port: 8001 2023-10-18T20:38:18.893 DEBUG:boto:Params: {} 2023-10-18T20:38:18.894 DEBUG:boto:Token: None 2023-10-18T20:38:18.894 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/?versions 2023-10-18T20:38:18.895 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kzYgb0Jbit9vy97WTsX1pDzxU8s= 2023-10-18T20:38:18.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kzYgb0Jbit9vy97WTsX1pDzxU8s=', 'Content-Length': '0'} 2023-10-18T20:38:18.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c84770897340422b-006530423a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.898 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:18.900 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:18.900 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:18.901 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.901 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.901 DEBUG:boto:Method: GET 2023-10-18T20:38:18.902 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:38:18.902 DEBUG:boto:Data: 2023-10-18T20:38:18.903 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.903 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:18.903 DEBUG:boto:Port: 8000 2023-10-18T20:38:18.904 DEBUG:boto:Params: {} 2023-10-18T20:38:18.904 DEBUG:boto:Token: None 2023-10-18T20:38:18.905 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/?versions 2023-10-18T20:38:18.906 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kzYgb0Jbit9vy97WTsX1pDzxU8s= 2023-10-18T20:38:18.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kzYgb0Jbit9vy97WTsX1pDzxU8s=', 'Content-Length': '0'} 2023-10-18T20:38:18.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000880947bd1266cab6-006530423a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.909 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:18.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:38:18.911 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:18.912 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:18.912 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.913 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.913 DEBUG:boto:Method: GET 2023-10-18T20:38:18.913 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.914 DEBUG:boto:Data: 2023-10-18T20:38:18.914 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:18.915 DEBUG:boto:Port: 8001 2023-10-18T20:38:18.916 DEBUG:boto:Params: {} 2023-10-18T20:38:18.916 DEBUG:boto:Token: None 2023-10-18T20:38:18.917 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yR3Io+/iJxyf8JfxGH+RQb+QDhg= 2023-10-18T20:38:18.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yR3Io+/iJxyf8JfxGH+RQb+QDhg=', 'Content-Length': '0'} 2023-10-18T20:38:18.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017d0497877ce4068-006530423a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.920 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:18.921 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:18.921 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.921 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.922 DEBUG:boto:Method: GET 2023-10-18T20:38:18.922 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.923 DEBUG:boto:Data: 2023-10-18T20:38:18.923 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.924 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:18.924 DEBUG:boto:Port: 8000 2023-10-18T20:38:18.925 DEBUG:boto:Params: {} 2023-10-18T20:38:18.925 DEBUG:boto:Token: None 2023-10-18T20:38:18.925 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yR3Io+/iJxyf8JfxGH+RQb+QDhg= 2023-10-18T20:38:18.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yR3Io+/iJxyf8JfxGH+RQb+QDhg=', 'Content-Length': '0'} 2023-10-18T20:38:18.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c722bdfb5745b84d-006530423a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.929 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:18.929 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:18.930 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.930 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.931 DEBUG:boto:Method: HEAD 2023-10-18T20:38:18.931 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.931 DEBUG:boto:Data: 2023-10-18T20:38:18.932 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.932 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:18.933 DEBUG:boto:Port: 8001 2023-10-18T20:38:18.933 DEBUG:boto:Params: {} 2023-10-18T20:38:18.933 DEBUG:boto:Token: None 2023-10-18T20:38:18.934 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.934 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:S/Q9QFUQYFHZsS6MDHJorCz0zgU= 2023-10-18T20:38:18.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:S/Q9QFUQYFHZsS6MDHJorCz0zgU=', 'Content-Length': '0'} 2023-10-18T20:38:18.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009645a87d155d10d0-006530423a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.937 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:18.938 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:18.938 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.938 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.939 DEBUG:boto:Method: HEAD 2023-10-18T20:38:18.939 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.940 DEBUG:boto:Data: 2023-10-18T20:38:18.940 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.940 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:18.941 DEBUG:boto:Port: 8000 2023-10-18T20:38:18.941 DEBUG:boto:Params: {} 2023-10-18T20:38:18.942 DEBUG:boto:Token: None 2023-10-18T20:38:18.942 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.943 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:S/Q9QFUQYFHZsS6MDHJorCz0zgU= 2023-10-18T20:38:18.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:S/Q9QFUQYFHZsS6MDHJorCz0zgU=', 'Content-Length': '0'} 2023-10-18T20:38:18.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029e8bfb3b760f2ab-006530423a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:38:18.946 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:18.946 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:18.946 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.947 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.947 DEBUG:boto:Method: GET 2023-10-18T20:38:18.948 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.948 DEBUG:boto:Data: 2023-10-18T20:38:18.949 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:18.949 DEBUG:boto:Port: 8001 2023-10-18T20:38:18.950 DEBUG:boto:Params: {} 2023-10-18T20:38:18.950 DEBUG:boto:Token: None 2023-10-18T20:38:18.951 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yR3Io+/iJxyf8JfxGH+RQb+QDhg= 2023-10-18T20:38:18.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yR3Io+/iJxyf8JfxGH+RQb+QDhg=', 'Content-Length': '0'} 2023-10-18T20:38:18.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ad7ba86c6eb614-006530423a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.954 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:18.955 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:18.955 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.956 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.956 DEBUG:boto:Method: GET 2023-10-18T20:38:18.957 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.957 DEBUG:boto:Data: 2023-10-18T20:38:18.958 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:18.958 DEBUG:boto:Port: 8000 2023-10-18T20:38:18.959 DEBUG:boto:Params: {} 2023-10-18T20:38:18.959 DEBUG:boto:Token: None 2023-10-18T20:38:18.960 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:38:18.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yR3Io+/iJxyf8JfxGH+RQb+QDhg= 2023-10-18T20:38:18.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yR3Io+/iJxyf8JfxGH+RQb+QDhg=', 'Content-Length': '0'} 2023-10-18T20:38:18.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857eca8ef8500ecf-006530423a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.963 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:18.964 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:18.964 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:38:18.965 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:38:18.965 DEBUG:boto:Method: GET 2023-10-18T20:38:18.965 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:38:18.966 DEBUG:boto:Data: 2023-10-18T20:38:18.966 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.967 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:18.967 DEBUG:boto:Port: 8001 2023-10-18T20:38:18.968 DEBUG:boto:Params: {} 2023-10-18T20:38:18.968 DEBUG:boto:Token: None 2023-10-18T20:38:18.969 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/ 2023-10-18T20:38:18.969 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O+JoUl9fN1dPhLr38rGcYDRjGAA= 2023-10-18T20:38:18.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O+JoUl9fN1dPhLr38rGcYDRjGAA=', 'Content-Length': '0'} 2023-10-18T20:38:18.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d109308020849d5-006530423a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.972 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:18.973 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:18.974 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:18.974 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:38:18.975 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:38:18.975 DEBUG:boto:Method: GET 2023-10-18T20:38:18.975 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:38:18.976 DEBUG:boto:Data: 2023-10-18T20:38:18.976 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:18.977 DEBUG:boto:Port: 8000 2023-10-18T20:38:18.978 DEBUG:boto:Params: {} 2023-10-18T20:38:18.978 DEBUG:boto:Token: None 2023-10-18T20:38:18.979 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/ 2023-10-18T20:38:18.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O+JoUl9fN1dPhLr38rGcYDRjGAA= 2023-10-18T20:38:18.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O+JoUl9fN1dPhLr38rGcYDRjGAA=', 'Content-Length': '0'} 2023-10-18T20:38:18.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000272461e5ee9da9ee-006530423a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.983 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:18.984 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:18.984 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:18.985 DEBUG:boto:Method: HEAD 2023-10-18T20:38:18.985 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:38:18.985 DEBUG:boto:Data: 2023-10-18T20:38:18.986 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.986 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:18.987 DEBUG:boto:Port: 8001 2023-10-18T20:38:18.987 DEBUG:boto:Params: {} 2023-10-18T20:38:18.988 DEBUG:boto:Token: None 2023-10-18T20:38:18.988 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/obj11 2023-10-18T20:38:18.988 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xFWdDXaWSTJIeCDSSP38XfI/xR4= 2023-10-18T20:38:18.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xFWdDXaWSTJIeCDSSP38XfI/xR4=', 'Content-Length': '0'} 2023-10-18T20:38:18.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc650bbeaa44f0e-006530423a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.991 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:18.992 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:18.992 DEBUG:boto:Method: HEAD 2023-10-18T20:38:18.993 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:38:18.993 DEBUG:boto:Data: 2023-10-18T20:38:18.993 DEBUG:boto:Headers: {} 2023-10-18T20:38:18.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:18.994 DEBUG:boto:Port: 8000 2023-10-18T20:38:18.994 DEBUG:boto:Params: {} 2023-10-18T20:38:18.995 DEBUG:boto:Token: None 2023-10-18T20:38:18.995 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:18 GMT /bofrnf-59/obj11 2023-10-18T20:38:18.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xFWdDXaWSTJIeCDSSP38XfI/xR4= 2023-10-18T20:38:18.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:18 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xFWdDXaWSTJIeCDSSP38XfI/xR4=', 'Content-Length': '0'} 2023-10-18T20:38:18.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000302c0052f0e345b3-006530423a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:18.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:38:18.999 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:19.000 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:19.000 DEBUG:boto:Method: GET 2023-10-18T20:38:19.000 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:38:19.001 DEBUG:boto:Data: 2023-10-18T20:38:19.001 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.002 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.002 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.003 DEBUG:boto:Params: {} 2023-10-18T20:38:19.003 DEBUG:boto:Token: None 2023-10-18T20:38:19.003 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj11 2023-10-18T20:38:19.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RPbMPaz9FyrTjCT7AjJuzIaBi1k= 2023-10-18T20:38:19.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RPbMPaz9FyrTjCT7AjJuzIaBi1k=', 'Content-Length': '0'} 2023-10-18T20:38:19.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016d014e192d5c6bb-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.007 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:38:19.009 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:38:19.009 DEBUG:boto:Method: GET 2023-10-18T20:38:19.010 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:38:19.010 DEBUG:boto:Data: 2023-10-18T20:38:19.011 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.011 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.011 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.012 DEBUG:boto:Params: {} 2023-10-18T20:38:19.012 DEBUG:boto:Token: None 2023-10-18T20:38:19.013 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj11 2023-10-18T20:38:19.013 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RPbMPaz9FyrTjCT7AjJuzIaBi1k= 2023-10-18T20:38:19.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RPbMPaz9FyrTjCT7AjJuzIaBi1k=', 'Content-Length': '0'} 2023-10-18T20:38:19.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8dc0b8e89880d7f-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.016 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.017 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.017 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:38:19.018 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:38:19.018 DEBUG:boto:Method: GET 2023-10-18T20:38:19.019 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:38:19.019 DEBUG:boto:Data: 2023-10-18T20:38:19.019 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.020 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.020 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.021 DEBUG:boto:Params: {} 2023-10-18T20:38:19.021 DEBUG:boto:Token: None 2023-10-18T20:38:19.021 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.022 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000952961974aebb15-006530423b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.066 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.067 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.067 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.068 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:38:19.068 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:38:19.069 DEBUG:boto:Method: GET 2023-10-18T20:38:19.069 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:38:19.070 DEBUG:boto:Data: 2023-10-18T20:38:19.070 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.070 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.071 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.071 DEBUG:boto:Params: {} 2023-10-18T20:38:19.072 DEBUG:boto:Token: None 2023-10-18T20:38:19.072 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.073 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000725bff1d222552ed-006530423b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.077 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:38:19.078 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.079 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.079 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.079 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.080 DEBUG:boto:Method: GET 2023-10-18T20:38:19.080 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.081 DEBUG:boto:Data: 2023-10-18T20:38:19.081 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.081 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.082 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.082 DEBUG:boto:Params: {} 2023-10-18T20:38:19.083 DEBUG:boto:Token: None 2023-10-18T20:38:19.083 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.084 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jnj44WuiEbEUlUj3OsFucM7HVuI= 2023-10-18T20:38:19.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jnj44WuiEbEUlUj3OsFucM7HVuI=', 'Content-Length': '0'} 2023-10-18T20:38:19.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007334703c2298e3c1-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.087 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.087 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.088 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.088 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.089 DEBUG:boto:Method: GET 2023-10-18T20:38:19.089 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.089 DEBUG:boto:Data: 2023-10-18T20:38:19.090 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.090 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.091 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.091 DEBUG:boto:Params: {} 2023-10-18T20:38:19.092 DEBUG:boto:Token: None 2023-10-18T20:38:19.092 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.093 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jnj44WuiEbEUlUj3OsFucM7HVuI= 2023-10-18T20:38:19.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jnj44WuiEbEUlUj3OsFucM7HVuI=', 'Content-Length': '0'} 2023-10-18T20:38:19.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a03dca0ff0485df-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.096 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.096 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.097 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.097 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.097 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.098 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.098 DEBUG:boto:Data: 2023-10-18T20:38:19.099 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.099 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.100 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.100 DEBUG:boto:Params: {} 2023-10-18T20:38:19.100 DEBUG:boto:Token: None 2023-10-18T20:38:19.101 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.101 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f9mwwqeSKZgXERW8StL79ibPqmY= 2023-10-18T20:38:19.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f9mwwqeSKZgXERW8StL79ibPqmY=', 'Content-Length': '0'} 2023-10-18T20:38:19.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ed861a4c48520fd-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.104 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.105 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.105 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.105 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.106 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.106 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.107 DEBUG:boto:Data: 2023-10-18T20:38:19.107 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.107 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.108 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.108 DEBUG:boto:Params: {} 2023-10-18T20:38:19.109 DEBUG:boto:Token: None 2023-10-18T20:38:19.109 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.110 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f9mwwqeSKZgXERW8StL79ibPqmY= 2023-10-18T20:38:19.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f9mwwqeSKZgXERW8StL79ibPqmY=', 'Content-Length': '0'} 2023-10-18T20:38:19.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f18e4a76233fb204-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:38:19.113 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.113 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.114 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.114 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.115 DEBUG:boto:Method: GET 2023-10-18T20:38:19.115 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.115 DEBUG:boto:Data: 2023-10-18T20:38:19.116 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.116 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.117 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.117 DEBUG:boto:Params: {} 2023-10-18T20:38:19.118 DEBUG:boto:Token: None 2023-10-18T20:38:19.118 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.119 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jnj44WuiEbEUlUj3OsFucM7HVuI= 2023-10-18T20:38:19.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jnj44WuiEbEUlUj3OsFucM7HVuI=', 'Content-Length': '0'} 2023-10-18T20:38:19.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a35ccb63b7bdda-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.122 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.122 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.122 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.123 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.123 DEBUG:boto:Method: GET 2023-10-18T20:38:19.124 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.124 DEBUG:boto:Data: 2023-10-18T20:38:19.125 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.125 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.126 DEBUG:boto:Params: {} 2023-10-18T20:38:19.126 DEBUG:boto:Token: None 2023-10-18T20:38:19.127 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:38:19.127 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jnj44WuiEbEUlUj3OsFucM7HVuI= 2023-10-18T20:38:19.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jnj44WuiEbEUlUj3OsFucM7HVuI=', 'Content-Length': '0'} 2023-10-18T20:38:19.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3e75c8d854082d6-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.174 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.175 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.176 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.176 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.176 DEBUG:boto:Method: GET 2023-10-18T20:38:19.177 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.177 DEBUG:boto:Data: 2023-10-18T20:38:19.178 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.179 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.179 DEBUG:boto:Params: {} 2023-10-18T20:38:19.180 DEBUG:boto:Token: None 2023-10-18T20:38:19.181 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.181 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0bde0cb80f98ac-006530423b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.185 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.186 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.187 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.187 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.188 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.188 DEBUG:boto:Method: GET 2023-10-18T20:38:19.188 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.189 DEBUG:boto:Data: 2023-10-18T20:38:19.189 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.190 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.191 DEBUG:boto:Params: {} 2023-10-18T20:38:19.191 DEBUG:boto:Token: None 2023-10-18T20:38:19.192 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da588d336ddd9bd7-006530423b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.196 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.197 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.198 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.198 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.199 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:38:19.199 DEBUG:boto:Data: 2023-10-18T20:38:19.199 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.200 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.200 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.201 DEBUG:boto:Params: {} 2023-10-18T20:38:19.201 DEBUG:boto:Token: None 2023-10-18T20:38:19.202 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17 2023-10-18T20:38:19.202 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8a5h1VyvwZMTZu7Y2frjVgJjTOs= 2023-10-18T20:38:19.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8a5h1VyvwZMTZu7Y2frjVgJjTOs=', 'Content-Length': '0'} 2023-10-18T20:38:19.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000008f898a9cdfe3c2-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.205 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.206 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.206 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.207 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:38:19.207 DEBUG:boto:Data: 2023-10-18T20:38:19.208 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.208 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.208 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.209 DEBUG:boto:Params: {} 2023-10-18T20:38:19.209 DEBUG:boto:Token: None 2023-10-18T20:38:19.210 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17 2023-10-18T20:38:19.210 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8a5h1VyvwZMTZu7Y2frjVgJjTOs= 2023-10-18T20:38:19.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8a5h1VyvwZMTZu7Y2frjVgJjTOs=', 'Content-Length': '0'} 2023-10-18T20:38:19.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c6857061ab3e75b-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:38:19.214 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.214 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.215 DEBUG:boto:Method: GET 2023-10-18T20:38:19.215 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:38:19.216 DEBUG:boto:Data: 2023-10-18T20:38:19.216 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.217 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.217 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.218 DEBUG:boto:Params: {} 2023-10-18T20:38:19.218 DEBUG:boto:Token: None 2023-10-18T20:38:19.219 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17 2023-10-18T20:38:19.219 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wfKpfe6erWPhTVTG2+8qJqg3rkI= 2023-10-18T20:38:19.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wfKpfe6erWPhTVTG2+8qJqg3rkI=', 'Content-Length': '0'} 2023-10-18T20:38:19.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bde6502fbfb72c1-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.223 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:38:19.224 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:38:19.225 DEBUG:boto:Method: GET 2023-10-18T20:38:19.226 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:38:19.226 DEBUG:boto:Data: 2023-10-18T20:38:19.227 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.228 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.229 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.230 DEBUG:boto:Params: {} 2023-10-18T20:38:19.230 DEBUG:boto:Token: None 2023-10-18T20:38:19.231 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj17 2023-10-18T20:38:19.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wfKpfe6erWPhTVTG2+8qJqg3rkI= 2023-10-18T20:38:19.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wfKpfe6erWPhTVTG2+8qJqg3rkI=', 'Content-Length': '0'} 2023-10-18T20:38:19.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000090b7ba17d95ac1c-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.234 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.235 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.235 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.236 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.236 DEBUG:boto:Method: GET 2023-10-18T20:38:19.236 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.237 DEBUG:boto:Data: 2023-10-18T20:38:19.237 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.238 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.238 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.239 DEBUG:boto:Params: {} 2023-10-18T20:38:19.239 DEBUG:boto:Token: None 2023-10-18T20:38:19.240 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.241 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093204d820f025008-006530423b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.244 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.245 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.245 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.246 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.246 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.247 DEBUG:boto:Method: GET 2023-10-18T20:38:19.247 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:38:19.247 DEBUG:boto:Data: 2023-10-18T20:38:19.248 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.249 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.249 DEBUG:boto:Params: {} 2023-10-18T20:38:19.250 DEBUG:boto:Token: None 2023-10-18T20:38:19.251 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c96eb2f3e1fffa-006530423b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.256 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:38:19.257 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.257 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.258 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.258 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.259 DEBUG:boto:Method: GET 2023-10-18T20:38:19.259 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.260 DEBUG:boto:Data: 2023-10-18T20:38:19.260 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.260 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.261 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.261 DEBUG:boto:Params: {} 2023-10-18T20:38:19.262 DEBUG:boto:Token: None 2023-10-18T20:38:19.262 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1odwGGPJn+W4edsqGMJpskaw72w= 2023-10-18T20:38:19.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1odwGGPJn+W4edsqGMJpskaw72w=', 'Content-Length': '0'} 2023-10-18T20:38:19.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007359648016491db-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.306 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.307 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.308 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.308 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.308 DEBUG:boto:Method: GET 2023-10-18T20:38:19.309 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.309 DEBUG:boto:Data: 2023-10-18T20:38:19.310 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.311 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.311 DEBUG:boto:Params: {} 2023-10-18T20:38:19.312 DEBUG:boto:Token: None 2023-10-18T20:38:19.312 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1odwGGPJn+W4edsqGMJpskaw72w= 2023-10-18T20:38:19.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1odwGGPJn+W4edsqGMJpskaw72w=', 'Content-Length': '0'} 2023-10-18T20:38:19.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db0c0042759d8be5-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.316 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.316 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.317 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.317 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.318 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.318 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.319 DEBUG:boto:Data: 2023-10-18T20:38:19.319 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.320 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.320 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.320 DEBUG:boto:Params: {} 2023-10-18T20:38:19.321 DEBUG:boto:Token: None 2023-10-18T20:38:19.322 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.323 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bj7+tp5rEOfqAM2mzNjARzjfa18= 2023-10-18T20:38:19.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bj7+tp5rEOfqAM2mzNjARzjfa18=', 'Content-Length': '0'} 2023-10-18T20:38:19.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dedf6aefc0512e79-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.326 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.327 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.327 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.328 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.328 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.329 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.329 DEBUG:boto:Data: 2023-10-18T20:38:19.330 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.330 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.331 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.331 DEBUG:boto:Params: {} 2023-10-18T20:38:19.332 DEBUG:boto:Token: None 2023-10-18T20:38:19.332 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.332 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bj7+tp5rEOfqAM2mzNjARzjfa18= 2023-10-18T20:38:19.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bj7+tp5rEOfqAM2mzNjARzjfa18=', 'Content-Length': '0'} 2023-10-18T20:38:19.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d1e13522e97acf-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:38:19.336 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.337 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.337 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.338 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.338 DEBUG:boto:Method: GET 2023-10-18T20:38:19.339 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.339 DEBUG:boto:Data: 2023-10-18T20:38:19.339 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.340 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.340 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.340 DEBUG:boto:Params: {} 2023-10-18T20:38:19.341 DEBUG:boto:Token: None 2023-10-18T20:38:19.341 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.342 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1odwGGPJn+W4edsqGMJpskaw72w= 2023-10-18T20:38:19.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1odwGGPJn+W4edsqGMJpskaw72w=', 'Content-Length': '0'} 2023-10-18T20:38:19.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000366b8de8c4b5fb78-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.345 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.346 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.346 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.347 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.347 DEBUG:boto:Method: GET 2023-10-18T20:38:19.347 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.348 DEBUG:boto:Data: 2023-10-18T20:38:19.348 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.349 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.350 DEBUG:boto:Params: {} 2023-10-18T20:38:19.350 DEBUG:boto:Token: None 2023-10-18T20:38:19.350 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:38:19.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1odwGGPJn+W4edsqGMJpskaw72w= 2023-10-18T20:38:19.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1odwGGPJn+W4edsqGMJpskaw72w=', 'Content-Length': '0'} 2023-10-18T20:38:19.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bce0ad13918612b6-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.398 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.399 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.399 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.399 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.400 DEBUG:boto:Method: GET 2023-10-18T20:38:19.400 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.401 DEBUG:boto:Data: 2023-10-18T20:38:19.401 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.402 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.402 DEBUG:boto:Params: {} 2023-10-18T20:38:19.403 DEBUG:boto:Token: None 2023-10-18T20:38:19.403 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af6d7fc63e0c025d-006530423b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.407 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.408 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.409 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.409 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.409 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.410 DEBUG:boto:Method: GET 2023-10-18T20:38:19.410 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.411 DEBUG:boto:Data: 2023-10-18T20:38:19.411 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.412 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.412 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.412 DEBUG:boto:Params: {} 2023-10-18T20:38:19.413 DEBUG:boto:Token: None 2023-10-18T20:38:19.413 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.414 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d23f851f477f01c1-006530423b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.418 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.418 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.419 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.419 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.420 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:38:19.420 DEBUG:boto:Data: 2023-10-18T20:38:19.421 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.421 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.421 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.422 DEBUG:boto:Params: {} 2023-10-18T20:38:19.422 DEBUG:boto:Token: None 2023-10-18T20:38:19.423 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19 2023-10-18T20:38:19.423 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cfb5MNzof+/kpdcQ0tCRS0yBwm4= 2023-10-18T20:38:19.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cfb5MNzof+/kpdcQ0tCRS0yBwm4=', 'Content-Length': '0'} 2023-10-18T20:38:19.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000038630cb2fc240f1-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.426 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.426 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.427 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.427 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:38:19.428 DEBUG:boto:Data: 2023-10-18T20:38:19.428 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.429 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.429 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.429 DEBUG:boto:Params: {} 2023-10-18T20:38:19.430 DEBUG:boto:Token: None 2023-10-18T20:38:19.430 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19 2023-10-18T20:38:19.431 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cfb5MNzof+/kpdcQ0tCRS0yBwm4= 2023-10-18T20:38:19.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cfb5MNzof+/kpdcQ0tCRS0yBwm4=', 'Content-Length': '0'} 2023-10-18T20:38:19.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000718ccc2e5f308da8-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:38:19.434 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.434 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.435 DEBUG:boto:Method: GET 2023-10-18T20:38:19.435 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:38:19.436 DEBUG:boto:Data: 2023-10-18T20:38:19.436 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.436 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.437 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.437 DEBUG:boto:Params: {} 2023-10-18T20:38:19.438 DEBUG:boto:Token: None 2023-10-18T20:38:19.438 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19 2023-10-18T20:38:19.439 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nNfme09qTRbOIA0kSrK1wo/UnZQ= 2023-10-18T20:38:19.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nNfme09qTRbOIA0kSrK1wo/UnZQ=', 'Content-Length': '0'} 2023-10-18T20:38:19.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ec50e3e3b683a4-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.442 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:38:19.442 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:38:19.443 DEBUG:boto:Method: GET 2023-10-18T20:38:19.443 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:38:19.444 DEBUG:boto:Data: 2023-10-18T20:38:19.444 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.445 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.446 DEBUG:boto:Params: {} 2023-10-18T20:38:19.446 DEBUG:boto:Token: None 2023-10-18T20:38:19.447 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj19 2023-10-18T20:38:19.447 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nNfme09qTRbOIA0kSrK1wo/UnZQ= 2023-10-18T20:38:19.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nNfme09qTRbOIA0kSrK1wo/UnZQ=', 'Content-Length': '0'} 2023-10-18T20:38:19.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d08fe40feb627d0-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.450 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.450 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.451 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.451 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.452 DEBUG:boto:Method: GET 2023-10-18T20:38:19.452 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.453 DEBUG:boto:Data: 2023-10-18T20:38:19.453 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.454 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.454 DEBUG:boto:Params: {} 2023-10-18T20:38:19.455 DEBUG:boto:Token: None 2023-10-18T20:38:19.455 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a07eec27578aea16-006530423b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.459 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.459 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.460 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.460 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.460 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.461 DEBUG:boto:Method: GET 2023-10-18T20:38:19.461 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:38:19.462 DEBUG:boto:Data: 2023-10-18T20:38:19.462 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.463 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.463 DEBUG:boto:Params: {} 2023-10-18T20:38:19.464 DEBUG:boto:Token: None 2023-10-18T20:38:19.464 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b562ae69ede1078-006530423b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.468 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:38:19.469 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.470 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.470 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.470 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.471 DEBUG:boto:Method: GET 2023-10-18T20:38:19.471 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.472 DEBUG:boto:Data: 2023-10-18T20:38:19.472 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.472 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.473 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.473 DEBUG:boto:Params: {} 2023-10-18T20:38:19.474 DEBUG:boto:Token: None 2023-10-18T20:38:19.474 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.475 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zXhTUBudyRMM9Hxz/WfYzQPutbE= 2023-10-18T20:38:19.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zXhTUBudyRMM9Hxz/WfYzQPutbE=', 'Content-Length': '0'} 2023-10-18T20:38:19.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cabfe26dcc58dc9b-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.518 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.533 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.533 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.534 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.534 DEBUG:boto:Method: GET 2023-10-18T20:38:19.535 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.535 DEBUG:boto:Data: 2023-10-18T20:38:19.535 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.536 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.536 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.537 DEBUG:boto:Params: {} 2023-10-18T20:38:19.537 DEBUG:boto:Token: None 2023-10-18T20:38:19.538 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.538 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zXhTUBudyRMM9Hxz/WfYzQPutbE= 2023-10-18T20:38:19.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zXhTUBudyRMM9Hxz/WfYzQPutbE=', 'Content-Length': '0'} 2023-10-18T20:38:19.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092ed897d86075907-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.541 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.542 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.542 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.543 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.543 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.544 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.544 DEBUG:boto:Data: 2023-10-18T20:38:19.544 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.545 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.545 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.546 DEBUG:boto:Params: {} 2023-10-18T20:38:19.546 DEBUG:boto:Token: None 2023-10-18T20:38:19.547 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.547 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:njMWuDA80ochpVAagm7jRtDIWy8= 2023-10-18T20:38:19.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:njMWuDA80ochpVAagm7jRtDIWy8=', 'Content-Length': '0'} 2023-10-18T20:38:19.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c85834e88c95a2b2-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.550 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.550 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.551 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.551 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.552 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.552 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.552 DEBUG:boto:Data: 2023-10-18T20:38:19.553 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.553 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.554 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.554 DEBUG:boto:Params: {} 2023-10-18T20:38:19.554 DEBUG:boto:Token: None 2023-10-18T20:38:19.555 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:njMWuDA80ochpVAagm7jRtDIWy8= 2023-10-18T20:38:19.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:njMWuDA80ochpVAagm7jRtDIWy8=', 'Content-Length': '0'} 2023-10-18T20:38:19.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab4425fde126b249-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:38:19.559 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.559 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.560 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.560 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.561 DEBUG:boto:Method: GET 2023-10-18T20:38:19.561 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.562 DEBUG:boto:Data: 2023-10-18T20:38:19.562 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.563 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.563 DEBUG:boto:Params: {} 2023-10-18T20:38:19.564 DEBUG:boto:Token: None 2023-10-18T20:38:19.564 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.565 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zXhTUBudyRMM9Hxz/WfYzQPutbE= 2023-10-18T20:38:19.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zXhTUBudyRMM9Hxz/WfYzQPutbE=', 'Content-Length': '0'} 2023-10-18T20:38:19.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd3d5c625a0cf89-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.568 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.568 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.569 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.569 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.570 DEBUG:boto:Method: GET 2023-10-18T20:38:19.570 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.571 DEBUG:boto:Data: 2023-10-18T20:38:19.571 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.572 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.572 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.572 DEBUG:boto:Params: {} 2023-10-18T20:38:19.573 DEBUG:boto:Token: None 2023-10-18T20:38:19.573 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:38:19.574 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zXhTUBudyRMM9Hxz/WfYzQPutbE= 2023-10-18T20:38:19.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zXhTUBudyRMM9Hxz/WfYzQPutbE=', 'Content-Length': '0'} 2023-10-18T20:38:19.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002d201a7c78d7575-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.577 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.577 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.578 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.578 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.579 DEBUG:boto:Method: GET 2023-10-18T20:38:19.579 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.580 DEBUG:boto:Data: 2023-10-18T20:38:19.580 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.580 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.581 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.581 DEBUG:boto:Params: {} 2023-10-18T20:38:19.582 DEBUG:boto:Token: None 2023-10-18T20:38:19.582 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.583 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bedfb82ca7ed14fc-006530423b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.586 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.587 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.587 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.588 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.588 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.589 DEBUG:boto:Method: GET 2023-10-18T20:38:19.589 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.590 DEBUG:boto:Data: 2023-10-18T20:38:19.590 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.591 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.591 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.592 DEBUG:boto:Params: {} 2023-10-18T20:38:19.592 DEBUG:boto:Token: None 2023-10-18T20:38:19.592 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.593 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e12d03989fddccb0-006530423b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.597 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.597 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.598 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.598 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.599 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:38:19.599 DEBUG:boto:Data: 2023-10-18T20:38:19.599 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.600 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.600 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.601 DEBUG:boto:Params: {} 2023-10-18T20:38:19.601 DEBUG:boto:Token: None 2023-10-18T20:38:19.601 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22 2023-10-18T20:38:19.602 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LeXkHvzN/LJ+05b/3nXdVKyjKWA= 2023-10-18T20:38:19.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LeXkHvzN/LJ+05b/3nXdVKyjKWA=', 'Content-Length': '0'} 2023-10-18T20:38:19.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000471c4677063c92ef-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.605 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.605 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.605 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.606 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:38:19.606 DEBUG:boto:Data: 2023-10-18T20:38:19.607 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.607 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.607 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.608 DEBUG:boto:Params: {} 2023-10-18T20:38:19.608 DEBUG:boto:Token: None 2023-10-18T20:38:19.609 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22 2023-10-18T20:38:19.609 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LeXkHvzN/LJ+05b/3nXdVKyjKWA= 2023-10-18T20:38:19.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LeXkHvzN/LJ+05b/3nXdVKyjKWA=', 'Content-Length': '0'} 2023-10-18T20:38:19.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082b1fb3ddb0569e0-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:38:19.612 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.613 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.613 DEBUG:boto:Method: GET 2023-10-18T20:38:19.614 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:38:19.614 DEBUG:boto:Data: 2023-10-18T20:38:19.615 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.615 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.615 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.616 DEBUG:boto:Params: {} 2023-10-18T20:38:19.616 DEBUG:boto:Token: None 2023-10-18T20:38:19.617 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22 2023-10-18T20:38:19.617 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h1mSOK3/UQTcSjWI5zucPbzLvlc= 2023-10-18T20:38:19.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h1mSOK3/UQTcSjWI5zucPbzLvlc=', 'Content-Length': '0'} 2023-10-18T20:38:19.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa6a745f3c54fc73-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.622 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:38:19.622 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:38:19.623 DEBUG:boto:Method: GET 2023-10-18T20:38:19.623 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:38:19.623 DEBUG:boto:Data: 2023-10-18T20:38:19.624 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.624 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.625 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.625 DEBUG:boto:Params: {} 2023-10-18T20:38:19.625 DEBUG:boto:Token: None 2023-10-18T20:38:19.626 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj22 2023-10-18T20:38:19.626 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h1mSOK3/UQTcSjWI5zucPbzLvlc= 2023-10-18T20:38:19.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h1mSOK3/UQTcSjWI5zucPbzLvlc=', 'Content-Length': '0'} 2023-10-18T20:38:19.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005754301dd2f6a2d8-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.629 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.630 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.630 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.631 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.631 DEBUG:boto:Method: GET 2023-10-18T20:38:19.632 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.632 DEBUG:boto:Data: 2023-10-18T20:38:19.633 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.633 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.633 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.634 DEBUG:boto:Params: {} 2023-10-18T20:38:19.634 DEBUG:boto:Token: None 2023-10-18T20:38:19.635 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.635 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097e5c2e0fb25df63-006530423b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.639 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.640 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.640 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.640 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.641 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.641 DEBUG:boto:Method: GET 2023-10-18T20:38:19.642 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:38:19.642 DEBUG:boto:Data: 2023-10-18T20:38:19.642 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.643 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.643 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.644 DEBUG:boto:Params: {} 2023-10-18T20:38:19.644 DEBUG:boto:Token: None 2023-10-18T20:38:19.645 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.645 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c33c62e0562280e-006530423b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.649 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:38:19.650 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.650 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.651 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.651 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.652 DEBUG:boto:Method: GET 2023-10-18T20:38:19.652 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.652 DEBUG:boto:Data: 2023-10-18T20:38:19.653 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.653 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.654 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.654 DEBUG:boto:Params: {} 2023-10-18T20:38:19.655 DEBUG:boto:Token: None 2023-10-18T20:38:19.655 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+suOYqSI7plMQ97QrNlkiFnWvoA= 2023-10-18T20:38:19.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+suOYqSI7plMQ97QrNlkiFnWvoA=', 'Content-Length': '0'} 2023-10-18T20:38:19.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e13b116b55008317-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.698 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.699 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.700 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.700 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.700 DEBUG:boto:Method: GET 2023-10-18T20:38:19.701 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.701 DEBUG:boto:Data: 2023-10-18T20:38:19.702 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.702 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.702 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.703 DEBUG:boto:Params: {} 2023-10-18T20:38:19.703 DEBUG:boto:Token: None 2023-10-18T20:38:19.704 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.704 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+suOYqSI7plMQ97QrNlkiFnWvoA= 2023-10-18T20:38:19.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+suOYqSI7plMQ97QrNlkiFnWvoA=', 'Content-Length': '0'} 2023-10-18T20:38:19.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d22f1192d9f3c0-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.708 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.708 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.709 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.709 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.709 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.710 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.710 DEBUG:boto:Data: 2023-10-18T20:38:19.711 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.711 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.712 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.712 DEBUG:boto:Params: {} 2023-10-18T20:38:19.712 DEBUG:boto:Token: None 2023-10-18T20:38:19.713 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.713 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P8FePy8O/wojjne8BwRcDfvOceo= 2023-10-18T20:38:19.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P8FePy8O/wojjne8BwRcDfvOceo=', 'Content-Length': '0'} 2023-10-18T20:38:19.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e13d0f855aaf86d-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.716 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.717 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.717 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.717 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.718 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.718 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.719 DEBUG:boto:Data: 2023-10-18T20:38:19.719 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.720 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.720 DEBUG:boto:Params: {} 2023-10-18T20:38:19.721 DEBUG:boto:Token: None 2023-10-18T20:38:19.721 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.722 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P8FePy8O/wojjne8BwRcDfvOceo= 2023-10-18T20:38:19.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P8FePy8O/wojjne8BwRcDfvOceo=', 'Content-Length': '0'} 2023-10-18T20:38:19.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea55d3eca8450a5d-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:38:19.725 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.725 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.726 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.726 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.727 DEBUG:boto:Method: GET 2023-10-18T20:38:19.727 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.727 DEBUG:boto:Data: 2023-10-18T20:38:19.728 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.729 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.729 DEBUG:boto:Params: {} 2023-10-18T20:38:19.730 DEBUG:boto:Token: None 2023-10-18T20:38:19.730 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.731 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+suOYqSI7plMQ97QrNlkiFnWvoA= 2023-10-18T20:38:19.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+suOYqSI7plMQ97QrNlkiFnWvoA=', 'Content-Length': '0'} 2023-10-18T20:38:19.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd6a9a9f8e27d2d-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.734 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.734 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.735 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.735 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.736 DEBUG:boto:Method: GET 2023-10-18T20:38:19.736 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.736 DEBUG:boto:Data: 2023-10-18T20:38:19.737 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.737 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.738 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.738 DEBUG:boto:Params: {} 2023-10-18T20:38:19.739 DEBUG:boto:Token: None 2023-10-18T20:38:19.739 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:38:19.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+suOYqSI7plMQ97QrNlkiFnWvoA= 2023-10-18T20:38:19.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+suOYqSI7plMQ97QrNlkiFnWvoA=', 'Content-Length': '0'} 2023-10-18T20:38:19.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000963f4676cf5ba90e-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.743 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.743 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.744 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.744 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.745 DEBUG:boto:Method: GET 2023-10-18T20:38:19.745 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.745 DEBUG:boto:Data: 2023-10-18T20:38:19.746 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.747 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.747 DEBUG:boto:Params: {} 2023-10-18T20:38:19.748 DEBUG:boto:Token: None 2023-10-18T20:38:19.748 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001400cbd2b0883a29-006530423b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.752 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.753 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.753 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.754 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.754 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.755 DEBUG:boto:Method: GET 2023-10-18T20:38:19.755 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.756 DEBUG:boto:Data: 2023-10-18T20:38:19.756 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.756 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.757 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.757 DEBUG:boto:Params: {} 2023-10-18T20:38:19.758 DEBUG:boto:Token: None 2023-10-18T20:38:19.758 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.759 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001c37af28a6be50e-006530423b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.802 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.803 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.804 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.804 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.805 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:38:19.805 DEBUG:boto:Data: 2023-10-18T20:38:19.806 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.806 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.807 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.807 DEBUG:boto:Params: {} 2023-10-18T20:38:19.808 DEBUG:boto:Token: None 2023-10-18T20:38:19.808 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33 2023-10-18T20:38:19.809 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5IYypXiki4tZM+hbsJolPc5Xr1g= 2023-10-18T20:38:19.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5IYypXiki4tZM+hbsJolPc5Xr1g=', 'Content-Length': '0'} 2023-10-18T20:38:19.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a69037947223c677-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.812 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.812 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.812 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.813 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:38:19.813 DEBUG:boto:Data: 2023-10-18T20:38:19.814 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.814 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.815 DEBUG:boto:Params: {} 2023-10-18T20:38:19.815 DEBUG:boto:Token: None 2023-10-18T20:38:19.816 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33 2023-10-18T20:38:19.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5IYypXiki4tZM+hbsJolPc5Xr1g= 2023-10-18T20:38:19.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5IYypXiki4tZM+hbsJolPc5Xr1g=', 'Content-Length': '0'} 2023-10-18T20:38:19.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eebb973c80a17400-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:38:19.819 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.820 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.820 DEBUG:boto:Method: GET 2023-10-18T20:38:19.821 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:38:19.821 DEBUG:boto:Data: 2023-10-18T20:38:19.822 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.822 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.822 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.823 DEBUG:boto:Params: {} 2023-10-18T20:38:19.823 DEBUG:boto:Token: None 2023-10-18T20:38:19.826 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33 2023-10-18T20:38:19.826 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sM8KLTFw/sQI8F+CkCfDr7aJCfk= 2023-10-18T20:38:19.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sM8KLTFw/sQI8F+CkCfDr7aJCfk=', 'Content-Length': '0'} 2023-10-18T20:38:19.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef021d962bc416a7-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.829 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:38:19.830 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:38:19.830 DEBUG:boto:Method: GET 2023-10-18T20:38:19.831 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:38:19.831 DEBUG:boto:Data: 2023-10-18T20:38:19.832 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.832 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.832 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.833 DEBUG:boto:Params: {} 2023-10-18T20:38:19.833 DEBUG:boto:Token: None 2023-10-18T20:38:19.834 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj33 2023-10-18T20:38:19.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sM8KLTFw/sQI8F+CkCfDr7aJCfk= 2023-10-18T20:38:19.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sM8KLTFw/sQI8F+CkCfDr7aJCfk=', 'Content-Length': '0'} 2023-10-18T20:38:19.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085f6d7640994c3dc-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.838 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.838 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.838 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.839 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.839 DEBUG:boto:Method: GET 2023-10-18T20:38:19.840 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.840 DEBUG:boto:Data: 2023-10-18T20:38:19.840 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.841 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.841 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.842 DEBUG:boto:Params: {} 2023-10-18T20:38:19.843 DEBUG:boto:Token: None 2023-10-18T20:38:19.843 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.843 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8f28e987517fc11-006530423b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.847 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.848 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:19.848 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:19.849 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.849 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.850 DEBUG:boto:Method: GET 2023-10-18T20:38:19.850 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:38:19.851 DEBUG:boto:Data: 2023-10-18T20:38:19.851 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.852 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.852 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.853 DEBUG:boto:Params: {} 2023-10-18T20:38:19.853 DEBUG:boto:Token: None 2023-10-18T20:38:19.854 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/ 2023-10-18T20:38:19.854 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8= 2023-10-18T20:38:19.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:veaHgJReGgOXU5z4ueME1QfWZf8=', 'Content-Length': '0'} 2023-10-18T20:38:19.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000897cc4f996def530-006530423b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.858 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:19.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:38:19.859 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:19.859 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:19.860 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.860 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.861 DEBUG:boto:Method: GET 2023-10-18T20:38:19.861 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.861 DEBUG:boto:Data: 2023-10-18T20:38:19.862 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.862 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.863 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.863 DEBUG:boto:Params: {} 2023-10-18T20:38:19.864 DEBUG:boto:Token: None 2023-10-18T20:38:19.864 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.865 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAyEPu4wrgMDAQeDhBCD0jFGZPA= 2023-10-18T20:38:19.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAyEPu4wrgMDAQeDhBCD0jFGZPA=', 'Content-Length': '0'} 2023-10-18T20:38:19.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa4cb8059c5e657-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.914 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:19.915 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:19.916 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.916 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.916 DEBUG:boto:Method: GET 2023-10-18T20:38:19.917 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.917 DEBUG:boto:Data: 2023-10-18T20:38:19.918 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.919 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.919 DEBUG:boto:Params: {} 2023-10-18T20:38:19.919 DEBUG:boto:Token: None 2023-10-18T20:38:19.920 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAyEPu4wrgMDAQeDhBCD0jFGZPA= 2023-10-18T20:38:19.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAyEPu4wrgMDAQeDhBCD0jFGZPA=', 'Content-Length': '0'} 2023-10-18T20:38:19.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d01bf53a79ca15-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.923 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:19.924 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:19.924 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.925 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.925 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.925 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.926 DEBUG:boto:Data: 2023-10-18T20:38:19.926 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.927 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.927 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.928 DEBUG:boto:Params: {} 2023-10-18T20:38:19.928 DEBUG:boto:Token: None 2023-10-18T20:38:19.929 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdIWU9wPF8rMrV1DtgxbzM2GQiE= 2023-10-18T20:38:19.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdIWU9wPF8rMrV1DtgxbzM2GQiE=', 'Content-Length': '0'} 2023-10-18T20:38:19.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c30942c41df61fe-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.932 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:19.932 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:19.933 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.933 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.934 DEBUG:boto:Method: HEAD 2023-10-18T20:38:19.934 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.935 DEBUG:boto:Data: 2023-10-18T20:38:19.935 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.936 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.936 DEBUG:boto:Params: {} 2023-10-18T20:38:19.937 DEBUG:boto:Token: None 2023-10-18T20:38:19.937 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.938 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DdIWU9wPF8rMrV1DtgxbzM2GQiE= 2023-10-18T20:38:19.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DdIWU9wPF8rMrV1DtgxbzM2GQiE=', 'Content-Length': '0'} 2023-10-18T20:38:19.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49bf19941b8cd71-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:38:19.941 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:19.941 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:19.942 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.942 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.942 DEBUG:boto:Method: GET 2023-10-18T20:38:19.943 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.943 DEBUG:boto:Data: 2023-10-18T20:38:19.944 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.944 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:19.945 DEBUG:boto:Port: 8001 2023-10-18T20:38:19.945 DEBUG:boto:Params: {} 2023-10-18T20:38:19.945 DEBUG:boto:Token: None 2023-10-18T20:38:19.946 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.946 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAyEPu4wrgMDAQeDhBCD0jFGZPA= 2023-10-18T20:38:19.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAyEPu4wrgMDAQeDhBCD0jFGZPA=', 'Content-Length': '0'} 2023-10-18T20:38:19.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e329f8aea466e916-006530423b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:19.950 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:19.950 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:19.951 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.951 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.951 DEBUG:boto:Method: GET 2023-10-18T20:38:19.952 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.952 DEBUG:boto:Data: 2023-10-18T20:38:19.953 DEBUG:boto:Headers: {} 2023-10-18T20:38:19.953 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:19.953 DEBUG:boto:Port: 8000 2023-10-18T20:38:19.954 DEBUG:boto:Params: {} 2023-10-18T20:38:19.954 DEBUG:boto:Token: None 2023-10-18T20:38:19.955 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:19 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:38:19.955 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xAyEPu4wrgMDAQeDhBCD0jFGZPA= 2023-10-18T20:38:19.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:19 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xAyEPu4wrgMDAQeDhBCD0jFGZPA=', 'Content-Length': '0'} 2023-10-18T20:38:19.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a411f8b59f797a72-006530423b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.002 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.003 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.003 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.004 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.004 DEBUG:boto:Method: GET 2023-10-18T20:38:20.005 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.005 DEBUG:boto:Data: 2023-10-18T20:38:20.006 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.006 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.006 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.007 DEBUG:boto:Params: {} 2023-10-18T20:38:20.007 DEBUG:boto:Token: None 2023-10-18T20:38:20.008 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.008 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f677c36e87a5e59-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.012 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.012 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.013 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.013 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.014 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.014 DEBUG:boto:Method: GET 2023-10-18T20:38:20.014 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.015 DEBUG:boto:Data: 2023-10-18T20:38:20.015 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.015 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.016 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.016 DEBUG:boto:Params: {} 2023-10-18T20:38:20.017 DEBUG:boto:Token: None 2023-10-18T20:38:20.017 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.018 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e04f5725708ad7e9-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.022 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.022 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:20.023 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:20.023 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.024 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:38:20.024 DEBUG:boto:Data: 2023-10-18T20:38:20.025 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.025 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.026 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.026 DEBUG:boto:Params: {} 2023-10-18T20:38:20.026 DEBUG:boto:Token: None 2023-10-18T20:38:20.027 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj34 2023-10-18T20:38:20.027 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lW3gPJzxTm0JFFMyUqTzMfctCy0= 2023-10-18T20:38:20.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lW3gPJzxTm0JFFMyUqTzMfctCy0=', 'Content-Length': '0'} 2023-10-18T20:38:20.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f443ab133c39698-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.030 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:20.031 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:20.033 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.033 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:38:20.034 DEBUG:boto:Data: 2023-10-18T20:38:20.034 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.035 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.035 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.035 DEBUG:boto:Params: {} 2023-10-18T20:38:20.036 DEBUG:boto:Token: None 2023-10-18T20:38:20.036 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj34 2023-10-18T20:38:20.037 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lW3gPJzxTm0JFFMyUqTzMfctCy0= 2023-10-18T20:38:20.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lW3gPJzxTm0JFFMyUqTzMfctCy0=', 'Content-Length': '0'} 2023-10-18T20:38:20.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036cf11534db97827-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:38:20.040 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:20.041 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:20.041 DEBUG:boto:Method: GET 2023-10-18T20:38:20.042 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:38:20.042 DEBUG:boto:Data: 2023-10-18T20:38:20.043 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.043 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.044 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.044 DEBUG:boto:Params: {} 2023-10-18T20:38:20.045 DEBUG:boto:Token: None 2023-10-18T20:38:20.045 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj34 2023-10-18T20:38:20.046 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YbqYuRmjXXH1Qf9Tn2qoq3f6XWw= 2023-10-18T20:38:20.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YbqYuRmjXXH1Qf9Tn2qoq3f6XWw=', 'Content-Length': '0'} 2023-10-18T20:38:20.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4a7cc83f62b603d-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.048 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:38:20.049 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:38:20.049 DEBUG:boto:Method: GET 2023-10-18T20:38:20.050 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:38:20.050 DEBUG:boto:Data: 2023-10-18T20:38:20.051 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.051 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.052 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.052 DEBUG:boto:Params: {} 2023-10-18T20:38:20.053 DEBUG:boto:Token: None 2023-10-18T20:38:20.053 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj34 2023-10-18T20:38:20.054 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YbqYuRmjXXH1Qf9Tn2qoq3f6XWw= 2023-10-18T20:38:20.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YbqYuRmjXXH1Qf9Tn2qoq3f6XWw=', 'Content-Length': '0'} 2023-10-18T20:38:20.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000366d2e821a5cb2b3-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.056 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.057 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.057 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.058 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.058 DEBUG:boto:Method: GET 2023-10-18T20:38:20.058 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.059 DEBUG:boto:Data: 2023-10-18T20:38:20.059 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.060 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.060 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.061 DEBUG:boto:Params: {} 2023-10-18T20:38:20.061 DEBUG:boto:Token: None 2023-10-18T20:38:20.062 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.062 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006418147b41728b77-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.065 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.066 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.066 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.067 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.067 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.068 DEBUG:boto:Method: GET 2023-10-18T20:38:20.068 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:38:20.069 DEBUG:boto:Data: 2023-10-18T20:38:20.069 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.069 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.070 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.070 DEBUG:boto:Params: {} 2023-10-18T20:38:20.071 DEBUG:boto:Token: None 2023-10-18T20:38:20.071 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.072 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007afaee2af46a9fd1-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.075 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:38:20.076 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.076 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.077 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.077 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.078 DEBUG:boto:Method: GET 2023-10-18T20:38:20.078 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.078 DEBUG:boto:Data: 2023-10-18T20:38:20.079 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.079 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.080 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.080 DEBUG:boto:Params: {} 2023-10-18T20:38:20.080 DEBUG:boto:Token: None 2023-10-18T20:38:20.081 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.081 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CnxMYav6hMhdXMGCEIlUcdaG4ik= 2023-10-18T20:38:20.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CnxMYav6hMhdXMGCEIlUcdaG4ik=', 'Content-Length': '0'} 2023-10-18T20:38:20.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ad8a18c5d6272b-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.085 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.085 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.086 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.086 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.086 DEBUG:boto:Method: GET 2023-10-18T20:38:20.087 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.087 DEBUG:boto:Data: 2023-10-18T20:38:20.088 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.088 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.088 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.089 DEBUG:boto:Params: {} 2023-10-18T20:38:20.089 DEBUG:boto:Token: None 2023-10-18T20:38:20.090 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.090 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CnxMYav6hMhdXMGCEIlUcdaG4ik= 2023-10-18T20:38:20.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CnxMYav6hMhdXMGCEIlUcdaG4ik=', 'Content-Length': '0'} 2023-10-18T20:38:20.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004747d8ae5ea6fc01-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.093 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.093 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.094 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.094 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.095 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.095 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.095 DEBUG:boto:Data: 2023-10-18T20:38:20.096 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.096 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.096 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.097 DEBUG:boto:Params: {} 2023-10-18T20:38:20.097 DEBUG:boto:Token: None 2023-10-18T20:38:20.098 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.098 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8XFpvoUAGCVAPaI43/UqN0nLLrc= 2023-10-18T20:38:20.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8XFpvoUAGCVAPaI43/UqN0nLLrc=', 'Content-Length': '0'} 2023-10-18T20:38:20.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aee977440787064-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.101 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.101 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.102 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.102 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.103 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.103 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.103 DEBUG:boto:Data: 2023-10-18T20:38:20.104 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.104 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.104 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.105 DEBUG:boto:Params: {} 2023-10-18T20:38:20.105 DEBUG:boto:Token: None 2023-10-18T20:38:20.106 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.106 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8XFpvoUAGCVAPaI43/UqN0nLLrc= 2023-10-18T20:38:20.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8XFpvoUAGCVAPaI43/UqN0nLLrc=', 'Content-Length': '0'} 2023-10-18T20:38:20.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c35c121995e3a0bf-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:38:20.110 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.110 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.110 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.111 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.111 DEBUG:boto:Method: GET 2023-10-18T20:38:20.112 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.112 DEBUG:boto:Data: 2023-10-18T20:38:20.112 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.113 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.113 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.114 DEBUG:boto:Params: {} 2023-10-18T20:38:20.114 DEBUG:boto:Token: None 2023-10-18T20:38:20.115 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CnxMYav6hMhdXMGCEIlUcdaG4ik= 2023-10-18T20:38:20.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CnxMYav6hMhdXMGCEIlUcdaG4ik=', 'Content-Length': '0'} 2023-10-18T20:38:20.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d075692056e1867-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.118 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.118 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.119 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.119 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.120 DEBUG:boto:Method: GET 2023-10-18T20:38:20.120 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.120 DEBUG:boto:Data: 2023-10-18T20:38:20.121 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.121 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.122 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.122 DEBUG:boto:Params: {} 2023-10-18T20:38:20.123 DEBUG:boto:Token: None 2023-10-18T20:38:20.123 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:38:20.124 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CnxMYav6hMhdXMGCEIlUcdaG4ik= 2023-10-18T20:38:20.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CnxMYav6hMhdXMGCEIlUcdaG4ik=', 'Content-Length': '0'} 2023-10-18T20:38:20.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a97f4734a03d2db9-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.127 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.127 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.128 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.128 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.128 DEBUG:boto:Method: GET 2023-10-18T20:38:20.129 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.129 DEBUG:boto:Data: 2023-10-18T20:38:20.130 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.130 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.131 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.131 DEBUG:boto:Params: {} 2023-10-18T20:38:20.131 DEBUG:boto:Token: None 2023-10-18T20:38:20.132 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.132 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9678001ee6ec188-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.136 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.136 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.137 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.137 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.137 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.138 DEBUG:boto:Method: GET 2023-10-18T20:38:20.138 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.139 DEBUG:boto:Data: 2023-10-18T20:38:20.139 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.139 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.140 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.140 DEBUG:boto:Params: {} 2023-10-18T20:38:20.140 DEBUG:boto:Token: None 2023-10-18T20:38:20.141 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.141 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f8ceae4daefc944-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.145 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.146 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.146 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.146 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.147 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:38:20.147 DEBUG:boto:Data: 2023-10-18T20:38:20.148 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.148 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.148 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.149 DEBUG:boto:Params: {} 2023-10-18T20:38:20.149 DEBUG:boto:Token: None 2023-10-18T20:38:20.149 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38 2023-10-18T20:38:20.150 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/iTWIlPytV9Zf+UO+RoicOfLbJE= 2023-10-18T20:38:20.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/iTWIlPytV9Zf+UO+RoicOfLbJE=', 'Content-Length': '0'} 2023-10-18T20:38:20.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001203b04e184934ac-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.152 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.153 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.153 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.154 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:38:20.154 DEBUG:boto:Data: 2023-10-18T20:38:20.154 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.155 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.155 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.155 DEBUG:boto:Params: {} 2023-10-18T20:38:20.156 DEBUG:boto:Token: None 2023-10-18T20:38:20.156 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38 2023-10-18T20:38:20.157 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/iTWIlPytV9Zf+UO+RoicOfLbJE= 2023-10-18T20:38:20.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/iTWIlPytV9Zf+UO+RoicOfLbJE=', 'Content-Length': '0'} 2023-10-18T20:38:20.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0253c345895bb14-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:38:20.160 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.160 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.161 DEBUG:boto:Method: GET 2023-10-18T20:38:20.161 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:38:20.162 DEBUG:boto:Data: 2023-10-18T20:38:20.162 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.162 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.163 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.163 DEBUG:boto:Params: {} 2023-10-18T20:38:20.164 DEBUG:boto:Token: None 2023-10-18T20:38:20.164 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38 2023-10-18T20:38:20.164 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:296Noik2HFI4+DgYz3vd3jamcDM= 2023-10-18T20:38:20.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:296Noik2HFI4+DgYz3vd3jamcDM=', 'Content-Length': '0'} 2023-10-18T20:38:20.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9fa7054c9fe237-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.210 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:38:20.211 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:38:20.211 DEBUG:boto:Method: GET 2023-10-18T20:38:20.211 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:38:20.212 DEBUG:boto:Data: 2023-10-18T20:38:20.212 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.213 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.213 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.214 DEBUG:boto:Params: {} 2023-10-18T20:38:20.214 DEBUG:boto:Token: None 2023-10-18T20:38:20.215 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj38 2023-10-18T20:38:20.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:296Noik2HFI4+DgYz3vd3jamcDM= 2023-10-18T20:38:20.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:296Noik2HFI4+DgYz3vd3jamcDM=', 'Content-Length': '0'} 2023-10-18T20:38:20.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd0b82c909447ecb-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.218 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.219 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.219 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.219 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.220 DEBUG:boto:Method: GET 2023-10-18T20:38:20.220 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.221 DEBUG:boto:Data: 2023-10-18T20:38:20.221 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.221 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.222 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.222 DEBUG:boto:Params: {} 2023-10-18T20:38:20.223 DEBUG:boto:Token: None 2023-10-18T20:38:20.223 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.224 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0bbf835665c3480-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.227 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.227 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.228 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.228 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.228 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.229 DEBUG:boto:Method: GET 2023-10-18T20:38:20.229 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:38:20.229 DEBUG:boto:Data: 2023-10-18T20:38:20.230 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.231 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.231 DEBUG:boto:Params: {} 2023-10-18T20:38:20.231 DEBUG:boto:Token: None 2023-10-18T20:38:20.232 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b92b5815349d41d7-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.236 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:38:20.237 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.237 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.237 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.238 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.238 DEBUG:boto:Method: GET 2023-10-18T20:38:20.238 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.239 DEBUG:boto:Data: 2023-10-18T20:38:20.239 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.240 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.240 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.240 DEBUG:boto:Params: {} 2023-10-18T20:38:20.241 DEBUG:boto:Token: None 2023-10-18T20:38:20.241 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.242 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GF7EKcK0x+yw07A8GAe3n1E+ixg= 2023-10-18T20:38:20.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GF7EKcK0x+yw07A8GAe3n1E+ixg=', 'Content-Length': '0'} 2023-10-18T20:38:20.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002333c6002b49e367-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.245 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.245 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.245 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.246 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.246 DEBUG:boto:Method: GET 2023-10-18T20:38:20.247 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.247 DEBUG:boto:Data: 2023-10-18T20:38:20.247 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.248 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.249 DEBUG:boto:Params: {} 2023-10-18T20:38:20.249 DEBUG:boto:Token: None 2023-10-18T20:38:20.250 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GF7EKcK0x+yw07A8GAe3n1E+ixg= 2023-10-18T20:38:20.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GF7EKcK0x+yw07A8GAe3n1E+ixg=', 'Content-Length': '0'} 2023-10-18T20:38:20.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5c6b68dbb13cf3-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.294 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.295 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.295 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.295 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.296 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.296 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.297 DEBUG:boto:Data: 2023-10-18T20:38:20.297 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.297 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.298 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.298 DEBUG:boto:Params: {} 2023-10-18T20:38:20.299 DEBUG:boto:Token: None 2023-10-18T20:38:20.299 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.300 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:goZa3AYfhnw2DTj+jZThnpPo9QY= 2023-10-18T20:38:20.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:goZa3AYfhnw2DTj+jZThnpPo9QY=', 'Content-Length': '0'} 2023-10-18T20:38:20.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c52f2cd42712b1-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.303 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.303 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.303 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.304 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.304 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.304 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.305 DEBUG:boto:Data: 2023-10-18T20:38:20.305 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.305 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.306 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.306 DEBUG:boto:Params: {} 2023-10-18T20:38:20.307 DEBUG:boto:Token: None 2023-10-18T20:38:20.307 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.307 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:goZa3AYfhnw2DTj+jZThnpPo9QY= 2023-10-18T20:38:20.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:goZa3AYfhnw2DTj+jZThnpPo9QY=', 'Content-Length': '0'} 2023-10-18T20:38:20.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000731bbd8ff80739f0-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:38:20.310 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.311 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.311 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.312 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.312 DEBUG:boto:Method: GET 2023-10-18T20:38:20.312 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.313 DEBUG:boto:Data: 2023-10-18T20:38:20.313 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.314 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.314 DEBUG:boto:Params: {} 2023-10-18T20:38:20.315 DEBUG:boto:Token: None 2023-10-18T20:38:20.315 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.316 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GF7EKcK0x+yw07A8GAe3n1E+ixg= 2023-10-18T20:38:20.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GF7EKcK0x+yw07A8GAe3n1E+ixg=', 'Content-Length': '0'} 2023-10-18T20:38:20.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000f605912bf7118c-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.318 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.319 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.319 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.320 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.320 DEBUG:boto:Method: GET 2023-10-18T20:38:20.320 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.321 DEBUG:boto:Data: 2023-10-18T20:38:20.321 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.322 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.322 DEBUG:boto:Params: {} 2023-10-18T20:38:20.323 DEBUG:boto:Token: None 2023-10-18T20:38:20.323 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:38:20.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GF7EKcK0x+yw07A8GAe3n1E+ixg= 2023-10-18T20:38:20.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GF7EKcK0x+yw07A8GAe3n1E+ixg=', 'Content-Length': '0'} 2023-10-18T20:38:20.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c398626a046c09c2-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.327 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.327 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.328 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.328 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.328 DEBUG:boto:Method: GET 2023-10-18T20:38:20.329 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.329 DEBUG:boto:Data: 2023-10-18T20:38:20.330 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.330 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.331 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.331 DEBUG:boto:Params: {} 2023-10-18T20:38:20.331 DEBUG:boto:Token: None 2023-10-18T20:38:20.332 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.332 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d819a6e297d3931f-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.336 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.336 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.337 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.337 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.337 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.338 DEBUG:boto:Method: GET 2023-10-18T20:38:20.338 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.339 DEBUG:boto:Data: 2023-10-18T20:38:20.339 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.339 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.340 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.340 DEBUG:boto:Params: {} 2023-10-18T20:38:20.341 DEBUG:boto:Token: None 2023-10-18T20:38:20.341 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.341 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a95f2bbca97a48-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.345 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.346 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.346 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.347 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.347 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:38:20.348 DEBUG:boto:Data: 2023-10-18T20:38:20.348 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.348 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.349 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.349 DEBUG:boto:Params: {} 2023-10-18T20:38:20.349 DEBUG:boto:Token: None 2023-10-18T20:38:20.350 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44 2023-10-18T20:38:20.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TYhs7OOMpyelKMbemAogRBaw09E= 2023-10-18T20:38:20.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TYhs7OOMpyelKMbemAogRBaw09E=', 'Content-Length': '0'} 2023-10-18T20:38:20.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9beca8df2d09679-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.353 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.353 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.354 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.354 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:38:20.355 DEBUG:boto:Data: 2023-10-18T20:38:20.355 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.355 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.356 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.356 DEBUG:boto:Params: {} 2023-10-18T20:38:20.356 DEBUG:boto:Token: None 2023-10-18T20:38:20.357 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44 2023-10-18T20:38:20.357 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TYhs7OOMpyelKMbemAogRBaw09E= 2023-10-18T20:38:20.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TYhs7OOMpyelKMbemAogRBaw09E=', 'Content-Length': '0'} 2023-10-18T20:38:20.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf54f4fdc7ef9cde-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:38:20.360 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.361 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.361 DEBUG:boto:Method: GET 2023-10-18T20:38:20.362 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:38:20.362 DEBUG:boto:Data: 2023-10-18T20:38:20.362 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.363 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.363 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.364 DEBUG:boto:Params: {} 2023-10-18T20:38:20.364 DEBUG:boto:Token: None 2023-10-18T20:38:20.364 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44 2023-10-18T20:38:20.365 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GYscU6V5eBsPQnSCFepluF9+WBA= 2023-10-18T20:38:20.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GYscU6V5eBsPQnSCFepluF9+WBA=', 'Content-Length': '0'} 2023-10-18T20:38:20.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008590b8cf2a69ad61-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.414 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:38:20.415 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:38:20.416 DEBUG:boto:Method: GET 2023-10-18T20:38:20.416 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:38:20.416 DEBUG:boto:Data: 2023-10-18T20:38:20.417 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.418 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.418 DEBUG:boto:Params: {} 2023-10-18T20:38:20.419 DEBUG:boto:Token: None 2023-10-18T20:38:20.419 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj44 2023-10-18T20:38:20.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GYscU6V5eBsPQnSCFepluF9+WBA= 2023-10-18T20:38:20.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GYscU6V5eBsPQnSCFepluF9+WBA=', 'Content-Length': '0'} 2023-10-18T20:38:20.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000875557d3ca9ef3fe-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.423 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.423 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.424 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.424 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.424 DEBUG:boto:Method: GET 2023-10-18T20:38:20.425 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.425 DEBUG:boto:Data: 2023-10-18T20:38:20.426 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.427 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.427 DEBUG:boto:Params: {} 2023-10-18T20:38:20.427 DEBUG:boto:Token: None 2023-10-18T20:38:20.428 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac0c05aee264ef40-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.432 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.433 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.433 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.434 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.434 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.434 DEBUG:boto:Method: GET 2023-10-18T20:38:20.435 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:38:20.435 DEBUG:boto:Data: 2023-10-18T20:38:20.436 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.436 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.437 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.437 DEBUG:boto:Params: {} 2023-10-18T20:38:20.437 DEBUG:boto:Token: None 2023-10-18T20:38:20.438 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.438 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000040a606b7a57275e-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.482 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:38:20.484 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.484 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.484 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.485 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.485 DEBUG:boto:Method: GET 2023-10-18T20:38:20.486 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.486 DEBUG:boto:Data: 2023-10-18T20:38:20.487 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.487 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.487 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.488 DEBUG:boto:Params: {} 2023-10-18T20:38:20.488 DEBUG:boto:Token: None 2023-10-18T20:38:20.489 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.489 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:napdPWG9Yg0yn2tQ1XKLl7qn7qo= 2023-10-18T20:38:20.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:napdPWG9Yg0yn2tQ1XKLl7qn7qo=', 'Content-Length': '0'} 2023-10-18T20:38:20.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e23df6909b18f178-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.493 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.493 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.493 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.494 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.494 DEBUG:boto:Method: GET 2023-10-18T20:38:20.495 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.495 DEBUG:boto:Data: 2023-10-18T20:38:20.496 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.496 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.496 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.497 DEBUG:boto:Params: {} 2023-10-18T20:38:20.497 DEBUG:boto:Token: None 2023-10-18T20:38:20.498 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.498 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:napdPWG9Yg0yn2tQ1XKLl7qn7qo= 2023-10-18T20:38:20.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:napdPWG9Yg0yn2tQ1XKLl7qn7qo=', 'Content-Length': '0'} 2023-10-18T20:38:20.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e768131d2604c20-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.502 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.502 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.502 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.503 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.503 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.504 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.504 DEBUG:boto:Data: 2023-10-18T20:38:20.504 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.505 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.505 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.506 DEBUG:boto:Params: {} 2023-10-18T20:38:20.506 DEBUG:boto:Token: None 2023-10-18T20:38:20.506 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.507 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vbkS/ROHf7jC29PWsZy3sYKsoxw= 2023-10-18T20:38:20.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vbkS/ROHf7jC29PWsZy3sYKsoxw=', 'Content-Length': '0'} 2023-10-18T20:38:20.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000585f35cbc40e5236-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.510 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.510 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.511 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.511 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.511 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.512 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.512 DEBUG:boto:Data: 2023-10-18T20:38:20.513 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.513 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.513 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.514 DEBUG:boto:Params: {} 2023-10-18T20:38:20.514 DEBUG:boto:Token: None 2023-10-18T20:38:20.515 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.515 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vbkS/ROHf7jC29PWsZy3sYKsoxw= 2023-10-18T20:38:20.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vbkS/ROHf7jC29PWsZy3sYKsoxw=', 'Content-Length': '0'} 2023-10-18T20:38:20.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7997e3c1df9fadb-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:38:20.518 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.519 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.519 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.520 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.520 DEBUG:boto:Method: GET 2023-10-18T20:38:20.521 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.521 DEBUG:boto:Data: 2023-10-18T20:38:20.522 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.522 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.523 DEBUG:boto:Params: {} 2023-10-18T20:38:20.523 DEBUG:boto:Token: None 2023-10-18T20:38:20.524 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:napdPWG9Yg0yn2tQ1XKLl7qn7qo= 2023-10-18T20:38:20.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:napdPWG9Yg0yn2tQ1XKLl7qn7qo=', 'Content-Length': '0'} 2023-10-18T20:38:20.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4cca75a5cda735-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.528 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.528 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.528 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.529 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.529 DEBUG:boto:Method: GET 2023-10-18T20:38:20.530 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.530 DEBUG:boto:Data: 2023-10-18T20:38:20.531 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.531 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.531 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.532 DEBUG:boto:Params: {} 2023-10-18T20:38:20.532 DEBUG:boto:Token: None 2023-10-18T20:38:20.533 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:38:20.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:napdPWG9Yg0yn2tQ1XKLl7qn7qo= 2023-10-18T20:38:20.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:napdPWG9Yg0yn2tQ1XKLl7qn7qo=', 'Content-Length': '0'} 2023-10-18T20:38:20.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f36d2f953ced0d8b-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.536 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.537 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.537 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.538 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.538 DEBUG:boto:Method: GET 2023-10-18T20:38:20.539 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.539 DEBUG:boto:Data: 2023-10-18T20:38:20.539 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.540 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.541 DEBUG:boto:Params: {} 2023-10-18T20:38:20.541 DEBUG:boto:Token: None 2023-10-18T20:38:20.542 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.542 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000546ce4fe20a9889d-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.546 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.546 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.547 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.547 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.548 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.548 DEBUG:boto:Method: GET 2023-10-18T20:38:20.549 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.549 DEBUG:boto:Data: 2023-10-18T20:38:20.549 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.550 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.551 DEBUG:boto:Params: {} 2023-10-18T20:38:20.551 DEBUG:boto:Token: None 2023-10-18T20:38:20.552 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.552 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000733468cd1ec9325-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.556 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.557 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.557 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.557 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.558 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:38:20.558 DEBUG:boto:Data: 2023-10-18T20:38:20.559 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.559 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.559 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.560 DEBUG:boto:Params: {} 2023-10-18T20:38:20.560 DEBUG:boto:Token: None 2023-10-18T20:38:20.561 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51 2023-10-18T20:38:20.561 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LzGiuw/VF0AtcsgS/DxzL4XoHv0= 2023-10-18T20:38:20.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LzGiuw/VF0AtcsgS/DxzL4XoHv0=', 'Content-Length': '0'} 2023-10-18T20:38:20.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d43b6ac1c34f0e-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.564 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.564 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.564 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.565 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:38:20.565 DEBUG:boto:Data: 2023-10-18T20:38:20.566 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.566 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.567 DEBUG:boto:Params: {} 2023-10-18T20:38:20.567 DEBUG:boto:Token: None 2023-10-18T20:38:20.568 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51 2023-10-18T20:38:20.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LzGiuw/VF0AtcsgS/DxzL4XoHv0= 2023-10-18T20:38:20.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LzGiuw/VF0AtcsgS/DxzL4XoHv0=', 'Content-Length': '0'} 2023-10-18T20:38:20.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ce2711ff962d54-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:38:20.572 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.572 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.573 DEBUG:boto:Method: GET 2023-10-18T20:38:20.573 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:38:20.574 DEBUG:boto:Data: 2023-10-18T20:38:20.574 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.574 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.575 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.575 DEBUG:boto:Params: {} 2023-10-18T20:38:20.576 DEBUG:boto:Token: None 2023-10-18T20:38:20.576 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51 2023-10-18T20:38:20.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zHdkMMDiUGeFRNSWuY7VGye7bf4= 2023-10-18T20:38:20.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zHdkMMDiUGeFRNSWuY7VGye7bf4=', 'Content-Length': '0'} 2023-10-18T20:38:20.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a439687cd928ee8-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.580 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:38:20.580 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:38:20.581 DEBUG:boto:Method: GET 2023-10-18T20:38:20.581 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:38:20.581 DEBUG:boto:Data: 2023-10-18T20:38:20.582 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.583 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.583 DEBUG:boto:Params: {} 2023-10-18T20:38:20.584 DEBUG:boto:Token: None 2023-10-18T20:38:20.584 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj51 2023-10-18T20:38:20.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zHdkMMDiUGeFRNSWuY7VGye7bf4= 2023-10-18T20:38:20.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zHdkMMDiUGeFRNSWuY7VGye7bf4=', 'Content-Length': '0'} 2023-10-18T20:38:20.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bec840659ef44bc-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.630 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.631 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.632 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.632 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.632 DEBUG:boto:Method: GET 2023-10-18T20:38:20.633 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.633 DEBUG:boto:Data: 2023-10-18T20:38:20.634 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.634 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.635 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.635 DEBUG:boto:Params: {} 2023-10-18T20:38:20.636 DEBUG:boto:Token: None 2023-10-18T20:38:20.636 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.637 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce87f2ce448efde1-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.640 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.641 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.641 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.642 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.642 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.643 DEBUG:boto:Method: GET 2023-10-18T20:38:20.643 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:38:20.644 DEBUG:boto:Data: 2023-10-18T20:38:20.644 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.644 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.645 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.645 DEBUG:boto:Params: {} 2023-10-18T20:38:20.646 DEBUG:boto:Token: None 2023-10-18T20:38:20.646 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.647 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4ec9df3e15d5ff-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.650 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:38:20.652 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.652 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.652 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.653 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.653 DEBUG:boto:Method: GET 2023-10-18T20:38:20.654 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.654 DEBUG:boto:Data: 2023-10-18T20:38:20.655 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.655 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.656 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.656 DEBUG:boto:Params: {} 2023-10-18T20:38:20.657 DEBUG:boto:Token: None 2023-10-18T20:38:20.657 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.658 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ragx0yqh049qbKy/OCHOh84vWXs= 2023-10-18T20:38:20.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ragx0yqh049qbKy/OCHOh84vWXs=', 'Content-Length': '0'} 2023-10-18T20:38:20.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfe94505d7c2d7c6-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.702 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.703 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.703 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.704 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.704 DEBUG:boto:Method: GET 2023-10-18T20:38:20.705 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.705 DEBUG:boto:Data: 2023-10-18T20:38:20.706 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.706 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.707 DEBUG:boto:Params: {} 2023-10-18T20:38:20.707 DEBUG:boto:Token: None 2023-10-18T20:38:20.708 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.708 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ragx0yqh049qbKy/OCHOh84vWXs= 2023-10-18T20:38:20.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ragx0yqh049qbKy/OCHOh84vWXs=', 'Content-Length': '0'} 2023-10-18T20:38:20.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000533d090e4b4865ab-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.712 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.712 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.712 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.713 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.713 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.714 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.714 DEBUG:boto:Data: 2023-10-18T20:38:20.715 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.715 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.716 DEBUG:boto:Params: {} 2023-10-18T20:38:20.716 DEBUG:boto:Token: None 2023-10-18T20:38:20.717 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3clqeDxYrIBeETB7RS9nfuy/EYE= 2023-10-18T20:38:20.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3clqeDxYrIBeETB7RS9nfuy/EYE=', 'Content-Length': '0'} 2023-10-18T20:38:20.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9d621decb3eb9dd-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.720 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.720 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.721 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.721 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.722 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.722 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.722 DEBUG:boto:Data: 2023-10-18T20:38:20.723 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.724 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.724 DEBUG:boto:Params: {} 2023-10-18T20:38:20.725 DEBUG:boto:Token: None 2023-10-18T20:38:20.725 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3clqeDxYrIBeETB7RS9nfuy/EYE= 2023-10-18T20:38:20.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3clqeDxYrIBeETB7RS9nfuy/EYE=', 'Content-Length': '0'} 2023-10-18T20:38:20.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049d0621b7871c0c2-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:38:20.729 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.729 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.730 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.730 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.731 DEBUG:boto:Method: GET 2023-10-18T20:38:20.731 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.732 DEBUG:boto:Data: 2023-10-18T20:38:20.732 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.732 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.733 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.733 DEBUG:boto:Params: {} 2023-10-18T20:38:20.734 DEBUG:boto:Token: None 2023-10-18T20:38:20.734 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ragx0yqh049qbKy/OCHOh84vWXs= 2023-10-18T20:38:20.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ragx0yqh049qbKy/OCHOh84vWXs=', 'Content-Length': '0'} 2023-10-18T20:38:20.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab6f5276856336ca-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.738 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.738 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.739 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.739 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.740 DEBUG:boto:Method: GET 2023-10-18T20:38:20.740 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.741 DEBUG:boto:Data: 2023-10-18T20:38:20.741 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.742 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.742 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.743 DEBUG:boto:Params: {} 2023-10-18T20:38:20.743 DEBUG:boto:Token: None 2023-10-18T20:38:20.744 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:38:20.744 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ragx0yqh049qbKy/OCHOh84vWXs= 2023-10-18T20:38:20.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ragx0yqh049qbKy/OCHOh84vWXs=', 'Content-Length': '0'} 2023-10-18T20:38:20.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003ee17c9f8ff8412-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.747 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.747 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.748 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.748 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.749 DEBUG:boto:Method: GET 2023-10-18T20:38:20.749 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.750 DEBUG:boto:Data: 2023-10-18T20:38:20.750 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.751 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.751 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.752 DEBUG:boto:Params: {} 2023-10-18T20:38:20.752 DEBUG:boto:Token: None 2023-10-18T20:38:20.753 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.753 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000130c873dead4271c-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.756 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.757 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.758 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.758 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.759 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.759 DEBUG:boto:Method: GET 2023-10-18T20:38:20.760 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.760 DEBUG:boto:Data: 2023-10-18T20:38:20.760 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.761 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.762 DEBUG:boto:Params: {} 2023-10-18T20:38:20.762 DEBUG:boto:Token: None 2023-10-18T20:38:20.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7c2983ff22c304d-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.810 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.811 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.812 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.812 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.813 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:38:20.813 DEBUG:boto:Data: 2023-10-18T20:38:20.813 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.814 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.815 DEBUG:boto:Params: {} 2023-10-18T20:38:20.815 DEBUG:boto:Token: None 2023-10-18T20:38:20.816 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55 2023-10-18T20:38:20.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0XLtsCn1qfccU8ygmyztnhPIA/g= 2023-10-18T20:38:20.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0XLtsCn1qfccU8ygmyztnhPIA/g=', 'Content-Length': '0'} 2023-10-18T20:38:20.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32f217249c0bc1e-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.819 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.820 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.820 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.821 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:38:20.821 DEBUG:boto:Data: 2023-10-18T20:38:20.821 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.822 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.822 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.823 DEBUG:boto:Params: {} 2023-10-18T20:38:20.823 DEBUG:boto:Token: None 2023-10-18T20:38:20.824 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55 2023-10-18T20:38:20.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0XLtsCn1qfccU8ygmyztnhPIA/g= 2023-10-18T20:38:20.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0XLtsCn1qfccU8ygmyztnhPIA/g=', 'Content-Length': '0'} 2023-10-18T20:38:20.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007043531866fd49fa-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:38:20.827 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.829 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.829 DEBUG:boto:Method: GET 2023-10-18T20:38:20.830 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:38:20.830 DEBUG:boto:Data: 2023-10-18T20:38:20.831 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.831 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.832 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.832 DEBUG:boto:Params: {} 2023-10-18T20:38:20.832 DEBUG:boto:Token: None 2023-10-18T20:38:20.833 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55 2023-10-18T20:38:20.833 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1B2Xd+oenDWIbZtOonrEX2JLNC4= 2023-10-18T20:38:20.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1B2Xd+oenDWIbZtOonrEX2JLNC4=', 'Content-Length': '0'} 2023-10-18T20:38:20.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad9be7a2f69c1d62-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.836 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:38:20.837 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:38:20.838 DEBUG:boto:Method: GET 2023-10-18T20:38:20.838 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:38:20.839 DEBUG:boto:Data: 2023-10-18T20:38:20.839 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.840 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.841 DEBUG:boto:Params: {} 2023-10-18T20:38:20.841 DEBUG:boto:Token: None 2023-10-18T20:38:20.842 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj55 2023-10-18T20:38:20.842 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1B2Xd+oenDWIbZtOonrEX2JLNC4= 2023-10-18T20:38:20.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1B2Xd+oenDWIbZtOonrEX2JLNC4=', 'Content-Length': '0'} 2023-10-18T20:38:20.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b2627babf7afe4-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.845 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.846 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.846 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.847 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.847 DEBUG:boto:Method: GET 2023-10-18T20:38:20.848 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.848 DEBUG:boto:Data: 2023-10-18T20:38:20.848 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.849 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.849 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.849 DEBUG:boto:Params: {} 2023-10-18T20:38:20.850 DEBUG:boto:Token: None 2023-10-18T20:38:20.851 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.851 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089dfbeaf59434632-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.854 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.855 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.856 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.856 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.857 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.857 DEBUG:boto:Method: GET 2023-10-18T20:38:20.858 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:38:20.858 DEBUG:boto:Data: 2023-10-18T20:38:20.858 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.859 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.860 DEBUG:boto:Params: {} 2023-10-18T20:38:20.860 DEBUG:boto:Token: None 2023-10-18T20:38:20.861 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.861 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000977b42b7dcc7d81c-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.865 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:38:20.866 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:20.867 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:20.867 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.867 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.868 DEBUG:boto:Method: GET 2023-10-18T20:38:20.868 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.869 DEBUG:boto:Data: 2023-10-18T20:38:20.869 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.870 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.870 DEBUG:boto:Params: {} 2023-10-18T20:38:20.871 DEBUG:boto:Token: None 2023-10-18T20:38:20.871 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4eE1/Gdvc4Rjk3i22lAEuUVTrJA= 2023-10-18T20:38:20.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4eE1/Gdvc4Rjk3i22lAEuUVTrJA=', 'Content-Length': '0'} 2023-10-18T20:38:20.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006a7f86de0ae20a0-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.875 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:20.875 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:20.876 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.876 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.877 DEBUG:boto:Method: GET 2023-10-18T20:38:20.877 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.877 DEBUG:boto:Data: 2023-10-18T20:38:20.878 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.878 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.879 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.879 DEBUG:boto:Params: {} 2023-10-18T20:38:20.880 DEBUG:boto:Token: None 2023-10-18T20:38:20.880 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.881 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4eE1/Gdvc4Rjk3i22lAEuUVTrJA= 2023-10-18T20:38:20.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4eE1/Gdvc4Rjk3i22lAEuUVTrJA=', 'Content-Length': '0'} 2023-10-18T20:38:20.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ef059090f0b11a3-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.884 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:20.884 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:20.885 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.885 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.885 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.886 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.886 DEBUG:boto:Data: 2023-10-18T20:38:20.887 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.887 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.887 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.888 DEBUG:boto:Params: {} 2023-10-18T20:38:20.888 DEBUG:boto:Token: None 2023-10-18T20:38:20.889 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.889 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rmamkQWyeaRtKWNSWsYTa2RJUzI= 2023-10-18T20:38:20.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rmamkQWyeaRtKWNSWsYTa2RJUzI=', 'Content-Length': '0'} 2023-10-18T20:38:20.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee5d56982a4e9692-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.892 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:20.892 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:20.893 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.893 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.893 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.894 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.894 DEBUG:boto:Data: 2023-10-18T20:38:20.895 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.895 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.896 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.896 DEBUG:boto:Params: {} 2023-10-18T20:38:20.896 DEBUG:boto:Token: None 2023-10-18T20:38:20.897 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.897 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rmamkQWyeaRtKWNSWsYTa2RJUzI= 2023-10-18T20:38:20.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rmamkQWyeaRtKWNSWsYTa2RJUzI=', 'Content-Length': '0'} 2023-10-18T20:38:20.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009952a5169c60e676-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:38:20.900 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:20.901 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:20.901 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.902 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.902 DEBUG:boto:Method: GET 2023-10-18T20:38:20.902 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.903 DEBUG:boto:Data: 2023-10-18T20:38:20.903 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.904 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.904 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.905 DEBUG:boto:Params: {} 2023-10-18T20:38:20.905 DEBUG:boto:Token: None 2023-10-18T20:38:20.905 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.906 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4eE1/Gdvc4Rjk3i22lAEuUVTrJA= 2023-10-18T20:38:20.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4eE1/Gdvc4Rjk3i22lAEuUVTrJA=', 'Content-Length': '0'} 2023-10-18T20:38:20.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007081d857f73f4d15-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.954 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:20.955 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:20.955 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.956 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.956 DEBUG:boto:Method: GET 2023-10-18T20:38:20.957 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.957 DEBUG:boto:Data: 2023-10-18T20:38:20.958 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.958 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.959 DEBUG:boto:Params: {} 2023-10-18T20:38:20.959 DEBUG:boto:Token: None 2023-10-18T20:38:20.960 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:38:20.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4eE1/Gdvc4Rjk3i22lAEuUVTrJA= 2023-10-18T20:38:20.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4eE1/Gdvc4Rjk3i22lAEuUVTrJA=', 'Content-Length': '0'} 2023-10-18T20:38:20.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f79c4e308f3c69c-006530423c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.964 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.965 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.966 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:38:20.966 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:38:20.967 DEBUG:boto:Method: GET 2023-10-18T20:38:20.967 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:38:20.968 DEBUG:boto:Data: 2023-10-18T20:38:20.968 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.969 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.969 DEBUG:boto:Params: {} 2023-10-18T20:38:20.970 DEBUG:boto:Token: None 2023-10-18T20:38:20.970 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.971 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054555a710df554c3-006530423c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.975 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.975 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:20.976 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:20.976 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:38:20.977 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:38:20.977 DEBUG:boto:Method: GET 2023-10-18T20:38:20.978 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:38:20.978 DEBUG:boto:Data: 2023-10-18T20:38:20.979 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.979 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.979 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.980 DEBUG:boto:Params: {} 2023-10-18T20:38:20.980 DEBUG:boto:Token: None 2023-10-18T20:38:20.981 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/ 2023-10-18T20:38:20.981 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s= 2023-10-18T20:38:20.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BSuNNpUMMnxs9dpZVUyRXAUmo4s=', 'Content-Length': '0'} 2023-10-18T20:38:20.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004473122d2b9194d0-006530423c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.985 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:20.986 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:20.986 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:20.987 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.987 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:38:20.988 DEBUG:boto:Data: 2023-10-18T20:38:20.988 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.988 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:20.989 DEBUG:boto:Port: 8001 2023-10-18T20:38:20.989 DEBUG:boto:Params: {} 2023-10-18T20:38:20.990 DEBUG:boto:Token: None 2023-10-18T20:38:20.990 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj57 2023-10-18T20:38:20.991 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T5PpC2PTaWxCbodk5af2TnY+TOw= 2023-10-18T20:38:20.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T5PpC2PTaWxCbodk5af2TnY+TOw=', 'Content-Length': '0'} 2023-10-18T20:38:20.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8815a5846a2d5a6-006530423c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:20.994 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:20.994 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:20.995 DEBUG:boto:Method: HEAD 2023-10-18T20:38:20.995 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:38:20.996 DEBUG:boto:Data: 2023-10-18T20:38:20.996 DEBUG:boto:Headers: {} 2023-10-18T20:38:20.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:20.997 DEBUG:boto:Port: 8000 2023-10-18T20:38:20.997 DEBUG:boto:Params: {} 2023-10-18T20:38:20.998 DEBUG:boto:Token: None 2023-10-18T20:38:20.999 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:20 GMT /bofrnf-59/obj57 2023-10-18T20:38:20.999 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T5PpC2PTaWxCbodk5af2TnY+TOw= 2023-10-18T20:38:21.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T5PpC2PTaWxCbodk5af2TnY+TOw=', 'Content-Length': '0'} 2023-10-18T20:38:21.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045fe84ad7978710b-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:38:21.003 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:21.003 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:21.003 DEBUG:boto:Method: GET 2023-10-18T20:38:21.004 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:38:21.004 DEBUG:boto:Data: 2023-10-18T20:38:21.005 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.005 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.005 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.006 DEBUG:boto:Params: {} 2023-10-18T20:38:21.006 DEBUG:boto:Token: None 2023-10-18T20:38:21.007 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj57 2023-10-18T20:38:21.007 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:61M1aJ05z9sGbn9J1dLq1j13Gr8= 2023-10-18T20:38:21.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:61M1aJ05z9sGbn9J1dLq1j13Gr8=', 'Content-Length': '0'} 2023-10-18T20:38:21.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bb5e65e7777407a-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.010 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:38:21.011 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:38:21.011 DEBUG:boto:Method: GET 2023-10-18T20:38:21.011 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:38:21.012 DEBUG:boto:Data: 2023-10-18T20:38:21.012 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.013 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.013 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.014 DEBUG:boto:Params: {} 2023-10-18T20:38:21.014 DEBUG:boto:Token: None 2023-10-18T20:38:21.015 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj57 2023-10-18T20:38:21.015 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:61M1aJ05z9sGbn9J1dLq1j13Gr8= 2023-10-18T20:38:21.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:61M1aJ05z9sGbn9J1dLq1j13Gr8=', 'Content-Length': '0'} 2023-10-18T20:38:21.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9569271bff92431-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.018 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.019 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.019 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:38:21.019 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:38:21.020 DEBUG:boto:Method: GET 2023-10-18T20:38:21.020 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:38:21.021 DEBUG:boto:Data: 2023-10-18T20:38:21.021 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.021 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.022 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.022 DEBUG:boto:Params: {} 2023-10-18T20:38:21.023 DEBUG:boto:Token: None 2023-10-18T20:38:21.023 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.024 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a4f4553cb647354-006530423d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.027 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.028 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.028 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.029 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:38:21.029 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:38:21.029 DEBUG:boto:Method: GET 2023-10-18T20:38:21.030 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:38:21.030 DEBUG:boto:Data: 2023-10-18T20:38:21.031 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.031 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.031 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.032 DEBUG:boto:Params: {} 2023-10-18T20:38:21.032 DEBUG:boto:Token: None 2023-10-18T20:38:21.033 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.033 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000511f6c330774b98c-006530423d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.038 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:38:21.039 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.040 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.040 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.041 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.042 DEBUG:boto:Method: GET 2023-10-18T20:38:21.042 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.043 DEBUG:boto:Data: 2023-10-18T20:38:21.044 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.044 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.045 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.046 DEBUG:boto:Params: {} 2023-10-18T20:38:21.047 DEBUG:boto:Token: None 2023-10-18T20:38:21.047 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.048 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HiXrEhTKF6CMnN1l8A78tHrz+tk= 2023-10-18T20:38:21.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HiXrEhTKF6CMnN1l8A78tHrz+tk=', 'Content-Length': '0'} 2023-10-18T20:38:21.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000668a72cbfcc54464-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.094 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.095 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.095 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.096 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.096 DEBUG:boto:Method: GET 2023-10-18T20:38:21.097 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.097 DEBUG:boto:Data: 2023-10-18T20:38:21.098 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.099 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.099 DEBUG:boto:Params: {} 2023-10-18T20:38:21.100 DEBUG:boto:Token: None 2023-10-18T20:38:21.100 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.101 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HiXrEhTKF6CMnN1l8A78tHrz+tk= 2023-10-18T20:38:21.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HiXrEhTKF6CMnN1l8A78tHrz+tk=', 'Content-Length': '0'} 2023-10-18T20:38:21.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccd5d8a123ae6422-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.104 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.105 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.105 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.106 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.106 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.107 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.107 DEBUG:boto:Data: 2023-10-18T20:38:21.108 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.108 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.108 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.109 DEBUG:boto:Params: {} 2023-10-18T20:38:21.109 DEBUG:boto:Token: None 2023-10-18T20:38:21.110 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.110 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rSxU78bBnYL5DMenKYDHtErVD80= 2023-10-18T20:38:21.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rSxU78bBnYL5DMenKYDHtErVD80=', 'Content-Length': '0'} 2023-10-18T20:38:21.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ab029db3c587d82-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.113 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.113 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.114 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.114 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.115 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.115 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.115 DEBUG:boto:Data: 2023-10-18T20:38:21.116 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.116 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.117 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.117 DEBUG:boto:Params: {} 2023-10-18T20:38:21.118 DEBUG:boto:Token: None 2023-10-18T20:38:21.118 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.119 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rSxU78bBnYL5DMenKYDHtErVD80= 2023-10-18T20:38:21.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rSxU78bBnYL5DMenKYDHtErVD80=', 'Content-Length': '0'} 2023-10-18T20:38:21.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424c5ba73cfa3ddf-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:38:21.122 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.122 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.123 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.123 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.124 DEBUG:boto:Method: GET 2023-10-18T20:38:21.124 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.124 DEBUG:boto:Data: 2023-10-18T20:38:21.125 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.126 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.126 DEBUG:boto:Params: {} 2023-10-18T20:38:21.127 DEBUG:boto:Token: None 2023-10-18T20:38:21.127 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.128 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HiXrEhTKF6CMnN1l8A78tHrz+tk= 2023-10-18T20:38:21.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HiXrEhTKF6CMnN1l8A78tHrz+tk=', 'Content-Length': '0'} 2023-10-18T20:38:21.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a13eca43bdcfea3-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.130 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.131 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.131 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.132 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.132 DEBUG:boto:Method: GET 2023-10-18T20:38:21.132 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.133 DEBUG:boto:Data: 2023-10-18T20:38:21.133 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.134 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.134 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.135 DEBUG:boto:Params: {} 2023-10-18T20:38:21.135 DEBUG:boto:Token: None 2023-10-18T20:38:21.135 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:38:21.136 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HiXrEhTKF6CMnN1l8A78tHrz+tk= 2023-10-18T20:38:21.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HiXrEhTKF6CMnN1l8A78tHrz+tk=', 'Content-Length': '0'} 2023-10-18T20:38:21.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014fe984659846f10-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.139 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.139 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.140 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.140 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.141 DEBUG:boto:Method: GET 2023-10-18T20:38:21.141 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.141 DEBUG:boto:Data: 2023-10-18T20:38:21.142 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.143 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.143 DEBUG:boto:Params: {} 2023-10-18T20:38:21.143 DEBUG:boto:Token: None 2023-10-18T20:38:21.144 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8f40c4238d40b91-006530423d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.148 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.149 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.149 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.150 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.150 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.150 DEBUG:boto:Method: GET 2023-10-18T20:38:21.151 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.151 DEBUG:boto:Data: 2023-10-18T20:38:21.152 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.152 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.153 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.153 DEBUG:boto:Params: {} 2023-10-18T20:38:21.154 DEBUG:boto:Token: None 2023-10-18T20:38:21.154 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.155 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009fe6f1e1668c492-006530423d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.198 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.199 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.200 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.200 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.201 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:38:21.201 DEBUG:boto:Data: 2023-10-18T20:38:21.201 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.202 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.202 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.203 DEBUG:boto:Params: {} 2023-10-18T20:38:21.203 DEBUG:boto:Token: None 2023-10-18T20:38:21.204 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68 2023-10-18T20:38:21.204 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z+FerohR7IIhA0HiQcA73LWOtrc= 2023-10-18T20:38:21.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z+FerohR7IIhA0HiQcA73LWOtrc=', 'Content-Length': '0'} 2023-10-18T20:38:21.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075b8ad5d081a74e0-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.207 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.208 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.208 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.209 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:38:21.209 DEBUG:boto:Data: 2023-10-18T20:38:21.209 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.210 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.211 DEBUG:boto:Params: {} 2023-10-18T20:38:21.212 DEBUG:boto:Token: None 2023-10-18T20:38:21.212 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68 2023-10-18T20:38:21.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z+FerohR7IIhA0HiQcA73LWOtrc= 2023-10-18T20:38:21.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z+FerohR7IIhA0HiQcA73LWOtrc=', 'Content-Length': '0'} 2023-10-18T20:38:21.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c96267aa918195d7-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:38:21.217 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.217 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.217 DEBUG:boto:Method: GET 2023-10-18T20:38:21.218 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:38:21.218 DEBUG:boto:Data: 2023-10-18T20:38:21.219 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.220 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.220 DEBUG:boto:Params: {} 2023-10-18T20:38:21.221 DEBUG:boto:Token: None 2023-10-18T20:38:21.221 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68 2023-10-18T20:38:21.222 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W3HtfWwD3XFPjmat1PzH/57MP3M= 2023-10-18T20:38:21.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W3HtfWwD3XFPjmat1PzH/57MP3M=', 'Content-Length': '0'} 2023-10-18T20:38:21.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a6e59ce1bdafdf4-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.225 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:38:21.226 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:38:21.226 DEBUG:boto:Method: GET 2023-10-18T20:38:21.227 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:38:21.227 DEBUG:boto:Data: 2023-10-18T20:38:21.228 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.228 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.229 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.229 DEBUG:boto:Params: {} 2023-10-18T20:38:21.229 DEBUG:boto:Token: None 2023-10-18T20:38:21.230 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj68 2023-10-18T20:38:21.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W3HtfWwD3XFPjmat1PzH/57MP3M= 2023-10-18T20:38:21.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W3HtfWwD3XFPjmat1PzH/57MP3M=', 'Content-Length': '0'} 2023-10-18T20:38:21.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba009a5413286442-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.234 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.234 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.235 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.235 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.235 DEBUG:boto:Method: GET 2023-10-18T20:38:21.236 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.236 DEBUG:boto:Data: 2023-10-18T20:38:21.237 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.237 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.238 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.238 DEBUG:boto:Params: {} 2023-10-18T20:38:21.239 DEBUG:boto:Token: None 2023-10-18T20:38:21.239 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.239 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c2bc1f148e66beb-006530423d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.243 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.243 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.244 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.244 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.245 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.245 DEBUG:boto:Method: GET 2023-10-18T20:38:21.246 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:38:21.246 DEBUG:boto:Data: 2023-10-18T20:38:21.246 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.247 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.247 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.248 DEBUG:boto:Params: {} 2023-10-18T20:38:21.248 DEBUG:boto:Token: None 2023-10-18T20:38:21.249 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.249 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095160cf7e1050c27-006530423d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.253 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:38:21.254 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.254 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.255 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.255 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.256 DEBUG:boto:Method: GET 2023-10-18T20:38:21.256 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.256 DEBUG:boto:Data: 2023-10-18T20:38:21.257 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.257 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.258 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.258 DEBUG:boto:Params: {} 2023-10-18T20:38:21.259 DEBUG:boto:Token: None 2023-10-18T20:38:21.259 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5iAm/72cjduzc2iCSjb54/syhpU= 2023-10-18T20:38:21.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5iAm/72cjduzc2iCSjb54/syhpU=', 'Content-Length': '0'} 2023-10-18T20:38:21.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096ef5015172a4b6e-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.302 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.303 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.303 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.304 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.304 DEBUG:boto:Method: GET 2023-10-18T20:38:21.305 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.305 DEBUG:boto:Data: 2023-10-18T20:38:21.306 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.306 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.307 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.307 DEBUG:boto:Params: {} 2023-10-18T20:38:21.307 DEBUG:boto:Token: None 2023-10-18T20:38:21.308 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.309 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5iAm/72cjduzc2iCSjb54/syhpU= 2023-10-18T20:38:21.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5iAm/72cjduzc2iCSjb54/syhpU=', 'Content-Length': '0'} 2023-10-18T20:38:21.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061be66ba2f9c8c44-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.312 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.312 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.313 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.313 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.314 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.314 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.315 DEBUG:boto:Data: 2023-10-18T20:38:21.315 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.315 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.316 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.316 DEBUG:boto:Params: {} 2023-10-18T20:38:21.317 DEBUG:boto:Token: None 2023-10-18T20:38:21.317 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.318 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lePPKw4Ej/tkIR/LclGecf1sIpo= 2023-10-18T20:38:21.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lePPKw4Ej/tkIR/LclGecf1sIpo=', 'Content-Length': '0'} 2023-10-18T20:38:21.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b2a4b17bebbbd8-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.321 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.321 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.321 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.322 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.322 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.323 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.323 DEBUG:boto:Data: 2023-10-18T20:38:21.323 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.324 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.324 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.325 DEBUG:boto:Params: {} 2023-10-18T20:38:21.325 DEBUG:boto:Token: None 2023-10-18T20:38:21.326 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.326 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lePPKw4Ej/tkIR/LclGecf1sIpo= 2023-10-18T20:38:21.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lePPKw4Ej/tkIR/LclGecf1sIpo=', 'Content-Length': '0'} 2023-10-18T20:38:21.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e24934a4df20331-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:38:21.329 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.330 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.330 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.331 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.331 DEBUG:boto:Method: GET 2023-10-18T20:38:21.331 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.332 DEBUG:boto:Data: 2023-10-18T20:38:21.332 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.333 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.333 DEBUG:boto:Params: {} 2023-10-18T20:38:21.334 DEBUG:boto:Token: None 2023-10-18T20:38:21.334 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.335 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5iAm/72cjduzc2iCSjb54/syhpU= 2023-10-18T20:38:21.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5iAm/72cjduzc2iCSjb54/syhpU=', 'Content-Length': '0'} 2023-10-18T20:38:21.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003809a7e1a3b3ea1b-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.338 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.338 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.339 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.339 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.339 DEBUG:boto:Method: GET 2023-10-18T20:38:21.340 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.340 DEBUG:boto:Data: 2023-10-18T20:38:21.341 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.341 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.341 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.342 DEBUG:boto:Params: {} 2023-10-18T20:38:21.342 DEBUG:boto:Token: None 2023-10-18T20:38:21.343 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:38:21.343 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5iAm/72cjduzc2iCSjb54/syhpU= 2023-10-18T20:38:21.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5iAm/72cjduzc2iCSjb54/syhpU=', 'Content-Length': '0'} 2023-10-18T20:38:21.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b0f10a267e73d2-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.386 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.387 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.387 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.388 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.388 DEBUG:boto:Method: GET 2023-10-18T20:38:21.389 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.389 DEBUG:boto:Data: 2023-10-18T20:38:21.390 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.391 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.391 DEBUG:boto:Params: {} 2023-10-18T20:38:21.391 DEBUG:boto:Token: None 2023-10-18T20:38:21.392 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015d847287ceea343-006530423d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.396 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.397 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.397 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.398 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.398 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.398 DEBUG:boto:Method: GET 2023-10-18T20:38:21.399 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.399 DEBUG:boto:Data: 2023-10-18T20:38:21.400 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.400 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.400 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.401 DEBUG:boto:Params: {} 2023-10-18T20:38:21.401 DEBUG:boto:Token: None 2023-10-18T20:38:21.402 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000017f9ec83d2480ff-006530423d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.406 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.407 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.407 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.408 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.408 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:38:21.409 DEBUG:boto:Data: 2023-10-18T20:38:21.409 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.410 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.410 DEBUG:boto:Params: {} 2023-10-18T20:38:21.411 DEBUG:boto:Token: None 2023-10-18T20:38:21.411 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76 2023-10-18T20:38:21.412 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VYrbuCskFq959BkG9QFwdhwcGUo= 2023-10-18T20:38:21.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VYrbuCskFq959BkG9QFwdhwcGUo=', 'Content-Length': '0'} 2023-10-18T20:38:21.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054a4ac663bd3a68c-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.415 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.415 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.416 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.416 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:38:21.416 DEBUG:boto:Data: 2023-10-18T20:38:21.417 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.418 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.418 DEBUG:boto:Params: {} 2023-10-18T20:38:21.419 DEBUG:boto:Token: None 2023-10-18T20:38:21.419 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76 2023-10-18T20:38:21.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VYrbuCskFq959BkG9QFwdhwcGUo= 2023-10-18T20:38:21.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VYrbuCskFq959BkG9QFwdhwcGUo=', 'Content-Length': '0'} 2023-10-18T20:38:21.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e69dd36cecd3a332-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:38:21.423 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.423 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.424 DEBUG:boto:Method: GET 2023-10-18T20:38:21.424 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:38:21.424 DEBUG:boto:Data: 2023-10-18T20:38:21.425 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.426 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.426 DEBUG:boto:Params: {} 2023-10-18T20:38:21.427 DEBUG:boto:Token: None 2023-10-18T20:38:21.427 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76 2023-10-18T20:38:21.427 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vd9ORnfjP+hM0j8fSpAVvm5i80k= 2023-10-18T20:38:21.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vd9ORnfjP+hM0j8fSpAVvm5i80k=', 'Content-Length': '0'} 2023-10-18T20:38:21.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000165659a124d3a419-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.430 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:38:21.431 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:38:21.431 DEBUG:boto:Method: GET 2023-10-18T20:38:21.432 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:38:21.432 DEBUG:boto:Data: 2023-10-18T20:38:21.433 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.433 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.434 DEBUG:boto:Params: {} 2023-10-18T20:38:21.434 DEBUG:boto:Token: None 2023-10-18T20:38:21.435 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj76 2023-10-18T20:38:21.435 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vd9ORnfjP+hM0j8fSpAVvm5i80k= 2023-10-18T20:38:21.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vd9ORnfjP+hM0j8fSpAVvm5i80k=', 'Content-Length': '0'} 2023-10-18T20:38:21.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3246ee1cd492cce-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.438 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.439 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.439 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.440 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.440 DEBUG:boto:Method: GET 2023-10-18T20:38:21.441 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.441 DEBUG:boto:Data: 2023-10-18T20:38:21.441 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.442 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.443 DEBUG:boto:Params: {} 2023-10-18T20:38:21.443 DEBUG:boto:Token: None 2023-10-18T20:38:21.444 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.444 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f69730eaa5c03401-006530423d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.448 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.448 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.449 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.449 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.450 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.450 DEBUG:boto:Method: GET 2023-10-18T20:38:21.451 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:38:21.451 DEBUG:boto:Data: 2023-10-18T20:38:21.452 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.453 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.453 DEBUG:boto:Params: {} 2023-10-18T20:38:21.454 DEBUG:boto:Token: None 2023-10-18T20:38:21.454 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd68c8679eda99a9-006530423d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.459 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:38:21.460 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.461 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.461 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.462 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.462 DEBUG:boto:Method: GET 2023-10-18T20:38:21.462 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.463 DEBUG:boto:Data: 2023-10-18T20:38:21.463 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.464 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.464 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.465 DEBUG:boto:Params: {} 2023-10-18T20:38:21.465 DEBUG:boto:Token: None 2023-10-18T20:38:21.466 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.466 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MAvqidQ+DOUAN+doegvjO81sro4= 2023-10-18T20:38:21.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MAvqidQ+DOUAN+doegvjO81sro4=', 'Content-Length': '0'} 2023-10-18T20:38:21.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223d28763162ba86-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.510 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.512 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.512 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.513 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.514 DEBUG:boto:Method: GET 2023-10-18T20:38:21.515 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.515 DEBUG:boto:Data: 2023-10-18T20:38:21.516 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.517 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.517 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.518 DEBUG:boto:Params: {} 2023-10-18T20:38:21.518 DEBUG:boto:Token: None 2023-10-18T20:38:21.519 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.519 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MAvqidQ+DOUAN+doegvjO81sro4= 2023-10-18T20:38:21.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MAvqidQ+DOUAN+doegvjO81sro4=', 'Content-Length': '0'} 2023-10-18T20:38:21.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002dd5330158fb93-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.522 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.523 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.523 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.523 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.524 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.524 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.525 DEBUG:boto:Data: 2023-10-18T20:38:21.525 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.526 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.526 DEBUG:boto:Params: {} 2023-10-18T20:38:21.527 DEBUG:boto:Token: None 2023-10-18T20:38:21.527 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.527 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0tqRlVXo+ThRyY9EspWhnU0d+Xs= 2023-10-18T20:38:21.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0tqRlVXo+ThRyY9EspWhnU0d+Xs=', 'Content-Length': '0'} 2023-10-18T20:38:21.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a69747af116bfda7-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.531 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.531 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.531 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.532 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.532 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.533 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.533 DEBUG:boto:Data: 2023-10-18T20:38:21.533 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.534 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.535 DEBUG:boto:Params: {} 2023-10-18T20:38:21.535 DEBUG:boto:Token: None 2023-10-18T20:38:21.536 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0tqRlVXo+ThRyY9EspWhnU0d+Xs= 2023-10-18T20:38:21.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0tqRlVXo+ThRyY9EspWhnU0d+Xs=', 'Content-Length': '0'} 2023-10-18T20:38:21.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004855e23edb9ab25b-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:38:21.540 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.541 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.541 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.542 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.543 DEBUG:boto:Method: GET 2023-10-18T20:38:21.543 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.544 DEBUG:boto:Data: 2023-10-18T20:38:21.545 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.545 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.546 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.547 DEBUG:boto:Params: {} 2023-10-18T20:38:21.547 DEBUG:boto:Token: None 2023-10-18T20:38:21.547 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.548 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MAvqidQ+DOUAN+doegvjO81sro4= 2023-10-18T20:38:21.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MAvqidQ+DOUAN+doegvjO81sro4=', 'Content-Length': '0'} 2023-10-18T20:38:21.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009268c9777164c4c3-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.551 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.551 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.552 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.552 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.553 DEBUG:boto:Method: GET 2023-10-18T20:38:21.553 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.553 DEBUG:boto:Data: 2023-10-18T20:38:21.554 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.554 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.555 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.555 DEBUG:boto:Params: {} 2023-10-18T20:38:21.555 DEBUG:boto:Token: None 2023-10-18T20:38:21.556 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:38:21.557 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MAvqidQ+DOUAN+doegvjO81sro4= 2023-10-18T20:38:21.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MAvqidQ+DOUAN+doegvjO81sro4=', 'Content-Length': '0'} 2023-10-18T20:38:21.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b60f89ef1415f405-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.560 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.560 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.560 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.561 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.561 DEBUG:boto:Method: GET 2023-10-18T20:38:21.562 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.562 DEBUG:boto:Data: 2023-10-18T20:38:21.562 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.563 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.563 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.564 DEBUG:boto:Params: {} 2023-10-18T20:38:21.564 DEBUG:boto:Token: None 2023-10-18T20:38:21.565 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.565 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000965566df285680da-006530423d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.569 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.569 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.570 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.570 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.571 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.571 DEBUG:boto:Method: GET 2023-10-18T20:38:21.572 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.572 DEBUG:boto:Data: 2023-10-18T20:38:21.573 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.573 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.574 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.575 DEBUG:boto:Params: {} 2023-10-18T20:38:21.576 DEBUG:boto:Token: None 2023-10-18T20:38:21.576 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067574c318b3b0d0d-006530423d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.622 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.624 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.624 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.625 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.625 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:38:21.626 DEBUG:boto:Data: 2023-10-18T20:38:21.626 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.627 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.627 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.627 DEBUG:boto:Params: {} 2023-10-18T20:38:21.628 DEBUG:boto:Token: None 2023-10-18T20:38:21.628 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85 2023-10-18T20:38:21.629 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uxi2MrjVN8ZZrB2bQdJVXMJLaIw= 2023-10-18T20:38:21.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uxi2MrjVN8ZZrB2bQdJVXMJLaIw=', 'Content-Length': '0'} 2023-10-18T20:38:21.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006021e3f6a174b39a-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.632 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.633 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.633 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.633 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:38:21.634 DEBUG:boto:Data: 2023-10-18T20:38:21.634 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.635 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.635 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.635 DEBUG:boto:Params: {} 2023-10-18T20:38:21.636 DEBUG:boto:Token: None 2023-10-18T20:38:21.636 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85 2023-10-18T20:38:21.637 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uxi2MrjVN8ZZrB2bQdJVXMJLaIw= 2023-10-18T20:38:21.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uxi2MrjVN8ZZrB2bQdJVXMJLaIw=', 'Content-Length': '0'} 2023-10-18T20:38:21.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829e0f9423ab8e4d-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:38:21.640 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.640 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.641 DEBUG:boto:Method: GET 2023-10-18T20:38:21.641 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:38:21.642 DEBUG:boto:Data: 2023-10-18T20:38:21.642 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.643 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.643 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.644 DEBUG:boto:Params: {} 2023-10-18T20:38:21.644 DEBUG:boto:Token: None 2023-10-18T20:38:21.645 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85 2023-10-18T20:38:21.645 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YyZAem+Sl0i1JP8YEWvbfZnk2Hc= 2023-10-18T20:38:21.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YyZAem+Sl0i1JP8YEWvbfZnk2Hc=', 'Content-Length': '0'} 2023-10-18T20:38:21.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000452a2fad9498de31-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.690 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:38:21.691 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:38:21.691 DEBUG:boto:Method: GET 2023-10-18T20:38:21.692 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:38:21.692 DEBUG:boto:Data: 2023-10-18T20:38:21.693 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.693 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.693 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.694 DEBUG:boto:Params: {} 2023-10-18T20:38:21.694 DEBUG:boto:Token: None 2023-10-18T20:38:21.695 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj85 2023-10-18T20:38:21.695 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YyZAem+Sl0i1JP8YEWvbfZnk2Hc= 2023-10-18T20:38:21.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YyZAem+Sl0i1JP8YEWvbfZnk2Hc=', 'Content-Length': '0'} 2023-10-18T20:38:21.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07d2e4f72e5883c-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.698 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.699 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.699 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.699 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.700 DEBUG:boto:Method: GET 2023-10-18T20:38:21.701 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.701 DEBUG:boto:Data: 2023-10-18T20:38:21.702 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.703 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.704 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.704 DEBUG:boto:Params: {} 2023-10-18T20:38:21.705 DEBUG:boto:Token: None 2023-10-18T20:38:21.706 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.707 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d749e77b0cd447df-006530423d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.711 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.711 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.712 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.712 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.712 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.713 DEBUG:boto:Method: GET 2023-10-18T20:38:21.713 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:38:21.714 DEBUG:boto:Data: 2023-10-18T20:38:21.714 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.715 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.715 DEBUG:boto:Params: {} 2023-10-18T20:38:21.716 DEBUG:boto:Token: None 2023-10-18T20:38:21.716 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a616df1bb6ca4800-006530423d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.720 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:38:21.721 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.722 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.722 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.722 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.723 DEBUG:boto:Method: GET 2023-10-18T20:38:21.723 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.724 DEBUG:boto:Data: 2023-10-18T20:38:21.724 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.724 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.725 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.725 DEBUG:boto:Params: {} 2023-10-18T20:38:21.726 DEBUG:boto:Token: None 2023-10-18T20:38:21.726 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.726 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+lZR9EeqEujRwL2hlGAkdmd/mV0= 2023-10-18T20:38:21.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+lZR9EeqEujRwL2hlGAkdmd/mV0=', 'Content-Length': '0'} 2023-10-18T20:38:21.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000553a2d0767dd38af-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.730 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.730 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.731 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.731 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.731 DEBUG:boto:Method: GET 2023-10-18T20:38:21.732 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.732 DEBUG:boto:Data: 2023-10-18T20:38:21.733 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.733 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.734 DEBUG:boto:Params: {} 2023-10-18T20:38:21.734 DEBUG:boto:Token: None 2023-10-18T20:38:21.735 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+lZR9EeqEujRwL2hlGAkdmd/mV0= 2023-10-18T20:38:21.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+lZR9EeqEujRwL2hlGAkdmd/mV0=', 'Content-Length': '0'} 2023-10-18T20:38:21.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c397c31d32c3d86-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.738 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.738 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.739 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.739 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.740 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.740 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.740 DEBUG:boto:Data: 2023-10-18T20:38:21.741 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.741 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.741 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.742 DEBUG:boto:Params: {} 2023-10-18T20:38:21.742 DEBUG:boto:Token: None 2023-10-18T20:38:21.743 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.743 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lU4xrxx4NCGHj0IYCTM7QvY83sc= 2023-10-18T20:38:21.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lU4xrxx4NCGHj0IYCTM7QvY83sc=', 'Content-Length': '0'} 2023-10-18T20:38:21.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040ab2281c42186dc-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.746 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.746 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.747 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.747 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.747 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.748 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.748 DEBUG:boto:Data: 2023-10-18T20:38:21.749 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.749 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.749 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.750 DEBUG:boto:Params: {} 2023-10-18T20:38:21.750 DEBUG:boto:Token: None 2023-10-18T20:38:21.750 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.751 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lU4xrxx4NCGHj0IYCTM7QvY83sc= 2023-10-18T20:38:21.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lU4xrxx4NCGHj0IYCTM7QvY83sc=', 'Content-Length': '0'} 2023-10-18T20:38:21.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca89ee2b20d8cab-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:38:21.755 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.755 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.755 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.756 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.756 DEBUG:boto:Method: GET 2023-10-18T20:38:21.757 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.757 DEBUG:boto:Data: 2023-10-18T20:38:21.757 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.758 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.758 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.759 DEBUG:boto:Params: {} 2023-10-18T20:38:21.759 DEBUG:boto:Token: None 2023-10-18T20:38:21.760 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.760 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+lZR9EeqEujRwL2hlGAkdmd/mV0= 2023-10-18T20:38:21.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+lZR9EeqEujRwL2hlGAkdmd/mV0=', 'Content-Length': '0'} 2023-10-18T20:38:21.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b57798d027c76373-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.763 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.763 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.764 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.764 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.765 DEBUG:boto:Method: GET 2023-10-18T20:38:21.765 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.766 DEBUG:boto:Data: 2023-10-18T20:38:21.766 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.767 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.767 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.768 DEBUG:boto:Params: {} 2023-10-18T20:38:21.768 DEBUG:boto:Token: None 2023-10-18T20:38:21.769 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:38:21.769 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+lZR9EeqEujRwL2hlGAkdmd/mV0= 2023-10-18T20:38:21.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+lZR9EeqEujRwL2hlGAkdmd/mV0=', 'Content-Length': '0'} 2023-10-18T20:38:21.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3b5c03953e97096-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.772 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.772 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.773 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.773 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.774 DEBUG:boto:Method: GET 2023-10-18T20:38:21.774 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.774 DEBUG:boto:Data: 2023-10-18T20:38:21.775 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.775 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.776 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.776 DEBUG:boto:Params: {} 2023-10-18T20:38:21.777 DEBUG:boto:Token: None 2023-10-18T20:38:21.778 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.779 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae0883fbaa0298e6-006530423d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.783 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.783 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.784 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.784 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.785 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.785 DEBUG:boto:Method: GET 2023-10-18T20:38:21.785 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.786 DEBUG:boto:Data: 2023-10-18T20:38:21.786 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.787 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.787 DEBUG:boto:Params: {} 2023-10-18T20:38:21.788 DEBUG:boto:Token: None 2023-10-18T20:38:21.788 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f30c46998d6b37-006530423d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.834 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.835 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.836 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.837 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.837 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:38:21.838 DEBUG:boto:Data: 2023-10-18T20:38:21.839 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.839 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.840 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.841 DEBUG:boto:Params: {} 2023-10-18T20:38:21.841 DEBUG:boto:Token: None 2023-10-18T20:38:21.842 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95 2023-10-18T20:38:21.843 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b3NEc9PJfNT/eaW1o15pSsZhKko= 2023-10-18T20:38:21.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b3NEc9PJfNT/eaW1o15pSsZhKko=', 'Content-Length': '0'} 2023-10-18T20:38:21.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b67c4d2bdb6e9b17-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.847 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.847 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.848 DEBUG:boto:Method: HEAD 2023-10-18T20:38:21.849 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:38:21.849 DEBUG:boto:Data: 2023-10-18T20:38:21.849 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.850 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.850 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.851 DEBUG:boto:Params: {} 2023-10-18T20:38:21.851 DEBUG:boto:Token: None 2023-10-18T20:38:21.852 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95 2023-10-18T20:38:21.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b3NEc9PJfNT/eaW1o15pSsZhKko= 2023-10-18T20:38:21.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b3NEc9PJfNT/eaW1o15pSsZhKko=', 'Content-Length': '0'} 2023-10-18T20:38:21.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d71766c72681d7d-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:38:21.856 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.856 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.857 DEBUG:boto:Method: GET 2023-10-18T20:38:21.857 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:38:21.858 DEBUG:boto:Data: 2023-10-18T20:38:21.858 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.859 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.860 DEBUG:boto:Params: {} 2023-10-18T20:38:21.860 DEBUG:boto:Token: None 2023-10-18T20:38:21.861 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95 2023-10-18T20:38:21.861 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CKOWWvQPt1E6wiXVyp4RWJu7msE= 2023-10-18T20:38:21.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CKOWWvQPt1E6wiXVyp4RWJu7msE=', 'Content-Length': '0'} 2023-10-18T20:38:21.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edc914db21cfde58-006530423d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.864 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:38:21.865 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:38:21.865 DEBUG:boto:Method: GET 2023-10-18T20:38:21.865 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:38:21.866 DEBUG:boto:Data: 2023-10-18T20:38:21.867 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.867 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.867 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.868 DEBUG:boto:Params: {} 2023-10-18T20:38:21.868 DEBUG:boto:Token: None 2023-10-18T20:38:21.869 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/obj95 2023-10-18T20:38:21.869 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CKOWWvQPt1E6wiXVyp4RWJu7msE= 2023-10-18T20:38:21.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CKOWWvQPt1E6wiXVyp4RWJu7msE=', 'Content-Length': '0'} 2023-10-18T20:38:21.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be5ad7ad82ad74a-006530423d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.872 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.873 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.873 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.874 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.874 DEBUG:boto:Method: GET 2023-10-18T20:38:21.875 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.875 DEBUG:boto:Data: 2023-10-18T20:38:21.875 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.876 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:38:21.876 DEBUG:boto:Port: 8001 2023-10-18T20:38:21.877 DEBUG:boto:Params: {} 2023-10-18T20:38:21.877 DEBUG:boto:Token: None 2023-10-18T20:38:21.877 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.878 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c95235cb4353f43-006530423d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.882 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.882 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:21.883 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:21.883 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.884 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.884 DEBUG:boto:Method: GET 2023-10-18T20:38:21.884 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:38:21.885 DEBUG:boto:Data: 2023-10-18T20:38:21.885 DEBUG:boto:Headers: {} 2023-10-18T20:38:21.886 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:21.886 DEBUG:boto:Port: 8000 2023-10-18T20:38:21.887 DEBUG:boto:Params: {} 2023-10-18T20:38:21.887 DEBUG:boto:Token: None 2023-10-18T20:38:21.888 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:38:21 GMT /bofrnf-59/ 2023-10-18T20:38:21.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0= 2023-10-18T20:38:21.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CH6GMDBVh7EmvDsAIMOBm2YDcJ0=', 'Content-Length': '0'} 2023-10-18T20:38:21.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000454b093fed6dba2c-006530423d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:21.892 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:38:21.893 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:38:21.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:38:21.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:38:21.894 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:38:21.953 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:38:21.953 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:38:22.000 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:38:22.000 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-59 2023-10-18T20:38:22.007 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 15 2023-10-18T20:38:22.053 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:22.054 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:22.055 DEBUG:boto:Method: HEAD 2023-10-18T20:38:22.055 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:38:22.056 DEBUG:boto:Data: 2023-10-18T20:38:22.056 DEBUG:boto:Headers: {} 2023-10-18T20:38:22.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.057 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.057 DEBUG:boto:Params: {} 2023-10-18T20:38:22.058 DEBUG:boto:Token: None 2023-10-18T20:38:22.058 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/ 2023-10-18T20:38:22.059 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k= 2023-10-18T20:38:22.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k=', 'Content-Length': '0'} 2023-10-18T20:38:22.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f44e5f9ab9b851e-006530423e-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.063 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:38:22.064 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:38:22.064 DEBUG:boto:Method: PUT 2023-10-18T20:38:22.065 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:38:22.065 DEBUG:boto:Data: 2023-10-18T20:38:22.065 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:38:22.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.066 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.067 DEBUG:boto:Params: {} 2023-10-18T20:38:22.067 DEBUG:boto:Token: None 2023-10-18T20:38:22.068 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/obj23 2023-10-18T20:38:22.069 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GVeakpQ3zc1olAXPMoLrZNTlwjA= 2023-10-18T20:38:22.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GVeakpQ3zc1olAXPMoLrZNTlwjA='} 2023-10-18T20:38:22.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007bc426eda6a0d1fe-006530423e-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.115 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:22.116 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:22.116 DEBUG:boto:Method: HEAD 2023-10-18T20:38:22.117 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:38:22.117 DEBUG:boto:Data: 2023-10-18T20:38:22.117 DEBUG:boto:Headers: {} 2023-10-18T20:38:22.118 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.118 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.119 DEBUG:boto:Params: {} 2023-10-18T20:38:22.119 DEBUG:boto:Token: None 2023-10-18T20:38:22.119 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/ 2023-10-18T20:38:22.120 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k= 2023-10-18T20:38:22.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k=', 'Content-Length': '0'} 2023-10-18T20:38:22.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000753948fc32ea0ef9-006530423e-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.124 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:38:22.125 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:38:22.125 DEBUG:boto:Method: PUT 2023-10-18T20:38:22.126 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:38:22.127 DEBUG:boto:Data: 2023-10-18T20:38:22.127 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:38:22.128 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.128 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.129 DEBUG:boto:Params: {} 2023-10-18T20:38:22.129 DEBUG:boto:Token: None 2023-10-18T20:38:22.130 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/obj46 2023-10-18T20:38:22.130 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t7qRXVEE4CBWzvnmHqgwJNYUDTs= 2023-10-18T20:38:22.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t7qRXVEE4CBWzvnmHqgwJNYUDTs='} 2023-10-18T20:38:22.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000998ad605eda958c5-006530423e-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.136 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:22.136 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:22.137 DEBUG:boto:Method: HEAD 2023-10-18T20:38:22.137 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:38:22.138 DEBUG:boto:Data: 2023-10-18T20:38:22.138 DEBUG:boto:Headers: {} 2023-10-18T20:38:22.139 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.139 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.139 DEBUG:boto:Params: {} 2023-10-18T20:38:22.140 DEBUG:boto:Token: None 2023-10-18T20:38:22.140 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/ 2023-10-18T20:38:22.140 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k= 2023-10-18T20:38:22.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k=', 'Content-Length': '0'} 2023-10-18T20:38:22.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b4ecd3ab7ceeee7-006530423e-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.144 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:38:22.144 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:38:22.145 DEBUG:boto:Method: PUT 2023-10-18T20:38:22.145 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:38:22.146 DEBUG:boto:Data: 2023-10-18T20:38:22.146 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:38:22.146 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.147 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.147 DEBUG:boto:Params: {} 2023-10-18T20:38:22.148 DEBUG:boto:Token: None 2023-10-18T20:38:22.148 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/obj69 2023-10-18T20:38:22.149 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:grvZd+o52mR1xBtAP4JBFE7m/XY= 2023-10-18T20:38:22.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:grvZd+o52mR1xBtAP4JBFE7m/XY='} 2023-10-18T20:38:22.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbb6468c28cd69e9-006530423e-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.199 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:22.199 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:22.200 DEBUG:boto:Method: HEAD 2023-10-18T20:38:22.200 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:38:22.201 DEBUG:boto:Data: 2023-10-18T20:38:22.201 DEBUG:boto:Headers: {} 2023-10-18T20:38:22.201 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.202 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.202 DEBUG:boto:Params: {} 2023-10-18T20:38:22.203 DEBUG:boto:Token: None 2023-10-18T20:38:22.203 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/ 2023-10-18T20:38:22.203 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k= 2023-10-18T20:38:22.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k=', 'Content-Length': '0'} 2023-10-18T20:38:22.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e9cc1e7108aeb10-006530423e-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.207 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:38:22.208 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:38:22.208 DEBUG:boto:Method: PUT 2023-10-18T20:38:22.209 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:38:22.209 DEBUG:boto:Data: 2023-10-18T20:38:22.209 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:38:22.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.210 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.211 DEBUG:boto:Params: {} 2023-10-18T20:38:22.211 DEBUG:boto:Token: None 2023-10-18T20:38:22.212 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/obj92 2023-10-18T20:38:22.212 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SNW7/Fsf5bAFcO2A/fRXcC1/p5I= 2023-10-18T20:38:22.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SNW7/Fsf5bAFcO2A/fRXcC1/p5I='} 2023-10-18T20:38:22.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000972bfe13b5be1463-006530423e-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.218 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:38:22.219 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:38:22.219 DEBUG:boto:Method: HEAD 2023-10-18T20:38:22.220 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:38:22.220 DEBUG:boto:Data: 2023-10-18T20:38:22.220 DEBUG:boto:Headers: {} 2023-10-18T20:38:22.221 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.221 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.222 DEBUG:boto:Params: {} 2023-10-18T20:38:22.222 DEBUG:boto:Token: None 2023-10-18T20:38:22.223 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/ 2023-10-18T20:38:22.223 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k= 2023-10-18T20:38:22.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BS32apwVwiNZ9fJu4iUKFAQaL9k=', 'Content-Length': '0'} 2023-10-18T20:38:22.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b15c891d9eec7e24-006530423e-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.227 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:38:22.227 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:38:22.228 DEBUG:boto:Method: PUT 2023-10-18T20:38:22.228 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:38:22.228 DEBUG:boto:Data: 2023-10-18T20:38:22.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:38:22.229 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:38:22.230 DEBUG:boto:Port: 8000 2023-10-18T20:38:22.230 DEBUG:boto:Params: {} 2023-10-18T20:38:22.231 DEBUG:boto:Token: None 2023-10-18T20:38:22.231 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:38:22 GMT /bofrnf-59/obj115 2023-10-18T20:38:22.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GAHpvZxdCf3qNWL18ksTvIx1Qaw= 2023-10-18T20:38:22.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:38:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GAHpvZxdCf3qNWL18ksTvIx1Qaw='} 2023-10-18T20:38:22.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d92a9ff79653f1d-006530423e-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:38:22.275 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-59 2023-10-18T20:38:22.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:38:22.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:38:22.276 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:38:22.333 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:38:22.334 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:38:22.368 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:22.366+0000 7fd8857b3780 1 waiting to reach latest gen 3, on 2.. 2023-10-18T20:38:27.369 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:27.368+0000 7fd8857b3780 1 waiting to reach latest gen 3, on 2.. 2023-10-18T20:38:32.370 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:32.369+0000 7fd8857b3780 1 waiting to reach latest gen 3, on 2.. 2023-10-18T20:38:37.371 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:37.369+0000 7fd8857b3780 1 waiting to reach latest gen 3, on 2.. 2023-10-18T20:38:42.373 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:42.372+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:38:42.374 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:38:42.374 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:38:47.375 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:47.373+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:38:47.375 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:38:47.376 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:38:52.377 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:52.375+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:38:52.377 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:38:52.377 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:38:57.378 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:38:57.377+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:38:57.379 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:38:57.379 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:02.380 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:02.377+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:39:02.380 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:39:02.380 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:07.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:07.379+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:39:07.382 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:39:07.382 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:12.383 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:12.381+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:39:12.384 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:39:12.384 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:17.385 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:17.384+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:39:17.386 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:39:17.386 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:22.387 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:22.386+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:39:22.388 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:39:22.388 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:27.389 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:27.387+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:39:27.389 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:39:27.389 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:32.390 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:32.389+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:39:32.391 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:32.391 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:37.392 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:37.391+0000 7fd8857b3780 1 waiting for incremental sync to catch up: 2023-10-18T20:39:37.393 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:37.393 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:42.394 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:42.393+0000 7fd8857b3780 1 bucket sync caught up with source: 2023-10-18T20:39:42.395 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:42.395 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.86.6, , , 00000000002.93.6, , 00000000002.85.6, , , , , , 00000000001.73.6, , 00000000001.54.6, , , , ] 2023-10-18T20:39:42.395 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:42.393+0000 7fd8857b3780 0 bucket checkpoint complete 2023-10-18T20:39:42.411 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-59 2023-10-18T20:39:42.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:39:42.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:39:42.413 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:39:42.475 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:39:42.475 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:39:42.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:42.511+0000 7f87f7e27780 1 bucket sync caught up with source: 2023-10-18T20:39:42.514 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.757.6, 00000000002.1177.6, 00000000001.1248.6, 00000000002.701.6, 00000000002.773.6, , , 00000000002.232.6, 00000000005.457.6, 00000000004.248.6] 2023-10-18T20:39:42.514 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.757.6, 00000000002.1177.6, 00000000001.1248.6, 00000000002.701.6, 00000000002.773.6, , , 00000000002.232.6, 00000000005.457.6, 00000000004.248.6] 2023-10-18T20:39:42.514 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:42.511+0000 7f87f7e27780 0 bucket checkpoint complete 2023-10-18T20:39:42.527 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:39:42.528 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.528 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.529 DEBUG:boto:Method: HEAD 2023-10-18T20:39:42.529 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:39:42.529 DEBUG:boto:Data: 2023-10-18T20:39:42.530 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.531 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.531 DEBUG:boto:Params: {} 2023-10-18T20:39:42.532 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:39:42.533 DEBUG:boto:Token: None 2023-10-18T20:39:42.534 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/ 2023-10-18T20:39:42.534 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNehs0PnG1M42+RnSIwvkDu84Rw= 2023-10-18T20:39:42.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNehs0PnG1M42+RnSIwvkDu84Rw=', 'Content-Length': '0'} 2023-10-18T20:39:42.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bce5b6de26fd86c4-006530428e-1af4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.540 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.540 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.541 DEBUG:boto:Method: HEAD 2023-10-18T20:39:42.541 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:39:42.542 DEBUG:boto:Data: 2023-10-18T20:39:42.542 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.543 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.543 DEBUG:boto:Params: {} 2023-10-18T20:39:42.544 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:39:42.544 DEBUG:boto:Token: None 2023-10-18T20:39:42.545 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/ 2023-10-18T20:39:42.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNehs0PnG1M42+RnSIwvkDu84Rw= 2023-10-18T20:39:42.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNehs0PnG1M42+RnSIwvkDu84Rw=', 'Content-Length': '0'} 2023-10-18T20:39:42.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c224ebdd03619c83-006530428e-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.550 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:39:42.551 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.551 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.552 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.552 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.553 DEBUG:boto:Method: GET 2023-10-18T20:39:42.553 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.553 DEBUG:boto:Data: 2023-10-18T20:39:42.554 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.554 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.555 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.555 DEBUG:boto:Params: {} 2023-10-18T20:39:42.556 DEBUG:boto:Token: None 2023-10-18T20:39:42.556 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/?versions 2023-10-18T20:39:42.557 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mPK81aQs/ckN1D1nIbCAlYoG12A= 2023-10-18T20:39:42.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mPK81aQs/ckN1D1nIbCAlYoG12A=', 'Content-Length': '0'} 2023-10-18T20:39:42.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000449f0859373f41b5-006530428e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.560 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:39:42.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:39:42.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:39:42.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:39:42.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:39:42.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:39:42.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:39:42.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:39:42.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:39:42.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:39:42.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:39:42.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:39:42.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:39:42.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:39:42.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:39:42.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:39:42.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:39:42.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:39:42.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:39:42.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:39:42.569 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:39:42.570 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.570 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.571 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.571 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.572 DEBUG:boto:Method: GET 2023-10-18T20:39:42.572 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.572 DEBUG:boto:Data: 2023-10-18T20:39:42.573 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.573 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.574 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.574 DEBUG:boto:Params: {} 2023-10-18T20:39:42.574 DEBUG:boto:Token: None 2023-10-18T20:39:42.575 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/?versions 2023-10-18T20:39:42.575 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mPK81aQs/ckN1D1nIbCAlYoG12A= 2023-10-18T20:39:42.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mPK81aQs/ckN1D1nIbCAlYoG12A=', 'Content-Length': '0'} 2023-10-18T20:39:42.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d97c78338fa63b4-006530428e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.579 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:39:42.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:39:42.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:39:42.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:39:42.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:39:42.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:39:42.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:39:42.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:39:42.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:39:42.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:39:42.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:39:42.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:39:42.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:39:42.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:39:42.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:39:42.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:39:42.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:39:42.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:39:42.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:39:42.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:39:42.588 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.589 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.589 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.589 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.590 DEBUG:boto:Method: GET 2023-10-18T20:39:42.590 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.591 DEBUG:boto:Data: 2023-10-18T20:39:42.591 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.592 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.592 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.592 DEBUG:boto:Params: {} 2023-10-18T20:39:42.593 DEBUG:boto:Token: None 2023-10-18T20:39:42.593 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/?versions 2023-10-18T20:39:42.594 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mPK81aQs/ckN1D1nIbCAlYoG12A= 2023-10-18T20:39:42.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mPK81aQs/ckN1D1nIbCAlYoG12A=', 'Content-Length': '0'} 2023-10-18T20:39:42.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a20863912162ae4e-006530428e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.597 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.599 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.599 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.600 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.600 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.600 DEBUG:boto:Method: GET 2023-10-18T20:39:42.601 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:39:42.601 DEBUG:boto:Data: 2023-10-18T20:39:42.602 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.603 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.603 DEBUG:boto:Params: {} 2023-10-18T20:39:42.604 DEBUG:boto:Token: None 2023-10-18T20:39:42.604 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/?versions 2023-10-18T20:39:42.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mPK81aQs/ckN1D1nIbCAlYoG12A= 2023-10-18T20:39:42.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mPK81aQs/ckN1D1nIbCAlYoG12A=', 'Content-Length': '0'} 2023-10-18T20:39:42.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000482c0574e385c372-006530428e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.608 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:39:42.610 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.611 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.611 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.612 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.612 DEBUG:boto:Method: GET 2023-10-18T20:39:42.612 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.613 DEBUG:boto:Data: 2023-10-18T20:39:42.613 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.614 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.614 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.615 DEBUG:boto:Params: {} 2023-10-18T20:39:42.615 DEBUG:boto:Token: None 2023-10-18T20:39:42.616 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.616 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FwezgWuB6EVxMrXbFrGYFCbeeJk= 2023-10-18T20:39:42.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FwezgWuB6EVxMrXbFrGYFCbeeJk=', 'Content-Length': '0'} 2023-10-18T20:39:42.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaa530455b7640d1-006530428e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.619 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.619 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.620 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.620 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.621 DEBUG:boto:Method: GET 2023-10-18T20:39:42.621 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.621 DEBUG:boto:Data: 2023-10-18T20:39:42.622 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.622 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.623 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.623 DEBUG:boto:Params: {} 2023-10-18T20:39:42.624 DEBUG:boto:Token: None 2023-10-18T20:39:42.624 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.625 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FwezgWuB6EVxMrXbFrGYFCbeeJk= 2023-10-18T20:39:42.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FwezgWuB6EVxMrXbFrGYFCbeeJk=', 'Content-Length': '0'} 2023-10-18T20:39:42.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095bf6a64bcc8d557-006530428e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.628 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.628 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.629 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.629 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.629 DEBUG:boto:Method: HEAD 2023-10-18T20:39:42.630 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.630 DEBUG:boto:Data: 2023-10-18T20:39:42.631 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.631 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.631 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.632 DEBUG:boto:Params: {} 2023-10-18T20:39:42.632 DEBUG:boto:Token: None 2023-10-18T20:39:42.633 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.633 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GXYkWLZvqkyzKSYHiDsPO+ukJfY= 2023-10-18T20:39:42.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GXYkWLZvqkyzKSYHiDsPO+ukJfY=', 'Content-Length': '0'} 2023-10-18T20:39:42.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092bfd875973b1488-006530428e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.636 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.637 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.637 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.638 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.638 DEBUG:boto:Method: HEAD 2023-10-18T20:39:42.638 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.639 DEBUG:boto:Data: 2023-10-18T20:39:42.639 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.640 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.640 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.640 DEBUG:boto:Params: {} 2023-10-18T20:39:42.641 DEBUG:boto:Token: None 2023-10-18T20:39:42.641 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.642 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GXYkWLZvqkyzKSYHiDsPO+ukJfY= 2023-10-18T20:39:42.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GXYkWLZvqkyzKSYHiDsPO+ukJfY=', 'Content-Length': '0'} 2023-10-18T20:39:42.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6bbd38dcf904e35-006530428e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:39:42.646 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.646 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.646 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.647 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.647 DEBUG:boto:Method: GET 2023-10-18T20:39:42.648 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.648 DEBUG:boto:Data: 2023-10-18T20:39:42.649 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.649 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.650 DEBUG:boto:Params: {} 2023-10-18T20:39:42.650 DEBUG:boto:Token: None 2023-10-18T20:39:42.651 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.651 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FwezgWuB6EVxMrXbFrGYFCbeeJk= 2023-10-18T20:39:42.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FwezgWuB6EVxMrXbFrGYFCbeeJk=', 'Content-Length': '0'} 2023-10-18T20:39:42.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011171f79c5cada72-006530428e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.694 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.695 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.696 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.696 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.696 DEBUG:boto:Method: GET 2023-10-18T20:39:42.697 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.697 DEBUG:boto:Data: 2023-10-18T20:39:42.698 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.699 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.699 DEBUG:boto:Params: {} 2023-10-18T20:39:42.700 DEBUG:boto:Token: None 2023-10-18T20:39:42.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:39:42.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FwezgWuB6EVxMrXbFrGYFCbeeJk= 2023-10-18T20:39:42.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FwezgWuB6EVxMrXbFrGYFCbeeJk=', 'Content-Length': '0'} 2023-10-18T20:39:42.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000701635c543f156ce-006530428e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.703 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.704 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.704 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.705 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.705 DEBUG:boto:Method: GET 2023-10-18T20:39:42.706 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.706 DEBUG:boto:Data: 2023-10-18T20:39:42.707 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.708 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.708 DEBUG:boto:Params: {} 2023-10-18T20:39:42.708 DEBUG:boto:Token: None 2023-10-18T20:39:42.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/ 2023-10-18T20:39:42.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M= 2023-10-18T20:39:42.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M=', 'Content-Length': '0'} 2023-10-18T20:39:42.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8dd8b12ce3e622-006530428e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.713 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.714 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.714 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.715 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.715 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.716 DEBUG:boto:Method: GET 2023-10-18T20:39:42.716 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.717 DEBUG:boto:Data: 2023-10-18T20:39:42.717 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.718 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.718 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.721 DEBUG:boto:Params: {} 2023-10-18T20:39:42.722 DEBUG:boto:Token: None 2023-10-18T20:39:42.722 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/ 2023-10-18T20:39:42.723 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M= 2023-10-18T20:39:42.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M=', 'Content-Length': '0'} 2023-10-18T20:39:42.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088c1d7bdb970ad18-006530428e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.727 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.728 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.728 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.729 DEBUG:boto:Method: HEAD 2023-10-18T20:39:42.729 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:39:42.730 DEBUG:boto:Data: 2023-10-18T20:39:42.730 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.730 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.731 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.731 DEBUG:boto:Params: {} 2023-10-18T20:39:42.732 DEBUG:boto:Token: None 2023-10-18T20:39:42.732 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11 2023-10-18T20:39:42.733 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z0ObP3MWk9Mi1wSsblk9QtPb/tI= 2023-10-18T20:39:42.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z0ObP3MWk9Mi1wSsblk9QtPb/tI=', 'Content-Length': '0'} 2023-10-18T20:39:42.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d43f4b69bdf761a-006530428e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.736 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.736 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.737 DEBUG:boto:Method: HEAD 2023-10-18T20:39:42.737 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:39:42.737 DEBUG:boto:Data: 2023-10-18T20:39:42.738 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.739 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.739 DEBUG:boto:Params: {} 2023-10-18T20:39:42.740 DEBUG:boto:Token: None 2023-10-18T20:39:42.740 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11 2023-10-18T20:39:42.741 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z0ObP3MWk9Mi1wSsblk9QtPb/tI= 2023-10-18T20:39:42.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z0ObP3MWk9Mi1wSsblk9QtPb/tI=', 'Content-Length': '0'} 2023-10-18T20:39:42.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066e435808559ea6d-006530428e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:39:42.744 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.744 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.745 DEBUG:boto:Method: GET 2023-10-18T20:39:42.745 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:39:42.746 DEBUG:boto:Data: 2023-10-18T20:39:42.746 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.747 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.748 DEBUG:boto:Params: {} 2023-10-18T20:39:42.748 DEBUG:boto:Token: None 2023-10-18T20:39:42.749 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11 2023-10-18T20:39:42.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DRZDzpcFfnOMuIQ7TyxUjGAzD5E= 2023-10-18T20:39:42.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DRZDzpcFfnOMuIQ7TyxUjGAzD5E=', 'Content-Length': '0'} 2023-10-18T20:39:42.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b18b6415a688077-006530428e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.752 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:39:42.753 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:39:42.753 DEBUG:boto:Method: GET 2023-10-18T20:39:42.753 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:39:42.754 DEBUG:boto:Data: 2023-10-18T20:39:42.754 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.755 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.755 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.756 DEBUG:boto:Params: {} 2023-10-18T20:39:42.756 DEBUG:boto:Token: None 2023-10-18T20:39:42.757 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj11 2023-10-18T20:39:42.757 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DRZDzpcFfnOMuIQ7TyxUjGAzD5E= 2023-10-18T20:39:42.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DRZDzpcFfnOMuIQ7TyxUjGAzD5E=', 'Content-Length': '0'} 2023-10-18T20:39:42.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b24e8d81bd32a1c-006530428e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.806 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.807 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.808 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.808 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.809 DEBUG:boto:Method: GET 2023-10-18T20:39:42.809 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.810 DEBUG:boto:Data: 2023-10-18T20:39:42.810 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.811 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.811 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.812 DEBUG:boto:Params: {} 2023-10-18T20:39:42.813 DEBUG:boto:Token: None 2023-10-18T20:39:42.813 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/ 2023-10-18T20:39:42.814 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M= 2023-10-18T20:39:42.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M=', 'Content-Length': '0'} 2023-10-18T20:39:42.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbe7b0986059e383-006530428e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.817 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.818 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.819 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.819 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.820 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.820 DEBUG:boto:Method: GET 2023-10-18T20:39:42.820 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:39:42.821 DEBUG:boto:Data: 2023-10-18T20:39:42.821 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.822 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.822 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.823 DEBUG:boto:Params: {} 2023-10-18T20:39:42.823 DEBUG:boto:Token: None 2023-10-18T20:39:42.824 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/ 2023-10-18T20:39:42.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M= 2023-10-18T20:39:42.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M=', 'Content-Length': '0'} 2023-10-18T20:39:42.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ae4fae8d295cd87-006530428e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.829 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:39:42.830 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:42.830 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:42.831 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.831 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.832 DEBUG:boto:Method: GET 2023-10-18T20:39:42.832 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.832 DEBUG:boto:Data: 2023-10-18T20:39:42.833 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.833 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.834 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.834 DEBUG:boto:Params: {} 2023-10-18T20:39:42.835 DEBUG:boto:Token: None 2023-10-18T20:39:42.835 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.836 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ON1+698dUk7r9yALia68Q5n90Do= 2023-10-18T20:39:42.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ON1+698dUk7r9yALia68Q5n90Do=', 'Content-Length': '0'} 2023-10-18T20:39:42.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0d429650e865ef2-006530428e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.839 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:42.839 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:42.840 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.840 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.841 DEBUG:boto:Method: GET 2023-10-18T20:39:42.841 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.842 DEBUG:boto:Data: 2023-10-18T20:39:42.842 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.843 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.844 DEBUG:boto:Params: {} 2023-10-18T20:39:42.844 DEBUG:boto:Token: None 2023-10-18T20:39:42.845 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.845 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ON1+698dUk7r9yALia68Q5n90Do= 2023-10-18T20:39:42.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ON1+698dUk7r9yALia68Q5n90Do=', 'Content-Length': '0'} 2023-10-18T20:39:42.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b43cec4d4695408-006530428e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.848 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:42.849 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:42.849 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.850 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.850 DEBUG:boto:Method: HEAD 2023-10-18T20:39:42.851 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.851 DEBUG:boto:Data: 2023-10-18T20:39:42.851 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.852 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.852 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.853 DEBUG:boto:Params: {} 2023-10-18T20:39:42.853 DEBUG:boto:Token: None 2023-10-18T20:39:42.854 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.854 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F+rfjF0+uIueZZ8SprUZQ4ojgQg= 2023-10-18T20:39:42.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F+rfjF0+uIueZZ8SprUZQ4ojgQg=', 'Content-Length': '0'} 2023-10-18T20:39:42.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5403c0e43509564-006530428e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.857 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:42.857 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:42.858 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.858 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.859 DEBUG:boto:Method: HEAD 2023-10-18T20:39:42.859 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.860 DEBUG:boto:Data: 2023-10-18T20:39:42.860 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.861 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.861 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.862 DEBUG:boto:Params: {} 2023-10-18T20:39:42.862 DEBUG:boto:Token: None 2023-10-18T20:39:42.863 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.863 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F+rfjF0+uIueZZ8SprUZQ4ojgQg= 2023-10-18T20:39:42.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F+rfjF0+uIueZZ8SprUZQ4ojgQg=', 'Content-Length': '0'} 2023-10-18T20:39:42.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ad4a4c891cb9f38-006530428e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:39:42.866 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:42.867 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:42.867 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.868 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.868 DEBUG:boto:Method: GET 2023-10-18T20:39:42.869 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.869 DEBUG:boto:Data: 2023-10-18T20:39:42.870 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.870 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.871 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.871 DEBUG:boto:Params: {} 2023-10-18T20:39:42.871 DEBUG:boto:Token: None 2023-10-18T20:39:42.872 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ON1+698dUk7r9yALia68Q5n90Do= 2023-10-18T20:39:42.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ON1+698dUk7r9yALia68Q5n90Do=', 'Content-Length': '0'} 2023-10-18T20:39:42.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000641dae6520d3c721-006530428e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.918 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:42.919 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:42.919 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.920 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.920 DEBUG:boto:Method: GET 2023-10-18T20:39:42.921 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.921 DEBUG:boto:Data: 2023-10-18T20:39:42.921 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.922 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.922 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.923 DEBUG:boto:Params: {} 2023-10-18T20:39:42.923 DEBUG:boto:Token: None 2023-10-18T20:39:42.924 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:39:42.924 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ON1+698dUk7r9yALia68Q5n90Do= 2023-10-18T20:39:42.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ON1+698dUk7r9yALia68Q5n90Do=', 'Content-Length': '0'} 2023-10-18T20:39:42.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa40c4dffb7b5b94-006530428e-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.930 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.930 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.931 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:39:42.931 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:39:42.932 DEBUG:boto:Method: GET 2023-10-18T20:39:42.932 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:39:42.932 DEBUG:boto:Data: 2023-10-18T20:39:42.933 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.933 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.934 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.934 DEBUG:boto:Params: {} 2023-10-18T20:39:42.935 DEBUG:boto:Token: None 2023-10-18T20:39:42.935 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/ 2023-10-18T20:39:42.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M= 2023-10-18T20:39:42.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M=', 'Content-Length': '0'} 2023-10-18T20:39:42.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e728b80b27f6936c-006530428e-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.939 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.940 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:42.940 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:42.941 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:39:42.941 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:39:42.942 DEBUG:boto:Method: GET 2023-10-18T20:39:42.942 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:39:42.943 DEBUG:boto:Data: 2023-10-18T20:39:42.943 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.943 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:42.944 DEBUG:boto:Port: 8000 2023-10-18T20:39:42.944 DEBUG:boto:Params: {} 2023-10-18T20:39:42.945 DEBUG:boto:Token: None 2023-10-18T20:39:42.945 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/ 2023-10-18T20:39:42.946 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M= 2023-10-18T20:39:42.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nSfKjVd0Rg1Gvnura3C94j7k30M=', 'Content-Length': '0'} 2023-10-18T20:39:42.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026bfafa34a49d928-006530428e-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.990 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:42.991 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:42.992 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:42.992 DEBUG:boto:Method: HEAD 2023-10-18T20:39:42.993 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:39:42.993 DEBUG:boto:Data: 2023-10-18T20:39:42.993 DEBUG:boto:Headers: {} 2023-10-18T20:39:42.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:42.994 DEBUG:boto:Port: 8001 2023-10-18T20:39:42.995 DEBUG:boto:Params: {} 2023-10-18T20:39:42.995 DEBUG:boto:Token: None 2023-10-18T20:39:42.995 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:42 GMT /bofrnf-59/obj115 2023-10-18T20:39:42.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FobHH8NuhE7KPM8KxqLmfj/oUDY= 2023-10-18T20:39:42.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FobHH8NuhE7KPM8KxqLmfj/oUDY=', 'Content-Length': '0'} 2023-10-18T20:39:42.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bb570b0bb1e38c5-006530428e-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:42.999 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:42.999 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:43.000 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.000 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:39:43.001 DEBUG:boto:Data: 2023-10-18T20:39:43.001 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.002 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.002 DEBUG:boto:Params: {} 2023-10-18T20:39:43.003 DEBUG:boto:Token: None 2023-10-18T20:39:43.003 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj115 2023-10-18T20:39:43.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+5n5ZnYRFZKHvQ3MFerXyL3jRZA= 2023-10-18T20:39:43.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+5n5ZnYRFZKHvQ3MFerXyL3jRZA=', 'Content-Length': '0'} 2023-10-18T20:39:43.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007651ab827e611cd4-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:39:43.007 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:43.007 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:43.008 DEBUG:boto:Method: GET 2023-10-18T20:39:43.008 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:39:43.009 DEBUG:boto:Data: 2023-10-18T20:39:43.009 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.009 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.010 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.010 DEBUG:boto:Params: {} 2023-10-18T20:39:43.011 DEBUG:boto:Token: None 2023-10-18T20:39:43.011 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj115 2023-10-18T20:39:43.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pF2Ow8R1AAjv/yE+dkI33noMD0A= 2023-10-18T20:39:43.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pF2Ow8R1AAjv/yE+dkI33noMD0A=', 'Content-Length': '0'} 2023-10-18T20:39:43.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51a10282fd132c1-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.015 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:39:43.015 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:39:43.015 DEBUG:boto:Method: GET 2023-10-18T20:39:43.016 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:39:43.016 DEBUG:boto:Data: 2023-10-18T20:39:43.017 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.017 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.017 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.018 DEBUG:boto:Params: {} 2023-10-18T20:39:43.018 DEBUG:boto:Token: None 2023-10-18T20:39:43.019 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj115 2023-10-18T20:39:43.019 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pF2Ow8R1AAjv/yE+dkI33noMD0A= 2023-10-18T20:39:43.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pF2Ow8R1AAjv/yE+dkI33noMD0A=', 'Content-Length': '0'} 2023-10-18T20:39:43.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4a9ec1c2cf35a98-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.023 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.023 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.023 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:39:43.024 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:39:43.024 DEBUG:boto:Method: GET 2023-10-18T20:39:43.025 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:39:43.025 DEBUG:boto:Data: 2023-10-18T20:39:43.025 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.026 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.026 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.027 DEBUG:boto:Params: {} 2023-10-18T20:39:43.027 DEBUG:boto:Token: None 2023-10-18T20:39:43.028 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.028 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6c2fd115be08856-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.032 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.032 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.033 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.033 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:39:43.034 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:39:43.034 DEBUG:boto:Method: GET 2023-10-18T20:39:43.035 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:39:43.035 DEBUG:boto:Data: 2023-10-18T20:39:43.036 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.036 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.036 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.037 DEBUG:boto:Params: {} 2023-10-18T20:39:43.037 DEBUG:boto:Token: None 2023-10-18T20:39:43.038 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.038 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f43074e661d78abd-006530428f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.042 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:39:43.044 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.044 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.044 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.045 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.045 DEBUG:boto:Method: GET 2023-10-18T20:39:43.046 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.046 DEBUG:boto:Data: 2023-10-18T20:39:43.046 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.047 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.047 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.048 DEBUG:boto:Params: {} 2023-10-18T20:39:43.048 DEBUG:boto:Token: None 2023-10-18T20:39:43.049 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nr9ei6JAmB1nfix0YXxHzNVmqnY= 2023-10-18T20:39:43.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nr9ei6JAmB1nfix0YXxHzNVmqnY=', 'Content-Length': '0'} 2023-10-18T20:39:43.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb9e095c9a7d201d-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.098 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.099 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.099 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.100 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.100 DEBUG:boto:Method: GET 2023-10-18T20:39:43.101 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.101 DEBUG:boto:Data: 2023-10-18T20:39:43.101 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.102 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.102 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.103 DEBUG:boto:Params: {} 2023-10-18T20:39:43.103 DEBUG:boto:Token: None 2023-10-18T20:39:43.104 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.104 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nr9ei6JAmB1nfix0YXxHzNVmqnY= 2023-10-18T20:39:43.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nr9ei6JAmB1nfix0YXxHzNVmqnY=', 'Content-Length': '0'} 2023-10-18T20:39:43.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1eb27bba35121f9-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.107 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.108 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.108 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.109 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.109 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.110 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.110 DEBUG:boto:Data: 2023-10-18T20:39:43.110 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.111 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.111 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.112 DEBUG:boto:Params: {} 2023-10-18T20:39:43.112 DEBUG:boto:Token: None 2023-10-18T20:39:43.113 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.113 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0k8A4Pf8Old6zZ0iDwUR13zILrQ= 2023-10-18T20:39:43.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0k8A4Pf8Old6zZ0iDwUR13zILrQ=', 'Content-Length': '0'} 2023-10-18T20:39:43.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d1cc60b23cce4d-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.116 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.117 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.117 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.118 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.118 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.119 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.119 DEBUG:boto:Data: 2023-10-18T20:39:43.120 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.120 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.120 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.121 DEBUG:boto:Params: {} 2023-10-18T20:39:43.121 DEBUG:boto:Token: None 2023-10-18T20:39:43.122 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.122 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0k8A4Pf8Old6zZ0iDwUR13zILrQ= 2023-10-18T20:39:43.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0k8A4Pf8Old6zZ0iDwUR13zILrQ=', 'Content-Length': '0'} 2023-10-18T20:39:43.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086e159374dcd44ca-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:39:43.125 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.126 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.126 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.127 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.127 DEBUG:boto:Method: GET 2023-10-18T20:39:43.128 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.128 DEBUG:boto:Data: 2023-10-18T20:39:43.129 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.129 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.129 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.130 DEBUG:boto:Params: {} 2023-10-18T20:39:43.133 DEBUG:boto:Token: None 2023-10-18T20:39:43.134 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.134 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nr9ei6JAmB1nfix0YXxHzNVmqnY= 2023-10-18T20:39:43.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nr9ei6JAmB1nfix0YXxHzNVmqnY=', 'Content-Length': '0'} 2023-10-18T20:39:43.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000449ab56e7f914a68-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.138 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.138 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.139 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.139 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.139 DEBUG:boto:Method: GET 2023-10-18T20:39:43.140 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.140 DEBUG:boto:Data: 2023-10-18T20:39:43.141 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.142 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.142 DEBUG:boto:Params: {} 2023-10-18T20:39:43.143 DEBUG:boto:Token: None 2023-10-18T20:39:43.143 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:39:43.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Nr9ei6JAmB1nfix0YXxHzNVmqnY= 2023-10-18T20:39:43.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Nr9ei6JAmB1nfix0YXxHzNVmqnY=', 'Content-Length': '0'} 2023-10-18T20:39:43.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006248bcbf219df944-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.147 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.147 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.148 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.148 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.149 DEBUG:boto:Method: GET 2023-10-18T20:39:43.149 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.150 DEBUG:boto:Data: 2023-10-18T20:39:43.150 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.151 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.151 DEBUG:boto:Params: {} 2023-10-18T20:39:43.152 DEBUG:boto:Token: None 2023-10-18T20:39:43.152 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039a10e8a12ff904c-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.156 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.157 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.157 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.158 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.158 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.159 DEBUG:boto:Method: GET 2023-10-18T20:39:43.159 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.159 DEBUG:boto:Data: 2023-10-18T20:39:43.160 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.160 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.161 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.161 DEBUG:boto:Params: {} 2023-10-18T20:39:43.162 DEBUG:boto:Token: None 2023-10-18T20:39:43.162 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002164c4333787754a-006530428f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.210 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.211 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.212 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.212 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.213 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:39:43.213 DEBUG:boto:Data: 2023-10-18T20:39:43.214 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.214 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.214 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.215 DEBUG:boto:Params: {} 2023-10-18T20:39:43.215 DEBUG:boto:Token: None 2023-10-18T20:39:43.216 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17 2023-10-18T20:39:43.216 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ULc6y98u+MbTCDFU0RIMBwzu2lE= 2023-10-18T20:39:43.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ULc6y98u+MbTCDFU0RIMBwzu2lE=', 'Content-Length': '0'} 2023-10-18T20:39:43.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf889a0ce490128-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.219 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.220 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.220 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.220 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:39:43.221 DEBUG:boto:Data: 2023-10-18T20:39:43.221 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.222 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.222 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.223 DEBUG:boto:Params: {} 2023-10-18T20:39:43.223 DEBUG:boto:Token: None 2023-10-18T20:39:43.224 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17 2023-10-18T20:39:43.225 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ULc6y98u+MbTCDFU0RIMBwzu2lE= 2023-10-18T20:39:43.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ULc6y98u+MbTCDFU0RIMBwzu2lE=', 'Content-Length': '0'} 2023-10-18T20:39:43.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df59cb89833c1ec-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:39:43.230 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.231 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.231 DEBUG:boto:Method: GET 2023-10-18T20:39:43.232 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:39:43.233 DEBUG:boto:Data: 2023-10-18T20:39:43.234 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.234 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.235 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.236 DEBUG:boto:Params: {} 2023-10-18T20:39:43.237 DEBUG:boto:Token: None 2023-10-18T20:39:43.237 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17 2023-10-18T20:39:43.238 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZG4WTDgMmtpqAjQ9PyqSBFkzY68= 2023-10-18T20:39:43.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZG4WTDgMmtpqAjQ9PyqSBFkzY68=', 'Content-Length': '0'} 2023-10-18T20:39:43.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff223bb6c477bfbc-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.243 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:39:43.243 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:39:43.244 DEBUG:boto:Method: GET 2023-10-18T20:39:43.245 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:39:43.246 DEBUG:boto:Data: 2023-10-18T20:39:43.246 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.247 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.248 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.249 DEBUG:boto:Params: {} 2023-10-18T20:39:43.249 DEBUG:boto:Token: None 2023-10-18T20:39:43.250 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj17 2023-10-18T20:39:43.251 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZG4WTDgMmtpqAjQ9PyqSBFkzY68= 2023-10-18T20:39:43.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZG4WTDgMmtpqAjQ9PyqSBFkzY68=', 'Content-Length': '0'} 2023-10-18T20:39:43.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ad1b9c9f2b25f23-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.255 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.255 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.256 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.256 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.257 DEBUG:boto:Method: GET 2023-10-18T20:39:43.257 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.258 DEBUG:boto:Data: 2023-10-18T20:39:43.258 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.259 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.259 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.260 DEBUG:boto:Params: {} 2023-10-18T20:39:43.260 DEBUG:boto:Token: None 2023-10-18T20:39:43.261 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.261 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b78e9de30a08196-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.265 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.266 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.266 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.267 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.267 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.268 DEBUG:boto:Method: GET 2023-10-18T20:39:43.268 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:39:43.268 DEBUG:boto:Data: 2023-10-18T20:39:43.269 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.270 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.270 DEBUG:boto:Params: {} 2023-10-18T20:39:43.271 DEBUG:boto:Token: None 2023-10-18T20:39:43.272 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.272 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e4b03eb24b3127c-006530428f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.277 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:39:43.278 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.278 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.279 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.279 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.280 DEBUG:boto:Method: GET 2023-10-18T20:39:43.280 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.281 DEBUG:boto:Data: 2023-10-18T20:39:43.281 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.281 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.282 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.282 DEBUG:boto:Params: {} 2023-10-18T20:39:43.283 DEBUG:boto:Token: None 2023-10-18T20:39:43.284 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lFAxjIqUYS4q3YaYXWWEF+jqJ2M= 2023-10-18T20:39:43.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lFAxjIqUYS4q3YaYXWWEF+jqJ2M=', 'Content-Length': '0'} 2023-10-18T20:39:43.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000623b12b09fa56585-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.287 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.287 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.288 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.288 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.289 DEBUG:boto:Method: GET 2023-10-18T20:39:43.289 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.289 DEBUG:boto:Data: 2023-10-18T20:39:43.290 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.290 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.291 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.291 DEBUG:boto:Params: {} 2023-10-18T20:39:43.292 DEBUG:boto:Token: None 2023-10-18T20:39:43.292 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.293 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lFAxjIqUYS4q3YaYXWWEF+jqJ2M= 2023-10-18T20:39:43.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lFAxjIqUYS4q3YaYXWWEF+jqJ2M=', 'Content-Length': '0'} 2023-10-18T20:39:43.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2fa064eed83a8bb-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.296 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.296 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.297 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.297 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.298 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.298 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.299 DEBUG:boto:Data: 2023-10-18T20:39:43.299 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.299 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.300 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.300 DEBUG:boto:Params: {} 2023-10-18T20:39:43.301 DEBUG:boto:Token: None 2023-10-18T20:39:43.301 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8LnIXV1QW6vf0KEFdpMnzNZbgis= 2023-10-18T20:39:43.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8LnIXV1QW6vf0KEFdpMnzNZbgis=', 'Content-Length': '0'} 2023-10-18T20:39:43.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f279354e21354d-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.304 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.304 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.305 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.305 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.306 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.306 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.307 DEBUG:boto:Data: 2023-10-18T20:39:43.307 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.307 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.308 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.308 DEBUG:boto:Params: {} 2023-10-18T20:39:43.308 DEBUG:boto:Token: None 2023-10-18T20:39:43.309 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.309 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8LnIXV1QW6vf0KEFdpMnzNZbgis= 2023-10-18T20:39:43.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8LnIXV1QW6vf0KEFdpMnzNZbgis=', 'Content-Length': '0'} 2023-10-18T20:39:43.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e909d7d1e64c99e0-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:39:43.313 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.313 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.313 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.314 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.314 DEBUG:boto:Method: GET 2023-10-18T20:39:43.315 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.315 DEBUG:boto:Data: 2023-10-18T20:39:43.315 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.316 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.316 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.317 DEBUG:boto:Params: {} 2023-10-18T20:39:43.317 DEBUG:boto:Token: None 2023-10-18T20:39:43.317 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.318 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lFAxjIqUYS4q3YaYXWWEF+jqJ2M= 2023-10-18T20:39:43.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lFAxjIqUYS4q3YaYXWWEF+jqJ2M=', 'Content-Length': '0'} 2023-10-18T20:39:43.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e9a451c49f6f75f-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.362 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.363 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.364 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.364 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.364 DEBUG:boto:Method: GET 2023-10-18T20:39:43.365 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.365 DEBUG:boto:Data: 2023-10-18T20:39:43.366 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.366 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.366 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.367 DEBUG:boto:Params: {} 2023-10-18T20:39:43.367 DEBUG:boto:Token: None 2023-10-18T20:39:43.368 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:39:43.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lFAxjIqUYS4q3YaYXWWEF+jqJ2M= 2023-10-18T20:39:43.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lFAxjIqUYS4q3YaYXWWEF+jqJ2M=', 'Content-Length': '0'} 2023-10-18T20:39:43.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f910d28e9a9f9a4d-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.372 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.372 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.373 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.373 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.374 DEBUG:boto:Method: GET 2023-10-18T20:39:43.374 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.375 DEBUG:boto:Data: 2023-10-18T20:39:43.375 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.375 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.376 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.376 DEBUG:boto:Params: {} 2023-10-18T20:39:43.377 DEBUG:boto:Token: None 2023-10-18T20:39:43.377 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.378 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080ae6a7c72ea8b60-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.381 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.382 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.382 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.383 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.383 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.384 DEBUG:boto:Method: GET 2023-10-18T20:39:43.384 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.384 DEBUG:boto:Data: 2023-10-18T20:39:43.385 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.385 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.386 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.386 DEBUG:boto:Params: {} 2023-10-18T20:39:43.387 DEBUG:boto:Token: None 2023-10-18T20:39:43.387 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bf18a302d8b2f05-006530428f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.434 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.435 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.436 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.436 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.437 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:39:43.437 DEBUG:boto:Data: 2023-10-18T20:39:43.437 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.438 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.438 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.439 DEBUG:boto:Params: {} 2023-10-18T20:39:43.439 DEBUG:boto:Token: None 2023-10-18T20:39:43.440 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19 2023-10-18T20:39:43.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TCU2fTsJISfM6Uu35Ip9XIC0HSQ= 2023-10-18T20:39:43.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TCU2fTsJISfM6Uu35Ip9XIC0HSQ=', 'Content-Length': '0'} 2023-10-18T20:39:43.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc8b47d9c60a219a-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.443 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.444 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.444 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.444 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:39:43.445 DEBUG:boto:Data: 2023-10-18T20:39:43.445 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.446 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.446 DEBUG:boto:Params: {} 2023-10-18T20:39:43.447 DEBUG:boto:Token: None 2023-10-18T20:39:43.448 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19 2023-10-18T20:39:43.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TCU2fTsJISfM6Uu35Ip9XIC0HSQ= 2023-10-18T20:39:43.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TCU2fTsJISfM6Uu35Ip9XIC0HSQ=', 'Content-Length': '0'} 2023-10-18T20:39:43.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cfa095bb312e318-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:39:43.451 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.452 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.452 DEBUG:boto:Method: GET 2023-10-18T20:39:43.452 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:39:43.453 DEBUG:boto:Data: 2023-10-18T20:39:43.453 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.454 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.454 DEBUG:boto:Params: {} 2023-10-18T20:39:43.455 DEBUG:boto:Token: None 2023-10-18T20:39:43.455 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19 2023-10-18T20:39:43.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PzxMsKH6OBOJEmKaQDgdAEOLYa0= 2023-10-18T20:39:43.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PzxMsKH6OBOJEmKaQDgdAEOLYa0=', 'Content-Length': '0'} 2023-10-18T20:39:43.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000456e3edb7c0b140-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.459 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:39:43.460 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:39:43.460 DEBUG:boto:Method: GET 2023-10-18T20:39:43.460 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:39:43.461 DEBUG:boto:Data: 2023-10-18T20:39:43.461 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.462 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.463 DEBUG:boto:Params: {} 2023-10-18T20:39:43.463 DEBUG:boto:Token: None 2023-10-18T20:39:43.464 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj19 2023-10-18T20:39:43.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PzxMsKH6OBOJEmKaQDgdAEOLYa0= 2023-10-18T20:39:43.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PzxMsKH6OBOJEmKaQDgdAEOLYa0=', 'Content-Length': '0'} 2023-10-18T20:39:43.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bbd89cec2508a01-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.467 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.468 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.468 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.469 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.469 DEBUG:boto:Method: GET 2023-10-18T20:39:43.469 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.470 DEBUG:boto:Data: 2023-10-18T20:39:43.470 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.471 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.471 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.472 DEBUG:boto:Params: {} 2023-10-18T20:39:43.472 DEBUG:boto:Token: None 2023-10-18T20:39:43.473 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.473 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df14ba049731c9c2-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.522 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.523 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.524 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.524 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.526 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.526 DEBUG:boto:Method: GET 2023-10-18T20:39:43.527 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:39:43.527 DEBUG:boto:Data: 2023-10-18T20:39:43.528 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.528 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.528 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.529 DEBUG:boto:Params: {} 2023-10-18T20:39:43.529 DEBUG:boto:Token: None 2023-10-18T20:39:43.530 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.530 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6a81368ee87b39c-006530428f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.535 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:39:43.536 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.537 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.537 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.538 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.538 DEBUG:boto:Method: GET 2023-10-18T20:39:43.539 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.539 DEBUG:boto:Data: 2023-10-18T20:39:43.539 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.540 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.541 DEBUG:boto:Params: {} 2023-10-18T20:39:43.541 DEBUG:boto:Token: None 2023-10-18T20:39:43.542 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.542 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0ZNUR4yMDfGGDPRmAWjB8bbfN1U= 2023-10-18T20:39:43.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0ZNUR4yMDfGGDPRmAWjB8bbfN1U=', 'Content-Length': '0'} 2023-10-18T20:39:43.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b395324141fcec1-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.545 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.546 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.546 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.547 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.547 DEBUG:boto:Method: GET 2023-10-18T20:39:43.548 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.548 DEBUG:boto:Data: 2023-10-18T20:39:43.548 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.549 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.549 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.550 DEBUG:boto:Params: {} 2023-10-18T20:39:43.550 DEBUG:boto:Token: None 2023-10-18T20:39:43.551 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0ZNUR4yMDfGGDPRmAWjB8bbfN1U= 2023-10-18T20:39:43.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0ZNUR4yMDfGGDPRmAWjB8bbfN1U=', 'Content-Length': '0'} 2023-10-18T20:39:43.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc9c625e8bc7cfa-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.554 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.555 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.555 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.555 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.556 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.556 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.557 DEBUG:boto:Data: 2023-10-18T20:39:43.557 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.557 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.558 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.558 DEBUG:boto:Params: {} 2023-10-18T20:39:43.559 DEBUG:boto:Token: None 2023-10-18T20:39:43.559 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T2s8CTi4UOrJUqG9mKt0rDty0vc= 2023-10-18T20:39:43.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T2s8CTi4UOrJUqG9mKt0rDty0vc=', 'Content-Length': '0'} 2023-10-18T20:39:43.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c92186854b8a503d-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.562 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.563 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.563 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.564 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.564 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.564 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.565 DEBUG:boto:Data: 2023-10-18T20:39:43.565 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.566 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.566 DEBUG:boto:Params: {} 2023-10-18T20:39:43.567 DEBUG:boto:Token: None 2023-10-18T20:39:43.567 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T2s8CTi4UOrJUqG9mKt0rDty0vc= 2023-10-18T20:39:43.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T2s8CTi4UOrJUqG9mKt0rDty0vc=', 'Content-Length': '0'} 2023-10-18T20:39:43.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b273034c7166fc-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:39:43.571 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.571 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.572 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.572 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.573 DEBUG:boto:Method: GET 2023-10-18T20:39:43.573 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.574 DEBUG:boto:Data: 2023-10-18T20:39:43.574 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.574 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.575 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.575 DEBUG:boto:Params: {} 2023-10-18T20:39:43.576 DEBUG:boto:Token: None 2023-10-18T20:39:43.576 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0ZNUR4yMDfGGDPRmAWjB8bbfN1U= 2023-10-18T20:39:43.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0ZNUR4yMDfGGDPRmAWjB8bbfN1U=', 'Content-Length': '0'} 2023-10-18T20:39:43.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b5bfccefb0badf-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.580 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.580 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.580 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.581 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.581 DEBUG:boto:Method: GET 2023-10-18T20:39:43.582 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.582 DEBUG:boto:Data: 2023-10-18T20:39:43.583 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.583 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.584 DEBUG:boto:Params: {} 2023-10-18T20:39:43.584 DEBUG:boto:Token: None 2023-10-18T20:39:43.585 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:39:43.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0ZNUR4yMDfGGDPRmAWjB8bbfN1U= 2023-10-18T20:39:43.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0ZNUR4yMDfGGDPRmAWjB8bbfN1U=', 'Content-Length': '0'} 2023-10-18T20:39:43.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025ea30954c8ef441-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.588 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.589 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.589 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.589 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.590 DEBUG:boto:Method: GET 2023-10-18T20:39:43.590 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.591 DEBUG:boto:Data: 2023-10-18T20:39:43.591 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.591 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.592 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.592 DEBUG:boto:Params: {} 2023-10-18T20:39:43.593 DEBUG:boto:Token: None 2023-10-18T20:39:43.593 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.594 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008362dac590889fe6-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.597 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.598 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.598 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.599 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.599 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.600 DEBUG:boto:Method: GET 2023-10-18T20:39:43.600 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.600 DEBUG:boto:Data: 2023-10-18T20:39:43.601 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.601 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.602 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.602 DEBUG:boto:Params: {} 2023-10-18T20:39:43.603 DEBUG:boto:Token: None 2023-10-18T20:39:43.603 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.603 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afbf26241dc833dd-006530428f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.607 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.608 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.609 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.609 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.609 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:39:43.610 DEBUG:boto:Data: 2023-10-18T20:39:43.610 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.611 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.611 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.611 DEBUG:boto:Params: {} 2023-10-18T20:39:43.612 DEBUG:boto:Token: None 2023-10-18T20:39:43.612 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22 2023-10-18T20:39:43.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eWRrDvW8J1KYMPM7ECNuRoOkCOA= 2023-10-18T20:39:43.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eWRrDvW8J1KYMPM7ECNuRoOkCOA=', 'Content-Length': '0'} 2023-10-18T20:39:43.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f9899e898f973c-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.616 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.616 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.616 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.617 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:39:43.617 DEBUG:boto:Data: 2023-10-18T20:39:43.618 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.618 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.618 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.619 DEBUG:boto:Params: {} 2023-10-18T20:39:43.619 DEBUG:boto:Token: None 2023-10-18T20:39:43.620 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22 2023-10-18T20:39:43.620 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eWRrDvW8J1KYMPM7ECNuRoOkCOA= 2023-10-18T20:39:43.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eWRrDvW8J1KYMPM7ECNuRoOkCOA=', 'Content-Length': '0'} 2023-10-18T20:39:43.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7d01fd8a279100e-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:39:43.624 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.624 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.624 DEBUG:boto:Method: GET 2023-10-18T20:39:43.625 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:39:43.625 DEBUG:boto:Data: 2023-10-18T20:39:43.626 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.626 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.627 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.627 DEBUG:boto:Params: {} 2023-10-18T20:39:43.627 DEBUG:boto:Token: None 2023-10-18T20:39:43.628 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22 2023-10-18T20:39:43.628 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w8kzrCr+1hJ1rdbT8bLMbpHOZd0= 2023-10-18T20:39:43.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w8kzrCr+1hJ1rdbT8bLMbpHOZd0=', 'Content-Length': '0'} 2023-10-18T20:39:43.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eca54adcfd1d51e-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.674 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:39:43.675 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:39:43.676 DEBUG:boto:Method: GET 2023-10-18T20:39:43.676 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:39:43.676 DEBUG:boto:Data: 2023-10-18T20:39:43.677 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.677 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.678 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.678 DEBUG:boto:Params: {} 2023-10-18T20:39:43.679 DEBUG:boto:Token: None 2023-10-18T20:39:43.679 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj22 2023-10-18T20:39:43.680 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w8kzrCr+1hJ1rdbT8bLMbpHOZd0= 2023-10-18T20:39:43.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w8kzrCr+1hJ1rdbT8bLMbpHOZd0=', 'Content-Length': '0'} 2023-10-18T20:39:43.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f5ad7bd6caab3dd-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.683 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.683 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.684 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.684 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.684 DEBUG:boto:Method: GET 2023-10-18T20:39:43.685 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.685 DEBUG:boto:Data: 2023-10-18T20:39:43.686 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.687 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.687 DEBUG:boto:Params: {} 2023-10-18T20:39:43.687 DEBUG:boto:Token: None 2023-10-18T20:39:43.688 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000270c817f2fe2170f-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.692 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.693 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.693 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.693 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.694 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.694 DEBUG:boto:Method: GET 2023-10-18T20:39:43.695 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:39:43.695 DEBUG:boto:Data: 2023-10-18T20:39:43.696 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.696 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.696 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.697 DEBUG:boto:Params: {} 2023-10-18T20:39:43.697 DEBUG:boto:Token: None 2023-10-18T20:39:43.698 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.698 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fd3eccfe572af7c-006530428f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.703 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:39:43.704 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.704 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.704 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.705 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.705 DEBUG:boto:Method: GET 2023-10-18T20:39:43.706 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.706 DEBUG:boto:Data: 2023-10-18T20:39:43.707 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.707 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.708 DEBUG:boto:Params: {} 2023-10-18T20:39:43.708 DEBUG:boto:Token: None 2023-10-18T20:39:43.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fyzmWXaxQn6gGfaXj/Qkp7a+1nA= 2023-10-18T20:39:43.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fyzmWXaxQn6gGfaXj/Qkp7a+1nA=', 'Content-Length': '0'} 2023-10-18T20:39:43.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008c9e36c59bc4be1-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.712 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.713 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.713 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.714 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.714 DEBUG:boto:Method: GET 2023-10-18T20:39:43.715 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.715 DEBUG:boto:Data: 2023-10-18T20:39:43.715 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.716 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.716 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.717 DEBUG:boto:Params: {} 2023-10-18T20:39:43.717 DEBUG:boto:Token: None 2023-10-18T20:39:43.718 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.718 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fyzmWXaxQn6gGfaXj/Qkp7a+1nA= 2023-10-18T20:39:43.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fyzmWXaxQn6gGfaXj/Qkp7a+1nA=', 'Content-Length': '0'} 2023-10-18T20:39:43.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b1b25d5da3727f7-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.721 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.722 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.722 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.723 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.723 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.723 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.724 DEBUG:boto:Data: 2023-10-18T20:39:43.724 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.724 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.725 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.725 DEBUG:boto:Params: {} 2023-10-18T20:39:43.726 DEBUG:boto:Token: None 2023-10-18T20:39:43.726 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.727 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HXsgvvs77JAEu7ByNQYmnToTAfo= 2023-10-18T20:39:43.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HXsgvvs77JAEu7ByNQYmnToTAfo=', 'Content-Length': '0'} 2023-10-18T20:39:43.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22a7b151d460699-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.730 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.730 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.730 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.731 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.731 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.732 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.732 DEBUG:boto:Data: 2023-10-18T20:39:43.732 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.733 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.734 DEBUG:boto:Params: {} 2023-10-18T20:39:43.734 DEBUG:boto:Token: None 2023-10-18T20:39:43.734 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HXsgvvs77JAEu7ByNQYmnToTAfo= 2023-10-18T20:39:43.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HXsgvvs77JAEu7ByNQYmnToTAfo=', 'Content-Length': '0'} 2023-10-18T20:39:43.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63b24b5b0e52ad8-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:39:43.738 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.738 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.739 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.739 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.740 DEBUG:boto:Method: GET 2023-10-18T20:39:43.740 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.740 DEBUG:boto:Data: 2023-10-18T20:39:43.741 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.741 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.742 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.742 DEBUG:boto:Params: {} 2023-10-18T20:39:43.743 DEBUG:boto:Token: None 2023-10-18T20:39:43.743 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.744 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fyzmWXaxQn6gGfaXj/Qkp7a+1nA= 2023-10-18T20:39:43.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fyzmWXaxQn6gGfaXj/Qkp7a+1nA=', 'Content-Length': '0'} 2023-10-18T20:39:43.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5378fc9b0a769ff-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.747 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.747 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.748 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.748 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.748 DEBUG:boto:Method: GET 2023-10-18T20:39:43.749 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.749 DEBUG:boto:Data: 2023-10-18T20:39:43.750 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.750 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.751 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.753 DEBUG:boto:Params: {} 2023-10-18T20:39:43.754 DEBUG:boto:Token: None 2023-10-18T20:39:43.754 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:39:43.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fyzmWXaxQn6gGfaXj/Qkp7a+1nA= 2023-10-18T20:39:43.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fyzmWXaxQn6gGfaXj/Qkp7a+1nA=', 'Content-Length': '0'} 2023-10-18T20:39:43.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063c559d4540521dc-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.758 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.758 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.758 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.759 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.759 DEBUG:boto:Method: GET 2023-10-18T20:39:43.760 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.760 DEBUG:boto:Data: 2023-10-18T20:39:43.761 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.761 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.762 DEBUG:boto:Params: {} 2023-10-18T20:39:43.762 DEBUG:boto:Token: None 2023-10-18T20:39:43.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074d0e9982bee0e00-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.806 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.807 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.808 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.808 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.809 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.809 DEBUG:boto:Method: GET 2023-10-18T20:39:43.810 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.810 DEBUG:boto:Data: 2023-10-18T20:39:43.811 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.811 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.811 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.812 DEBUG:boto:Params: {} 2023-10-18T20:39:43.812 DEBUG:boto:Token: None 2023-10-18T20:39:43.813 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.813 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec9dd325a8aefe2e-006530428f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.817 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.818 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.818 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.819 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.819 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:39:43.819 DEBUG:boto:Data: 2023-10-18T20:39:43.820 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.820 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.821 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.821 DEBUG:boto:Params: {} 2023-10-18T20:39:43.821 DEBUG:boto:Token: None 2023-10-18T20:39:43.822 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23 2023-10-18T20:39:43.822 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W3XgWARx+V+CINcwC49N4tl5gIk= 2023-10-18T20:39:43.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W3XgWARx+V+CINcwC49N4tl5gIk=', 'Content-Length': '0'} 2023-10-18T20:39:43.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001143d4bbb377c40e-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.825 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.826 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.826 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.826 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:39:43.827 DEBUG:boto:Data: 2023-10-18T20:39:43.827 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.828 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.828 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.828 DEBUG:boto:Params: {} 2023-10-18T20:39:43.829 DEBUG:boto:Token: None 2023-10-18T20:39:43.829 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23 2023-10-18T20:39:43.830 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W3XgWARx+V+CINcwC49N4tl5gIk= 2023-10-18T20:39:43.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W3XgWARx+V+CINcwC49N4tl5gIk=', 'Content-Length': '0'} 2023-10-18T20:39:43.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e651f79205650496-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:39:43.833 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.834 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.834 DEBUG:boto:Method: GET 2023-10-18T20:39:43.834 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:39:43.835 DEBUG:boto:Data: 2023-10-18T20:39:43.835 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.836 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.836 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.836 DEBUG:boto:Params: {} 2023-10-18T20:39:43.837 DEBUG:boto:Token: None 2023-10-18T20:39:43.837 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23 2023-10-18T20:39:43.838 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:op0P54GFB/TOAf9i0JL6WQpuJu8= 2023-10-18T20:39:43.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:op0P54GFB/TOAf9i0JL6WQpuJu8=', 'Content-Length': '0'} 2023-10-18T20:39:43.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cf2e2e4a851248a-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.841 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:39:43.841 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:39:43.842 DEBUG:boto:Method: GET 2023-10-18T20:39:43.842 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:39:43.843 DEBUG:boto:Data: 2023-10-18T20:39:43.843 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.844 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.844 DEBUG:boto:Params: {} 2023-10-18T20:39:43.845 DEBUG:boto:Token: None 2023-10-18T20:39:43.845 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj23 2023-10-18T20:39:43.846 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:op0P54GFB/TOAf9i0JL6WQpuJu8= 2023-10-18T20:39:43.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:op0P54GFB/TOAf9i0JL6WQpuJu8=', 'Content-Length': '0'} 2023-10-18T20:39:43.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000515a55de3e0f07dc-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.849 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.849 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.849 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.850 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.850 DEBUG:boto:Method: GET 2023-10-18T20:39:43.851 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.851 DEBUG:boto:Data: 2023-10-18T20:39:43.852 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.852 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.852 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.853 DEBUG:boto:Params: {} 2023-10-18T20:39:43.853 DEBUG:boto:Token: None 2023-10-18T20:39:43.854 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.854 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000281cb07d469ba792-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.858 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.858 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.859 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.859 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.860 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.860 DEBUG:boto:Method: GET 2023-10-18T20:39:43.861 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:39:43.861 DEBUG:boto:Data: 2023-10-18T20:39:43.861 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.862 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.862 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.863 DEBUG:boto:Params: {} 2023-10-18T20:39:43.863 DEBUG:boto:Token: None 2023-10-18T20:39:43.864 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.864 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f193c96474ddebc-006530428f-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.868 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:43.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:39:43.869 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:43.870 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:43.870 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.870 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.871 DEBUG:boto:Method: GET 2023-10-18T20:39:43.871 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.872 DEBUG:boto:Data: 2023-10-18T20:39:43.872 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.872 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.873 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.873 DEBUG:boto:Params: {} 2023-10-18T20:39:43.874 DEBUG:boto:Token: None 2023-10-18T20:39:43.874 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.875 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yzvkZ8xthaOa3vOjTVK8P+twmeQ= 2023-10-18T20:39:43.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yzvkZ8xthaOa3vOjTVK8P+twmeQ=', 'Content-Length': '0'} 2023-10-18T20:39:43.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019ee7dd289537298-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.878 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:43.878 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:43.879 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.879 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.879 DEBUG:boto:Method: GET 2023-10-18T20:39:43.880 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.880 DEBUG:boto:Data: 2023-10-18T20:39:43.881 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.881 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.882 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.882 DEBUG:boto:Params: {} 2023-10-18T20:39:43.883 DEBUG:boto:Token: None 2023-10-18T20:39:43.883 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yzvkZ8xthaOa3vOjTVK8P+twmeQ= 2023-10-18T20:39:43.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yzvkZ8xthaOa3vOjTVK8P+twmeQ=', 'Content-Length': '0'} 2023-10-18T20:39:43.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b934dec198d673-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.930 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:43.931 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:43.931 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.932 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.932 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.933 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.933 DEBUG:boto:Data: 2023-10-18T20:39:43.934 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.934 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.934 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.935 DEBUG:boto:Params: {} 2023-10-18T20:39:43.935 DEBUG:boto:Token: None 2023-10-18T20:39:43.936 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PTpGXL0m4z24+GaS8pfgLpkyw5k= 2023-10-18T20:39:43.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PTpGXL0m4z24+GaS8pfgLpkyw5k=', 'Content-Length': '0'} 2023-10-18T20:39:43.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8dafcaa5e9c09c5-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.939 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:43.940 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:43.940 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.940 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.941 DEBUG:boto:Method: HEAD 2023-10-18T20:39:43.941 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.941 DEBUG:boto:Data: 2023-10-18T20:39:43.942 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.942 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.943 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.943 DEBUG:boto:Params: {} 2023-10-18T20:39:43.944 DEBUG:boto:Token: None 2023-10-18T20:39:43.944 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.945 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PTpGXL0m4z24+GaS8pfgLpkyw5k= 2023-10-18T20:39:43.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PTpGXL0m4z24+GaS8pfgLpkyw5k=', 'Content-Length': '0'} 2023-10-18T20:39:43.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be759866bafd9353-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:39:43.948 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:43.948 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:43.949 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.949 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.949 DEBUG:boto:Method: GET 2023-10-18T20:39:43.950 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.950 DEBUG:boto:Data: 2023-10-18T20:39:43.951 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.951 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.952 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.952 DEBUG:boto:Params: {} 2023-10-18T20:39:43.952 DEBUG:boto:Token: None 2023-10-18T20:39:43.953 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.953 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yzvkZ8xthaOa3vOjTVK8P+twmeQ= 2023-10-18T20:39:43.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yzvkZ8xthaOa3vOjTVK8P+twmeQ=', 'Content-Length': '0'} 2023-10-18T20:39:43.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000768f7b99b1bb4a5f-006530428f-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.959 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:43.960 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:43.960 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.961 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.961 DEBUG:boto:Method: GET 2023-10-18T20:39:43.961 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.962 DEBUG:boto:Data: 2023-10-18T20:39:43.962 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.963 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:43.963 DEBUG:boto:Port: 8000 2023-10-18T20:39:43.964 DEBUG:boto:Params: {} 2023-10-18T20:39:43.964 DEBUG:boto:Token: None 2023-10-18T20:39:43.965 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:39:43.965 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yzvkZ8xthaOa3vOjTVK8P+twmeQ= 2023-10-18T20:39:43.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yzvkZ8xthaOa3vOjTVK8P+twmeQ=', 'Content-Length': '0'} 2023-10-18T20:39:43.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bdcd6084f5cf8a8-006530428f-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:43.968 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:43.968 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:43.969 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:39:43.969 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:39:43.969 DEBUG:boto:Method: GET 2023-10-18T20:39:43.970 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:39:43.970 DEBUG:boto:Data: 2023-10-18T20:39:43.971 DEBUG:boto:Headers: {} 2023-10-18T20:39:43.971 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:43.971 DEBUG:boto:Port: 8001 2023-10-18T20:39:43.972 DEBUG:boto:Params: {} 2023-10-18T20:39:43.972 DEBUG:boto:Token: None 2023-10-18T20:39:43.973 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:43 GMT /bofrnf-59/ 2023-10-18T20:39:43.973 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4= 2023-10-18T20:39:43.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AqNo+3ACJeKuXRy9Bo+IKChFHD4=', 'Content-Length': '0'} 2023-10-18T20:39:43.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f08a3306c0131de4-006530428f-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.018 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.019 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.020 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.020 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:39:44.021 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:39:44.021 DEBUG:boto:Method: GET 2023-10-18T20:39:44.022 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:39:44.022 DEBUG:boto:Data: 2023-10-18T20:39:44.022 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.023 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.023 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.024 DEBUG:boto:Params: {} 2023-10-18T20:39:44.024 DEBUG:boto:Token: None 2023-10-18T20:39:44.025 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.025 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c92d61a66102e6-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.029 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.030 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:44.030 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:44.031 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.031 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:39:44.032 DEBUG:boto:Data: 2023-10-18T20:39:44.032 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.032 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.033 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.033 DEBUG:boto:Params: {} 2023-10-18T20:39:44.034 DEBUG:boto:Token: None 2023-10-18T20:39:44.034 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj33 2023-10-18T20:39:44.035 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+s+9ZWiWaz9wZ38jBf/9GXllHiQ= 2023-10-18T20:39:44.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+s+9ZWiWaz9wZ38jBf/9GXllHiQ=', 'Content-Length': '0'} 2023-10-18T20:39:44.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0fb28fdce96e836-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.038 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:44.038 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:44.038 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.039 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:39:44.039 DEBUG:boto:Data: 2023-10-18T20:39:44.040 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.040 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.040 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.041 DEBUG:boto:Params: {} 2023-10-18T20:39:44.041 DEBUG:boto:Token: None 2023-10-18T20:39:44.042 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj33 2023-10-18T20:39:44.042 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+s+9ZWiWaz9wZ38jBf/9GXllHiQ= 2023-10-18T20:39:44.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+s+9ZWiWaz9wZ38jBf/9GXllHiQ=', 'Content-Length': '0'} 2023-10-18T20:39:44.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe249f9cb4475134-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:39:44.045 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:44.046 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:44.046 DEBUG:boto:Method: GET 2023-10-18T20:39:44.046 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:39:44.047 DEBUG:boto:Data: 2023-10-18T20:39:44.047 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.048 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.048 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.049 DEBUG:boto:Params: {} 2023-10-18T20:39:44.049 DEBUG:boto:Token: None 2023-10-18T20:39:44.049 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj33 2023-10-18T20:39:44.050 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LXgEdazZ5WMSXeXI+MZCJvwN4Hs= 2023-10-18T20:39:44.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LXgEdazZ5WMSXeXI+MZCJvwN4Hs=', 'Content-Length': '0'} 2023-10-18T20:39:44.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068e5fb58021a6347-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.053 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:39:44.053 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:39:44.054 DEBUG:boto:Method: GET 2023-10-18T20:39:44.054 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:39:44.055 DEBUG:boto:Data: 2023-10-18T20:39:44.055 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.056 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.056 DEBUG:boto:Params: {} 2023-10-18T20:39:44.057 DEBUG:boto:Token: None 2023-10-18T20:39:44.057 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj33 2023-10-18T20:39:44.058 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LXgEdazZ5WMSXeXI+MZCJvwN4Hs= 2023-10-18T20:39:44.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LXgEdazZ5WMSXeXI+MZCJvwN4Hs=', 'Content-Length': '0'} 2023-10-18T20:39:44.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b9242027022cb2-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.061 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.061 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.062 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:39:44.062 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:39:44.063 DEBUG:boto:Method: GET 2023-10-18T20:39:44.063 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:39:44.063 DEBUG:boto:Data: 2023-10-18T20:39:44.064 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.065 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.065 DEBUG:boto:Params: {} 2023-10-18T20:39:44.066 DEBUG:boto:Token: None 2023-10-18T20:39:44.066 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.066 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335a08fd6aa109bf-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.070 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.071 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.071 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.071 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:39:44.072 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:39:44.072 DEBUG:boto:Method: GET 2023-10-18T20:39:44.073 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:39:44.073 DEBUG:boto:Data: 2023-10-18T20:39:44.074 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.074 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.074 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.075 DEBUG:boto:Params: {} 2023-10-18T20:39:44.075 DEBUG:boto:Token: None 2023-10-18T20:39:44.076 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.076 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8da982d0b1272ee-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.080 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:39:44.081 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.082 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.082 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.083 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.083 DEBUG:boto:Method: GET 2023-10-18T20:39:44.083 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.084 DEBUG:boto:Data: 2023-10-18T20:39:44.084 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.085 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.085 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.086 DEBUG:boto:Params: {} 2023-10-18T20:39:44.086 DEBUG:boto:Token: None 2023-10-18T20:39:44.087 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.087 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iODyJcq6iGeG2T3aFFtEX8BZBrY= 2023-10-18T20:39:44.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iODyJcq6iGeG2T3aFFtEX8BZBrY=', 'Content-Length': '0'} 2023-10-18T20:39:44.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f43f299aeca04c5-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.090 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.091 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.091 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.092 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.092 DEBUG:boto:Method: GET 2023-10-18T20:39:44.092 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.093 DEBUG:boto:Data: 2023-10-18T20:39:44.093 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.094 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.094 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.095 DEBUG:boto:Params: {} 2023-10-18T20:39:44.095 DEBUG:boto:Token: None 2023-10-18T20:39:44.096 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.096 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iODyJcq6iGeG2T3aFFtEX8BZBrY= 2023-10-18T20:39:44.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iODyJcq6iGeG2T3aFFtEX8BZBrY=', 'Content-Length': '0'} 2023-10-18T20:39:44.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000529b82fb74fbea87-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.146 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.147 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.147 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.148 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.148 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.149 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.149 DEBUG:boto:Data: 2023-10-18T20:39:44.149 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.150 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.151 DEBUG:boto:Params: {} 2023-10-18T20:39:44.151 DEBUG:boto:Token: None 2023-10-18T20:39:44.152 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orfin4paY7UmDwqvR1/60+vSvuQ= 2023-10-18T20:39:44.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orfin4paY7UmDwqvR1/60+vSvuQ=', 'Content-Length': '0'} 2023-10-18T20:39:44.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029137e5c594c4805-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.155 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.156 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.156 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.157 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.157 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.157 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.158 DEBUG:boto:Data: 2023-10-18T20:39:44.158 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.159 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.161 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.162 DEBUG:boto:Params: {} 2023-10-18T20:39:44.162 DEBUG:boto:Token: None 2023-10-18T20:39:44.163 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:orfin4paY7UmDwqvR1/60+vSvuQ= 2023-10-18T20:39:44.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:orfin4paY7UmDwqvR1/60+vSvuQ=', 'Content-Length': '0'} 2023-10-18T20:39:44.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027e94aab6aa6044f-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:39:44.167 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.167 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.168 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.168 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.169 DEBUG:boto:Method: GET 2023-10-18T20:39:44.169 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.169 DEBUG:boto:Data: 2023-10-18T20:39:44.170 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.170 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.171 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.171 DEBUG:boto:Params: {} 2023-10-18T20:39:44.172 DEBUG:boto:Token: None 2023-10-18T20:39:44.172 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.173 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iODyJcq6iGeG2T3aFFtEX8BZBrY= 2023-10-18T20:39:44.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iODyJcq6iGeG2T3aFFtEX8BZBrY=', 'Content-Length': '0'} 2023-10-18T20:39:44.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7218d353af84d10-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.176 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.176 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.177 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.177 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.177 DEBUG:boto:Method: GET 2023-10-18T20:39:44.178 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.178 DEBUG:boto:Data: 2023-10-18T20:39:44.179 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.179 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.180 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.180 DEBUG:boto:Params: {} 2023-10-18T20:39:44.180 DEBUG:boto:Token: None 2023-10-18T20:39:44.181 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:39:44.181 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iODyJcq6iGeG2T3aFFtEX8BZBrY= 2023-10-18T20:39:44.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iODyJcq6iGeG2T3aFFtEX8BZBrY=', 'Content-Length': '0'} 2023-10-18T20:39:44.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b996171573d09d-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.185 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.185 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.186 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.186 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.186 DEBUG:boto:Method: GET 2023-10-18T20:39:44.187 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.187 DEBUG:boto:Data: 2023-10-18T20:39:44.188 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.188 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.188 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.189 DEBUG:boto:Params: {} 2023-10-18T20:39:44.189 DEBUG:boto:Token: None 2023-10-18T20:39:44.190 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.190 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6708f05b5bce698-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.235 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.236 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.236 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.236 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.237 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.237 DEBUG:boto:Method: GET 2023-10-18T20:39:44.238 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.238 DEBUG:boto:Data: 2023-10-18T20:39:44.239 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.239 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.239 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.240 DEBUG:boto:Params: {} 2023-10-18T20:39:44.240 DEBUG:boto:Token: None 2023-10-18T20:39:44.241 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.241 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5c391579deb9804-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.246 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.246 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.247 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.247 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.248 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:39:44.248 DEBUG:boto:Data: 2023-10-18T20:39:44.248 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.249 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.249 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.250 DEBUG:boto:Params: {} 2023-10-18T20:39:44.250 DEBUG:boto:Token: None 2023-10-18T20:39:44.250 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34 2023-10-18T20:39:44.251 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:66oW1apmap5sYYV4ThaVWMm/zQQ= 2023-10-18T20:39:44.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:66oW1apmap5sYYV4ThaVWMm/zQQ=', 'Content-Length': '0'} 2023-10-18T20:39:44.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f8cc29aa64da54-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.254 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.255 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.255 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.255 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:39:44.256 DEBUG:boto:Data: 2023-10-18T20:39:44.256 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.257 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.257 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.257 DEBUG:boto:Params: {} 2023-10-18T20:39:44.258 DEBUG:boto:Token: None 2023-10-18T20:39:44.258 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34 2023-10-18T20:39:44.259 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:66oW1apmap5sYYV4ThaVWMm/zQQ= 2023-10-18T20:39:44.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:66oW1apmap5sYYV4ThaVWMm/zQQ=', 'Content-Length': '0'} 2023-10-18T20:39:44.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000663a1a57c84814e4-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:39:44.262 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.262 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.263 DEBUG:boto:Method: GET 2023-10-18T20:39:44.263 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:39:44.264 DEBUG:boto:Data: 2023-10-18T20:39:44.264 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.265 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.265 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.266 DEBUG:boto:Params: {} 2023-10-18T20:39:44.266 DEBUG:boto:Token: None 2023-10-18T20:39:44.266 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34 2023-10-18T20:39:44.267 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BdUeJqRyjSFak7EzDKI7col+8YQ= 2023-10-18T20:39:44.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BdUeJqRyjSFak7EzDKI7col+8YQ=', 'Content-Length': '0'} 2023-10-18T20:39:44.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf99a9380405a58e-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.270 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:39:44.270 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:39:44.271 DEBUG:boto:Method: GET 2023-10-18T20:39:44.271 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:39:44.272 DEBUG:boto:Data: 2023-10-18T20:39:44.272 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.272 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.273 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.273 DEBUG:boto:Params: {} 2023-10-18T20:39:44.274 DEBUG:boto:Token: None 2023-10-18T20:39:44.274 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj34 2023-10-18T20:39:44.275 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BdUeJqRyjSFak7EzDKI7col+8YQ= 2023-10-18T20:39:44.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BdUeJqRyjSFak7EzDKI7col+8YQ=', 'Content-Length': '0'} 2023-10-18T20:39:44.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065cb7ba10d16d032-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.278 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.278 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.279 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.279 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.279 DEBUG:boto:Method: GET 2023-10-18T20:39:44.280 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.281 DEBUG:boto:Data: 2023-10-18T20:39:44.281 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.281 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.282 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.282 DEBUG:boto:Params: {} 2023-10-18T20:39:44.283 DEBUG:boto:Token: None 2023-10-18T20:39:44.284 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d00be15937c3149d-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.287 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.288 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.288 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.289 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.289 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.290 DEBUG:boto:Method: GET 2023-10-18T20:39:44.290 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:39:44.290 DEBUG:boto:Data: 2023-10-18T20:39:44.291 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.291 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.292 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.292 DEBUG:boto:Params: {} 2023-10-18T20:39:44.293 DEBUG:boto:Token: None 2023-10-18T20:39:44.293 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.294 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bb3ffd11df18bb8-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.298 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:39:44.299 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.299 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.300 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.300 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.301 DEBUG:boto:Method: GET 2023-10-18T20:39:44.301 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.302 DEBUG:boto:Data: 2023-10-18T20:39:44.302 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.303 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.303 DEBUG:boto:Params: {} 2023-10-18T20:39:44.304 DEBUG:boto:Token: None 2023-10-18T20:39:44.305 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g2042hDx9tBLNqJJTAN2XKYDHEk= 2023-10-18T20:39:44.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g2042hDx9tBLNqJJTAN2XKYDHEk=', 'Content-Length': '0'} 2023-10-18T20:39:44.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc15d1dc48b5f17a-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.308 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.309 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.309 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.309 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.310 DEBUG:boto:Method: GET 2023-10-18T20:39:44.310 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.311 DEBUG:boto:Data: 2023-10-18T20:39:44.311 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.312 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.312 DEBUG:boto:Params: {} 2023-10-18T20:39:44.313 DEBUG:boto:Token: None 2023-10-18T20:39:44.313 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.314 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g2042hDx9tBLNqJJTAN2XKYDHEk= 2023-10-18T20:39:44.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g2042hDx9tBLNqJJTAN2XKYDHEk=', 'Content-Length': '0'} 2023-10-18T20:39:44.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0c7eeeae9111a0b-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.317 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.317 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.318 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.318 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.319 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.319 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.319 DEBUG:boto:Data: 2023-10-18T20:39:44.320 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.320 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.321 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.321 DEBUG:boto:Params: {} 2023-10-18T20:39:44.321 DEBUG:boto:Token: None 2023-10-18T20:39:44.322 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.322 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SG3nS+4dDXkmKJe2IxJ0JNZmJ6k= 2023-10-18T20:39:44.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SG3nS+4dDXkmKJe2IxJ0JNZmJ6k=', 'Content-Length': '0'} 2023-10-18T20:39:44.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cca7e2fb11cd21f-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.325 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.325 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.326 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.326 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.327 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.327 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.327 DEBUG:boto:Data: 2023-10-18T20:39:44.328 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.329 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.329 DEBUG:boto:Params: {} 2023-10-18T20:39:44.329 DEBUG:boto:Token: None 2023-10-18T20:39:44.330 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SG3nS+4dDXkmKJe2IxJ0JNZmJ6k= 2023-10-18T20:39:44.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SG3nS+4dDXkmKJe2IxJ0JNZmJ6k=', 'Content-Length': '0'} 2023-10-18T20:39:44.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06561bd1e555de1-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:39:44.333 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.334 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.334 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.335 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.335 DEBUG:boto:Method: GET 2023-10-18T20:39:44.336 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.336 DEBUG:boto:Data: 2023-10-18T20:39:44.336 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.337 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.337 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.338 DEBUG:boto:Params: {} 2023-10-18T20:39:44.338 DEBUG:boto:Token: None 2023-10-18T20:39:44.339 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.339 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g2042hDx9tBLNqJJTAN2XKYDHEk= 2023-10-18T20:39:44.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g2042hDx9tBLNqJJTAN2XKYDHEk=', 'Content-Length': '0'} 2023-10-18T20:39:44.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d5b9d2ebdeb90e2-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.342 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.343 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.343 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.344 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.344 DEBUG:boto:Method: GET 2023-10-18T20:39:44.344 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.345 DEBUG:boto:Data: 2023-10-18T20:39:44.345 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.346 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.347 DEBUG:boto:Params: {} 2023-10-18T20:39:44.347 DEBUG:boto:Token: None 2023-10-18T20:39:44.348 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:39:44.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g2042hDx9tBLNqJJTAN2XKYDHEk= 2023-10-18T20:39:44.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g2042hDx9tBLNqJJTAN2XKYDHEk=', 'Content-Length': '0'} 2023-10-18T20:39:44.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006810857209119bf0-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.351 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.352 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.352 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.352 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.353 DEBUG:boto:Method: GET 2023-10-18T20:39:44.353 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.354 DEBUG:boto:Data: 2023-10-18T20:39:44.354 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.355 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.355 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.355 DEBUG:boto:Params: {} 2023-10-18T20:39:44.356 DEBUG:boto:Token: None 2023-10-18T20:39:44.356 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.357 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003353d3d2bba0ebf0-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.360 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.361 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.361 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.362 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.362 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.362 DEBUG:boto:Method: GET 2023-10-18T20:39:44.363 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.363 DEBUG:boto:Data: 2023-10-18T20:39:44.364 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.364 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.364 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.365 DEBUG:boto:Params: {} 2023-10-18T20:39:44.365 DEBUG:boto:Token: None 2023-10-18T20:39:44.366 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.366 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fac2c6c62808dc3b-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.370 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.371 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.371 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.372 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.372 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:39:44.372 DEBUG:boto:Data: 2023-10-18T20:39:44.373 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.374 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.374 DEBUG:boto:Params: {} 2023-10-18T20:39:44.375 DEBUG:boto:Token: None 2023-10-18T20:39:44.375 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38 2023-10-18T20:39:44.376 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ukmX8e+Gkqpqz5SwWlrftZoLlKE= 2023-10-18T20:39:44.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ukmX8e+Gkqpqz5SwWlrftZoLlKE=', 'Content-Length': '0'} 2023-10-18T20:39:44.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000414a98958ba5e548-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.378 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.379 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.379 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.380 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:39:44.380 DEBUG:boto:Data: 2023-10-18T20:39:44.381 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.381 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.382 DEBUG:boto:Params: {} 2023-10-18T20:39:44.383 DEBUG:boto:Token: None 2023-10-18T20:39:44.383 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38 2023-10-18T20:39:44.384 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ukmX8e+Gkqpqz5SwWlrftZoLlKE= 2023-10-18T20:39:44.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ukmX8e+Gkqpqz5SwWlrftZoLlKE=', 'Content-Length': '0'} 2023-10-18T20:39:44.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000022313e41151afbf-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:39:44.387 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.387 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.388 DEBUG:boto:Method: GET 2023-10-18T20:39:44.388 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:39:44.389 DEBUG:boto:Data: 2023-10-18T20:39:44.389 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.390 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.390 DEBUG:boto:Params: {} 2023-10-18T20:39:44.391 DEBUG:boto:Token: None 2023-10-18T20:39:44.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38 2023-10-18T20:39:44.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2oe+RNg6EtBSeBucwsf3xldRTRk= 2023-10-18T20:39:44.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2oe+RNg6EtBSeBucwsf3xldRTRk=', 'Content-Length': '0'} 2023-10-18T20:39:44.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045a39428d61adb3b-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.434 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:39:44.435 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:39:44.436 DEBUG:boto:Method: GET 2023-10-18T20:39:44.436 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:39:44.437 DEBUG:boto:Data: 2023-10-18T20:39:44.437 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.438 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.438 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.438 DEBUG:boto:Params: {} 2023-10-18T20:39:44.439 DEBUG:boto:Token: None 2023-10-18T20:39:44.439 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj38 2023-10-18T20:39:44.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2oe+RNg6EtBSeBucwsf3xldRTRk= 2023-10-18T20:39:44.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2oe+RNg6EtBSeBucwsf3xldRTRk=', 'Content-Length': '0'} 2023-10-18T20:39:44.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e54b26d3bcb559-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.444 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.444 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.445 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.446 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.446 DEBUG:boto:Method: GET 2023-10-18T20:39:44.447 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.447 DEBUG:boto:Data: 2023-10-18T20:39:44.448 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.448 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.449 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.449 DEBUG:boto:Params: {} 2023-10-18T20:39:44.450 DEBUG:boto:Token: None 2023-10-18T20:39:44.450 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.451 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f65ad779b2898619-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.455 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.455 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.456 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.456 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.456 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.457 DEBUG:boto:Method: GET 2023-10-18T20:39:44.457 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:39:44.458 DEBUG:boto:Data: 2023-10-18T20:39:44.458 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.459 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.459 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.460 DEBUG:boto:Params: {} 2023-10-18T20:39:44.460 DEBUG:boto:Token: None 2023-10-18T20:39:44.461 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.461 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026788752cc17001a-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.510 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:39:44.512 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.512 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.513 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.513 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.514 DEBUG:boto:Method: GET 2023-10-18T20:39:44.514 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.515 DEBUG:boto:Data: 2023-10-18T20:39:44.515 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.516 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.516 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.517 DEBUG:boto:Params: {} 2023-10-18T20:39:44.517 DEBUG:boto:Token: None 2023-10-18T20:39:44.518 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.518 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0zHXwIUlfSPamgJeNdb1VnQ10d8= 2023-10-18T20:39:44.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0zHXwIUlfSPamgJeNdb1VnQ10d8=', 'Content-Length': '0'} 2023-10-18T20:39:44.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045dd13b752ffa627-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.522 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.522 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.523 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.523 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.523 DEBUG:boto:Method: GET 2023-10-18T20:39:44.524 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.524 DEBUG:boto:Data: 2023-10-18T20:39:44.525 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.526 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.526 DEBUG:boto:Params: {} 2023-10-18T20:39:44.527 DEBUG:boto:Token: None 2023-10-18T20:39:44.527 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.528 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0zHXwIUlfSPamgJeNdb1VnQ10d8= 2023-10-18T20:39:44.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0zHXwIUlfSPamgJeNdb1VnQ10d8=', 'Content-Length': '0'} 2023-10-18T20:39:44.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000625a50c1a570098f-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.531 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.531 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.531 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.532 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.532 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.533 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.533 DEBUG:boto:Data: 2023-10-18T20:39:44.533 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.534 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.535 DEBUG:boto:Params: {} 2023-10-18T20:39:44.535 DEBUG:boto:Token: None 2023-10-18T20:39:44.536 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1s5/TqJtLKKtQbiIdKLVARflN3A= 2023-10-18T20:39:44.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1s5/TqJtLKKtQbiIdKLVARflN3A=', 'Content-Length': '0'} 2023-10-18T20:39:44.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f7ac2d7eb4d300-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.539 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.540 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.541 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.541 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.542 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.542 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.543 DEBUG:boto:Data: 2023-10-18T20:39:44.544 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.544 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.545 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.545 DEBUG:boto:Params: {} 2023-10-18T20:39:44.546 DEBUG:boto:Token: None 2023-10-18T20:39:44.546 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.547 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1s5/TqJtLKKtQbiIdKLVARflN3A= 2023-10-18T20:39:44.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1s5/TqJtLKKtQbiIdKLVARflN3A=', 'Content-Length': '0'} 2023-10-18T20:39:44.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040c00ee5ff25497f-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:39:44.550 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.551 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.551 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.552 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.552 DEBUG:boto:Method: GET 2023-10-18T20:39:44.553 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.553 DEBUG:boto:Data: 2023-10-18T20:39:44.554 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.554 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.555 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.555 DEBUG:boto:Params: {} 2023-10-18T20:39:44.555 DEBUG:boto:Token: None 2023-10-18T20:39:44.556 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.556 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0zHXwIUlfSPamgJeNdb1VnQ10d8= 2023-10-18T20:39:44.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0zHXwIUlfSPamgJeNdb1VnQ10d8=', 'Content-Length': '0'} 2023-10-18T20:39:44.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000442ba8b98bf6fea4-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.560 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.560 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.561 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.562 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.562 DEBUG:boto:Method: GET 2023-10-18T20:39:44.563 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.564 DEBUG:boto:Data: 2023-10-18T20:39:44.565 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.566 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.567 DEBUG:boto:Params: {} 2023-10-18T20:39:44.568 DEBUG:boto:Token: None 2023-10-18T20:39:44.569 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:39:44.570 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0zHXwIUlfSPamgJeNdb1VnQ10d8= 2023-10-18T20:39:44.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0zHXwIUlfSPamgJeNdb1VnQ10d8=', 'Content-Length': '0'} 2023-10-18T20:39:44.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d24a29393c8bbf50-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.575 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.576 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.576 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.577 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.577 DEBUG:boto:Method: GET 2023-10-18T20:39:44.578 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.578 DEBUG:boto:Data: 2023-10-18T20:39:44.578 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.579 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.579 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.580 DEBUG:boto:Params: {} 2023-10-18T20:39:44.580 DEBUG:boto:Token: None 2023-10-18T20:39:44.581 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036a4bd47187ac6de-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.585 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.586 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.586 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.587 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.587 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.587 DEBUG:boto:Method: GET 2023-10-18T20:39:44.588 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.588 DEBUG:boto:Data: 2023-10-18T20:39:44.589 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.589 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.590 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.590 DEBUG:boto:Params: {} 2023-10-18T20:39:44.591 DEBUG:boto:Token: None 2023-10-18T20:39:44.591 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.592 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f0a3f4457570ad9-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.596 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.597 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.597 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.598 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.598 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:39:44.599 DEBUG:boto:Data: 2023-10-18T20:39:44.599 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.599 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.600 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.600 DEBUG:boto:Params: {} 2023-10-18T20:39:44.601 DEBUG:boto:Token: None 2023-10-18T20:39:44.601 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44 2023-10-18T20:39:44.602 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9Jm4RWBtfmR/tMaibHyDNr4f9PI= 2023-10-18T20:39:44.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9Jm4RWBtfmR/tMaibHyDNr4f9PI=', 'Content-Length': '0'} 2023-10-18T20:39:44.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca345ee13fd0a88-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.605 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.605 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.605 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.606 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:39:44.606 DEBUG:boto:Data: 2023-10-18T20:39:44.607 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.607 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.607 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.608 DEBUG:boto:Params: {} 2023-10-18T20:39:44.608 DEBUG:boto:Token: None 2023-10-18T20:39:44.609 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44 2023-10-18T20:39:44.609 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9Jm4RWBtfmR/tMaibHyDNr4f9PI= 2023-10-18T20:39:44.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9Jm4RWBtfmR/tMaibHyDNr4f9PI=', 'Content-Length': '0'} 2023-10-18T20:39:44.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c511012f2a88bf36-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:39:44.613 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.613 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.614 DEBUG:boto:Method: GET 2023-10-18T20:39:44.614 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:39:44.614 DEBUG:boto:Data: 2023-10-18T20:39:44.615 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.615 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.616 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.616 DEBUG:boto:Params: {} 2023-10-18T20:39:44.617 DEBUG:boto:Token: None 2023-10-18T20:39:44.617 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44 2023-10-18T20:39:44.618 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uhvbcaIjMbhuz+46FJQTjP1e/Eo= 2023-10-18T20:39:44.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uhvbcaIjMbhuz+46FJQTjP1e/Eo=', 'Content-Length': '0'} 2023-10-18T20:39:44.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b383541297fe8a2b-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.621 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:39:44.621 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:39:44.622 DEBUG:boto:Method: GET 2023-10-18T20:39:44.622 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:39:44.623 DEBUG:boto:Data: 2023-10-18T20:39:44.623 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.623 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.624 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.624 DEBUG:boto:Params: {} 2023-10-18T20:39:44.625 DEBUG:boto:Token: None 2023-10-18T20:39:44.625 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj44 2023-10-18T20:39:44.626 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uhvbcaIjMbhuz+46FJQTjP1e/Eo= 2023-10-18T20:39:44.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uhvbcaIjMbhuz+46FJQTjP1e/Eo=', 'Content-Length': '0'} 2023-10-18T20:39:44.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d1f0948ecb9bbae-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.674 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.675 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.676 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.676 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.677 DEBUG:boto:Method: GET 2023-10-18T20:39:44.677 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.677 DEBUG:boto:Data: 2023-10-18T20:39:44.678 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.678 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.679 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.679 DEBUG:boto:Params: {} 2023-10-18T20:39:44.680 DEBUG:boto:Token: None 2023-10-18T20:39:44.680 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.681 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d18374bad63c6b-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.684 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.685 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.686 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.687 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.688 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.689 DEBUG:boto:Method: GET 2023-10-18T20:39:44.690 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:39:44.690 DEBUG:boto:Data: 2023-10-18T20:39:44.691 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.692 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.693 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.694 DEBUG:boto:Params: {} 2023-10-18T20:39:44.695 DEBUG:boto:Token: None 2023-10-18T20:39:44.695 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.696 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000528dd71c72151f03-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.700 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:39:44.702 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.702 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.703 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.703 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.703 DEBUG:boto:Method: GET 2023-10-18T20:39:44.704 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.704 DEBUG:boto:Data: 2023-10-18T20:39:44.705 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.705 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.705 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.706 DEBUG:boto:Params: {} 2023-10-18T20:39:44.707 DEBUG:boto:Token: None 2023-10-18T20:39:44.707 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.708 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WRMOLNZWMwobxRmM7c0qxNj49wE= 2023-10-18T20:39:44.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WRMOLNZWMwobxRmM7c0qxNj49wE=', 'Content-Length': '0'} 2023-10-18T20:39:44.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037875235cc3021dd-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.758 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.759 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.760 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.760 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.761 DEBUG:boto:Method: GET 2023-10-18T20:39:44.761 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.762 DEBUG:boto:Data: 2023-10-18T20:39:44.762 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.763 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.763 DEBUG:boto:Params: {} 2023-10-18T20:39:44.764 DEBUG:boto:Token: None 2023-10-18T20:39:44.764 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.765 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WRMOLNZWMwobxRmM7c0qxNj49wE= 2023-10-18T20:39:44.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WRMOLNZWMwobxRmM7c0qxNj49wE=', 'Content-Length': '0'} 2023-10-18T20:39:44.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c47cb75e9de90c4-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.768 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.769 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.769 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.770 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.771 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.771 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.772 DEBUG:boto:Data: 2023-10-18T20:39:44.773 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.774 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.777 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.778 DEBUG:boto:Params: {} 2023-10-18T20:39:44.779 DEBUG:boto:Token: None 2023-10-18T20:39:44.779 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VeQzru1eOAL/fYSpR+2DE/D9OTc= 2023-10-18T20:39:44.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VeQzru1eOAL/fYSpR+2DE/D9OTc=', 'Content-Length': '0'} 2023-10-18T20:39:44.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000458a8768fd572d47-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.784 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.784 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.784 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.785 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.785 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.786 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.786 DEBUG:boto:Data: 2023-10-18T20:39:44.787 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.788 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.788 DEBUG:boto:Params: {} 2023-10-18T20:39:44.788 DEBUG:boto:Token: None 2023-10-18T20:39:44.789 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.790 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VeQzru1eOAL/fYSpR+2DE/D9OTc= 2023-10-18T20:39:44.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VeQzru1eOAL/fYSpR+2DE/D9OTc=', 'Content-Length': '0'} 2023-10-18T20:39:44.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000147d0132e3bfc868-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:39:44.793 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.794 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.794 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.794 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.795 DEBUG:boto:Method: GET 2023-10-18T20:39:44.795 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.796 DEBUG:boto:Data: 2023-10-18T20:39:44.796 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.797 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.797 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.797 DEBUG:boto:Params: {} 2023-10-18T20:39:44.798 DEBUG:boto:Token: None 2023-10-18T20:39:44.798 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WRMOLNZWMwobxRmM7c0qxNj49wE= 2023-10-18T20:39:44.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WRMOLNZWMwobxRmM7c0qxNj49wE=', 'Content-Length': '0'} 2023-10-18T20:39:44.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bae2726c660565bf-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.802 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.802 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.803 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.803 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.804 DEBUG:boto:Method: GET 2023-10-18T20:39:44.804 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.805 DEBUG:boto:Data: 2023-10-18T20:39:44.805 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.806 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.806 DEBUG:boto:Params: {} 2023-10-18T20:39:44.807 DEBUG:boto:Token: None 2023-10-18T20:39:44.807 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:39:44.808 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WRMOLNZWMwobxRmM7c0qxNj49wE= 2023-10-18T20:39:44.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WRMOLNZWMwobxRmM7c0qxNj49wE=', 'Content-Length': '0'} 2023-10-18T20:39:44.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005655ed6ded574af3-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.811 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.811 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.812 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.812 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.812 DEBUG:boto:Method: GET 2023-10-18T20:39:44.813 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.813 DEBUG:boto:Data: 2023-10-18T20:39:44.814 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.815 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.815 DEBUG:boto:Params: {} 2023-10-18T20:39:44.815 DEBUG:boto:Token: None 2023-10-18T20:39:44.816 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcd2c62db05a464b-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.820 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.821 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.821 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.822 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.822 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.822 DEBUG:boto:Method: GET 2023-10-18T20:39:44.823 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.823 DEBUG:boto:Data: 2023-10-18T20:39:44.824 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.824 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.824 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.825 DEBUG:boto:Params: {} 2023-10-18T20:39:44.825 DEBUG:boto:Token: None 2023-10-18T20:39:44.826 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.826 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076131ece6a4c548c-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.830 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.831 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.831 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.832 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.832 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:39:44.833 DEBUG:boto:Data: 2023-10-18T20:39:44.833 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.833 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.834 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.834 DEBUG:boto:Params: {} 2023-10-18T20:39:44.835 DEBUG:boto:Token: None 2023-10-18T20:39:44.835 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46 2023-10-18T20:39:44.836 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:30EJZ7Smu1m/2vsh79E1JRP+F28= 2023-10-18T20:39:44.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:30EJZ7Smu1m/2vsh79E1JRP+F28=', 'Content-Length': '0'} 2023-10-18T20:39:44.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b43fedc051270139-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.839 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.839 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.840 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.840 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:39:44.840 DEBUG:boto:Data: 2023-10-18T20:39:44.841 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.841 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.842 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.842 DEBUG:boto:Params: {} 2023-10-18T20:39:44.842 DEBUG:boto:Token: None 2023-10-18T20:39:44.843 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46 2023-10-18T20:39:44.843 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:30EJZ7Smu1m/2vsh79E1JRP+F28= 2023-10-18T20:39:44.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:30EJZ7Smu1m/2vsh79E1JRP+F28=', 'Content-Length': '0'} 2023-10-18T20:39:44.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000809258de8dc9db3d-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:39:44.847 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.847 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.848 DEBUG:boto:Method: GET 2023-10-18T20:39:44.848 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:39:44.849 DEBUG:boto:Data: 2023-10-18T20:39:44.849 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.849 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.850 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.850 DEBUG:boto:Params: {} 2023-10-18T20:39:44.851 DEBUG:boto:Token: None 2023-10-18T20:39:44.851 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46 2023-10-18T20:39:44.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lQM54pWxnZQ6iccPj6hD//qVsQs= 2023-10-18T20:39:44.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lQM54pWxnZQ6iccPj6hD//qVsQs=', 'Content-Length': '0'} 2023-10-18T20:39:44.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001897b2d53c5d4d17-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.855 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:39:44.855 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:39:44.856 DEBUG:boto:Method: GET 2023-10-18T20:39:44.856 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:39:44.857 DEBUG:boto:Data: 2023-10-18T20:39:44.857 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.857 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.858 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.858 DEBUG:boto:Params: {} 2023-10-18T20:39:44.859 DEBUG:boto:Token: None 2023-10-18T20:39:44.859 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj46 2023-10-18T20:39:44.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lQM54pWxnZQ6iccPj6hD//qVsQs= 2023-10-18T20:39:44.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lQM54pWxnZQ6iccPj6hD//qVsQs=', 'Content-Length': '0'} 2023-10-18T20:39:44.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc646b4fd7f4ca26-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.863 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.863 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.864 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.864 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.865 DEBUG:boto:Method: GET 2023-10-18T20:39:44.865 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.865 DEBUG:boto:Data: 2023-10-18T20:39:44.866 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.866 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.867 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.867 DEBUG:boto:Params: {} 2023-10-18T20:39:44.868 DEBUG:boto:Token: None 2023-10-18T20:39:44.868 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.869 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f263515ce365e6e-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.918 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.919 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.920 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.920 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.921 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.921 DEBUG:boto:Method: GET 2023-10-18T20:39:44.922 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:39:44.922 DEBUG:boto:Data: 2023-10-18T20:39:44.923 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.924 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.924 DEBUG:boto:Params: {} 2023-10-18T20:39:44.925 DEBUG:boto:Token: None 2023-10-18T20:39:44.925 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010bf203224b93747-0065304290-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.930 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:39:44.931 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:44.932 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:44.932 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.933 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.933 DEBUG:boto:Method: GET 2023-10-18T20:39:44.934 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.934 DEBUG:boto:Data: 2023-10-18T20:39:44.935 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.935 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.936 DEBUG:boto:Params: {} 2023-10-18T20:39:44.936 DEBUG:boto:Token: None 2023-10-18T20:39:44.937 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.937 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ErrfWI833AeVdZD32Y+V5oPFscU= 2023-10-18T20:39:44.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ErrfWI833AeVdZD32Y+V5oPFscU=', 'Content-Length': '0'} 2023-10-18T20:39:44.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1d5f70e656c801-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.940 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:44.941 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:44.941 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.942 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.942 DEBUG:boto:Method: GET 2023-10-18T20:39:44.943 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.943 DEBUG:boto:Data: 2023-10-18T20:39:44.944 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.944 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.945 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.945 DEBUG:boto:Params: {} 2023-10-18T20:39:44.945 DEBUG:boto:Token: None 2023-10-18T20:39:44.946 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.946 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ErrfWI833AeVdZD32Y+V5oPFscU= 2023-10-18T20:39:44.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ErrfWI833AeVdZD32Y+V5oPFscU=', 'Content-Length': '0'} 2023-10-18T20:39:44.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007850df302998192d-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.950 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:44.950 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:44.950 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.951 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.951 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.952 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.952 DEBUG:boto:Data: 2023-10-18T20:39:44.952 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.953 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.953 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.954 DEBUG:boto:Params: {} 2023-10-18T20:39:44.954 DEBUG:boto:Token: None 2023-10-18T20:39:44.955 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.955 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CX9Yh7QB2LeR3U8XcAat3npgu80= 2023-10-18T20:39:44.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CX9Yh7QB2LeR3U8XcAat3npgu80=', 'Content-Length': '0'} 2023-10-18T20:39:44.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ea372623ec83b85-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.958 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:44.958 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:44.959 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.959 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.960 DEBUG:boto:Method: HEAD 2023-10-18T20:39:44.960 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.961 DEBUG:boto:Data: 2023-10-18T20:39:44.961 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.962 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.962 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.962 DEBUG:boto:Params: {} 2023-10-18T20:39:44.963 DEBUG:boto:Token: None 2023-10-18T20:39:44.963 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.964 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CX9Yh7QB2LeR3U8XcAat3npgu80= 2023-10-18T20:39:44.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CX9Yh7QB2LeR3U8XcAat3npgu80=', 'Content-Length': '0'} 2023-10-18T20:39:44.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018d10ba62563ecbc-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:39:44.967 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:44.968 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:44.968 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.969 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.969 DEBUG:boto:Method: GET 2023-10-18T20:39:44.970 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.970 DEBUG:boto:Data: 2023-10-18T20:39:44.971 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.971 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.971 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.972 DEBUG:boto:Params: {} 2023-10-18T20:39:44.972 DEBUG:boto:Token: None 2023-10-18T20:39:44.973 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.973 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ErrfWI833AeVdZD32Y+V5oPFscU= 2023-10-18T20:39:44.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ErrfWI833AeVdZD32Y+V5oPFscU=', 'Content-Length': '0'} 2023-10-18T20:39:44.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078d7d2805f1c5ed7-0065304290-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.977 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:44.977 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:44.978 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.978 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.979 DEBUG:boto:Method: GET 2023-10-18T20:39:44.982 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.983 DEBUG:boto:Data: 2023-10-18T20:39:44.983 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.983 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:44.984 DEBUG:boto:Port: 8000 2023-10-18T20:39:44.984 DEBUG:boto:Params: {} 2023-10-18T20:39:44.985 DEBUG:boto:Token: None 2023-10-18T20:39:44.986 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:39:44.986 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ErrfWI833AeVdZD32Y+V5oPFscU= 2023-10-18T20:39:44.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ErrfWI833AeVdZD32Y+V5oPFscU=', 'Content-Length': '0'} 2023-10-18T20:39:44.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091af596e5775c66b-0065304290-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.989 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.990 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:44.990 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:39:44.990 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:39:44.991 DEBUG:boto:Method: GET 2023-10-18T20:39:44.991 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:39:44.992 DEBUG:boto:Data: 2023-10-18T20:39:44.992 DEBUG:boto:Headers: {} 2023-10-18T20:39:44.993 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:44.993 DEBUG:boto:Port: 8001 2023-10-18T20:39:44.993 DEBUG:boto:Params: {} 2023-10-18T20:39:44.994 DEBUG:boto:Token: None 2023-10-18T20:39:44.994 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:44 GMT /bofrnf-59/ 2023-10-18T20:39:44.995 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE= 2023-10-18T20:39:44.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bhKGrCdKwyPdCDEOshjwPbMqKNE=', 'Content-Length': '0'} 2023-10-18T20:39:44.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e806437fc65d817c-0065304290-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:44.998 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:44.999 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:44.999 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.000 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:39:45.000 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:39:45.001 DEBUG:boto:Method: GET 2023-10-18T20:39:45.001 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:39:45.002 DEBUG:boto:Data: 2023-10-18T20:39:45.002 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.003 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.003 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.003 DEBUG:boto:Params: {} 2023-10-18T20:39:45.004 DEBUG:boto:Token: None 2023-10-18T20:39:45.004 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.005 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089b2e330d7f9d64d-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.009 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.009 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:45.010 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:45.010 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.011 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:39:45.011 DEBUG:boto:Data: 2023-10-18T20:39:45.011 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.012 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.012 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.013 DEBUG:boto:Params: {} 2023-10-18T20:39:45.013 DEBUG:boto:Token: None 2023-10-18T20:39:45.014 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj51 2023-10-18T20:39:45.014 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z5mBjhUhSQw6sUv09tYm6RK0x7c= 2023-10-18T20:39:45.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z5mBjhUhSQw6sUv09tYm6RK0x7c=', 'Content-Length': '0'} 2023-10-18T20:39:45.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033945fdc55f24371-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.017 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:45.018 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:45.018 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.019 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:39:45.019 DEBUG:boto:Data: 2023-10-18T20:39:45.019 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.020 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.020 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.021 DEBUG:boto:Params: {} 2023-10-18T20:39:45.021 DEBUG:boto:Token: None 2023-10-18T20:39:45.022 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj51 2023-10-18T20:39:45.022 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z5mBjhUhSQw6sUv09tYm6RK0x7c= 2023-10-18T20:39:45.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z5mBjhUhSQw6sUv09tYm6RK0x7c=', 'Content-Length': '0'} 2023-10-18T20:39:45.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000894ea8a2f10b457-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:39:45.025 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:45.026 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:45.026 DEBUG:boto:Method: GET 2023-10-18T20:39:45.027 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:39:45.027 DEBUG:boto:Data: 2023-10-18T20:39:45.027 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.028 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.028 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.029 DEBUG:boto:Params: {} 2023-10-18T20:39:45.029 DEBUG:boto:Token: None 2023-10-18T20:39:45.030 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj51 2023-10-18T20:39:45.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X+kvZyECYQKhuDGoJHsWIpwgWQw= 2023-10-18T20:39:45.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X+kvZyECYQKhuDGoJHsWIpwgWQw=', 'Content-Length': '0'} 2023-10-18T20:39:45.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa0bf44a657a8fa7-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.033 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:39:45.034 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:39:45.034 DEBUG:boto:Method: GET 2023-10-18T20:39:45.034 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:39:45.035 DEBUG:boto:Data: 2023-10-18T20:39:45.035 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.036 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.036 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.037 DEBUG:boto:Params: {} 2023-10-18T20:39:45.037 DEBUG:boto:Token: None 2023-10-18T20:39:45.037 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj51 2023-10-18T20:39:45.038 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X+kvZyECYQKhuDGoJHsWIpwgWQw= 2023-10-18T20:39:45.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X+kvZyECYQKhuDGoJHsWIpwgWQw=', 'Content-Length': '0'} 2023-10-18T20:39:45.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d29d796d76d0366-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.082 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.083 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.083 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:39:45.084 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:39:45.084 DEBUG:boto:Method: GET 2023-10-18T20:39:45.085 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:39:45.085 DEBUG:boto:Data: 2023-10-18T20:39:45.086 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.086 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.086 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.087 DEBUG:boto:Params: {} 2023-10-18T20:39:45.087 DEBUG:boto:Token: None 2023-10-18T20:39:45.088 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.088 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f6dc79ae2e0b7c0-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.092 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.093 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.093 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.093 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:39:45.094 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:39:45.094 DEBUG:boto:Method: GET 2023-10-18T20:39:45.095 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:39:45.095 DEBUG:boto:Data: 2023-10-18T20:39:45.096 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.096 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.096 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.097 DEBUG:boto:Params: {} 2023-10-18T20:39:45.097 DEBUG:boto:Token: None 2023-10-18T20:39:45.098 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.098 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000040b73368744770d-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.102 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:39:45.103 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.104 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.104 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.105 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.105 DEBUG:boto:Method: GET 2023-10-18T20:39:45.106 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.106 DEBUG:boto:Data: 2023-10-18T20:39:45.106 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.107 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.107 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.108 DEBUG:boto:Params: {} 2023-10-18T20:39:45.108 DEBUG:boto:Token: None 2023-10-18T20:39:45.109 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.109 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w5unHxzqP8U3cwZRMVPWl5ZxZTw= 2023-10-18T20:39:45.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w5unHxzqP8U3cwZRMVPWl5ZxZTw=', 'Content-Length': '0'} 2023-10-18T20:39:45.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0154181c6ae3dfa-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.112 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.113 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.113 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.114 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.114 DEBUG:boto:Method: GET 2023-10-18T20:39:45.115 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.115 DEBUG:boto:Data: 2023-10-18T20:39:45.115 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.116 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.116 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.117 DEBUG:boto:Params: {} 2023-10-18T20:39:45.117 DEBUG:boto:Token: None 2023-10-18T20:39:45.118 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.118 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w5unHxzqP8U3cwZRMVPWl5ZxZTw= 2023-10-18T20:39:45.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w5unHxzqP8U3cwZRMVPWl5ZxZTw=', 'Content-Length': '0'} 2023-10-18T20:39:45.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000402e571fc3fb9608-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.121 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.122 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.122 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.122 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.123 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.123 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.124 DEBUG:boto:Data: 2023-10-18T20:39:45.124 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.124 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.125 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.125 DEBUG:boto:Params: {} 2023-10-18T20:39:45.126 DEBUG:boto:Token: None 2023-10-18T20:39:45.126 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.127 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YV6Rrdx70ClkQByVmvZLsS2IlN0= 2023-10-18T20:39:45.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YV6Rrdx70ClkQByVmvZLsS2IlN0=', 'Content-Length': '0'} 2023-10-18T20:39:45.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e780f39be8182db-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.129 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.130 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.130 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.130 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.131 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.131 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.132 DEBUG:boto:Data: 2023-10-18T20:39:45.132 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.132 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.133 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.133 DEBUG:boto:Params: {} 2023-10-18T20:39:45.134 DEBUG:boto:Token: None 2023-10-18T20:39:45.134 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YV6Rrdx70ClkQByVmvZLsS2IlN0= 2023-10-18T20:39:45.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YV6Rrdx70ClkQByVmvZLsS2IlN0=', 'Content-Length': '0'} 2023-10-18T20:39:45.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc05d71e16c83c7e-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:39:45.138 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.138 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.139 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.139 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.139 DEBUG:boto:Method: GET 2023-10-18T20:39:45.140 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.140 DEBUG:boto:Data: 2023-10-18T20:39:45.141 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.141 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.142 DEBUG:boto:Params: {} 2023-10-18T20:39:45.142 DEBUG:boto:Token: None 2023-10-18T20:39:45.143 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.143 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w5unHxzqP8U3cwZRMVPWl5ZxZTw= 2023-10-18T20:39:45.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w5unHxzqP8U3cwZRMVPWl5ZxZTw=', 'Content-Length': '0'} 2023-10-18T20:39:45.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c961532903b98b7-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.146 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.147 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.147 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.148 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.148 DEBUG:boto:Method: GET 2023-10-18T20:39:45.149 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.149 DEBUG:boto:Data: 2023-10-18T20:39:45.149 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.150 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.151 DEBUG:boto:Params: {} 2023-10-18T20:39:45.151 DEBUG:boto:Token: None 2023-10-18T20:39:45.152 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:39:45.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w5unHxzqP8U3cwZRMVPWl5ZxZTw= 2023-10-18T20:39:45.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w5unHxzqP8U3cwZRMVPWl5ZxZTw=', 'Content-Length': '0'} 2023-10-18T20:39:45.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a374ca978370cb25-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.155 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.156 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.156 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.156 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.157 DEBUG:boto:Method: GET 2023-10-18T20:39:45.157 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.158 DEBUG:boto:Data: 2023-10-18T20:39:45.158 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.159 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.159 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.159 DEBUG:boto:Params: {} 2023-10-18T20:39:45.160 DEBUG:boto:Token: None 2023-10-18T20:39:45.160 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009558835e76d1b98c-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.164 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.165 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.165 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.166 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.166 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.167 DEBUG:boto:Method: GET 2023-10-18T20:39:45.167 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.167 DEBUG:boto:Data: 2023-10-18T20:39:45.168 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.168 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.169 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.169 DEBUG:boto:Params: {} 2023-10-18T20:39:45.169 DEBUG:boto:Token: None 2023-10-18T20:39:45.170 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.170 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000916625964cc541b0-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.174 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.175 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.176 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.176 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.176 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:39:45.177 DEBUG:boto:Data: 2023-10-18T20:39:45.177 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.178 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.178 DEBUG:boto:Params: {} 2023-10-18T20:39:45.179 DEBUG:boto:Token: None 2023-10-18T20:39:45.179 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55 2023-10-18T20:39:45.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+dWWa2OCDBqdlwLf/IxY/ztFA7c= 2023-10-18T20:39:45.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+dWWa2OCDBqdlwLf/IxY/ztFA7c=', 'Content-Length': '0'} 2023-10-18T20:39:45.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008725e0da730524d4-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.182 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.183 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.183 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.184 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:39:45.184 DEBUG:boto:Data: 2023-10-18T20:39:45.184 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.185 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.186 DEBUG:boto:Params: {} 2023-10-18T20:39:45.189 DEBUG:boto:Token: None 2023-10-18T20:39:45.190 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55 2023-10-18T20:39:45.190 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+dWWa2OCDBqdlwLf/IxY/ztFA7c= 2023-10-18T20:39:45.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+dWWa2OCDBqdlwLf/IxY/ztFA7c=', 'Content-Length': '0'} 2023-10-18T20:39:45.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f25b9decf07463f-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:39:45.194 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.194 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.195 DEBUG:boto:Method: GET 2023-10-18T20:39:45.195 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:39:45.195 DEBUG:boto:Data: 2023-10-18T20:39:45.196 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.196 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.197 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.197 DEBUG:boto:Params: {} 2023-10-18T20:39:45.198 DEBUG:boto:Token: None 2023-10-18T20:39:45.198 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55 2023-10-18T20:39:45.198 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sXBK2mTsMM8Dv46r3oLF+9ldRR8= 2023-10-18T20:39:45.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sXBK2mTsMM8Dv46r3oLF+9ldRR8=', 'Content-Length': '0'} 2023-10-18T20:39:45.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a777122c56e071a-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.246 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:39:45.247 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:39:45.247 DEBUG:boto:Method: GET 2023-10-18T20:39:45.248 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:39:45.248 DEBUG:boto:Data: 2023-10-18T20:39:45.249 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.249 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.250 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.250 DEBUG:boto:Params: {} 2023-10-18T20:39:45.251 DEBUG:boto:Token: None 2023-10-18T20:39:45.251 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj55 2023-10-18T20:39:45.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sXBK2mTsMM8Dv46r3oLF+9ldRR8= 2023-10-18T20:39:45.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sXBK2mTsMM8Dv46r3oLF+9ldRR8=', 'Content-Length': '0'} 2023-10-18T20:39:45.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092ee27c7d7353a11-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.255 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.255 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.256 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.256 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.256 DEBUG:boto:Method: GET 2023-10-18T20:39:45.257 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.257 DEBUG:boto:Data: 2023-10-18T20:39:45.258 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.258 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.259 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.259 DEBUG:boto:Params: {} 2023-10-18T20:39:45.260 DEBUG:boto:Token: None 2023-10-18T20:39:45.260 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.261 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000777c0543808cd2d0-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.264 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.265 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.265 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.266 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.266 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.266 DEBUG:boto:Method: GET 2023-10-18T20:39:45.267 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:39:45.267 DEBUG:boto:Data: 2023-10-18T20:39:45.268 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.268 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.269 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.269 DEBUG:boto:Params: {} 2023-10-18T20:39:45.269 DEBUG:boto:Token: None 2023-10-18T20:39:45.270 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.270 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b37606fcc5b44167-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.314 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:39:45.316 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.316 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.316 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.317 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.317 DEBUG:boto:Method: GET 2023-10-18T20:39:45.318 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.318 DEBUG:boto:Data: 2023-10-18T20:39:45.319 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.319 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.320 DEBUG:boto:Params: {} 2023-10-18T20:39:45.320 DEBUG:boto:Token: None 2023-10-18T20:39:45.321 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X2D15zzMrpVXlb789jnIzsfQDuc= 2023-10-18T20:39:45.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X2D15zzMrpVXlb789jnIzsfQDuc=', 'Content-Length': '0'} 2023-10-18T20:39:45.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000788f4c4d2d797acf-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.324 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.325 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.325 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.326 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.326 DEBUG:boto:Method: GET 2023-10-18T20:39:45.326 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.327 DEBUG:boto:Data: 2023-10-18T20:39:45.327 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.328 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.329 DEBUG:boto:Params: {} 2023-10-18T20:39:45.329 DEBUG:boto:Token: None 2023-10-18T20:39:45.330 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X2D15zzMrpVXlb789jnIzsfQDuc= 2023-10-18T20:39:45.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X2D15zzMrpVXlb789jnIzsfQDuc=', 'Content-Length': '0'} 2023-10-18T20:39:45.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc218b8b0c1d4557-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.333 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.334 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.334 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.334 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.335 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.335 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.336 DEBUG:boto:Data: 2023-10-18T20:39:45.336 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.336 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.337 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.337 DEBUG:boto:Params: {} 2023-10-18T20:39:45.338 DEBUG:boto:Token: None 2023-10-18T20:39:45.338 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.339 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dqsBEBhhxmpCCnulAfKDs5hGcs0= 2023-10-18T20:39:45.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dqsBEBhhxmpCCnulAfKDs5hGcs0=', 'Content-Length': '0'} 2023-10-18T20:39:45.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056984df677ed0c2c-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.341 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.342 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.342 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.342 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.343 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.343 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.344 DEBUG:boto:Data: 2023-10-18T20:39:45.344 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.344 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.345 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.345 DEBUG:boto:Params: {} 2023-10-18T20:39:45.346 DEBUG:boto:Token: None 2023-10-18T20:39:45.346 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.347 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dqsBEBhhxmpCCnulAfKDs5hGcs0= 2023-10-18T20:39:45.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dqsBEBhhxmpCCnulAfKDs5hGcs0=', 'Content-Length': '0'} 2023-10-18T20:39:45.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041b2e1ec07525c34-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:39:45.350 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.350 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.351 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.351 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.352 DEBUG:boto:Method: GET 2023-10-18T20:39:45.352 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.352 DEBUG:boto:Data: 2023-10-18T20:39:45.353 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.353 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.354 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.354 DEBUG:boto:Params: {} 2023-10-18T20:39:45.355 DEBUG:boto:Token: None 2023-10-18T20:39:45.355 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X2D15zzMrpVXlb789jnIzsfQDuc= 2023-10-18T20:39:45.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X2D15zzMrpVXlb789jnIzsfQDuc=', 'Content-Length': '0'} 2023-10-18T20:39:45.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aaa9bd3e30ed3df-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.358 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.359 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.359 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.360 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.360 DEBUG:boto:Method: GET 2023-10-18T20:39:45.360 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.361 DEBUG:boto:Data: 2023-10-18T20:39:45.361 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.362 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.362 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.363 DEBUG:boto:Params: {} 2023-10-18T20:39:45.363 DEBUG:boto:Token: None 2023-10-18T20:39:45.364 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:39:45.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X2D15zzMrpVXlb789jnIzsfQDuc= 2023-10-18T20:39:45.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X2D15zzMrpVXlb789jnIzsfQDuc=', 'Content-Length': '0'} 2023-10-18T20:39:45.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f774aa90bb0550e0-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.367 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.368 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.368 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.368 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.369 DEBUG:boto:Method: GET 2023-10-18T20:39:45.369 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.370 DEBUG:boto:Data: 2023-10-18T20:39:45.370 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.371 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.371 DEBUG:boto:Params: {} 2023-10-18T20:39:45.372 DEBUG:boto:Token: None 2023-10-18T20:39:45.372 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047b456dd02958307-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.376 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.377 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.377 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.378 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.378 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.378 DEBUG:boto:Method: GET 2023-10-18T20:39:45.379 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.379 DEBUG:boto:Data: 2023-10-18T20:39:45.380 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.380 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.381 DEBUG:boto:Params: {} 2023-10-18T20:39:45.381 DEBUG:boto:Token: None 2023-10-18T20:39:45.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057a1ffb4baaee9ce-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.386 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.387 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.387 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.388 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.388 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:39:45.389 DEBUG:boto:Data: 2023-10-18T20:39:45.389 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.390 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.390 DEBUG:boto:Params: {} 2023-10-18T20:39:45.393 DEBUG:boto:Token: None 2023-10-18T20:39:45.394 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57 2023-10-18T20:39:45.394 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iaUVkX6J1J0HMK5ZArfdKL/uK34= 2023-10-18T20:39:45.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iaUVkX6J1J0HMK5ZArfdKL/uK34=', 'Content-Length': '0'} 2023-10-18T20:39:45.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d33c668a77f1ecb2-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.397 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.397 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.398 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.398 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:39:45.398 DEBUG:boto:Data: 2023-10-18T20:39:45.399 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.400 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.400 DEBUG:boto:Params: {} 2023-10-18T20:39:45.400 DEBUG:boto:Token: None 2023-10-18T20:39:45.401 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57 2023-10-18T20:39:45.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iaUVkX6J1J0HMK5ZArfdKL/uK34= 2023-10-18T20:39:45.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iaUVkX6J1J0HMK5ZArfdKL/uK34=', 'Content-Length': '0'} 2023-10-18T20:39:45.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f24d53329bf0aec5-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:39:45.405 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.405 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.406 DEBUG:boto:Method: GET 2023-10-18T20:39:45.406 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:39:45.407 DEBUG:boto:Data: 2023-10-18T20:39:45.407 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.408 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.408 DEBUG:boto:Params: {} 2023-10-18T20:39:45.409 DEBUG:boto:Token: None 2023-10-18T20:39:45.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57 2023-10-18T20:39:45.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zn/nNfVEBpwMYafGJCRKA85q7xg= 2023-10-18T20:39:45.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zn/nNfVEBpwMYafGJCRKA85q7xg=', 'Content-Length': '0'} 2023-10-18T20:39:45.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f09ace28c459b1c-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.413 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:39:45.413 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:39:45.414 DEBUG:boto:Method: GET 2023-10-18T20:39:45.414 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:39:45.415 DEBUG:boto:Data: 2023-10-18T20:39:45.415 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.416 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.416 DEBUG:boto:Params: {} 2023-10-18T20:39:45.417 DEBUG:boto:Token: None 2023-10-18T20:39:45.417 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj57 2023-10-18T20:39:45.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Zn/nNfVEBpwMYafGJCRKA85q7xg= 2023-10-18T20:39:45.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Zn/nNfVEBpwMYafGJCRKA85q7xg=', 'Content-Length': '0'} 2023-10-18T20:39:45.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1de1609ef7bee77-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.421 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.421 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.421 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.422 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.422 DEBUG:boto:Method: GET 2023-10-18T20:39:45.423 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.423 DEBUG:boto:Data: 2023-10-18T20:39:45.424 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.424 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.424 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.425 DEBUG:boto:Params: {} 2023-10-18T20:39:45.425 DEBUG:boto:Token: None 2023-10-18T20:39:45.426 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.426 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbc2ab92a4154366-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.429 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.430 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.430 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.431 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.431 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.432 DEBUG:boto:Method: GET 2023-10-18T20:39:45.432 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:39:45.433 DEBUG:boto:Data: 2023-10-18T20:39:45.433 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.434 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.434 DEBUG:boto:Params: {} 2023-10-18T20:39:45.435 DEBUG:boto:Token: None 2023-10-18T20:39:45.435 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.436 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009efc2cf8ffead998-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.439 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:39:45.441 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.441 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.441 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.442 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.442 DEBUG:boto:Method: GET 2023-10-18T20:39:45.443 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.443 DEBUG:boto:Data: 2023-10-18T20:39:45.443 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.444 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.444 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.445 DEBUG:boto:Params: {} 2023-10-18T20:39:45.445 DEBUG:boto:Token: None 2023-10-18T20:39:45.446 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.446 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0O5crj9QbApDIf9hvipHxU2ki/k= 2023-10-18T20:39:45.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0O5crj9QbApDIf9hvipHxU2ki/k=', 'Content-Length': '0'} 2023-10-18T20:39:45.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000721e11e65fdf748-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.449 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.450 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.450 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.451 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.451 DEBUG:boto:Method: GET 2023-10-18T20:39:45.452 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.452 DEBUG:boto:Data: 2023-10-18T20:39:45.452 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.453 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.454 DEBUG:boto:Params: {} 2023-10-18T20:39:45.454 DEBUG:boto:Token: None 2023-10-18T20:39:45.455 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0O5crj9QbApDIf9hvipHxU2ki/k= 2023-10-18T20:39:45.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0O5crj9QbApDIf9hvipHxU2ki/k=', 'Content-Length': '0'} 2023-10-18T20:39:45.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003b549ba5721c337-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.502 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.503 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.503 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.504 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.505 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.506 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.506 DEBUG:boto:Data: 2023-10-18T20:39:45.506 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.507 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.508 DEBUG:boto:Params: {} 2023-10-18T20:39:45.508 DEBUG:boto:Token: None 2023-10-18T20:39:45.509 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JHdjpPmFbZ5zZblxwR4ojgXVH2Q= 2023-10-18T20:39:45.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JHdjpPmFbZ5zZblxwR4ojgXVH2Q=', 'Content-Length': '0'} 2023-10-18T20:39:45.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc0d3e8022d2a0ba-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.512 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.513 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.513 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.513 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.514 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.514 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.515 DEBUG:boto:Data: 2023-10-18T20:39:45.515 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.515 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.516 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.516 DEBUG:boto:Params: {} 2023-10-18T20:39:45.517 DEBUG:boto:Token: None 2023-10-18T20:39:45.517 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.517 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JHdjpPmFbZ5zZblxwR4ojgXVH2Q= 2023-10-18T20:39:45.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JHdjpPmFbZ5zZblxwR4ojgXVH2Q=', 'Content-Length': '0'} 2023-10-18T20:39:45.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7f9080add1d6233-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:39:45.521 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.521 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.522 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.522 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.522 DEBUG:boto:Method: GET 2023-10-18T20:39:45.523 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.523 DEBUG:boto:Data: 2023-10-18T20:39:45.524 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.524 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.525 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.525 DEBUG:boto:Params: {} 2023-10-18T20:39:45.525 DEBUG:boto:Token: None 2023-10-18T20:39:45.526 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.526 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0O5crj9QbApDIf9hvipHxU2ki/k= 2023-10-18T20:39:45.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0O5crj9QbApDIf9hvipHxU2ki/k=', 'Content-Length': '0'} 2023-10-18T20:39:45.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b9145b3ceb6f5a-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.529 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.530 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.530 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.530 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.531 DEBUG:boto:Method: GET 2023-10-18T20:39:45.531 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.532 DEBUG:boto:Data: 2023-10-18T20:39:45.532 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.533 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.533 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.533 DEBUG:boto:Params: {} 2023-10-18T20:39:45.534 DEBUG:boto:Token: None 2023-10-18T20:39:45.534 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:39:45.535 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0O5crj9QbApDIf9hvipHxU2ki/k= 2023-10-18T20:39:45.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0O5crj9QbApDIf9hvipHxU2ki/k=', 'Content-Length': '0'} 2023-10-18T20:39:45.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005866a10f9216de40-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.538 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.538 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.539 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.539 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.540 DEBUG:boto:Method: GET 2023-10-18T20:39:45.540 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.541 DEBUG:boto:Data: 2023-10-18T20:39:45.541 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.542 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.542 DEBUG:boto:Params: {} 2023-10-18T20:39:45.543 DEBUG:boto:Token: None 2023-10-18T20:39:45.543 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000360117f2ab92ffef-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.547 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.547 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.548 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.548 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.549 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.549 DEBUG:boto:Method: GET 2023-10-18T20:39:45.550 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.550 DEBUG:boto:Data: 2023-10-18T20:39:45.550 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.551 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.551 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.552 DEBUG:boto:Params: {} 2023-10-18T20:39:45.552 DEBUG:boto:Token: None 2023-10-18T20:39:45.553 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b0e42a07e39be12-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.557 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.558 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.558 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.558 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.559 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:39:45.559 DEBUG:boto:Data: 2023-10-18T20:39:45.560 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.560 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.560 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.561 DEBUG:boto:Params: {} 2023-10-18T20:39:45.561 DEBUG:boto:Token: None 2023-10-18T20:39:45.562 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68 2023-10-18T20:39:45.562 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxK8h8CIYmRlVzVVfWBPl02JLno= 2023-10-18T20:39:45.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxK8h8CIYmRlVzVVfWBPl02JLno=', 'Content-Length': '0'} 2023-10-18T20:39:45.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058d3577652da4d76-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.565 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.565 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.566 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.566 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:39:45.566 DEBUG:boto:Data: 2023-10-18T20:39:45.567 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.567 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.568 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.568 DEBUG:boto:Params: {} 2023-10-18T20:39:45.568 DEBUG:boto:Token: None 2023-10-18T20:39:45.569 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68 2023-10-18T20:39:45.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxK8h8CIYmRlVzVVfWBPl02JLno= 2023-10-18T20:39:45.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxK8h8CIYmRlVzVVfWBPl02JLno=', 'Content-Length': '0'} 2023-10-18T20:39:45.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eccdcf104476cfa8-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:39:45.572 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.573 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.573 DEBUG:boto:Method: GET 2023-10-18T20:39:45.574 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:39:45.574 DEBUG:boto:Data: 2023-10-18T20:39:45.575 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.575 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.575 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.576 DEBUG:boto:Params: {} 2023-10-18T20:39:45.576 DEBUG:boto:Token: None 2023-10-18T20:39:45.577 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68 2023-10-18T20:39:45.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zILyBDytzZfi2xMGVz2sfPLbOyI= 2023-10-18T20:39:45.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zILyBDytzZfi2xMGVz2sfPLbOyI=', 'Content-Length': '0'} 2023-10-18T20:39:45.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc826715d875709f-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.626 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:39:45.627 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:39:45.628 DEBUG:boto:Method: GET 2023-10-18T20:39:45.628 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:39:45.628 DEBUG:boto:Data: 2023-10-18T20:39:45.629 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.629 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.630 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.630 DEBUG:boto:Params: {} 2023-10-18T20:39:45.631 DEBUG:boto:Token: None 2023-10-18T20:39:45.631 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj68 2023-10-18T20:39:45.632 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zILyBDytzZfi2xMGVz2sfPLbOyI= 2023-10-18T20:39:45.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zILyBDytzZfi2xMGVz2sfPLbOyI=', 'Content-Length': '0'} 2023-10-18T20:39:45.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eddb2551e57585d1-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.635 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.635 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.636 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.636 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.636 DEBUG:boto:Method: GET 2023-10-18T20:39:45.637 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.637 DEBUG:boto:Data: 2023-10-18T20:39:45.638 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.638 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.639 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.639 DEBUG:boto:Params: {} 2023-10-18T20:39:45.639 DEBUG:boto:Token: None 2023-10-18T20:39:45.640 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.640 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006931878140a9e26e-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.644 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.644 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.645 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.645 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.646 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.646 DEBUG:boto:Method: GET 2023-10-18T20:39:45.646 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:39:45.647 DEBUG:boto:Data: 2023-10-18T20:39:45.647 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.648 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.648 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.649 DEBUG:boto:Params: {} 2023-10-18T20:39:45.649 DEBUG:boto:Token: None 2023-10-18T20:39:45.649 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.650 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d0b25071941749-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.654 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:39:45.655 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.656 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.656 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.657 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.657 DEBUG:boto:Method: GET 2023-10-18T20:39:45.657 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.658 DEBUG:boto:Data: 2023-10-18T20:39:45.658 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.659 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.660 DEBUG:boto:Params: {} 2023-10-18T20:39:45.660 DEBUG:boto:Token: None 2023-10-18T20:39:45.661 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.661 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y8tZiwb4aCdmfLfSJWfvK/l1byk= 2023-10-18T20:39:45.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y8tZiwb4aCdmfLfSJWfvK/l1byk=', 'Content-Length': '0'} 2023-10-18T20:39:45.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f83f025c806fd502-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.664 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.665 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.665 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.665 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.666 DEBUG:boto:Method: GET 2023-10-18T20:39:45.666 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.667 DEBUG:boto:Data: 2023-10-18T20:39:45.667 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.668 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.668 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.668 DEBUG:boto:Params: {} 2023-10-18T20:39:45.669 DEBUG:boto:Token: None 2023-10-18T20:39:45.669 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.670 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y8tZiwb4aCdmfLfSJWfvK/l1byk= 2023-10-18T20:39:45.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y8tZiwb4aCdmfLfSJWfvK/l1byk=', 'Content-Length': '0'} 2023-10-18T20:39:45.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d1c04e7c09c8292-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.673 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.673 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.674 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.674 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.675 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.675 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.675 DEBUG:boto:Data: 2023-10-18T20:39:45.676 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.676 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.677 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.677 DEBUG:boto:Params: {} 2023-10-18T20:39:45.677 DEBUG:boto:Token: None 2023-10-18T20:39:45.678 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.678 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LnykiHt6+muB6PBTujpsZC17os8= 2023-10-18T20:39:45.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LnykiHt6+muB6PBTujpsZC17os8=', 'Content-Length': '0'} 2023-10-18T20:39:45.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adf357294df42e1d-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.681 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.682 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.682 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.682 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.683 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.683 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.684 DEBUG:boto:Data: 2023-10-18T20:39:45.684 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.684 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.685 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.685 DEBUG:boto:Params: {} 2023-10-18T20:39:45.686 DEBUG:boto:Token: None 2023-10-18T20:39:45.686 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.687 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LnykiHt6+muB6PBTujpsZC17os8= 2023-10-18T20:39:45.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LnykiHt6+muB6PBTujpsZC17os8=', 'Content-Length': '0'} 2023-10-18T20:39:45.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003fb06ae1afb3e07-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:39:45.690 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.690 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.691 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.691 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.692 DEBUG:boto:Method: GET 2023-10-18T20:39:45.692 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.693 DEBUG:boto:Data: 2023-10-18T20:39:45.693 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.693 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.694 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.694 DEBUG:boto:Params: {} 2023-10-18T20:39:45.695 DEBUG:boto:Token: None 2023-10-18T20:39:45.695 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.696 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y8tZiwb4aCdmfLfSJWfvK/l1byk= 2023-10-18T20:39:45.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y8tZiwb4aCdmfLfSJWfvK/l1byk=', 'Content-Length': '0'} 2023-10-18T20:39:45.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026109a7affda42e5-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.699 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.699 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.700 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.700 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.701 DEBUG:boto:Method: GET 2023-10-18T20:39:45.701 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.701 DEBUG:boto:Data: 2023-10-18T20:39:45.702 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.702 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.703 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.703 DEBUG:boto:Params: {} 2023-10-18T20:39:45.704 DEBUG:boto:Token: None 2023-10-18T20:39:45.704 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:39:45.705 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y8tZiwb4aCdmfLfSJWfvK/l1byk= 2023-10-18T20:39:45.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y8tZiwb4aCdmfLfSJWfvK/l1byk=', 'Content-Length': '0'} 2023-10-18T20:39:45.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073e155fece2617d1-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.707 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.708 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.708 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.709 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.709 DEBUG:boto:Method: GET 2023-10-18T20:39:45.710 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.710 DEBUG:boto:Data: 2023-10-18T20:39:45.710 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.711 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.711 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.712 DEBUG:boto:Params: {} 2023-10-18T20:39:45.712 DEBUG:boto:Token: None 2023-10-18T20:39:45.713 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.713 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5db3f9d82f167e0-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.758 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.759 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.760 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.760 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.761 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.761 DEBUG:boto:Method: GET 2023-10-18T20:39:45.761 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.762 DEBUG:boto:Data: 2023-10-18T20:39:45.762 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.763 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.764 DEBUG:boto:Params: {} 2023-10-18T20:39:45.764 DEBUG:boto:Token: None 2023-10-18T20:39:45.765 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.765 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7eb34a4c5f114a9-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.770 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.770 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.771 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.771 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.771 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:39:45.772 DEBUG:boto:Data: 2023-10-18T20:39:45.772 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.773 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.773 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.774 DEBUG:boto:Params: {} 2023-10-18T20:39:45.774 DEBUG:boto:Token: None 2023-10-18T20:39:45.775 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69 2023-10-18T20:39:45.775 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L30HAqO1maRYmi85xt2Y30rU1HY= 2023-10-18T20:39:45.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L30HAqO1maRYmi85xt2Y30rU1HY=', 'Content-Length': '0'} 2023-10-18T20:39:45.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d83aff8eb48ff7-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.779 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.779 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.779 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.780 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:39:45.780 DEBUG:boto:Data: 2023-10-18T20:39:45.781 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.782 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.782 DEBUG:boto:Params: {} 2023-10-18T20:39:45.782 DEBUG:boto:Token: None 2023-10-18T20:39:45.783 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69 2023-10-18T20:39:45.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L30HAqO1maRYmi85xt2Y30rU1HY= 2023-10-18T20:39:45.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L30HAqO1maRYmi85xt2Y30rU1HY=', 'Content-Length': '0'} 2023-10-18T20:39:45.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f22b35e81d2ff77-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:39:45.787 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.787 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.788 DEBUG:boto:Method: GET 2023-10-18T20:39:45.788 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:39:45.789 DEBUG:boto:Data: 2023-10-18T20:39:45.789 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.790 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.790 DEBUG:boto:Params: {} 2023-10-18T20:39:45.791 DEBUG:boto:Token: None 2023-10-18T20:39:45.791 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69 2023-10-18T20:39:45.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PqTGrsvb4BjEb2txVCBdRGeromg= 2023-10-18T20:39:45.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PqTGrsvb4BjEb2txVCBdRGeromg=', 'Content-Length': '0'} 2023-10-18T20:39:45.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008df88d843a45d152-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.795 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:39:45.795 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:39:45.796 DEBUG:boto:Method: GET 2023-10-18T20:39:45.796 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:39:45.797 DEBUG:boto:Data: 2023-10-18T20:39:45.797 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.797 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.798 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.798 DEBUG:boto:Params: {} 2023-10-18T20:39:45.799 DEBUG:boto:Token: None 2023-10-18T20:39:45.799 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj69 2023-10-18T20:39:45.800 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PqTGrsvb4BjEb2txVCBdRGeromg= 2023-10-18T20:39:45.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PqTGrsvb4BjEb2txVCBdRGeromg=', 'Content-Length': '0'} 2023-10-18T20:39:45.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000394b67a5a5d6a60d-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.805 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.806 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.807 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.808 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.808 DEBUG:boto:Method: GET 2023-10-18T20:39:45.809 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.809 DEBUG:boto:Data: 2023-10-18T20:39:45.809 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.810 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.811 DEBUG:boto:Params: {} 2023-10-18T20:39:45.811 DEBUG:boto:Token: None 2023-10-18T20:39:45.812 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000582d1a0c915f4e27-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.816 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.816 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.817 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.817 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.818 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.818 DEBUG:boto:Method: GET 2023-10-18T20:39:45.818 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:39:45.819 DEBUG:boto:Data: 2023-10-18T20:39:45.819 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.820 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.820 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.821 DEBUG:boto:Params: {} 2023-10-18T20:39:45.821 DEBUG:boto:Token: None 2023-10-18T20:39:45.821 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.822 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb1f11f55d2f94c6-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.826 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:39:45.827 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:45.827 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:45.828 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.828 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.829 DEBUG:boto:Method: GET 2023-10-18T20:39:45.829 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.829 DEBUG:boto:Data: 2023-10-18T20:39:45.830 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.831 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.831 DEBUG:boto:Params: {} 2023-10-18T20:39:45.832 DEBUG:boto:Token: None 2023-10-18T20:39:45.832 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.833 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g8XsPdYPb3ZJwcCCQHGg/aWVITw= 2023-10-18T20:39:45.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g8XsPdYPb3ZJwcCCQHGg/aWVITw=', 'Content-Length': '0'} 2023-10-18T20:39:45.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed9d9d2256fbd957-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.878 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:45.879 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:45.879 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.880 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.880 DEBUG:boto:Method: GET 2023-10-18T20:39:45.881 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.881 DEBUG:boto:Data: 2023-10-18T20:39:45.882 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.882 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.883 DEBUG:boto:Params: {} 2023-10-18T20:39:45.883 DEBUG:boto:Token: None 2023-10-18T20:39:45.884 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g8XsPdYPb3ZJwcCCQHGg/aWVITw= 2023-10-18T20:39:45.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g8XsPdYPb3ZJwcCCQHGg/aWVITw=', 'Content-Length': '0'} 2023-10-18T20:39:45.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000561045edb47e5fac-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.887 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:45.888 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:45.888 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.888 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.889 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.889 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.890 DEBUG:boto:Data: 2023-10-18T20:39:45.890 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.890 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.891 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.891 DEBUG:boto:Params: {} 2023-10-18T20:39:45.892 DEBUG:boto:Token: None 2023-10-18T20:39:45.892 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.893 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jIBpIdlm/a3N6KTmfpGczgG2Qmg= 2023-10-18T20:39:45.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jIBpIdlm/a3N6KTmfpGczgG2Qmg=', 'Content-Length': '0'} 2023-10-18T20:39:45.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001c42d87e601c320-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.896 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:45.896 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:45.896 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.897 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.897 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.897 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.898 DEBUG:boto:Data: 2023-10-18T20:39:45.898 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.899 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.899 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.900 DEBUG:boto:Params: {} 2023-10-18T20:39:45.900 DEBUG:boto:Token: None 2023-10-18T20:39:45.900 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.901 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jIBpIdlm/a3N6KTmfpGczgG2Qmg= 2023-10-18T20:39:45.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jIBpIdlm/a3N6KTmfpGczgG2Qmg=', 'Content-Length': '0'} 2023-10-18T20:39:45.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c52230c97ec679-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:39:45.904 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:45.905 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:45.905 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.905 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.906 DEBUG:boto:Method: GET 2023-10-18T20:39:45.906 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.907 DEBUG:boto:Data: 2023-10-18T20:39:45.907 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.908 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.908 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.908 DEBUG:boto:Params: {} 2023-10-18T20:39:45.909 DEBUG:boto:Token: None 2023-10-18T20:39:45.909 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.910 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g8XsPdYPb3ZJwcCCQHGg/aWVITw= 2023-10-18T20:39:45.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g8XsPdYPb3ZJwcCCQHGg/aWVITw=', 'Content-Length': '0'} 2023-10-18T20:39:45.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c13481c03796182a-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.913 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:45.913 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:45.914 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.914 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.915 DEBUG:boto:Method: GET 2023-10-18T20:39:45.915 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.916 DEBUG:boto:Data: 2023-10-18T20:39:45.916 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.916 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.917 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.917 DEBUG:boto:Params: {} 2023-10-18T20:39:45.918 DEBUG:boto:Token: None 2023-10-18T20:39:45.918 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:39:45.919 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g8XsPdYPb3ZJwcCCQHGg/aWVITw= 2023-10-18T20:39:45.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g8XsPdYPb3ZJwcCCQHGg/aWVITw=', 'Content-Length': '0'} 2023-10-18T20:39:45.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000447c98bd8e2e6c1f-0065304291-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.962 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.963 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.963 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:39:45.964 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:39:45.964 DEBUG:boto:Method: GET 2023-10-18T20:39:45.965 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:39:45.965 DEBUG:boto:Data: 2023-10-18T20:39:45.966 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.966 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.967 DEBUG:boto:Params: {} 2023-10-18T20:39:45.967 DEBUG:boto:Token: None 2023-10-18T20:39:45.968 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e3145f151091c84-0065304291-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.972 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.973 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:45.973 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:45.973 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:39:45.974 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:39:45.974 DEBUG:boto:Method: GET 2023-10-18T20:39:45.975 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:39:45.975 DEBUG:boto:Data: 2023-10-18T20:39:45.975 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.976 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.976 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.977 DEBUG:boto:Params: {} 2023-10-18T20:39:45.978 DEBUG:boto:Token: None 2023-10-18T20:39:45.978 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/ 2023-10-18T20:39:45.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc= 2023-10-18T20:39:45.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c3MrUZhuKdfOZG+wXOZ2s9JfgPc=', 'Content-Length': '0'} 2023-10-18T20:39:45.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088e0a7a786bd84dc-0065304291-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.983 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:45.984 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:45.985 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:45.986 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.986 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:39:45.987 DEBUG:boto:Data: 2023-10-18T20:39:45.988 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.988 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:45.988 DEBUG:boto:Port: 8001 2023-10-18T20:39:45.989 DEBUG:boto:Params: {} 2023-10-18T20:39:45.989 DEBUG:boto:Token: None 2023-10-18T20:39:45.990 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj76 2023-10-18T20:39:45.990 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Vk6zdzfgfEbW4QeN64Zf6X+yLbo= 2023-10-18T20:39:45.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vk6zdzfgfEbW4QeN64Zf6X+yLbo=', 'Content-Length': '0'} 2023-10-18T20:39:45.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d0cf7380842843a-0065304291-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:45.994 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:45.994 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:45.995 DEBUG:boto:Method: HEAD 2023-10-18T20:39:45.995 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:39:45.996 DEBUG:boto:Data: 2023-10-18T20:39:45.997 DEBUG:boto:Headers: {} 2023-10-18T20:39:45.997 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:45.998 DEBUG:boto:Port: 8000 2023-10-18T20:39:45.998 DEBUG:boto:Params: {} 2023-10-18T20:39:45.999 DEBUG:boto:Token: None 2023-10-18T20:39:45.999 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:45 GMT /bofrnf-59/obj76 2023-10-18T20:39:46.000 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Vk6zdzfgfEbW4QeN64Zf6X+yLbo= 2023-10-18T20:39:46.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vk6zdzfgfEbW4QeN64Zf6X+yLbo=', 'Content-Length': '0'} 2023-10-18T20:39:46.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bb5bc31a1d66af0-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:39:46.007 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:46.007 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:46.008 DEBUG:boto:Method: GET 2023-10-18T20:39:46.008 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:39:46.009 DEBUG:boto:Data: 2023-10-18T20:39:46.009 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.010 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.010 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.010 DEBUG:boto:Params: {} 2023-10-18T20:39:46.011 DEBUG:boto:Token: None 2023-10-18T20:39:46.011 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj76 2023-10-18T20:39:46.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d3QYRMeGYMt+N2v9ZbppIe4zY7s= 2023-10-18T20:39:46.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d3QYRMeGYMt+N2v9ZbppIe4zY7s=', 'Content-Length': '0'} 2023-10-18T20:39:46.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001dbb15e7544a2b9-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.062 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:39:46.063 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:39:46.063 DEBUG:boto:Method: GET 2023-10-18T20:39:46.064 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:39:46.064 DEBUG:boto:Data: 2023-10-18T20:39:46.065 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.065 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.066 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.066 DEBUG:boto:Params: {} 2023-10-18T20:39:46.067 DEBUG:boto:Token: None 2023-10-18T20:39:46.068 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj76 2023-10-18T20:39:46.068 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d3QYRMeGYMt+N2v9ZbppIe4zY7s= 2023-10-18T20:39:46.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d3QYRMeGYMt+N2v9ZbppIe4zY7s=', 'Content-Length': '0'} 2023-10-18T20:39:46.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e7cf290803fd941-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.071 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.072 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.072 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:39:46.073 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:39:46.073 DEBUG:boto:Method: GET 2023-10-18T20:39:46.073 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:39:46.074 DEBUG:boto:Data: 2023-10-18T20:39:46.074 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.075 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.075 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.076 DEBUG:boto:Params: {} 2023-10-18T20:39:46.076 DEBUG:boto:Token: None 2023-10-18T20:39:46.077 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.077 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14b2a579667b66d-0065304292-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.081 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.082 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.082 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.083 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:39:46.083 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:39:46.084 DEBUG:boto:Method: GET 2023-10-18T20:39:46.085 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:39:46.085 DEBUG:boto:Data: 2023-10-18T20:39:46.086 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.087 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.088 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.089 DEBUG:boto:Params: {} 2023-10-18T20:39:46.089 DEBUG:boto:Token: None 2023-10-18T20:39:46.090 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.091 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f75aa402e245531-0065304292-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.142 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:39:46.144 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.145 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.145 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.146 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.146 DEBUG:boto:Method: GET 2023-10-18T20:39:46.147 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.147 DEBUG:boto:Data: 2023-10-18T20:39:46.148 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.148 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.149 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.150 DEBUG:boto:Params: {} 2023-10-18T20:39:46.151 DEBUG:boto:Token: None 2023-10-18T20:39:46.152 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WVzDdooIC4SvoammOicI+LL+d0s= 2023-10-18T20:39:46.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WVzDdooIC4SvoammOicI+LL+d0s=', 'Content-Length': '0'} 2023-10-18T20:39:46.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9aaaaed40e9c41-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.157 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.158 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.159 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.160 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.160 DEBUG:boto:Method: GET 2023-10-18T20:39:46.162 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.163 DEBUG:boto:Data: 2023-10-18T20:39:46.163 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.164 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.164 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.165 DEBUG:boto:Params: {} 2023-10-18T20:39:46.165 DEBUG:boto:Token: None 2023-10-18T20:39:46.166 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.166 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WVzDdooIC4SvoammOicI+LL+d0s= 2023-10-18T20:39:46.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WVzDdooIC4SvoammOicI+LL+d0s=', 'Content-Length': '0'} 2023-10-18T20:39:46.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d191c1bc2a6e0c7-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.169 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.170 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.170 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.171 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.171 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.172 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.172 DEBUG:boto:Data: 2023-10-18T20:39:46.172 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.173 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.173 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.174 DEBUG:boto:Params: {} 2023-10-18T20:39:46.174 DEBUG:boto:Token: None 2023-10-18T20:39:46.174 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.175 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QPBxNIeKhLlCAh6LBQuAFhPZsZ0= 2023-10-18T20:39:46.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QPBxNIeKhLlCAh6LBQuAFhPZsZ0=', 'Content-Length': '0'} 2023-10-18T20:39:46.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d20f1d29f44e5418-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.178 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.178 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.179 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.179 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.179 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.180 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.180 DEBUG:boto:Data: 2023-10-18T20:39:46.180 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.181 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.181 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.182 DEBUG:boto:Params: {} 2023-10-18T20:39:46.182 DEBUG:boto:Token: None 2023-10-18T20:39:46.182 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QPBxNIeKhLlCAh6LBQuAFhPZsZ0= 2023-10-18T20:39:46.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QPBxNIeKhLlCAh6LBQuAFhPZsZ0=', 'Content-Length': '0'} 2023-10-18T20:39:46.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041f5ba0acf3c70c9-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:39:46.186 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.187 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.187 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.188 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.188 DEBUG:boto:Method: GET 2023-10-18T20:39:46.188 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.189 DEBUG:boto:Data: 2023-10-18T20:39:46.189 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.190 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.191 DEBUG:boto:Params: {} 2023-10-18T20:39:46.191 DEBUG:boto:Token: None 2023-10-18T20:39:46.192 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WVzDdooIC4SvoammOicI+LL+d0s= 2023-10-18T20:39:46.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WVzDdooIC4SvoammOicI+LL+d0s=', 'Content-Length': '0'} 2023-10-18T20:39:46.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000207437b3da1998f8-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.195 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.195 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.196 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.196 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.197 DEBUG:boto:Method: GET 2023-10-18T20:39:46.197 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.198 DEBUG:boto:Data: 2023-10-18T20:39:46.198 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.199 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.200 DEBUG:boto:Params: {} 2023-10-18T20:39:46.200 DEBUG:boto:Token: None 2023-10-18T20:39:46.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:39:46.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WVzDdooIC4SvoammOicI+LL+d0s= 2023-10-18T20:39:46.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WVzDdooIC4SvoammOicI+LL+d0s=', 'Content-Length': '0'} 2023-10-18T20:39:46.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000334d5dcaeb5cb599-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.204 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.205 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.205 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.206 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.206 DEBUG:boto:Method: GET 2023-10-18T20:39:46.207 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.207 DEBUG:boto:Data: 2023-10-18T20:39:46.207 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.208 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.208 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.209 DEBUG:boto:Params: {} 2023-10-18T20:39:46.209 DEBUG:boto:Token: None 2023-10-18T20:39:46.210 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab36eaba9827c5b-0065304292-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.219 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.220 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.220 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.221 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.221 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.221 DEBUG:boto:Method: GET 2023-10-18T20:39:46.222 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.222 DEBUG:boto:Data: 2023-10-18T20:39:46.223 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.223 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.224 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.224 DEBUG:boto:Params: {} 2023-10-18T20:39:46.225 DEBUG:boto:Token: None 2023-10-18T20:39:46.225 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.226 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e27b959c354bcd8a-0065304292-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.230 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.231 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.231 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.231 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.232 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:39:46.233 DEBUG:boto:Data: 2023-10-18T20:39:46.233 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.233 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.234 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.234 DEBUG:boto:Params: {} 2023-10-18T20:39:46.235 DEBUG:boto:Token: None 2023-10-18T20:39:46.235 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85 2023-10-18T20:39:46.236 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yur6LX8a3xMlRDo1i1wmT6+PwsM= 2023-10-18T20:39:46.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yur6LX8a3xMlRDo1i1wmT6+PwsM=', 'Content-Length': '0'} 2023-10-18T20:39:46.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea77dcb19c986272-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.239 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.239 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.240 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.240 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:39:46.241 DEBUG:boto:Data: 2023-10-18T20:39:46.241 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.241 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.242 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.242 DEBUG:boto:Params: {} 2023-10-18T20:39:46.243 DEBUG:boto:Token: None 2023-10-18T20:39:46.243 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85 2023-10-18T20:39:46.243 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yur6LX8a3xMlRDo1i1wmT6+PwsM= 2023-10-18T20:39:46.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yur6LX8a3xMlRDo1i1wmT6+PwsM=', 'Content-Length': '0'} 2023-10-18T20:39:46.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f954536239157b6-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:39:46.247 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.248 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.249 DEBUG:boto:Method: GET 2023-10-18T20:39:46.250 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:39:46.250 DEBUG:boto:Data: 2023-10-18T20:39:46.251 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.251 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.251 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.252 DEBUG:boto:Params: {} 2023-10-18T20:39:46.252 DEBUG:boto:Token: None 2023-10-18T20:39:46.253 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85 2023-10-18T20:39:46.253 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OtXFGPi1oPeuX3br86m4nWJDIfg= 2023-10-18T20:39:46.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OtXFGPi1oPeuX3br86m4nWJDIfg=', 'Content-Length': '0'} 2023-10-18T20:39:46.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000577574bd62b21795-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.302 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:39:46.303 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:39:46.303 DEBUG:boto:Method: GET 2023-10-18T20:39:46.304 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:39:46.304 DEBUG:boto:Data: 2023-10-18T20:39:46.305 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.305 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.305 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.306 DEBUG:boto:Params: {} 2023-10-18T20:39:46.306 DEBUG:boto:Token: None 2023-10-18T20:39:46.307 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj85 2023-10-18T20:39:46.307 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OtXFGPi1oPeuX3br86m4nWJDIfg= 2023-10-18T20:39:46.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OtXFGPi1oPeuX3br86m4nWJDIfg=', 'Content-Length': '0'} 2023-10-18T20:39:46.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090916abcb3a9afd1-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.311 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.312 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.312 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.312 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.313 DEBUG:boto:Method: GET 2023-10-18T20:39:46.313 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.314 DEBUG:boto:Data: 2023-10-18T20:39:46.315 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.316 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.316 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.316 DEBUG:boto:Params: {} 2023-10-18T20:39:46.317 DEBUG:boto:Token: None 2023-10-18T20:39:46.317 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.318 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000557432c6a9939f20-0065304292-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.322 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.323 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.323 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.324 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.324 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.325 DEBUG:boto:Method: GET 2023-10-18T20:39:46.325 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:39:46.325 DEBUG:boto:Data: 2023-10-18T20:39:46.326 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.326 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.327 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.327 DEBUG:boto:Params: {} 2023-10-18T20:39:46.328 DEBUG:boto:Token: None 2023-10-18T20:39:46.328 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.329 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009343b7a5b88f6a95-0065304292-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.333 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:39:46.334 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.335 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.335 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.336 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.336 DEBUG:boto:Method: GET 2023-10-18T20:39:46.336 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.337 DEBUG:boto:Data: 2023-10-18T20:39:46.337 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.338 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.339 DEBUG:boto:Params: {} 2023-10-18T20:39:46.339 DEBUG:boto:Token: None 2023-10-18T20:39:46.340 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qaW8vOCeb7+AnP8s8PP+l5Q1BPE= 2023-10-18T20:39:46.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qaW8vOCeb7+AnP8s8PP+l5Q1BPE=', 'Content-Length': '0'} 2023-10-18T20:39:46.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005242bb0aed020d27-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.344 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.344 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.345 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.345 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.346 DEBUG:boto:Method: GET 2023-10-18T20:39:46.346 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.346 DEBUG:boto:Data: 2023-10-18T20:39:46.347 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.347 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.348 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.348 DEBUG:boto:Params: {} 2023-10-18T20:39:46.349 DEBUG:boto:Token: None 2023-10-18T20:39:46.349 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.350 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qaW8vOCeb7+AnP8s8PP+l5Q1BPE= 2023-10-18T20:39:46.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qaW8vOCeb7+AnP8s8PP+l5Q1BPE=', 'Content-Length': '0'} 2023-10-18T20:39:46.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd61a74eb6da9948-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.353 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.353 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.354 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.354 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.354 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.355 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.355 DEBUG:boto:Data: 2023-10-18T20:39:46.356 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.356 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.356 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.357 DEBUG:boto:Params: {} 2023-10-18T20:39:46.357 DEBUG:boto:Token: None 2023-10-18T20:39:46.358 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:heqI3oTu07jxYKGZeXYSysn73jQ= 2023-10-18T20:39:46.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:heqI3oTu07jxYKGZeXYSysn73jQ=', 'Content-Length': '0'} 2023-10-18T20:39:46.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000731a11c49ca26e9b-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.361 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.362 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.362 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.362 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.363 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.363 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.364 DEBUG:boto:Data: 2023-10-18T20:39:46.364 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.364 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.365 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.365 DEBUG:boto:Params: {} 2023-10-18T20:39:46.366 DEBUG:boto:Token: None 2023-10-18T20:39:46.366 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.366 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:heqI3oTu07jxYKGZeXYSysn73jQ= 2023-10-18T20:39:46.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:heqI3oTu07jxYKGZeXYSysn73jQ=', 'Content-Length': '0'} 2023-10-18T20:39:46.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7b933dd8116b3e1-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:39:46.370 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.370 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.371 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.371 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.371 DEBUG:boto:Method: GET 2023-10-18T20:39:46.372 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.372 DEBUG:boto:Data: 2023-10-18T20:39:46.373 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.373 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.374 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.374 DEBUG:boto:Params: {} 2023-10-18T20:39:46.374 DEBUG:boto:Token: None 2023-10-18T20:39:46.375 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.375 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qaW8vOCeb7+AnP8s8PP+l5Q1BPE= 2023-10-18T20:39:46.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qaW8vOCeb7+AnP8s8PP+l5Q1BPE=', 'Content-Length': '0'} 2023-10-18T20:39:46.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002353fdb5b11bde7f-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.378 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.379 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.379 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.380 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.380 DEBUG:boto:Method: GET 2023-10-18T20:39:46.381 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.381 DEBUG:boto:Data: 2023-10-18T20:39:46.381 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.382 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.382 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.383 DEBUG:boto:Params: {} 2023-10-18T20:39:46.383 DEBUG:boto:Token: None 2023-10-18T20:39:46.384 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:39:46.384 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qaW8vOCeb7+AnP8s8PP+l5Q1BPE= 2023-10-18T20:39:46.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qaW8vOCeb7+AnP8s8PP+l5Q1BPE=', 'Content-Length': '0'} 2023-10-18T20:39:46.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efcc981882fc48b0-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.387 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.387 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.388 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.388 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.389 DEBUG:boto:Method: GET 2023-10-18T20:39:46.389 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.389 DEBUG:boto:Data: 2023-10-18T20:39:46.390 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.391 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.391 DEBUG:boto:Params: {} 2023-10-18T20:39:46.392 DEBUG:boto:Token: None 2023-10-18T20:39:46.392 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.393 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ab62e33124e22d9-0065304292-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.396 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.397 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.397 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.398 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.398 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.398 DEBUG:boto:Method: GET 2023-10-18T20:39:46.399 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.399 DEBUG:boto:Data: 2023-10-18T20:39:46.400 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.400 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.400 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.401 DEBUG:boto:Params: {} 2023-10-18T20:39:46.401 DEBUG:boto:Token: None 2023-10-18T20:39:46.402 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c59dba3855e5d760-0065304292-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.406 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.407 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.407 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.408 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.408 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:39:46.409 DEBUG:boto:Data: 2023-10-18T20:39:46.409 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.410 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.410 DEBUG:boto:Params: {} 2023-10-18T20:39:46.411 DEBUG:boto:Token: None 2023-10-18T20:39:46.411 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92 2023-10-18T20:39:46.412 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wNxArKSO3Ssdfz21yj1DUX8Zyv0= 2023-10-18T20:39:46.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wNxArKSO3Ssdfz21yj1DUX8Zyv0=', 'Content-Length': '0'} 2023-10-18T20:39:46.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd7b4c59fdb98346-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.414 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.415 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.415 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.416 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:39:46.416 DEBUG:boto:Data: 2023-10-18T20:39:46.417 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.417 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.418 DEBUG:boto:Params: {} 2023-10-18T20:39:46.418 DEBUG:boto:Token: None 2023-10-18T20:39:46.419 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92 2023-10-18T20:39:46.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wNxArKSO3Ssdfz21yj1DUX8Zyv0= 2023-10-18T20:39:46.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wNxArKSO3Ssdfz21yj1DUX8Zyv0=', 'Content-Length': '0'} 2023-10-18T20:39:46.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc05dc9b1998c9a0-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:39:46.422 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.423 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.423 DEBUG:boto:Method: GET 2023-10-18T20:39:46.424 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:39:46.424 DEBUG:boto:Data: 2023-10-18T20:39:46.424 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.425 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.426 DEBUG:boto:Params: {} 2023-10-18T20:39:46.426 DEBUG:boto:Token: None 2023-10-18T20:39:46.427 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92 2023-10-18T20:39:46.427 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h9hDXB5N98JOvOYzk9ebcYr0Yfw= 2023-10-18T20:39:46.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h9hDXB5N98JOvOYzk9ebcYr0Yfw=', 'Content-Length': '0'} 2023-10-18T20:39:46.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b8705a26cec2322-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.474 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:39:46.475 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:39:46.475 DEBUG:boto:Method: GET 2023-10-18T20:39:46.476 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:39:46.476 DEBUG:boto:Data: 2023-10-18T20:39:46.477 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.477 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.478 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.478 DEBUG:boto:Params: {} 2023-10-18T20:39:46.479 DEBUG:boto:Token: None 2023-10-18T20:39:46.479 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj92 2023-10-18T20:39:46.480 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h9hDXB5N98JOvOYzk9ebcYr0Yfw= 2023-10-18T20:39:46.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h9hDXB5N98JOvOYzk9ebcYr0Yfw=', 'Content-Length': '0'} 2023-10-18T20:39:46.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff1c8e535b454ea-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.483 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.483 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.484 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.484 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.484 DEBUG:boto:Method: GET 2023-10-18T20:39:46.485 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.485 DEBUG:boto:Data: 2023-10-18T20:39:46.486 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.486 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.486 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.487 DEBUG:boto:Params: {} 2023-10-18T20:39:46.487 DEBUG:boto:Token: None 2023-10-18T20:39:46.488 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.488 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec33e6ccd61f62ad-0065304292-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.492 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.492 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.493 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.493 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.494 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.494 DEBUG:boto:Method: GET 2023-10-18T20:39:46.494 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:39:46.495 DEBUG:boto:Data: 2023-10-18T20:39:46.495 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.496 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.496 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.496 DEBUG:boto:Params: {} 2023-10-18T20:39:46.497 DEBUG:boto:Token: None 2023-10-18T20:39:46.497 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.498 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a35f5610829c0709-0065304292-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.502 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:39:46.503 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.504 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.504 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.504 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.505 DEBUG:boto:Method: GET 2023-10-18T20:39:46.505 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.506 DEBUG:boto:Data: 2023-10-18T20:39:46.506 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.507 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.507 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.507 DEBUG:boto:Params: {} 2023-10-18T20:39:46.508 DEBUG:boto:Token: None 2023-10-18T20:39:46.508 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.509 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qwYfqXXCG6ZmzIhl0VLENDxOScc= 2023-10-18T20:39:46.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qwYfqXXCG6ZmzIhl0VLENDxOScc=', 'Content-Length': '0'} 2023-10-18T20:39:46.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029628425e01a52bb-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.512 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.512 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.513 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.513 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.513 DEBUG:boto:Method: GET 2023-10-18T20:39:46.514 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.514 DEBUG:boto:Data: 2023-10-18T20:39:46.515 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.515 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.516 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.516 DEBUG:boto:Params: {} 2023-10-18T20:39:46.516 DEBUG:boto:Token: None 2023-10-18T20:39:46.517 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.517 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qwYfqXXCG6ZmzIhl0VLENDxOScc= 2023-10-18T20:39:46.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qwYfqXXCG6ZmzIhl0VLENDxOScc=', 'Content-Length': '0'} 2023-10-18T20:39:46.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083922e55d5f977b6-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.520 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.521 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.521 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.522 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.522 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.522 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.523 DEBUG:boto:Data: 2023-10-18T20:39:46.523 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.524 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.524 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.524 DEBUG:boto:Params: {} 2023-10-18T20:39:46.525 DEBUG:boto:Token: None 2023-10-18T20:39:46.525 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.526 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zYjmWIsii2bkPSHPTu39rvpS+30= 2023-10-18T20:39:46.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zYjmWIsii2bkPSHPTu39rvpS+30=', 'Content-Length': '0'} 2023-10-18T20:39:46.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083f740e02badb3a7-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.528 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.529 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.529 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.530 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.530 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.531 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.531 DEBUG:boto:Data: 2023-10-18T20:39:46.531 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.532 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.532 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.532 DEBUG:boto:Params: {} 2023-10-18T20:39:46.533 DEBUG:boto:Token: None 2023-10-18T20:39:46.533 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.534 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zYjmWIsii2bkPSHPTu39rvpS+30= 2023-10-18T20:39:46.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zYjmWIsii2bkPSHPTu39rvpS+30=', 'Content-Length': '0'} 2023-10-18T20:39:46.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001811da8a97fe65df-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:39:46.537 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.538 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.538 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.539 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.539 DEBUG:boto:Method: GET 2023-10-18T20:39:46.539 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.540 DEBUG:boto:Data: 2023-10-18T20:39:46.540 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.541 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.541 DEBUG:boto:Params: {} 2023-10-18T20:39:46.542 DEBUG:boto:Token: None 2023-10-18T20:39:46.542 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qwYfqXXCG6ZmzIhl0VLENDxOScc= 2023-10-18T20:39:46.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qwYfqXXCG6ZmzIhl0VLENDxOScc=', 'Content-Length': '0'} 2023-10-18T20:39:46.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc760a8581b9d28-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.590 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.591 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.591 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.592 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.592 DEBUG:boto:Method: GET 2023-10-18T20:39:46.593 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.593 DEBUG:boto:Data: 2023-10-18T20:39:46.593 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.594 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.594 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.595 DEBUG:boto:Params: {} 2023-10-18T20:39:46.595 DEBUG:boto:Token: None 2023-10-18T20:39:46.596 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:39:46.596 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qwYfqXXCG6ZmzIhl0VLENDxOScc= 2023-10-18T20:39:46.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qwYfqXXCG6ZmzIhl0VLENDxOScc=', 'Content-Length': '0'} 2023-10-18T20:39:46.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068bffce305e8f64e-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.599 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.600 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.600 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.601 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.601 DEBUG:boto:Method: GET 2023-10-18T20:39:46.602 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.602 DEBUG:boto:Data: 2023-10-18T20:39:46.602 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.603 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.603 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.604 DEBUG:boto:Params: {} 2023-10-18T20:39:46.604 DEBUG:boto:Token: None 2023-10-18T20:39:46.605 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c42fa717184d7f2-0065304292-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.608 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.609 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.610 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.610 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.610 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.611 DEBUG:boto:Method: GET 2023-10-18T20:39:46.611 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.612 DEBUG:boto:Data: 2023-10-18T20:39:46.612 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.612 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.613 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.613 DEBUG:boto:Params: {} 2023-10-18T20:39:46.614 DEBUG:boto:Token: None 2023-10-18T20:39:46.615 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042737b77cd9121cc-0065304292-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.662 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.663 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.664 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.664 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.665 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:39:46.665 DEBUG:boto:Data: 2023-10-18T20:39:46.665 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.666 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.666 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.667 DEBUG:boto:Params: {} 2023-10-18T20:39:46.667 DEBUG:boto:Token: None 2023-10-18T20:39:46.667 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95 2023-10-18T20:39:46.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mF1r2Mt0ssebzZYeSYAexFgOZCE= 2023-10-18T20:39:46.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mF1r2Mt0ssebzZYeSYAexFgOZCE=', 'Content-Length': '0'} 2023-10-18T20:39:46.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca175483db0159f0-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.671 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.672 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.672 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.672 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:39:46.673 DEBUG:boto:Data: 2023-10-18T20:39:46.673 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.674 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.674 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.674 DEBUG:boto:Params: {} 2023-10-18T20:39:46.675 DEBUG:boto:Token: None 2023-10-18T20:39:46.675 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95 2023-10-18T20:39:46.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mF1r2Mt0ssebzZYeSYAexFgOZCE= 2023-10-18T20:39:46.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mF1r2Mt0ssebzZYeSYAexFgOZCE=', 'Content-Length': '0'} 2023-10-18T20:39:46.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000647aba5d58d92530-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:39:46.679 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.679 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.680 DEBUG:boto:Method: GET 2023-10-18T20:39:46.680 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:39:46.681 DEBUG:boto:Data: 2023-10-18T20:39:46.681 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.682 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.682 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.682 DEBUG:boto:Params: {} 2023-10-18T20:39:46.683 DEBUG:boto:Token: None 2023-10-18T20:39:46.683 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95 2023-10-18T20:39:46.684 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6NfFBOVnUH5QTYrtIoTwjDevkzg= 2023-10-18T20:39:46.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6NfFBOVnUH5QTYrtIoTwjDevkzg=', 'Content-Length': '0'} 2023-10-18T20:39:46.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df128edcfa96839-0065304292-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.687 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:39:46.687 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:39:46.688 DEBUG:boto:Method: GET 2023-10-18T20:39:46.688 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:39:46.688 DEBUG:boto:Data: 2023-10-18T20:39:46.689 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.690 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.690 DEBUG:boto:Params: {} 2023-10-18T20:39:46.691 DEBUG:boto:Token: None 2023-10-18T20:39:46.691 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj95 2023-10-18T20:39:46.691 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6NfFBOVnUH5QTYrtIoTwjDevkzg= 2023-10-18T20:39:46.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6NfFBOVnUH5QTYrtIoTwjDevkzg=', 'Content-Length': '0'} 2023-10-18T20:39:46.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da44b652412c5bb9-0065304292-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.695 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.695 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.695 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.696 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.696 DEBUG:boto:Method: GET 2023-10-18T20:39:46.697 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.697 DEBUG:boto:Data: 2023-10-18T20:39:46.698 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:39:46.698 DEBUG:boto:Port: 8001 2023-10-18T20:39:46.699 DEBUG:boto:Params: {} 2023-10-18T20:39:46.699 DEBUG:boto:Token: None 2023-10-18T20:39:46.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.700 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065d06f521738d461-0065304292-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.704 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.705 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.705 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.705 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.706 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.706 DEBUG:boto:Method: GET 2023-10-18T20:39:46.707 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:39:46.707 DEBUG:boto:Data: 2023-10-18T20:39:46.707 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.708 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.708 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.709 DEBUG:boto:Params: {} 2023-10-18T20:39:46.709 DEBUG:boto:Token: None 2023-10-18T20:39:46.710 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU= 2023-10-18T20:39:46.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LxSv1hitro/xkazuXzhcTyGFRlU=', 'Content-Length': '0'} 2023-10-18T20:39:46.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c3c43eaa02d51f4-0065304292-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.714 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:39:46.715 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:39:46.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:39:46.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:39:46.717 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:39:46.773 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:39:46.773 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:39:46.821 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:39:46.821 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-59 2023-10-18T20:39:46.831 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 20 2023-10-18T20:39:46.877 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.877 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.878 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.878 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:39:46.879 DEBUG:boto:Data: 2023-10-18T20:39:46.879 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.880 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.880 DEBUG:boto:Params: {} 2023-10-18T20:39:46.881 DEBUG:boto:Token: None 2023-10-18T20:39:46.881 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.882 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5yemV5JRHTkWNn7hXbfaHBM00Dw= 2023-10-18T20:39:46.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5yemV5JRHTkWNn7hXbfaHBM00Dw=', 'Content-Length': '0'} 2023-10-18T20:39:46.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000105d2e21698acc9e-0065304292-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.886 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:39:46.887 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:39:46.887 DEBUG:boto:Method: PUT 2023-10-18T20:39:46.888 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:39:46.888 DEBUG:boto:Data: 2023-10-18T20:39:46.889 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:39:46.889 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.889 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.890 DEBUG:boto:Params: {} 2023-10-18T20:39:46.890 DEBUG:boto:Token: None 2023-10-18T20:39:46.891 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj29 2023-10-18T20:39:46.891 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:inmTtWUonrdK++sPcApeY6PtUGo= 2023-10-18T20:39:46.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:inmTtWUonrdK++sPcApeY6PtUGo='} 2023-10-18T20:39:46.897 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de9ff0b6326c6d19-0065304292-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.898 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.898 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.899 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.899 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:39:46.899 DEBUG:boto:Data: 2023-10-18T20:39:46.900 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.900 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.901 DEBUG:boto:Params: {} 2023-10-18T20:39:46.901 DEBUG:boto:Token: None 2023-10-18T20:39:46.902 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.902 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5yemV5JRHTkWNn7hXbfaHBM00Dw= 2023-10-18T20:39:46.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5yemV5JRHTkWNn7hXbfaHBM00Dw=', 'Content-Length': '0'} 2023-10-18T20:39:46.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005df6ade1e44a504e-0065304292-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.906 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:39:46.907 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:39:46.907 DEBUG:boto:Method: PUT 2023-10-18T20:39:46.908 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:39:46.908 DEBUG:boto:Data: 2023-10-18T20:39:46.908 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:39:46.909 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.909 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.910 DEBUG:boto:Params: {} 2023-10-18T20:39:46.910 DEBUG:boto:Token: None 2023-10-18T20:39:46.911 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj58 2023-10-18T20:39:46.911 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h9vg64a5PhirUCk28CkI6YrwAko= 2023-10-18T20:39:46.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h9vg64a5PhirUCk28CkI6YrwAko='} 2023-10-18T20:39:46.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000865c8de8336b96-0065304292-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.917 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.918 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.918 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.919 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:39:46.919 DEBUG:boto:Data: 2023-10-18T20:39:46.919 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.920 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.920 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.921 DEBUG:boto:Params: {} 2023-10-18T20:39:46.921 DEBUG:boto:Token: None 2023-10-18T20:39:46.921 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5yemV5JRHTkWNn7hXbfaHBM00Dw= 2023-10-18T20:39:46.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5yemV5JRHTkWNn7hXbfaHBM00Dw=', 'Content-Length': '0'} 2023-10-18T20:39:46.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051cf0d9e19d3242e-0065304292-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.926 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:39:46.926 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:39:46.927 DEBUG:boto:Method: PUT 2023-10-18T20:39:46.927 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:39:46.927 DEBUG:boto:Data: 2023-10-18T20:39:46.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:39:46.928 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.929 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.929 DEBUG:boto:Params: {} 2023-10-18T20:39:46.930 DEBUG:boto:Token: None 2023-10-18T20:39:46.930 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj87 2023-10-18T20:39:46.930 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nYYLdJSgfS3cxaQ2E5uecSWi8ic= 2023-10-18T20:39:46.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nYYLdJSgfS3cxaQ2E5uecSWi8ic='} 2023-10-18T20:39:46.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017e06506ba90232a-0065304292-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.975 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.976 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.976 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.977 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:39:46.977 DEBUG:boto:Data: 2023-10-18T20:39:46.977 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.978 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.978 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.979 DEBUG:boto:Params: {} 2023-10-18T20:39:46.979 DEBUG:boto:Token: None 2023-10-18T20:39:46.980 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/ 2023-10-18T20:39:46.980 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5yemV5JRHTkWNn7hXbfaHBM00Dw= 2023-10-18T20:39:46.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5yemV5JRHTkWNn7hXbfaHBM00Dw=', 'Content-Length': '0'} 2023-10-18T20:39:46.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a562f5c172610d8-0065304292-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.984 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:39:46.985 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:39:46.985 DEBUG:boto:Method: PUT 2023-10-18T20:39:46.986 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:39:46.986 DEBUG:boto:Data: 2023-10-18T20:39:46.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:39:46.987 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:46.987 DEBUG:boto:Port: 8000 2023-10-18T20:39:46.988 DEBUG:boto:Params: {} 2023-10-18T20:39:46.989 DEBUG:boto:Token: None 2023-10-18T20:39:46.989 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:39:46 GMT /bofrnf-59/obj116 2023-10-18T20:39:46.990 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6NLTE9cuYTojAyXiw9xHWf/fedg= 2023-10-18T20:39:46.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:39:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6NLTE9cuYTojAyXiw9xHWf/fedg='} 2023-10-18T20:39:46.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ab014c25186eb57-0065304292-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:46.996 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:39:46.997 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:39:46.997 DEBUG:boto:Method: HEAD 2023-10-18T20:39:46.998 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:39:46.998 DEBUG:boto:Data: 2023-10-18T20:39:46.999 DEBUG:boto:Headers: {} 2023-10-18T20:39:46.999 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:47.000 DEBUG:boto:Port: 8000 2023-10-18T20:39:47.000 DEBUG:boto:Params: {} 2023-10-18T20:39:47.000 DEBUG:boto:Token: None 2023-10-18T20:39:47.001 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:39:47 GMT /bofrnf-59/ 2023-10-18T20:39:47.001 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZAM/QqMgmYrCd7dKUhsrityuo98= 2023-10-18T20:39:47.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:39:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZAM/QqMgmYrCd7dKUhsrityuo98=', 'Content-Length': '0'} 2023-10-18T20:39:47.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5d5a9f9c5c18c81-0065304293-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:47.006 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:39:47.006 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:39:47.007 DEBUG:boto:Method: PUT 2023-10-18T20:39:47.007 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:39:47.008 DEBUG:boto:Data: 2023-10-18T20:39:47.008 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:39:47.009 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:39:47.009 DEBUG:boto:Port: 8000 2023-10-18T20:39:47.009 DEBUG:boto:Params: {} 2023-10-18T20:39:47.010 DEBUG:boto:Token: None 2023-10-18T20:39:47.010 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:39:47 GMT /bofrnf-59/obj145 2023-10-18T20:39:47.011 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sdTGlpwu2hf2jrufM86K9wcjS7o= 2023-10-18T20:39:47.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:39:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sdTGlpwu2hf2jrufM86K9wcjS7o='} 2023-10-18T20:39:47.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c894a8498d199ffa-0065304293-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:39:47.055 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-59 2023-10-18T20:39:47.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:39:47.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:39:47.057 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:39:47.112 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:39:47.113 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:39:47.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:47.145+0000 7ff95e51e780 1 waiting to reach latest gen 4, on 3.. 2023-10-18T20:39:52.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:52.145+0000 7ff95e51e780 1 waiting to reach latest gen 4, on 3.. 2023-10-18T20:39:57.148 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:39:57.146+0000 7ff95e51e780 1 waiting to reach latest gen 4, on 3.. 2023-10-18T20:40:02.151 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:02.148+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:02.151 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:02.151 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:07.153 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:07.151+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:07.153 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:07.153 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:12.155 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:12.153+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:12.155 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:12.155 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:17.157 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:17.156+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:17.157 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:17.158 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:22.159 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:22.158+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:22.160 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:22.160 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:27.161 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:27.160+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:27.162 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:27.162 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:32.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:32.162+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:32.164 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:32.165 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:37.165 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:37.164+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:37.166 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:37.166 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:42.167 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:42.166+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:42.168 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:42.168 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:47.170 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:47.167+0000 7ff95e51e780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:47.170 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000001.98.6, , , , , , , , , , , , , , , , 00000000002.73.6, , , , , , , , , , , ] 2023-10-18T20:40:47.170 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:52.172 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:52.169+0000 7ff95e51e780 1 bucket sync caught up with source: 2023-10-18T20:40:52.172 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:52.172 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [00000000002.60.6, 00000000001.98.6, , , , , 00000000001.75.6, , , , , , , , , , , 00000000002.73.6, 00000000001.106.6, , , , , , , , , , ] 2023-10-18T20:40:52.173 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:52.169+0000 7ff95e51e780 0 bucket checkpoint complete 2023-10-18T20:40:52.189 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-59 2023-10-18T20:40:52.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:40:52.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:40:52.192 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:40:52.254 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:40:52.254 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:40:52.292 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:52.290+0000 7f774da52780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:52.292 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.757.6, 00000000002.1177.6, 00000000001.1248.6, 00000000003.706.6, 00000000002.773.6, , 00000000001.863.6, 00000000002.232.6, 00000000005.457.6, 00000000004.248.6] 2023-10-18T20:40:52.292 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.757.6, 00000000002.1177.6, 00000000001.1248.6, 00000000003.706.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000002.232.6, 00000000005.457.6, 00000000004.248.6] 2023-10-18T20:40:57.293 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:40:57.291+0000 7f774da52780 1 waiting for incremental sync to catch up: 2023-10-18T20:40:57.294 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.757.6, 00000000002.1177.6, 00000000001.1248.6, 00000000003.706.6, 00000000002.773.6, , 00000000001.863.6, 00000000002.232.6, 00000000005.457.6, 00000000004.248.6] 2023-10-18T20:40:57.294 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.757.6, 00000000002.1177.6, 00000000001.1248.6, 00000000003.706.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000002.232.6, 00000000005.457.6, 00000000004.248.6] 2023-10-18T20:41:02.295 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:02.293+0000 7f774da52780 1 bucket sync caught up with source: 2023-10-18T20:41:02.295 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000002.757.6, 00000000002.1177.6, 00000000001.1248.6, 00000000003.706.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000002.232.6, 00000000005.457.6, 00000000004.248.6] 2023-10-18T20:41:02.295 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000002.757.6, 00000000002.1177.6, 00000000001.1248.6, 00000000003.706.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000002.232.6, 00000000005.457.6, 00000000004.248.6] 2023-10-18T20:41:02.296 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:02.293+0000 7f774da52780 0 bucket checkpoint complete 2023-10-18T20:41:02.312 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:41:02.312 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.313 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.313 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.314 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:41:02.314 DEBUG:boto:Data: 2023-10-18T20:41:02.314 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.315 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.315 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.315 DEBUG:boto:Params: {} 2023-10-18T20:41:02.316 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:41:02.317 DEBUG:boto:Token: None 2023-10-18T20:41:02.317 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.318 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3tfQVkLQNd14TMDgFZGvIFk8S9o= 2023-10-18T20:41:02.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3tfQVkLQNd14TMDgFZGvIFk8S9o=', 'Content-Length': '0'} 2023-10-18T20:41:02.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000799f8368485c3ab4-00653042de-1af4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.323 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.324 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.324 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.324 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:41:02.325 DEBUG:boto:Data: 2023-10-18T20:41:02.325 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.326 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.326 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.326 DEBUG:boto:Params: {} 2023-10-18T20:41:02.327 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:41:02.327 DEBUG:boto:Token: None 2023-10-18T20:41:02.328 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3tfQVkLQNd14TMDgFZGvIFk8S9o= 2023-10-18T20:41:02.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3tfQVkLQNd14TMDgFZGvIFk8S9o=', 'Content-Length': '0'} 2023-10-18T20:41:02.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f05bea1e0362cf5a-00653042de-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.334 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:41:02.335 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.335 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.336 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.336 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.336 DEBUG:boto:Method: GET 2023-10-18T20:41:02.337 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.337 DEBUG:boto:Data: 2023-10-18T20:41:02.337 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.338 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.339 DEBUG:boto:Params: {} 2023-10-18T20:41:02.339 DEBUG:boto:Token: None 2023-10-18T20:41:02.339 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/?versions 2023-10-18T20:41:02.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:N2E6VWUGPTUhiN3UNL/YTRXs8JE= 2023-10-18T20:41:02.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:N2E6VWUGPTUhiN3UNL/YTRXs8JE=', 'Content-Length': '0'} 2023-10-18T20:41:02.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc43c2ea587f05e-00653042de-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.343 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:41:02.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:41:02.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-18T20:41:02.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-18T20:41:02.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:41:02.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:41:02.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:41:02.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:41:02.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-18T20:41:02.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:41:02.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:41:02.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:41:02.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:41:02.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:41:02.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:41:02.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:41:02.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:41:02.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-18T20:41:02.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:41:02.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:41:02.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:41:02.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:41:02.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-18T20:41:02.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:41:02.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:41:02.354 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:41:02.355 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.355 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.356 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.356 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.356 DEBUG:boto:Method: GET 2023-10-18T20:41:02.357 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.357 DEBUG:boto:Data: 2023-10-18T20:41:02.357 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.358 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.359 DEBUG:boto:Params: {} 2023-10-18T20:41:02.359 DEBUG:boto:Token: None 2023-10-18T20:41:02.359 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/?versions 2023-10-18T20:41:02.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:N2E6VWUGPTUhiN3UNL/YTRXs8JE= 2023-10-18T20:41:02.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:N2E6VWUGPTUhiN3UNL/YTRXs8JE=', 'Content-Length': '0'} 2023-10-18T20:41:02.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8ad1551d4006aa-00653042de-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.365 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:41:02.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:41:02.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-18T20:41:02.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-18T20:41:02.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:41:02.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:41:02.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:41:02.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:41:02.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-18T20:41:02.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:41:02.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:41:02.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:41:02.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:41:02.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:41:02.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:41:02.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:41:02.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:41:02.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-18T20:41:02.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:41:02.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:41:02.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:41:02.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:41:02.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-18T20:41:02.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:41:02.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:41:02.375 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.376 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.376 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.376 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.377 DEBUG:boto:Method: GET 2023-10-18T20:41:02.377 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.378 DEBUG:boto:Data: 2023-10-18T20:41:02.378 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.378 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.379 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.379 DEBUG:boto:Params: {} 2023-10-18T20:41:02.380 DEBUG:boto:Token: None 2023-10-18T20:41:02.380 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/?versions 2023-10-18T20:41:02.381 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:N2E6VWUGPTUhiN3UNL/YTRXs8JE= 2023-10-18T20:41:02.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:N2E6VWUGPTUhiN3UNL/YTRXs8JE=', 'Content-Length': '0'} 2023-10-18T20:41:02.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8451004fb2bea3c-00653042de-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.384 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.386 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.386 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.387 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.387 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.387 DEBUG:boto:Method: GET 2023-10-18T20:41:02.388 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:41:02.388 DEBUG:boto:Data: 2023-10-18T20:41:02.388 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.389 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.390 DEBUG:boto:Params: {} 2023-10-18T20:41:02.390 DEBUG:boto:Token: None 2023-10-18T20:41:02.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/?versions 2023-10-18T20:41:02.391 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:N2E6VWUGPTUhiN3UNL/YTRXs8JE= 2023-10-18T20:41:02.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:N2E6VWUGPTUhiN3UNL/YTRXs8JE=', 'Content-Length': '0'} 2023-10-18T20:41:02.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b6059c457ca3017-00653042de-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.396 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:41:02.398 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.398 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.399 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.399 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.400 DEBUG:boto:Method: GET 2023-10-18T20:41:02.400 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.401 DEBUG:boto:Data: 2023-10-18T20:41:02.401 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.402 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.402 DEBUG:boto:Params: {} 2023-10-18T20:41:02.403 DEBUG:boto:Token: None 2023-10-18T20:41:02.403 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LCb00ixWsRVC0nRF9qiWqqt82WU= 2023-10-18T20:41:02.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LCb00ixWsRVC0nRF9qiWqqt82WU=', 'Content-Length': '0'} 2023-10-18T20:41:02.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009caedfb74709fdcd-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.450 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.451 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.451 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.452 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.452 DEBUG:boto:Method: GET 2023-10-18T20:41:02.453 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.453 DEBUG:boto:Data: 2023-10-18T20:41:02.453 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.454 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.455 DEBUG:boto:Params: {} 2023-10-18T20:41:02.455 DEBUG:boto:Token: None 2023-10-18T20:41:02.457 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.457 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LCb00ixWsRVC0nRF9qiWqqt82WU= 2023-10-18T20:41:02.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LCb00ixWsRVC0nRF9qiWqqt82WU=', 'Content-Length': '0'} 2023-10-18T20:41:02.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e2b39aa57014b61-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.461 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.461 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.462 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.462 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.462 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.463 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.463 DEBUG:boto:Data: 2023-10-18T20:41:02.464 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.464 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.464 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.465 DEBUG:boto:Params: {} 2023-10-18T20:41:02.465 DEBUG:boto:Token: None 2023-10-18T20:41:02.466 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.466 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L7JCXpEmgLII8qTURZeDlzinLbQ= 2023-10-18T20:41:02.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L7JCXpEmgLII8qTURZeDlzinLbQ=', 'Content-Length': '0'} 2023-10-18T20:41:02.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d31edd6fd2424be-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.469 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.469 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.470 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.470 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.471 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.471 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.472 DEBUG:boto:Data: 2023-10-18T20:41:02.472 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.472 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.473 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.473 DEBUG:boto:Params: {} 2023-10-18T20:41:02.474 DEBUG:boto:Token: None 2023-10-18T20:41:02.474 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.475 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L7JCXpEmgLII8qTURZeDlzinLbQ= 2023-10-18T20:41:02.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L7JCXpEmgLII8qTURZeDlzinLbQ=', 'Content-Length': '0'} 2023-10-18T20:41:02.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000911f9dab20d474d4-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:41:02.478 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.479 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.479 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.479 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.480 DEBUG:boto:Method: GET 2023-10-18T20:41:02.480 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.481 DEBUG:boto:Data: 2023-10-18T20:41:02.481 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.482 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.482 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.482 DEBUG:boto:Params: {} 2023-10-18T20:41:02.483 DEBUG:boto:Token: None 2023-10-18T20:41:02.483 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.484 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LCb00ixWsRVC0nRF9qiWqqt82WU= 2023-10-18T20:41:02.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LCb00ixWsRVC0nRF9qiWqqt82WU=', 'Content-Length': '0'} 2023-10-18T20:41:02.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70d4e4a18ec3e86-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.488 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.488 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.489 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.489 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.489 DEBUG:boto:Method: GET 2023-10-18T20:41:02.490 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.490 DEBUG:boto:Data: 2023-10-18T20:41:02.491 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.491 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.492 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.492 DEBUG:boto:Params: {} 2023-10-18T20:41:02.493 DEBUG:boto:Token: None 2023-10-18T20:41:02.493 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:41:02.494 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LCb00ixWsRVC0nRF9qiWqqt82WU= 2023-10-18T20:41:02.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LCb00ixWsRVC0nRF9qiWqqt82WU=', 'Content-Length': '0'} 2023-10-18T20:41:02.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6c861e7e00b1c9-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.497 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.498 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.498 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.498 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.499 DEBUG:boto:Method: GET 2023-10-18T20:41:02.499 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.500 DEBUG:boto:Data: 2023-10-18T20:41:02.500 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.501 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.501 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.502 DEBUG:boto:Params: {} 2023-10-18T20:41:02.502 DEBUG:boto:Token: None 2023-10-18T20:41:02.503 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.503 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c4cc36b7632eb70-00653042de-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.507 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.508 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.509 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.509 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.509 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.510 DEBUG:boto:Method: GET 2023-10-18T20:41:02.510 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.511 DEBUG:boto:Data: 2023-10-18T20:41:02.511 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.512 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.512 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.512 DEBUG:boto:Params: {} 2023-10-18T20:41:02.513 DEBUG:boto:Token: None 2023-10-18T20:41:02.513 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.514 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfb2e212b3688870-00653042de-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.519 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.520 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.520 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.521 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.521 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:41:02.521 DEBUG:boto:Data: 2023-10-18T20:41:02.522 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.523 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.523 DEBUG:boto:Params: {} 2023-10-18T20:41:02.523 DEBUG:boto:Token: None 2023-10-18T20:41:02.524 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11 2023-10-18T20:41:02.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sb7kgttr9S8n5L8x4KEZxYv8CPQ= 2023-10-18T20:41:02.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sb7kgttr9S8n5L8x4KEZxYv8CPQ=', 'Content-Length': '0'} 2023-10-18T20:41:02.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd0d700fc671d4d-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.527 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.527 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.528 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.528 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:41:02.529 DEBUG:boto:Data: 2023-10-18T20:41:02.529 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.529 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.530 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.530 DEBUG:boto:Params: {} 2023-10-18T20:41:02.530 DEBUG:boto:Token: None 2023-10-18T20:41:02.531 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11 2023-10-18T20:41:02.531 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sb7kgttr9S8n5L8x4KEZxYv8CPQ= 2023-10-18T20:41:02.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sb7kgttr9S8n5L8x4KEZxYv8CPQ=', 'Content-Length': '0'} 2023-10-18T20:41:02.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff66a10d582c4f1-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:41:02.535 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.535 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.535 DEBUG:boto:Method: GET 2023-10-18T20:41:02.536 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:41:02.536 DEBUG:boto:Data: 2023-10-18T20:41:02.537 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.537 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.538 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.538 DEBUG:boto:Params: {} 2023-10-18T20:41:02.538 DEBUG:boto:Token: None 2023-10-18T20:41:02.539 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11 2023-10-18T20:41:02.539 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PnQ9GcaYOnJyhPhIrs3lm2dN8cg= 2023-10-18T20:41:02.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PnQ9GcaYOnJyhPhIrs3lm2dN8cg=', 'Content-Length': '0'} 2023-10-18T20:41:02.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2030a11030b664a-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.582 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:41:02.583 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:41:02.583 DEBUG:boto:Method: GET 2023-10-18T20:41:02.584 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:41:02.584 DEBUG:boto:Data: 2023-10-18T20:41:02.585 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.585 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.586 DEBUG:boto:Params: {} 2023-10-18T20:41:02.586 DEBUG:boto:Token: None 2023-10-18T20:41:02.587 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj11 2023-10-18T20:41:02.588 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PnQ9GcaYOnJyhPhIrs3lm2dN8cg= 2023-10-18T20:41:02.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PnQ9GcaYOnJyhPhIrs3lm2dN8cg=', 'Content-Length': '0'} 2023-10-18T20:41:02.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ebc146c00d17bfb-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.591 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.592 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.592 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.593 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.593 DEBUG:boto:Method: GET 2023-10-18T20:41:02.593 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.594 DEBUG:boto:Data: 2023-10-18T20:41:02.594 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.595 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.595 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.596 DEBUG:boto:Params: {} 2023-10-18T20:41:02.596 DEBUG:boto:Token: None 2023-10-18T20:41:02.596 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.597 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c23949cff557dc30-00653042de-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.601 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.602 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.602 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.603 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.603 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.603 DEBUG:boto:Method: GET 2023-10-18T20:41:02.604 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:41:02.604 DEBUG:boto:Data: 2023-10-18T20:41:02.605 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.605 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.605 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.606 DEBUG:boto:Params: {} 2023-10-18T20:41:02.606 DEBUG:boto:Token: None 2023-10-18T20:41:02.607 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.607 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbb9ccebd795d48f-00653042de-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.654 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:41:02.656 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.656 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.656 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.657 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.657 DEBUG:boto:Method: GET 2023-10-18T20:41:02.658 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.658 DEBUG:boto:Data: 2023-10-18T20:41:02.659 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.659 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.661 DEBUG:boto:Params: {} 2023-10-18T20:41:02.661 DEBUG:boto:Token: None 2023-10-18T20:41:02.662 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.663 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QBw3N7zLrKEATGv2B/kvs/foXgc= 2023-10-18T20:41:02.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QBw3N7zLrKEATGv2B/kvs/foXgc=', 'Content-Length': '0'} 2023-10-18T20:41:02.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705f03669a362aa2-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.666 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.666 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.667 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.667 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.668 DEBUG:boto:Method: GET 2023-10-18T20:41:02.668 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.668 DEBUG:boto:Data: 2023-10-18T20:41:02.669 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.669 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.670 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.670 DEBUG:boto:Params: {} 2023-10-18T20:41:02.671 DEBUG:boto:Token: None 2023-10-18T20:41:02.671 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.671 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QBw3N7zLrKEATGv2B/kvs/foXgc= 2023-10-18T20:41:02.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QBw3N7zLrKEATGv2B/kvs/foXgc=', 'Content-Length': '0'} 2023-10-18T20:41:02.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085eaa143465a3b45-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.675 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.675 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.676 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.676 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.676 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.677 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.677 DEBUG:boto:Data: 2023-10-18T20:41:02.678 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.678 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.679 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.679 DEBUG:boto:Params: {} 2023-10-18T20:41:02.679 DEBUG:boto:Token: None 2023-10-18T20:41:02.680 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.680 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pM0SOD1j68bwZW17ptC6PKGw0lg= 2023-10-18T20:41:02.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pM0SOD1j68bwZW17ptC6PKGw0lg=', 'Content-Length': '0'} 2023-10-18T20:41:02.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000369a8dd8df43c708-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.683 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.683 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.684 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.684 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.684 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.685 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.685 DEBUG:boto:Data: 2023-10-18T20:41:02.686 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.687 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.687 DEBUG:boto:Params: {} 2023-10-18T20:41:02.687 DEBUG:boto:Token: None 2023-10-18T20:41:02.688 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pM0SOD1j68bwZW17ptC6PKGw0lg= 2023-10-18T20:41:02.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pM0SOD1j68bwZW17ptC6PKGw0lg=', 'Content-Length': '0'} 2023-10-18T20:41:02.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005307b0566f769844-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:41:02.692 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.692 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.693 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.693 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.694 DEBUG:boto:Method: GET 2023-10-18T20:41:02.694 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.694 DEBUG:boto:Data: 2023-10-18T20:41:02.695 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.695 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.696 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.696 DEBUG:boto:Params: {} 2023-10-18T20:41:02.697 DEBUG:boto:Token: None 2023-10-18T20:41:02.697 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.698 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QBw3N7zLrKEATGv2B/kvs/foXgc= 2023-10-18T20:41:02.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QBw3N7zLrKEATGv2B/kvs/foXgc=', 'Content-Length': '0'} 2023-10-18T20:41:02.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ee0a7b5e2c50b1e-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.701 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.701 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.702 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.702 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.703 DEBUG:boto:Method: GET 2023-10-18T20:41:02.703 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.703 DEBUG:boto:Data: 2023-10-18T20:41:02.704 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.704 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.705 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.705 DEBUG:boto:Params: {} 2023-10-18T20:41:02.705 DEBUG:boto:Token: None 2023-10-18T20:41:02.706 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:41:02.706 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QBw3N7zLrKEATGv2B/kvs/foXgc= 2023-10-18T20:41:02.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QBw3N7zLrKEATGv2B/kvs/foXgc=', 'Content-Length': '0'} 2023-10-18T20:41:02.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079fa62ab62e00d6f-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.750 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.751 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.751 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.751 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.752 DEBUG:boto:Method: GET 2023-10-18T20:41:02.752 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.753 DEBUG:boto:Data: 2023-10-18T20:41:02.753 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.754 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.754 DEBUG:boto:Params: {} 2023-10-18T20:41:02.755 DEBUG:boto:Token: None 2023-10-18T20:41:02.755 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d214d197e148c5ea-00653042de-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.759 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.760 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.760 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.761 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.761 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.762 DEBUG:boto:Method: GET 2023-10-18T20:41:02.762 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.762 DEBUG:boto:Data: 2023-10-18T20:41:02.763 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.764 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.764 DEBUG:boto:Params: {} 2023-10-18T20:41:02.765 DEBUG:boto:Token: None 2023-10-18T20:41:02.765 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.766 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057ce7ab58a7bb9a0-00653042de-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.770 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.771 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.771 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.772 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.772 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:41:02.773 DEBUG:boto:Data: 2023-10-18T20:41:02.773 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.773 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.774 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.774 DEBUG:boto:Params: {} 2023-10-18T20:41:02.775 DEBUG:boto:Token: None 2023-10-18T20:41:02.775 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115 2023-10-18T20:41:02.776 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r450ObHx8UrJQew+tadcmY3i/Wk= 2023-10-18T20:41:02.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r450ObHx8UrJQew+tadcmY3i/Wk=', 'Content-Length': '0'} 2023-10-18T20:41:02.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000116b75fb066fdf12-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.779 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.779 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.779 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.780 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:41:02.780 DEBUG:boto:Data: 2023-10-18T20:41:02.780 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.781 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.782 DEBUG:boto:Params: {} 2023-10-18T20:41:02.782 DEBUG:boto:Token: None 2023-10-18T20:41:02.782 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115 2023-10-18T20:41:02.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r450ObHx8UrJQew+tadcmY3i/Wk= 2023-10-18T20:41:02.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r450ObHx8UrJQew+tadcmY3i/Wk=', 'Content-Length': '0'} 2023-10-18T20:41:02.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d86126c1c2ede884-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:41:02.787 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.787 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.787 DEBUG:boto:Method: GET 2023-10-18T20:41:02.788 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:41:02.788 DEBUG:boto:Data: 2023-10-18T20:41:02.789 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.789 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.790 DEBUG:boto:Params: {} 2023-10-18T20:41:02.790 DEBUG:boto:Token: None 2023-10-18T20:41:02.791 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115 2023-10-18T20:41:02.791 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pQo/CbKtecDDY0AkXiKluvpnNag= 2023-10-18T20:41:02.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pQo/CbKtecDDY0AkXiKluvpnNag=', 'Content-Length': '0'} 2023-10-18T20:41:02.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000022e96d2f8a5d5e1-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.834 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:41:02.835 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:41:02.835 DEBUG:boto:Method: GET 2023-10-18T20:41:02.836 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:41:02.836 DEBUG:boto:Data: 2023-10-18T20:41:02.836 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.837 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.837 DEBUG:boto:Params: {} 2023-10-18T20:41:02.838 DEBUG:boto:Token: None 2023-10-18T20:41:02.838 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj115 2023-10-18T20:41:02.839 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pQo/CbKtecDDY0AkXiKluvpnNag= 2023-10-18T20:41:02.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pQo/CbKtecDDY0AkXiKluvpnNag=', 'Content-Length': '0'} 2023-10-18T20:41:02.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd6bf4c11f76e5fc-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.842 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.843 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.843 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.843 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.844 DEBUG:boto:Method: GET 2023-10-18T20:41:02.844 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.844 DEBUG:boto:Data: 2023-10-18T20:41:02.845 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.845 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.846 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.846 DEBUG:boto:Params: {} 2023-10-18T20:41:02.846 DEBUG:boto:Token: None 2023-10-18T20:41:02.847 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.847 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba0774b296e1df1a-00653042de-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.850 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.851 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.851 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.852 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.852 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.853 DEBUG:boto:Method: GET 2023-10-18T20:41:02.853 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:41:02.853 DEBUG:boto:Data: 2023-10-18T20:41:02.854 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.854 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.855 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.855 DEBUG:boto:Params: {} 2023-10-18T20:41:02.855 DEBUG:boto:Token: None 2023-10-18T20:41:02.856 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.856 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000674cdec9323af34d-00653042de-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.861 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:41:02.862 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.862 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.863 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.863 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.865 DEBUG:boto:Method: GET 2023-10-18T20:41:02.865 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.866 DEBUG:boto:Data: 2023-10-18T20:41:02.866 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.866 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.867 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.867 DEBUG:boto:Params: {} 2023-10-18T20:41:02.868 DEBUG:boto:Token: None 2023-10-18T20:41:02.868 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.869 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kECgM34wI8PnXEbf5mwYMtUghWA= 2023-10-18T20:41:02.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kECgM34wI8PnXEbf5mwYMtUghWA=', 'Content-Length': '0'} 2023-10-18T20:41:02.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091309369d37c3b1c-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.872 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.872 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.873 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.873 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.873 DEBUG:boto:Method: GET 2023-10-18T20:41:02.874 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.874 DEBUG:boto:Data: 2023-10-18T20:41:02.875 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.876 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.876 DEBUG:boto:Params: {} 2023-10-18T20:41:02.876 DEBUG:boto:Token: None 2023-10-18T20:41:02.877 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.877 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kECgM34wI8PnXEbf5mwYMtUghWA= 2023-10-18T20:41:02.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kECgM34wI8PnXEbf5mwYMtUghWA=', 'Content-Length': '0'} 2023-10-18T20:41:02.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7c06b9074adc91f-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.881 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.881 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.881 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.882 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.882 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.883 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.883 DEBUG:boto:Data: 2023-10-18T20:41:02.883 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.884 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.884 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.884 DEBUG:boto:Params: {} 2023-10-18T20:41:02.885 DEBUG:boto:Token: None 2023-10-18T20:41:02.885 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.886 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nVBMY8V6iKMT43Msxem789NyQnw= 2023-10-18T20:41:02.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nVBMY8V6iKMT43Msxem789NyQnw=', 'Content-Length': '0'} 2023-10-18T20:41:02.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793d8dbd0b54743c-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.889 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.889 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.890 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.890 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.890 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.891 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.891 DEBUG:boto:Data: 2023-10-18T20:41:02.891 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.892 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.892 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.892 DEBUG:boto:Params: {} 2023-10-18T20:41:02.893 DEBUG:boto:Token: None 2023-10-18T20:41:02.893 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.894 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nVBMY8V6iKMT43Msxem789NyQnw= 2023-10-18T20:41:02.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nVBMY8V6iKMT43Msxem789NyQnw=', 'Content-Length': '0'} 2023-10-18T20:41:02.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5215c270e0a6ac-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:41:02.897 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.897 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.898 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.898 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.899 DEBUG:boto:Method: GET 2023-10-18T20:41:02.899 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.899 DEBUG:boto:Data: 2023-10-18T20:41:02.900 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.900 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.901 DEBUG:boto:Params: {} 2023-10-18T20:41:02.901 DEBUG:boto:Token: None 2023-10-18T20:41:02.902 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.902 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kECgM34wI8PnXEbf5mwYMtUghWA= 2023-10-18T20:41:02.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kECgM34wI8PnXEbf5mwYMtUghWA=', 'Content-Length': '0'} 2023-10-18T20:41:02.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b6350c8dd0deb7f-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.905 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.905 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.906 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.906 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.907 DEBUG:boto:Method: GET 2023-10-18T20:41:02.907 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.908 DEBUG:boto:Data: 2023-10-18T20:41:02.908 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.908 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.909 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.909 DEBUG:boto:Params: {} 2023-10-18T20:41:02.910 DEBUG:boto:Token: None 2023-10-18T20:41:02.910 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:41:02.911 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kECgM34wI8PnXEbf5mwYMtUghWA= 2023-10-18T20:41:02.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kECgM34wI8PnXEbf5mwYMtUghWA=', 'Content-Length': '0'} 2023-10-18T20:41:02.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000622fcafad6b088dd-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.914 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.914 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.914 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:41:02.915 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:41:02.915 DEBUG:boto:Method: GET 2023-10-18T20:41:02.916 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:41:02.916 DEBUG:boto:Data: 2023-10-18T20:41:02.917 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.917 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.918 DEBUG:boto:Params: {} 2023-10-18T20:41:02.918 DEBUG:boto:Token: None 2023-10-18T20:41:02.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.919 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a187e74bbca73133-00653042de-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.923 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.923 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:02.924 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:02.924 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:41:02.925 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:41:02.925 DEBUG:boto:Method: GET 2023-10-18T20:41:02.925 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:41:02.926 DEBUG:boto:Data: 2023-10-18T20:41:02.926 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.926 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.927 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.927 DEBUG:boto:Params: {} 2023-10-18T20:41:02.928 DEBUG:boto:Token: None 2023-10-18T20:41:02.928 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/ 2023-10-18T20:41:02.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4= 2023-10-18T20:41:02.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PRKI604pTJS/ssILg+oU/YUSTx4=', 'Content-Length': '0'} 2023-10-18T20:41:02.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa0938aefed8884-00653042de-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.934 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:02.934 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.935 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.935 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.935 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:41:02.936 DEBUG:boto:Data: 2023-10-18T20:41:02.936 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.937 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.937 DEBUG:boto:Params: {} 2023-10-18T20:41:02.938 DEBUG:boto:Token: None 2023-10-18T20:41:02.938 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj116 2023-10-18T20:41:02.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5BsbPedKviCASTPhEcpYx9FmwUQ= 2023-10-18T20:41:02.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5BsbPedKviCASTPhEcpYx9FmwUQ=', 'Content-Length': '0'} 2023-10-18T20:41:02.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82ef30003eeac3c-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.942 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.942 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.943 DEBUG:boto:Method: HEAD 2023-10-18T20:41:02.943 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:41:02.944 DEBUG:boto:Data: 2023-10-18T20:41:02.944 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.944 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:02.945 DEBUG:boto:Port: 8000 2023-10-18T20:41:02.945 DEBUG:boto:Params: {} 2023-10-18T20:41:02.946 DEBUG:boto:Token: None 2023-10-18T20:41:02.946 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj116 2023-10-18T20:41:02.946 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5BsbPedKviCASTPhEcpYx9FmwUQ= 2023-10-18T20:41:02.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5BsbPedKviCASTPhEcpYx9FmwUQ=', 'Content-Length': '0'} 2023-10-18T20:41:02.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceecac79ecafae01-00653042de-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:41:02.950 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.950 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.951 DEBUG:boto:Method: GET 2023-10-18T20:41:02.951 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:41:02.952 DEBUG:boto:Data: 2023-10-18T20:41:02.952 DEBUG:boto:Headers: {} 2023-10-18T20:41:02.953 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:02.953 DEBUG:boto:Port: 8001 2023-10-18T20:41:02.953 DEBUG:boto:Params: {} 2023-10-18T20:41:02.954 DEBUG:boto:Token: None 2023-10-18T20:41:02.954 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:02 GMT /bofrnf-59/obj116 2023-10-18T20:41:02.955 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xF2idzNot95UUhEKF7KRV6beVKs= 2023-10-18T20:41:02.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:02 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xF2idzNot95UUhEKF7KRV6beVKs=', 'Content-Length': '0'} 2023-10-18T20:41:02.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfa4da62416a59bd-00653042de-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:02.998 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:41:02.999 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:41:02.999 DEBUG:boto:Method: GET 2023-10-18T20:41:03.000 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:41:03.000 DEBUG:boto:Data: 2023-10-18T20:41:03.000 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.001 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.002 DEBUG:boto:Params: {} 2023-10-18T20:41:03.002 DEBUG:boto:Token: None 2023-10-18T20:41:03.003 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj116 2023-10-18T20:41:03.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pq7mFLL9kHwZ5AtVlf2V0Ascwhs= 2023-10-18T20:41:03.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pq7mFLL9kHwZ5AtVlf2V0Ascwhs=', 'Content-Length': '0'} 2023-10-18T20:41:03.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b3136272af04bdf-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.006 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.007 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.007 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:41:03.007 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:41:03.008 DEBUG:boto:Method: GET 2023-10-18T20:41:03.008 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:41:03.009 DEBUG:boto:Data: 2023-10-18T20:41:03.009 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.009 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.010 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.010 DEBUG:boto:Params: {} 2023-10-18T20:41:03.011 DEBUG:boto:Token: None 2023-10-18T20:41:03.011 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036363e7ceda8c209-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.015 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.016 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.016 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.016 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:41:03.017 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:41:03.017 DEBUG:boto:Method: GET 2023-10-18T20:41:03.017 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:41:03.018 DEBUG:boto:Data: 2023-10-18T20:41:03.018 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.019 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.019 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.020 DEBUG:boto:Params: {} 2023-10-18T20:41:03.020 DEBUG:boto:Token: None 2023-10-18T20:41:03.021 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.021 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000685c8a1085680d8f-00653042df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.026 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:41:03.027 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.027 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.028 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.028 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.029 DEBUG:boto:Method: GET 2023-10-18T20:41:03.029 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.029 DEBUG:boto:Data: 2023-10-18T20:41:03.030 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.030 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.031 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.031 DEBUG:boto:Params: {} 2023-10-18T20:41:03.031 DEBUG:boto:Token: None 2023-10-18T20:41:03.032 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Na9vTG9xy9hDAcnBYV+w0gYNEfw= 2023-10-18T20:41:03.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Na9vTG9xy9hDAcnBYV+w0gYNEfw=', 'Content-Length': '0'} 2023-10-18T20:41:03.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004054d7d482f96982-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.036 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.036 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.037 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.037 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.037 DEBUG:boto:Method: GET 2023-10-18T20:41:03.038 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.038 DEBUG:boto:Data: 2023-10-18T20:41:03.039 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.039 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.039 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.040 DEBUG:boto:Params: {} 2023-10-18T20:41:03.040 DEBUG:boto:Token: None 2023-10-18T20:41:03.041 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.041 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Na9vTG9xy9hDAcnBYV+w0gYNEfw= 2023-10-18T20:41:03.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Na9vTG9xy9hDAcnBYV+w0gYNEfw=', 'Content-Length': '0'} 2023-10-18T20:41:03.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a534376b7d3c3c-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.044 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.045 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.045 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.045 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.046 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.046 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.046 DEBUG:boto:Data: 2023-10-18T20:41:03.047 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.047 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.047 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.048 DEBUG:boto:Params: {} 2023-10-18T20:41:03.048 DEBUG:boto:Token: None 2023-10-18T20:41:03.049 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.049 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx6dezO/AffX++ADtI48sC64++o= 2023-10-18T20:41:03.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx6dezO/AffX++ADtI48sC64++o=', 'Content-Length': '0'} 2023-10-18T20:41:03.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091e44f1b007c0ffe-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.052 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.053 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.053 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.053 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.054 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.054 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.054 DEBUG:boto:Data: 2023-10-18T20:41:03.055 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.055 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.056 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.056 DEBUG:boto:Params: {} 2023-10-18T20:41:03.056 DEBUG:boto:Token: None 2023-10-18T20:41:03.057 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.057 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Jx6dezO/AffX++ADtI48sC64++o= 2023-10-18T20:41:03.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Jx6dezO/AffX++ADtI48sC64++o=', 'Content-Length': '0'} 2023-10-18T20:41:03.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b250300a93c606a0-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:41:03.061 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.061 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.062 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.062 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.063 DEBUG:boto:Method: GET 2023-10-18T20:41:03.063 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.064 DEBUG:boto:Data: 2023-10-18T20:41:03.064 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.065 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.065 DEBUG:boto:Params: {} 2023-10-18T20:41:03.066 DEBUG:boto:Token: None 2023-10-18T20:41:03.066 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.069 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Na9vTG9xy9hDAcnBYV+w0gYNEfw= 2023-10-18T20:41:03.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Na9vTG9xy9hDAcnBYV+w0gYNEfw=', 'Content-Length': '0'} 2023-10-18T20:41:03.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000879a7a8f3f60f0f8-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.114 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.115 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.115 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.116 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.116 DEBUG:boto:Method: GET 2023-10-18T20:41:03.116 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.117 DEBUG:boto:Data: 2023-10-18T20:41:03.117 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.117 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.118 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.118 DEBUG:boto:Params: {} 2023-10-18T20:41:03.119 DEBUG:boto:Token: None 2023-10-18T20:41:03.119 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:41:03.120 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Na9vTG9xy9hDAcnBYV+w0gYNEfw= 2023-10-18T20:41:03.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Na9vTG9xy9hDAcnBYV+w0gYNEfw=', 'Content-Length': '0'} 2023-10-18T20:41:03.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006215e4658e35a2b-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.123 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.123 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.124 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.124 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.125 DEBUG:boto:Method: GET 2023-10-18T20:41:03.125 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.125 DEBUG:boto:Data: 2023-10-18T20:41:03.126 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.126 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.127 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.127 DEBUG:boto:Params: {} 2023-10-18T20:41:03.128 DEBUG:boto:Token: None 2023-10-18T20:41:03.128 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.129 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d639e99d0f4c4a9c-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.132 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.133 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.133 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.134 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.134 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.135 DEBUG:boto:Method: GET 2023-10-18T20:41:03.135 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.136 DEBUG:boto:Data: 2023-10-18T20:41:03.136 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.136 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.137 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.137 DEBUG:boto:Params: {} 2023-10-18T20:41:03.138 DEBUG:boto:Token: None 2023-10-18T20:41:03.138 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.139 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbd2afc7baef7cc-00653042df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.143 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.144 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.144 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.145 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.145 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:41:03.146 DEBUG:boto:Data: 2023-10-18T20:41:03.146 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.146 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.147 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.147 DEBUG:boto:Params: {} 2023-10-18T20:41:03.148 DEBUG:boto:Token: None 2023-10-18T20:41:03.148 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145 2023-10-18T20:41:03.148 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eXBzzBngwESZJFLe8J41Ruj2dbk= 2023-10-18T20:41:03.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eXBzzBngwESZJFLe8J41Ruj2dbk=', 'Content-Length': '0'} 2023-10-18T20:41:03.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae36433ea01b2b73-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.152 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.152 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.152 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.153 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:41:03.153 DEBUG:boto:Data: 2023-10-18T20:41:03.154 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.154 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.155 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.155 DEBUG:boto:Params: {} 2023-10-18T20:41:03.155 DEBUG:boto:Token: None 2023-10-18T20:41:03.156 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145 2023-10-18T20:41:03.156 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eXBzzBngwESZJFLe8J41Ruj2dbk= 2023-10-18T20:41:03.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eXBzzBngwESZJFLe8J41Ruj2dbk=', 'Content-Length': '0'} 2023-10-18T20:41:03.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d44a5d7b6b5e885-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:41:03.160 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.160 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.160 DEBUG:boto:Method: GET 2023-10-18T20:41:03.161 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:41:03.161 DEBUG:boto:Data: 2023-10-18T20:41:03.162 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.162 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.163 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.163 DEBUG:boto:Params: {} 2023-10-18T20:41:03.163 DEBUG:boto:Token: None 2023-10-18T20:41:03.164 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145 2023-10-18T20:41:03.164 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gECzUGqf21LvzYq5TYBIpacq2OQ= 2023-10-18T20:41:03.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gECzUGqf21LvzYq5TYBIpacq2OQ=', 'Content-Length': '0'} 2023-10-18T20:41:03.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674f966c16fc716e-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.210 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:41:03.211 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:41:03.211 DEBUG:boto:Method: GET 2023-10-18T20:41:03.212 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:41:03.212 DEBUG:boto:Data: 2023-10-18T20:41:03.212 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.213 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.213 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.214 DEBUG:boto:Params: {} 2023-10-18T20:41:03.214 DEBUG:boto:Token: None 2023-10-18T20:41:03.215 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj145 2023-10-18T20:41:03.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gECzUGqf21LvzYq5TYBIpacq2OQ= 2023-10-18T20:41:03.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gECzUGqf21LvzYq5TYBIpacq2OQ=', 'Content-Length': '0'} 2023-10-18T20:41:03.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38c6ce7c255ff04-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.218 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.219 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.219 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.220 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.220 DEBUG:boto:Method: GET 2023-10-18T20:41:03.220 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.221 DEBUG:boto:Data: 2023-10-18T20:41:03.221 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.222 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.222 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.222 DEBUG:boto:Params: {} 2023-10-18T20:41:03.223 DEBUG:boto:Token: None 2023-10-18T20:41:03.223 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.224 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062ecc835c2e6edad-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.228 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.228 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.229 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.229 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.230 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.230 DEBUG:boto:Method: GET 2023-10-18T20:41:03.230 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:41:03.231 DEBUG:boto:Data: 2023-10-18T20:41:03.231 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.232 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.232 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.232 DEBUG:boto:Params: {} 2023-10-18T20:41:03.233 DEBUG:boto:Token: None 2023-10-18T20:41:03.233 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.234 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005871d42250765aca-00653042df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.238 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:41:03.239 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.240 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.240 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.241 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.241 DEBUG:boto:Method: GET 2023-10-18T20:41:03.241 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.242 DEBUG:boto:Data: 2023-10-18T20:41:03.242 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.243 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.243 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.243 DEBUG:boto:Params: {} 2023-10-18T20:41:03.244 DEBUG:boto:Token: None 2023-10-18T20:41:03.244 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.245 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:53ob11Mp3+CiEkYa9gP4k5RNhWM= 2023-10-18T20:41:03.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:53ob11Mp3+CiEkYa9gP4k5RNhWM=', 'Content-Length': '0'} 2023-10-18T20:41:03.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df196462fbb922f2-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.248 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.248 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.249 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.249 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.250 DEBUG:boto:Method: GET 2023-10-18T20:41:03.250 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.250 DEBUG:boto:Data: 2023-10-18T20:41:03.251 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.251 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.252 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.252 DEBUG:boto:Params: {} 2023-10-18T20:41:03.252 DEBUG:boto:Token: None 2023-10-18T20:41:03.253 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.253 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:53ob11Mp3+CiEkYa9gP4k5RNhWM= 2023-10-18T20:41:03.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:53ob11Mp3+CiEkYa9gP4k5RNhWM=', 'Content-Length': '0'} 2023-10-18T20:41:03.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f562fed2971cb8a-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.256 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.257 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.257 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.258 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.258 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.259 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.259 DEBUG:boto:Data: 2023-10-18T20:41:03.259 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.260 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.260 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.261 DEBUG:boto:Params: {} 2023-10-18T20:41:03.261 DEBUG:boto:Token: None 2023-10-18T20:41:03.261 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.262 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHJb5pVH8Qpb5IEafRKjBgRQqdU= 2023-10-18T20:41:03.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHJb5pVH8Qpb5IEafRKjBgRQqdU=', 'Content-Length': '0'} 2023-10-18T20:41:03.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005188a8c4ffdef22c-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.265 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.265 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.266 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.266 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.267 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.267 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.267 DEBUG:boto:Data: 2023-10-18T20:41:03.268 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.268 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.269 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.269 DEBUG:boto:Params: {} 2023-10-18T20:41:03.269 DEBUG:boto:Token: None 2023-10-18T20:41:03.270 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.270 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHJb5pVH8Qpb5IEafRKjBgRQqdU= 2023-10-18T20:41:03.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHJb5pVH8Qpb5IEafRKjBgRQqdU=', 'Content-Length': '0'} 2023-10-18T20:41:03.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cb8761e31c52605-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:41:03.276 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.276 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.276 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.277 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.277 DEBUG:boto:Method: GET 2023-10-18T20:41:03.278 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.278 DEBUG:boto:Data: 2023-10-18T20:41:03.279 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.280 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.280 DEBUG:boto:Params: {} 2023-10-18T20:41:03.280 DEBUG:boto:Token: None 2023-10-18T20:41:03.281 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.282 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:53ob11Mp3+CiEkYa9gP4k5RNhWM= 2023-10-18T20:41:03.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:53ob11Mp3+CiEkYa9gP4k5RNhWM=', 'Content-Length': '0'} 2023-10-18T20:41:03.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070f5fd4387b67c1f-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.330 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.331 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.331 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.331 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.332 DEBUG:boto:Method: GET 2023-10-18T20:41:03.332 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.333 DEBUG:boto:Data: 2023-10-18T20:41:03.333 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.334 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.334 DEBUG:boto:Params: {} 2023-10-18T20:41:03.335 DEBUG:boto:Token: None 2023-10-18T20:41:03.336 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:41:03.336 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:53ob11Mp3+CiEkYa9gP4k5RNhWM= 2023-10-18T20:41:03.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:53ob11Mp3+CiEkYa9gP4k5RNhWM=', 'Content-Length': '0'} 2023-10-18T20:41:03.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cefb6b387c8e4ca1-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.339 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.340 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.340 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.341 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.341 DEBUG:boto:Method: GET 2023-10-18T20:41:03.342 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.342 DEBUG:boto:Data: 2023-10-18T20:41:03.343 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.343 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.344 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.344 DEBUG:boto:Params: {} 2023-10-18T20:41:03.345 DEBUG:boto:Token: None 2023-10-18T20:41:03.345 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.346 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a96ba3160d942154-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.349 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.350 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.350 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.351 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.351 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.352 DEBUG:boto:Method: GET 2023-10-18T20:41:03.352 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.353 DEBUG:boto:Data: 2023-10-18T20:41:03.353 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.353 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.354 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.354 DEBUG:boto:Params: {} 2023-10-18T20:41:03.355 DEBUG:boto:Token: None 2023-10-18T20:41:03.355 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6bd23747bd39565-00653042df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.402 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.403 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.403 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.404 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.404 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:41:03.404 DEBUG:boto:Data: 2023-10-18T20:41:03.405 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.405 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.406 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.406 DEBUG:boto:Params: {} 2023-10-18T20:41:03.406 DEBUG:boto:Token: None 2023-10-18T20:41:03.407 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17 2023-10-18T20:41:03.407 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hj4gGhC5jiNU+qaYCBVml6mRXDE= 2023-10-18T20:41:03.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hj4gGhC5jiNU+qaYCBVml6mRXDE=', 'Content-Length': '0'} 2023-10-18T20:41:03.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000493f0138e9d7d734-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.410 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.410 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.411 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.411 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:41:03.412 DEBUG:boto:Data: 2023-10-18T20:41:03.412 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.412 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.413 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.413 DEBUG:boto:Params: {} 2023-10-18T20:41:03.413 DEBUG:boto:Token: None 2023-10-18T20:41:03.414 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17 2023-10-18T20:41:03.414 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hj4gGhC5jiNU+qaYCBVml6mRXDE= 2023-10-18T20:41:03.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hj4gGhC5jiNU+qaYCBVml6mRXDE=', 'Content-Length': '0'} 2023-10-18T20:41:03.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008169dce094a30aae-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:41:03.418 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.418 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.418 DEBUG:boto:Method: GET 2023-10-18T20:41:03.419 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:41:03.419 DEBUG:boto:Data: 2023-10-18T20:41:03.420 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.420 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.421 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.421 DEBUG:boto:Params: {} 2023-10-18T20:41:03.422 DEBUG:boto:Token: None 2023-10-18T20:41:03.422 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17 2023-10-18T20:41:03.423 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBgRD5ydyIrDPeOqbLMJUdlrRTM= 2023-10-18T20:41:03.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBgRD5ydyIrDPeOqbLMJUdlrRTM=', 'Content-Length': '0'} 2023-10-18T20:41:03.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682e7acfe649a33b-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.426 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:41:03.426 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:41:03.427 DEBUG:boto:Method: GET 2023-10-18T20:41:03.427 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:41:03.428 DEBUG:boto:Data: 2023-10-18T20:41:03.428 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.428 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.429 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.429 DEBUG:boto:Params: {} 2023-10-18T20:41:03.430 DEBUG:boto:Token: None 2023-10-18T20:41:03.430 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj17 2023-10-18T20:41:03.430 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBgRD5ydyIrDPeOqbLMJUdlrRTM= 2023-10-18T20:41:03.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBgRD5ydyIrDPeOqbLMJUdlrRTM=', 'Content-Length': '0'} 2023-10-18T20:41:03.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000757c1cabf88e0e78-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.434 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.434 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.435 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.435 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.435 DEBUG:boto:Method: GET 2023-10-18T20:41:03.436 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.436 DEBUG:boto:Data: 2023-10-18T20:41:03.437 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.437 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.437 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.438 DEBUG:boto:Params: {} 2023-10-18T20:41:03.438 DEBUG:boto:Token: None 2023-10-18T20:41:03.439 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.439 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e5f571c71231421-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.443 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.443 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.444 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.444 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.445 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.445 DEBUG:boto:Method: GET 2023-10-18T20:41:03.446 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:41:03.446 DEBUG:boto:Data: 2023-10-18T20:41:03.446 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.447 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.447 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.448 DEBUG:boto:Params: {} 2023-10-18T20:41:03.448 DEBUG:boto:Token: None 2023-10-18T20:41:03.449 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.449 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3df6dca73c7b570-00653042df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.453 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:41:03.454 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.455 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.455 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.456 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.456 DEBUG:boto:Method: GET 2023-10-18T20:41:03.456 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.457 DEBUG:boto:Data: 2023-10-18T20:41:03.457 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.458 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.458 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.459 DEBUG:boto:Params: {} 2023-10-18T20:41:03.459 DEBUG:boto:Token: None 2023-10-18T20:41:03.460 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/yJkoUEntyNUbFjvOHv41MDAr/w= 2023-10-18T20:41:03.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/yJkoUEntyNUbFjvOHv41MDAr/w=', 'Content-Length': '0'} 2023-10-18T20:41:03.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae58ff2c70504863-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.463 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.464 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.464 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.464 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.465 DEBUG:boto:Method: GET 2023-10-18T20:41:03.465 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.466 DEBUG:boto:Data: 2023-10-18T20:41:03.466 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.466 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.467 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.467 DEBUG:boto:Params: {} 2023-10-18T20:41:03.468 DEBUG:boto:Token: None 2023-10-18T20:41:03.468 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.468 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/yJkoUEntyNUbFjvOHv41MDAr/w= 2023-10-18T20:41:03.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/yJkoUEntyNUbFjvOHv41MDAr/w=', 'Content-Length': '0'} 2023-10-18T20:41:03.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023072401477d9283-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.471 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.472 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.472 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.472 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.473 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.473 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.473 DEBUG:boto:Data: 2023-10-18T20:41:03.474 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.475 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.477 DEBUG:boto:Params: {} 2023-10-18T20:41:03.478 DEBUG:boto:Token: None 2023-10-18T20:41:03.478 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.479 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gu0HB8vbd5vOr3LPzG2mzY2QmM0= 2023-10-18T20:41:03.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gu0HB8vbd5vOr3LPzG2mzY2QmM0=', 'Content-Length': '0'} 2023-10-18T20:41:03.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000144747aad6478ee1-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.482 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.482 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.483 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.483 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.483 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.484 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.484 DEBUG:boto:Data: 2023-10-18T20:41:03.484 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.485 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.485 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.485 DEBUG:boto:Params: {} 2023-10-18T20:41:03.486 DEBUG:boto:Token: None 2023-10-18T20:41:03.486 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.487 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gu0HB8vbd5vOr3LPzG2mzY2QmM0= 2023-10-18T20:41:03.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gu0HB8vbd5vOr3LPzG2mzY2QmM0=', 'Content-Length': '0'} 2023-10-18T20:41:03.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aca19e411ab5c00d-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:41:03.490 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.490 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.491 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.491 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.492 DEBUG:boto:Method: GET 2023-10-18T20:41:03.492 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.492 DEBUG:boto:Data: 2023-10-18T20:41:03.493 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.493 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.494 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.494 DEBUG:boto:Params: {} 2023-10-18T20:41:03.494 DEBUG:boto:Token: None 2023-10-18T20:41:03.495 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.495 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/yJkoUEntyNUbFjvOHv41MDAr/w= 2023-10-18T20:41:03.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/yJkoUEntyNUbFjvOHv41MDAr/w=', 'Content-Length': '0'} 2023-10-18T20:41:03.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9fc38c153f63865-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.538 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.539 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.539 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.540 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.540 DEBUG:boto:Method: GET 2023-10-18T20:41:03.540 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.541 DEBUG:boto:Data: 2023-10-18T20:41:03.541 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.542 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.542 DEBUG:boto:Params: {} 2023-10-18T20:41:03.543 DEBUG:boto:Token: None 2023-10-18T20:41:03.543 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:41:03.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/yJkoUEntyNUbFjvOHv41MDAr/w= 2023-10-18T20:41:03.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/yJkoUEntyNUbFjvOHv41MDAr/w=', 'Content-Length': '0'} 2023-10-18T20:41:03.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004701bdd6eaa73494-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.546 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.547 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.547 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.548 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.548 DEBUG:boto:Method: GET 2023-10-18T20:41:03.549 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.549 DEBUG:boto:Data: 2023-10-18T20:41:03.549 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.550 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.551 DEBUG:boto:Params: {} 2023-10-18T20:41:03.551 DEBUG:boto:Token: None 2023-10-18T20:41:03.552 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.552 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a0f84fc598bffa5-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.555 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.556 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.557 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.557 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.557 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.558 DEBUG:boto:Method: GET 2023-10-18T20:41:03.558 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.559 DEBUG:boto:Data: 2023-10-18T20:41:03.559 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.560 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.560 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.561 DEBUG:boto:Params: {} 2023-10-18T20:41:03.561 DEBUG:boto:Token: None 2023-10-18T20:41:03.561 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.562 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f6f7680601158f-00653042df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.567 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.567 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.568 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.568 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.568 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:41:03.569 DEBUG:boto:Data: 2023-10-18T20:41:03.569 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.569 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.570 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.570 DEBUG:boto:Params: {} 2023-10-18T20:41:03.571 DEBUG:boto:Token: None 2023-10-18T20:41:03.571 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19 2023-10-18T20:41:03.571 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yfQjS5H1J1yUVuFIji7gEs81wC4= 2023-10-18T20:41:03.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yfQjS5H1J1yUVuFIji7gEs81wC4=', 'Content-Length': '0'} 2023-10-18T20:41:03.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e54cca429695b3-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.574 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.575 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.575 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.575 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:41:03.576 DEBUG:boto:Data: 2023-10-18T20:41:03.576 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.576 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.577 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.577 DEBUG:boto:Params: {} 2023-10-18T20:41:03.577 DEBUG:boto:Token: None 2023-10-18T20:41:03.578 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19 2023-10-18T20:41:03.578 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yfQjS5H1J1yUVuFIji7gEs81wC4= 2023-10-18T20:41:03.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yfQjS5H1J1yUVuFIji7gEs81wC4=', 'Content-Length': '0'} 2023-10-18T20:41:03.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34350c223cb52bb-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:41:03.581 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.582 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.582 DEBUG:boto:Method: GET 2023-10-18T20:41:03.583 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:41:03.583 DEBUG:boto:Data: 2023-10-18T20:41:03.584 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.584 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.584 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.585 DEBUG:boto:Params: {} 2023-10-18T20:41:03.585 DEBUG:boto:Token: None 2023-10-18T20:41:03.585 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19 2023-10-18T20:41:03.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LaYoJ/crNdVq6m8nv7Tf0NGGADc= 2023-10-18T20:41:03.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LaYoJ/crNdVq6m8nv7Tf0NGGADc=', 'Content-Length': '0'} 2023-10-18T20:41:03.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbb3d7e2e978f20d-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.589 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:41:03.589 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:41:03.590 DEBUG:boto:Method: GET 2023-10-18T20:41:03.590 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:41:03.590 DEBUG:boto:Data: 2023-10-18T20:41:03.591 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.591 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.592 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.592 DEBUG:boto:Params: {} 2023-10-18T20:41:03.593 DEBUG:boto:Token: None 2023-10-18T20:41:03.593 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj19 2023-10-18T20:41:03.593 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LaYoJ/crNdVq6m8nv7Tf0NGGADc= 2023-10-18T20:41:03.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LaYoJ/crNdVq6m8nv7Tf0NGGADc=', 'Content-Length': '0'} 2023-10-18T20:41:03.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deb75bc334f905f3-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.638 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.639 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.639 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.639 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.640 DEBUG:boto:Method: GET 2023-10-18T20:41:03.640 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.641 DEBUG:boto:Data: 2023-10-18T20:41:03.641 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.642 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.642 DEBUG:boto:Params: {} 2023-10-18T20:41:03.643 DEBUG:boto:Token: None 2023-10-18T20:41:03.643 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3a33c34b501efe5-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.647 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.647 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.648 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.648 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.649 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.649 DEBUG:boto:Method: GET 2023-10-18T20:41:03.649 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:41:03.650 DEBUG:boto:Data: 2023-10-18T20:41:03.650 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.651 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.651 DEBUG:boto:Params: {} 2023-10-18T20:41:03.652 DEBUG:boto:Token: None 2023-10-18T20:41:03.652 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8eb0ad3da7eba36-00653042df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.657 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:41:03.658 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.659 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.659 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.660 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.660 DEBUG:boto:Method: GET 2023-10-18T20:41:03.660 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.661 DEBUG:boto:Data: 2023-10-18T20:41:03.661 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.662 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.662 DEBUG:boto:Params: {} 2023-10-18T20:41:03.663 DEBUG:boto:Token: None 2023-10-18T20:41:03.663 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F0M3efevviZ8BpGlm44sA/36JZg= 2023-10-18T20:41:03.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F0M3efevviZ8BpGlm44sA/36JZg=', 'Content-Length': '0'} 2023-10-18T20:41:03.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d1232c2b73cd4d6-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.706 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.707 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.707 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.708 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.708 DEBUG:boto:Method: GET 2023-10-18T20:41:03.708 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.709 DEBUG:boto:Data: 2023-10-18T20:41:03.709 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.709 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.710 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.710 DEBUG:boto:Params: {} 2023-10-18T20:41:03.711 DEBUG:boto:Token: None 2023-10-18T20:41:03.711 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F0M3efevviZ8BpGlm44sA/36JZg= 2023-10-18T20:41:03.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F0M3efevviZ8BpGlm44sA/36JZg=', 'Content-Length': '0'} 2023-10-18T20:41:03.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087c84139b9459df0-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.714 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.715 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.715 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.715 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.716 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.716 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.717 DEBUG:boto:Data: 2023-10-18T20:41:03.717 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.717 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.718 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.718 DEBUG:boto:Params: {} 2023-10-18T20:41:03.719 DEBUG:boto:Token: None 2023-10-18T20:41:03.719 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.720 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3OfbBkAXg06ZSpMPsegmY0OIJjY= 2023-10-18T20:41:03.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3OfbBkAXg06ZSpMPsegmY0OIJjY=', 'Content-Length': '0'} 2023-10-18T20:41:03.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a26b258c47323241-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.722 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.723 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.723 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.724 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.724 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.725 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.725 DEBUG:boto:Data: 2023-10-18T20:41:03.725 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.726 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.726 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.727 DEBUG:boto:Params: {} 2023-10-18T20:41:03.727 DEBUG:boto:Token: None 2023-10-18T20:41:03.727 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3OfbBkAXg06ZSpMPsegmY0OIJjY= 2023-10-18T20:41:03.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3OfbBkAXg06ZSpMPsegmY0OIJjY=', 'Content-Length': '0'} 2023-10-18T20:41:03.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000163be37c92b8cff1-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:41:03.731 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.731 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.732 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.732 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.733 DEBUG:boto:Method: GET 2023-10-18T20:41:03.733 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.734 DEBUG:boto:Data: 2023-10-18T20:41:03.734 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.734 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.735 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.735 DEBUG:boto:Params: {} 2023-10-18T20:41:03.736 DEBUG:boto:Token: None 2023-10-18T20:41:03.736 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F0M3efevviZ8BpGlm44sA/36JZg= 2023-10-18T20:41:03.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F0M3efevviZ8BpGlm44sA/36JZg=', 'Content-Length': '0'} 2023-10-18T20:41:03.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e12234041484ce38-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.740 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.740 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.740 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.741 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.741 DEBUG:boto:Method: GET 2023-10-18T20:41:03.741 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.742 DEBUG:boto:Data: 2023-10-18T20:41:03.742 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.743 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.743 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.743 DEBUG:boto:Params: {} 2023-10-18T20:41:03.744 DEBUG:boto:Token: None 2023-10-18T20:41:03.744 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:41:03.745 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F0M3efevviZ8BpGlm44sA/36JZg= 2023-10-18T20:41:03.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F0M3efevviZ8BpGlm44sA/36JZg=', 'Content-Length': '0'} 2023-10-18T20:41:03.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000669f538431cc61a9-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.747 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.748 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.748 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.749 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.749 DEBUG:boto:Method: GET 2023-10-18T20:41:03.750 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.750 DEBUG:boto:Data: 2023-10-18T20:41:03.750 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.751 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.751 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.752 DEBUG:boto:Params: {} 2023-10-18T20:41:03.752 DEBUG:boto:Token: None 2023-10-18T20:41:03.752 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.753 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a1eb439c31b29cd-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.756 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.757 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.757 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.758 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.758 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.759 DEBUG:boto:Method: GET 2023-10-18T20:41:03.759 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.759 DEBUG:boto:Data: 2023-10-18T20:41:03.760 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.760 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.760 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.761 DEBUG:boto:Params: {} 2023-10-18T20:41:03.761 DEBUG:boto:Token: None 2023-10-18T20:41:03.762 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.762 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5439fe5a06203ab-00653042df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.806 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.807 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.808 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.808 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.809 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:41:03.809 DEBUG:boto:Data: 2023-10-18T20:41:03.809 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.810 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.811 DEBUG:boto:Params: {} 2023-10-18T20:41:03.811 DEBUG:boto:Token: None 2023-10-18T20:41:03.812 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22 2023-10-18T20:41:03.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OLOYeio74N7y3DaJXweGOyr+gnA= 2023-10-18T20:41:03.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OLOYeio74N7y3DaJXweGOyr+gnA=', 'Content-Length': '0'} 2023-10-18T20:41:03.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000addbc860b9140cea-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.815 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.815 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.816 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.816 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:41:03.816 DEBUG:boto:Data: 2023-10-18T20:41:03.817 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.817 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.818 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.818 DEBUG:boto:Params: {} 2023-10-18T20:41:03.818 DEBUG:boto:Token: None 2023-10-18T20:41:03.819 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22 2023-10-18T20:41:03.819 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OLOYeio74N7y3DaJXweGOyr+gnA= 2023-10-18T20:41:03.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OLOYeio74N7y3DaJXweGOyr+gnA=', 'Content-Length': '0'} 2023-10-18T20:41:03.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006be482c51efa52db-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:41:03.823 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.823 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.823 DEBUG:boto:Method: GET 2023-10-18T20:41:03.824 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:41:03.824 DEBUG:boto:Data: 2023-10-18T20:41:03.824 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.825 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.825 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.826 DEBUG:boto:Params: {} 2023-10-18T20:41:03.826 DEBUG:boto:Token: None 2023-10-18T20:41:03.826 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22 2023-10-18T20:41:03.827 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Bk0nYsX5dozn2ITDXCsfRUOVfM= 2023-10-18T20:41:03.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Bk0nYsX5dozn2ITDXCsfRUOVfM=', 'Content-Length': '0'} 2023-10-18T20:41:03.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf1c7dfc72b8056b-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.870 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:41:03.871 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:41:03.871 DEBUG:boto:Method: GET 2023-10-18T20:41:03.872 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:41:03.872 DEBUG:boto:Data: 2023-10-18T20:41:03.873 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.873 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.874 DEBUG:boto:Params: {} 2023-10-18T20:41:03.874 DEBUG:boto:Token: None 2023-10-18T20:41:03.875 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj22 2023-10-18T20:41:03.875 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Bk0nYsX5dozn2ITDXCsfRUOVfM= 2023-10-18T20:41:03.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Bk0nYsX5dozn2ITDXCsfRUOVfM=', 'Content-Length': '0'} 2023-10-18T20:41:03.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000569d1ebc7026b76d-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.879 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.879 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.879 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.880 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.880 DEBUG:boto:Method: GET 2023-10-18T20:41:03.881 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.881 DEBUG:boto:Data: 2023-10-18T20:41:03.881 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.882 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.882 DEBUG:boto:Params: {} 2023-10-18T20:41:03.883 DEBUG:boto:Token: None 2023-10-18T20:41:03.883 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009868f6984489b975-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.888 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.888 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.889 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.889 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.890 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.890 DEBUG:boto:Method: GET 2023-10-18T20:41:03.890 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:41:03.891 DEBUG:boto:Data: 2023-10-18T20:41:03.891 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.891 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.892 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.892 DEBUG:boto:Params: {} 2023-10-18T20:41:03.893 DEBUG:boto:Token: None 2023-10-18T20:41:03.893 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.894 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f069181545cbc00-00653042df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.898 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:03.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:41:03.899 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:03.899 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:03.900 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.900 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.900 DEBUG:boto:Method: GET 2023-10-18T20:41:03.901 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.901 DEBUG:boto:Data: 2023-10-18T20:41:03.902 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.902 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.902 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.903 DEBUG:boto:Params: {} 2023-10-18T20:41:03.903 DEBUG:boto:Token: None 2023-10-18T20:41:03.904 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.904 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O1QEoKAi3d3qULr0WNP2ttHx5qA= 2023-10-18T20:41:03.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O1QEoKAi3d3qULr0WNP2ttHx5qA=', 'Content-Length': '0'} 2023-10-18T20:41:03.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a6110e3ebd6999-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.907 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:03.907 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:03.908 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.908 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.909 DEBUG:boto:Method: GET 2023-10-18T20:41:03.909 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.909 DEBUG:boto:Data: 2023-10-18T20:41:03.910 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.910 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.911 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.911 DEBUG:boto:Params: {} 2023-10-18T20:41:03.911 DEBUG:boto:Token: None 2023-10-18T20:41:03.912 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.912 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O1QEoKAi3d3qULr0WNP2ttHx5qA= 2023-10-18T20:41:03.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O1QEoKAi3d3qULr0WNP2ttHx5qA=', 'Content-Length': '0'} 2023-10-18T20:41:03.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004965f0a5e19dd3de-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.915 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:03.916 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:03.916 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.916 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.917 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.917 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.917 DEBUG:boto:Data: 2023-10-18T20:41:03.918 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.919 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.919 DEBUG:boto:Params: {} 2023-10-18T20:41:03.919 DEBUG:boto:Token: None 2023-10-18T20:41:03.920 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sxw7+4HqeLzm09/faqAUqkVLXaY= 2023-10-18T20:41:03.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sxw7+4HqeLzm09/faqAUqkVLXaY=', 'Content-Length': '0'} 2023-10-18T20:41:03.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1be69e4b13d3ca5-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.923 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:03.923 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:03.924 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.924 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.925 DEBUG:boto:Method: HEAD 2023-10-18T20:41:03.925 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.925 DEBUG:boto:Data: 2023-10-18T20:41:03.926 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.926 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.926 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.927 DEBUG:boto:Params: {} 2023-10-18T20:41:03.927 DEBUG:boto:Token: None 2023-10-18T20:41:03.928 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.928 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sxw7+4HqeLzm09/faqAUqkVLXaY= 2023-10-18T20:41:03.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sxw7+4HqeLzm09/faqAUqkVLXaY=', 'Content-Length': '0'} 2023-10-18T20:41:03.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c110469822d78675-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:41:03.931 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:03.931 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:03.932 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.932 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.933 DEBUG:boto:Method: GET 2023-10-18T20:41:03.933 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.934 DEBUG:boto:Data: 2023-10-18T20:41:03.934 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.934 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.935 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.935 DEBUG:boto:Params: {} 2023-10-18T20:41:03.936 DEBUG:boto:Token: None 2023-10-18T20:41:03.936 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.937 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O1QEoKAi3d3qULr0WNP2ttHx5qA= 2023-10-18T20:41:03.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O1QEoKAi3d3qULr0WNP2ttHx5qA=', 'Content-Length': '0'} 2023-10-18T20:41:03.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b5735c187b7ec20-00653042df-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.982 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:03.983 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:03.983 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.983 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.984 DEBUG:boto:Method: GET 2023-10-18T20:41:03.984 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.985 DEBUG:boto:Data: 2023-10-18T20:41:03.985 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.985 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:03.986 DEBUG:boto:Port: 8000 2023-10-18T20:41:03.986 DEBUG:boto:Params: {} 2023-10-18T20:41:03.987 DEBUG:boto:Token: None 2023-10-18T20:41:03.987 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:41:03.987 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O1QEoKAi3d3qULr0WNP2ttHx5qA= 2023-10-18T20:41:03.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O1QEoKAi3d3qULr0WNP2ttHx5qA=', 'Content-Length': '0'} 2023-10-18T20:41:03.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c738401d5f29401-00653042df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:03.991 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:03.991 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:03.991 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:41:03.992 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:41:03.992 DEBUG:boto:Method: GET 2023-10-18T20:41:03.993 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:41:03.993 DEBUG:boto:Data: 2023-10-18T20:41:03.993 DEBUG:boto:Headers: {} 2023-10-18T20:41:03.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:03.994 DEBUG:boto:Port: 8001 2023-10-18T20:41:03.995 DEBUG:boto:Params: {} 2023-10-18T20:41:03.995 DEBUG:boto:Token: None 2023-10-18T20:41:03.996 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:03 GMT /bofrnf-59/ 2023-10-18T20:41:03.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk= 2023-10-18T20:41:03.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rdc3PnL+nGeajJWtOQCwCb9K1pk=', 'Content-Length': '0'} 2023-10-18T20:41:03.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f068fad9b6b9981a-00653042df-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.000 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.000 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.001 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.001 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:41:04.002 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:41:04.002 DEBUG:boto:Method: GET 2023-10-18T20:41:04.003 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:41:04.003 DEBUG:boto:Data: 2023-10-18T20:41:04.003 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.004 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.004 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.005 DEBUG:boto:Params: {} 2023-10-18T20:41:04.005 DEBUG:boto:Token: None 2023-10-18T20:41:04.006 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.006 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8279ae31fb84c69-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.010 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.011 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:04.012 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:04.012 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.012 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:41:04.013 DEBUG:boto:Data: 2023-10-18T20:41:04.013 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.014 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.014 DEBUG:boto:Params: {} 2023-10-18T20:41:04.015 DEBUG:boto:Token: None 2023-10-18T20:41:04.015 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj23 2023-10-18T20:41:04.016 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TGTk6Q00DvSIhDqqqNrs0rYlUuA= 2023-10-18T20:41:04.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TGTk6Q00DvSIhDqqqNrs0rYlUuA=', 'Content-Length': '0'} 2023-10-18T20:41:04.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024acfb271e863730-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.019 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:04.019 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:04.020 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.020 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:41:04.021 DEBUG:boto:Data: 2023-10-18T20:41:04.021 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.021 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.022 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.022 DEBUG:boto:Params: {} 2023-10-18T20:41:04.023 DEBUG:boto:Token: None 2023-10-18T20:41:04.023 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj23 2023-10-18T20:41:04.024 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TGTk6Q00DvSIhDqqqNrs0rYlUuA= 2023-10-18T20:41:04.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TGTk6Q00DvSIhDqqqNrs0rYlUuA=', 'Content-Length': '0'} 2023-10-18T20:41:04.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046e6c29414474a1a-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:41:04.027 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:04.027 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:04.028 DEBUG:boto:Method: GET 2023-10-18T20:41:04.028 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:41:04.029 DEBUG:boto:Data: 2023-10-18T20:41:04.029 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.029 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.030 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.030 DEBUG:boto:Params: {} 2023-10-18T20:41:04.031 DEBUG:boto:Token: None 2023-10-18T20:41:04.031 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj23 2023-10-18T20:41:04.031 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XfxXNYZQW71RtLTCuCs2nJd2eZ4= 2023-10-18T20:41:04.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XfxXNYZQW71RtLTCuCs2nJd2eZ4=', 'Content-Length': '0'} 2023-10-18T20:41:04.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007041924a00d3ae5-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.078 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:41:04.079 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:41:04.079 DEBUG:boto:Method: GET 2023-10-18T20:41:04.080 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:41:04.080 DEBUG:boto:Data: 2023-10-18T20:41:04.080 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.081 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.081 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.081 DEBUG:boto:Params: {} 2023-10-18T20:41:04.082 DEBUG:boto:Token: None 2023-10-18T20:41:04.082 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj23 2023-10-18T20:41:04.083 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XfxXNYZQW71RtLTCuCs2nJd2eZ4= 2023-10-18T20:41:04.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XfxXNYZQW71RtLTCuCs2nJd2eZ4=', 'Content-Length': '0'} 2023-10-18T20:41:04.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e67bc77df6e0e2f9-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.086 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.086 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.087 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:41:04.087 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:41:04.087 DEBUG:boto:Method: GET 2023-10-18T20:41:04.088 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:41:04.088 DEBUG:boto:Data: 2023-10-18T20:41:04.089 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.089 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.089 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.090 DEBUG:boto:Params: {} 2023-10-18T20:41:04.090 DEBUG:boto:Token: None 2023-10-18T20:41:04.091 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.093 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017ffe58fe27165e4-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.096 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.097 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.097 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.098 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:41:04.098 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:41:04.098 DEBUG:boto:Method: GET 2023-10-18T20:41:04.099 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:41:04.099 DEBUG:boto:Data: 2023-10-18T20:41:04.100 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.100 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.100 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.101 DEBUG:boto:Params: {} 2023-10-18T20:41:04.101 DEBUG:boto:Token: None 2023-10-18T20:41:04.102 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.102 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6965f343a3331bf-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.106 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:41:04.107 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.108 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.108 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.109 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.109 DEBUG:boto:Method: GET 2023-10-18T20:41:04.110 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.110 DEBUG:boto:Data: 2023-10-18T20:41:04.110 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.111 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.111 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.112 DEBUG:boto:Params: {} 2023-10-18T20:41:04.112 DEBUG:boto:Token: None 2023-10-18T20:41:04.113 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.113 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MQ45GzAYPQpiZftuIU0v5D7wY7Q= 2023-10-18T20:41:04.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MQ45GzAYPQpiZftuIU0v5D7wY7Q=', 'Content-Length': '0'} 2023-10-18T20:41:04.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c275388c328645-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.116 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.117 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.117 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.118 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.118 DEBUG:boto:Method: GET 2023-10-18T20:41:04.118 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.119 DEBUG:boto:Data: 2023-10-18T20:41:04.119 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.120 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.120 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.120 DEBUG:boto:Params: {} 2023-10-18T20:41:04.121 DEBUG:boto:Token: None 2023-10-18T20:41:04.121 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.122 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MQ45GzAYPQpiZftuIU0v5D7wY7Q= 2023-10-18T20:41:04.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MQ45GzAYPQpiZftuIU0v5D7wY7Q=', 'Content-Length': '0'} 2023-10-18T20:41:04.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db85f7b1827cdf78-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.125 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.126 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.126 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.126 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.127 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.127 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.128 DEBUG:boto:Data: 2023-10-18T20:41:04.128 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.128 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.129 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.129 DEBUG:boto:Params: {} 2023-10-18T20:41:04.129 DEBUG:boto:Token: None 2023-10-18T20:41:04.130 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.130 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fiwKulMmUaKYrf3VWeIMtNd3BUs= 2023-10-18T20:41:04.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fiwKulMmUaKYrf3VWeIMtNd3BUs=', 'Content-Length': '0'} 2023-10-18T20:41:04.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b4e13d178b7a55e-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.133 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.133 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.134 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.134 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.135 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.135 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.135 DEBUG:boto:Data: 2023-10-18T20:41:04.136 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.136 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.137 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.137 DEBUG:boto:Params: {} 2023-10-18T20:41:04.137 DEBUG:boto:Token: None 2023-10-18T20:41:04.138 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.138 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fiwKulMmUaKYrf3VWeIMtNd3BUs= 2023-10-18T20:41:04.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fiwKulMmUaKYrf3VWeIMtNd3BUs=', 'Content-Length': '0'} 2023-10-18T20:41:04.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c74cf507fdb8433-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:41:04.142 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.142 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.143 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.143 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.143 DEBUG:boto:Method: GET 2023-10-18T20:41:04.144 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.144 DEBUG:boto:Data: 2023-10-18T20:41:04.145 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.145 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.146 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.146 DEBUG:boto:Params: {} 2023-10-18T20:41:04.146 DEBUG:boto:Token: None 2023-10-18T20:41:04.147 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.147 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MQ45GzAYPQpiZftuIU0v5D7wY7Q= 2023-10-18T20:41:04.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MQ45GzAYPQpiZftuIU0v5D7wY7Q=', 'Content-Length': '0'} 2023-10-18T20:41:04.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000192952d66b5610f6-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.151 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.151 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.151 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.152 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.152 DEBUG:boto:Method: GET 2023-10-18T20:41:04.153 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.153 DEBUG:boto:Data: 2023-10-18T20:41:04.154 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.154 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.154 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.155 DEBUG:boto:Params: {} 2023-10-18T20:41:04.155 DEBUG:boto:Token: None 2023-10-18T20:41:04.156 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:41:04.156 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MQ45GzAYPQpiZftuIU0v5D7wY7Q= 2023-10-18T20:41:04.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MQ45GzAYPQpiZftuIU0v5D7wY7Q=', 'Content-Length': '0'} 2023-10-18T20:41:04.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070cf150c93a33711-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.160 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.160 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.160 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.161 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.161 DEBUG:boto:Method: GET 2023-10-18T20:41:04.162 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.162 DEBUG:boto:Data: 2023-10-18T20:41:04.163 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.163 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.163 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.164 DEBUG:boto:Params: {} 2023-10-18T20:41:04.164 DEBUG:boto:Token: None 2023-10-18T20:41:04.165 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.165 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019ef25395f2d8b5f-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.169 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.169 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.170 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.170 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.170 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.171 DEBUG:boto:Method: GET 2023-10-18T20:41:04.171 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.172 DEBUG:boto:Data: 2023-10-18T20:41:04.172 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.172 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.173 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.173 DEBUG:boto:Params: {} 2023-10-18T20:41:04.174 DEBUG:boto:Token: None 2023-10-18T20:41:04.174 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.174 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13b919816bc971a-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.222 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.223 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.223 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.224 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.224 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:41:04.225 DEBUG:boto:Data: 2023-10-18T20:41:04.225 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.225 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.226 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.226 DEBUG:boto:Params: {} 2023-10-18T20:41:04.226 DEBUG:boto:Token: None 2023-10-18T20:41:04.227 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29 2023-10-18T20:41:04.227 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V0p6XCRWS+wmL+hoI4vcizp207M= 2023-10-18T20:41:04.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V0p6XCRWS+wmL+hoI4vcizp207M=', 'Content-Length': '0'} 2023-10-18T20:41:04.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c38a7937e211c99a-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.230 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.231 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.231 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.231 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:41:04.232 DEBUG:boto:Data: 2023-10-18T20:41:04.232 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.233 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.233 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.233 DEBUG:boto:Params: {} 2023-10-18T20:41:04.234 DEBUG:boto:Token: None 2023-10-18T20:41:04.234 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29 2023-10-18T20:41:04.235 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V0p6XCRWS+wmL+hoI4vcizp207M= 2023-10-18T20:41:04.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V0p6XCRWS+wmL+hoI4vcizp207M=', 'Content-Length': '0'} 2023-10-18T20:41:04.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eac59329776e8972-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:41:04.238 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.238 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.239 DEBUG:boto:Method: GET 2023-10-18T20:41:04.239 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:41:04.240 DEBUG:boto:Data: 2023-10-18T20:41:04.240 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.240 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.241 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.241 DEBUG:boto:Params: {} 2023-10-18T20:41:04.241 DEBUG:boto:Token: None 2023-10-18T20:41:04.242 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29 2023-10-18T20:41:04.242 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GkA+H61M3xDtiTztXdR+nd2ydbw= 2023-10-18T20:41:04.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GkA+H61M3xDtiTztXdR+nd2ydbw=', 'Content-Length': '0'} 2023-10-18T20:41:04.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000273fcc50fdff2df8-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.245 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:41:04.246 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:41:04.246 DEBUG:boto:Method: GET 2023-10-18T20:41:04.246 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:41:04.247 DEBUG:boto:Data: 2023-10-18T20:41:04.247 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.248 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.249 DEBUG:boto:Params: {} 2023-10-18T20:41:04.249 DEBUG:boto:Token: None 2023-10-18T20:41:04.249 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj29 2023-10-18T20:41:04.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GkA+H61M3xDtiTztXdR+nd2ydbw= 2023-10-18T20:41:04.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GkA+H61M3xDtiTztXdR+nd2ydbw=', 'Content-Length': '0'} 2023-10-18T20:41:04.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fe01dfbe6210130-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.253 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.253 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.254 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.254 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.255 DEBUG:boto:Method: GET 2023-10-18T20:41:04.255 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.255 DEBUG:boto:Data: 2023-10-18T20:41:04.256 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.256 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.257 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.257 DEBUG:boto:Params: {} 2023-10-18T20:41:04.257 DEBUG:boto:Token: None 2023-10-18T20:41:04.258 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.258 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000519aa36b3854583a-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.262 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.262 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.263 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.263 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.264 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.264 DEBUG:boto:Method: GET 2023-10-18T20:41:04.264 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:41:04.265 DEBUG:boto:Data: 2023-10-18T20:41:04.265 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.266 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.266 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.266 DEBUG:boto:Params: {} 2023-10-18T20:41:04.267 DEBUG:boto:Token: None 2023-10-18T20:41:04.267 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.268 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000062e94681f68660d-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.272 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:41:04.273 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.273 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.274 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.274 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.275 DEBUG:boto:Method: GET 2023-10-18T20:41:04.275 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.275 DEBUG:boto:Data: 2023-10-18T20:41:04.276 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.276 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.276 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.277 DEBUG:boto:Params: {} 2023-10-18T20:41:04.277 DEBUG:boto:Token: None 2023-10-18T20:41:04.278 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.278 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i1kTPnN2LvtxO/aUxFjsFU0bj5Q= 2023-10-18T20:41:04.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i1kTPnN2LvtxO/aUxFjsFU0bj5Q=', 'Content-Length': '0'} 2023-10-18T20:41:04.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022b1e48c0f605b5a-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.281 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.281 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.282 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.282 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.283 DEBUG:boto:Method: GET 2023-10-18T20:41:04.283 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.284 DEBUG:boto:Data: 2023-10-18T20:41:04.284 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.284 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.285 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.285 DEBUG:boto:Params: {} 2023-10-18T20:41:04.286 DEBUG:boto:Token: None 2023-10-18T20:41:04.286 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.287 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i1kTPnN2LvtxO/aUxFjsFU0bj5Q= 2023-10-18T20:41:04.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i1kTPnN2LvtxO/aUxFjsFU0bj5Q=', 'Content-Length': '0'} 2023-10-18T20:41:04.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a79a72107bb64f-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.290 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.290 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.291 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.291 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.291 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.292 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.292 DEBUG:boto:Data: 2023-10-18T20:41:04.293 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.293 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.293 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.294 DEBUG:boto:Params: {} 2023-10-18T20:41:04.294 DEBUG:boto:Token: None 2023-10-18T20:41:04.295 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.295 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:19pf7vqC+/9HYyMi3PN7Wa6Bi6U= 2023-10-18T20:41:04.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:19pf7vqC+/9HYyMi3PN7Wa6Bi6U=', 'Content-Length': '0'} 2023-10-18T20:41:04.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021fac9a127ac51b0-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.300 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.300 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.300 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.301 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.301 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.302 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.302 DEBUG:boto:Data: 2023-10-18T20:41:04.302 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.303 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.303 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.304 DEBUG:boto:Params: {} 2023-10-18T20:41:04.304 DEBUG:boto:Token: None 2023-10-18T20:41:04.304 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:19pf7vqC+/9HYyMi3PN7Wa6Bi6U= 2023-10-18T20:41:04.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:19pf7vqC+/9HYyMi3PN7Wa6Bi6U=', 'Content-Length': '0'} 2023-10-18T20:41:04.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b04a424af56bf0f-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:41:04.308 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.309 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.309 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.309 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.310 DEBUG:boto:Method: GET 2023-10-18T20:41:04.310 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.311 DEBUG:boto:Data: 2023-10-18T20:41:04.311 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.312 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.312 DEBUG:boto:Params: {} 2023-10-18T20:41:04.313 DEBUG:boto:Token: None 2023-10-18T20:41:04.313 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.314 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i1kTPnN2LvtxO/aUxFjsFU0bj5Q= 2023-10-18T20:41:04.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i1kTPnN2LvtxO/aUxFjsFU0bj5Q=', 'Content-Length': '0'} 2023-10-18T20:41:04.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9e33b66c31850c2-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.358 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.359 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.359 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.360 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.360 DEBUG:boto:Method: GET 2023-10-18T20:41:04.361 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.361 DEBUG:boto:Data: 2023-10-18T20:41:04.362 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.362 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.363 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.363 DEBUG:boto:Params: {} 2023-10-18T20:41:04.364 DEBUG:boto:Token: None 2023-10-18T20:41:04.364 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:41:04.364 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i1kTPnN2LvtxO/aUxFjsFU0bj5Q= 2023-10-18T20:41:04.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i1kTPnN2LvtxO/aUxFjsFU0bj5Q=', 'Content-Length': '0'} 2023-10-18T20:41:04.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d5a9d169a94efe1-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.368 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.368 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.368 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.369 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.369 DEBUG:boto:Method: GET 2023-10-18T20:41:04.370 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.370 DEBUG:boto:Data: 2023-10-18T20:41:04.370 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.371 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.371 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.372 DEBUG:boto:Params: {} 2023-10-18T20:41:04.372 DEBUG:boto:Token: None 2023-10-18T20:41:04.373 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc0a4c0d1fe8b911-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.377 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.377 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.378 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.378 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.379 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.379 DEBUG:boto:Method: GET 2023-10-18T20:41:04.379 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.380 DEBUG:boto:Data: 2023-10-18T20:41:04.380 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.381 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.382 DEBUG:boto:Params: {} 2023-10-18T20:41:04.382 DEBUG:boto:Token: None 2023-10-18T20:41:04.383 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f99cb707d8f9d27c-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.430 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.431 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.431 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.432 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.432 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:41:04.433 DEBUG:boto:Data: 2023-10-18T20:41:04.433 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.434 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.434 DEBUG:boto:Params: {} 2023-10-18T20:41:04.435 DEBUG:boto:Token: None 2023-10-18T20:41:04.435 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33 2023-10-18T20:41:04.436 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VnzvYpc7AGGDxNDF+0itn1TaqqM= 2023-10-18T20:41:04.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VnzvYpc7AGGDxNDF+0itn1TaqqM=', 'Content-Length': '0'} 2023-10-18T20:41:04.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7c9947d074450f6-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.439 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.439 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.440 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.440 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:41:04.440 DEBUG:boto:Data: 2023-10-18T20:41:04.441 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.441 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.442 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.442 DEBUG:boto:Params: {} 2023-10-18T20:41:04.443 DEBUG:boto:Token: None 2023-10-18T20:41:04.443 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33 2023-10-18T20:41:04.443 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VnzvYpc7AGGDxNDF+0itn1TaqqM= 2023-10-18T20:41:04.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VnzvYpc7AGGDxNDF+0itn1TaqqM=', 'Content-Length': '0'} 2023-10-18T20:41:04.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c277c076fe1ddf5c-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:41:04.447 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.447 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.448 DEBUG:boto:Method: GET 2023-10-18T20:41:04.448 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:41:04.449 DEBUG:boto:Data: 2023-10-18T20:41:04.449 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.449 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.450 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.450 DEBUG:boto:Params: {} 2023-10-18T20:41:04.451 DEBUG:boto:Token: None 2023-10-18T20:41:04.451 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33 2023-10-18T20:41:04.452 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5xLFKF31NG22thmu/uX5w6G/GtE= 2023-10-18T20:41:04.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5xLFKF31NG22thmu/uX5w6G/GtE=', 'Content-Length': '0'} 2023-10-18T20:41:04.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007efd22d1bdff5134-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.455 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:41:04.455 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:41:04.455 DEBUG:boto:Method: GET 2023-10-18T20:41:04.456 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:41:04.456 DEBUG:boto:Data: 2023-10-18T20:41:04.457 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.458 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.458 DEBUG:boto:Params: {} 2023-10-18T20:41:04.459 DEBUG:boto:Token: None 2023-10-18T20:41:04.459 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj33 2023-10-18T20:41:04.459 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5xLFKF31NG22thmu/uX5w6G/GtE= 2023-10-18T20:41:04.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5xLFKF31NG22thmu/uX5w6G/GtE=', 'Content-Length': '0'} 2023-10-18T20:41:04.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2bf64fffc04e620-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.463 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.463 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.463 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.464 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.464 DEBUG:boto:Method: GET 2023-10-18T20:41:04.465 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.465 DEBUG:boto:Data: 2023-10-18T20:41:04.465 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.466 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.466 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.467 DEBUG:boto:Params: {} 2023-10-18T20:41:04.467 DEBUG:boto:Token: None 2023-10-18T20:41:04.467 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.468 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007060c4a4809ac090-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.471 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.472 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.472 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.473 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.473 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.473 DEBUG:boto:Method: GET 2023-10-18T20:41:04.474 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:41:04.474 DEBUG:boto:Data: 2023-10-18T20:41:04.475 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.475 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.475 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.476 DEBUG:boto:Params: {} 2023-10-18T20:41:04.476 DEBUG:boto:Token: None 2023-10-18T20:41:04.477 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.477 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006254b5d6c58b009b-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.482 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:41:04.483 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.483 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.483 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.484 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.484 DEBUG:boto:Method: GET 2023-10-18T20:41:04.485 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.485 DEBUG:boto:Data: 2023-10-18T20:41:04.485 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.486 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.486 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.487 DEBUG:boto:Params: {} 2023-10-18T20:41:04.487 DEBUG:boto:Token: None 2023-10-18T20:41:04.487 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.488 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5CPE1nOiA/kIZRhfsRV2ciszz/k= 2023-10-18T20:41:04.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5CPE1nOiA/kIZRhfsRV2ciszz/k=', 'Content-Length': '0'} 2023-10-18T20:41:04.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000295be0a47883cac5-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.491 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.492 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.492 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.492 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.493 DEBUG:boto:Method: GET 2023-10-18T20:41:04.493 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.493 DEBUG:boto:Data: 2023-10-18T20:41:04.494 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.495 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.495 DEBUG:boto:Params: {} 2023-10-18T20:41:04.495 DEBUG:boto:Token: None 2023-10-18T20:41:04.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.496 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5CPE1nOiA/kIZRhfsRV2ciszz/k= 2023-10-18T20:41:04.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5CPE1nOiA/kIZRhfsRV2ciszz/k=', 'Content-Length': '0'} 2023-10-18T20:41:04.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f675a45971e69f-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.499 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.501 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.501 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.501 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.502 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.502 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.503 DEBUG:boto:Data: 2023-10-18T20:41:04.503 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.504 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.504 DEBUG:boto:Params: {} 2023-10-18T20:41:04.504 DEBUG:boto:Token: None 2023-10-18T20:41:04.505 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eKLOgnCpTkkN40dk7oE2y99i4tI= 2023-10-18T20:41:04.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eKLOgnCpTkkN40dk7oE2y99i4tI=', 'Content-Length': '0'} 2023-10-18T20:41:04.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dac383104898c527-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.508 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.509 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.509 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.509 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.510 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.510 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.511 DEBUG:boto:Data: 2023-10-18T20:41:04.511 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.511 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.512 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.512 DEBUG:boto:Params: {} 2023-10-18T20:41:04.512 DEBUG:boto:Token: None 2023-10-18T20:41:04.513 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.513 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eKLOgnCpTkkN40dk7oE2y99i4tI= 2023-10-18T20:41:04.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eKLOgnCpTkkN40dk7oE2y99i4tI=', 'Content-Length': '0'} 2023-10-18T20:41:04.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000518491bfe6e8bbb6-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:41:04.517 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.517 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.518 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.518 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.518 DEBUG:boto:Method: GET 2023-10-18T20:41:04.519 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.519 DEBUG:boto:Data: 2023-10-18T20:41:04.520 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.520 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.521 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.521 DEBUG:boto:Params: {} 2023-10-18T20:41:04.521 DEBUG:boto:Token: None 2023-10-18T20:41:04.522 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.522 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5CPE1nOiA/kIZRhfsRV2ciszz/k= 2023-10-18T20:41:04.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5CPE1nOiA/kIZRhfsRV2ciszz/k=', 'Content-Length': '0'} 2023-10-18T20:41:04.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332ea0c9a7c24e8a-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.566 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.567 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.567 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.568 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.568 DEBUG:boto:Method: GET 2023-10-18T20:41:04.568 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.569 DEBUG:boto:Data: 2023-10-18T20:41:04.569 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.570 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.570 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.570 DEBUG:boto:Params: {} 2023-10-18T20:41:04.571 DEBUG:boto:Token: None 2023-10-18T20:41:04.571 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:41:04.572 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5CPE1nOiA/kIZRhfsRV2ciszz/k= 2023-10-18T20:41:04.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5CPE1nOiA/kIZRhfsRV2ciszz/k=', 'Content-Length': '0'} 2023-10-18T20:41:04.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002126610f38c360a2-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.575 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.575 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.576 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.576 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.576 DEBUG:boto:Method: GET 2023-10-18T20:41:04.577 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.577 DEBUG:boto:Data: 2023-10-18T20:41:04.578 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.578 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.578 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.579 DEBUG:boto:Params: {} 2023-10-18T20:41:04.579 DEBUG:boto:Token: None 2023-10-18T20:41:04.580 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.580 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000040c7b760ec957e9-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.584 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.584 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.585 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.585 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.586 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.586 DEBUG:boto:Method: GET 2023-10-18T20:41:04.587 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.587 DEBUG:boto:Data: 2023-10-18T20:41:04.587 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.588 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.588 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.589 DEBUG:boto:Params: {} 2023-10-18T20:41:04.589 DEBUG:boto:Token: None 2023-10-18T20:41:04.590 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.590 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb0e79dc7a98e32-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.594 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.595 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.595 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.596 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.596 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:41:04.597 DEBUG:boto:Data: 2023-10-18T20:41:04.597 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.598 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.598 DEBUG:boto:Params: {} 2023-10-18T20:41:04.599 DEBUG:boto:Token: None 2023-10-18T20:41:04.599 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34 2023-10-18T20:41:04.600 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sqDfeX/cOzJFnwboGcfqDyly+a0= 2023-10-18T20:41:04.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sqDfeX/cOzJFnwboGcfqDyly+a0=', 'Content-Length': '0'} 2023-10-18T20:41:04.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001556f9fe2d11d69b-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.603 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.603 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.604 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.604 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:41:04.604 DEBUG:boto:Data: 2023-10-18T20:41:04.605 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.605 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.606 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.606 DEBUG:boto:Params: {} 2023-10-18T20:41:04.607 DEBUG:boto:Token: None 2023-10-18T20:41:04.607 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34 2023-10-18T20:41:04.607 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sqDfeX/cOzJFnwboGcfqDyly+a0= 2023-10-18T20:41:04.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sqDfeX/cOzJFnwboGcfqDyly+a0=', 'Content-Length': '0'} 2023-10-18T20:41:04.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ab17ee70b404765-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:41:04.611 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.611 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.612 DEBUG:boto:Method: GET 2023-10-18T20:41:04.612 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:41:04.612 DEBUG:boto:Data: 2023-10-18T20:41:04.613 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.613 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.614 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.614 DEBUG:boto:Params: {} 2023-10-18T20:41:04.614 DEBUG:boto:Token: None 2023-10-18T20:41:04.615 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34 2023-10-18T20:41:04.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+CRTjrfDEz7nyUK+qao3HjCMf5w= 2023-10-18T20:41:04.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+CRTjrfDEz7nyUK+qao3HjCMf5w=', 'Content-Length': '0'} 2023-10-18T20:41:04.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d68a1a9caf8d74ef-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.618 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:41:04.619 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:41:04.619 DEBUG:boto:Method: GET 2023-10-18T20:41:04.620 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:41:04.620 DEBUG:boto:Data: 2023-10-18T20:41:04.620 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.621 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.621 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.622 DEBUG:boto:Params: {} 2023-10-18T20:41:04.622 DEBUG:boto:Token: None 2023-10-18T20:41:04.623 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj34 2023-10-18T20:41:04.623 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+CRTjrfDEz7nyUK+qao3HjCMf5w= 2023-10-18T20:41:04.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+CRTjrfDEz7nyUK+qao3HjCMf5w=', 'Content-Length': '0'} 2023-10-18T20:41:04.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc88dc6b485e2c46-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.626 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.626 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.627 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.627 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.628 DEBUG:boto:Method: GET 2023-10-18T20:41:04.628 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.628 DEBUG:boto:Data: 2023-10-18T20:41:04.629 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.629 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.630 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.630 DEBUG:boto:Params: {} 2023-10-18T20:41:04.630 DEBUG:boto:Token: None 2023-10-18T20:41:04.631 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.631 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003892c94d7771963a-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.635 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.635 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.636 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.636 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.637 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.637 DEBUG:boto:Method: GET 2023-10-18T20:41:04.637 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:41:04.638 DEBUG:boto:Data: 2023-10-18T20:41:04.638 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.639 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.639 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.640 DEBUG:boto:Params: {} 2023-10-18T20:41:04.640 DEBUG:boto:Token: None 2023-10-18T20:41:04.640 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.641 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c68ec3f4738d596-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.645 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:41:04.646 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.647 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.647 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.648 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.648 DEBUG:boto:Method: GET 2023-10-18T20:41:04.648 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.649 DEBUG:boto:Data: 2023-10-18T20:41:04.649 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.650 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.650 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.650 DEBUG:boto:Params: {} 2023-10-18T20:41:04.651 DEBUG:boto:Token: None 2023-10-18T20:41:04.651 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:92qqCXPnrvoSZmht3Y66EosrW3Y= 2023-10-18T20:41:04.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:92qqCXPnrvoSZmht3Y66EosrW3Y=', 'Content-Length': '0'} 2023-10-18T20:41:04.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2766b278f2cca3-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.698 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.699 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.699 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.700 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.700 DEBUG:boto:Method: GET 2023-10-18T20:41:04.700 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.701 DEBUG:boto:Data: 2023-10-18T20:41:04.701 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.702 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.702 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.702 DEBUG:boto:Params: {} 2023-10-18T20:41:04.705 DEBUG:boto:Token: None 2023-10-18T20:41:04.705 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.706 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:92qqCXPnrvoSZmht3Y66EosrW3Y= 2023-10-18T20:41:04.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:92qqCXPnrvoSZmht3Y66EosrW3Y=', 'Content-Length': '0'} 2023-10-18T20:41:04.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6c0b548d07c3f48-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.709 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.709 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.710 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.710 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.711 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.711 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.712 DEBUG:boto:Data: 2023-10-18T20:41:04.712 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.712 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.713 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.713 DEBUG:boto:Params: {} 2023-10-18T20:41:04.714 DEBUG:boto:Token: None 2023-10-18T20:41:04.714 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.714 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9VHcl098Gn9V2pJlE2/BUpYwVzA= 2023-10-18T20:41:04.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9VHcl098Gn9V2pJlE2/BUpYwVzA=', 'Content-Length': '0'} 2023-10-18T20:41:04.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055b8f7f76e5cb291-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.717 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.718 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.718 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.719 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.719 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.719 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.720 DEBUG:boto:Data: 2023-10-18T20:41:04.720 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.721 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.721 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.721 DEBUG:boto:Params: {} 2023-10-18T20:41:04.722 DEBUG:boto:Token: None 2023-10-18T20:41:04.722 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.723 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9VHcl098Gn9V2pJlE2/BUpYwVzA= 2023-10-18T20:41:04.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9VHcl098Gn9V2pJlE2/BUpYwVzA=', 'Content-Length': '0'} 2023-10-18T20:41:04.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c591c5543420616c-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:41:04.726 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.727 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.727 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.727 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.728 DEBUG:boto:Method: GET 2023-10-18T20:41:04.728 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.729 DEBUG:boto:Data: 2023-10-18T20:41:04.729 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.729 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.730 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.730 DEBUG:boto:Params: {} 2023-10-18T20:41:04.731 DEBUG:boto:Token: None 2023-10-18T20:41:04.731 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.732 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:92qqCXPnrvoSZmht3Y66EosrW3Y= 2023-10-18T20:41:04.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:92qqCXPnrvoSZmht3Y66EosrW3Y=', 'Content-Length': '0'} 2023-10-18T20:41:04.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f202c9c4e211ec52-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.735 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.735 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.735 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.736 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.736 DEBUG:boto:Method: GET 2023-10-18T20:41:04.737 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.737 DEBUG:boto:Data: 2023-10-18T20:41:04.738 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.738 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.739 DEBUG:boto:Params: {} 2023-10-18T20:41:04.739 DEBUG:boto:Token: None 2023-10-18T20:41:04.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:41:04.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:92qqCXPnrvoSZmht3Y66EosrW3Y= 2023-10-18T20:41:04.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:92qqCXPnrvoSZmht3Y66EosrW3Y=', 'Content-Length': '0'} 2023-10-18T20:41:04.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000880e0484005f3141-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.743 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.744 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.744 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.745 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.745 DEBUG:boto:Method: GET 2023-10-18T20:41:04.745 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.746 DEBUG:boto:Data: 2023-10-18T20:41:04.746 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.747 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.748 DEBUG:boto:Params: {} 2023-10-18T20:41:04.748 DEBUG:boto:Token: None 2023-10-18T20:41:04.749 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098afbd0e108d3834-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.752 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.753 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.753 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.754 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.754 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.755 DEBUG:boto:Method: GET 2023-10-18T20:41:04.755 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.756 DEBUG:boto:Data: 2023-10-18T20:41:04.756 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.756 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.757 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.757 DEBUG:boto:Params: {} 2023-10-18T20:41:04.758 DEBUG:boto:Token: None 2023-10-18T20:41:04.758 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.759 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb41eadf5c5a31e-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.806 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.807 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.808 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.808 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.808 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:41:04.809 DEBUG:boto:Data: 2023-10-18T20:41:04.809 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.810 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.810 DEBUG:boto:Params: {} 2023-10-18T20:41:04.811 DEBUG:boto:Token: None 2023-10-18T20:41:04.811 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38 2023-10-18T20:41:04.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uJY4+dbzJxX+hQHOppF0iNk/V54= 2023-10-18T20:41:04.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJY4+dbzJxX+hQHOppF0iNk/V54=', 'Content-Length': '0'} 2023-10-18T20:41:04.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffc3adc61080a75d-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.815 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.815 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.816 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.816 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:41:04.817 DEBUG:boto:Data: 2023-10-18T20:41:04.817 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.817 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.818 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.818 DEBUG:boto:Params: {} 2023-10-18T20:41:04.819 DEBUG:boto:Token: None 2023-10-18T20:41:04.819 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38 2023-10-18T20:41:04.820 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uJY4+dbzJxX+hQHOppF0iNk/V54= 2023-10-18T20:41:04.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJY4+dbzJxX+hQHOppF0iNk/V54=', 'Content-Length': '0'} 2023-10-18T20:41:04.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370fce5a812ab16f-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:41:04.823 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.824 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.824 DEBUG:boto:Method: GET 2023-10-18T20:41:04.824 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:41:04.825 DEBUG:boto:Data: 2023-10-18T20:41:04.825 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.826 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.826 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.827 DEBUG:boto:Params: {} 2023-10-18T20:41:04.827 DEBUG:boto:Token: None 2023-10-18T20:41:04.827 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38 2023-10-18T20:41:04.828 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ETV/b6Jg3IYFggOfSUhSeDjpmrY= 2023-10-18T20:41:04.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ETV/b6Jg3IYFggOfSUhSeDjpmrY=', 'Content-Length': '0'} 2023-10-18T20:41:04.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a259cca661b0cdb3-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.831 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:41:04.831 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:41:04.832 DEBUG:boto:Method: GET 2023-10-18T20:41:04.832 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:41:04.833 DEBUG:boto:Data: 2023-10-18T20:41:04.833 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.833 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.834 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.834 DEBUG:boto:Params: {} 2023-10-18T20:41:04.835 DEBUG:boto:Token: None 2023-10-18T20:41:04.835 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj38 2023-10-18T20:41:04.836 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ETV/b6Jg3IYFggOfSUhSeDjpmrY= 2023-10-18T20:41:04.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ETV/b6Jg3IYFggOfSUhSeDjpmrY=', 'Content-Length': '0'} 2023-10-18T20:41:04.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000985d29a3b73f5d3c-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.839 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.839 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.839 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.840 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.840 DEBUG:boto:Method: GET 2023-10-18T20:41:04.841 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.841 DEBUG:boto:Data: 2023-10-18T20:41:04.841 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.842 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.842 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.843 DEBUG:boto:Params: {} 2023-10-18T20:41:04.843 DEBUG:boto:Token: None 2023-10-18T20:41:04.843 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.844 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a9253cc8f2e3b84-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.890 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.891 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.892 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.892 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.892 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.893 DEBUG:boto:Method: GET 2023-10-18T20:41:04.893 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:41:04.894 DEBUG:boto:Data: 2023-10-18T20:41:04.894 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.894 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.895 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.895 DEBUG:boto:Params: {} 2023-10-18T20:41:04.896 DEBUG:boto:Token: None 2023-10-18T20:41:04.896 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.897 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c26cb26552b8981c-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.901 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:41:04.902 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:04.903 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:04.903 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.903 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.904 DEBUG:boto:Method: GET 2023-10-18T20:41:04.904 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.905 DEBUG:boto:Data: 2023-10-18T20:41:04.905 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.905 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.906 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.906 DEBUG:boto:Params: {} 2023-10-18T20:41:04.907 DEBUG:boto:Token: None 2023-10-18T20:41:04.907 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.908 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MFBQHS+skmGTTeIVgrhTBgEN9Pk= 2023-10-18T20:41:04.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MFBQHS+skmGTTeIVgrhTBgEN9Pk=', 'Content-Length': '0'} 2023-10-18T20:41:04.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2883269ed2de705-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.911 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:04.911 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:04.913 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.913 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.913 DEBUG:boto:Method: GET 2023-10-18T20:41:04.914 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.914 DEBUG:boto:Data: 2023-10-18T20:41:04.915 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.915 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.916 DEBUG:boto:Params: {} 2023-10-18T20:41:04.916 DEBUG:boto:Token: None 2023-10-18T20:41:04.917 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MFBQHS+skmGTTeIVgrhTBgEN9Pk= 2023-10-18T20:41:04.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MFBQHS+skmGTTeIVgrhTBgEN9Pk=', 'Content-Length': '0'} 2023-10-18T20:41:04.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002af48d678f115e8-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.920 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:04.921 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:04.921 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.921 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.922 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.922 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.922 DEBUG:boto:Data: 2023-10-18T20:41:04.923 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.924 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.924 DEBUG:boto:Params: {} 2023-10-18T20:41:04.924 DEBUG:boto:Token: None 2023-10-18T20:41:04.925 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.925 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16SOLLrJAKLZpT/ZHtYoNaYCGhw= 2023-10-18T20:41:04.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16SOLLrJAKLZpT/ZHtYoNaYCGhw=', 'Content-Length': '0'} 2023-10-18T20:41:04.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b0fa394f33c3e21-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.928 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:04.928 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:04.929 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.929 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.929 DEBUG:boto:Method: HEAD 2023-10-18T20:41:04.930 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.930 DEBUG:boto:Data: 2023-10-18T20:41:04.931 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.931 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.932 DEBUG:boto:Params: {} 2023-10-18T20:41:04.932 DEBUG:boto:Token: None 2023-10-18T20:41:04.933 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:16SOLLrJAKLZpT/ZHtYoNaYCGhw= 2023-10-18T20:41:04.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:16SOLLrJAKLZpT/ZHtYoNaYCGhw=', 'Content-Length': '0'} 2023-10-18T20:41:04.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db17e5fc0f9be010-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:41:04.937 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:04.937 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:04.937 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.938 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.938 DEBUG:boto:Method: GET 2023-10-18T20:41:04.939 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.939 DEBUG:boto:Data: 2023-10-18T20:41:04.940 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.940 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.941 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.941 DEBUG:boto:Params: {} 2023-10-18T20:41:04.942 DEBUG:boto:Token: None 2023-10-18T20:41:04.942 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.943 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MFBQHS+skmGTTeIVgrhTBgEN9Pk= 2023-10-18T20:41:04.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MFBQHS+skmGTTeIVgrhTBgEN9Pk=', 'Content-Length': '0'} 2023-10-18T20:41:04.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008663a0b3d40b271-00653042e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.946 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:04.946 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:04.947 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.947 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.947 DEBUG:boto:Method: GET 2023-10-18T20:41:04.948 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.948 DEBUG:boto:Data: 2023-10-18T20:41:04.949 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.949 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.950 DEBUG:boto:Params: {} 2023-10-18T20:41:04.950 DEBUG:boto:Token: None 2023-10-18T20:41:04.951 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:41:04.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MFBQHS+skmGTTeIVgrhTBgEN9Pk= 2023-10-18T20:41:04.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MFBQHS+skmGTTeIVgrhTBgEN9Pk=', 'Content-Length': '0'} 2023-10-18T20:41:04.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e55dbd7b4527bcc-00653042e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.955 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.955 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.956 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:41:04.956 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:41:04.957 DEBUG:boto:Method: GET 2023-10-18T20:41:04.957 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:41:04.957 DEBUG:boto:Data: 2023-10-18T20:41:04.958 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:04.959 DEBUG:boto:Port: 8001 2023-10-18T20:41:04.959 DEBUG:boto:Params: {} 2023-10-18T20:41:04.960 DEBUG:boto:Token: None 2023-10-18T20:41:04.960 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.961 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d4b811dfb1d600-00653042e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:04.964 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:04.965 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:04.965 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:04.966 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:41:04.966 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:41:04.966 DEBUG:boto:Method: GET 2023-10-18T20:41:04.967 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:41:04.967 DEBUG:boto:Data: 2023-10-18T20:41:04.968 DEBUG:boto:Headers: {} 2023-10-18T20:41:04.968 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:04.968 DEBUG:boto:Port: 8000 2023-10-18T20:41:04.969 DEBUG:boto:Params: {} 2023-10-18T20:41:04.969 DEBUG:boto:Token: None 2023-10-18T20:41:04.970 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:04 GMT /bofrnf-59/ 2023-10-18T20:41:04.970 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI= 2023-10-18T20:41:04.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IBTz+SfHZ8mBNExBvGwAMKkkdzI=', 'Content-Length': '0'} 2023-10-18T20:41:04.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000371fd94f3b693b52-00653042e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.018 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.019 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:05.019 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:05.020 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.020 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:41:05.021 DEBUG:boto:Data: 2023-10-18T20:41:05.021 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.021 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.022 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.022 DEBUG:boto:Params: {} 2023-10-18T20:41:05.023 DEBUG:boto:Token: None 2023-10-18T20:41:05.023 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj44 2023-10-18T20:41:05.024 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+hMOFvvPUu4qQQOiOjPjqc87ss= 2023-10-18T20:41:05.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+hMOFvvPUu4qQQOiOjPjqc87ss=', 'Content-Length': '0'} 2023-10-18T20:41:05.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074e0acb1c152c195-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.027 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:05.027 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:05.028 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.028 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:41:05.028 DEBUG:boto:Data: 2023-10-18T20:41:05.029 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.029 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.030 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.030 DEBUG:boto:Params: {} 2023-10-18T20:41:05.031 DEBUG:boto:Token: None 2023-10-18T20:41:05.031 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj44 2023-10-18T20:41:05.031 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+hMOFvvPUu4qQQOiOjPjqc87ss= 2023-10-18T20:41:05.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+hMOFvvPUu4qQQOiOjPjqc87ss=', 'Content-Length': '0'} 2023-10-18T20:41:05.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a005a60125b98d8-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:41:05.035 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:05.035 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:05.036 DEBUG:boto:Method: GET 2023-10-18T20:41:05.036 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:41:05.037 DEBUG:boto:Data: 2023-10-18T20:41:05.037 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.038 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.038 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.038 DEBUG:boto:Params: {} 2023-10-18T20:41:05.039 DEBUG:boto:Token: None 2023-10-18T20:41:05.039 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj44 2023-10-18T20:41:05.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ppgsNX16HUi+Cl5X6JqN0Syi6M0= 2023-10-18T20:41:05.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ppgsNX16HUi+Cl5X6JqN0Syi6M0=', 'Content-Length': '0'} 2023-10-18T20:41:05.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7045bb6a233af2b-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.086 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:41:05.087 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:41:05.087 DEBUG:boto:Method: GET 2023-10-18T20:41:05.088 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:41:05.088 DEBUG:boto:Data: 2023-10-18T20:41:05.089 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.089 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.089 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.090 DEBUG:boto:Params: {} 2023-10-18T20:41:05.090 DEBUG:boto:Token: None 2023-10-18T20:41:05.091 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj44 2023-10-18T20:41:05.091 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ppgsNX16HUi+Cl5X6JqN0Syi6M0= 2023-10-18T20:41:05.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ppgsNX16HUi+Cl5X6JqN0Syi6M0=', 'Content-Length': '0'} 2023-10-18T20:41:05.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d42d83eaa20d13-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.094 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.095 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.095 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:41:05.096 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:41:05.096 DEBUG:boto:Method: GET 2023-10-18T20:41:05.097 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:41:05.097 DEBUG:boto:Data: 2023-10-18T20:41:05.097 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.098 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.099 DEBUG:boto:Params: {} 2023-10-18T20:41:05.099 DEBUG:boto:Token: None 2023-10-18T20:41:05.100 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe0649e8a5de917-00653042e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.104 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.104 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.105 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.105 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:41:05.106 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:41:05.106 DEBUG:boto:Method: GET 2023-10-18T20:41:05.107 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:41:05.107 DEBUG:boto:Data: 2023-10-18T20:41:05.107 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.108 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.108 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.109 DEBUG:boto:Params: {} 2023-10-18T20:41:05.109 DEBUG:boto:Token: None 2023-10-18T20:41:05.110 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.110 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001582a2c1910c6825-00653042e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.115 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:41:05.117 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.118 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.118 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.119 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.119 DEBUG:boto:Method: GET 2023-10-18T20:41:05.119 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.120 DEBUG:boto:Data: 2023-10-18T20:41:05.120 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.121 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.121 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.121 DEBUG:boto:Params: {} 2023-10-18T20:41:05.122 DEBUG:boto:Token: None 2023-10-18T20:41:05.122 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.123 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c7vMd5voXErqix9QlaCgL4kZ8ek= 2023-10-18T20:41:05.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c7vMd5voXErqix9QlaCgL4kZ8ek=', 'Content-Length': '0'} 2023-10-18T20:41:05.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e77de068d43c174a-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.126 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.127 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.127 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.127 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.128 DEBUG:boto:Method: GET 2023-10-18T20:41:05.128 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.129 DEBUG:boto:Data: 2023-10-18T20:41:05.129 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.130 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.130 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.131 DEBUG:boto:Params: {} 2023-10-18T20:41:05.131 DEBUG:boto:Token: None 2023-10-18T20:41:05.132 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.132 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c7vMd5voXErqix9QlaCgL4kZ8ek= 2023-10-18T20:41:05.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c7vMd5voXErqix9QlaCgL4kZ8ek=', 'Content-Length': '0'} 2023-10-18T20:41:05.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5ae48543366ed79-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.136 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.136 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.137 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.137 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.137 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.138 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.138 DEBUG:boto:Data: 2023-10-18T20:41:05.138 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.139 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.139 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.140 DEBUG:boto:Params: {} 2023-10-18T20:41:05.140 DEBUG:boto:Token: None 2023-10-18T20:41:05.140 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.141 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eYCmjQULi4DwqbanmQQFoUZ1+d8= 2023-10-18T20:41:05.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eYCmjQULi4DwqbanmQQFoUZ1+d8=', 'Content-Length': '0'} 2023-10-18T20:41:05.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f83078f68bf626c-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.144 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.144 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.145 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.145 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.146 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.146 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.146 DEBUG:boto:Data: 2023-10-18T20:41:05.147 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.147 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.147 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.148 DEBUG:boto:Params: {} 2023-10-18T20:41:05.148 DEBUG:boto:Token: None 2023-10-18T20:41:05.149 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.149 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eYCmjQULi4DwqbanmQQFoUZ1+d8= 2023-10-18T20:41:05.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eYCmjQULi4DwqbanmQQFoUZ1+d8=', 'Content-Length': '0'} 2023-10-18T20:41:05.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d966775946b4bbdb-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:41:05.153 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.153 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.153 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.154 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.154 DEBUG:boto:Method: GET 2023-10-18T20:41:05.155 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.155 DEBUG:boto:Data: 2023-10-18T20:41:05.156 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.156 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.156 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.157 DEBUG:boto:Params: {} 2023-10-18T20:41:05.157 DEBUG:boto:Token: None 2023-10-18T20:41:05.158 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.158 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c7vMd5voXErqix9QlaCgL4kZ8ek= 2023-10-18T20:41:05.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c7vMd5voXErqix9QlaCgL4kZ8ek=', 'Content-Length': '0'} 2023-10-18T20:41:05.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047775bec3c8ba61e-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.202 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.203 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.203 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.204 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.204 DEBUG:boto:Method: GET 2023-10-18T20:41:05.204 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.205 DEBUG:boto:Data: 2023-10-18T20:41:05.205 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.206 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.206 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.206 DEBUG:boto:Params: {} 2023-10-18T20:41:05.207 DEBUG:boto:Token: None 2023-10-18T20:41:05.207 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:41:05.208 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c7vMd5voXErqix9QlaCgL4kZ8ek= 2023-10-18T20:41:05.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c7vMd5voXErqix9QlaCgL4kZ8ek=', 'Content-Length': '0'} 2023-10-18T20:41:05.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000763c008ce1a21147-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.211 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.212 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.212 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.212 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.213 DEBUG:boto:Method: GET 2023-10-18T20:41:05.213 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.213 DEBUG:boto:Data: 2023-10-18T20:41:05.214 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.214 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.215 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.215 DEBUG:boto:Params: {} 2023-10-18T20:41:05.216 DEBUG:boto:Token: None 2023-10-18T20:41:05.216 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.216 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068fa6425f9a218e8-00653042e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.220 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.221 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.221 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.221 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.222 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.222 DEBUG:boto:Method: GET 2023-10-18T20:41:05.223 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.223 DEBUG:boto:Data: 2023-10-18T20:41:05.223 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.224 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.224 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.225 DEBUG:boto:Params: {} 2023-10-18T20:41:05.225 DEBUG:boto:Token: None 2023-10-18T20:41:05.225 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.226 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001752cbf59c4def09-00653042e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.230 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.231 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.231 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.232 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.232 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:41:05.233 DEBUG:boto:Data: 2023-10-18T20:41:05.233 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.233 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.234 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.234 DEBUG:boto:Params: {} 2023-10-18T20:41:05.235 DEBUG:boto:Token: None 2023-10-18T20:41:05.235 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46 2023-10-18T20:41:05.235 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8CU1K8BmhWd3xCVjbJqQd4KvtE8= 2023-10-18T20:41:05.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8CU1K8BmhWd3xCVjbJqQd4KvtE8=', 'Content-Length': '0'} 2023-10-18T20:41:05.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f2931a4d67df08-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.238 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.238 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.239 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.239 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:41:05.240 DEBUG:boto:Data: 2023-10-18T20:41:05.240 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.240 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.241 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.241 DEBUG:boto:Params: {} 2023-10-18T20:41:05.241 DEBUG:boto:Token: None 2023-10-18T20:41:05.242 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46 2023-10-18T20:41:05.242 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8CU1K8BmhWd3xCVjbJqQd4KvtE8= 2023-10-18T20:41:05.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8CU1K8BmhWd3xCVjbJqQd4KvtE8=', 'Content-Length': '0'} 2023-10-18T20:41:05.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000353e441fc0dccb29-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:41:05.246 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.246 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.247 DEBUG:boto:Method: GET 2023-10-18T20:41:05.247 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:41:05.247 DEBUG:boto:Data: 2023-10-18T20:41:05.248 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.249 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.249 DEBUG:boto:Params: {} 2023-10-18T20:41:05.250 DEBUG:boto:Token: None 2023-10-18T20:41:05.250 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46 2023-10-18T20:41:05.251 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kZ4x9tnfKQ4A/rtVkE0h78DG+Sc= 2023-10-18T20:41:05.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kZ4x9tnfKQ4A/rtVkE0h78DG+Sc=', 'Content-Length': '0'} 2023-10-18T20:41:05.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d222b9ac830f947-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.254 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:41:05.254 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:41:05.255 DEBUG:boto:Method: GET 2023-10-18T20:41:05.255 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:41:05.255 DEBUG:boto:Data: 2023-10-18T20:41:05.256 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.256 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.257 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.257 DEBUG:boto:Params: {} 2023-10-18T20:41:05.258 DEBUG:boto:Token: None 2023-10-18T20:41:05.258 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj46 2023-10-18T20:41:05.259 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kZ4x9tnfKQ4A/rtVkE0h78DG+Sc= 2023-10-18T20:41:05.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kZ4x9tnfKQ4A/rtVkE0h78DG+Sc=', 'Content-Length': '0'} 2023-10-18T20:41:05.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea94a0b5dd39a83-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.262 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.262 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.263 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.263 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.264 DEBUG:boto:Method: GET 2023-10-18T20:41:05.264 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.265 DEBUG:boto:Data: 2023-10-18T20:41:05.265 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.265 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.266 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.266 DEBUG:boto:Params: {} 2023-10-18T20:41:05.267 DEBUG:boto:Token: None 2023-10-18T20:41:05.267 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.268 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003911d2051da043ed-00653042e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.271 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.272 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.272 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.272 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.273 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.273 DEBUG:boto:Method: GET 2023-10-18T20:41:05.274 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:41:05.274 DEBUG:boto:Data: 2023-10-18T20:41:05.275 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.275 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.275 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.276 DEBUG:boto:Params: {} 2023-10-18T20:41:05.276 DEBUG:boto:Token: None 2023-10-18T20:41:05.277 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.277 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b8de6a6b428c4e-00653042e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.281 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:41:05.282 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.283 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.283 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.284 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.284 DEBUG:boto:Method: GET 2023-10-18T20:41:05.285 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.285 DEBUG:boto:Data: 2023-10-18T20:41:05.285 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.286 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.286 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.287 DEBUG:boto:Params: {} 2023-10-18T20:41:05.287 DEBUG:boto:Token: None 2023-10-18T20:41:05.287 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.288 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Awv6xgxWcBXjl7gmCizdByIpcUw= 2023-10-18T20:41:05.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Awv6xgxWcBXjl7gmCizdByIpcUw=', 'Content-Length': '0'} 2023-10-18T20:41:05.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e6559a1a62c3cd8-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.334 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.335 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.336 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.336 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.336 DEBUG:boto:Method: GET 2023-10-18T20:41:05.337 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.337 DEBUG:boto:Data: 2023-10-18T20:41:05.338 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.338 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.339 DEBUG:boto:Params: {} 2023-10-18T20:41:05.339 DEBUG:boto:Token: None 2023-10-18T20:41:05.340 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Awv6xgxWcBXjl7gmCizdByIpcUw= 2023-10-18T20:41:05.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Awv6xgxWcBXjl7gmCizdByIpcUw=', 'Content-Length': '0'} 2023-10-18T20:41:05.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8bd163da90cf29b-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.343 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.344 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.344 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.344 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.345 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.345 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.346 DEBUG:boto:Data: 2023-10-18T20:41:05.346 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.347 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.347 DEBUG:boto:Params: {} 2023-10-18T20:41:05.347 DEBUG:boto:Token: None 2023-10-18T20:41:05.348 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q1nKYhoYNygQcgwXnkdZ5V9NYGk= 2023-10-18T20:41:05.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q1nKYhoYNygQcgwXnkdZ5V9NYGk=', 'Content-Length': '0'} 2023-10-18T20:41:05.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006321fc468becadce-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.351 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.352 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.352 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.352 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.353 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.353 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.354 DEBUG:boto:Data: 2023-10-18T20:41:05.354 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.354 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.355 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.355 DEBUG:boto:Params: {} 2023-10-18T20:41:05.355 DEBUG:boto:Token: None 2023-10-18T20:41:05.356 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q1nKYhoYNygQcgwXnkdZ5V9NYGk= 2023-10-18T20:41:05.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q1nKYhoYNygQcgwXnkdZ5V9NYGk=', 'Content-Length': '0'} 2023-10-18T20:41:05.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb595ba4c1bd6e97-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:41:05.360 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.360 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.361 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.361 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.361 DEBUG:boto:Method: GET 2023-10-18T20:41:05.362 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.362 DEBUG:boto:Data: 2023-10-18T20:41:05.363 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.363 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.364 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.364 DEBUG:boto:Params: {} 2023-10-18T20:41:05.364 DEBUG:boto:Token: None 2023-10-18T20:41:05.365 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.365 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Awv6xgxWcBXjl7gmCizdByIpcUw= 2023-10-18T20:41:05.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Awv6xgxWcBXjl7gmCizdByIpcUw=', 'Content-Length': '0'} 2023-10-18T20:41:05.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbe516df4af06d26-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.369 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.369 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.369 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.370 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.370 DEBUG:boto:Method: GET 2023-10-18T20:41:05.371 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.371 DEBUG:boto:Data: 2023-10-18T20:41:05.371 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.372 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.372 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.373 DEBUG:boto:Params: {} 2023-10-18T20:41:05.373 DEBUG:boto:Token: None 2023-10-18T20:41:05.374 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:41:05.374 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Awv6xgxWcBXjl7gmCizdByIpcUw= 2023-10-18T20:41:05.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Awv6xgxWcBXjl7gmCizdByIpcUw=', 'Content-Length': '0'} 2023-10-18T20:41:05.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085a5f6e00f505161-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.418 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.419 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.419 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.420 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.420 DEBUG:boto:Method: GET 2023-10-18T20:41:05.420 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.421 DEBUG:boto:Data: 2023-10-18T20:41:05.421 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.422 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.422 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.423 DEBUG:boto:Params: {} 2023-10-18T20:41:05.423 DEBUG:boto:Token: None 2023-10-18T20:41:05.424 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.424 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac6d50fcc80d5c8a-00653042e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.428 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.428 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.429 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.429 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.430 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.430 DEBUG:boto:Method: GET 2023-10-18T20:41:05.431 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.431 DEBUG:boto:Data: 2023-10-18T20:41:05.431 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.432 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.432 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.432 DEBUG:boto:Params: {} 2023-10-18T20:41:05.433 DEBUG:boto:Token: None 2023-10-18T20:41:05.433 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.434 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e56572373c567d-00653042e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.439 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.439 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.440 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.440 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.441 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:41:05.441 DEBUG:boto:Data: 2023-10-18T20:41:05.441 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.442 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.443 DEBUG:boto:Params: {} 2023-10-18T20:41:05.443 DEBUG:boto:Token: None 2023-10-18T20:41:05.444 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51 2023-10-18T20:41:05.444 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HHxGaVBAFjaUEa+nDuja8nt5M0U= 2023-10-18T20:41:05.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HHxGaVBAFjaUEa+nDuja8nt5M0U=', 'Content-Length': '0'} 2023-10-18T20:41:05.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efe8c213f49c1e00-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.447 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.447 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.448 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.448 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:41:05.449 DEBUG:boto:Data: 2023-10-18T20:41:05.449 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.449 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.450 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.450 DEBUG:boto:Params: {} 2023-10-18T20:41:05.451 DEBUG:boto:Token: None 2023-10-18T20:41:05.451 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51 2023-10-18T20:41:05.452 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HHxGaVBAFjaUEa+nDuja8nt5M0U= 2023-10-18T20:41:05.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HHxGaVBAFjaUEa+nDuja8nt5M0U=', 'Content-Length': '0'} 2023-10-18T20:41:05.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c60b11fd696b4768-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:41:05.455 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.455 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.456 DEBUG:boto:Method: GET 2023-10-18T20:41:05.456 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:41:05.456 DEBUG:boto:Data: 2023-10-18T20:41:05.457 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.458 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.458 DEBUG:boto:Params: {} 2023-10-18T20:41:05.459 DEBUG:boto:Token: None 2023-10-18T20:41:05.459 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51 2023-10-18T20:41:05.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7LW2aQ4MixIX5w2Wv8E0LWA+YGo= 2023-10-18T20:41:05.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7LW2aQ4MixIX5w2Wv8E0LWA+YGo=', 'Content-Length': '0'} 2023-10-18T20:41:05.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4154c12d41930ef-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.506 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:41:05.507 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:41:05.507 DEBUG:boto:Method: GET 2023-10-18T20:41:05.508 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:41:05.508 DEBUG:boto:Data: 2023-10-18T20:41:05.508 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.509 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.509 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.510 DEBUG:boto:Params: {} 2023-10-18T20:41:05.510 DEBUG:boto:Token: None 2023-10-18T20:41:05.511 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj51 2023-10-18T20:41:05.511 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7LW2aQ4MixIX5w2Wv8E0LWA+YGo= 2023-10-18T20:41:05.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7LW2aQ4MixIX5w2Wv8E0LWA+YGo=', 'Content-Length': '0'} 2023-10-18T20:41:05.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab38f2e5a2c463a8-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.514 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.515 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.515 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.515 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.516 DEBUG:boto:Method: GET 2023-10-18T20:41:05.516 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.517 DEBUG:boto:Data: 2023-10-18T20:41:05.517 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.517 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.518 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.518 DEBUG:boto:Params: {} 2023-10-18T20:41:05.519 DEBUG:boto:Token: None 2023-10-18T20:41:05.519 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c32c59029c615ae8-00653042e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.523 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.524 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.524 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.525 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.525 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.526 DEBUG:boto:Method: GET 2023-10-18T20:41:05.526 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:41:05.527 DEBUG:boto:Data: 2023-10-18T20:41:05.529 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.530 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.531 DEBUG:boto:Params: {} 2023-10-18T20:41:05.531 DEBUG:boto:Token: None 2023-10-18T20:41:05.531 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86159d13d1d4770-00653042e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.536 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:41:05.537 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.538 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.538 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.539 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.539 DEBUG:boto:Method: GET 2023-10-18T20:41:05.539 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.540 DEBUG:boto:Data: 2023-10-18T20:41:05.540 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.541 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.541 DEBUG:boto:Params: {} 2023-10-18T20:41:05.542 DEBUG:boto:Token: None 2023-10-18T20:41:05.542 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/KK+hC1CqzRtbv+K1UVSIoEADWc= 2023-10-18T20:41:05.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/KK+hC1CqzRtbv+K1UVSIoEADWc=', 'Content-Length': '0'} 2023-10-18T20:41:05.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6716bf2072accb-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.546 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.546 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.547 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.547 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.548 DEBUG:boto:Method: GET 2023-10-18T20:41:05.548 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.548 DEBUG:boto:Data: 2023-10-18T20:41:05.549 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.549 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.550 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.550 DEBUG:boto:Params: {} 2023-10-18T20:41:05.551 DEBUG:boto:Token: None 2023-10-18T20:41:05.551 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/KK+hC1CqzRtbv+K1UVSIoEADWc= 2023-10-18T20:41:05.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/KK+hC1CqzRtbv+K1UVSIoEADWc=', 'Content-Length': '0'} 2023-10-18T20:41:05.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d19eca1dd9572e2d-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.594 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.595 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.595 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.596 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.596 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.597 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.597 DEBUG:boto:Data: 2023-10-18T20:41:05.597 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.598 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.598 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.598 DEBUG:boto:Params: {} 2023-10-18T20:41:05.599 DEBUG:boto:Token: None 2023-10-18T20:41:05.599 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.600 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2svb8BjB6+2yHqdgLugIo01v5eI= 2023-10-18T20:41:05.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2svb8BjB6+2yHqdgLugIo01v5eI=', 'Content-Length': '0'} 2023-10-18T20:41:05.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f0e4855492717e8-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.603 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.603 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.604 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.604 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.604 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.605 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.605 DEBUG:boto:Data: 2023-10-18T20:41:05.605 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.606 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.606 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.606 DEBUG:boto:Params: {} 2023-10-18T20:41:05.607 DEBUG:boto:Token: None 2023-10-18T20:41:05.607 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2svb8BjB6+2yHqdgLugIo01v5eI= 2023-10-18T20:41:05.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2svb8BjB6+2yHqdgLugIo01v5eI=', 'Content-Length': '0'} 2023-10-18T20:41:05.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e85e870d0a72713e-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:41:05.611 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.612 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.612 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.612 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.613 DEBUG:boto:Method: GET 2023-10-18T20:41:05.613 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.613 DEBUG:boto:Data: 2023-10-18T20:41:05.614 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.614 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.615 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.615 DEBUG:boto:Params: {} 2023-10-18T20:41:05.615 DEBUG:boto:Token: None 2023-10-18T20:41:05.616 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.616 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/KK+hC1CqzRtbv+K1UVSIoEADWc= 2023-10-18T20:41:05.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/KK+hC1CqzRtbv+K1UVSIoEADWc=', 'Content-Length': '0'} 2023-10-18T20:41:05.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227acb02cd619ce2-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.619 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.620 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.620 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.620 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.621 DEBUG:boto:Method: GET 2023-10-18T20:41:05.621 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.621 DEBUG:boto:Data: 2023-10-18T20:41:05.622 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.622 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.623 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.623 DEBUG:boto:Params: {} 2023-10-18T20:41:05.624 DEBUG:boto:Token: None 2023-10-18T20:41:05.624 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:41:05.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/KK+hC1CqzRtbv+K1UVSIoEADWc= 2023-10-18T20:41:05.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/KK+hC1CqzRtbv+K1UVSIoEADWc=', 'Content-Length': '0'} 2023-10-18T20:41:05.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000174a4521bcd8bb5e-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.627 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.628 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.628 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.628 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.629 DEBUG:boto:Method: GET 2023-10-18T20:41:05.629 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.630 DEBUG:boto:Data: 2023-10-18T20:41:05.630 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.631 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.631 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.631 DEBUG:boto:Params: {} 2023-10-18T20:41:05.632 DEBUG:boto:Token: None 2023-10-18T20:41:05.632 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.633 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000985fec2c21977234-00653042e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.678 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.679 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.679 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.680 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.680 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.681 DEBUG:boto:Method: GET 2023-10-18T20:41:05.681 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.681 DEBUG:boto:Data: 2023-10-18T20:41:05.682 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.682 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.683 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.683 DEBUG:boto:Params: {} 2023-10-18T20:41:05.683 DEBUG:boto:Token: None 2023-10-18T20:41:05.684 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.684 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a884878a87adc794-00653042e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.689 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.690 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.690 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.690 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.691 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:41:05.691 DEBUG:boto:Data: 2023-10-18T20:41:05.692 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.692 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.692 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.693 DEBUG:boto:Params: {} 2023-10-18T20:41:05.693 DEBUG:boto:Token: None 2023-10-18T20:41:05.693 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55 2023-10-18T20:41:05.694 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EiBHTpipKLaf1cpc8ahM9oI2IKQ= 2023-10-18T20:41:05.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EiBHTpipKLaf1cpc8ahM9oI2IKQ=', 'Content-Length': '0'} 2023-10-18T20:41:05.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc6104321933dc84-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.697 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.697 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.698 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.698 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:41:05.699 DEBUG:boto:Data: 2023-10-18T20:41:05.699 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.699 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.700 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.700 DEBUG:boto:Params: {} 2023-10-18T20:41:05.701 DEBUG:boto:Token: None 2023-10-18T20:41:05.701 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55 2023-10-18T20:41:05.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EiBHTpipKLaf1cpc8ahM9oI2IKQ= 2023-10-18T20:41:05.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EiBHTpipKLaf1cpc8ahM9oI2IKQ=', 'Content-Length': '0'} 2023-10-18T20:41:05.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000934c7b093713cfe1-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:41:05.705 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.705 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.706 DEBUG:boto:Method: GET 2023-10-18T20:41:05.706 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:41:05.706 DEBUG:boto:Data: 2023-10-18T20:41:05.707 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.708 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.708 DEBUG:boto:Params: {} 2023-10-18T20:41:05.709 DEBUG:boto:Token: None 2023-10-18T20:41:05.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55 2023-10-18T20:41:05.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GhWoMAoJ30LfC+Bv1LQyosRcBAU= 2023-10-18T20:41:05.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GhWoMAoJ30LfC+Bv1LQyosRcBAU=', 'Content-Length': '0'} 2023-10-18T20:41:05.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fdda43c011c7f44-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.713 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:41:05.713 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:41:05.713 DEBUG:boto:Method: GET 2023-10-18T20:41:05.714 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:41:05.714 DEBUG:boto:Data: 2023-10-18T20:41:05.715 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.716 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.716 DEBUG:boto:Params: {} 2023-10-18T20:41:05.717 DEBUG:boto:Token: None 2023-10-18T20:41:05.717 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj55 2023-10-18T20:41:05.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GhWoMAoJ30LfC+Bv1LQyosRcBAU= 2023-10-18T20:41:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GhWoMAoJ30LfC+Bv1LQyosRcBAU=', 'Content-Length': '0'} 2023-10-18T20:41:05.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225dad63deceafce-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.720 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.721 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.721 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.722 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.722 DEBUG:boto:Method: GET 2023-10-18T20:41:05.723 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.723 DEBUG:boto:Data: 2023-10-18T20:41:05.724 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.724 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.724 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.725 DEBUG:boto:Params: {} 2023-10-18T20:41:05.725 DEBUG:boto:Token: None 2023-10-18T20:41:05.726 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.726 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b3f8e44462e4607-00653042e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.730 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.731 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.731 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.732 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.732 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.733 DEBUG:boto:Method: GET 2023-10-18T20:41:05.733 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:41:05.734 DEBUG:boto:Data: 2023-10-18T20:41:05.734 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.735 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.735 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.735 DEBUG:boto:Params: {} 2023-10-18T20:41:05.736 DEBUG:boto:Token: None 2023-10-18T20:41:05.736 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051125ec2d5ce871f-00653042e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.742 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:41:05.743 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.743 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.744 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.744 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.744 DEBUG:boto:Method: GET 2023-10-18T20:41:05.745 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.745 DEBUG:boto:Data: 2023-10-18T20:41:05.745 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.746 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.747 DEBUG:boto:Params: {} 2023-10-18T20:41:05.747 DEBUG:boto:Token: None 2023-10-18T20:41:05.748 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2UeYjkFT6QSmdQ/xaBJUbcg9tBI= 2023-10-18T20:41:05.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2UeYjkFT6QSmdQ/xaBJUbcg9tBI=', 'Content-Length': '0'} 2023-10-18T20:41:05.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037244a1b0269f8a9-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.751 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.751 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.752 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.752 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.753 DEBUG:boto:Method: GET 2023-10-18T20:41:05.753 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.753 DEBUG:boto:Data: 2023-10-18T20:41:05.754 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.754 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.755 DEBUG:boto:Params: {} 2023-10-18T20:41:05.755 DEBUG:boto:Token: None 2023-10-18T20:41:05.756 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2UeYjkFT6QSmdQ/xaBJUbcg9tBI= 2023-10-18T20:41:05.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2UeYjkFT6QSmdQ/xaBJUbcg9tBI=', 'Content-Length': '0'} 2023-10-18T20:41:05.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d15ab825f2c560-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.802 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.803 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.803 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.804 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.804 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.805 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.805 DEBUG:boto:Data: 2023-10-18T20:41:05.805 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.806 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.806 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.807 DEBUG:boto:Params: {} 2023-10-18T20:41:05.807 DEBUG:boto:Token: None 2023-10-18T20:41:05.808 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.808 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTmV/ZfRU3iForWUOpYspCpFfrk= 2023-10-18T20:41:05.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTmV/ZfRU3iForWUOpYspCpFfrk=', 'Content-Length': '0'} 2023-10-18T20:41:05.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a015652038f14ca4-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.811 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.812 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.812 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.812 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.813 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.813 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.814 DEBUG:boto:Data: 2023-10-18T20:41:05.814 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.815 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.815 DEBUG:boto:Params: {} 2023-10-18T20:41:05.816 DEBUG:boto:Token: None 2023-10-18T20:41:05.816 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KTmV/ZfRU3iForWUOpYspCpFfrk= 2023-10-18T20:41:05.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTmV/ZfRU3iForWUOpYspCpFfrk=', 'Content-Length': '0'} 2023-10-18T20:41:05.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009411a7d5d5ff6948-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:41:05.820 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.820 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.821 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.821 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.821 DEBUG:boto:Method: GET 2023-10-18T20:41:05.822 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.822 DEBUG:boto:Data: 2023-10-18T20:41:05.823 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.823 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.823 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.824 DEBUG:boto:Params: {} 2023-10-18T20:41:05.824 DEBUG:boto:Token: None 2023-10-18T20:41:05.825 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.825 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2UeYjkFT6QSmdQ/xaBJUbcg9tBI= 2023-10-18T20:41:05.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2UeYjkFT6QSmdQ/xaBJUbcg9tBI=', 'Content-Length': '0'} 2023-10-18T20:41:05.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf5f5ff43faefdba-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.828 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.828 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.829 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.829 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.830 DEBUG:boto:Method: GET 2023-10-18T20:41:05.830 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.831 DEBUG:boto:Data: 2023-10-18T20:41:05.831 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.831 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.832 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.832 DEBUG:boto:Params: {} 2023-10-18T20:41:05.833 DEBUG:boto:Token: None 2023-10-18T20:41:05.833 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:41:05.833 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2UeYjkFT6QSmdQ/xaBJUbcg9tBI= 2023-10-18T20:41:05.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2UeYjkFT6QSmdQ/xaBJUbcg9tBI=', 'Content-Length': '0'} 2023-10-18T20:41:05.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc5113368c72234-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.837 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.837 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.838 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.838 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.838 DEBUG:boto:Method: GET 2023-10-18T20:41:05.839 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.839 DEBUG:boto:Data: 2023-10-18T20:41:05.840 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.841 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.841 DEBUG:boto:Params: {} 2023-10-18T20:41:05.841 DEBUG:boto:Token: None 2023-10-18T20:41:05.842 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.842 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f5cd2e890c2942c-00653042e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.846 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.847 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.847 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.848 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.848 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.848 DEBUG:boto:Method: GET 2023-10-18T20:41:05.849 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.849 DEBUG:boto:Data: 2023-10-18T20:41:05.850 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.850 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.850 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.851 DEBUG:boto:Params: {} 2023-10-18T20:41:05.851 DEBUG:boto:Token: None 2023-10-18T20:41:05.852 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277164456d173858-00653042e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.856 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.857 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.857 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.858 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.858 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:41:05.858 DEBUG:boto:Data: 2023-10-18T20:41:05.859 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.860 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.860 DEBUG:boto:Params: {} 2023-10-18T20:41:05.860 DEBUG:boto:Token: None 2023-10-18T20:41:05.861 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57 2023-10-18T20:41:05.861 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xddix3YvZwhxpaR/1ZWlkMp+ZOw= 2023-10-18T20:41:05.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xddix3YvZwhxpaR/1ZWlkMp+ZOw=', 'Content-Length': '0'} 2023-10-18T20:41:05.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092b817ad9d5702b0-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.864 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.864 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.865 DEBUG:boto:Method: HEAD 2023-10-18T20:41:05.865 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:41:05.865 DEBUG:boto:Data: 2023-10-18T20:41:05.866 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.866 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.867 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.867 DEBUG:boto:Params: {} 2023-10-18T20:41:05.867 DEBUG:boto:Token: None 2023-10-18T20:41:05.868 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57 2023-10-18T20:41:05.868 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xddix3YvZwhxpaR/1ZWlkMp+ZOw= 2023-10-18T20:41:05.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xddix3YvZwhxpaR/1ZWlkMp+ZOw=', 'Content-Length': '0'} 2023-10-18T20:41:05.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f2d0c2a703f9500-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:41:05.871 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.872 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.872 DEBUG:boto:Method: GET 2023-10-18T20:41:05.872 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:41:05.873 DEBUG:boto:Data: 2023-10-18T20:41:05.873 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.874 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.874 DEBUG:boto:Params: {} 2023-10-18T20:41:05.875 DEBUG:boto:Token: None 2023-10-18T20:41:05.875 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57 2023-10-18T20:41:05.876 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3PxjEsV84Ms2d8vIg1wte1IEkqs= 2023-10-18T20:41:05.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3PxjEsV84Ms2d8vIg1wte1IEkqs=', 'Content-Length': '0'} 2023-10-18T20:41:05.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000064cf15b31bf9d6a-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.879 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:41:05.879 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:41:05.879 DEBUG:boto:Method: GET 2023-10-18T20:41:05.880 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:41:05.880 DEBUG:boto:Data: 2023-10-18T20:41:05.881 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.881 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.881 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.882 DEBUG:boto:Params: {} 2023-10-18T20:41:05.882 DEBUG:boto:Token: None 2023-10-18T20:41:05.883 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj57 2023-10-18T20:41:05.883 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3PxjEsV84Ms2d8vIg1wte1IEkqs= 2023-10-18T20:41:05.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3PxjEsV84Ms2d8vIg1wte1IEkqs=', 'Content-Length': '0'} 2023-10-18T20:41:05.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe6febbf5200fafa-00653042e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.926 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.927 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.927 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.927 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.928 DEBUG:boto:Method: GET 2023-10-18T20:41:05.928 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.929 DEBUG:boto:Data: 2023-10-18T20:41:05.929 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.930 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.930 DEBUG:boto:Params: {} 2023-10-18T20:41:05.931 DEBUG:boto:Token: None 2023-10-18T20:41:05.931 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.932 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ddd84587cc2f1f5-00653042e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.935 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.936 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:05.936 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:05.937 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.937 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.937 DEBUG:boto:Method: GET 2023-10-18T20:41:05.938 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:41:05.938 DEBUG:boto:Data: 2023-10-18T20:41:05.939 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.939 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:05.939 DEBUG:boto:Port: 8000 2023-10-18T20:41:05.941 DEBUG:boto:Params: {} 2023-10-18T20:41:05.942 DEBUG:boto:Token: None 2023-10-18T20:41:05.942 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/ 2023-10-18T20:41:05.943 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8= 2023-10-18T20:41:05.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0WTDDalaQ0O2ulBEblcbkW4hIC8=', 'Content-Length': '0'} 2023-10-18T20:41:05.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004109d3ced43b2e7f-00653042e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.947 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:05.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:41:05.948 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:05.948 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:05.949 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:05.949 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:05.950 DEBUG:boto:Method: GET 2023-10-18T20:41:05.950 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:05.950 DEBUG:boto:Data: 2023-10-18T20:41:05.951 DEBUG:boto:Headers: {} 2023-10-18T20:41:05.951 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:05.952 DEBUG:boto:Port: 8001 2023-10-18T20:41:05.952 DEBUG:boto:Params: {} 2023-10-18T20:41:05.952 DEBUG:boto:Token: None 2023-10-18T20:41:05.953 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:05 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:05.953 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UQ5XG4B/TPqJFox2uZGJbyedMZU= 2023-10-18T20:41:05.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UQ5XG4B/TPqJFox2uZGJbyedMZU=', 'Content-Length': '0'} 2023-10-18T20:41:05.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89bcb95f639557f-00653042e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:05.998 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:05.999 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:05.999 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:05.999 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.000 DEBUG:boto:Method: GET 2023-10-18T20:41:06.000 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.001 DEBUG:boto:Data: 2023-10-18T20:41:06.001 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.002 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.002 DEBUG:boto:Params: {} 2023-10-18T20:41:06.003 DEBUG:boto:Token: None 2023-10-18T20:41:06.003 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tKz2BAmX6hgwIZmN1uZhES/e0Kw= 2023-10-18T20:41:06.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tKz2BAmX6hgwIZmN1uZhES/e0Kw=', 'Content-Length': '0'} 2023-10-18T20:41:06.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9264a6f5d60109-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.007 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:06.007 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:06.008 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.008 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.009 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.009 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.009 DEBUG:boto:Data: 2023-10-18T20:41:06.010 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.010 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.011 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.011 DEBUG:boto:Params: {} 2023-10-18T20:41:06.011 DEBUG:boto:Token: None 2023-10-18T20:41:06.012 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0H3M2OKmUjBplWqLGU0Vnoo+YJs= 2023-10-18T20:41:06.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0H3M2OKmUjBplWqLGU0Vnoo+YJs=', 'Content-Length': '0'} 2023-10-18T20:41:06.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de8d24068e58120-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.015 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:06.016 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:06.016 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.016 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.017 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.017 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.018 DEBUG:boto:Data: 2023-10-18T20:41:06.018 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.018 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.019 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.019 DEBUG:boto:Params: {} 2023-10-18T20:41:06.020 DEBUG:boto:Token: None 2023-10-18T20:41:06.020 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.021 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0H3M2OKmUjBplWqLGU0Vnoo+YJs= 2023-10-18T20:41:06.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0H3M2OKmUjBplWqLGU0Vnoo+YJs=', 'Content-Length': '0'} 2023-10-18T20:41:06.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b354802817ffe2-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:41:06.024 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:06.024 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:06.025 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.025 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.025 DEBUG:boto:Method: GET 2023-10-18T20:41:06.026 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.026 DEBUG:boto:Data: 2023-10-18T20:41:06.027 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.027 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.028 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.028 DEBUG:boto:Params: {} 2023-10-18T20:41:06.028 DEBUG:boto:Token: None 2023-10-18T20:41:06.029 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.029 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tKz2BAmX6hgwIZmN1uZhES/e0Kw= 2023-10-18T20:41:06.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tKz2BAmX6hgwIZmN1uZhES/e0Kw=', 'Content-Length': '0'} 2023-10-18T20:41:06.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec8c1b7e129eae2-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.033 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:06.033 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:06.033 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.034 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.034 DEBUG:boto:Method: GET 2023-10-18T20:41:06.035 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.035 DEBUG:boto:Data: 2023-10-18T20:41:06.035 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.036 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.036 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.037 DEBUG:boto:Params: {} 2023-10-18T20:41:06.037 DEBUG:boto:Token: None 2023-10-18T20:41:06.037 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:41:06.038 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tKz2BAmX6hgwIZmN1uZhES/e0Kw= 2023-10-18T20:41:06.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tKz2BAmX6hgwIZmN1uZhES/e0Kw=', 'Content-Length': '0'} 2023-10-18T20:41:06.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a26b6690e88a4b0e-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.082 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.083 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.084 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.085 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.085 DEBUG:boto:Method: GET 2023-10-18T20:41:06.086 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.087 DEBUG:boto:Data: 2023-10-18T20:41:06.087 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.088 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.089 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.089 DEBUG:boto:Params: {} 2023-10-18T20:41:06.090 DEBUG:boto:Token: None 2023-10-18T20:41:06.091 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.091 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af504adac3acec9-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.097 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.098 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.099 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.099 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.100 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.100 DEBUG:boto:Method: GET 2023-10-18T20:41:06.101 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.102 DEBUG:boto:Data: 2023-10-18T20:41:06.103 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.103 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.104 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.105 DEBUG:boto:Params: {} 2023-10-18T20:41:06.105 DEBUG:boto:Token: None 2023-10-18T20:41:06.106 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.107 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000784773e7d4f8fcae-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.112 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.113 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:06.113 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:06.114 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.114 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:41:06.114 DEBUG:boto:Data: 2023-10-18T20:41:06.115 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.115 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.116 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.116 DEBUG:boto:Params: {} 2023-10-18T20:41:06.117 DEBUG:boto:Token: None 2023-10-18T20:41:06.117 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj58 2023-10-18T20:41:06.117 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cCK5ak8LVma8fJNjclm6NaKK0Hw= 2023-10-18T20:41:06.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cCK5ak8LVma8fJNjclm6NaKK0Hw=', 'Content-Length': '0'} 2023-10-18T20:41:06.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0655cd25bd05511-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.120 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:06.121 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:06.121 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.121 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:41:06.122 DEBUG:boto:Data: 2023-10-18T20:41:06.122 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.123 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.123 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.123 DEBUG:boto:Params: {} 2023-10-18T20:41:06.124 DEBUG:boto:Token: None 2023-10-18T20:41:06.124 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj58 2023-10-18T20:41:06.124 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cCK5ak8LVma8fJNjclm6NaKK0Hw= 2023-10-18T20:41:06.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cCK5ak8LVma8fJNjclm6NaKK0Hw=', 'Content-Length': '0'} 2023-10-18T20:41:06.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004248ff1003a03536-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:41:06.128 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:06.128 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:06.129 DEBUG:boto:Method: GET 2023-10-18T20:41:06.129 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:41:06.129 DEBUG:boto:Data: 2023-10-18T20:41:06.130 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.130 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.131 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.131 DEBUG:boto:Params: {} 2023-10-18T20:41:06.132 DEBUG:boto:Token: None 2023-10-18T20:41:06.132 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj58 2023-10-18T20:41:06.132 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e860lUmP6V8tZFNys9h6/7h1Ggg= 2023-10-18T20:41:06.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e860lUmP6V8tZFNys9h6/7h1Ggg=', 'Content-Length': '0'} 2023-10-18T20:41:06.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c90b9a080d2ffd05-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.178 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:41:06.179 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:41:06.179 DEBUG:boto:Method: GET 2023-10-18T20:41:06.180 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:41:06.180 DEBUG:boto:Data: 2023-10-18T20:41:06.181 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.181 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.181 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.182 DEBUG:boto:Params: {} 2023-10-18T20:41:06.182 DEBUG:boto:Token: None 2023-10-18T20:41:06.183 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj58 2023-10-18T20:41:06.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:e860lUmP6V8tZFNys9h6/7h1Ggg= 2023-10-18T20:41:06.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:e860lUmP6V8tZFNys9h6/7h1Ggg=', 'Content-Length': '0'} 2023-10-18T20:41:06.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1ccd564b66ae01d-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.187 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.187 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.188 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.188 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.188 DEBUG:boto:Method: GET 2023-10-18T20:41:06.189 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.189 DEBUG:boto:Data: 2023-10-18T20:41:06.189 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.190 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.191 DEBUG:boto:Params: {} 2023-10-18T20:41:06.191 DEBUG:boto:Token: None 2023-10-18T20:41:06.192 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b62e90887d713b2-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.196 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.196 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.197 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.197 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.197 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.198 DEBUG:boto:Method: GET 2023-10-18T20:41:06.198 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:41:06.199 DEBUG:boto:Data: 2023-10-18T20:41:06.199 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.200 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.200 DEBUG:boto:Params: {} 2023-10-18T20:41:06.201 DEBUG:boto:Token: None 2023-10-18T20:41:06.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.202 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c1f8a1964c9a86d-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.206 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:41:06.207 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.208 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.208 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.209 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.209 DEBUG:boto:Method: GET 2023-10-18T20:41:06.209 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.210 DEBUG:boto:Data: 2023-10-18T20:41:06.210 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.211 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.211 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.211 DEBUG:boto:Params: {} 2023-10-18T20:41:06.212 DEBUG:boto:Token: None 2023-10-18T20:41:06.212 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/O6rajwDHuCgezhjAqB2GqRInec= 2023-10-18T20:41:06.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/O6rajwDHuCgezhjAqB2GqRInec=', 'Content-Length': '0'} 2023-10-18T20:41:06.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a2eb0e1560396b9-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.216 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.216 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.217 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.217 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.217 DEBUG:boto:Method: GET 2023-10-18T20:41:06.218 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.218 DEBUG:boto:Data: 2023-10-18T20:41:06.219 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.219 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.220 DEBUG:boto:Params: {} 2023-10-18T20:41:06.220 DEBUG:boto:Token: None 2023-10-18T20:41:06.221 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/O6rajwDHuCgezhjAqB2GqRInec= 2023-10-18T20:41:06.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/O6rajwDHuCgezhjAqB2GqRInec=', 'Content-Length': '0'} 2023-10-18T20:41:06.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd8dd8e1e1002d8b-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.224 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.225 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.225 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.225 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.226 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.226 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.227 DEBUG:boto:Data: 2023-10-18T20:41:06.227 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.227 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.228 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.228 DEBUG:boto:Params: {} 2023-10-18T20:41:06.228 DEBUG:boto:Token: None 2023-10-18T20:41:06.229 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.229 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:owzluKNX3bdX87O1WHB//fYb+DQ= 2023-10-18T20:41:06.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:owzluKNX3bdX87O1WHB//fYb+DQ=', 'Content-Length': '0'} 2023-10-18T20:41:06.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f5a4b32449eb48-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.232 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.233 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.233 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.233 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.234 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.234 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.235 DEBUG:boto:Data: 2023-10-18T20:41:06.235 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.236 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.236 DEBUG:boto:Params: {} 2023-10-18T20:41:06.237 DEBUG:boto:Token: None 2023-10-18T20:41:06.237 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.238 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:owzluKNX3bdX87O1WHB//fYb+DQ= 2023-10-18T20:41:06.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:owzluKNX3bdX87O1WHB//fYb+DQ=', 'Content-Length': '0'} 2023-10-18T20:41:06.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b21d82609643a0d-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:41:06.241 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.241 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.242 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.242 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.243 DEBUG:boto:Method: GET 2023-10-18T20:41:06.243 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.244 DEBUG:boto:Data: 2023-10-18T20:41:06.244 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.244 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.245 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.245 DEBUG:boto:Params: {} 2023-10-18T20:41:06.246 DEBUG:boto:Token: None 2023-10-18T20:41:06.246 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.247 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/O6rajwDHuCgezhjAqB2GqRInec= 2023-10-18T20:41:06.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/O6rajwDHuCgezhjAqB2GqRInec=', 'Content-Length': '0'} 2023-10-18T20:41:06.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bc89e69310e2525-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.290 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.291 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.291 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.292 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.292 DEBUG:boto:Method: GET 2023-10-18T20:41:06.292 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.293 DEBUG:boto:Data: 2023-10-18T20:41:06.293 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.294 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.294 DEBUG:boto:Params: {} 2023-10-18T20:41:06.295 DEBUG:boto:Token: None 2023-10-18T20:41:06.295 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:41:06.296 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/O6rajwDHuCgezhjAqB2GqRInec= 2023-10-18T20:41:06.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/O6rajwDHuCgezhjAqB2GqRInec=', 'Content-Length': '0'} 2023-10-18T20:41:06.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004802ed8cd068e092-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.299 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.299 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.300 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.300 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.301 DEBUG:boto:Method: GET 2023-10-18T20:41:06.301 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.301 DEBUG:boto:Data: 2023-10-18T20:41:06.302 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.303 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.303 DEBUG:boto:Params: {} 2023-10-18T20:41:06.303 DEBUG:boto:Token: None 2023-10-18T20:41:06.304 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c052d4d17adbcd2c-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.308 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.309 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.309 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.310 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.310 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.311 DEBUG:boto:Method: GET 2023-10-18T20:41:06.311 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.311 DEBUG:boto:Data: 2023-10-18T20:41:06.312 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.313 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.313 DEBUG:boto:Params: {} 2023-10-18T20:41:06.313 DEBUG:boto:Token: None 2023-10-18T20:41:06.314 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.314 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b84a5c2052a60d1e-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.362 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.363 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.363 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.364 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.364 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:41:06.365 DEBUG:boto:Data: 2023-10-18T20:41:06.365 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.365 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.366 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.366 DEBUG:boto:Params: {} 2023-10-18T20:41:06.367 DEBUG:boto:Token: None 2023-10-18T20:41:06.367 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68 2023-10-18T20:41:06.368 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oqCj+zbOqWs/G5leM3c413HJNKU= 2023-10-18T20:41:06.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oqCj+zbOqWs/G5leM3c413HJNKU=', 'Content-Length': '0'} 2023-10-18T20:41:06.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e8cf6c4f70cee2-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.370 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.371 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.371 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.372 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:41:06.372 DEBUG:boto:Data: 2023-10-18T20:41:06.372 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.373 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.373 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.373 DEBUG:boto:Params: {} 2023-10-18T20:41:06.374 DEBUG:boto:Token: None 2023-10-18T20:41:06.374 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68 2023-10-18T20:41:06.375 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oqCj+zbOqWs/G5leM3c413HJNKU= 2023-10-18T20:41:06.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oqCj+zbOqWs/G5leM3c413HJNKU=', 'Content-Length': '0'} 2023-10-18T20:41:06.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d256655102ebb5b-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:41:06.378 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.379 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.379 DEBUG:boto:Method: GET 2023-10-18T20:41:06.379 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:41:06.380 DEBUG:boto:Data: 2023-10-18T20:41:06.380 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.381 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.381 DEBUG:boto:Params: {} 2023-10-18T20:41:06.382 DEBUG:boto:Token: None 2023-10-18T20:41:06.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68 2023-10-18T20:41:06.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8dLwJFn4lUEsuBLTuFoeM/cRgnE= 2023-10-18T20:41:06.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8dLwJFn4lUEsuBLTuFoeM/cRgnE=', 'Content-Length': '0'} 2023-10-18T20:41:06.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63b3cd67d44219d-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.386 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:41:06.386 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:41:06.387 DEBUG:boto:Method: GET 2023-10-18T20:41:06.387 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:41:06.388 DEBUG:boto:Data: 2023-10-18T20:41:06.388 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.388 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.389 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.389 DEBUG:boto:Params: {} 2023-10-18T20:41:06.389 DEBUG:boto:Token: None 2023-10-18T20:41:06.390 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj68 2023-10-18T20:41:06.390 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8dLwJFn4lUEsuBLTuFoeM/cRgnE= 2023-10-18T20:41:06.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8dLwJFn4lUEsuBLTuFoeM/cRgnE=', 'Content-Length': '0'} 2023-10-18T20:41:06.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3b17cc9c442c6f-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.393 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.394 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.394 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.395 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.395 DEBUG:boto:Method: GET 2023-10-18T20:41:06.395 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.396 DEBUG:boto:Data: 2023-10-18T20:41:06.396 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.396 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.397 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.397 DEBUG:boto:Params: {} 2023-10-18T20:41:06.398 DEBUG:boto:Token: None 2023-10-18T20:41:06.398 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.399 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bae39144f92119d5-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.402 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.403 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.403 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.404 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.404 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.405 DEBUG:boto:Method: GET 2023-10-18T20:41:06.405 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:41:06.405 DEBUG:boto:Data: 2023-10-18T20:41:06.406 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.406 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.407 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.407 DEBUG:boto:Params: {} 2023-10-18T20:41:06.408 DEBUG:boto:Token: None 2023-10-18T20:41:06.408 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.409 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f54e4617f6a57a8-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.413 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:41:06.414 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.415 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.415 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.416 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.416 DEBUG:boto:Method: GET 2023-10-18T20:41:06.417 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.418 DEBUG:boto:Data: 2023-10-18T20:41:06.418 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.419 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.419 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.419 DEBUG:boto:Params: {} 2023-10-18T20:41:06.420 DEBUG:boto:Token: None 2023-10-18T20:41:06.420 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.421 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:alY4C+MfOiXOWGVExzCNbe8WHMI= 2023-10-18T20:41:06.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:alY4C+MfOiXOWGVExzCNbe8WHMI=', 'Content-Length': '0'} 2023-10-18T20:41:06.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088786b4991b7f745-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.424 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.425 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.425 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.426 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.426 DEBUG:boto:Method: GET 2023-10-18T20:41:06.427 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.427 DEBUG:boto:Data: 2023-10-18T20:41:06.427 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.428 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.428 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.429 DEBUG:boto:Params: {} 2023-10-18T20:41:06.429 DEBUG:boto:Token: None 2023-10-18T20:41:06.430 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.430 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:alY4C+MfOiXOWGVExzCNbe8WHMI= 2023-10-18T20:41:06.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:alY4C+MfOiXOWGVExzCNbe8WHMI=', 'Content-Length': '0'} 2023-10-18T20:41:06.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb596076643659b5-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.433 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.434 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.434 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.434 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.435 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.435 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.435 DEBUG:boto:Data: 2023-10-18T20:41:06.436 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.436 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.437 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.437 DEBUG:boto:Params: {} 2023-10-18T20:41:06.437 DEBUG:boto:Token: None 2023-10-18T20:41:06.438 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.438 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oxo4arv/v+NqlQtS+Iz/BKZpOqQ= 2023-10-18T20:41:06.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oxo4arv/v+NqlQtS+Iz/BKZpOqQ=', 'Content-Length': '0'} 2023-10-18T20:41:06.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a09bc212215f2f4-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.441 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.442 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.442 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.443 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.443 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.443 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.444 DEBUG:boto:Data: 2023-10-18T20:41:06.444 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.444 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.445 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.445 DEBUG:boto:Params: {} 2023-10-18T20:41:06.446 DEBUG:boto:Token: None 2023-10-18T20:41:06.446 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.446 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oxo4arv/v+NqlQtS+Iz/BKZpOqQ= 2023-10-18T20:41:06.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oxo4arv/v+NqlQtS+Iz/BKZpOqQ=', 'Content-Length': '0'} 2023-10-18T20:41:06.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2e08e25fda93d3e-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:41:06.450 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.450 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.451 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.451 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.451 DEBUG:boto:Method: GET 2023-10-18T20:41:06.452 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.452 DEBUG:boto:Data: 2023-10-18T20:41:06.453 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.453 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.454 DEBUG:boto:Params: {} 2023-10-18T20:41:06.454 DEBUG:boto:Token: None 2023-10-18T20:41:06.455 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:alY4C+MfOiXOWGVExzCNbe8WHMI= 2023-10-18T20:41:06.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:alY4C+MfOiXOWGVExzCNbe8WHMI=', 'Content-Length': '0'} 2023-10-18T20:41:06.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e3eb975b70b19e9-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.458 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.459 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.459 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.459 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.460 DEBUG:boto:Method: GET 2023-10-18T20:41:06.460 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.461 DEBUG:boto:Data: 2023-10-18T20:41:06.461 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.461 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.462 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.462 DEBUG:boto:Params: {} 2023-10-18T20:41:06.463 DEBUG:boto:Token: None 2023-10-18T20:41:06.463 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:41:06.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:alY4C+MfOiXOWGVExzCNbe8WHMI= 2023-10-18T20:41:06.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:alY4C+MfOiXOWGVExzCNbe8WHMI=', 'Content-Length': '0'} 2023-10-18T20:41:06.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6208a4aac2ed0a0-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.467 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.467 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.468 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.468 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.468 DEBUG:boto:Method: GET 2023-10-18T20:41:06.469 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.469 DEBUG:boto:Data: 2023-10-18T20:41:06.470 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.470 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.471 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.471 DEBUG:boto:Params: {} 2023-10-18T20:41:06.471 DEBUG:boto:Token: None 2023-10-18T20:41:06.472 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.472 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aff526834fa0b85-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.476 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.477 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.477 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.478 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.478 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.478 DEBUG:boto:Method: GET 2023-10-18T20:41:06.479 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.479 DEBUG:boto:Data: 2023-10-18T20:41:06.480 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.480 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.481 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.481 DEBUG:boto:Params: {} 2023-10-18T20:41:06.482 DEBUG:boto:Token: None 2023-10-18T20:41:06.482 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.483 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd51ae9a05f95e2-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.530 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.531 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.532 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.532 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.533 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:41:06.533 DEBUG:boto:Data: 2023-10-18T20:41:06.533 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.534 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.534 DEBUG:boto:Params: {} 2023-10-18T20:41:06.535 DEBUG:boto:Token: None 2023-10-18T20:41:06.535 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69 2023-10-18T20:41:06.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HlqmHYTN51H2H6afL0CQIxjGG8I= 2023-10-18T20:41:06.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HlqmHYTN51H2H6afL0CQIxjGG8I=', 'Content-Length': '0'} 2023-10-18T20:41:06.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b62afa68a84d82-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.539 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.539 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.540 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.540 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:41:06.540 DEBUG:boto:Data: 2023-10-18T20:41:06.541 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.542 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.542 DEBUG:boto:Params: {} 2023-10-18T20:41:06.542 DEBUG:boto:Token: None 2023-10-18T20:41:06.543 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69 2023-10-18T20:41:06.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HlqmHYTN51H2H6afL0CQIxjGG8I= 2023-10-18T20:41:06.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HlqmHYTN51H2H6afL0CQIxjGG8I=', 'Content-Length': '0'} 2023-10-18T20:41:06.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de040f3bc0fb8f90-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:41:06.547 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.547 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.548 DEBUG:boto:Method: GET 2023-10-18T20:41:06.548 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:41:06.549 DEBUG:boto:Data: 2023-10-18T20:41:06.549 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.549 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.550 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.550 DEBUG:boto:Params: {} 2023-10-18T20:41:06.551 DEBUG:boto:Token: None 2023-10-18T20:41:06.551 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69 2023-10-18T20:41:06.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g5mrBFxF429ilnEhsJGcnhHC9Mk= 2023-10-18T20:41:06.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g5mrBFxF429ilnEhsJGcnhHC9Mk=', 'Content-Length': '0'} 2023-10-18T20:41:06.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fd00b2c87dd5f6b-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.556 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:41:06.557 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:41:06.557 DEBUG:boto:Method: GET 2023-10-18T20:41:06.558 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:41:06.558 DEBUG:boto:Data: 2023-10-18T20:41:06.559 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.559 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.559 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.560 DEBUG:boto:Params: {} 2023-10-18T20:41:06.560 DEBUG:boto:Token: None 2023-10-18T20:41:06.561 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj69 2023-10-18T20:41:06.561 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g5mrBFxF429ilnEhsJGcnhHC9Mk= 2023-10-18T20:41:06.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g5mrBFxF429ilnEhsJGcnhHC9Mk=', 'Content-Length': '0'} 2023-10-18T20:41:06.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b133b7f6b0337d7-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.564 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.565 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.565 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.566 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.566 DEBUG:boto:Method: GET 2023-10-18T20:41:06.567 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.567 DEBUG:boto:Data: 2023-10-18T20:41:06.567 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.568 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.568 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.569 DEBUG:boto:Params: {} 2023-10-18T20:41:06.569 DEBUG:boto:Token: None 2023-10-18T20:41:06.570 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.570 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082ea06da1929dad7-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.574 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.575 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.575 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.575 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.576 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.576 DEBUG:boto:Method: GET 2023-10-18T20:41:06.577 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:41:06.577 DEBUG:boto:Data: 2023-10-18T20:41:06.577 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.578 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.578 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.579 DEBUG:boto:Params: {} 2023-10-18T20:41:06.579 DEBUG:boto:Token: None 2023-10-18T20:41:06.580 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.580 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a24ea8a7ab0d2a97-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.584 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:41:06.585 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.586 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.586 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.587 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.587 DEBUG:boto:Method: GET 2023-10-18T20:41:06.587 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.588 DEBUG:boto:Data: 2023-10-18T20:41:06.588 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.588 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.589 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.589 DEBUG:boto:Params: {} 2023-10-18T20:41:06.590 DEBUG:boto:Token: None 2023-10-18T20:41:06.590 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.591 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B5yWhwVMAzzEX6hbDR+mWcOt/zM= 2023-10-18T20:41:06.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B5yWhwVMAzzEX6hbDR+mWcOt/zM=', 'Content-Length': '0'} 2023-10-18T20:41:06.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000454f4e5c9e0749ed-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.594 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.594 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.595 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.595 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.595 DEBUG:boto:Method: GET 2023-10-18T20:41:06.596 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.596 DEBUG:boto:Data: 2023-10-18T20:41:06.597 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.597 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.598 DEBUG:boto:Params: {} 2023-10-18T20:41:06.598 DEBUG:boto:Token: None 2023-10-18T20:41:06.599 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.599 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B5yWhwVMAzzEX6hbDR+mWcOt/zM= 2023-10-18T20:41:06.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B5yWhwVMAzzEX6hbDR+mWcOt/zM=', 'Content-Length': '0'} 2023-10-18T20:41:06.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007436a54e0e6471fe-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.602 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.603 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.603 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.603 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.604 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.604 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.604 DEBUG:boto:Data: 2023-10-18T20:41:06.605 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.605 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.605 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.606 DEBUG:boto:Params: {} 2023-10-18T20:41:06.606 DEBUG:boto:Token: None 2023-10-18T20:41:06.607 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.607 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AsU8MkVxi24//b8YUPj0dSIh898= 2023-10-18T20:41:06.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AsU8MkVxi24//b8YUPj0dSIh898=', 'Content-Length': '0'} 2023-10-18T20:41:06.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d33ec55a2f2c2e0-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.610 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.610 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.611 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.611 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.612 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.612 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.612 DEBUG:boto:Data: 2023-10-18T20:41:06.613 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.613 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.613 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.614 DEBUG:boto:Params: {} 2023-10-18T20:41:06.614 DEBUG:boto:Token: None 2023-10-18T20:41:06.615 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AsU8MkVxi24//b8YUPj0dSIh898= 2023-10-18T20:41:06.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AsU8MkVxi24//b8YUPj0dSIh898=', 'Content-Length': '0'} 2023-10-18T20:41:06.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064421831cd7e01e3-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:41:06.619 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.619 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.619 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.620 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.620 DEBUG:boto:Method: GET 2023-10-18T20:41:06.620 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.621 DEBUG:boto:Data: 2023-10-18T20:41:06.621 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.622 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.622 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.622 DEBUG:boto:Params: {} 2023-10-18T20:41:06.623 DEBUG:boto:Token: None 2023-10-18T20:41:06.623 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B5yWhwVMAzzEX6hbDR+mWcOt/zM= 2023-10-18T20:41:06.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B5yWhwVMAzzEX6hbDR+mWcOt/zM=', 'Content-Length': '0'} 2023-10-18T20:41:06.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72fdb09b0eedc1b-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.627 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.627 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.628 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.628 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.628 DEBUG:boto:Method: GET 2023-10-18T20:41:06.629 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.629 DEBUG:boto:Data: 2023-10-18T20:41:06.630 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.631 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.631 DEBUG:boto:Params: {} 2023-10-18T20:41:06.631 DEBUG:boto:Token: None 2023-10-18T20:41:06.632 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:41:06.633 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B5yWhwVMAzzEX6hbDR+mWcOt/zM= 2023-10-18T20:41:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B5yWhwVMAzzEX6hbDR+mWcOt/zM=', 'Content-Length': '0'} 2023-10-18T20:41:06.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c8975b276f6ab9b-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.636 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.636 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.636 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.637 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.637 DEBUG:boto:Method: GET 2023-10-18T20:41:06.638 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.638 DEBUG:boto:Data: 2023-10-18T20:41:06.639 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.639 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.639 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.640 DEBUG:boto:Params: {} 2023-10-18T20:41:06.640 DEBUG:boto:Token: None 2023-10-18T20:41:06.641 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.641 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3074a8d7b45c0d2-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.686 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.687 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.687 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.688 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.688 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.689 DEBUG:boto:Method: GET 2023-10-18T20:41:06.689 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.689 DEBUG:boto:Data: 2023-10-18T20:41:06.690 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.691 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.691 DEBUG:boto:Params: {} 2023-10-18T20:41:06.691 DEBUG:boto:Token: None 2023-10-18T20:41:06.692 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f28a8e63030e2de-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.697 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.698 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.698 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.698 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.699 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:41:06.699 DEBUG:boto:Data: 2023-10-18T20:41:06.699 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.700 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.700 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.701 DEBUG:boto:Params: {} 2023-10-18T20:41:06.701 DEBUG:boto:Token: None 2023-10-18T20:41:06.701 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76 2023-10-18T20:41:06.702 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jD9BvESo3/oojiRy978pLci4jcc= 2023-10-18T20:41:06.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jD9BvESo3/oojiRy978pLci4jcc=', 'Content-Length': '0'} 2023-10-18T20:41:06.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e57c115ad59539-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.705 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.705 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.706 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.706 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:41:06.707 DEBUG:boto:Data: 2023-10-18T20:41:06.707 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.708 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.708 DEBUG:boto:Params: {} 2023-10-18T20:41:06.709 DEBUG:boto:Token: None 2023-10-18T20:41:06.709 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76 2023-10-18T20:41:06.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jD9BvESo3/oojiRy978pLci4jcc= 2023-10-18T20:41:06.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jD9BvESo3/oojiRy978pLci4jcc=', 'Content-Length': '0'} 2023-10-18T20:41:06.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c00545e435b2af-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:41:06.713 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.714 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.714 DEBUG:boto:Method: GET 2023-10-18T20:41:06.714 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:41:06.715 DEBUG:boto:Data: 2023-10-18T20:41:06.715 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.716 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.716 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.716 DEBUG:boto:Params: {} 2023-10-18T20:41:06.717 DEBUG:boto:Token: None 2023-10-18T20:41:06.717 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76 2023-10-18T20:41:06.718 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xZRCZNFxPyHncLQ8TMJJSb/HbQg= 2023-10-18T20:41:06.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xZRCZNFxPyHncLQ8TMJJSb/HbQg=', 'Content-Length': '0'} 2023-10-18T20:41:06.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a62fbee27a9c2b-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.721 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:41:06.721 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:41:06.722 DEBUG:boto:Method: GET 2023-10-18T20:41:06.722 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:41:06.722 DEBUG:boto:Data: 2023-10-18T20:41:06.723 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.724 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.724 DEBUG:boto:Params: {} 2023-10-18T20:41:06.724 DEBUG:boto:Token: None 2023-10-18T20:41:06.725 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj76 2023-10-18T20:41:06.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xZRCZNFxPyHncLQ8TMJJSb/HbQg= 2023-10-18T20:41:06.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xZRCZNFxPyHncLQ8TMJJSb/HbQg=', 'Content-Length': '0'} 2023-10-18T20:41:06.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e1e1cce1d99dd9-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.770 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.771 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.771 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.772 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.772 DEBUG:boto:Method: GET 2023-10-18T20:41:06.772 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.773 DEBUG:boto:Data: 2023-10-18T20:41:06.773 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.774 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.774 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.774 DEBUG:boto:Params: {} 2023-10-18T20:41:06.775 DEBUG:boto:Token: None 2023-10-18T20:41:06.775 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.776 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b507dbc4d9be1cd3-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.779 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.779 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.780 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.780 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.781 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.781 DEBUG:boto:Method: GET 2023-10-18T20:41:06.781 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:41:06.782 DEBUG:boto:Data: 2023-10-18T20:41:06.782 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.783 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.783 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.783 DEBUG:boto:Params: {} 2023-10-18T20:41:06.784 DEBUG:boto:Token: None 2023-10-18T20:41:06.784 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.785 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0991a7d24d30596-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.789 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:41:06.790 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.791 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.791 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.791 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.792 DEBUG:boto:Method: GET 2023-10-18T20:41:06.792 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.793 DEBUG:boto:Data: 2023-10-18T20:41:06.793 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.793 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.794 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.794 DEBUG:boto:Params: {} 2023-10-18T20:41:06.795 DEBUG:boto:Token: None 2023-10-18T20:41:06.795 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.796 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6q5s9+XzTy6W7JWM7/Da8Qpf4pI= 2023-10-18T20:41:06.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6q5s9+XzTy6W7JWM7/Da8Qpf4pI=', 'Content-Length': '0'} 2023-10-18T20:41:06.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004436fdecdbd6b86d-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.799 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.799 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.800 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.800 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.801 DEBUG:boto:Method: GET 2023-10-18T20:41:06.801 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.801 DEBUG:boto:Data: 2023-10-18T20:41:06.802 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.802 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.803 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.803 DEBUG:boto:Params: {} 2023-10-18T20:41:06.803 DEBUG:boto:Token: None 2023-10-18T20:41:06.804 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.804 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6q5s9+XzTy6W7JWM7/Da8Qpf4pI= 2023-10-18T20:41:06.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6q5s9+XzTy6W7JWM7/Da8Qpf4pI=', 'Content-Length': '0'} 2023-10-18T20:41:06.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083d41b30253bbea3-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.807 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.808 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.808 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.809 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.809 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.809 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.810 DEBUG:boto:Data: 2023-10-18T20:41:06.810 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.811 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.811 DEBUG:boto:Params: {} 2023-10-18T20:41:06.812 DEBUG:boto:Token: None 2023-10-18T20:41:06.812 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.813 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IISi8BHakOLTceKllqLOk3T9yJw= 2023-10-18T20:41:06.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IISi8BHakOLTceKllqLOk3T9yJw=', 'Content-Length': '0'} 2023-10-18T20:41:06.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842dbd9e1d42fccb-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.816 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.816 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.817 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.817 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.817 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.818 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.818 DEBUG:boto:Data: 2023-10-18T20:41:06.819 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.819 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.819 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.820 DEBUG:boto:Params: {} 2023-10-18T20:41:06.820 DEBUG:boto:Token: None 2023-10-18T20:41:06.821 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IISi8BHakOLTceKllqLOk3T9yJw= 2023-10-18T20:41:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IISi8BHakOLTceKllqLOk3T9yJw=', 'Content-Length': '0'} 2023-10-18T20:41:06.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000695ec305f332260f-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:41:06.824 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.825 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.825 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.825 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.826 DEBUG:boto:Method: GET 2023-10-18T20:41:06.826 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.827 DEBUG:boto:Data: 2023-10-18T20:41:06.827 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.827 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.828 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.828 DEBUG:boto:Params: {} 2023-10-18T20:41:06.829 DEBUG:boto:Token: None 2023-10-18T20:41:06.829 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.830 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6q5s9+XzTy6W7JWM7/Da8Qpf4pI= 2023-10-18T20:41:06.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6q5s9+XzTy6W7JWM7/Da8Qpf4pI=', 'Content-Length': '0'} 2023-10-18T20:41:06.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02ec8861e25eec3-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.874 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.875 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.875 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.876 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.876 DEBUG:boto:Method: GET 2023-10-18T20:41:06.876 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.877 DEBUG:boto:Data: 2023-10-18T20:41:06.877 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.878 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.878 DEBUG:boto:Params: {} 2023-10-18T20:41:06.879 DEBUG:boto:Token: None 2023-10-18T20:41:06.879 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:41:06.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6q5s9+XzTy6W7JWM7/Da8Qpf4pI= 2023-10-18T20:41:06.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6q5s9+XzTy6W7JWM7/Da8Qpf4pI=', 'Content-Length': '0'} 2023-10-18T20:41:06.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d25c590c104ae76-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.883 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.883 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.883 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.884 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.884 DEBUG:boto:Method: GET 2023-10-18T20:41:06.885 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.885 DEBUG:boto:Data: 2023-10-18T20:41:06.885 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.886 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.886 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.886 DEBUG:boto:Params: {} 2023-10-18T20:41:06.887 DEBUG:boto:Token: None 2023-10-18T20:41:06.887 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d55d365263a59a-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.891 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.892 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.892 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.893 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.893 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.894 DEBUG:boto:Method: GET 2023-10-18T20:41:06.894 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.894 DEBUG:boto:Data: 2023-10-18T20:41:06.895 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.895 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.896 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.896 DEBUG:boto:Params: {} 2023-10-18T20:41:06.897 DEBUG:boto:Token: None 2023-10-18T20:41:06.897 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.898 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b92daa07e54a802-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.902 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.903 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.903 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.904 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.904 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:41:06.905 DEBUG:boto:Data: 2023-10-18T20:41:06.905 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.905 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.906 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.906 DEBUG:boto:Params: {} 2023-10-18T20:41:06.906 DEBUG:boto:Token: None 2023-10-18T20:41:06.907 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85 2023-10-18T20:41:06.907 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZNlXWarSN9mXGA3SrkH3MpXGSEU= 2023-10-18T20:41:06.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZNlXWarSN9mXGA3SrkH3MpXGSEU=', 'Content-Length': '0'} 2023-10-18T20:41:06.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cce2a39c333f81bd-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.910 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.911 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.911 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.911 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:41:06.912 DEBUG:boto:Data: 2023-10-18T20:41:06.912 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.913 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.913 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.913 DEBUG:boto:Params: {} 2023-10-18T20:41:06.914 DEBUG:boto:Token: None 2023-10-18T20:41:06.914 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85 2023-10-18T20:41:06.915 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZNlXWarSN9mXGA3SrkH3MpXGSEU= 2023-10-18T20:41:06.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZNlXWarSN9mXGA3SrkH3MpXGSEU=', 'Content-Length': '0'} 2023-10-18T20:41:06.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000628eb88b85ece0-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:41:06.918 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.918 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.919 DEBUG:boto:Method: GET 2023-10-18T20:41:06.919 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:41:06.920 DEBUG:boto:Data: 2023-10-18T20:41:06.920 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.920 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.921 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.921 DEBUG:boto:Params: {} 2023-10-18T20:41:06.922 DEBUG:boto:Token: None 2023-10-18T20:41:06.922 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85 2023-10-18T20:41:06.923 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eFQEesCmQ/ANYWHyBLuF7iKlrzo= 2023-10-18T20:41:06.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eFQEesCmQ/ANYWHyBLuF7iKlrzo=', 'Content-Length': '0'} 2023-10-18T20:41:06.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2b0030b98a958a-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.926 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:41:06.926 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:41:06.927 DEBUG:boto:Method: GET 2023-10-18T20:41:06.927 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:41:06.928 DEBUG:boto:Data: 2023-10-18T20:41:06.928 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.928 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.929 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.929 DEBUG:boto:Params: {} 2023-10-18T20:41:06.930 DEBUG:boto:Token: None 2023-10-18T20:41:06.930 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj85 2023-10-18T20:41:06.931 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eFQEesCmQ/ANYWHyBLuF7iKlrzo= 2023-10-18T20:41:06.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eFQEesCmQ/ANYWHyBLuF7iKlrzo=', 'Content-Length': '0'} 2023-10-18T20:41:06.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000839fa783bee10910-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.933 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.934 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.934 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.935 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.935 DEBUG:boto:Method: GET 2023-10-18T20:41:06.935 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.936 DEBUG:boto:Data: 2023-10-18T20:41:06.936 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.936 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.937 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.937 DEBUG:boto:Params: {} 2023-10-18T20:41:06.938 DEBUG:boto:Token: None 2023-10-18T20:41:06.938 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8f37d318f76564d-00653042e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.942 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.943 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:06.943 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:06.943 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.944 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.944 DEBUG:boto:Method: GET 2023-10-18T20:41:06.944 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:41:06.945 DEBUG:boto:Data: 2023-10-18T20:41:06.945 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.946 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.946 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.946 DEBUG:boto:Params: {} 2023-10-18T20:41:06.947 DEBUG:boto:Token: None 2023-10-18T20:41:06.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/ 2023-10-18T20:41:06.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8= 2023-10-18T20:41:06.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:szGwwVWsVJAyusGdBZFG355xqp8=', 'Content-Length': '0'} 2023-10-18T20:41:06.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000063d257b97db503f-00653042e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.952 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:06.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:41:06.953 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:06.954 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:06.954 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.954 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.955 DEBUG:boto:Method: GET 2023-10-18T20:41:06.955 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.956 DEBUG:boto:Data: 2023-10-18T20:41:06.956 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.956 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.957 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.957 DEBUG:boto:Params: {} 2023-10-18T20:41:06.958 DEBUG:boto:Token: None 2023-10-18T20:41:06.958 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bmMOLgTRbLFEhxg+IgPYmQi1qbY= 2023-10-18T20:41:06.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bmMOLgTRbLFEhxg+IgPYmQi1qbY=', 'Content-Length': '0'} 2023-10-18T20:41:06.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e516f20cf6825be1-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.962 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:06.963 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:06.963 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.963 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.964 DEBUG:boto:Method: GET 2023-10-18T20:41:06.964 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.965 DEBUG:boto:Data: 2023-10-18T20:41:06.965 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.965 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.966 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.966 DEBUG:boto:Params: {} 2023-10-18T20:41:06.967 DEBUG:boto:Token: None 2023-10-18T20:41:06.967 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bmMOLgTRbLFEhxg+IgPYmQi1qbY= 2023-10-18T20:41:06.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bmMOLgTRbLFEhxg+IgPYmQi1qbY=', 'Content-Length': '0'} 2023-10-18T20:41:06.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff88385892616c7-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.971 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:06.971 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:06.972 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.972 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.972 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.973 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.973 DEBUG:boto:Data: 2023-10-18T20:41:06.973 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.974 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.975 DEBUG:boto:Params: {} 2023-10-18T20:41:06.975 DEBUG:boto:Token: None 2023-10-18T20:41:06.976 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.976 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YXJPN9Zlb9+UaoQkydKn/Q5WSCw= 2023-10-18T20:41:06.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YXJPN9Zlb9+UaoQkydKn/Q5WSCw=', 'Content-Length': '0'} 2023-10-18T20:41:06.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8dab6573a89f489-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.979 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:06.979 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:06.980 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.980 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.981 DEBUG:boto:Method: HEAD 2023-10-18T20:41:06.981 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.981 DEBUG:boto:Data: 2023-10-18T20:41:06.982 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.982 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:06.982 DEBUG:boto:Port: 8000 2023-10-18T20:41:06.983 DEBUG:boto:Params: {} 2023-10-18T20:41:06.983 DEBUG:boto:Token: None 2023-10-18T20:41:06.984 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.984 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YXJPN9Zlb9+UaoQkydKn/Q5WSCw= 2023-10-18T20:41:06.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YXJPN9Zlb9+UaoQkydKn/Q5WSCw=', 'Content-Length': '0'} 2023-10-18T20:41:06.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f59fd027c9a816-00653042e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:06.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:41:06.988 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:06.988 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:06.989 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.989 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.989 DEBUG:boto:Method: GET 2023-10-18T20:41:06.990 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.990 DEBUG:boto:Data: 2023-10-18T20:41:06.991 DEBUG:boto:Headers: {} 2023-10-18T20:41:06.991 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:06.991 DEBUG:boto:Port: 8001 2023-10-18T20:41:06.992 DEBUG:boto:Params: {} 2023-10-18T20:41:06.992 DEBUG:boto:Token: None 2023-10-18T20:41:06.993 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:06 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:06.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bmMOLgTRbLFEhxg+IgPYmQi1qbY= 2023-10-18T20:41:06.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bmMOLgTRbLFEhxg+IgPYmQi1qbY=', 'Content-Length': '0'} 2023-10-18T20:41:06.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007084d0f3fd52fe30-00653042e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.038 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:07.039 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:07.039 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:07.040 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:41:07.040 DEBUG:boto:Method: GET 2023-10-18T20:41:07.041 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:07.041 DEBUG:boto:Data: 2023-10-18T20:41:07.041 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.042 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.043 DEBUG:boto:Params: {} 2023-10-18T20:41:07.043 DEBUG:boto:Token: None 2023-10-18T20:41:07.043 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:41:07.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PZWR7CtcU9TtR66Tci1uij5yiLA= 2023-10-18T20:41:07.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PZWR7CtcU9TtR66Tci1uij5yiLA=', 'Content-Length': '0'} 2023-10-18T20:41:07.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003060dbd40c7dbc33-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.047 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.048 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.048 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.048 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.049 DEBUG:boto:Method: GET 2023-10-18T20:41:07.049 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.049 DEBUG:boto:Data: 2023-10-18T20:41:07.050 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.051 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.051 DEBUG:boto:Params: {} 2023-10-18T20:41:07.051 DEBUG:boto:Token: None 2023-10-18T20:41:07.052 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083e213d2cf80c52a-00653042e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.056 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.057 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.057 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.058 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.058 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.058 DEBUG:boto:Method: GET 2023-10-18T20:41:07.059 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.059 DEBUG:boto:Data: 2023-10-18T20:41:07.060 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.060 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.060 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.061 DEBUG:boto:Params: {} 2023-10-18T20:41:07.061 DEBUG:boto:Token: None 2023-10-18T20:41:07.062 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.062 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027cdebb2e12366de-00653042e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.110 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.111 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:07.111 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:07.112 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.112 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:41:07.113 DEBUG:boto:Data: 2023-10-18T20:41:07.113 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.113 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.114 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.114 DEBUG:boto:Params: {} 2023-10-18T20:41:07.114 DEBUG:boto:Token: None 2023-10-18T20:41:07.115 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj87 2023-10-18T20:41:07.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01dQ2GFLHGy0h4eWfCy8IYLnBro= 2023-10-18T20:41:07.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01dQ2GFLHGy0h4eWfCy8IYLnBro=', 'Content-Length': '0'} 2023-10-18T20:41:07.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d586158b55f1baa-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.118 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:07.119 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:07.119 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.119 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:41:07.120 DEBUG:boto:Data: 2023-10-18T20:41:07.120 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.120 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.121 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.121 DEBUG:boto:Params: {} 2023-10-18T20:41:07.122 DEBUG:boto:Token: None 2023-10-18T20:41:07.122 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj87 2023-10-18T20:41:07.122 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:01dQ2GFLHGy0h4eWfCy8IYLnBro= 2023-10-18T20:41:07.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:01dQ2GFLHGy0h4eWfCy8IYLnBro=', 'Content-Length': '0'} 2023-10-18T20:41:07.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049169813305b000e-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:41:07.126 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:07.126 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:07.127 DEBUG:boto:Method: GET 2023-10-18T20:41:07.127 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:41:07.127 DEBUG:boto:Data: 2023-10-18T20:41:07.128 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.128 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.128 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.129 DEBUG:boto:Params: {} 2023-10-18T20:41:07.129 DEBUG:boto:Token: None 2023-10-18T20:41:07.130 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj87 2023-10-18T20:41:07.130 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O6jVzh665982wVRhre1kqIDYkLw= 2023-10-18T20:41:07.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O6jVzh665982wVRhre1kqIDYkLw=', 'Content-Length': '0'} 2023-10-18T20:41:07.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000321942be68757d40-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.133 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:41:07.134 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:41:07.134 DEBUG:boto:Method: GET 2023-10-18T20:41:07.134 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:41:07.135 DEBUG:boto:Data: 2023-10-18T20:41:07.135 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.136 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.136 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.136 DEBUG:boto:Params: {} 2023-10-18T20:41:07.137 DEBUG:boto:Token: None 2023-10-18T20:41:07.137 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj87 2023-10-18T20:41:07.138 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O6jVzh665982wVRhre1kqIDYkLw= 2023-10-18T20:41:07.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O6jVzh665982wVRhre1kqIDYkLw=', 'Content-Length': '0'} 2023-10-18T20:41:07.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64e548027e4a4c6-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.141 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.141 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.142 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.142 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.143 DEBUG:boto:Method: GET 2023-10-18T20:41:07.143 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.143 DEBUG:boto:Data: 2023-10-18T20:41:07.144 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.144 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.145 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.145 DEBUG:boto:Params: {} 2023-10-18T20:41:07.145 DEBUG:boto:Token: None 2023-10-18T20:41:07.146 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.146 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fe87d44f1505063-00653042e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.150 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.151 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.151 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.152 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.152 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.152 DEBUG:boto:Method: GET 2023-10-18T20:41:07.153 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:41:07.153 DEBUG:boto:Data: 2023-10-18T20:41:07.154 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.154 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.155 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.155 DEBUG:boto:Params: {} 2023-10-18T20:41:07.155 DEBUG:boto:Token: None 2023-10-18T20:41:07.156 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.156 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085080bf712752e65-00653042e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.161 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:41:07.162 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.162 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.163 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.163 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.164 DEBUG:boto:Method: GET 2023-10-18T20:41:07.164 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.165 DEBUG:boto:Data: 2023-10-18T20:41:07.165 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.165 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.166 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.166 DEBUG:boto:Params: {} 2023-10-18T20:41:07.167 DEBUG:boto:Token: None 2023-10-18T20:41:07.169 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.170 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AazuRTEx1cEv59+CBc3MK3UNxxU= 2023-10-18T20:41:07.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AazuRTEx1cEv59+CBc3MK3UNxxU=', 'Content-Length': '0'} 2023-10-18T20:41:07.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40dbb2bf2c2542f-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.173 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.173 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.174 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.174 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.175 DEBUG:boto:Method: GET 2023-10-18T20:41:07.175 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.176 DEBUG:boto:Data: 2023-10-18T20:41:07.176 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.176 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.177 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.177 DEBUG:boto:Params: {} 2023-10-18T20:41:07.178 DEBUG:boto:Token: None 2023-10-18T20:41:07.178 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.179 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AazuRTEx1cEv59+CBc3MK3UNxxU= 2023-10-18T20:41:07.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AazuRTEx1cEv59+CBc3MK3UNxxU=', 'Content-Length': '0'} 2023-10-18T20:41:07.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfc00955d24d1a72-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.182 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.182 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.183 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.183 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.183 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.184 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.184 DEBUG:boto:Data: 2023-10-18T20:41:07.185 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.185 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.186 DEBUG:boto:Params: {} 2023-10-18T20:41:07.186 DEBUG:boto:Token: None 2023-10-18T20:41:07.186 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.187 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JRk0ghp2nL2mCqxOWp/WaWN3PmQ= 2023-10-18T20:41:07.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JRk0ghp2nL2mCqxOWp/WaWN3PmQ=', 'Content-Length': '0'} 2023-10-18T20:41:07.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8271233b9bfbdc-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.190 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.190 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.191 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.191 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.191 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.192 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.192 DEBUG:boto:Data: 2023-10-18T20:41:07.192 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.193 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.193 DEBUG:boto:Params: {} 2023-10-18T20:41:07.194 DEBUG:boto:Token: None 2023-10-18T20:41:07.194 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.195 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JRk0ghp2nL2mCqxOWp/WaWN3PmQ= 2023-10-18T20:41:07.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JRk0ghp2nL2mCqxOWp/WaWN3PmQ=', 'Content-Length': '0'} 2023-10-18T20:41:07.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b32b6f290472e1d2-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:41:07.198 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.198 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.199 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.199 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.200 DEBUG:boto:Method: GET 2023-10-18T20:41:07.200 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.200 DEBUG:boto:Data: 2023-10-18T20:41:07.201 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.201 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.202 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.202 DEBUG:boto:Params: {} 2023-10-18T20:41:07.202 DEBUG:boto:Token: None 2023-10-18T20:41:07.203 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.203 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AazuRTEx1cEv59+CBc3MK3UNxxU= 2023-10-18T20:41:07.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AazuRTEx1cEv59+CBc3MK3UNxxU=', 'Content-Length': '0'} 2023-10-18T20:41:07.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c5ca144f94f4876-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.206 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.207 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.207 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.208 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.208 DEBUG:boto:Method: GET 2023-10-18T20:41:07.208 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.209 DEBUG:boto:Data: 2023-10-18T20:41:07.209 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.210 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.210 DEBUG:boto:Params: {} 2023-10-18T20:41:07.211 DEBUG:boto:Token: None 2023-10-18T20:41:07.211 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:41:07.212 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AazuRTEx1cEv59+CBc3MK3UNxxU= 2023-10-18T20:41:07.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AazuRTEx1cEv59+CBc3MK3UNxxU=', 'Content-Length': '0'} 2023-10-18T20:41:07.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3816a6bfce0bc1f-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.215 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.215 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.216 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.216 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.217 DEBUG:boto:Method: GET 2023-10-18T20:41:07.217 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.217 DEBUG:boto:Data: 2023-10-18T20:41:07.218 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.219 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.219 DEBUG:boto:Params: {} 2023-10-18T20:41:07.220 DEBUG:boto:Token: None 2023-10-18T20:41:07.220 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dd23f851548c1dd-00653042e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.224 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.225 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.225 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.226 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.226 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.226 DEBUG:boto:Method: GET 2023-10-18T20:41:07.227 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.227 DEBUG:boto:Data: 2023-10-18T20:41:07.228 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.228 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.229 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.229 DEBUG:boto:Params: {} 2023-10-18T20:41:07.229 DEBUG:boto:Token: None 2023-10-18T20:41:07.230 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.230 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c6e42fc0427a5d6-00653042e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.278 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.279 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.279 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.280 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.280 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:41:07.281 DEBUG:boto:Data: 2023-10-18T20:41:07.281 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.282 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.282 DEBUG:boto:Params: {} 2023-10-18T20:41:07.283 DEBUG:boto:Token: None 2023-10-18T20:41:07.283 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92 2023-10-18T20:41:07.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IFaeEejgbs5IJdUzDApmQU46NXA= 2023-10-18T20:41:07.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IFaeEejgbs5IJdUzDApmQU46NXA=', 'Content-Length': '0'} 2023-10-18T20:41:07.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c281fb433d29425e-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.287 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.287 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.288 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.288 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:41:07.288 DEBUG:boto:Data: 2023-10-18T20:41:07.289 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.289 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.290 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.290 DEBUG:boto:Params: {} 2023-10-18T20:41:07.291 DEBUG:boto:Token: None 2023-10-18T20:41:07.291 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92 2023-10-18T20:41:07.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IFaeEejgbs5IJdUzDApmQU46NXA= 2023-10-18T20:41:07.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IFaeEejgbs5IJdUzDApmQU46NXA=', 'Content-Length': '0'} 2023-10-18T20:41:07.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ac784c5769f275c-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:41:07.295 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.295 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.296 DEBUG:boto:Method: GET 2023-10-18T20:41:07.296 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:41:07.297 DEBUG:boto:Data: 2023-10-18T20:41:07.297 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.298 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.298 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.298 DEBUG:boto:Params: {} 2023-10-18T20:41:07.299 DEBUG:boto:Token: None 2023-10-18T20:41:07.299 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92 2023-10-18T20:41:07.300 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hQ/0OCANpgV8VS4nk/8C60NJAeA= 2023-10-18T20:41:07.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hQ/0OCANpgV8VS4nk/8C60NJAeA=', 'Content-Length': '0'} 2023-10-18T20:41:07.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c191fc5fa86e64c-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.303 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:41:07.303 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:41:07.304 DEBUG:boto:Method: GET 2023-10-18T20:41:07.304 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:41:07.304 DEBUG:boto:Data: 2023-10-18T20:41:07.305 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.305 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.306 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.306 DEBUG:boto:Params: {} 2023-10-18T20:41:07.307 DEBUG:boto:Token: None 2023-10-18T20:41:07.307 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj92 2023-10-18T20:41:07.308 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hQ/0OCANpgV8VS4nk/8C60NJAeA= 2023-10-18T20:41:07.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hQ/0OCANpgV8VS4nk/8C60NJAeA=', 'Content-Length': '0'} 2023-10-18T20:41:07.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ddd1b8c87039edd-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.311 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.311 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.312 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.312 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.312 DEBUG:boto:Method: GET 2023-10-18T20:41:07.313 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.313 DEBUG:boto:Data: 2023-10-18T20:41:07.313 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.314 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.315 DEBUG:boto:Params: {} 2023-10-18T20:41:07.315 DEBUG:boto:Token: None 2023-10-18T20:41:07.316 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.316 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c84c22f4dbedf8b-00653042e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.320 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.320 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.321 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.321 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.321 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.322 DEBUG:boto:Method: GET 2023-10-18T20:41:07.322 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:41:07.323 DEBUG:boto:Data: 2023-10-18T20:41:07.323 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.324 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.324 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.324 DEBUG:boto:Params: {} 2023-10-18T20:41:07.325 DEBUG:boto:Token: None 2023-10-18T20:41:07.325 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.326 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000868336f607101aa0-00653042e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.331 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:41:07.332 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.332 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.332 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.333 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.333 DEBUG:boto:Method: GET 2023-10-18T20:41:07.333 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.334 DEBUG:boto:Data: 2023-10-18T20:41:07.334 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.335 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.335 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.335 DEBUG:boto:Params: {} 2023-10-18T20:41:07.336 DEBUG:boto:Token: None 2023-10-18T20:41:07.336 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9/k4UjvhP8nfQ1hZSUt0Yrs1Y6M= 2023-10-18T20:41:07.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9/k4UjvhP8nfQ1hZSUt0Yrs1Y6M=', 'Content-Length': '0'} 2023-10-18T20:41:07.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003459ce449683664b-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.382 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.383 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.383 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.384 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.384 DEBUG:boto:Method: GET 2023-10-18T20:41:07.384 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.385 DEBUG:boto:Data: 2023-10-18T20:41:07.385 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.386 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.387 DEBUG:boto:Params: {} 2023-10-18T20:41:07.387 DEBUG:boto:Token: None 2023-10-18T20:41:07.388 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9/k4UjvhP8nfQ1hZSUt0Yrs1Y6M= 2023-10-18T20:41:07.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9/k4UjvhP8nfQ1hZSUt0Yrs1Y6M=', 'Content-Length': '0'} 2023-10-18T20:41:07.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2aaa39bd70f9815-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.391 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.391 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.392 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.392 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.392 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.393 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.393 DEBUG:boto:Data: 2023-10-18T20:41:07.393 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.394 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.394 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.395 DEBUG:boto:Params: {} 2023-10-18T20:41:07.395 DEBUG:boto:Token: None 2023-10-18T20:41:07.395 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.396 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gKIUItFaZ5DlelSrct/B40DckLg= 2023-10-18T20:41:07.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gKIUItFaZ5DlelSrct/B40DckLg=', 'Content-Length': '0'} 2023-10-18T20:41:07.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63be90f7597a8ce-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.399 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.399 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.400 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.400 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.400 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.401 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.401 DEBUG:boto:Data: 2023-10-18T20:41:07.401 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.402 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.402 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.402 DEBUG:boto:Params: {} 2023-10-18T20:41:07.403 DEBUG:boto:Token: None 2023-10-18T20:41:07.403 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gKIUItFaZ5DlelSrct/B40DckLg= 2023-10-18T20:41:07.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gKIUItFaZ5DlelSrct/B40DckLg=', 'Content-Length': '0'} 2023-10-18T20:41:07.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63c7c8421365245-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:41:07.407 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.408 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.408 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.408 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.409 DEBUG:boto:Method: GET 2023-10-18T20:41:07.409 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.409 DEBUG:boto:Data: 2023-10-18T20:41:07.410 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.410 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.411 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.411 DEBUG:boto:Params: {} 2023-10-18T20:41:07.411 DEBUG:boto:Token: None 2023-10-18T20:41:07.412 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.412 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9/k4UjvhP8nfQ1hZSUt0Yrs1Y6M= 2023-10-18T20:41:07.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9/k4UjvhP8nfQ1hZSUt0Yrs1Y6M=', 'Content-Length': '0'} 2023-10-18T20:41:07.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca1bad3e95aa6f5d-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.415 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.416 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.416 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.417 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.417 DEBUG:boto:Method: GET 2023-10-18T20:41:07.417 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.418 DEBUG:boto:Data: 2023-10-18T20:41:07.418 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.419 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.419 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.419 DEBUG:boto:Params: {} 2023-10-18T20:41:07.420 DEBUG:boto:Token: None 2023-10-18T20:41:07.420 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:41:07.421 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9/k4UjvhP8nfQ1hZSUt0Yrs1Y6M= 2023-10-18T20:41:07.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9/k4UjvhP8nfQ1hZSUt0Yrs1Y6M=', 'Content-Length': '0'} 2023-10-18T20:41:07.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9bb595ccee5dbb5-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.466 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.467 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.467 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.467 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.468 DEBUG:boto:Method: GET 2023-10-18T20:41:07.468 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.469 DEBUG:boto:Data: 2023-10-18T20:41:07.469 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.469 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.470 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.470 DEBUG:boto:Params: {} 2023-10-18T20:41:07.471 DEBUG:boto:Token: None 2023-10-18T20:41:07.471 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.471 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000abeec02cdf5cce4-00653042e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.475 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.475 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.476 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.476 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.477 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.477 DEBUG:boto:Method: GET 2023-10-18T20:41:07.477 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.478 DEBUG:boto:Data: 2023-10-18T20:41:07.478 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.479 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.479 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.479 DEBUG:boto:Params: {} 2023-10-18T20:41:07.480 DEBUG:boto:Token: None 2023-10-18T20:41:07.480 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.481 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075b9c48297435638-00653042e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.485 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.486 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.486 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.487 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.487 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:41:07.487 DEBUG:boto:Data: 2023-10-18T20:41:07.488 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.488 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.489 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.489 DEBUG:boto:Params: {} 2023-10-18T20:41:07.489 DEBUG:boto:Token: None 2023-10-18T20:41:07.490 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95 2023-10-18T20:41:07.490 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XA3UVAmBRbjFJS4E2IOfv5/Uhoc= 2023-10-18T20:41:07.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XA3UVAmBRbjFJS4E2IOfv5/Uhoc=', 'Content-Length': '0'} 2023-10-18T20:41:07.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000864d78bef730f983-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.493 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.494 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.494 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.495 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:41:07.495 DEBUG:boto:Data: 2023-10-18T20:41:07.495 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.496 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.496 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.496 DEBUG:boto:Params: {} 2023-10-18T20:41:07.497 DEBUG:boto:Token: None 2023-10-18T20:41:07.497 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95 2023-10-18T20:41:07.498 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XA3UVAmBRbjFJS4E2IOfv5/Uhoc= 2023-10-18T20:41:07.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XA3UVAmBRbjFJS4E2IOfv5/Uhoc=', 'Content-Length': '0'} 2023-10-18T20:41:07.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000499595c5d0e2ada9-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:41:07.501 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.502 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.502 DEBUG:boto:Method: GET 2023-10-18T20:41:07.502 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:41:07.503 DEBUG:boto:Data: 2023-10-18T20:41:07.503 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.504 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.504 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.504 DEBUG:boto:Params: {} 2023-10-18T20:41:07.505 DEBUG:boto:Token: None 2023-10-18T20:41:07.505 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95 2023-10-18T20:41:07.506 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vhM4RWuuAmD49BfSfiXYaV7czsA= 2023-10-18T20:41:07.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vhM4RWuuAmD49BfSfiXYaV7czsA=', 'Content-Length': '0'} 2023-10-18T20:41:07.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000142cd421c5830e2c-00653042e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.509 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:41:07.509 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:41:07.510 DEBUG:boto:Method: GET 2023-10-18T20:41:07.510 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:41:07.511 DEBUG:boto:Data: 2023-10-18T20:41:07.511 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.511 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.512 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.512 DEBUG:boto:Params: {} 2023-10-18T20:41:07.512 DEBUG:boto:Token: None 2023-10-18T20:41:07.513 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj95 2023-10-18T20:41:07.513 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vhM4RWuuAmD49BfSfiXYaV7czsA= 2023-10-18T20:41:07.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vhM4RWuuAmD49BfSfiXYaV7czsA=', 'Content-Length': '0'} 2023-10-18T20:41:07.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042341767cb0a04eb-00653042e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.517 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.517 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.517 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.518 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.518 DEBUG:boto:Method: GET 2023-10-18T20:41:07.519 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.519 DEBUG:boto:Data: 2023-10-18T20:41:07.519 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.520 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:41:07.520 DEBUG:boto:Port: 8001 2023-10-18T20:41:07.521 DEBUG:boto:Params: {} 2023-10-18T20:41:07.521 DEBUG:boto:Token: None 2023-10-18T20:41:07.522 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.522 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d2c8463569ee77d-00653042e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.525 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.526 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.527 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.527 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.527 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.528 DEBUG:boto:Method: GET 2023-10-18T20:41:07.528 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:41:07.529 DEBUG:boto:Data: 2023-10-18T20:41:07.529 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.530 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.531 DEBUG:boto:Params: {} 2023-10-18T20:41:07.531 DEBUG:boto:Token: None 2023-10-18T20:41:07.531 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs= 2023-10-18T20:41:07.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t6zNDgT865UslLjLjEHVcRmMxDs=', 'Content-Length': '0'} 2023-10-18T20:41:07.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062f5949446065c87-00653042e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.536 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:41:07.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:41:07.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:41:07.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:41:07.539 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:41:07.598 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:41:07.598 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:41:07.651 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:41:07.652 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-59 2023-10-18T20:41:07.662 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 25 2023-10-18T20:41:07.717 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.717 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.718 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.718 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:41:07.718 DEBUG:boto:Data: 2023-10-18T20:41:07.719 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.719 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.720 DEBUG:boto:Params: {} 2023-10-18T20:41:07.720 DEBUG:boto:Token: None 2023-10-18T20:41:07.721 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0= 2023-10-18T20:41:07.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0=', 'Content-Length': '0'} 2023-10-18T20:41:07.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009eeeb8d7999a6501-00653042e3-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.725 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:41:07.726 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:41:07.726 DEBUG:boto:Method: PUT 2023-10-18T20:41:07.727 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:41:07.727 DEBUG:boto:Data: 2023-10-18T20:41:07.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:41:07.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.728 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.729 DEBUG:boto:Params: {} 2023-10-18T20:41:07.729 DEBUG:boto:Token: None 2023-10-18T20:41:07.730 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj31 2023-10-18T20:41:07.730 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tvTwcK5JiTIC6+2/Y2aLXarpxWE= 2023-10-18T20:41:07.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tvTwcK5JiTIC6+2/Y2aLXarpxWE='} 2023-10-18T20:41:07.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000227c298c01274d23-00653042e3-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.736 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.737 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.737 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.737 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:41:07.738 DEBUG:boto:Data: 2023-10-18T20:41:07.738 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.739 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.739 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.739 DEBUG:boto:Params: {} 2023-10-18T20:41:07.740 DEBUG:boto:Token: None 2023-10-18T20:41:07.740 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.741 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0= 2023-10-18T20:41:07.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0=', 'Content-Length': '0'} 2023-10-18T20:41:07.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baf87c22b35c9817-00653042e3-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.745 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:41:07.745 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:41:07.746 DEBUG:boto:Method: PUT 2023-10-18T20:41:07.746 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:41:07.747 DEBUG:boto:Data: 2023-10-18T20:41:07.747 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:41:07.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.748 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.748 DEBUG:boto:Params: {} 2023-10-18T20:41:07.749 DEBUG:boto:Token: None 2023-10-18T20:41:07.749 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj62 2023-10-18T20:41:07.750 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1fmuhkkasRrkstSbrMBDW62gFjk= 2023-10-18T20:41:07.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1fmuhkkasRrkstSbrMBDW62gFjk='} 2023-10-18T20:41:07.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019f6a28b3678bb1b-00653042e3-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.795 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.795 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.796 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.796 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:41:07.796 DEBUG:boto:Data: 2023-10-18T20:41:07.797 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.797 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.797 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.798 DEBUG:boto:Params: {} 2023-10-18T20:41:07.798 DEBUG:boto:Token: None 2023-10-18T20:41:07.799 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0= 2023-10-18T20:41:07.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0=', 'Content-Length': '0'} 2023-10-18T20:41:07.803 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ad920b93e8373d8-00653042e3-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.803 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:41:07.804 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:41:07.804 DEBUG:boto:Method: PUT 2023-10-18T20:41:07.805 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:41:07.805 DEBUG:boto:Data: 2023-10-18T20:41:07.805 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:41:07.806 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.806 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.807 DEBUG:boto:Params: {} 2023-10-18T20:41:07.807 DEBUG:boto:Token: None 2023-10-18T20:41:07.807 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj93 2023-10-18T20:41:07.808 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f8nfdk8BiUk1VNZmCh2osegYLEg= 2023-10-18T20:41:07.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f8nfdk8BiUk1VNZmCh2osegYLEg='} 2023-10-18T20:41:07.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc2da9da03211959-00653042e3-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.814 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.815 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.815 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.815 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:41:07.816 DEBUG:boto:Data: 2023-10-18T20:41:07.816 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.817 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.817 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.817 DEBUG:boto:Params: {} 2023-10-18T20:41:07.818 DEBUG:boto:Token: None 2023-10-18T20:41:07.818 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.819 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0= 2023-10-18T20:41:07.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0=', 'Content-Length': '0'} 2023-10-18T20:41:07.822 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008071e74f6a4f6b4-00653042e3-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.823 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:41:07.823 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:41:07.823 DEBUG:boto:Method: PUT 2023-10-18T20:41:07.824 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:41:07.824 DEBUG:boto:Data: 2023-10-18T20:41:07.824 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:41:07.825 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.825 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.825 DEBUG:boto:Params: {} 2023-10-18T20:41:07.826 DEBUG:boto:Token: None 2023-10-18T20:41:07.826 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj124 2023-10-18T20:41:07.827 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EKo+RBsc+Z/1oBSj44At6f8cVlg= 2023-10-18T20:41:07.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EKo+RBsc+Z/1oBSj44At6f8cVlg='} 2023-10-18T20:41:07.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001afdd1792f9016f5-00653042e3-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.875 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:41:07.875 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:41:07.876 DEBUG:boto:Method: HEAD 2023-10-18T20:41:07.876 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:41:07.877 DEBUG:boto:Data: 2023-10-18T20:41:07.877 DEBUG:boto:Headers: {} 2023-10-18T20:41:07.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.878 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.878 DEBUG:boto:Params: {} 2023-10-18T20:41:07.879 DEBUG:boto:Token: None 2023-10-18T20:41:07.879 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/ 2023-10-18T20:41:07.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0= 2023-10-18T20:41:07.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Zdw2SHj6BTwQb2wd7HxlDPM3Q0=', 'Content-Length': '0'} 2023-10-18T20:41:07.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d38e2ce3e8150f65-00653042e3-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.884 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:41:07.884 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:41:07.884 DEBUG:boto:Method: PUT 2023-10-18T20:41:07.885 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:41:07.885 DEBUG:boto:Data: 2023-10-18T20:41:07.886 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:41:07.886 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:41:07.887 DEBUG:boto:Port: 8000 2023-10-18T20:41:07.887 DEBUG:boto:Params: {} 2023-10-18T20:41:07.887 DEBUG:boto:Token: None 2023-10-18T20:41:07.888 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:41:07 GMT /bofrnf-59/obj155 2023-10-18T20:41:07.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SlG6eypJrx0X/2wfT53w5S+G1Us= 2023-10-18T20:41:07.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:41:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SlG6eypJrx0X/2wfT53w5S+G1Us='} 2023-10-18T20:41:07.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca38725ad6d8bd92-00653042e3-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:41:07.894 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-59 2023-10-18T20:41:07.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:41:07.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:41:07.896 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:41:07.952 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:41:07.952 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:41:07.987 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:07.985+0000 7fda43c89780 1 waiting to reach latest gen 5, on 4.. 2023-10-18T20:41:12.987 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:12.986+0000 7fda43c89780 1 waiting to reach latest gen 5, on 4.. 2023-10-18T20:41:17.988 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:17.986+0000 7fda43c89780 1 waiting to reach latest gen 5, on 4.. 2023-10-18T20:41:22.989 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:22.987+0000 7fda43c89780 1 waiting to reach latest gen 5, on 4.. 2023-10-18T20:41:27.991 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:27.990+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:41:27.992 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:41:27.992 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:32.993 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:32.992+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:41:32.994 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:32.994 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:37.995 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:37.993+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:41:37.996 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:37.996 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:42.998 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:42.995+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:41:42.998 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:42.999 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:48.000 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:47.998+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:41:48.001 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:48.001 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:53.002 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:53.001+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:41:53.003 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:53.003 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:58.005 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:41:58.003+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:41:58.005 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:41:58.006 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:03.007 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:03.005+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:03.007 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:03.007 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:08.009 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:08.008+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:08.010 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:08.010 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:13.011 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:13.010+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:13.012 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:13.013 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:18.014 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:18.012+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:18.014 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:18.014 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:23.016 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:23.014+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:23.016 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:23.016 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:28.018 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:28.016+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:28.019 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:28.019 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:33.020 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:33.018+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:33.021 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:33.021 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:38.023 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:38.020+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:38.023 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:38.023 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:43.025 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:43.023+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:43.025 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:43.025 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:48.027 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:48.026+0000 7fda43c89780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:48.028 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:48.028 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:53.029 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:53.028+0000 7fda43c89780 1 bucket sync caught up with source: 2023-10-18T20:42:53.030 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:53.030 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , 00000000002.53.6, , 00000000001.100.6, , 00000000001.70.6, , , , , , , , , , 00000000001.93.6, , , , , , , , , , , , , , 00000000001.68.6] 2023-10-18T20:42:53.030 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:53.028+0000 7fda43c89780 0 bucket checkpoint complete 2023-10-18T20:42:53.047 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-59 2023-10-18T20:42:53.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:42:53.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:42:53.049 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:42:53.110 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:42:53.111 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:42:53.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:53.146+0000 7fce8f0d2780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:53.147 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000003.706.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000003.260.6, 00000000006.464.6, 00000000004.248.6] 2023-10-18T20:42:53.148 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000004.710.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000003.260.6, 00000000006.464.6, 00000000004.248.6] 2023-10-18T20:42:58.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:42:58.147+0000 7fce8f0d2780 1 waiting for incremental sync to catch up: 2023-10-18T20:42:58.149 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000003.706.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000003.260.6, 00000000006.464.6, 00000000004.248.6] 2023-10-18T20:42:58.149 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000004.710.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000003.260.6, 00000000006.464.6, 00000000004.248.6] 2023-10-18T20:43:03.150 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:03.148+0000 7fce8f0d2780 1 bucket sync caught up with source: 2023-10-18T20:43:03.150 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000004.710.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000003.260.6, 00000000006.464.6, 00000000004.248.6] 2023-10-18T20:43:03.151 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000004.710.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000003.260.6, 00000000006.464.6, 00000000004.248.6] 2023-10-18T20:43:03.151 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:03.148+0000 7fce8f0d2780 0 bucket checkpoint complete 2023-10-18T20:43:03.167 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:43:03.168 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.168 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.169 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.169 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:43:03.169 DEBUG:boto:Data: 2023-10-18T20:43:03.170 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.170 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.171 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.171 DEBUG:boto:Params: {} 2023-10-18T20:43:03.172 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:43:03.172 DEBUG:boto:Token: None 2023-10-18T20:43:03.173 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.173 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iF+4errr7W8MIkKvYLuhgqaCPJ8= 2023-10-18T20:43:03.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iF+4errr7W8MIkKvYLuhgqaCPJ8=', 'Content-Length': '0'} 2023-10-18T20:43:03.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039ce85d664303218-0065304357-1af4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.179 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.179 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.180 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.180 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:43:03.180 DEBUG:boto:Data: 2023-10-18T20:43:03.181 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.181 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.182 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.182 DEBUG:boto:Params: {} 2023-10-18T20:43:03.183 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:43:03.183 DEBUG:boto:Token: None 2023-10-18T20:43:03.184 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.184 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iF+4errr7W8MIkKvYLuhgqaCPJ8= 2023-10-18T20:43:03.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iF+4errr7W8MIkKvYLuhgqaCPJ8=', 'Content-Length': '0'} 2023-10-18T20:43:03.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfec79ad7ca1c6b4-0065304357-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.190 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:43:03.191 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.191 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.191 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.192 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.192 DEBUG:boto:Method: GET 2023-10-18T20:43:03.192 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.193 DEBUG:boto:Data: 2023-10-18T20:43:03.193 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.194 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.194 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.194 DEBUG:boto:Params: {} 2023-10-18T20:43:03.195 DEBUG:boto:Token: None 2023-10-18T20:43:03.195 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/?versions 2023-10-18T20:43:03.196 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2sb3AePk0uqjfp2XYH7de5EEHnE= 2023-10-18T20:43:03.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2sb3AePk0uqjfp2XYH7de5EEHnE=', 'Content-Length': '0'} 2023-10-18T20:43:03.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b30b8bc2a9eee10-0065304357-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.200 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:43:03.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:43:03.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-18T20:43:03.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-18T20:43:03.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-18T20:43:03.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-18T20:43:03.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:43:03.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:43:03.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:43:03.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:43:03.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-18T20:43:03.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-18T20:43:03.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:43:03.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:43:03.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:43:03.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:43:03.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:43:03.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:43:03.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:43:03.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:43:03.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-18T20:43:03.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-18T20:43:03.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:43:03.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:43:03.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:43:03.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:43:03.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-18T20:43:03.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:43:03.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-18T20:43:03.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:43:03.212 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:43:03.213 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.213 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.214 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.214 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.214 DEBUG:boto:Method: GET 2023-10-18T20:43:03.215 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.215 DEBUG:boto:Data: 2023-10-18T20:43:03.216 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.216 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.216 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.217 DEBUG:boto:Params: {} 2023-10-18T20:43:03.217 DEBUG:boto:Token: None 2023-10-18T20:43:03.218 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/?versions 2023-10-18T20:43:03.218 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2sb3AePk0uqjfp2XYH7de5EEHnE= 2023-10-18T20:43:03.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2sb3AePk0uqjfp2XYH7de5EEHnE=', 'Content-Length': '0'} 2023-10-18T20:43:03.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff043020dbdaa441-0065304357-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.223 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:43:03.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:43:03.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-18T20:43:03.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-18T20:43:03.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-18T20:43:03.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-18T20:43:03.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:43:03.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:43:03.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:43:03.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:43:03.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-18T20:43:03.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-18T20:43:03.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:43:03.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:43:03.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:43:03.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:43:03.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:43:03.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:43:03.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:43:03.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:43:03.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-18T20:43:03.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-18T20:43:03.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:43:03.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:43:03.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:43:03.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:43:03.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-18T20:43:03.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:43:03.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-18T20:43:03.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:43:03.235 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.235 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.236 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.236 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.236 DEBUG:boto:Method: GET 2023-10-18T20:43:03.237 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.237 DEBUG:boto:Data: 2023-10-18T20:43:03.237 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.238 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.238 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.239 DEBUG:boto:Params: {} 2023-10-18T20:43:03.239 DEBUG:boto:Token: None 2023-10-18T20:43:03.239 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/?versions 2023-10-18T20:43:03.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2sb3AePk0uqjfp2XYH7de5EEHnE= 2023-10-18T20:43:03.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2sb3AePk0uqjfp2XYH7de5EEHnE=', 'Content-Length': '0'} 2023-10-18T20:43:03.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f00a4ab4a74c41e-0065304357-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.244 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.245 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.246 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.246 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.247 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.247 DEBUG:boto:Method: GET 2023-10-18T20:43:03.248 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:43:03.248 DEBUG:boto:Data: 2023-10-18T20:43:03.249 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.249 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.250 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.250 DEBUG:boto:Params: {} 2023-10-18T20:43:03.251 DEBUG:boto:Token: None 2023-10-18T20:43:03.251 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/?versions 2023-10-18T20:43:03.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2sb3AePk0uqjfp2XYH7de5EEHnE= 2023-10-18T20:43:03.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2sb3AePk0uqjfp2XYH7de5EEHnE=', 'Content-Length': '0'} 2023-10-18T20:43:03.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bb8afd5e55da388-0065304357-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.257 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:43:03.259 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.260 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.260 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.260 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.261 DEBUG:boto:Method: GET 2023-10-18T20:43:03.261 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.261 DEBUG:boto:Data: 2023-10-18T20:43:03.262 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.262 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.263 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.263 DEBUG:boto:Params: {} 2023-10-18T20:43:03.264 DEBUG:boto:Token: None 2023-10-18T20:43:03.264 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.264 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f2N+Qg1CbyDwbftssMhkmdwLJ4s= 2023-10-18T20:43:03.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f2N+Qg1CbyDwbftssMhkmdwLJ4s=', 'Content-Length': '0'} 2023-10-18T20:43:03.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008df384d3255ebb23-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.267 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.268 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.268 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.269 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.269 DEBUG:boto:Method: GET 2023-10-18T20:43:03.270 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.270 DEBUG:boto:Data: 2023-10-18T20:43:03.270 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.271 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.271 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.272 DEBUG:boto:Params: {} 2023-10-18T20:43:03.272 DEBUG:boto:Token: None 2023-10-18T20:43:03.272 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.273 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f2N+Qg1CbyDwbftssMhkmdwLJ4s= 2023-10-18T20:43:03.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f2N+Qg1CbyDwbftssMhkmdwLJ4s=', 'Content-Length': '0'} 2023-10-18T20:43:03.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a169134d6e688f-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.276 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.276 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.277 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.277 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.277 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.278 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.278 DEBUG:boto:Data: 2023-10-18T20:43:03.279 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.279 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.280 DEBUG:boto:Params: {} 2023-10-18T20:43:03.280 DEBUG:boto:Token: None 2023-10-18T20:43:03.281 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.281 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6cxlr1+FPYtLdu2omedGCXrscOw= 2023-10-18T20:43:03.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6cxlr1+FPYtLdu2omedGCXrscOw=', 'Content-Length': '0'} 2023-10-18T20:43:03.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005691193d45dd23ab-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.284 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.285 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.285 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.285 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.286 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.286 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.286 DEBUG:boto:Data: 2023-10-18T20:43:03.287 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.287 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.288 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.288 DEBUG:boto:Params: {} 2023-10-18T20:43:03.288 DEBUG:boto:Token: None 2023-10-18T20:43:03.289 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.290 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6cxlr1+FPYtLdu2omedGCXrscOw= 2023-10-18T20:43:03.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6cxlr1+FPYtLdu2omedGCXrscOw=', 'Content-Length': '0'} 2023-10-18T20:43:03.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb711167aa9d31b-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:43:03.293 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.294 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.294 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.294 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.295 DEBUG:boto:Method: GET 2023-10-18T20:43:03.295 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.296 DEBUG:boto:Data: 2023-10-18T20:43:03.296 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.296 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.297 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.297 DEBUG:boto:Params: {} 2023-10-18T20:43:03.298 DEBUG:boto:Token: None 2023-10-18T20:43:03.298 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.299 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f2N+Qg1CbyDwbftssMhkmdwLJ4s= 2023-10-18T20:43:03.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f2N+Qg1CbyDwbftssMhkmdwLJ4s=', 'Content-Length': '0'} 2023-10-18T20:43:03.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f50390b6a61905b5-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.342 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.343 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.343 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.344 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.344 DEBUG:boto:Method: GET 2023-10-18T20:43:03.344 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.345 DEBUG:boto:Data: 2023-10-18T20:43:03.345 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.346 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.347 DEBUG:boto:Params: {} 2023-10-18T20:43:03.347 DEBUG:boto:Token: None 2023-10-18T20:43:03.348 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:43:03.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f2N+Qg1CbyDwbftssMhkmdwLJ4s= 2023-10-18T20:43:03.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f2N+Qg1CbyDwbftssMhkmdwLJ4s=', 'Content-Length': '0'} 2023-10-18T20:43:03.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008869f4e1a6f28526-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.352 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.352 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.353 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.353 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.354 DEBUG:boto:Method: GET 2023-10-18T20:43:03.354 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.354 DEBUG:boto:Data: 2023-10-18T20:43:03.355 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.355 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.356 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.356 DEBUG:boto:Params: {} 2023-10-18T20:43:03.357 DEBUG:boto:Token: None 2023-10-18T20:43:03.357 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af182176761c108b-0065304357-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.362 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.362 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.363 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.363 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.364 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.364 DEBUG:boto:Method: GET 2023-10-18T20:43:03.364 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.365 DEBUG:boto:Data: 2023-10-18T20:43:03.365 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.365 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.366 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.366 DEBUG:boto:Params: {} 2023-10-18T20:43:03.367 DEBUG:boto:Token: None 2023-10-18T20:43:03.367 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.368 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678c4ed43ae6f7bc-0065304357-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.418 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.419 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.419 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.420 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.420 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:43:03.420 DEBUG:boto:Data: 2023-10-18T20:43:03.421 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.421 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.422 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.422 DEBUG:boto:Params: {} 2023-10-18T20:43:03.422 DEBUG:boto:Token: None 2023-10-18T20:43:03.423 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11 2023-10-18T20:43:03.423 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:igLLszqajKdkx7IiUUjXxMyCX6M= 2023-10-18T20:43:03.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:igLLszqajKdkx7IiUUjXxMyCX6M=', 'Content-Length': '0'} 2023-10-18T20:43:03.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008407e073b2244edb-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.426 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.427 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.427 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.427 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:43:03.428 DEBUG:boto:Data: 2023-10-18T20:43:03.428 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.428 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.429 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.429 DEBUG:boto:Params: {} 2023-10-18T20:43:03.430 DEBUG:boto:Token: None 2023-10-18T20:43:03.430 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11 2023-10-18T20:43:03.431 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:igLLszqajKdkx7IiUUjXxMyCX6M= 2023-10-18T20:43:03.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:igLLszqajKdkx7IiUUjXxMyCX6M=', 'Content-Length': '0'} 2023-10-18T20:43:03.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000550e70ac79d63534-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:43:03.434 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.435 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.435 DEBUG:boto:Method: GET 2023-10-18T20:43:03.435 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:43:03.436 DEBUG:boto:Data: 2023-10-18T20:43:03.436 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.436 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.437 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.437 DEBUG:boto:Params: {} 2023-10-18T20:43:03.438 DEBUG:boto:Token: None 2023-10-18T20:43:03.438 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11 2023-10-18T20:43:03.439 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BKT+jp8Tz8vv/VbSTjH5dpwfOXk= 2023-10-18T20:43:03.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BKT+jp8Tz8vv/VbSTjH5dpwfOXk=', 'Content-Length': '0'} 2023-10-18T20:43:03.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008df0382bf4311dda-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.441 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:43:03.442 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:43:03.442 DEBUG:boto:Method: GET 2023-10-18T20:43:03.443 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:43:03.443 DEBUG:boto:Data: 2023-10-18T20:43:03.444 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.444 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.444 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.445 DEBUG:boto:Params: {} 2023-10-18T20:43:03.445 DEBUG:boto:Token: None 2023-10-18T20:43:03.445 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj11 2023-10-18T20:43:03.446 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BKT+jp8Tz8vv/VbSTjH5dpwfOXk= 2023-10-18T20:43:03.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BKT+jp8Tz8vv/VbSTjH5dpwfOXk=', 'Content-Length': '0'} 2023-10-18T20:43:03.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076dd3e8112f1d674-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.449 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.449 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.450 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.450 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.451 DEBUG:boto:Method: GET 2023-10-18T20:43:03.451 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.451 DEBUG:boto:Data: 2023-10-18T20:43:03.452 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.453 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.453 DEBUG:boto:Params: {} 2023-10-18T20:43:03.453 DEBUG:boto:Token: None 2023-10-18T20:43:03.454 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.454 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a74b497be270d42-0065304357-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.458 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.458 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.459 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.459 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.460 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.460 DEBUG:boto:Method: GET 2023-10-18T20:43:03.461 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:43:03.461 DEBUG:boto:Data: 2023-10-18T20:43:03.461 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.462 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.462 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.463 DEBUG:boto:Params: {} 2023-10-18T20:43:03.463 DEBUG:boto:Token: None 2023-10-18T20:43:03.464 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.464 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b4f6a26085d0fb-0065304357-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.469 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:43:03.470 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.470 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.471 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.471 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.472 DEBUG:boto:Method: GET 2023-10-18T20:43:03.472 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.472 DEBUG:boto:Data: 2023-10-18T20:43:03.473 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.473 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.473 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.474 DEBUG:boto:Params: {} 2023-10-18T20:43:03.474 DEBUG:boto:Token: None 2023-10-18T20:43:03.475 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.475 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wn7hPn5jOOJSd4jHk0+VFpuYhXU= 2023-10-18T20:43:03.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wn7hPn5jOOJSd4jHk0+VFpuYhXU=', 'Content-Length': '0'} 2023-10-18T20:43:03.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035669f34f322de64-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.478 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.479 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.479 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.479 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.480 DEBUG:boto:Method: GET 2023-10-18T20:43:03.480 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.481 DEBUG:boto:Data: 2023-10-18T20:43:03.481 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.482 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.482 DEBUG:boto:Params: {} 2023-10-18T20:43:03.483 DEBUG:boto:Token: None 2023-10-18T20:43:03.483 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.484 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wn7hPn5jOOJSd4jHk0+VFpuYhXU= 2023-10-18T20:43:03.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wn7hPn5jOOJSd4jHk0+VFpuYhXU=', 'Content-Length': '0'} 2023-10-18T20:43:03.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000885b68b73104414e-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.487 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.487 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.488 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.488 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.488 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.489 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.489 DEBUG:boto:Data: 2023-10-18T20:43:03.489 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.490 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.490 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.491 DEBUG:boto:Params: {} 2023-10-18T20:43:03.491 DEBUG:boto:Token: None 2023-10-18T20:43:03.491 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.492 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fXyWgjHRubXfCm8yKKEXzHDAaFg= 2023-10-18T20:43:03.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fXyWgjHRubXfCm8yKKEXzHDAaFg=', 'Content-Length': '0'} 2023-10-18T20:43:03.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c8f79beecc7b638-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.495 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.495 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.496 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.496 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.496 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.497 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.497 DEBUG:boto:Data: 2023-10-18T20:43:03.497 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.498 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.499 DEBUG:boto:Params: {} 2023-10-18T20:43:03.499 DEBUG:boto:Token: None 2023-10-18T20:43:03.499 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.500 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fXyWgjHRubXfCm8yKKEXzHDAaFg= 2023-10-18T20:43:03.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fXyWgjHRubXfCm8yKKEXzHDAaFg=', 'Content-Length': '0'} 2023-10-18T20:43:03.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e112d5c1b45fe1-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:43:03.503 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.503 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.504 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.504 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.504 DEBUG:boto:Method: GET 2023-10-18T20:43:03.505 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.505 DEBUG:boto:Data: 2023-10-18T20:43:03.506 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.506 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.506 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.507 DEBUG:boto:Params: {} 2023-10-18T20:43:03.507 DEBUG:boto:Token: None 2023-10-18T20:43:03.508 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.508 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wn7hPn5jOOJSd4jHk0+VFpuYhXU= 2023-10-18T20:43:03.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wn7hPn5jOOJSd4jHk0+VFpuYhXU=', 'Content-Length': '0'} 2023-10-18T20:43:03.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de980b3125b1059f-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.511 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.512 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.512 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.512 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.513 DEBUG:boto:Method: GET 2023-10-18T20:43:03.513 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.514 DEBUG:boto:Data: 2023-10-18T20:43:03.514 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.515 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.515 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.515 DEBUG:boto:Params: {} 2023-10-18T20:43:03.516 DEBUG:boto:Token: None 2023-10-18T20:43:03.516 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:43:03.517 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wn7hPn5jOOJSd4jHk0+VFpuYhXU= 2023-10-18T20:43:03.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wn7hPn5jOOJSd4jHk0+VFpuYhXU=', 'Content-Length': '0'} 2023-10-18T20:43:03.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007927dc91eb663cb8-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.562 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.563 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.563 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.564 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.564 DEBUG:boto:Method: GET 2023-10-18T20:43:03.564 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.565 DEBUG:boto:Data: 2023-10-18T20:43:03.565 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.566 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.566 DEBUG:boto:Params: {} 2023-10-18T20:43:03.567 DEBUG:boto:Token: None 2023-10-18T20:43:03.567 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd525d88ffaf87b2-0065304357-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.572 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.572 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.573 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.573 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.574 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.574 DEBUG:boto:Method: GET 2023-10-18T20:43:03.575 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.575 DEBUG:boto:Data: 2023-10-18T20:43:03.575 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.576 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.576 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.576 DEBUG:boto:Params: {} 2023-10-18T20:43:03.577 DEBUG:boto:Token: None 2023-10-18T20:43:03.577 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.578 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6f9cced98157d9d-0065304357-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.582 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.583 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.583 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.584 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.584 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:43:03.584 DEBUG:boto:Data: 2023-10-18T20:43:03.585 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.585 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.586 DEBUG:boto:Params: {} 2023-10-18T20:43:03.586 DEBUG:boto:Token: None 2023-10-18T20:43:03.587 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115 2023-10-18T20:43:03.587 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1wNzEf6nIvdE6quWxG7JClDQp5c= 2023-10-18T20:43:03.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1wNzEf6nIvdE6quWxG7JClDQp5c=', 'Content-Length': '0'} 2023-10-18T20:43:03.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbad29bfdc6e1f32-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.590 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.591 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.591 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.591 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:43:03.592 DEBUG:boto:Data: 2023-10-18T20:43:03.592 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.593 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.593 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.593 DEBUG:boto:Params: {} 2023-10-18T20:43:03.594 DEBUG:boto:Token: None 2023-10-18T20:43:03.594 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115 2023-10-18T20:43:03.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1wNzEf6nIvdE6quWxG7JClDQp5c= 2023-10-18T20:43:03.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1wNzEf6nIvdE6quWxG7JClDQp5c=', 'Content-Length': '0'} 2023-10-18T20:43:03.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056d9e26f99dbe269-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:43:03.598 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.598 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.599 DEBUG:boto:Method: GET 2023-10-18T20:43:03.599 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:43:03.599 DEBUG:boto:Data: 2023-10-18T20:43:03.600 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.600 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.601 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.601 DEBUG:boto:Params: {} 2023-10-18T20:43:03.602 DEBUG:boto:Token: None 2023-10-18T20:43:03.602 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115 2023-10-18T20:43:03.602 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tueF1qovjpS5xzMiP2rgpA4IYyQ= 2023-10-18T20:43:03.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tueF1qovjpS5xzMiP2rgpA4IYyQ=', 'Content-Length': '0'} 2023-10-18T20:43:03.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008984cf6fb68c5d54-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.650 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:43:03.651 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:43:03.651 DEBUG:boto:Method: GET 2023-10-18T20:43:03.652 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:43:03.652 DEBUG:boto:Data: 2023-10-18T20:43:03.653 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.653 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.654 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.654 DEBUG:boto:Params: {} 2023-10-18T20:43:03.655 DEBUG:boto:Token: None 2023-10-18T20:43:03.655 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj115 2023-10-18T20:43:03.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tueF1qovjpS5xzMiP2rgpA4IYyQ= 2023-10-18T20:43:03.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tueF1qovjpS5xzMiP2rgpA4IYyQ=', 'Content-Length': '0'} 2023-10-18T20:43:03.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f846f16012218fd-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.659 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.659 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.660 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.660 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.661 DEBUG:boto:Method: GET 2023-10-18T20:43:03.661 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.662 DEBUG:boto:Data: 2023-10-18T20:43:03.662 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.663 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.663 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.664 DEBUG:boto:Params: {} 2023-10-18T20:43:03.664 DEBUG:boto:Token: None 2023-10-18T20:43:03.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005153d52f6854483f-0065304357-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.668 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.669 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.670 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.670 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.671 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.671 DEBUG:boto:Method: GET 2023-10-18T20:43:03.671 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:43:03.672 DEBUG:boto:Data: 2023-10-18T20:43:03.672 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.673 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.673 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.674 DEBUG:boto:Params: {} 2023-10-18T20:43:03.674 DEBUG:boto:Token: None 2023-10-18T20:43:03.675 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.675 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ca40d5eba4e9eb-0065304357-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.680 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:43:03.682 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.682 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.683 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.683 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.683 DEBUG:boto:Method: GET 2023-10-18T20:43:03.684 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.684 DEBUG:boto:Data: 2023-10-18T20:43:03.685 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.685 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.685 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.686 DEBUG:boto:Params: {} 2023-10-18T20:43:03.686 DEBUG:boto:Token: None 2023-10-18T20:43:03.687 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.687 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fqtIIQVmNlppx87hbRlGpBxjLsU= 2023-10-18T20:43:03.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fqtIIQVmNlppx87hbRlGpBxjLsU=', 'Content-Length': '0'} 2023-10-18T20:43:03.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000722f327ba87b8790-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.691 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.691 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.692 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.692 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.693 DEBUG:boto:Method: GET 2023-10-18T20:43:03.693 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.694 DEBUG:boto:Data: 2023-10-18T20:43:03.694 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.694 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.695 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.695 DEBUG:boto:Params: {} 2023-10-18T20:43:03.696 DEBUG:boto:Token: None 2023-10-18T20:43:03.696 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.697 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fqtIIQVmNlppx87hbRlGpBxjLsU= 2023-10-18T20:43:03.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fqtIIQVmNlppx87hbRlGpBxjLsU=', 'Content-Length': '0'} 2023-10-18T20:43:03.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce5d60196f35248-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.742 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.743 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.743 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.744 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.744 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.745 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.745 DEBUG:boto:Data: 2023-10-18T20:43:03.745 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.746 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.746 DEBUG:boto:Params: {} 2023-10-18T20:43:03.747 DEBUG:boto:Token: None 2023-10-18T20:43:03.747 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RUx+fnunhRSnQZvIVUx7AsA9EOs= 2023-10-18T20:43:03.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RUx+fnunhRSnQZvIVUx7AsA9EOs=', 'Content-Length': '0'} 2023-10-18T20:43:03.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032b3293255df7766-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.751 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.751 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.752 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.752 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.752 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.753 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.753 DEBUG:boto:Data: 2023-10-18T20:43:03.753 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.754 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.755 DEBUG:boto:Params: {} 2023-10-18T20:43:03.755 DEBUG:boto:Token: None 2023-10-18T20:43:03.755 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RUx+fnunhRSnQZvIVUx7AsA9EOs= 2023-10-18T20:43:03.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RUx+fnunhRSnQZvIVUx7AsA9EOs=', 'Content-Length': '0'} 2023-10-18T20:43:03.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000192764b9edeea737-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:43:03.759 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.760 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.760 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.760 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.761 DEBUG:boto:Method: GET 2023-10-18T20:43:03.761 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.762 DEBUG:boto:Data: 2023-10-18T20:43:03.762 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.762 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.763 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.763 DEBUG:boto:Params: {} 2023-10-18T20:43:03.764 DEBUG:boto:Token: None 2023-10-18T20:43:03.764 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.765 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fqtIIQVmNlppx87hbRlGpBxjLsU= 2023-10-18T20:43:03.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fqtIIQVmNlppx87hbRlGpBxjLsU=', 'Content-Length': '0'} 2023-10-18T20:43:03.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbd05000032a818c-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.767 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.768 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.768 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.769 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.769 DEBUG:boto:Method: GET 2023-10-18T20:43:03.769 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.770 DEBUG:boto:Data: 2023-10-18T20:43:03.770 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.771 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.771 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.772 DEBUG:boto:Params: {} 2023-10-18T20:43:03.772 DEBUG:boto:Token: None 2023-10-18T20:43:03.773 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:43:03.773 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fqtIIQVmNlppx87hbRlGpBxjLsU= 2023-10-18T20:43:03.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fqtIIQVmNlppx87hbRlGpBxjLsU=', 'Content-Length': '0'} 2023-10-18T20:43:03.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b21870ddfce87b5-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.776 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.776 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.777 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.777 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.778 DEBUG:boto:Method: GET 2023-10-18T20:43:03.778 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.779 DEBUG:boto:Data: 2023-10-18T20:43:03.779 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.780 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.780 DEBUG:boto:Params: {} 2023-10-18T20:43:03.781 DEBUG:boto:Token: None 2023-10-18T20:43:03.781 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f352c76db1fe79df-0065304357-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.826 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.827 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.828 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.828 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.828 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.829 DEBUG:boto:Method: GET 2023-10-18T20:43:03.829 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.829 DEBUG:boto:Data: 2023-10-18T20:43:03.830 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.831 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.831 DEBUG:boto:Params: {} 2023-10-18T20:43:03.831 DEBUG:boto:Token: None 2023-10-18T20:43:03.832 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208f2bb4acc0a45e-0065304357-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.837 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.838 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.838 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.839 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.839 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:43:03.840 DEBUG:boto:Data: 2023-10-18T20:43:03.840 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.841 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.841 DEBUG:boto:Params: {} 2023-10-18T20:43:03.841 DEBUG:boto:Token: None 2023-10-18T20:43:03.842 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116 2023-10-18T20:43:03.842 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jPxZ3dhaR/mIfMs1LxLNg1jcOQw= 2023-10-18T20:43:03.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jPxZ3dhaR/mIfMs1LxLNg1jcOQw=', 'Content-Length': '0'} 2023-10-18T20:43:03.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf475654ebcfd8ac-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.845 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.846 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.846 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.847 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:43:03.847 DEBUG:boto:Data: 2023-10-18T20:43:03.847 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.848 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.848 DEBUG:boto:Params: {} 2023-10-18T20:43:03.849 DEBUG:boto:Token: None 2023-10-18T20:43:03.849 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116 2023-10-18T20:43:03.850 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jPxZ3dhaR/mIfMs1LxLNg1jcOQw= 2023-10-18T20:43:03.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jPxZ3dhaR/mIfMs1LxLNg1jcOQw=', 'Content-Length': '0'} 2023-10-18T20:43:03.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eae6f33b153d4323-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:43:03.853 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.853 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.854 DEBUG:boto:Method: GET 2023-10-18T20:43:03.854 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:43:03.854 DEBUG:boto:Data: 2023-10-18T20:43:03.855 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.856 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.856 DEBUG:boto:Params: {} 2023-10-18T20:43:03.857 DEBUG:boto:Token: None 2023-10-18T20:43:03.857 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116 2023-10-18T20:43:03.857 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tjrbPWqRj5lM0Uqoexe58LJTilw= 2023-10-18T20:43:03.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tjrbPWqRj5lM0Uqoexe58LJTilw=', 'Content-Length': '0'} 2023-10-18T20:43:03.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc9b31e56f67ab7d-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.860 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:43:03.861 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:43:03.861 DEBUG:boto:Method: GET 2023-10-18T20:43:03.862 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:43:03.862 DEBUG:boto:Data: 2023-10-18T20:43:03.863 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.863 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.864 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.864 DEBUG:boto:Params: {} 2023-10-18T20:43:03.865 DEBUG:boto:Token: None 2023-10-18T20:43:03.865 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj116 2023-10-18T20:43:03.866 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tjrbPWqRj5lM0Uqoexe58LJTilw= 2023-10-18T20:43:03.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tjrbPWqRj5lM0Uqoexe58LJTilw=', 'Content-Length': '0'} 2023-10-18T20:43:03.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea97aad06c43d489-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.914 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.915 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.915 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.916 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.916 DEBUG:boto:Method: GET 2023-10-18T20:43:03.916 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.917 DEBUG:boto:Data: 2023-10-18T20:43:03.917 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.918 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.919 DEBUG:boto:Params: {} 2023-10-18T20:43:03.919 DEBUG:boto:Token: None 2023-10-18T20:43:03.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003233f0689709ae7f-0065304357-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.923 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.924 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.925 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.925 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.925 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.926 DEBUG:boto:Method: GET 2023-10-18T20:43:03.926 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:43:03.927 DEBUG:boto:Data: 2023-10-18T20:43:03.927 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.927 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.928 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.928 DEBUG:boto:Params: {} 2023-10-18T20:43:03.929 DEBUG:boto:Token: None 2023-10-18T20:43:03.929 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.930 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078eb022e360e3137-0065304357-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.934 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:03.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-18T20:43:03.935 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:03.936 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:03.936 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.937 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.937 DEBUG:boto:Method: GET 2023-10-18T20:43:03.937 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.938 DEBUG:boto:Data: 2023-10-18T20:43:03.938 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.939 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.939 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.940 DEBUG:boto:Params: {} 2023-10-18T20:43:03.940 DEBUG:boto:Token: None 2023-10-18T20:43:03.941 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.941 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GJNFvhG5iRuOEqhBBkz0/DzOU8E= 2023-10-18T20:43:03.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GJNFvhG5iRuOEqhBBkz0/DzOU8E=', 'Content-Length': '0'} 2023-10-18T20:43:03.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1b2846ed7622a9c-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.945 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:03.945 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:03.945 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.946 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.946 DEBUG:boto:Method: GET 2023-10-18T20:43:03.947 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.947 DEBUG:boto:Data: 2023-10-18T20:43:03.948 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.948 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.948 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.949 DEBUG:boto:Params: {} 2023-10-18T20:43:03.949 DEBUG:boto:Token: None 2023-10-18T20:43:03.950 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.950 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GJNFvhG5iRuOEqhBBkz0/DzOU8E= 2023-10-18T20:43:03.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GJNFvhG5iRuOEqhBBkz0/DzOU8E=', 'Content-Length': '0'} 2023-10-18T20:43:03.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7da862377d1ba89-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.954 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:03.954 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:03.954 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.955 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.955 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.955 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.956 DEBUG:boto:Data: 2023-10-18T20:43:03.956 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.957 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.957 DEBUG:boto:Params: {} 2023-10-18T20:43:03.958 DEBUG:boto:Token: None 2023-10-18T20:43:03.958 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+yAHIz+Z22JsmycDsRoO5fHWSEU= 2023-10-18T20:43:03.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+yAHIz+Z22JsmycDsRoO5fHWSEU=', 'Content-Length': '0'} 2023-10-18T20:43:03.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e256f0efeacdc9-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.961 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:03.962 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:03.962 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.963 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.963 DEBUG:boto:Method: HEAD 2023-10-18T20:43:03.963 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.964 DEBUG:boto:Data: 2023-10-18T20:43:03.964 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.964 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.965 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.965 DEBUG:boto:Params: {} 2023-10-18T20:43:03.966 DEBUG:boto:Token: None 2023-10-18T20:43:03.966 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.967 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+yAHIz+Z22JsmycDsRoO5fHWSEU= 2023-10-18T20:43:03.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+yAHIz+Z22JsmycDsRoO5fHWSEU=', 'Content-Length': '0'} 2023-10-18T20:43:03.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012a63d69f57b536b-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-18T20:43:03.970 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:03.970 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:03.971 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.971 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.972 DEBUG:boto:Method: GET 2023-10-18T20:43:03.972 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.972 DEBUG:boto:Data: 2023-10-18T20:43:03.973 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.973 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.974 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.974 DEBUG:boto:Params: {} 2023-10-18T20:43:03.975 DEBUG:boto:Token: None 2023-10-18T20:43:03.975 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.976 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GJNFvhG5iRuOEqhBBkz0/DzOU8E= 2023-10-18T20:43:03.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GJNFvhG5iRuOEqhBBkz0/DzOU8E=', 'Content-Length': '0'} 2023-10-18T20:43:03.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e002f22a42d9e13-0065304357-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.979 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:03.979 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:03.980 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.980 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.981 DEBUG:boto:Method: GET 2023-10-18T20:43:03.981 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.981 DEBUG:boto:Data: 2023-10-18T20:43:03.982 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.982 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:03.983 DEBUG:boto:Port: 8000 2023-10-18T20:43:03.983 DEBUG:boto:Params: {} 2023-10-18T20:43:03.984 DEBUG:boto:Token: None 2023-10-18T20:43:03.984 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:43:03.985 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GJNFvhG5iRuOEqhBBkz0/DzOU8E= 2023-10-18T20:43:03.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GJNFvhG5iRuOEqhBBkz0/DzOU8E=', 'Content-Length': '0'} 2023-10-18T20:43:03.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c420284025e7bc29-0065304357-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:03.988 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:03.988 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:03.989 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:43:03.989 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:43:03.989 DEBUG:boto:Method: GET 2023-10-18T20:43:03.990 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:43:03.990 DEBUG:boto:Data: 2023-10-18T20:43:03.991 DEBUG:boto:Headers: {} 2023-10-18T20:43:03.991 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:03.991 DEBUG:boto:Port: 8001 2023-10-18T20:43:03.992 DEBUG:boto:Params: {} 2023-10-18T20:43:03.992 DEBUG:boto:Token: None 2023-10-18T20:43:03.993 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:03 GMT /bofrnf-59/ 2023-10-18T20:43:03.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc= 2023-10-18T20:43:03.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:03 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GqR+wrXA7Lfrep8Z2kUgHIYo9vc=', 'Content-Length': '0'} 2023-10-18T20:43:03.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cce5cc20d1ec755-0065304357-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.038 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.039 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.040 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.040 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:43:04.040 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:43:04.041 DEBUG:boto:Method: GET 2023-10-18T20:43:04.041 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:43:04.042 DEBUG:boto:Data: 2023-10-18T20:43:04.042 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.043 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.043 DEBUG:boto:Params: {} 2023-10-18T20:43:04.044 DEBUG:boto:Token: None 2023-10-18T20:43:04.044 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.045 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099e1e3e492ec36ef-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.050 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.050 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:04.051 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:04.051 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.052 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:43:04.052 DEBUG:boto:Data: 2023-10-18T20:43:04.053 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.053 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.053 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.054 DEBUG:boto:Params: {} 2023-10-18T20:43:04.054 DEBUG:boto:Token: None 2023-10-18T20:43:04.055 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj124 2023-10-18T20:43:04.055 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1lXWjnM9ZXljwmxE8Zhy+lherZ4= 2023-10-18T20:43:04.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1lXWjnM9ZXljwmxE8Zhy+lherZ4=', 'Content-Length': '0'} 2023-10-18T20:43:04.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcade3e605e667cd-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.058 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:04.059 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:04.059 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.059 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:43:04.060 DEBUG:boto:Data: 2023-10-18T20:43:04.060 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.060 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.061 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.061 DEBUG:boto:Params: {} 2023-10-18T20:43:04.062 DEBUG:boto:Token: None 2023-10-18T20:43:04.062 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj124 2023-10-18T20:43:04.062 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1lXWjnM9ZXljwmxE8Zhy+lherZ4= 2023-10-18T20:43:04.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1lXWjnM9ZXljwmxE8Zhy+lherZ4=', 'Content-Length': '0'} 2023-10-18T20:43:04.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc57ac93eedbba1-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-18T20:43:04.066 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:04.067 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:04.067 DEBUG:boto:Method: GET 2023-10-18T20:43:04.068 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:43:04.068 DEBUG:boto:Data: 2023-10-18T20:43:04.068 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.069 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.069 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.070 DEBUG:boto:Params: {} 2023-10-18T20:43:04.070 DEBUG:boto:Token: None 2023-10-18T20:43:04.071 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj124 2023-10-18T20:43:04.071 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fWAYRHrb7kC6Fb/GxZUSj5FNZXE= 2023-10-18T20:43:04.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fWAYRHrb7kC6Fb/GxZUSj5FNZXE=', 'Content-Length': '0'} 2023-10-18T20:43:04.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e144fdebebe4a1c6-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.074 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:43:04.074 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:43:04.075 DEBUG:boto:Method: GET 2023-10-18T20:43:04.075 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:43:04.076 DEBUG:boto:Data: 2023-10-18T20:43:04.076 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.076 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.077 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.077 DEBUG:boto:Params: {} 2023-10-18T20:43:04.078 DEBUG:boto:Token: None 2023-10-18T20:43:04.078 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj124 2023-10-18T20:43:04.079 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fWAYRHrb7kC6Fb/GxZUSj5FNZXE= 2023-10-18T20:43:04.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fWAYRHrb7kC6Fb/GxZUSj5FNZXE=', 'Content-Length': '0'} 2023-10-18T20:43:04.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac80c7d8c3804c36-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.082 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.082 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.083 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:43:04.083 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:43:04.084 DEBUG:boto:Method: GET 2023-10-18T20:43:04.084 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:43:04.084 DEBUG:boto:Data: 2023-10-18T20:43:04.085 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.085 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.086 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.086 DEBUG:boto:Params: {} 2023-10-18T20:43:04.087 DEBUG:boto:Token: None 2023-10-18T20:43:04.087 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.087 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f76750d9e240bcec-0065304358-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.091 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.091 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.092 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.092 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:43:04.093 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:43:04.093 DEBUG:boto:Method: GET 2023-10-18T20:43:04.093 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:43:04.094 DEBUG:boto:Data: 2023-10-18T20:43:04.094 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.095 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.095 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.095 DEBUG:boto:Params: {} 2023-10-18T20:43:04.096 DEBUG:boto:Token: None 2023-10-18T20:43:04.096 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.097 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000649667240a42ebd3-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.101 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:43:04.102 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.103 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.104 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.104 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.104 DEBUG:boto:Method: GET 2023-10-18T20:43:04.105 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.105 DEBUG:boto:Data: 2023-10-18T20:43:04.106 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.106 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.107 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.107 DEBUG:boto:Params: {} 2023-10-18T20:43:04.107 DEBUG:boto:Token: None 2023-10-18T20:43:04.108 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PJx0KTbIrBYs+jpOVNACj3Nen/s= 2023-10-18T20:43:04.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PJx0KTbIrBYs+jpOVNACj3Nen/s=', 'Content-Length': '0'} 2023-10-18T20:43:04.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004480bbb4ad44ec89-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.112 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.113 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.113 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.113 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.114 DEBUG:boto:Method: GET 2023-10-18T20:43:04.114 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.115 DEBUG:boto:Data: 2023-10-18T20:43:04.115 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.116 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.116 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.117 DEBUG:boto:Params: {} 2023-10-18T20:43:04.117 DEBUG:boto:Token: None 2023-10-18T20:43:04.118 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.118 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PJx0KTbIrBYs+jpOVNACj3Nen/s= 2023-10-18T20:43:04.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PJx0KTbIrBYs+jpOVNACj3Nen/s=', 'Content-Length': '0'} 2023-10-18T20:43:04.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ff45ac4672c932-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.166 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.167 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.167 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.168 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.168 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.168 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.169 DEBUG:boto:Data: 2023-10-18T20:43:04.169 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.170 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.170 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.170 DEBUG:boto:Params: {} 2023-10-18T20:43:04.171 DEBUG:boto:Token: None 2023-10-18T20:43:04.171 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.172 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:85saNlrwZyQ2AZ/0Tf8IONEpsT4= 2023-10-18T20:43:04.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:85saNlrwZyQ2AZ/0Tf8IONEpsT4=', 'Content-Length': '0'} 2023-10-18T20:43:04.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004275d73c833577c1-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.174 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.175 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.177 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.177 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.178 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.178 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.179 DEBUG:boto:Data: 2023-10-18T20:43:04.179 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.179 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.180 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.180 DEBUG:boto:Params: {} 2023-10-18T20:43:04.181 DEBUG:boto:Token: None 2023-10-18T20:43:04.181 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.181 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:85saNlrwZyQ2AZ/0Tf8IONEpsT4= 2023-10-18T20:43:04.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:85saNlrwZyQ2AZ/0Tf8IONEpsT4=', 'Content-Length': '0'} 2023-10-18T20:43:04.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c9261dd8a64f112-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:43:04.185 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.186 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.186 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.186 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.187 DEBUG:boto:Method: GET 2023-10-18T20:43:04.187 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.188 DEBUG:boto:Data: 2023-10-18T20:43:04.188 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.188 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.190 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.190 DEBUG:boto:Params: {} 2023-10-18T20:43:04.191 DEBUG:boto:Token: None 2023-10-18T20:43:04.191 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PJx0KTbIrBYs+jpOVNACj3Nen/s= 2023-10-18T20:43:04.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PJx0KTbIrBYs+jpOVNACj3Nen/s=', 'Content-Length': '0'} 2023-10-18T20:43:04.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd8c6bdba5cc97ce-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.195 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.195 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.196 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.196 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.197 DEBUG:boto:Method: GET 2023-10-18T20:43:04.197 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.197 DEBUG:boto:Data: 2023-10-18T20:43:04.198 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.198 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.199 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.199 DEBUG:boto:Params: {} 2023-10-18T20:43:04.200 DEBUG:boto:Token: None 2023-10-18T20:43:04.200 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:43:04.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PJx0KTbIrBYs+jpOVNACj3Nen/s= 2023-10-18T20:43:04.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PJx0KTbIrBYs+jpOVNACj3Nen/s=', 'Content-Length': '0'} 2023-10-18T20:43:04.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a13009742dc783-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.204 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.204 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.204 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.205 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.205 DEBUG:boto:Method: GET 2023-10-18T20:43:04.206 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.206 DEBUG:boto:Data: 2023-10-18T20:43:04.206 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.207 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.207 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.208 DEBUG:boto:Params: {} 2023-10-18T20:43:04.208 DEBUG:boto:Token: None 2023-10-18T20:43:04.208 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.209 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3e084d7e2e4baba-0065304358-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.254 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.255 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.255 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.256 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.256 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.257 DEBUG:boto:Method: GET 2023-10-18T20:43:04.257 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.257 DEBUG:boto:Data: 2023-10-18T20:43:04.258 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.258 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.259 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.259 DEBUG:boto:Params: {} 2023-10-18T20:43:04.259 DEBUG:boto:Token: None 2023-10-18T20:43:04.260 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000268542e953c33d28-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.265 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.266 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.266 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.267 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.267 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:43:04.268 DEBUG:boto:Data: 2023-10-18T20:43:04.268 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.268 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.269 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.269 DEBUG:boto:Params: {} 2023-10-18T20:43:04.270 DEBUG:boto:Token: None 2023-10-18T20:43:04.270 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145 2023-10-18T20:43:04.271 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mqoPLSGyGTxtPAJOYmNQ9441WKc= 2023-10-18T20:43:04.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mqoPLSGyGTxtPAJOYmNQ9441WKc=', 'Content-Length': '0'} 2023-10-18T20:43:04.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a673e658ec62d8a7-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.274 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.274 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.275 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.275 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:43:04.275 DEBUG:boto:Data: 2023-10-18T20:43:04.276 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.276 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.276 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.277 DEBUG:boto:Params: {} 2023-10-18T20:43:04.277 DEBUG:boto:Token: None 2023-10-18T20:43:04.277 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145 2023-10-18T20:43:04.278 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mqoPLSGyGTxtPAJOYmNQ9441WKc= 2023-10-18T20:43:04.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mqoPLSGyGTxtPAJOYmNQ9441WKc=', 'Content-Length': '0'} 2023-10-18T20:43:04.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a3b592613c6e599-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:43:04.282 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.282 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.282 DEBUG:boto:Method: GET 2023-10-18T20:43:04.283 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:43:04.283 DEBUG:boto:Data: 2023-10-18T20:43:04.284 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.284 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.284 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.285 DEBUG:boto:Params: {} 2023-10-18T20:43:04.285 DEBUG:boto:Token: None 2023-10-18T20:43:04.286 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145 2023-10-18T20:43:04.286 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I2ry80mom39a+xTaTPvk0LePP6g= 2023-10-18T20:43:04.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I2ry80mom39a+xTaTPvk0LePP6g=', 'Content-Length': '0'} 2023-10-18T20:43:04.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b04d5c3f8c93eae2-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.289 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:43:04.290 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:43:04.290 DEBUG:boto:Method: GET 2023-10-18T20:43:04.290 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:43:04.291 DEBUG:boto:Data: 2023-10-18T20:43:04.291 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.292 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.292 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.292 DEBUG:boto:Params: {} 2023-10-18T20:43:04.293 DEBUG:boto:Token: None 2023-10-18T20:43:04.293 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj145 2023-10-18T20:43:04.294 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:I2ry80mom39a+xTaTPvk0LePP6g= 2023-10-18T20:43:04.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:I2ry80mom39a+xTaTPvk0LePP6g=', 'Content-Length': '0'} 2023-10-18T20:43:04.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050ff41538837f650-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.297 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.297 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.298 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.298 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.299 DEBUG:boto:Method: GET 2023-10-18T20:43:04.299 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.299 DEBUG:boto:Data: 2023-10-18T20:43:04.300 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.300 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.300 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.301 DEBUG:boto:Params: {} 2023-10-18T20:43:04.301 DEBUG:boto:Token: None 2023-10-18T20:43:04.302 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.302 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f028d97253183b1-0065304358-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.306 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.307 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.307 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.308 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.308 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.308 DEBUG:boto:Method: GET 2023-10-18T20:43:04.309 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:43:04.309 DEBUG:boto:Data: 2023-10-18T20:43:04.310 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.310 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.311 DEBUG:boto:Params: {} 2023-10-18T20:43:04.311 DEBUG:boto:Token: None 2023-10-18T20:43:04.312 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f862773baf7dfc42-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.317 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-18T20:43:04.318 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.318 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.319 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.319 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.320 DEBUG:boto:Method: GET 2023-10-18T20:43:04.320 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.320 DEBUG:boto:Data: 2023-10-18T20:43:04.321 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.321 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.322 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.322 DEBUG:boto:Params: {} 2023-10-18T20:43:04.322 DEBUG:boto:Token: None 2023-10-18T20:43:04.323 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.323 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5x8iy2WgpJV1hIUARKI+kd17rTI= 2023-10-18T20:43:04.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5x8iy2WgpJV1hIUARKI+kd17rTI=', 'Content-Length': '0'} 2023-10-18T20:43:04.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf2f1d26ff96029-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.327 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.327 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.327 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.328 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.328 DEBUG:boto:Method: GET 2023-10-18T20:43:04.329 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.329 DEBUG:boto:Data: 2023-10-18T20:43:04.329 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.330 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.330 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.331 DEBUG:boto:Params: {} 2023-10-18T20:43:04.331 DEBUG:boto:Token: None 2023-10-18T20:43:04.332 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.332 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5x8iy2WgpJV1hIUARKI+kd17rTI= 2023-10-18T20:43:04.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5x8iy2WgpJV1hIUARKI+kd17rTI=', 'Content-Length': '0'} 2023-10-18T20:43:04.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007520935686d19ae3-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.378 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.379 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.379 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.380 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.380 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.381 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.381 DEBUG:boto:Data: 2023-10-18T20:43:04.381 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.382 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.382 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.382 DEBUG:boto:Params: {} 2023-10-18T20:43:04.383 DEBUG:boto:Token: None 2023-10-18T20:43:04.383 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.384 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q3FwD16PpdjDS+wG2gmmo5vqQjE= 2023-10-18T20:43:04.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q3FwD16PpdjDS+wG2gmmo5vqQjE=', 'Content-Length': '0'} 2023-10-18T20:43:04.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f649d2b0782b0748-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.387 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.387 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.388 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.388 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.389 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.389 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.389 DEBUG:boto:Data: 2023-10-18T20:43:04.390 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.391 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.391 DEBUG:boto:Params: {} 2023-10-18T20:43:04.392 DEBUG:boto:Token: None 2023-10-18T20:43:04.392 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.393 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q3FwD16PpdjDS+wG2gmmo5vqQjE= 2023-10-18T20:43:04.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q3FwD16PpdjDS+wG2gmmo5vqQjE=', 'Content-Length': '0'} 2023-10-18T20:43:04.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4873e02c3d6f0c4-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-18T20:43:04.396 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.397 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.397 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.397 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.398 DEBUG:boto:Method: GET 2023-10-18T20:43:04.398 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.399 DEBUG:boto:Data: 2023-10-18T20:43:04.399 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.400 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.400 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.400 DEBUG:boto:Params: {} 2023-10-18T20:43:04.401 DEBUG:boto:Token: None 2023-10-18T20:43:04.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5x8iy2WgpJV1hIUARKI+kd17rTI= 2023-10-18T20:43:04.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5x8iy2WgpJV1hIUARKI+kd17rTI=', 'Content-Length': '0'} 2023-10-18T20:43:04.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ae72b0691d82737-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.405 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.406 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.406 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.406 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.407 DEBUG:boto:Method: GET 2023-10-18T20:43:04.407 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.408 DEBUG:boto:Data: 2023-10-18T20:43:04.408 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.409 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.409 DEBUG:boto:Params: {} 2023-10-18T20:43:04.410 DEBUG:boto:Token: None 2023-10-18T20:43:04.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:43:04.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5x8iy2WgpJV1hIUARKI+kd17rTI= 2023-10-18T20:43:04.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5x8iy2WgpJV1hIUARKI+kd17rTI=', 'Content-Length': '0'} 2023-10-18T20:43:04.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087f408872ce9e304-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.414 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.415 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.415 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.416 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.416 DEBUG:boto:Method: GET 2023-10-18T20:43:04.417 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.417 DEBUG:boto:Data: 2023-10-18T20:43:04.417 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.418 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.419 DEBUG:boto:Params: {} 2023-10-18T20:43:04.419 DEBUG:boto:Token: None 2023-10-18T20:43:04.420 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a406a41ed226e6-0065304358-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.424 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.424 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.425 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.425 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.426 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.426 DEBUG:boto:Method: GET 2023-10-18T20:43:04.426 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.427 DEBUG:boto:Data: 2023-10-18T20:43:04.427 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.428 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.428 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.429 DEBUG:boto:Params: {} 2023-10-18T20:43:04.429 DEBUG:boto:Token: None 2023-10-18T20:43:04.430 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.430 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042f1901a7692826c-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.435 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.436 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.437 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.437 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.438 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:43:04.438 DEBUG:boto:Data: 2023-10-18T20:43:04.438 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.439 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.439 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.439 DEBUG:boto:Params: {} 2023-10-18T20:43:04.440 DEBUG:boto:Token: None 2023-10-18T20:43:04.440 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155 2023-10-18T20:43:04.441 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dfA4pF+8dxHvKO0d9NQp7/wA7zY= 2023-10-18T20:43:04.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dfA4pF+8dxHvKO0d9NQp7/wA7zY=', 'Content-Length': '0'} 2023-10-18T20:43:04.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef7c367fc9f8522-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.444 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.444 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.445 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.445 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:43:04.446 DEBUG:boto:Data: 2023-10-18T20:43:04.446 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.447 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.447 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.447 DEBUG:boto:Params: {} 2023-10-18T20:43:04.448 DEBUG:boto:Token: None 2023-10-18T20:43:04.448 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155 2023-10-18T20:43:04.449 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dfA4pF+8dxHvKO0d9NQp7/wA7zY= 2023-10-18T20:43:04.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dfA4pF+8dxHvKO0d9NQp7/wA7zY=', 'Content-Length': '0'} 2023-10-18T20:43:04.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063a76405f693aa32-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-18T20:43:04.452 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.452 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.453 DEBUG:boto:Method: GET 2023-10-18T20:43:04.453 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:43:04.454 DEBUG:boto:Data: 2023-10-18T20:43:04.454 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.455 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.455 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.455 DEBUG:boto:Params: {} 2023-10-18T20:43:04.456 DEBUG:boto:Token: None 2023-10-18T20:43:04.456 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155 2023-10-18T20:43:04.457 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vsXZoAjA+2ypHRE/LRj+F/5GYyo= 2023-10-18T20:43:04.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vsXZoAjA+2ypHRE/LRj+F/5GYyo=', 'Content-Length': '0'} 2023-10-18T20:43:04.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c56c90031f1684dc-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.506 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:43:04.507 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:43:04.507 DEBUG:boto:Method: GET 2023-10-18T20:43:04.507 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:43:04.508 DEBUG:boto:Data: 2023-10-18T20:43:04.508 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.509 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.509 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.509 DEBUG:boto:Params: {} 2023-10-18T20:43:04.510 DEBUG:boto:Token: None 2023-10-18T20:43:04.510 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj155 2023-10-18T20:43:04.511 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vsXZoAjA+2ypHRE/LRj+F/5GYyo= 2023-10-18T20:43:04.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vsXZoAjA+2ypHRE/LRj+F/5GYyo=', 'Content-Length': '0'} 2023-10-18T20:43:04.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0419c4605f2c6cf-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.514 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.515 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.515 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.515 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.516 DEBUG:boto:Method: GET 2023-10-18T20:43:04.516 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.516 DEBUG:boto:Data: 2023-10-18T20:43:04.517 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.517 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.518 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.518 DEBUG:boto:Params: {} 2023-10-18T20:43:04.518 DEBUG:boto:Token: None 2023-10-18T20:43:04.519 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.519 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7f09bccab4a6ea8-0065304358-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.523 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.524 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.524 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.524 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.525 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.525 DEBUG:boto:Method: GET 2023-10-18T20:43:04.526 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:43:04.526 DEBUG:boto:Data: 2023-10-18T20:43:04.526 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.527 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.527 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.528 DEBUG:boto:Params: {} 2023-10-18T20:43:04.528 DEBUG:boto:Token: None 2023-10-18T20:43:04.528 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.529 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c36c0aa2c6c55f8-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.534 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:43:04.535 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.535 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.536 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.536 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.536 DEBUG:boto:Method: GET 2023-10-18T20:43:04.537 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.537 DEBUG:boto:Data: 2023-10-18T20:43:04.538 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.538 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.539 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.539 DEBUG:boto:Params: {} 2023-10-18T20:43:04.539 DEBUG:boto:Token: None 2023-10-18T20:43:04.540 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.540 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nJRDuuJ4N7EKhAuxAfU+44BAU6Q= 2023-10-18T20:43:04.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nJRDuuJ4N7EKhAuxAfU+44BAU6Q=', 'Content-Length': '0'} 2023-10-18T20:43:04.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca484e5c79035c37-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.543 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.544 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.544 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.545 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.545 DEBUG:boto:Method: GET 2023-10-18T20:43:04.545 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.546 DEBUG:boto:Data: 2023-10-18T20:43:04.546 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.547 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.547 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.547 DEBUG:boto:Params: {} 2023-10-18T20:43:04.548 DEBUG:boto:Token: None 2023-10-18T20:43:04.548 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.549 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nJRDuuJ4N7EKhAuxAfU+44BAU6Q= 2023-10-18T20:43:04.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nJRDuuJ4N7EKhAuxAfU+44BAU6Q=', 'Content-Length': '0'} 2023-10-18T20:43:04.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833290644c833b05-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.552 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.552 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.552 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.553 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.553 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.553 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.554 DEBUG:boto:Data: 2023-10-18T20:43:04.554 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.555 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.555 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.555 DEBUG:boto:Params: {} 2023-10-18T20:43:04.556 DEBUG:boto:Token: None 2023-10-18T20:43:04.556 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.557 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z5C0uVc2aXg4hkjFEKZ3NSDXkwU= 2023-10-18T20:43:04.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z5C0uVc2aXg4hkjFEKZ3NSDXkwU=', 'Content-Length': '0'} 2023-10-18T20:43:04.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc3cf61b9ba7d0d-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.559 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.560 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.560 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.560 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.561 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.561 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.561 DEBUG:boto:Data: 2023-10-18T20:43:04.562 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.563 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.563 DEBUG:boto:Params: {} 2023-10-18T20:43:04.563 DEBUG:boto:Token: None 2023-10-18T20:43:04.564 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z5C0uVc2aXg4hkjFEKZ3NSDXkwU= 2023-10-18T20:43:04.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z5C0uVc2aXg4hkjFEKZ3NSDXkwU=', 'Content-Length': '0'} 2023-10-18T20:43:04.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bc92550f63fa7c0-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:43:04.568 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.568 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.569 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.569 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.569 DEBUG:boto:Method: GET 2023-10-18T20:43:04.570 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.570 DEBUG:boto:Data: 2023-10-18T20:43:04.570 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.571 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.571 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.572 DEBUG:boto:Params: {} 2023-10-18T20:43:04.572 DEBUG:boto:Token: None 2023-10-18T20:43:04.572 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.573 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nJRDuuJ4N7EKhAuxAfU+44BAU6Q= 2023-10-18T20:43:04.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nJRDuuJ4N7EKhAuxAfU+44BAU6Q=', 'Content-Length': '0'} 2023-10-18T20:43:04.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201819b99c220056-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.618 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.619 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.619 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.620 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.620 DEBUG:boto:Method: GET 2023-10-18T20:43:04.620 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.621 DEBUG:boto:Data: 2023-10-18T20:43:04.621 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.622 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.622 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.622 DEBUG:boto:Params: {} 2023-10-18T20:43:04.623 DEBUG:boto:Token: None 2023-10-18T20:43:04.624 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:43:04.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nJRDuuJ4N7EKhAuxAfU+44BAU6Q= 2023-10-18T20:43:04.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nJRDuuJ4N7EKhAuxAfU+44BAU6Q=', 'Content-Length': '0'} 2023-10-18T20:43:04.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a5cfc41d3f9f05-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.627 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.628 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.628 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.629 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.629 DEBUG:boto:Method: GET 2023-10-18T20:43:04.629 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.630 DEBUG:boto:Data: 2023-10-18T20:43:04.630 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.631 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.631 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.631 DEBUG:boto:Params: {} 2023-10-18T20:43:04.632 DEBUG:boto:Token: None 2023-10-18T20:43:04.632 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.633 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db6fa3430ed3ebf0-0065304358-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.636 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.637 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.638 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.638 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.639 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.639 DEBUG:boto:Method: GET 2023-10-18T20:43:04.639 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.640 DEBUG:boto:Data: 2023-10-18T20:43:04.640 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.641 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.641 DEBUG:boto:Params: {} 2023-10-18T20:43:04.642 DEBUG:boto:Token: None 2023-10-18T20:43:04.642 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000addd11be61fe5c18-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.648 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.649 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.649 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.649 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.650 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:43:04.650 DEBUG:boto:Data: 2023-10-18T20:43:04.651 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.651 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.652 DEBUG:boto:Params: {} 2023-10-18T20:43:04.652 DEBUG:boto:Token: None 2023-10-18T20:43:04.652 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17 2023-10-18T20:43:04.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yu8RTRmBp6TpLlCrvKnanTQRyHM= 2023-10-18T20:43:04.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yu8RTRmBp6TpLlCrvKnanTQRyHM=', 'Content-Length': '0'} 2023-10-18T20:43:04.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090ecad4801184f02-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.656 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.656 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.657 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.657 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:43:04.657 DEBUG:boto:Data: 2023-10-18T20:43:04.658 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.658 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.659 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.659 DEBUG:boto:Params: {} 2023-10-18T20:43:04.659 DEBUG:boto:Token: None 2023-10-18T20:43:04.660 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17 2023-10-18T20:43:04.660 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yu8RTRmBp6TpLlCrvKnanTQRyHM= 2023-10-18T20:43:04.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yu8RTRmBp6TpLlCrvKnanTQRyHM=', 'Content-Length': '0'} 2023-10-18T20:43:04.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087968d5a3bdc40a2-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:43:04.664 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.664 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.664 DEBUG:boto:Method: GET 2023-10-18T20:43:04.665 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:43:04.665 DEBUG:boto:Data: 2023-10-18T20:43:04.666 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.666 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.667 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.667 DEBUG:boto:Params: {} 2023-10-18T20:43:04.667 DEBUG:boto:Token: None 2023-10-18T20:43:04.668 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17 2023-10-18T20:43:04.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zZEQ4710nWDu7HaTXRh6IRN1TWE= 2023-10-18T20:43:04.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zZEQ4710nWDu7HaTXRh6IRN1TWE=', 'Content-Length': '0'} 2023-10-18T20:43:04.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094ac67c032efe330-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.671 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:43:04.672 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:43:04.672 DEBUG:boto:Method: GET 2023-10-18T20:43:04.673 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:43:04.673 DEBUG:boto:Data: 2023-10-18T20:43:04.674 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.674 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.674 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.675 DEBUG:boto:Params: {} 2023-10-18T20:43:04.675 DEBUG:boto:Token: None 2023-10-18T20:43:04.676 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj17 2023-10-18T20:43:04.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zZEQ4710nWDu7HaTXRh6IRN1TWE= 2023-10-18T20:43:04.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zZEQ4710nWDu7HaTXRh6IRN1TWE=', 'Content-Length': '0'} 2023-10-18T20:43:04.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bcffac97773f8ec-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.722 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.723 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.723 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.724 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.724 DEBUG:boto:Method: GET 2023-10-18T20:43:04.724 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.725 DEBUG:boto:Data: 2023-10-18T20:43:04.725 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.726 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.726 DEBUG:boto:Params: {} 2023-10-18T20:43:04.727 DEBUG:boto:Token: None 2023-10-18T20:43:04.727 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cff01733f749338-0065304358-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.731 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.732 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.732 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.732 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.733 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.733 DEBUG:boto:Method: GET 2023-10-18T20:43:04.734 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:43:04.734 DEBUG:boto:Data: 2023-10-18T20:43:04.735 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.735 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.735 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.736 DEBUG:boto:Params: {} 2023-10-18T20:43:04.736 DEBUG:boto:Token: None 2023-10-18T20:43:04.737 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff1c8d63da9ee2b8-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.742 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:43:04.743 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.744 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.744 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.744 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.745 DEBUG:boto:Method: GET 2023-10-18T20:43:04.745 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.745 DEBUG:boto:Data: 2023-10-18T20:43:04.746 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.747 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.747 DEBUG:boto:Params: {} 2023-10-18T20:43:04.747 DEBUG:boto:Token: None 2023-10-18T20:43:04.748 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lUrId420K8uVDdWi/rIICleTB5k= 2023-10-18T20:43:04.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lUrId420K8uVDdWi/rIICleTB5k=', 'Content-Length': '0'} 2023-10-18T20:43:04.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b93e180a52596d2-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.794 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.795 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.795 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.796 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.796 DEBUG:boto:Method: GET 2023-10-18T20:43:04.797 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.797 DEBUG:boto:Data: 2023-10-18T20:43:04.797 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.798 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.798 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.799 DEBUG:boto:Params: {} 2023-10-18T20:43:04.799 DEBUG:boto:Token: None 2023-10-18T20:43:04.800 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.800 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lUrId420K8uVDdWi/rIICleTB5k= 2023-10-18T20:43:04.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lUrId420K8uVDdWi/rIICleTB5k=', 'Content-Length': '0'} 2023-10-18T20:43:04.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000931b5e6187378f1e-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.804 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.804 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.804 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.805 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.805 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.805 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.806 DEBUG:boto:Data: 2023-10-18T20:43:04.806 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.807 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.807 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.807 DEBUG:boto:Params: {} 2023-10-18T20:43:04.808 DEBUG:boto:Token: None 2023-10-18T20:43:04.808 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.809 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bwx8oZ1StmyOIbYcJP63Lg64+ww= 2023-10-18T20:43:04.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bwx8oZ1StmyOIbYcJP63Lg64+ww=', 'Content-Length': '0'} 2023-10-18T20:43:04.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2efb248e1af685-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.811 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.812 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.812 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.813 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.813 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.813 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.814 DEBUG:boto:Data: 2023-10-18T20:43:04.814 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.815 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.815 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.815 DEBUG:boto:Params: {} 2023-10-18T20:43:04.816 DEBUG:boto:Token: None 2023-10-18T20:43:04.816 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bwx8oZ1StmyOIbYcJP63Lg64+ww= 2023-10-18T20:43:04.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bwx8oZ1StmyOIbYcJP63Lg64+ww=', 'Content-Length': '0'} 2023-10-18T20:43:04.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b202ee5ef4df004-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:43:04.820 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.820 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.821 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.821 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.822 DEBUG:boto:Method: GET 2023-10-18T20:43:04.822 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.822 DEBUG:boto:Data: 2023-10-18T20:43:04.823 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.823 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.824 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.824 DEBUG:boto:Params: {} 2023-10-18T20:43:04.825 DEBUG:boto:Token: None 2023-10-18T20:43:04.825 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.826 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lUrId420K8uVDdWi/rIICleTB5k= 2023-10-18T20:43:04.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lUrId420K8uVDdWi/rIICleTB5k=', 'Content-Length': '0'} 2023-10-18T20:43:04.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feb903b84e66543f-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.829 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.829 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.830 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.830 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.831 DEBUG:boto:Method: GET 2023-10-18T20:43:04.831 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.831 DEBUG:boto:Data: 2023-10-18T20:43:04.832 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.832 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.833 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.833 DEBUG:boto:Params: {} 2023-10-18T20:43:04.834 DEBUG:boto:Token: None 2023-10-18T20:43:04.834 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:43:04.835 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lUrId420K8uVDdWi/rIICleTB5k= 2023-10-18T20:43:04.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lUrId420K8uVDdWi/rIICleTB5k=', 'Content-Length': '0'} 2023-10-18T20:43:04.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dd20297760aa34e-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.878 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.879 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.879 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.880 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.880 DEBUG:boto:Method: GET 2023-10-18T20:43:04.881 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.881 DEBUG:boto:Data: 2023-10-18T20:43:04.882 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.882 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.883 DEBUG:boto:Params: {} 2023-10-18T20:43:04.883 DEBUG:boto:Token: None 2023-10-18T20:43:04.884 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080abaf010c3068d0-0065304358-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.888 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.889 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.889 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.890 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.890 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.890 DEBUG:boto:Method: GET 2023-10-18T20:43:04.891 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.891 DEBUG:boto:Data: 2023-10-18T20:43:04.892 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.892 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.892 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.893 DEBUG:boto:Params: {} 2023-10-18T20:43:04.893 DEBUG:boto:Token: None 2023-10-18T20:43:04.894 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.894 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c23e09ade4e8be8-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.899 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.900 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.900 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.901 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.901 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:43:04.901 DEBUG:boto:Data: 2023-10-18T20:43:04.902 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.902 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.903 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.903 DEBUG:boto:Params: {} 2023-10-18T20:43:04.904 DEBUG:boto:Token: None 2023-10-18T20:43:04.904 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19 2023-10-18T20:43:04.904 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6ooZaF4WGKJCZqa1wNHCzwU+mV4= 2023-10-18T20:43:04.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6ooZaF4WGKJCZqa1wNHCzwU+mV4=', 'Content-Length': '0'} 2023-10-18T20:43:04.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5fe33d9b00652e2-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.907 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.908 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.908 DEBUG:boto:Method: HEAD 2023-10-18T20:43:04.908 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:43:04.909 DEBUG:boto:Data: 2023-10-18T20:43:04.909 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.909 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.910 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.910 DEBUG:boto:Params: {} 2023-10-18T20:43:04.911 DEBUG:boto:Token: None 2023-10-18T20:43:04.911 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19 2023-10-18T20:43:04.911 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6ooZaF4WGKJCZqa1wNHCzwU+mV4= 2023-10-18T20:43:04.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6ooZaF4WGKJCZqa1wNHCzwU+mV4=', 'Content-Length': '0'} 2023-10-18T20:43:04.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000def88f6da54234ff-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:43:04.915 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.915 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.916 DEBUG:boto:Method: GET 2023-10-18T20:43:04.916 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:43:04.917 DEBUG:boto:Data: 2023-10-18T20:43:04.917 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.918 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.918 DEBUG:boto:Params: {} 2023-10-18T20:43:04.919 DEBUG:boto:Token: None 2023-10-18T20:43:04.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19 2023-10-18T20:43:04.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xH0lJMXqWWxM7z5lt1vrXoJYhMg= 2023-10-18T20:43:04.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xH0lJMXqWWxM7z5lt1vrXoJYhMg=', 'Content-Length': '0'} 2023-10-18T20:43:04.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201b6fa0520dc87a-0065304358-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.966 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:43:04.967 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:43:04.967 DEBUG:boto:Method: GET 2023-10-18T20:43:04.968 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:43:04.968 DEBUG:boto:Data: 2023-10-18T20:43:04.968 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.969 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.969 DEBUG:boto:Params: {} 2023-10-18T20:43:04.970 DEBUG:boto:Token: None 2023-10-18T20:43:04.970 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/obj19 2023-10-18T20:43:04.971 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xH0lJMXqWWxM7z5lt1vrXoJYhMg= 2023-10-18T20:43:04.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xH0lJMXqWWxM7z5lt1vrXoJYhMg=', 'Content-Length': '0'} 2023-10-18T20:43:04.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5d7130d9ef7af1-0065304358-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.974 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.974 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.975 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.975 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.975 DEBUG:boto:Method: GET 2023-10-18T20:43:04.976 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.976 DEBUG:boto:Data: 2023-10-18T20:43:04.977 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.978 DEBUG:boto:Port: 8001 2023-10-18T20:43:04.978 DEBUG:boto:Params: {} 2023-10-18T20:43:04.978 DEBUG:boto:Token: None 2023-10-18T20:43:04.979 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eb65604ceba1fa8-0065304358-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.983 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.984 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:04.984 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:04.984 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.985 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.985 DEBUG:boto:Method: GET 2023-10-18T20:43:04.985 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:43:04.986 DEBUG:boto:Data: 2023-10-18T20:43:04.986 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.987 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:04.987 DEBUG:boto:Port: 8000 2023-10-18T20:43:04.988 DEBUG:boto:Params: {} 2023-10-18T20:43:04.988 DEBUG:boto:Token: None 2023-10-18T20:43:04.989 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:04 GMT /bofrnf-59/ 2023-10-18T20:43:04.989 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k= 2023-10-18T20:43:04.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b7VSWrxwz37TJ0vEdwsnW2hXQ+k=', 'Content-Length': '0'} 2023-10-18T20:43:04.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b99ddb4b6edbd1e8-0065304358-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:04.994 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:04.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:43:04.995 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:04.996 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:04.996 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:04.996 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:04.997 DEBUG:boto:Method: GET 2023-10-18T20:43:04.997 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:04.998 DEBUG:boto:Data: 2023-10-18T20:43:04.998 DEBUG:boto:Headers: {} 2023-10-18T20:43:04.999 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:04.999 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.000 DEBUG:boto:Params: {} 2023-10-18T20:43:05.000 DEBUG:boto:Token: None 2023-10-18T20:43:05.001 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.001 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GPNZ2BPRLajWjz1g2+JLvu1n/tM= 2023-10-18T20:43:05.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GPNZ2BPRLajWjz1g2+JLvu1n/tM=', 'Content-Length': '0'} 2023-10-18T20:43:05.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a288228f250bde3d-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.004 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:05.005 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:05.005 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.006 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.006 DEBUG:boto:Method: GET 2023-10-18T20:43:05.006 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.007 DEBUG:boto:Data: 2023-10-18T20:43:05.007 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.008 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.008 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.008 DEBUG:boto:Params: {} 2023-10-18T20:43:05.009 DEBUG:boto:Token: None 2023-10-18T20:43:05.009 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.010 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GPNZ2BPRLajWjz1g2+JLvu1n/tM= 2023-10-18T20:43:05.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GPNZ2BPRLajWjz1g2+JLvu1n/tM=', 'Content-Length': '0'} 2023-10-18T20:43:05.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4d6b008a2594c5-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.013 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:05.013 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:05.014 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.014 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.015 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.015 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.015 DEBUG:boto:Data: 2023-10-18T20:43:05.016 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.016 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.017 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.017 DEBUG:boto:Params: {} 2023-10-18T20:43:05.017 DEBUG:boto:Token: None 2023-10-18T20:43:05.018 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.018 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gWd0WC9xHOPJi6Xfhtd8iXgqv5Q= 2023-10-18T20:43:05.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gWd0WC9xHOPJi6Xfhtd8iXgqv5Q=', 'Content-Length': '0'} 2023-10-18T20:43:05.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b060cadc8442f6-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.021 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:05.022 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:05.022 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.022 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.023 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.023 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.024 DEBUG:boto:Data: 2023-10-18T20:43:05.024 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.024 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.025 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.025 DEBUG:boto:Params: {} 2023-10-18T20:43:05.026 DEBUG:boto:Token: None 2023-10-18T20:43:05.026 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.027 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gWd0WC9xHOPJi6Xfhtd8iXgqv5Q= 2023-10-18T20:43:05.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gWd0WC9xHOPJi6Xfhtd8iXgqv5Q=', 'Content-Length': '0'} 2023-10-18T20:43:05.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007119735cb7656a7d-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:43:05.030 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:05.031 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:05.031 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.031 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.032 DEBUG:boto:Method: GET 2023-10-18T20:43:05.032 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.033 DEBUG:boto:Data: 2023-10-18T20:43:05.033 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.033 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.034 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.034 DEBUG:boto:Params: {} 2023-10-18T20:43:05.035 DEBUG:boto:Token: None 2023-10-18T20:43:05.035 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GPNZ2BPRLajWjz1g2+JLvu1n/tM= 2023-10-18T20:43:05.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GPNZ2BPRLajWjz1g2+JLvu1n/tM=', 'Content-Length': '0'} 2023-10-18T20:43:05.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e3aecc8e435baf2-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.039 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:05.039 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:05.040 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.040 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.040 DEBUG:boto:Method: GET 2023-10-18T20:43:05.041 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.041 DEBUG:boto:Data: 2023-10-18T20:43:05.042 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.042 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.043 DEBUG:boto:Params: {} 2023-10-18T20:43:05.043 DEBUG:boto:Token: None 2023-10-18T20:43:05.044 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:43:05.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GPNZ2BPRLajWjz1g2+JLvu1n/tM= 2023-10-18T20:43:05.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GPNZ2BPRLajWjz1g2+JLvu1n/tM=', 'Content-Length': '0'} 2023-10-18T20:43:05.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b045e5bb38c5db0d-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.047 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.048 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.048 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.049 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.049 DEBUG:boto:Method: GET 2023-10-18T20:43:05.050 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.050 DEBUG:boto:Data: 2023-10-18T20:43:05.050 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.051 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.051 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.052 DEBUG:boto:Params: {} 2023-10-18T20:43:05.052 DEBUG:boto:Token: None 2023-10-18T20:43:05.053 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.053 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f652c7c5915286-0065304359-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.102 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.103 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.104 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.104 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.104 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.105 DEBUG:boto:Method: GET 2023-10-18T20:43:05.105 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.106 DEBUG:boto:Data: 2023-10-18T20:43:05.106 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.106 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.107 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.107 DEBUG:boto:Params: {} 2023-10-18T20:43:05.108 DEBUG:boto:Token: None 2023-10-18T20:43:05.108 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031128c732d3c27df-0065304359-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.113 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.114 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:05.114 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:05.115 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.115 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:43:05.116 DEBUG:boto:Data: 2023-10-18T20:43:05.116 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.116 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.117 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.117 DEBUG:boto:Params: {} 2023-10-18T20:43:05.118 DEBUG:boto:Token: None 2023-10-18T20:43:05.118 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22 2023-10-18T20:43:05.119 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:86G2o5C54nKAVetHR1Ip7aEC9hQ= 2023-10-18T20:43:05.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:86G2o5C54nKAVetHR1Ip7aEC9hQ=', 'Content-Length': '0'} 2023-10-18T20:43:05.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a84e0cf8998a5f-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.122 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:05.122 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:05.122 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.123 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:43:05.123 DEBUG:boto:Data: 2023-10-18T20:43:05.124 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.124 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.125 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.125 DEBUG:boto:Params: {} 2023-10-18T20:43:05.125 DEBUG:boto:Token: None 2023-10-18T20:43:05.126 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22 2023-10-18T20:43:05.126 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:86G2o5C54nKAVetHR1Ip7aEC9hQ= 2023-10-18T20:43:05.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:86G2o5C54nKAVetHR1Ip7aEC9hQ=', 'Content-Length': '0'} 2023-10-18T20:43:05.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1dd4487e40acf11-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:43:05.130 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:05.131 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:05.131 DEBUG:boto:Method: GET 2023-10-18T20:43:05.131 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:43:05.132 DEBUG:boto:Data: 2023-10-18T20:43:05.132 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.133 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.133 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.134 DEBUG:boto:Params: {} 2023-10-18T20:43:05.134 DEBUG:boto:Token: None 2023-10-18T20:43:05.135 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22 2023-10-18T20:43:05.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1j9XddAbmzF0MEyYzH+/EVPIuMs= 2023-10-18T20:43:05.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1j9XddAbmzF0MEyYzH+/EVPIuMs=', 'Content-Length': '0'} 2023-10-18T20:43:05.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6acd37b1e30d14d-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.138 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:43:05.139 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:43:05.139 DEBUG:boto:Method: GET 2023-10-18T20:43:05.139 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:43:05.140 DEBUG:boto:Data: 2023-10-18T20:43:05.140 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.141 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.141 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.141 DEBUG:boto:Params: {} 2023-10-18T20:43:05.142 DEBUG:boto:Token: None 2023-10-18T20:43:05.142 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj22 2023-10-18T20:43:05.143 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1j9XddAbmzF0MEyYzH+/EVPIuMs= 2023-10-18T20:43:05.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1j9XddAbmzF0MEyYzH+/EVPIuMs=', 'Content-Length': '0'} 2023-10-18T20:43:05.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cf39c008741c62d-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.186 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.187 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.187 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.188 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.188 DEBUG:boto:Method: GET 2023-10-18T20:43:05.189 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.189 DEBUG:boto:Data: 2023-10-18T20:43:05.190 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.191 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.191 DEBUG:boto:Params: {} 2023-10-18T20:43:05.192 DEBUG:boto:Token: None 2023-10-18T20:43:05.192 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.193 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de9c98077b133d7f-0065304359-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.197 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.197 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.198 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.198 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.199 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.199 DEBUG:boto:Method: GET 2023-10-18T20:43:05.200 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:43:05.200 DEBUG:boto:Data: 2023-10-18T20:43:05.201 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.201 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.202 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.202 DEBUG:boto:Params: {} 2023-10-18T20:43:05.203 DEBUG:boto:Token: None 2023-10-18T20:43:05.203 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.204 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043a76de0bba0f062-0065304359-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.209 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:43:05.210 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.210 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.211 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.211 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.212 DEBUG:boto:Method: GET 2023-10-18T20:43:05.212 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.213 DEBUG:boto:Data: 2023-10-18T20:43:05.213 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.213 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.214 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.214 DEBUG:boto:Params: {} 2023-10-18T20:43:05.215 DEBUG:boto:Token: None 2023-10-18T20:43:05.216 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.216 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:odtLFWWcEW8Hjs82cYbHUdIByp8= 2023-10-18T20:43:05.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:odtLFWWcEW8Hjs82cYbHUdIByp8=', 'Content-Length': '0'} 2023-10-18T20:43:05.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d03e84402fc26097-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.220 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.220 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.221 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.221 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.222 DEBUG:boto:Method: GET 2023-10-18T20:43:05.222 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.223 DEBUG:boto:Data: 2023-10-18T20:43:05.223 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.224 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.224 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.224 DEBUG:boto:Params: {} 2023-10-18T20:43:05.225 DEBUG:boto:Token: None 2023-10-18T20:43:05.225 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.226 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:odtLFWWcEW8Hjs82cYbHUdIByp8= 2023-10-18T20:43:05.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:odtLFWWcEW8Hjs82cYbHUdIByp8=', 'Content-Length': '0'} 2023-10-18T20:43:05.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfc32e8a6f3fb922-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.229 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.230 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.230 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.230 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.231 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.232 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.232 DEBUG:boto:Data: 2023-10-18T20:43:05.232 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.233 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.233 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.234 DEBUG:boto:Params: {} 2023-10-18T20:43:05.234 DEBUG:boto:Token: None 2023-10-18T20:43:05.234 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.235 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4np8cZdaRnBXxtlgM1K0qusQEpk= 2023-10-18T20:43:05.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4np8cZdaRnBXxtlgM1K0qusQEpk=', 'Content-Length': '0'} 2023-10-18T20:43:05.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016fc2791ad237f6e-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.238 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.238 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.239 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.239 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.240 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.240 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.240 DEBUG:boto:Data: 2023-10-18T20:43:05.241 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.241 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.241 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.243 DEBUG:boto:Params: {} 2023-10-18T20:43:05.243 DEBUG:boto:Token: None 2023-10-18T20:43:05.244 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4np8cZdaRnBXxtlgM1K0qusQEpk= 2023-10-18T20:43:05.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4np8cZdaRnBXxtlgM1K0qusQEpk=', 'Content-Length': '0'} 2023-10-18T20:43:05.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000044dda38952c43e5-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:43:05.248 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.248 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.249 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.249 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.249 DEBUG:boto:Method: GET 2023-10-18T20:43:05.250 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.250 DEBUG:boto:Data: 2023-10-18T20:43:05.251 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.251 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.251 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.252 DEBUG:boto:Params: {} 2023-10-18T20:43:05.252 DEBUG:boto:Token: None 2023-10-18T20:43:05.253 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.253 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:odtLFWWcEW8Hjs82cYbHUdIByp8= 2023-10-18T20:43:05.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:odtLFWWcEW8Hjs82cYbHUdIByp8=', 'Content-Length': '0'} 2023-10-18T20:43:05.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7cd5e81b8bca14d-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.298 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.299 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.300 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.300 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.300 DEBUG:boto:Method: GET 2023-10-18T20:43:05.301 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.301 DEBUG:boto:Data: 2023-10-18T20:43:05.302 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.302 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.303 DEBUG:boto:Params: {} 2023-10-18T20:43:05.303 DEBUG:boto:Token: None 2023-10-18T20:43:05.304 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:43:05.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:odtLFWWcEW8Hjs82cYbHUdIByp8= 2023-10-18T20:43:05.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:odtLFWWcEW8Hjs82cYbHUdIByp8=', 'Content-Length': '0'} 2023-10-18T20:43:05.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086125f9e283fd2bc-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.308 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.308 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.309 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.309 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.309 DEBUG:boto:Method: GET 2023-10-18T20:43:05.310 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.310 DEBUG:boto:Data: 2023-10-18T20:43:05.311 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.311 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.311 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.312 DEBUG:boto:Params: {} 2023-10-18T20:43:05.312 DEBUG:boto:Token: None 2023-10-18T20:43:05.313 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaa6940f11ed3591-0065304359-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.317 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.318 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.318 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.318 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.319 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.319 DEBUG:boto:Method: GET 2023-10-18T20:43:05.320 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.320 DEBUG:boto:Data: 2023-10-18T20:43:05.320 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.321 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.321 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.322 DEBUG:boto:Params: {} 2023-10-18T20:43:05.322 DEBUG:boto:Token: None 2023-10-18T20:43:05.323 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.323 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5d3786a45c31c93-0065304359-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.370 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.371 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.371 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.372 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.372 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:43:05.372 DEBUG:boto:Data: 2023-10-18T20:43:05.373 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.373 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.374 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.374 DEBUG:boto:Params: {} 2023-10-18T20:43:05.374 DEBUG:boto:Token: None 2023-10-18T20:43:05.375 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23 2023-10-18T20:43:05.375 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HrekGTVJYgyyZV4HmcX7BytOV6I= 2023-10-18T20:43:05.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HrekGTVJYgyyZV4HmcX7BytOV6I=', 'Content-Length': '0'} 2023-10-18T20:43:05.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080330c65ae1bd4e1-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.379 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.379 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.379 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.380 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:43:05.380 DEBUG:boto:Data: 2023-10-18T20:43:05.381 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.381 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.382 DEBUG:boto:Params: {} 2023-10-18T20:43:05.382 DEBUG:boto:Token: None 2023-10-18T20:43:05.383 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23 2023-10-18T20:43:05.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HrekGTVJYgyyZV4HmcX7BytOV6I= 2023-10-18T20:43:05.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HrekGTVJYgyyZV4HmcX7BytOV6I=', 'Content-Length': '0'} 2023-10-18T20:43:05.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ad1997da254477-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:43:05.387 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.387 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.387 DEBUG:boto:Method: GET 2023-10-18T20:43:05.388 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:43:05.388 DEBUG:boto:Data: 2023-10-18T20:43:05.389 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.390 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.390 DEBUG:boto:Params: {} 2023-10-18T20:43:05.390 DEBUG:boto:Token: None 2023-10-18T20:43:05.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23 2023-10-18T20:43:05.391 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bJrOqZUJBlsSmcp0VPIPaWlXhik= 2023-10-18T20:43:05.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bJrOqZUJBlsSmcp0VPIPaWlXhik=', 'Content-Length': '0'} 2023-10-18T20:43:05.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f68132655e3a516-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.395 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:43:05.395 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:43:05.395 DEBUG:boto:Method: GET 2023-10-18T20:43:05.396 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:43:05.396 DEBUG:boto:Data: 2023-10-18T20:43:05.397 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.397 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.397 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.398 DEBUG:boto:Params: {} 2023-10-18T20:43:05.398 DEBUG:boto:Token: None 2023-10-18T20:43:05.399 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj23 2023-10-18T20:43:05.399 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bJrOqZUJBlsSmcp0VPIPaWlXhik= 2023-10-18T20:43:05.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bJrOqZUJBlsSmcp0VPIPaWlXhik=', 'Content-Length': '0'} 2023-10-18T20:43:05.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec790e24cd3e88be-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.402 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.403 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.403 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.403 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.405 DEBUG:boto:Method: GET 2023-10-18T20:43:05.405 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.406 DEBUG:boto:Data: 2023-10-18T20:43:05.406 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.407 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.407 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.407 DEBUG:boto:Params: {} 2023-10-18T20:43:05.408 DEBUG:boto:Token: None 2023-10-18T20:43:05.408 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.409 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34c44253aed5662-0065304359-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.412 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.413 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.413 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.414 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.414 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.414 DEBUG:boto:Method: GET 2023-10-18T20:43:05.415 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:43:05.415 DEBUG:boto:Data: 2023-10-18T20:43:05.415 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.416 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.416 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.417 DEBUG:boto:Params: {} 2023-10-18T20:43:05.417 DEBUG:boto:Token: None 2023-10-18T20:43:05.418 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5e7d73b982a6110-0065304359-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.425 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:43:05.426 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.427 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.427 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.428 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.428 DEBUG:boto:Method: GET 2023-10-18T20:43:05.428 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.429 DEBUG:boto:Data: 2023-10-18T20:43:05.429 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.430 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.430 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.430 DEBUG:boto:Params: {} 2023-10-18T20:43:05.431 DEBUG:boto:Token: None 2023-10-18T20:43:05.431 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.432 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYTGzZcMYBqfIhj/gq2b1XQ8NN4= 2023-10-18T20:43:05.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYTGzZcMYBqfIhj/gq2b1XQ8NN4=', 'Content-Length': '0'} 2023-10-18T20:43:05.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e97c3dae1305b8-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.435 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.435 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.436 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.436 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.436 DEBUG:boto:Method: GET 2023-10-18T20:43:05.437 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.437 DEBUG:boto:Data: 2023-10-18T20:43:05.437 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.438 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.438 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.439 DEBUG:boto:Params: {} 2023-10-18T20:43:05.440 DEBUG:boto:Token: None 2023-10-18T20:43:05.440 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.441 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYTGzZcMYBqfIhj/gq2b1XQ8NN4= 2023-10-18T20:43:05.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYTGzZcMYBqfIhj/gq2b1XQ8NN4=', 'Content-Length': '0'} 2023-10-18T20:43:05.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011056d2a326e5f3f-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.444 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.444 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.445 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.445 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.446 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.446 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.446 DEBUG:boto:Data: 2023-10-18T20:43:05.447 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.447 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.448 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.448 DEBUG:boto:Params: {} 2023-10-18T20:43:05.449 DEBUG:boto:Token: None 2023-10-18T20:43:05.449 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.449 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q7Ksv3NIsuaCgivx8onIhhzzF8A= 2023-10-18T20:43:05.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q7Ksv3NIsuaCgivx8onIhhzzF8A=', 'Content-Length': '0'} 2023-10-18T20:43:05.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000043e55179f8fe54b-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.452 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.453 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.453 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.454 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.454 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.454 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.455 DEBUG:boto:Data: 2023-10-18T20:43:05.455 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.456 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.456 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.456 DEBUG:boto:Params: {} 2023-10-18T20:43:05.457 DEBUG:boto:Token: None 2023-10-18T20:43:05.457 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.458 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q7Ksv3NIsuaCgivx8onIhhzzF8A= 2023-10-18T20:43:05.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q7Ksv3NIsuaCgivx8onIhhzzF8A=', 'Content-Length': '0'} 2023-10-18T20:43:05.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6fee2c633bc9e4f-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:43:05.461 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.462 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.462 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.463 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.463 DEBUG:boto:Method: GET 2023-10-18T20:43:05.463 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.464 DEBUG:boto:Data: 2023-10-18T20:43:05.464 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.465 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.465 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.465 DEBUG:boto:Params: {} 2023-10-18T20:43:05.466 DEBUG:boto:Token: None 2023-10-18T20:43:05.466 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.467 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYTGzZcMYBqfIhj/gq2b1XQ8NN4= 2023-10-18T20:43:05.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYTGzZcMYBqfIhj/gq2b1XQ8NN4=', 'Content-Length': '0'} 2023-10-18T20:43:05.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f10343d056b5643-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.510 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.511 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.511 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.512 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.512 DEBUG:boto:Method: GET 2023-10-18T20:43:05.513 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.513 DEBUG:boto:Data: 2023-10-18T20:43:05.514 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.514 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.514 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.515 DEBUG:boto:Params: {} 2023-10-18T20:43:05.515 DEBUG:boto:Token: None 2023-10-18T20:43:05.516 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:43:05.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYTGzZcMYBqfIhj/gq2b1XQ8NN4= 2023-10-18T20:43:05.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYTGzZcMYBqfIhj/gq2b1XQ8NN4=', 'Content-Length': '0'} 2023-10-18T20:43:05.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8aaf1815b02e523-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.520 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.520 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.520 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.521 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.521 DEBUG:boto:Method: GET 2023-10-18T20:43:05.522 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.522 DEBUG:boto:Data: 2023-10-18T20:43:05.523 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.523 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.523 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.524 DEBUG:boto:Params: {} 2023-10-18T20:43:05.524 DEBUG:boto:Token: None 2023-10-18T20:43:05.525 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.525 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008266b19e3e9582f8-0065304359-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.529 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.529 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.530 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.530 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.531 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.531 DEBUG:boto:Method: GET 2023-10-18T20:43:05.532 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.532 DEBUG:boto:Data: 2023-10-18T20:43:05.532 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.533 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.533 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.534 DEBUG:boto:Params: {} 2023-10-18T20:43:05.534 DEBUG:boto:Token: None 2023-10-18T20:43:05.535 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.535 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000afc9ab97031f30a-0065304359-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.540 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.540 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.541 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.541 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.542 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:43:05.542 DEBUG:boto:Data: 2023-10-18T20:43:05.542 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.543 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.543 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.544 DEBUG:boto:Params: {} 2023-10-18T20:43:05.544 DEBUG:boto:Token: None 2023-10-18T20:43:05.545 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29 2023-10-18T20:43:05.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:15S+xzOGKetAyp0D8YfmJhq4FNc= 2023-10-18T20:43:05.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:15S+xzOGKetAyp0D8YfmJhq4FNc=', 'Content-Length': '0'} 2023-10-18T20:43:05.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caaf90a5edc82260-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.548 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.549 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.549 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.549 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:43:05.550 DEBUG:boto:Data: 2023-10-18T20:43:05.550 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.551 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.551 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.551 DEBUG:boto:Params: {} 2023-10-18T20:43:05.552 DEBUG:boto:Token: None 2023-10-18T20:43:05.552 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29 2023-10-18T20:43:05.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:15S+xzOGKetAyp0D8YfmJhq4FNc= 2023-10-18T20:43:05.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:15S+xzOGKetAyp0D8YfmJhq4FNc=', 'Content-Length': '0'} 2023-10-18T20:43:05.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000797de6f18b941b2b-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:43:05.556 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.556 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.557 DEBUG:boto:Method: GET 2023-10-18T20:43:05.557 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:43:05.558 DEBUG:boto:Data: 2023-10-18T20:43:05.558 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.559 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.559 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.559 DEBUG:boto:Params: {} 2023-10-18T20:43:05.560 DEBUG:boto:Token: None 2023-10-18T20:43:05.560 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29 2023-10-18T20:43:05.561 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6ur+0tlfxQUMNAI6+N6B2bXaYkE= 2023-10-18T20:43:05.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6ur+0tlfxQUMNAI6+N6B2bXaYkE=', 'Content-Length': '0'} 2023-10-18T20:43:05.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8231085ab9b2965-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.564 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:43:05.565 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:43:05.565 DEBUG:boto:Method: GET 2023-10-18T20:43:05.565 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:43:05.566 DEBUG:boto:Data: 2023-10-18T20:43:05.566 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.567 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.567 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.568 DEBUG:boto:Params: {} 2023-10-18T20:43:05.568 DEBUG:boto:Token: None 2023-10-18T20:43:05.568 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj29 2023-10-18T20:43:05.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6ur+0tlfxQUMNAI6+N6B2bXaYkE= 2023-10-18T20:43:05.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6ur+0tlfxQUMNAI6+N6B2bXaYkE=', 'Content-Length': '0'} 2023-10-18T20:43:05.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e6a12cc524f412-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.618 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.619 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.619 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.620 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.620 DEBUG:boto:Method: GET 2023-10-18T20:43:05.621 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.621 DEBUG:boto:Data: 2023-10-18T20:43:05.621 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.622 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.622 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.623 DEBUG:boto:Params: {} 2023-10-18T20:43:05.623 DEBUG:boto:Token: None 2023-10-18T20:43:05.624 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c145880ef2bc166c-0065304359-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.628 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.629 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.629 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.629 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.630 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.630 DEBUG:boto:Method: GET 2023-10-18T20:43:05.631 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:43:05.631 DEBUG:boto:Data: 2023-10-18T20:43:05.631 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.632 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.632 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.632 DEBUG:boto:Params: {} 2023-10-18T20:43:05.633 DEBUG:boto:Token: None 2023-10-18T20:43:05.633 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.634 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a09bd5d82b9ff12-0065304359-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.639 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-18T20:43:05.640 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.640 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.641 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.641 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.641 DEBUG:boto:Method: GET 2023-10-18T20:43:05.642 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.642 DEBUG:boto:Data: 2023-10-18T20:43:05.643 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.643 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.643 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.644 DEBUG:boto:Params: {} 2023-10-18T20:43:05.644 DEBUG:boto:Token: None 2023-10-18T20:43:05.645 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.645 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3bfKlwBYIiiPkeCrHY/NjWO75/k= 2023-10-18T20:43:05.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3bfKlwBYIiiPkeCrHY/NjWO75/k=', 'Content-Length': '0'} 2023-10-18T20:43:05.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a6be49df92ea513-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.649 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.649 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.649 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.650 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.650 DEBUG:boto:Method: GET 2023-10-18T20:43:05.651 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.651 DEBUG:boto:Data: 2023-10-18T20:43:05.651 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.652 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.652 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.652 DEBUG:boto:Params: {} 2023-10-18T20:43:05.653 DEBUG:boto:Token: None 2023-10-18T20:43:05.653 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.654 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3bfKlwBYIiiPkeCrHY/NjWO75/k= 2023-10-18T20:43:05.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3bfKlwBYIiiPkeCrHY/NjWO75/k=', 'Content-Length': '0'} 2023-10-18T20:43:05.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b80b771c9146cdb5-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.657 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.657 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.658 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.658 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.659 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.659 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.659 DEBUG:boto:Data: 2023-10-18T20:43:05.660 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.660 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.660 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.661 DEBUG:boto:Params: {} 2023-10-18T20:43:05.661 DEBUG:boto:Token: None 2023-10-18T20:43:05.661 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QSLFTSYM73IlHCmGy9qYH65CQyU= 2023-10-18T20:43:05.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QSLFTSYM73IlHCmGy9qYH65CQyU=', 'Content-Length': '0'} 2023-10-18T20:43:05.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000383ae86d65f7f520-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.665 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.665 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.666 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.666 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.666 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.667 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.667 DEBUG:boto:Data: 2023-10-18T20:43:05.667 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.668 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.668 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.668 DEBUG:boto:Params: {} 2023-10-18T20:43:05.669 DEBUG:boto:Token: None 2023-10-18T20:43:05.669 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.670 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QSLFTSYM73IlHCmGy9qYH65CQyU= 2023-10-18T20:43:05.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QSLFTSYM73IlHCmGy9qYH65CQyU=', 'Content-Length': '0'} 2023-10-18T20:43:05.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084298802f49d2cf9-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-18T20:43:05.673 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.673 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.674 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.674 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.675 DEBUG:boto:Method: GET 2023-10-18T20:43:05.675 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.675 DEBUG:boto:Data: 2023-10-18T20:43:05.676 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.676 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.677 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.677 DEBUG:boto:Params: {} 2023-10-18T20:43:05.677 DEBUG:boto:Token: None 2023-10-18T20:43:05.678 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3bfKlwBYIiiPkeCrHY/NjWO75/k= 2023-10-18T20:43:05.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3bfKlwBYIiiPkeCrHY/NjWO75/k=', 'Content-Length': '0'} 2023-10-18T20:43:05.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2355179ec1e1150-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.682 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.682 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.682 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.683 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.683 DEBUG:boto:Method: GET 2023-10-18T20:43:05.684 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.684 DEBUG:boto:Data: 2023-10-18T20:43:05.685 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.685 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.685 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.686 DEBUG:boto:Params: {} 2023-10-18T20:43:05.686 DEBUG:boto:Token: None 2023-10-18T20:43:05.687 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:43:05.687 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3bfKlwBYIiiPkeCrHY/NjWO75/k= 2023-10-18T20:43:05.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3bfKlwBYIiiPkeCrHY/NjWO75/k=', 'Content-Length': '0'} 2023-10-18T20:43:05.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003996fcc8690069d3-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.690 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.691 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.691 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.692 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.692 DEBUG:boto:Method: GET 2023-10-18T20:43:05.692 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.693 DEBUG:boto:Data: 2023-10-18T20:43:05.693 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.694 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.694 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.695 DEBUG:boto:Params: {} 2023-10-18T20:43:05.695 DEBUG:boto:Token: None 2023-10-18T20:43:05.695 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.696 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d430a202b8f9c802-0065304359-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.699 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.700 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.700 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.701 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.701 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.701 DEBUG:boto:Method: GET 2023-10-18T20:43:05.702 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.702 DEBUG:boto:Data: 2023-10-18T20:43:05.703 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.703 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.703 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.704 DEBUG:boto:Params: {} 2023-10-18T20:43:05.704 DEBUG:boto:Token: None 2023-10-18T20:43:05.705 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.705 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c355effa70a7a51d-0065304359-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.710 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.710 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.711 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.711 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.712 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:43:05.712 DEBUG:boto:Data: 2023-10-18T20:43:05.712 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.713 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.713 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.714 DEBUG:boto:Params: {} 2023-10-18T20:43:05.714 DEBUG:boto:Token: None 2023-10-18T20:43:05.714 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31 2023-10-18T20:43:05.715 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IESC5Q6SBscIT+CEmbbLc43Q1sU= 2023-10-18T20:43:05.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IESC5Q6SBscIT+CEmbbLc43Q1sU=', 'Content-Length': '0'} 2023-10-18T20:43:05.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dfc440281bab9b1-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.718 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.718 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.719 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.719 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:43:05.719 DEBUG:boto:Data: 2023-10-18T20:43:05.720 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.720 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.720 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.721 DEBUG:boto:Params: {} 2023-10-18T20:43:05.721 DEBUG:boto:Token: None 2023-10-18T20:43:05.722 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31 2023-10-18T20:43:05.722 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IESC5Q6SBscIT+CEmbbLc43Q1sU= 2023-10-18T20:43:05.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IESC5Q6SBscIT+CEmbbLc43Q1sU=', 'Content-Length': '0'} 2023-10-18T20:43:05.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c99afb6f957a1c03-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-18T20:43:05.725 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.726 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.726 DEBUG:boto:Method: GET 2023-10-18T20:43:05.727 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:43:05.727 DEBUG:boto:Data: 2023-10-18T20:43:05.727 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.728 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.729 DEBUG:boto:Params: {} 2023-10-18T20:43:05.729 DEBUG:boto:Token: None 2023-10-18T20:43:05.729 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31 2023-10-18T20:43:05.730 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H94HeZOefAVORx2rU1zS/NTevMM= 2023-10-18T20:43:05.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H94HeZOefAVORx2rU1zS/NTevMM=', 'Content-Length': '0'} 2023-10-18T20:43:05.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c38c3630c645d496-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.774 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:43:05.775 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:43:05.775 DEBUG:boto:Method: GET 2023-10-18T20:43:05.776 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:43:05.776 DEBUG:boto:Data: 2023-10-18T20:43:05.776 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.777 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.777 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.778 DEBUG:boto:Params: {} 2023-10-18T20:43:05.778 DEBUG:boto:Token: None 2023-10-18T20:43:05.779 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj31 2023-10-18T20:43:05.779 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H94HeZOefAVORx2rU1zS/NTevMM= 2023-10-18T20:43:05.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H94HeZOefAVORx2rU1zS/NTevMM=', 'Content-Length': '0'} 2023-10-18T20:43:05.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf801ea06ab7cd83-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.782 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.782 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.783 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.783 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.784 DEBUG:boto:Method: GET 2023-10-18T20:43:05.784 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.784 DEBUG:boto:Data: 2023-10-18T20:43:05.785 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.785 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.785 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.786 DEBUG:boto:Params: {} 2023-10-18T20:43:05.786 DEBUG:boto:Token: None 2023-10-18T20:43:05.787 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.787 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c5a9c1ff015cf92-0065304359-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.791 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.791 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.792 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.792 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.792 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.793 DEBUG:boto:Method: GET 2023-10-18T20:43:05.793 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:43:05.793 DEBUG:boto:Data: 2023-10-18T20:43:05.794 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.794 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.795 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.795 DEBUG:boto:Params: {} 2023-10-18T20:43:05.795 DEBUG:boto:Token: None 2023-10-18T20:43:05.796 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.796 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013dea5662cff5783-0065304359-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.801 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:43:05.802 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.803 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.803 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.803 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.804 DEBUG:boto:Method: GET 2023-10-18T20:43:05.804 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.804 DEBUG:boto:Data: 2023-10-18T20:43:05.805 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.806 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.806 DEBUG:boto:Params: {} 2023-10-18T20:43:05.807 DEBUG:boto:Token: None 2023-10-18T20:43:05.807 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.807 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vC8Uh/LMfvJFLr6tt7ACWcYU8ko= 2023-10-18T20:43:05.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vC8Uh/LMfvJFLr6tt7ACWcYU8ko=', 'Content-Length': '0'} 2023-10-18T20:43:05.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000484d4bd1f425b176-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.811 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.811 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.812 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.812 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.812 DEBUG:boto:Method: GET 2023-10-18T20:43:05.813 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.813 DEBUG:boto:Data: 2023-10-18T20:43:05.814 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.814 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.815 DEBUG:boto:Params: {} 2023-10-18T20:43:05.815 DEBUG:boto:Token: None 2023-10-18T20:43:05.816 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vC8Uh/LMfvJFLr6tt7ACWcYU8ko= 2023-10-18T20:43:05.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vC8Uh/LMfvJFLr6tt7ACWcYU8ko=', 'Content-Length': '0'} 2023-10-18T20:43:05.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0671c3ea145dde1-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.866 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.867 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.867 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.867 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.868 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.868 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.869 DEBUG:boto:Data: 2023-10-18T20:43:05.869 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.870 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.870 DEBUG:boto:Params: {} 2023-10-18T20:43:05.871 DEBUG:boto:Token: None 2023-10-18T20:43:05.871 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.871 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aM2/n7JSAJY3WxsK7c4JZoq8SBY= 2023-10-18T20:43:05.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aM2/n7JSAJY3WxsK7c4JZoq8SBY=', 'Content-Length': '0'} 2023-10-18T20:43:05.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a8afc4e0b4997e4-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.874 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.875 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.875 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.875 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.876 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.876 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.877 DEBUG:boto:Data: 2023-10-18T20:43:05.877 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.878 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.878 DEBUG:boto:Params: {} 2023-10-18T20:43:05.879 DEBUG:boto:Token: None 2023-10-18T20:43:05.879 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aM2/n7JSAJY3WxsK7c4JZoq8SBY= 2023-10-18T20:43:05.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aM2/n7JSAJY3WxsK7c4JZoq8SBY=', 'Content-Length': '0'} 2023-10-18T20:43:05.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ec919166e29f57-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:43:05.883 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.883 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.884 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.884 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.884 DEBUG:boto:Method: GET 2023-10-18T20:43:05.885 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.885 DEBUG:boto:Data: 2023-10-18T20:43:05.886 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.886 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.886 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.887 DEBUG:boto:Params: {} 2023-10-18T20:43:05.887 DEBUG:boto:Token: None 2023-10-18T20:43:05.888 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vC8Uh/LMfvJFLr6tt7ACWcYU8ko= 2023-10-18T20:43:05.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vC8Uh/LMfvJFLr6tt7ACWcYU8ko=', 'Content-Length': '0'} 2023-10-18T20:43:05.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6d8837465de0a0a-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.891 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.891 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.892 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.892 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.893 DEBUG:boto:Method: GET 2023-10-18T20:43:05.893 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.893 DEBUG:boto:Data: 2023-10-18T20:43:05.894 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.894 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.895 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.895 DEBUG:boto:Params: {} 2023-10-18T20:43:05.895 DEBUG:boto:Token: None 2023-10-18T20:43:05.896 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:43:05.896 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vC8Uh/LMfvJFLr6tt7ACWcYU8ko= 2023-10-18T20:43:05.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vC8Uh/LMfvJFLr6tt7ACWcYU8ko=', 'Content-Length': '0'} 2023-10-18T20:43:05.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feacd90691092864-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.899 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.900 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.900 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:43:05.901 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:43:05.901 DEBUG:boto:Method: GET 2023-10-18T20:43:05.901 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:43:05.902 DEBUG:boto:Data: 2023-10-18T20:43:05.902 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.903 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.903 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.903 DEBUG:boto:Params: {} 2023-10-18T20:43:05.904 DEBUG:boto:Token: None 2023-10-18T20:43:05.904 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.905 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002def74a04fe68a4f-0065304359-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.954 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.955 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:05.955 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:05.956 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:43:05.956 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:43:05.957 DEBUG:boto:Method: GET 2023-10-18T20:43:05.957 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:43:05.957 DEBUG:boto:Data: 2023-10-18T20:43:05.958 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.959 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.959 DEBUG:boto:Params: {} 2023-10-18T20:43:05.959 DEBUG:boto:Token: None 2023-10-18T20:43:05.960 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/ 2023-10-18T20:43:05.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI= 2023-10-18T20:43:05.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IlL1HdoOnqArXFjX8b31tTfejbI=', 'Content-Length': '0'} 2023-10-18T20:43:05.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250744ebec5f597f-0065304359-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.965 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:05.966 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.967 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.967 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.967 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:43:05.968 DEBUG:boto:Data: 2023-10-18T20:43:05.968 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.969 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.969 DEBUG:boto:Params: {} 2023-10-18T20:43:05.970 DEBUG:boto:Token: None 2023-10-18T20:43:05.970 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33 2023-10-18T20:43:05.971 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5QJ6uyJVaqxpsRqcJnlIJTtDOaA= 2023-10-18T20:43:05.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5QJ6uyJVaqxpsRqcJnlIJTtDOaA=', 'Content-Length': '0'} 2023-10-18T20:43:05.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441b143b8b2f9827-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.974 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.975 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.975 DEBUG:boto:Method: HEAD 2023-10-18T20:43:05.975 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:43:05.976 DEBUG:boto:Data: 2023-10-18T20:43:05.976 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.977 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.978 DEBUG:boto:Params: {} 2023-10-18T20:43:05.978 DEBUG:boto:Token: None 2023-10-18T20:43:05.978 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33 2023-10-18T20:43:05.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5QJ6uyJVaqxpsRqcJnlIJTtDOaA= 2023-10-18T20:43:05.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5QJ6uyJVaqxpsRqcJnlIJTtDOaA=', 'Content-Length': '0'} 2023-10-18T20:43:05.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e3e6f07d0dde30b-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:43:05.983 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.983 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.984 DEBUG:boto:Method: GET 2023-10-18T20:43:05.984 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:43:05.985 DEBUG:boto:Data: 2023-10-18T20:43:05.985 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.986 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:05.986 DEBUG:boto:Port: 8001 2023-10-18T20:43:05.987 DEBUG:boto:Params: {} 2023-10-18T20:43:05.987 DEBUG:boto:Token: None 2023-10-18T20:43:05.988 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33 2023-10-18T20:43:05.989 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MIk5jhlxh1KQmnTdXtCdwhf4TwI= 2023-10-18T20:43:05.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MIk5jhlxh1KQmnTdXtCdwhf4TwI=', 'Content-Length': '0'} 2023-10-18T20:43:05.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000886f856681825a9e-0065304359-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:05.992 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:43:05.992 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:43:05.993 DEBUG:boto:Method: GET 2023-10-18T20:43:05.993 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:43:05.994 DEBUG:boto:Data: 2023-10-18T20:43:05.994 DEBUG:boto:Headers: {} 2023-10-18T20:43:05.995 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:05.995 DEBUG:boto:Port: 8000 2023-10-18T20:43:05.996 DEBUG:boto:Params: {} 2023-10-18T20:43:05.996 DEBUG:boto:Token: None 2023-10-18T20:43:05.996 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:05 GMT /bofrnf-59/obj33 2023-10-18T20:43:05.997 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MIk5jhlxh1KQmnTdXtCdwhf4TwI= 2023-10-18T20:43:05.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MIk5jhlxh1KQmnTdXtCdwhf4TwI=', 'Content-Length': '0'} 2023-10-18T20:43:06.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d320f12ae2f8480-0065304359-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.000 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.001 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.001 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:43:06.002 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:43:06.002 DEBUG:boto:Method: GET 2023-10-18T20:43:06.003 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:43:06.003 DEBUG:boto:Data: 2023-10-18T20:43:06.003 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.004 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.004 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.004 DEBUG:boto:Params: {} 2023-10-18T20:43:06.005 DEBUG:boto:Token: None 2023-10-18T20:43:06.005 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.006 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eba7f47310a7047-006530435a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.009 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.010 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.010 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.011 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:43:06.011 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:43:06.012 DEBUG:boto:Method: GET 2023-10-18T20:43:06.012 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:43:06.012 DEBUG:boto:Data: 2023-10-18T20:43:06.013 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.013 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.014 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.014 DEBUG:boto:Params: {} 2023-10-18T20:43:06.015 DEBUG:boto:Token: None 2023-10-18T20:43:06.015 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.015 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a586d046d58efeda-006530435a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.020 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:43:06.021 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.022 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.022 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.023 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.023 DEBUG:boto:Method: GET 2023-10-18T20:43:06.023 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.024 DEBUG:boto:Data: 2023-10-18T20:43:06.024 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.025 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.025 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.025 DEBUG:boto:Params: {} 2023-10-18T20:43:06.026 DEBUG:boto:Token: None 2023-10-18T20:43:06.026 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.027 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6wUsOj9sU+lpH42l6b+uOoTZqYw= 2023-10-18T20:43:06.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6wUsOj9sU+lpH42l6b+uOoTZqYw=', 'Content-Length': '0'} 2023-10-18T20:43:06.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2826880552fc188-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.030 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.031 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.031 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.032 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.032 DEBUG:boto:Method: GET 2023-10-18T20:43:06.032 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.033 DEBUG:boto:Data: 2023-10-18T20:43:06.033 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.034 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.034 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.035 DEBUG:boto:Params: {} 2023-10-18T20:43:06.035 DEBUG:boto:Token: None 2023-10-18T20:43:06.036 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6wUsOj9sU+lpH42l6b+uOoTZqYw= 2023-10-18T20:43:06.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6wUsOj9sU+lpH42l6b+uOoTZqYw=', 'Content-Length': '0'} 2023-10-18T20:43:06.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004907e5fb26f93d0f-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.082 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.083 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.083 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.084 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.084 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.084 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.085 DEBUG:boto:Data: 2023-10-18T20:43:06.085 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.086 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.086 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.086 DEBUG:boto:Params: {} 2023-10-18T20:43:06.087 DEBUG:boto:Token: None 2023-10-18T20:43:06.087 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.088 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tc9qEfyOCMjsNeZ0MoyD+p60xXM= 2023-10-18T20:43:06.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tc9qEfyOCMjsNeZ0MoyD+p60xXM=', 'Content-Length': '0'} 2023-10-18T20:43:06.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2c02d3382b12fed-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.091 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.091 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.092 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.092 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.093 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.093 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.093 DEBUG:boto:Data: 2023-10-18T20:43:06.094 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.094 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.095 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.095 DEBUG:boto:Params: {} 2023-10-18T20:43:06.095 DEBUG:boto:Token: None 2023-10-18T20:43:06.096 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.096 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tc9qEfyOCMjsNeZ0MoyD+p60xXM= 2023-10-18T20:43:06.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tc9qEfyOCMjsNeZ0MoyD+p60xXM=', 'Content-Length': '0'} 2023-10-18T20:43:06.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b5f313e5fb7f7c3-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:43:06.100 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.100 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.101 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.101 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.101 DEBUG:boto:Method: GET 2023-10-18T20:43:06.102 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.102 DEBUG:boto:Data: 2023-10-18T20:43:06.103 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.103 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.103 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.104 DEBUG:boto:Params: {} 2023-10-18T20:43:06.104 DEBUG:boto:Token: None 2023-10-18T20:43:06.105 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.105 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6wUsOj9sU+lpH42l6b+uOoTZqYw= 2023-10-18T20:43:06.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6wUsOj9sU+lpH42l6b+uOoTZqYw=', 'Content-Length': '0'} 2023-10-18T20:43:06.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edd67ea182d10547-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.154 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.155 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.155 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.156 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.156 DEBUG:boto:Method: GET 2023-10-18T20:43:06.157 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.157 DEBUG:boto:Data: 2023-10-18T20:43:06.157 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.158 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.159 DEBUG:boto:Params: {} 2023-10-18T20:43:06.159 DEBUG:boto:Token: None 2023-10-18T20:43:06.160 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:43:06.160 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6wUsOj9sU+lpH42l6b+uOoTZqYw= 2023-10-18T20:43:06.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6wUsOj9sU+lpH42l6b+uOoTZqYw=', 'Content-Length': '0'} 2023-10-18T20:43:06.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e76c0cc2afc6f73c-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.164 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.165 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.165 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.166 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.166 DEBUG:boto:Method: GET 2023-10-18T20:43:06.167 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.167 DEBUG:boto:Data: 2023-10-18T20:43:06.167 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.168 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.168 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.169 DEBUG:boto:Params: {} 2023-10-18T20:43:06.169 DEBUG:boto:Token: None 2023-10-18T20:43:06.170 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.170 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f74fc8dfd0de8090-006530435a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.174 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.174 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.175 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.175 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.176 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.176 DEBUG:boto:Method: GET 2023-10-18T20:43:06.177 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.177 DEBUG:boto:Data: 2023-10-18T20:43:06.178 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.179 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.179 DEBUG:boto:Params: {} 2023-10-18T20:43:06.179 DEBUG:boto:Token: None 2023-10-18T20:43:06.180 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061caf946223c8cc9-006530435a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.185 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.186 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.186 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.187 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.187 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:43:06.188 DEBUG:boto:Data: 2023-10-18T20:43:06.188 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.189 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.189 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.189 DEBUG:boto:Params: {} 2023-10-18T20:43:06.190 DEBUG:boto:Token: None 2023-10-18T20:43:06.190 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34 2023-10-18T20:43:06.191 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HOce/TPQoIdKxw9m2ZSKw8c3iyA= 2023-10-18T20:43:06.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HOce/TPQoIdKxw9m2ZSKw8c3iyA=', 'Content-Length': '0'} 2023-10-18T20:43:06.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071cd43d06f0b08be-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.194 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.194 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.195 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.195 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:43:06.195 DEBUG:boto:Data: 2023-10-18T20:43:06.196 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.196 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.197 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.197 DEBUG:boto:Params: {} 2023-10-18T20:43:06.197 DEBUG:boto:Token: None 2023-10-18T20:43:06.198 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34 2023-10-18T20:43:06.198 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HOce/TPQoIdKxw9m2ZSKw8c3iyA= 2023-10-18T20:43:06.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HOce/TPQoIdKxw9m2ZSKw8c3iyA=', 'Content-Length': '0'} 2023-10-18T20:43:06.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000030ca4bc59f4171e-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:43:06.202 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.203 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.203 DEBUG:boto:Method: GET 2023-10-18T20:43:06.203 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:43:06.204 DEBUG:boto:Data: 2023-10-18T20:43:06.204 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.205 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.205 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.206 DEBUG:boto:Params: {} 2023-10-18T20:43:06.206 DEBUG:boto:Token: None 2023-10-18T20:43:06.206 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34 2023-10-18T20:43:06.207 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Fs7PaOZTYmzydEkkKQK1h6sB8u4= 2023-10-18T20:43:06.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Fs7PaOZTYmzydEkkKQK1h6sB8u4=', 'Content-Length': '0'} 2023-10-18T20:43:06.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b09d2f36fc9b8d3f-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.210 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:43:06.211 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:43:06.211 DEBUG:boto:Method: GET 2023-10-18T20:43:06.211 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:43:06.212 DEBUG:boto:Data: 2023-10-18T20:43:06.212 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.213 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.213 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.214 DEBUG:boto:Params: {} 2023-10-18T20:43:06.214 DEBUG:boto:Token: None 2023-10-18T20:43:06.215 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj34 2023-10-18T20:43:06.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Fs7PaOZTYmzydEkkKQK1h6sB8u4= 2023-10-18T20:43:06.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Fs7PaOZTYmzydEkkKQK1h6sB8u4=', 'Content-Length': '0'} 2023-10-18T20:43:06.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbd925a96b0e73f6-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.258 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.259 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.259 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.260 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.260 DEBUG:boto:Method: GET 2023-10-18T20:43:06.261 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.261 DEBUG:boto:Data: 2023-10-18T20:43:06.261 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.262 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.262 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.263 DEBUG:boto:Params: {} 2023-10-18T20:43:06.263 DEBUG:boto:Token: None 2023-10-18T20:43:06.264 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.264 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecf033aa67e2f3f5-006530435a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.268 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.269 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.269 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.269 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.270 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.270 DEBUG:boto:Method: GET 2023-10-18T20:43:06.271 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:43:06.271 DEBUG:boto:Data: 2023-10-18T20:43:06.271 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.272 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.272 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.273 DEBUG:boto:Params: {} 2023-10-18T20:43:06.273 DEBUG:boto:Token: None 2023-10-18T20:43:06.274 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.274 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbd9c2f8fbbb38f9-006530435a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.279 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:43:06.280 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.281 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.281 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.281 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.282 DEBUG:boto:Method: GET 2023-10-18T20:43:06.282 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.283 DEBUG:boto:Data: 2023-10-18T20:43:06.283 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.284 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.284 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.284 DEBUG:boto:Params: {} 2023-10-18T20:43:06.285 DEBUG:boto:Token: None 2023-10-18T20:43:06.285 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.286 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDArXwzcBuKQh8ntOZiiLtTSUKI= 2023-10-18T20:43:06.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDArXwzcBuKQh8ntOZiiLtTSUKI=', 'Content-Length': '0'} 2023-10-18T20:43:06.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000623826c3a012ec00-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.289 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.290 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.290 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.291 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.291 DEBUG:boto:Method: GET 2023-10-18T20:43:06.291 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.292 DEBUG:boto:Data: 2023-10-18T20:43:06.292 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.293 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.293 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.293 DEBUG:boto:Params: {} 2023-10-18T20:43:06.294 DEBUG:boto:Token: None 2023-10-18T20:43:06.294 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.295 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDArXwzcBuKQh8ntOZiiLtTSUKI= 2023-10-18T20:43:06.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDArXwzcBuKQh8ntOZiiLtTSUKI=', 'Content-Length': '0'} 2023-10-18T20:43:06.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001abd59f11fc2f163-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.298 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.299 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.299 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.300 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.300 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.300 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.301 DEBUG:boto:Data: 2023-10-18T20:43:06.301 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.302 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.302 DEBUG:boto:Params: {} 2023-10-18T20:43:06.303 DEBUG:boto:Token: None 2023-10-18T20:43:06.303 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oe2ZLYaOe73TaL04PDnQD8VemIY= 2023-10-18T20:43:06.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oe2ZLYaOe73TaL04PDnQD8VemIY=', 'Content-Length': '0'} 2023-10-18T20:43:06.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004149deebac6e3c23-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.307 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.307 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.308 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.308 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.308 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.309 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.309 DEBUG:boto:Data: 2023-10-18T20:43:06.310 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.310 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.311 DEBUG:boto:Params: {} 2023-10-18T20:43:06.311 DEBUG:boto:Token: None 2023-10-18T20:43:06.312 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oe2ZLYaOe73TaL04PDnQD8VemIY= 2023-10-18T20:43:06.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oe2ZLYaOe73TaL04PDnQD8VemIY=', 'Content-Length': '0'} 2023-10-18T20:43:06.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7f686399406d59-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:43:06.315 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.316 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.316 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.317 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.317 DEBUG:boto:Method: GET 2023-10-18T20:43:06.317 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.318 DEBUG:boto:Data: 2023-10-18T20:43:06.318 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.319 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.319 DEBUG:boto:Params: {} 2023-10-18T20:43:06.320 DEBUG:boto:Token: None 2023-10-18T20:43:06.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDArXwzcBuKQh8ntOZiiLtTSUKI= 2023-10-18T20:43:06.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDArXwzcBuKQh8ntOZiiLtTSUKI=', 'Content-Length': '0'} 2023-10-18T20:43:06.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008251a2097185c2d0-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.366 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.367 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.368 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.368 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.368 DEBUG:boto:Method: GET 2023-10-18T20:43:06.369 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.369 DEBUG:boto:Data: 2023-10-18T20:43:06.370 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.371 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.371 DEBUG:boto:Params: {} 2023-10-18T20:43:06.372 DEBUG:boto:Token: None 2023-10-18T20:43:06.372 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:43:06.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDArXwzcBuKQh8ntOZiiLtTSUKI= 2023-10-18T20:43:06.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDArXwzcBuKQh8ntOZiiLtTSUKI=', 'Content-Length': '0'} 2023-10-18T20:43:06.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c4b2701c2b2cf7-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.376 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.377 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.377 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.377 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.378 DEBUG:boto:Method: GET 2023-10-18T20:43:06.378 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.379 DEBUG:boto:Data: 2023-10-18T20:43:06.379 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.379 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.380 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.380 DEBUG:boto:Params: {} 2023-10-18T20:43:06.381 DEBUG:boto:Token: None 2023-10-18T20:43:06.381 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c050051f89c3ef5b-006530435a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.385 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.386 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.386 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.387 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.387 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.388 DEBUG:boto:Method: GET 2023-10-18T20:43:06.388 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.388 DEBUG:boto:Data: 2023-10-18T20:43:06.389 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.390 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.390 DEBUG:boto:Params: {} 2023-10-18T20:43:06.391 DEBUG:boto:Token: None 2023-10-18T20:43:06.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a4608eeaf09053-006530435a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.438 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.439 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.440 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.440 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.441 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:43:06.441 DEBUG:boto:Data: 2023-10-18T20:43:06.441 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.442 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.443 DEBUG:boto:Params: {} 2023-10-18T20:43:06.443 DEBUG:boto:Token: None 2023-10-18T20:43:06.444 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38 2023-10-18T20:43:06.444 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1z2IDrLnbBm8rpCtpcbjRRC+z6M= 2023-10-18T20:43:06.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1z2IDrLnbBm8rpCtpcbjRRC+z6M=', 'Content-Length': '0'} 2023-10-18T20:43:06.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000852fd899fbadbeac-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.447 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.448 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.448 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.449 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:43:06.449 DEBUG:boto:Data: 2023-10-18T20:43:06.450 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.450 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.451 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.451 DEBUG:boto:Params: {} 2023-10-18T20:43:06.451 DEBUG:boto:Token: None 2023-10-18T20:43:06.452 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38 2023-10-18T20:43:06.452 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1z2IDrLnbBm8rpCtpcbjRRC+z6M= 2023-10-18T20:43:06.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1z2IDrLnbBm8rpCtpcbjRRC+z6M=', 'Content-Length': '0'} 2023-10-18T20:43:06.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a301643ad4c9ec3-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:43:06.456 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.456 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.457 DEBUG:boto:Method: GET 2023-10-18T20:43:06.457 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:43:06.458 DEBUG:boto:Data: 2023-10-18T20:43:06.458 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.459 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.459 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.459 DEBUG:boto:Params: {} 2023-10-18T20:43:06.460 DEBUG:boto:Token: None 2023-10-18T20:43:06.460 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38 2023-10-18T20:43:06.461 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nztqDtWDKmhDl6+oYu0CsAp0UpM= 2023-10-18T20:43:06.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nztqDtWDKmhDl6+oYu0CsAp0UpM=', 'Content-Length': '0'} 2023-10-18T20:43:06.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1f803309b68ddb-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.464 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:43:06.464 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:43:06.465 DEBUG:boto:Method: GET 2023-10-18T20:43:06.465 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:43:06.466 DEBUG:boto:Data: 2023-10-18T20:43:06.466 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.467 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.467 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.467 DEBUG:boto:Params: {} 2023-10-18T20:43:06.468 DEBUG:boto:Token: None 2023-10-18T20:43:06.468 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj38 2023-10-18T20:43:06.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nztqDtWDKmhDl6+oYu0CsAp0UpM= 2023-10-18T20:43:06.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nztqDtWDKmhDl6+oYu0CsAp0UpM=', 'Content-Length': '0'} 2023-10-18T20:43:06.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c28b17ade955f8b0-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.472 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.473 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.473 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.473 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.474 DEBUG:boto:Method: GET 2023-10-18T20:43:06.474 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.475 DEBUG:boto:Data: 2023-10-18T20:43:06.475 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.476 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.476 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.477 DEBUG:boto:Params: {} 2023-10-18T20:43:06.477 DEBUG:boto:Token: None 2023-10-18T20:43:06.477 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.478 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000533188aa656cf0-006530435a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.526 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.527 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.528 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.528 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.529 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.529 DEBUG:boto:Method: GET 2023-10-18T20:43:06.529 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:43:06.530 DEBUG:boto:Data: 2023-10-18T20:43:06.530 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.531 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.531 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.531 DEBUG:boto:Params: {} 2023-10-18T20:43:06.532 DEBUG:boto:Token: None 2023-10-18T20:43:06.532 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d93bd27a680858d-006530435a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.538 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:43:06.539 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.540 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.540 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.541 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.541 DEBUG:boto:Method: GET 2023-10-18T20:43:06.541 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.542 DEBUG:boto:Data: 2023-10-18T20:43:06.542 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.543 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.543 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.544 DEBUG:boto:Params: {} 2023-10-18T20:43:06.544 DEBUG:boto:Token: None 2023-10-18T20:43:06.545 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dzuc5/xWoEezsc07G96vhj0T98I= 2023-10-18T20:43:06.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dzuc5/xWoEezsc07G96vhj0T98I=', 'Content-Length': '0'} 2023-10-18T20:43:06.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d872b76147d221-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.548 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.549 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.549 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.549 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.550 DEBUG:boto:Method: GET 2023-10-18T20:43:06.550 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.551 DEBUG:boto:Data: 2023-10-18T20:43:06.551 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.552 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.553 DEBUG:boto:Params: {} 2023-10-18T20:43:06.553 DEBUG:boto:Token: None 2023-10-18T20:43:06.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.554 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dzuc5/xWoEezsc07G96vhj0T98I= 2023-10-18T20:43:06.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dzuc5/xWoEezsc07G96vhj0T98I=', 'Content-Length': '0'} 2023-10-18T20:43:06.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085bc444aff607366-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.557 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.558 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.558 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.559 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.559 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.559 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.560 DEBUG:boto:Data: 2023-10-18T20:43:06.560 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.561 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.561 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.561 DEBUG:boto:Params: {} 2023-10-18T20:43:06.562 DEBUG:boto:Token: None 2023-10-18T20:43:06.562 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.563 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0wv+CvW5/Chdr2rINUPKhP3LRNI= 2023-10-18T20:43:06.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0wv+CvW5/Chdr2rINUPKhP3LRNI=', 'Content-Length': '0'} 2023-10-18T20:43:06.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6733cba9565b555-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.566 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.566 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.567 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.567 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.568 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.568 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.569 DEBUG:boto:Data: 2023-10-18T20:43:06.569 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.570 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.570 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.570 DEBUG:boto:Params: {} 2023-10-18T20:43:06.571 DEBUG:boto:Token: None 2023-10-18T20:43:06.571 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.572 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0wv+CvW5/Chdr2rINUPKhP3LRNI= 2023-10-18T20:43:06.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0wv+CvW5/Chdr2rINUPKhP3LRNI=', 'Content-Length': '0'} 2023-10-18T20:43:06.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c107f0d05c89629c-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:43:06.576 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.576 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.577 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.577 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.577 DEBUG:boto:Method: GET 2023-10-18T20:43:06.578 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.578 DEBUG:boto:Data: 2023-10-18T20:43:06.579 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.579 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.580 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.580 DEBUG:boto:Params: {} 2023-10-18T20:43:06.580 DEBUG:boto:Token: None 2023-10-18T20:43:06.581 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dzuc5/xWoEezsc07G96vhj0T98I= 2023-10-18T20:43:06.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dzuc5/xWoEezsc07G96vhj0T98I=', 'Content-Length': '0'} 2023-10-18T20:43:06.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f5fba820e1f203e-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.585 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.585 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.586 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.586 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.587 DEBUG:boto:Method: GET 2023-10-18T20:43:06.587 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.588 DEBUG:boto:Data: 2023-10-18T20:43:06.588 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.589 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.589 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.589 DEBUG:boto:Params: {} 2023-10-18T20:43:06.590 DEBUG:boto:Token: None 2023-10-18T20:43:06.590 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:43:06.591 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dzuc5/xWoEezsc07G96vhj0T98I= 2023-10-18T20:43:06.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dzuc5/xWoEezsc07G96vhj0T98I=', 'Content-Length': '0'} 2023-10-18T20:43:06.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae76f4f14dbd2274-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.634 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.635 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.635 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.636 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.636 DEBUG:boto:Method: GET 2023-10-18T20:43:06.637 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.637 DEBUG:boto:Data: 2023-10-18T20:43:06.638 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.638 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.638 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.639 DEBUG:boto:Params: {} 2023-10-18T20:43:06.639 DEBUG:boto:Token: None 2023-10-18T20:43:06.640 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.640 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013f74af350d69b04-006530435a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.644 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.645 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.645 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.646 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.646 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.647 DEBUG:boto:Method: GET 2023-10-18T20:43:06.647 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.648 DEBUG:boto:Data: 2023-10-18T20:43:06.648 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.648 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.649 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.649 DEBUG:boto:Params: {} 2023-10-18T20:43:06.650 DEBUG:boto:Token: None 2023-10-18T20:43:06.650 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.651 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0835f653fcd4616-006530435a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.656 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.657 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.657 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.658 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.658 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:43:06.658 DEBUG:boto:Data: 2023-10-18T20:43:06.659 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.660 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.660 DEBUG:boto:Params: {} 2023-10-18T20:43:06.660 DEBUG:boto:Token: None 2023-10-18T20:43:06.661 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44 2023-10-18T20:43:06.661 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xRmsA+WZRBFMEgFJoJ4Cn54Abss= 2023-10-18T20:43:06.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xRmsA+WZRBFMEgFJoJ4Cn54Abss=', 'Content-Length': '0'} 2023-10-18T20:43:06.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000316b92655502f7d4-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.665 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.665 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.666 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.666 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:43:06.667 DEBUG:boto:Data: 2023-10-18T20:43:06.667 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.667 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.668 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.668 DEBUG:boto:Params: {} 2023-10-18T20:43:06.669 DEBUG:boto:Token: None 2023-10-18T20:43:06.669 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44 2023-10-18T20:43:06.669 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xRmsA+WZRBFMEgFJoJ4Cn54Abss= 2023-10-18T20:43:06.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xRmsA+WZRBFMEgFJoJ4Cn54Abss=', 'Content-Length': '0'} 2023-10-18T20:43:06.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0783350a5045cae-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:43:06.673 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.673 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.674 DEBUG:boto:Method: GET 2023-10-18T20:43:06.674 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:43:06.674 DEBUG:boto:Data: 2023-10-18T20:43:06.675 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.675 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.676 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.676 DEBUG:boto:Params: {} 2023-10-18T20:43:06.676 DEBUG:boto:Token: None 2023-10-18T20:43:06.677 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44 2023-10-18T20:43:06.677 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0B/MbQVCRHtoG56qOPSNfhkh0LQ= 2023-10-18T20:43:06.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0B/MbQVCRHtoG56qOPSNfhkh0LQ=', 'Content-Length': '0'} 2023-10-18T20:43:06.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005365e7acc103db26-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.680 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:43:06.681 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:43:06.681 DEBUG:boto:Method: GET 2023-10-18T20:43:06.681 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:43:06.682 DEBUG:boto:Data: 2023-10-18T20:43:06.682 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.683 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.683 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.684 DEBUG:boto:Params: {} 2023-10-18T20:43:06.684 DEBUG:boto:Token: None 2023-10-18T20:43:06.684 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj44 2023-10-18T20:43:06.685 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0B/MbQVCRHtoG56qOPSNfhkh0LQ= 2023-10-18T20:43:06.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0B/MbQVCRHtoG56qOPSNfhkh0LQ=', 'Content-Length': '0'} 2023-10-18T20:43:06.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000571d4318a7cc902e-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.688 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.688 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.689 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.689 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.689 DEBUG:boto:Method: GET 2023-10-18T20:43:06.690 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.690 DEBUG:boto:Data: 2023-10-18T20:43:06.691 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.691 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.691 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.692 DEBUG:boto:Params: {} 2023-10-18T20:43:06.692 DEBUG:boto:Token: None 2023-10-18T20:43:06.693 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.693 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0ddcff7818c763d-006530435a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.697 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.697 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.698 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.698 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.699 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.699 DEBUG:boto:Method: GET 2023-10-18T20:43:06.699 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:43:06.700 DEBUG:boto:Data: 2023-10-18T20:43:06.700 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.701 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.701 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.702 DEBUG:boto:Params: {} 2023-10-18T20:43:06.702 DEBUG:boto:Token: None 2023-10-18T20:43:06.703 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.703 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006959390a9bfc90d6-006530435a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.750 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:43:06.751 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.752 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.752 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.752 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.753 DEBUG:boto:Method: GET 2023-10-18T20:43:06.753 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.754 DEBUG:boto:Data: 2023-10-18T20:43:06.754 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.755 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.755 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.755 DEBUG:boto:Params: {} 2023-10-18T20:43:06.756 DEBUG:boto:Token: None 2023-10-18T20:43:06.756 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.757 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0agsBFOjhYGz6wbvWV6tDvopusI= 2023-10-18T20:43:06.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0agsBFOjhYGz6wbvWV6tDvopusI=', 'Content-Length': '0'} 2023-10-18T20:43:06.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351b28173f5e5710-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.760 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.760 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.761 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.761 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.762 DEBUG:boto:Method: GET 2023-10-18T20:43:06.762 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.763 DEBUG:boto:Data: 2023-10-18T20:43:06.763 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.764 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.764 DEBUG:boto:Params: {} 2023-10-18T20:43:06.765 DEBUG:boto:Token: None 2023-10-18T20:43:06.765 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.766 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0agsBFOjhYGz6wbvWV6tDvopusI= 2023-10-18T20:43:06.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0agsBFOjhYGz6wbvWV6tDvopusI=', 'Content-Length': '0'} 2023-10-18T20:43:06.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022180481a5403833-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.769 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.769 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.770 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.770 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.771 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.771 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.771 DEBUG:boto:Data: 2023-10-18T20:43:06.772 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.772 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.772 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.773 DEBUG:boto:Params: {} 2023-10-18T20:43:06.773 DEBUG:boto:Token: None 2023-10-18T20:43:06.774 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b4MagNgn/A8RC7asl+0KezVOnHo= 2023-10-18T20:43:06.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b4MagNgn/A8RC7asl+0KezVOnHo=', 'Content-Length': '0'} 2023-10-18T20:43:06.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001383b93d0bde184c-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.777 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.777 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.778 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.778 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.779 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.779 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.779 DEBUG:boto:Data: 2023-10-18T20:43:06.780 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.780 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.780 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.781 DEBUG:boto:Params: {} 2023-10-18T20:43:06.781 DEBUG:boto:Token: None 2023-10-18T20:43:06.782 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b4MagNgn/A8RC7asl+0KezVOnHo= 2023-10-18T20:43:06.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b4MagNgn/A8RC7asl+0KezVOnHo=', 'Content-Length': '0'} 2023-10-18T20:43:06.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005efc885daec8ed33-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:43:06.785 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.786 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.786 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.787 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.787 DEBUG:boto:Method: GET 2023-10-18T20:43:06.788 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.788 DEBUG:boto:Data: 2023-10-18T20:43:06.788 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.789 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.790 DEBUG:boto:Params: {} 2023-10-18T20:43:06.790 DEBUG:boto:Token: None 2023-10-18T20:43:06.791 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.791 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0agsBFOjhYGz6wbvWV6tDvopusI= 2023-10-18T20:43:06.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0agsBFOjhYGz6wbvWV6tDvopusI=', 'Content-Length': '0'} 2023-10-18T20:43:06.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000558e23287715418f-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.842 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.843 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.843 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.844 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.844 DEBUG:boto:Method: GET 2023-10-18T20:43:06.845 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.845 DEBUG:boto:Data: 2023-10-18T20:43:06.846 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.846 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.847 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.847 DEBUG:boto:Params: {} 2023-10-18T20:43:06.847 DEBUG:boto:Token: None 2023-10-18T20:43:06.848 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:43:06.848 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0agsBFOjhYGz6wbvWV6tDvopusI= 2023-10-18T20:43:06.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0agsBFOjhYGz6wbvWV6tDvopusI=', 'Content-Length': '0'} 2023-10-18T20:43:06.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000523279f016fe6a48-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.852 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.852 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.853 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.853 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.853 DEBUG:boto:Method: GET 2023-10-18T20:43:06.854 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.854 DEBUG:boto:Data: 2023-10-18T20:43:06.855 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.856 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.856 DEBUG:boto:Params: {} 2023-10-18T20:43:06.857 DEBUG:boto:Token: None 2023-10-18T20:43:06.857 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.858 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001254054fcb01cd93-006530435a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.861 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.862 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.862 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.863 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.863 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.864 DEBUG:boto:Method: GET 2023-10-18T20:43:06.864 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.864 DEBUG:boto:Data: 2023-10-18T20:43:06.865 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.866 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.866 DEBUG:boto:Params: {} 2023-10-18T20:43:06.867 DEBUG:boto:Token: None 2023-10-18T20:43:06.867 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.868 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002920a97a1a2b18bf-006530435a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.872 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.873 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.873 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.874 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.874 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:43:06.874 DEBUG:boto:Data: 2023-10-18T20:43:06.875 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.876 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.876 DEBUG:boto:Params: {} 2023-10-18T20:43:06.876 DEBUG:boto:Token: None 2023-10-18T20:43:06.877 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46 2023-10-18T20:43:06.877 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lkHzLImEB2pgf2XBd1hnmWGfzKg= 2023-10-18T20:43:06.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lkHzLImEB2pgf2XBd1hnmWGfzKg=', 'Content-Length': '0'} 2023-10-18T20:43:06.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5047248b9c77d08-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.880 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.881 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.881 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.881 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:43:06.882 DEBUG:boto:Data: 2023-10-18T20:43:06.882 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.883 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.883 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.883 DEBUG:boto:Params: {} 2023-10-18T20:43:06.884 DEBUG:boto:Token: None 2023-10-18T20:43:06.884 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46 2023-10-18T20:43:06.885 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lkHzLImEB2pgf2XBd1hnmWGfzKg= 2023-10-18T20:43:06.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lkHzLImEB2pgf2XBd1hnmWGfzKg=', 'Content-Length': '0'} 2023-10-18T20:43:06.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee8b8ba05ccd44b7-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:43:06.888 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.888 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.889 DEBUG:boto:Method: GET 2023-10-18T20:43:06.889 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:43:06.890 DEBUG:boto:Data: 2023-10-18T20:43:06.890 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.891 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.891 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.891 DEBUG:boto:Params: {} 2023-10-18T20:43:06.892 DEBUG:boto:Token: None 2023-10-18T20:43:06.892 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46 2023-10-18T20:43:06.893 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gUIR/mMiTL4dlSjAqK3Gr36Gu+0= 2023-10-18T20:43:06.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gUIR/mMiTL4dlSjAqK3Gr36Gu+0=', 'Content-Length': '0'} 2023-10-18T20:43:06.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000584e01d3f17d6474-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.896 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:43:06.897 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:43:06.897 DEBUG:boto:Method: GET 2023-10-18T20:43:06.898 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:43:06.898 DEBUG:boto:Data: 2023-10-18T20:43:06.899 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.899 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.899 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.900 DEBUG:boto:Params: {} 2023-10-18T20:43:06.900 DEBUG:boto:Token: None 2023-10-18T20:43:06.901 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj46 2023-10-18T20:43:06.901 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gUIR/mMiTL4dlSjAqK3Gr36Gu+0= 2023-10-18T20:43:06.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gUIR/mMiTL4dlSjAqK3Gr36Gu+0=', 'Content-Length': '0'} 2023-10-18T20:43:06.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab446d7d85d19ba4-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.946 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.947 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.947 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.948 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.948 DEBUG:boto:Method: GET 2023-10-18T20:43:06.949 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.949 DEBUG:boto:Data: 2023-10-18T20:43:06.949 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.950 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.950 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.950 DEBUG:boto:Params: {} 2023-10-18T20:43:06.951 DEBUG:boto:Token: None 2023-10-18T20:43:06.951 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.952 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4f3dadd77805982-006530435a-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.955 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.956 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:06.956 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:06.957 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.957 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.958 DEBUG:boto:Method: GET 2023-10-18T20:43:06.958 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:43:06.959 DEBUG:boto:Data: 2023-10-18T20:43:06.959 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.959 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.960 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.960 DEBUG:boto:Params: {} 2023-10-18T20:43:06.961 DEBUG:boto:Token: None 2023-10-18T20:43:06.961 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/ 2023-10-18T20:43:06.962 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM= 2023-10-18T20:43:06.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DUBproWGe/WS7kjv7GEDTLF8FxM=', 'Content-Length': '0'} 2023-10-18T20:43:06.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b3dc5263bbe70b1-006530435a-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.966 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:06.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:43:06.967 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:06.968 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:06.968 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.969 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.969 DEBUG:boto:Method: GET 2023-10-18T20:43:06.969 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.970 DEBUG:boto:Data: 2023-10-18T20:43:06.970 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.971 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.971 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.971 DEBUG:boto:Params: {} 2023-10-18T20:43:06.972 DEBUG:boto:Token: None 2023-10-18T20:43:06.972 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.973 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O+5YNCq5NifADzTwTj7WwVHnjRE= 2023-10-18T20:43:06.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O+5YNCq5NifADzTwTj7WwVHnjRE=', 'Content-Length': '0'} 2023-10-18T20:43:06.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067680a25a145d1a6-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.976 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:06.976 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:06.977 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.977 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.978 DEBUG:boto:Method: GET 2023-10-18T20:43:06.978 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.979 DEBUG:boto:Data: 2023-10-18T20:43:06.979 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.979 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.980 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.980 DEBUG:boto:Params: {} 2023-10-18T20:43:06.980 DEBUG:boto:Token: None 2023-10-18T20:43:06.981 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.981 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O+5YNCq5NifADzTwTj7WwVHnjRE= 2023-10-18T20:43:06.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O+5YNCq5NifADzTwTj7WwVHnjRE=', 'Content-Length': '0'} 2023-10-18T20:43:06.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b63e18c074c4166-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.984 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:06.985 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:06.985 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.986 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.986 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.986 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.987 DEBUG:boto:Data: 2023-10-18T20:43:06.987 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.987 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:06.988 DEBUG:boto:Port: 8001 2023-10-18T20:43:06.988 DEBUG:boto:Params: {} 2023-10-18T20:43:06.989 DEBUG:boto:Token: None 2023-10-18T20:43:06.989 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.990 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:56OzHrAMtEqpUzzADvzAshwhi2g= 2023-10-18T20:43:06.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:56OzHrAMtEqpUzzADvzAshwhi2g=', 'Content-Length': '0'} 2023-10-18T20:43:06.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d65f292aa99653cf-006530435a-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:06.993 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:06.993 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:06.993 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.994 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.994 DEBUG:boto:Method: HEAD 2023-10-18T20:43:06.995 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.995 DEBUG:boto:Data: 2023-10-18T20:43:06.995 DEBUG:boto:Headers: {} 2023-10-18T20:43:06.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:06.996 DEBUG:boto:Port: 8000 2023-10-18T20:43:06.997 DEBUG:boto:Params: {} 2023-10-18T20:43:06.997 DEBUG:boto:Token: None 2023-10-18T20:43:06.997 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:06 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:06.998 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:56OzHrAMtEqpUzzADvzAshwhi2g= 2023-10-18T20:43:06.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:06 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:56OzHrAMtEqpUzzADvzAshwhi2g=', 'Content-Length': '0'} 2023-10-18T20:43:07.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4f5733428ea53f5-006530435a-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:43:07.001 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:07.002 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:07.002 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:07.003 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:07.003 DEBUG:boto:Method: GET 2023-10-18T20:43:07.003 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:07.004 DEBUG:boto:Data: 2023-10-18T20:43:07.004 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.004 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.005 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.005 DEBUG:boto:Params: {} 2023-10-18T20:43:07.006 DEBUG:boto:Token: None 2023-10-18T20:43:07.006 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:07.007 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6vuHALhINWhyVmLCKAUEHHNjL0= 2023-10-18T20:43:07.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6vuHALhINWhyVmLCKAUEHHNjL0=', 'Content-Length': '0'} 2023-10-18T20:43:07.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004053744e4ca4d11f-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.050 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:07.051 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:07.052 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:07.052 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:43:07.052 DEBUG:boto:Method: GET 2023-10-18T20:43:07.053 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:07.053 DEBUG:boto:Data: 2023-10-18T20:43:07.054 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.054 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.054 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.055 DEBUG:boto:Params: {} 2023-10-18T20:43:07.055 DEBUG:boto:Token: None 2023-10-18T20:43:07.056 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:43:07.056 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:H6vuHALhINWhyVmLCKAUEHHNjL0= 2023-10-18T20:43:07.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:H6vuHALhINWhyVmLCKAUEHHNjL0=', 'Content-Length': '0'} 2023-10-18T20:43:07.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eefe79e34df26d8b-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.060 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.060 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.061 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.061 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.061 DEBUG:boto:Method: GET 2023-10-18T20:43:07.062 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.062 DEBUG:boto:Data: 2023-10-18T20:43:07.063 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.063 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.064 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.064 DEBUG:boto:Params: {} 2023-10-18T20:43:07.064 DEBUG:boto:Token: None 2023-10-18T20:43:07.065 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.065 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad505ace37de76d-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.069 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.070 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.070 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.071 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.071 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.071 DEBUG:boto:Method: GET 2023-10-18T20:43:07.072 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.072 DEBUG:boto:Data: 2023-10-18T20:43:07.073 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.074 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.074 DEBUG:boto:Params: {} 2023-10-18T20:43:07.074 DEBUG:boto:Token: None 2023-10-18T20:43:07.075 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.075 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bdead0820ead333-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.080 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.081 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:07.081 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:07.082 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.082 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:43:07.082 DEBUG:boto:Data: 2023-10-18T20:43:07.083 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.083 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.084 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.084 DEBUG:boto:Params: {} 2023-10-18T20:43:07.084 DEBUG:boto:Token: None 2023-10-18T20:43:07.085 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj51 2023-10-18T20:43:07.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZkDlq9MYrjmWytPVXgu05MUbJNs= 2023-10-18T20:43:07.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZkDlq9MYrjmWytPVXgu05MUbJNs=', 'Content-Length': '0'} 2023-10-18T20:43:07.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f93a222eca58ba0-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.088 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:07.089 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:07.089 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.089 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:43:07.090 DEBUG:boto:Data: 2023-10-18T20:43:07.090 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.091 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.091 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.091 DEBUG:boto:Params: {} 2023-10-18T20:43:07.092 DEBUG:boto:Token: None 2023-10-18T20:43:07.092 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj51 2023-10-18T20:43:07.093 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZkDlq9MYrjmWytPVXgu05MUbJNs= 2023-10-18T20:43:07.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZkDlq9MYrjmWytPVXgu05MUbJNs=', 'Content-Length': '0'} 2023-10-18T20:43:07.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d31eaba738b7bdf6-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:43:07.096 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:07.097 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:07.097 DEBUG:boto:Method: GET 2023-10-18T20:43:07.097 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:43:07.098 DEBUG:boto:Data: 2023-10-18T20:43:07.098 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.099 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.099 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.100 DEBUG:boto:Params: {} 2023-10-18T20:43:07.100 DEBUG:boto:Token: None 2023-10-18T20:43:07.101 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj51 2023-10-18T20:43:07.101 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eoB6W126UjjeoiRU9OP6QTyxXXY= 2023-10-18T20:43:07.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eoB6W126UjjeoiRU9OP6QTyxXXY=', 'Content-Length': '0'} 2023-10-18T20:43:07.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081583f3f0be1d56d-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.104 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:43:07.105 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:43:07.105 DEBUG:boto:Method: GET 2023-10-18T20:43:07.105 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:43:07.106 DEBUG:boto:Data: 2023-10-18T20:43:07.106 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.107 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.107 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.108 DEBUG:boto:Params: {} 2023-10-18T20:43:07.108 DEBUG:boto:Token: None 2023-10-18T20:43:07.109 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj51 2023-10-18T20:43:07.109 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eoB6W126UjjeoiRU9OP6QTyxXXY= 2023-10-18T20:43:07.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eoB6W126UjjeoiRU9OP6QTyxXXY=', 'Content-Length': '0'} 2023-10-18T20:43:07.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d45b2a500c94fe-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.158 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.159 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.159 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.160 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.160 DEBUG:boto:Method: GET 2023-10-18T20:43:07.160 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.161 DEBUG:boto:Data: 2023-10-18T20:43:07.161 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.161 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.162 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.162 DEBUG:boto:Params: {} 2023-10-18T20:43:07.163 DEBUG:boto:Token: None 2023-10-18T20:43:07.163 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.164 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000137c93a9e19302c7-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.167 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.168 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.168 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.169 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.169 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.170 DEBUG:boto:Method: GET 2023-10-18T20:43:07.170 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:43:07.170 DEBUG:boto:Data: 2023-10-18T20:43:07.171 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.171 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.172 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.172 DEBUG:boto:Params: {} 2023-10-18T20:43:07.172 DEBUG:boto:Token: None 2023-10-18T20:43:07.173 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.173 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b353235b6139a2c-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.178 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:43:07.179 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.180 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.180 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.180 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.181 DEBUG:boto:Method: GET 2023-10-18T20:43:07.181 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.181 DEBUG:boto:Data: 2023-10-18T20:43:07.182 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.182 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.183 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.183 DEBUG:boto:Params: {} 2023-10-18T20:43:07.184 DEBUG:boto:Token: None 2023-10-18T20:43:07.184 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.184 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zzC/FSkhSsjcIOBAeoWnTb58w2U= 2023-10-18T20:43:07.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zzC/FSkhSsjcIOBAeoWnTb58w2U=', 'Content-Length': '0'} 2023-10-18T20:43:07.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009b0b043b745c999-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.188 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.188 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.189 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.189 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.189 DEBUG:boto:Method: GET 2023-10-18T20:43:07.190 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.190 DEBUG:boto:Data: 2023-10-18T20:43:07.191 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.191 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.192 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.192 DEBUG:boto:Params: {} 2023-10-18T20:43:07.192 DEBUG:boto:Token: None 2023-10-18T20:43:07.193 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.193 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zzC/FSkhSsjcIOBAeoWnTb58w2U= 2023-10-18T20:43:07.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zzC/FSkhSsjcIOBAeoWnTb58w2U=', 'Content-Length': '0'} 2023-10-18T20:43:07.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000334c25ea4b6aad33-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.197 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.197 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.198 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.198 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.198 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.199 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.199 DEBUG:boto:Data: 2023-10-18T20:43:07.200 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.200 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.200 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.201 DEBUG:boto:Params: {} 2023-10-18T20:43:07.201 DEBUG:boto:Token: None 2023-10-18T20:43:07.202 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.202 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:k5yxCIPUXls7wg/1+ipKDMZAN8c= 2023-10-18T20:43:07.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:k5yxCIPUXls7wg/1+ipKDMZAN8c=', 'Content-Length': '0'} 2023-10-18T20:43:07.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b14c7af7631eb40-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.205 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.206 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.206 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.207 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.207 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.207 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.208 DEBUG:boto:Data: 2023-10-18T20:43:07.208 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.208 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.209 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.209 DEBUG:boto:Params: {} 2023-10-18T20:43:07.210 DEBUG:boto:Token: None 2023-10-18T20:43:07.210 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:k5yxCIPUXls7wg/1+ipKDMZAN8c= 2023-10-18T20:43:07.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:k5yxCIPUXls7wg/1+ipKDMZAN8c=', 'Content-Length': '0'} 2023-10-18T20:43:07.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4be1855056a54bf-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:43:07.215 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.215 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.215 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.216 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.216 DEBUG:boto:Method: GET 2023-10-18T20:43:07.217 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.217 DEBUG:boto:Data: 2023-10-18T20:43:07.218 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.218 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.219 DEBUG:boto:Params: {} 2023-10-18T20:43:07.219 DEBUG:boto:Token: None 2023-10-18T20:43:07.220 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zzC/FSkhSsjcIOBAeoWnTb58w2U= 2023-10-18T20:43:07.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zzC/FSkhSsjcIOBAeoWnTb58w2U=', 'Content-Length': '0'} 2023-10-18T20:43:07.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b4411e456eda28-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.270 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.271 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.271 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.272 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.272 DEBUG:boto:Method: GET 2023-10-18T20:43:07.272 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.273 DEBUG:boto:Data: 2023-10-18T20:43:07.273 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.274 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.274 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.274 DEBUG:boto:Params: {} 2023-10-18T20:43:07.275 DEBUG:boto:Token: None 2023-10-18T20:43:07.275 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:43:07.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zzC/FSkhSsjcIOBAeoWnTb58w2U= 2023-10-18T20:43:07.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zzC/FSkhSsjcIOBAeoWnTb58w2U=', 'Content-Length': '0'} 2023-10-18T20:43:07.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d99da7d4948c762-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.279 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.279 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.280 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.280 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.281 DEBUG:boto:Method: GET 2023-10-18T20:43:07.281 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.281 DEBUG:boto:Data: 2023-10-18T20:43:07.282 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.283 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.283 DEBUG:boto:Params: {} 2023-10-18T20:43:07.284 DEBUG:boto:Token: None 2023-10-18T20:43:07.284 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.285 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc7b79e9ab17d0c-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.288 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.289 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.289 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.290 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.290 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.291 DEBUG:boto:Method: GET 2023-10-18T20:43:07.291 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.291 DEBUG:boto:Data: 2023-10-18T20:43:07.292 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.292 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.293 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.293 DEBUG:boto:Params: {} 2023-10-18T20:43:07.293 DEBUG:boto:Token: None 2023-10-18T20:43:07.294 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.295 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d1896637e12cff1-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.299 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.300 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.300 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.301 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.301 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:43:07.302 DEBUG:boto:Data: 2023-10-18T20:43:07.302 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.303 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.303 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.303 DEBUG:boto:Params: {} 2023-10-18T20:43:07.304 DEBUG:boto:Token: None 2023-10-18T20:43:07.304 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55 2023-10-18T20:43:07.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MDNfmiRKEnXxrpfDj8MafR/Gy0c= 2023-10-18T20:43:07.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MDNfmiRKEnXxrpfDj8MafR/Gy0c=', 'Content-Length': '0'} 2023-10-18T20:43:07.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfb46b915d35f219-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.308 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.308 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.308 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.309 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:43:07.309 DEBUG:boto:Data: 2023-10-18T20:43:07.310 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.310 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.311 DEBUG:boto:Params: {} 2023-10-18T20:43:07.311 DEBUG:boto:Token: None 2023-10-18T20:43:07.312 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55 2023-10-18T20:43:07.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MDNfmiRKEnXxrpfDj8MafR/Gy0c= 2023-10-18T20:43:07.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MDNfmiRKEnXxrpfDj8MafR/Gy0c=', 'Content-Length': '0'} 2023-10-18T20:43:07.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d19de23062774cc-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:43:07.315 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.316 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.316 DEBUG:boto:Method: GET 2023-10-18T20:43:07.317 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:43:07.317 DEBUG:boto:Data: 2023-10-18T20:43:07.318 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.318 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.319 DEBUG:boto:Params: {} 2023-10-18T20:43:07.319 DEBUG:boto:Token: None 2023-10-18T20:43:07.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55 2023-10-18T20:43:07.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:716K64CSpIlGEZWV9Jp7U6T/890= 2023-10-18T20:43:07.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:716K64CSpIlGEZWV9Jp7U6T/890=', 'Content-Length': '0'} 2023-10-18T20:43:07.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002af36870f8ceb304-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.366 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:43:07.367 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:43:07.367 DEBUG:boto:Method: GET 2023-10-18T20:43:07.368 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:43:07.368 DEBUG:boto:Data: 2023-10-18T20:43:07.368 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.369 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.369 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.369 DEBUG:boto:Params: {} 2023-10-18T20:43:07.370 DEBUG:boto:Token: None 2023-10-18T20:43:07.371 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj55 2023-10-18T20:43:07.371 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:716K64CSpIlGEZWV9Jp7U6T/890= 2023-10-18T20:43:07.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:716K64CSpIlGEZWV9Jp7U6T/890=', 'Content-Length': '0'} 2023-10-18T20:43:07.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027976bf56e04815f-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.374 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.375 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.375 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.375 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.376 DEBUG:boto:Method: GET 2023-10-18T20:43:07.376 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.377 DEBUG:boto:Data: 2023-10-18T20:43:07.377 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.378 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.378 DEBUG:boto:Params: {} 2023-10-18T20:43:07.379 DEBUG:boto:Token: None 2023-10-18T20:43:07.379 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.380 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062372bf99f750f9b-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.383 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.384 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.384 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.385 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.385 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.386 DEBUG:boto:Method: GET 2023-10-18T20:43:07.386 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:43:07.387 DEBUG:boto:Data: 2023-10-18T20:43:07.387 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.388 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.388 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.388 DEBUG:boto:Params: {} 2023-10-18T20:43:07.389 DEBUG:boto:Token: None 2023-10-18T20:43:07.389 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.390 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000da6b1f3bf3f5a89-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.395 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:43:07.396 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.396 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.397 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.397 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.398 DEBUG:boto:Method: GET 2023-10-18T20:43:07.398 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.398 DEBUG:boto:Data: 2023-10-18T20:43:07.399 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.400 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.400 DEBUG:boto:Params: {} 2023-10-18T20:43:07.400 DEBUG:boto:Token: None 2023-10-18T20:43:07.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YceSJiQZnKNnsZVhntkAU5LZP1Y= 2023-10-18T20:43:07.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YceSJiQZnKNnsZVhntkAU5LZP1Y=', 'Content-Length': '0'} 2023-10-18T20:43:07.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c4aa67540c24b13-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.405 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.405 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.405 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.406 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.406 DEBUG:boto:Method: GET 2023-10-18T20:43:07.407 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.407 DEBUG:boto:Data: 2023-10-18T20:43:07.407 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.408 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.408 DEBUG:boto:Params: {} 2023-10-18T20:43:07.409 DEBUG:boto:Token: None 2023-10-18T20:43:07.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YceSJiQZnKNnsZVhntkAU5LZP1Y= 2023-10-18T20:43:07.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YceSJiQZnKNnsZVhntkAU5LZP1Y=', 'Content-Length': '0'} 2023-10-18T20:43:07.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd6bedd77bead647-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.413 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.413 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.413 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.414 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.414 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.415 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.415 DEBUG:boto:Data: 2023-10-18T20:43:07.415 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.416 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.416 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.417 DEBUG:boto:Params: {} 2023-10-18T20:43:07.417 DEBUG:boto:Token: None 2023-10-18T20:43:07.417 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hEuXvkY+Ifg20pu9DeARXvTxBmo= 2023-10-18T20:43:07.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hEuXvkY+Ifg20pu9DeARXvTxBmo=', 'Content-Length': '0'} 2023-10-18T20:43:07.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baca9c4f2c90aa45-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.421 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.421 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.422 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.422 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.423 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.423 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.423 DEBUG:boto:Data: 2023-10-18T20:43:07.424 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.424 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.424 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.425 DEBUG:boto:Params: {} 2023-10-18T20:43:07.425 DEBUG:boto:Token: None 2023-10-18T20:43:07.426 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.426 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hEuXvkY+Ifg20pu9DeARXvTxBmo= 2023-10-18T20:43:07.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hEuXvkY+Ifg20pu9DeARXvTxBmo=', 'Content-Length': '0'} 2023-10-18T20:43:07.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002381c056a8ee8438-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:43:07.430 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.430 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.431 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.431 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.431 DEBUG:boto:Method: GET 2023-10-18T20:43:07.432 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.432 DEBUG:boto:Data: 2023-10-18T20:43:07.433 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.434 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.434 DEBUG:boto:Params: {} 2023-10-18T20:43:07.434 DEBUG:boto:Token: None 2023-10-18T20:43:07.435 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.435 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YceSJiQZnKNnsZVhntkAU5LZP1Y= 2023-10-18T20:43:07.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YceSJiQZnKNnsZVhntkAU5LZP1Y=', 'Content-Length': '0'} 2023-10-18T20:43:07.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794e4a34209c1559-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.438 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.439 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.439 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.440 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.440 DEBUG:boto:Method: GET 2023-10-18T20:43:07.440 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.441 DEBUG:boto:Data: 2023-10-18T20:43:07.441 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.442 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.443 DEBUG:boto:Params: {} 2023-10-18T20:43:07.443 DEBUG:boto:Token: None 2023-10-18T20:43:07.445 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:43:07.445 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YceSJiQZnKNnsZVhntkAU5LZP1Y= 2023-10-18T20:43:07.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YceSJiQZnKNnsZVhntkAU5LZP1Y=', 'Content-Length': '0'} 2023-10-18T20:43:07.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b59411df426667-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.448 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.448 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.449 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.449 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.450 DEBUG:boto:Method: GET 2023-10-18T20:43:07.450 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.450 DEBUG:boto:Data: 2023-10-18T20:43:07.451 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.451 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.452 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.452 DEBUG:boto:Params: {} 2023-10-18T20:43:07.452 DEBUG:boto:Token: None 2023-10-18T20:43:07.453 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.453 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037054a561c923a97-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.457 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.458 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.458 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.459 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.459 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.460 DEBUG:boto:Method: GET 2023-10-18T20:43:07.460 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.460 DEBUG:boto:Data: 2023-10-18T20:43:07.461 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.461 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.461 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.462 DEBUG:boto:Params: {} 2023-10-18T20:43:07.462 DEBUG:boto:Token: None 2023-10-18T20:43:07.463 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.463 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000017836297076e601-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.468 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.469 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.469 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.470 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.470 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:43:07.471 DEBUG:boto:Data: 2023-10-18T20:43:07.471 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.472 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.472 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.472 DEBUG:boto:Params: {} 2023-10-18T20:43:07.473 DEBUG:boto:Token: None 2023-10-18T20:43:07.473 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57 2023-10-18T20:43:07.474 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q0/1kgDjmqN/1IWV+DQLrvBgbjU= 2023-10-18T20:43:07.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q0/1kgDjmqN/1IWV+DQLrvBgbjU=', 'Content-Length': '0'} 2023-10-18T20:43:07.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee4d6f9682800b93-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.477 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.477 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.478 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.478 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:43:07.479 DEBUG:boto:Data: 2023-10-18T20:43:07.479 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.479 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.480 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.480 DEBUG:boto:Params: {} 2023-10-18T20:43:07.481 DEBUG:boto:Token: None 2023-10-18T20:43:07.481 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57 2023-10-18T20:43:07.481 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q0/1kgDjmqN/1IWV+DQLrvBgbjU= 2023-10-18T20:43:07.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q0/1kgDjmqN/1IWV+DQLrvBgbjU=', 'Content-Length': '0'} 2023-10-18T20:43:07.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e67501247d6bc093-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:43:07.485 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.485 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.486 DEBUG:boto:Method: GET 2023-10-18T20:43:07.486 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:43:07.487 DEBUG:boto:Data: 2023-10-18T20:43:07.487 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.487 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.488 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.488 DEBUG:boto:Params: {} 2023-10-18T20:43:07.489 DEBUG:boto:Token: None 2023-10-18T20:43:07.489 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57 2023-10-18T20:43:07.490 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nL7UmaD6sXEZSdbhaWNgmI8Mpbo= 2023-10-18T20:43:07.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nL7UmaD6sXEZSdbhaWNgmI8Mpbo=', 'Content-Length': '0'} 2023-10-18T20:43:07.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000845548f764ffcfad-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.493 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:43:07.493 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:43:07.494 DEBUG:boto:Method: GET 2023-10-18T20:43:07.494 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:43:07.494 DEBUG:boto:Data: 2023-10-18T20:43:07.495 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.495 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.496 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.496 DEBUG:boto:Params: {} 2023-10-18T20:43:07.497 DEBUG:boto:Token: None 2023-10-18T20:43:07.497 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj57 2023-10-18T20:43:07.498 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nL7UmaD6sXEZSdbhaWNgmI8Mpbo= 2023-10-18T20:43:07.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nL7UmaD6sXEZSdbhaWNgmI8Mpbo=', 'Content-Length': '0'} 2023-10-18T20:43:07.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b8678c9e4aef855-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.501 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.501 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.502 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.502 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.503 DEBUG:boto:Method: GET 2023-10-18T20:43:07.503 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.503 DEBUG:boto:Data: 2023-10-18T20:43:07.504 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.504 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.505 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.505 DEBUG:boto:Params: {} 2023-10-18T20:43:07.506 DEBUG:boto:Token: None 2023-10-18T20:43:07.506 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.506 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000945d5107a372a664-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.510 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.511 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.511 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.512 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.512 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.512 DEBUG:boto:Method: GET 2023-10-18T20:43:07.513 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:43:07.513 DEBUG:boto:Data: 2023-10-18T20:43:07.514 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.514 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.514 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.515 DEBUG:boto:Params: {} 2023-10-18T20:43:07.515 DEBUG:boto:Token: None 2023-10-18T20:43:07.516 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.516 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000959ee8d19114e612-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.521 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:43:07.522 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.522 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.523 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.523 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.524 DEBUG:boto:Method: GET 2023-10-18T20:43:07.524 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.524 DEBUG:boto:Data: 2023-10-18T20:43:07.525 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.525 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.526 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.526 DEBUG:boto:Params: {} 2023-10-18T20:43:07.526 DEBUG:boto:Token: None 2023-10-18T20:43:07.527 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.527 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:462kjjZDlubdYanTjtNav47xeos= 2023-10-18T20:43:07.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:462kjjZDlubdYanTjtNav47xeos=', 'Content-Length': '0'} 2023-10-18T20:43:07.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370da0d4633df6b1-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.530 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.531 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.531 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.532 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.532 DEBUG:boto:Method: GET 2023-10-18T20:43:07.532 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.533 DEBUG:boto:Data: 2023-10-18T20:43:07.533 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.534 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.534 DEBUG:boto:Params: {} 2023-10-18T20:43:07.535 DEBUG:boto:Token: None 2023-10-18T20:43:07.535 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:462kjjZDlubdYanTjtNav47xeos= 2023-10-18T20:43:07.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:462kjjZDlubdYanTjtNav47xeos=', 'Content-Length': '0'} 2023-10-18T20:43:07.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdfbf0a676fc2767-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.539 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.539 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.540 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.540 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.541 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.541 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.541 DEBUG:boto:Data: 2023-10-18T20:43:07.542 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.542 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.543 DEBUG:boto:Params: {} 2023-10-18T20:43:07.543 DEBUG:boto:Token: None 2023-10-18T20:43:07.544 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8bGzy9uREzuDSPxf+FU9vPFCC70= 2023-10-18T20:43:07.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8bGzy9uREzuDSPxf+FU9vPFCC70=', 'Content-Length': '0'} 2023-10-18T20:43:07.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af6035166cfc9d15-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.547 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.547 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.548 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.548 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.548 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.549 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.549 DEBUG:boto:Data: 2023-10-18T20:43:07.550 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.550 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.551 DEBUG:boto:Params: {} 2023-10-18T20:43:07.551 DEBUG:boto:Token: None 2023-10-18T20:43:07.552 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.552 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8bGzy9uREzuDSPxf+FU9vPFCC70= 2023-10-18T20:43:07.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8bGzy9uREzuDSPxf+FU9vPFCC70=', 'Content-Length': '0'} 2023-10-18T20:43:07.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ff002af6cdd1c8-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:43:07.555 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.556 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.556 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.557 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.557 DEBUG:boto:Method: GET 2023-10-18T20:43:07.557 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.558 DEBUG:boto:Data: 2023-10-18T20:43:07.558 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.559 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.559 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.559 DEBUG:boto:Params: {} 2023-10-18T20:43:07.560 DEBUG:boto:Token: None 2023-10-18T20:43:07.560 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.561 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:462kjjZDlubdYanTjtNav47xeos= 2023-10-18T20:43:07.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:462kjjZDlubdYanTjtNav47xeos=', 'Content-Length': '0'} 2023-10-18T20:43:07.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6c3457200fed5bb-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.610 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.611 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.611 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.612 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.612 DEBUG:boto:Method: GET 2023-10-18T20:43:07.612 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.613 DEBUG:boto:Data: 2023-10-18T20:43:07.613 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.614 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.614 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.615 DEBUG:boto:Params: {} 2023-10-18T20:43:07.615 DEBUG:boto:Token: None 2023-10-18T20:43:07.616 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:43:07.616 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:462kjjZDlubdYanTjtNav47xeos= 2023-10-18T20:43:07.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:462kjjZDlubdYanTjtNav47xeos=', 'Content-Length': '0'} 2023-10-18T20:43:07.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c87a1d0c737fc5-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.619 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.620 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.620 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.621 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.621 DEBUG:boto:Method: GET 2023-10-18T20:43:07.621 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.622 DEBUG:boto:Data: 2023-10-18T20:43:07.622 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.623 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.623 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.624 DEBUG:boto:Params: {} 2023-10-18T20:43:07.624 DEBUG:boto:Token: None 2023-10-18T20:43:07.625 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.625 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2c240445338608-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.629 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.629 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.630 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.630 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.631 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.631 DEBUG:boto:Method: GET 2023-10-18T20:43:07.631 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.632 DEBUG:boto:Data: 2023-10-18T20:43:07.632 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.633 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.633 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.634 DEBUG:boto:Params: {} 2023-10-18T20:43:07.634 DEBUG:boto:Token: None 2023-10-18T20:43:07.635 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.635 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4205b5aa01c7d9c-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.640 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.641 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.641 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.642 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.642 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:43:07.642 DEBUG:boto:Data: 2023-10-18T20:43:07.643 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.643 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.644 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.644 DEBUG:boto:Params: {} 2023-10-18T20:43:07.645 DEBUG:boto:Token: None 2023-10-18T20:43:07.645 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58 2023-10-18T20:43:07.645 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMTun//Zr8mJIxjrSxwENpLEC+k= 2023-10-18T20:43:07.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMTun//Zr8mJIxjrSxwENpLEC+k=', 'Content-Length': '0'} 2023-10-18T20:43:07.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d386a7d51064e5-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.649 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.649 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.649 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.650 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:43:07.650 DEBUG:boto:Data: 2023-10-18T20:43:07.651 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.652 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.652 DEBUG:boto:Params: {} 2023-10-18T20:43:07.652 DEBUG:boto:Token: None 2023-10-18T20:43:07.653 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58 2023-10-18T20:43:07.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DMTun//Zr8mJIxjrSxwENpLEC+k= 2023-10-18T20:43:07.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DMTun//Zr8mJIxjrSxwENpLEC+k=', 'Content-Length': '0'} 2023-10-18T20:43:07.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071e96490dbef0200-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:43:07.657 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.657 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.657 DEBUG:boto:Method: GET 2023-10-18T20:43:07.658 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:43:07.658 DEBUG:boto:Data: 2023-10-18T20:43:07.659 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.660 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.660 DEBUG:boto:Params: {} 2023-10-18T20:43:07.660 DEBUG:boto:Token: None 2023-10-18T20:43:07.661 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58 2023-10-18T20:43:07.661 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g5+jeQJc2/3siGqmf+K/B79FXtY= 2023-10-18T20:43:07.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g5+jeQJc2/3siGqmf+K/B79FXtY=', 'Content-Length': '0'} 2023-10-18T20:43:07.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ad7fff40a728b4-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.664 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:43:07.665 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:43:07.665 DEBUG:boto:Method: GET 2023-10-18T20:43:07.666 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:43:07.666 DEBUG:boto:Data: 2023-10-18T20:43:07.666 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.667 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.667 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.668 DEBUG:boto:Params: {} 2023-10-18T20:43:07.668 DEBUG:boto:Token: None 2023-10-18T20:43:07.668 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj58 2023-10-18T20:43:07.669 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g5+jeQJc2/3siGqmf+K/B79FXtY= 2023-10-18T20:43:07.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g5+jeQJc2/3siGqmf+K/B79FXtY=', 'Content-Length': '0'} 2023-10-18T20:43:07.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a89039a28922c6-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.672 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.673 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.673 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.674 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.674 DEBUG:boto:Method: GET 2023-10-18T20:43:07.674 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.675 DEBUG:boto:Data: 2023-10-18T20:43:07.675 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.676 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.676 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.677 DEBUG:boto:Params: {} 2023-10-18T20:43:07.677 DEBUG:boto:Token: None 2023-10-18T20:43:07.677 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.678 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a88dd67bcdc3ec47-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.682 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.682 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.683 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.683 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.684 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.684 DEBUG:boto:Method: GET 2023-10-18T20:43:07.685 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:43:07.685 DEBUG:boto:Data: 2023-10-18T20:43:07.685 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.686 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.687 DEBUG:boto:Params: {} 2023-10-18T20:43:07.687 DEBUG:boto:Token: None 2023-10-18T20:43:07.688 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001224e6d5a0ca4e79-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.693 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-18T20:43:07.694 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.694 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.695 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.695 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.695 DEBUG:boto:Method: GET 2023-10-18T20:43:07.696 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.696 DEBUG:boto:Data: 2023-10-18T20:43:07.697 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.697 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.697 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.698 DEBUG:boto:Params: {} 2023-10-18T20:43:07.698 DEBUG:boto:Token: None 2023-10-18T20:43:07.699 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.699 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5FDNYUuJAkDlQ/qdCPxKt54XJ0M= 2023-10-18T20:43:07.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5FDNYUuJAkDlQ/qdCPxKt54XJ0M=', 'Content-Length': '0'} 2023-10-18T20:43:07.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000278bdecc017176d4-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.703 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.703 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.704 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.704 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.705 DEBUG:boto:Method: GET 2023-10-18T20:43:07.705 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.706 DEBUG:boto:Data: 2023-10-18T20:43:07.706 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.707 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.707 DEBUG:boto:Params: {} 2023-10-18T20:43:07.708 DEBUG:boto:Token: None 2023-10-18T20:43:07.708 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5FDNYUuJAkDlQ/qdCPxKt54XJ0M= 2023-10-18T20:43:07.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5FDNYUuJAkDlQ/qdCPxKt54XJ0M=', 'Content-Length': '0'} 2023-10-18T20:43:07.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000815b6942ab4bc4f3-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.758 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.759 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.759 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.760 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.760 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.760 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.761 DEBUG:boto:Data: 2023-10-18T20:43:07.761 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.762 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.762 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.762 DEBUG:boto:Params: {} 2023-10-18T20:43:07.763 DEBUG:boto:Token: None 2023-10-18T20:43:07.763 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GaDpxFZkqMLpx6k9L7N8aaO/2HY= 2023-10-18T20:43:07.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GaDpxFZkqMLpx6k9L7N8aaO/2HY=', 'Content-Length': '0'} 2023-10-18T20:43:07.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033ed517ff34ec78f-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.767 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.767 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.768 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.768 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.768 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.769 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.769 DEBUG:boto:Data: 2023-10-18T20:43:07.769 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.770 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.771 DEBUG:boto:Params: {} 2023-10-18T20:43:07.771 DEBUG:boto:Token: None 2023-10-18T20:43:07.772 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GaDpxFZkqMLpx6k9L7N8aaO/2HY= 2023-10-18T20:43:07.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GaDpxFZkqMLpx6k9L7N8aaO/2HY=', 'Content-Length': '0'} 2023-10-18T20:43:07.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029a8c97969e417a6-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-18T20:43:07.776 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.776 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.777 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.777 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.777 DEBUG:boto:Method: GET 2023-10-18T20:43:07.778 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.778 DEBUG:boto:Data: 2023-10-18T20:43:07.779 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.780 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.780 DEBUG:boto:Params: {} 2023-10-18T20:43:07.781 DEBUG:boto:Token: None 2023-10-18T20:43:07.781 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5FDNYUuJAkDlQ/qdCPxKt54XJ0M= 2023-10-18T20:43:07.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5FDNYUuJAkDlQ/qdCPxKt54XJ0M=', 'Content-Length': '0'} 2023-10-18T20:43:07.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cce2cbd347df46a-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.785 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.785 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.786 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.786 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.787 DEBUG:boto:Method: GET 2023-10-18T20:43:07.787 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.787 DEBUG:boto:Data: 2023-10-18T20:43:07.788 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.788 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.788 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.789 DEBUG:boto:Params: {} 2023-10-18T20:43:07.789 DEBUG:boto:Token: None 2023-10-18T20:43:07.790 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:43:07.790 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5FDNYUuJAkDlQ/qdCPxKt54XJ0M= 2023-10-18T20:43:07.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5FDNYUuJAkDlQ/qdCPxKt54XJ0M=', 'Content-Length': '0'} 2023-10-18T20:43:07.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c66d2e2b3ccf80-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.793 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.794 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.794 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.794 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.795 DEBUG:boto:Method: GET 2023-10-18T20:43:07.795 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.796 DEBUG:boto:Data: 2023-10-18T20:43:07.796 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.796 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.797 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.797 DEBUG:boto:Params: {} 2023-10-18T20:43:07.798 DEBUG:boto:Token: None 2023-10-18T20:43:07.798 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084e729c5353435cf-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.802 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.802 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.803 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.803 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.804 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.804 DEBUG:boto:Method: GET 2023-10-18T20:43:07.805 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.805 DEBUG:boto:Data: 2023-10-18T20:43:07.805 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.806 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.806 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.806 DEBUG:boto:Params: {} 2023-10-18T20:43:07.807 DEBUG:boto:Token: None 2023-10-18T20:43:07.807 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.808 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cbd22f1d78eb18d-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.813 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.813 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.814 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.814 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.815 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:43:07.815 DEBUG:boto:Data: 2023-10-18T20:43:07.815 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.816 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.816 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.817 DEBUG:boto:Params: {} 2023-10-18T20:43:07.817 DEBUG:boto:Token: None 2023-10-18T20:43:07.817 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62 2023-10-18T20:43:07.818 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/mp2JC1FqyKXSfM5uNtXSwQ2Xo= 2023-10-18T20:43:07.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/mp2JC1FqyKXSfM5uNtXSwQ2Xo=', 'Content-Length': '0'} 2023-10-18T20:43:07.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e02660d06ead93f-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.821 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.822 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.822 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.823 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:43:07.823 DEBUG:boto:Data: 2023-10-18T20:43:07.823 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.824 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.824 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.825 DEBUG:boto:Params: {} 2023-10-18T20:43:07.825 DEBUG:boto:Token: None 2023-10-18T20:43:07.825 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62 2023-10-18T20:43:07.826 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/mp2JC1FqyKXSfM5uNtXSwQ2Xo= 2023-10-18T20:43:07.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/mp2JC1FqyKXSfM5uNtXSwQ2Xo=', 'Content-Length': '0'} 2023-10-18T20:43:07.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5757afb91ed6fb-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-18T20:43:07.829 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.829 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.830 DEBUG:boto:Method: GET 2023-10-18T20:43:07.830 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:43:07.831 DEBUG:boto:Data: 2023-10-18T20:43:07.831 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.831 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.832 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.832 DEBUG:boto:Params: {} 2023-10-18T20:43:07.833 DEBUG:boto:Token: None 2023-10-18T20:43:07.833 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62 2023-10-18T20:43:07.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ilVK43B/Mnwv8KquY/4F3lQoWOo= 2023-10-18T20:43:07.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ilVK43B/Mnwv8KquY/4F3lQoWOo=', 'Content-Length': '0'} 2023-10-18T20:43:07.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff36649c9b4d868-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.837 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:43:07.837 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:43:07.837 DEBUG:boto:Method: GET 2023-10-18T20:43:07.838 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:43:07.838 DEBUG:boto:Data: 2023-10-18T20:43:07.839 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.839 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.839 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.840 DEBUG:boto:Params: {} 2023-10-18T20:43:07.840 DEBUG:boto:Token: None 2023-10-18T20:43:07.841 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj62 2023-10-18T20:43:07.841 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ilVK43B/Mnwv8KquY/4F3lQoWOo= 2023-10-18T20:43:07.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ilVK43B/Mnwv8KquY/4F3lQoWOo=', 'Content-Length': '0'} 2023-10-18T20:43:07.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011c289ef2ed17ff3-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.886 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.887 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.887 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.888 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.888 DEBUG:boto:Method: GET 2023-10-18T20:43:07.889 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.889 DEBUG:boto:Data: 2023-10-18T20:43:07.889 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.890 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.890 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.891 DEBUG:boto:Params: {} 2023-10-18T20:43:07.891 DEBUG:boto:Token: None 2023-10-18T20:43:07.892 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.892 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f55ecf9071a8e6a-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.896 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.896 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.897 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.897 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.898 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.898 DEBUG:boto:Method: GET 2023-10-18T20:43:07.898 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:43:07.899 DEBUG:boto:Data: 2023-10-18T20:43:07.899 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.900 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.901 DEBUG:boto:Params: {} 2023-10-18T20:43:07.901 DEBUG:boto:Token: None 2023-10-18T20:43:07.902 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.902 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f263c8c4a0986110-006530435b-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.907 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:07.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:43:07.908 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:07.908 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:07.909 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.909 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.910 DEBUG:boto:Method: GET 2023-10-18T20:43:07.910 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.911 DEBUG:boto:Data: 2023-10-18T20:43:07.911 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.911 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.912 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.912 DEBUG:boto:Params: {} 2023-10-18T20:43:07.913 DEBUG:boto:Token: None 2023-10-18T20:43:07.913 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.914 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AvKEDccMqeTHzV1BFYftKecbiZM= 2023-10-18T20:43:07.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AvKEDccMqeTHzV1BFYftKecbiZM=', 'Content-Length': '0'} 2023-10-18T20:43:07.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080cded5f1c008a1f-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.917 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:07.917 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:07.918 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.918 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.919 DEBUG:boto:Method: GET 2023-10-18T20:43:07.919 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.919 DEBUG:boto:Data: 2023-10-18T20:43:07.920 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.920 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.920 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.921 DEBUG:boto:Params: {} 2023-10-18T20:43:07.921 DEBUG:boto:Token: None 2023-10-18T20:43:07.922 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AvKEDccMqeTHzV1BFYftKecbiZM= 2023-10-18T20:43:07.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AvKEDccMqeTHzV1BFYftKecbiZM=', 'Content-Length': '0'} 2023-10-18T20:43:07.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd7fbfcc661bece-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.925 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:07.926 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:07.926 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.927 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.927 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.927 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.928 DEBUG:boto:Data: 2023-10-18T20:43:07.928 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.928 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.929 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.929 DEBUG:boto:Params: {} 2023-10-18T20:43:07.930 DEBUG:boto:Token: None 2023-10-18T20:43:07.930 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.930 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SZXqfQdx03Fcd4ik49DAqKw44Jk= 2023-10-18T20:43:07.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SZXqfQdx03Fcd4ik49DAqKw44Jk=', 'Content-Length': '0'} 2023-10-18T20:43:07.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26321c8aaf517ce-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.933 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:07.934 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:07.934 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.935 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.935 DEBUG:boto:Method: HEAD 2023-10-18T20:43:07.935 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.936 DEBUG:boto:Data: 2023-10-18T20:43:07.936 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.937 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.937 DEBUG:boto:Params: {} 2023-10-18T20:43:07.938 DEBUG:boto:Token: None 2023-10-18T20:43:07.938 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SZXqfQdx03Fcd4ik49DAqKw44Jk= 2023-10-18T20:43:07.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SZXqfQdx03Fcd4ik49DAqKw44Jk=', 'Content-Length': '0'} 2023-10-18T20:43:07.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2287b90b6f273cb-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:43:07.943 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:07.943 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:07.943 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.944 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.944 DEBUG:boto:Method: GET 2023-10-18T20:43:07.945 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.945 DEBUG:boto:Data: 2023-10-18T20:43:07.945 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.946 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.946 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.947 DEBUG:boto:Params: {} 2023-10-18T20:43:07.947 DEBUG:boto:Token: None 2023-10-18T20:43:07.948 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AvKEDccMqeTHzV1BFYftKecbiZM= 2023-10-18T20:43:07.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AvKEDccMqeTHzV1BFYftKecbiZM=', 'Content-Length': '0'} 2023-10-18T20:43:07.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000622a833a217249b3-006530435b-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.951 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:07.952 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:07.952 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.953 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.953 DEBUG:boto:Method: GET 2023-10-18T20:43:07.953 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.954 DEBUG:boto:Data: 2023-10-18T20:43:07.954 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.955 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:07.955 DEBUG:boto:Port: 8000 2023-10-18T20:43:07.955 DEBUG:boto:Params: {} 2023-10-18T20:43:07.956 DEBUG:boto:Token: None 2023-10-18T20:43:07.956 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:43:07.957 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AvKEDccMqeTHzV1BFYftKecbiZM= 2023-10-18T20:43:07.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AvKEDccMqeTHzV1BFYftKecbiZM=', 'Content-Length': '0'} 2023-10-18T20:43:07.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003769ea0d37634c8a-006530435b-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:07.960 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:07.961 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:07.961 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:43:07.962 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:43:07.962 DEBUG:boto:Method: GET 2023-10-18T20:43:07.962 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:43:07.963 DEBUG:boto:Data: 2023-10-18T20:43:07.963 DEBUG:boto:Headers: {} 2023-10-18T20:43:07.964 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:07.964 DEBUG:boto:Port: 8001 2023-10-18T20:43:07.964 DEBUG:boto:Params: {} 2023-10-18T20:43:07.965 DEBUG:boto:Token: None 2023-10-18T20:43:07.965 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:07 GMT /bofrnf-59/ 2023-10-18T20:43:07.966 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus= 2023-10-18T20:43:07.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ChW6DPrWpU3+PQpRsMXiHBvuSus=', 'Content-Length': '0'} 2023-10-18T20:43:07.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d217fded01ed576-006530435b-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.010 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.011 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.012 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.012 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:43:08.013 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:43:08.013 DEBUG:boto:Method: GET 2023-10-18T20:43:08.014 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:43:08.014 DEBUG:boto:Data: 2023-10-18T20:43:08.015 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.015 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.015 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.016 DEBUG:boto:Params: {} 2023-10-18T20:43:08.016 DEBUG:boto:Token: None 2023-10-18T20:43:08.017 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.017 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1af75e8a61e27d-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.023 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.023 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:08.024 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:08.024 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.025 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:43:08.025 DEBUG:boto:Data: 2023-10-18T20:43:08.026 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.026 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.026 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.027 DEBUG:boto:Params: {} 2023-10-18T20:43:08.027 DEBUG:boto:Token: None 2023-10-18T20:43:08.028 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj68 2023-10-18T20:43:08.028 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ORxX7+LsJWJby2hl//qOllkM3BE= 2023-10-18T20:43:08.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ORxX7+LsJWJby2hl//qOllkM3BE=', 'Content-Length': '0'} 2023-10-18T20:43:08.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b9eb7e6f3fd1e39-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.031 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:08.032 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:08.032 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.033 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:43:08.033 DEBUG:boto:Data: 2023-10-18T20:43:08.033 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.034 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.034 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.035 DEBUG:boto:Params: {} 2023-10-18T20:43:08.035 DEBUG:boto:Token: None 2023-10-18T20:43:08.036 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj68 2023-10-18T20:43:08.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ORxX7+LsJWJby2hl//qOllkM3BE= 2023-10-18T20:43:08.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ORxX7+LsJWJby2hl//qOllkM3BE=', 'Content-Length': '0'} 2023-10-18T20:43:08.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001de8501e862724a4-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:43:08.039 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:08.040 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:08.040 DEBUG:boto:Method: GET 2023-10-18T20:43:08.041 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:43:08.041 DEBUG:boto:Data: 2023-10-18T20:43:08.042 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.043 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.043 DEBUG:boto:Params: {} 2023-10-18T20:43:08.044 DEBUG:boto:Token: None 2023-10-18T20:43:08.044 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj68 2023-10-18T20:43:08.045 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F2N3Y87UJdlMUKkSQQtEQPu3Ci4= 2023-10-18T20:43:08.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F2N3Y87UJdlMUKkSQQtEQPu3Ci4=', 'Content-Length': '0'} 2023-10-18T20:43:08.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d1c0f3dc5e9816e-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.048 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:43:08.048 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:43:08.049 DEBUG:boto:Method: GET 2023-10-18T20:43:08.049 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:43:08.049 DEBUG:boto:Data: 2023-10-18T20:43:08.050 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.051 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.051 DEBUG:boto:Params: {} 2023-10-18T20:43:08.052 DEBUG:boto:Token: None 2023-10-18T20:43:08.052 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj68 2023-10-18T20:43:08.053 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F2N3Y87UJdlMUKkSQQtEQPu3Ci4= 2023-10-18T20:43:08.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F2N3Y87UJdlMUKkSQQtEQPu3Ci4=', 'Content-Length': '0'} 2023-10-18T20:43:08.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fa829ddb409164b-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.098 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.099 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.099 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:43:08.100 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:43:08.100 DEBUG:boto:Method: GET 2023-10-18T20:43:08.101 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:43:08.101 DEBUG:boto:Data: 2023-10-18T20:43:08.102 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.102 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.103 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.103 DEBUG:boto:Params: {} 2023-10-18T20:43:08.104 DEBUG:boto:Token: None 2023-10-18T20:43:08.104 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.105 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019c3d4bc28785011-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.108 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.109 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.109 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.110 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:43:08.110 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:43:08.111 DEBUG:boto:Method: GET 2023-10-18T20:43:08.111 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:43:08.112 DEBUG:boto:Data: 2023-10-18T20:43:08.112 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.112 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.113 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.113 DEBUG:boto:Params: {} 2023-10-18T20:43:08.114 DEBUG:boto:Token: None 2023-10-18T20:43:08.114 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000163500718a8c8dbc-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.120 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:43:08.121 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.121 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.122 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.122 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.123 DEBUG:boto:Method: GET 2023-10-18T20:43:08.123 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.123 DEBUG:boto:Data: 2023-10-18T20:43:08.124 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.124 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.125 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.125 DEBUG:boto:Params: {} 2023-10-18T20:43:08.125 DEBUG:boto:Token: None 2023-10-18T20:43:08.126 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.126 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yKW6rHXiNApBKabI7asHQWshIxw= 2023-10-18T20:43:08.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yKW6rHXiNApBKabI7asHQWshIxw=', 'Content-Length': '0'} 2023-10-18T20:43:08.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021854c2ee684769e-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.174 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.175 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.175 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.176 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.176 DEBUG:boto:Method: GET 2023-10-18T20:43:08.176 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.177 DEBUG:boto:Data: 2023-10-18T20:43:08.177 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.178 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.179 DEBUG:boto:Params: {} 2023-10-18T20:43:08.179 DEBUG:boto:Token: None 2023-10-18T20:43:08.180 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yKW6rHXiNApBKabI7asHQWshIxw= 2023-10-18T20:43:08.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yKW6rHXiNApBKabI7asHQWshIxw=', 'Content-Length': '0'} 2023-10-18T20:43:08.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009053b7c20f3dc3a9-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.183 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.184 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.184 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.184 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.185 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.185 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.186 DEBUG:boto:Data: 2023-10-18T20:43:08.186 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.187 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.187 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.187 DEBUG:boto:Params: {} 2023-10-18T20:43:08.188 DEBUG:boto:Token: None 2023-10-18T20:43:08.188 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.189 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+l3R8YSTAZZhEAYR+HPI6hp8inU= 2023-10-18T20:43:08.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+l3R8YSTAZZhEAYR+HPI6hp8inU=', 'Content-Length': '0'} 2023-10-18T20:43:08.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e5dc46f42a13e1f-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.192 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.192 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.193 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.193 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.194 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.194 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.194 DEBUG:boto:Data: 2023-10-18T20:43:08.195 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.195 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.196 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.196 DEBUG:boto:Params: {} 2023-10-18T20:43:08.196 DEBUG:boto:Token: None 2023-10-18T20:43:08.197 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.197 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+l3R8YSTAZZhEAYR+HPI6hp8inU= 2023-10-18T20:43:08.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+l3R8YSTAZZhEAYR+HPI6hp8inU=', 'Content-Length': '0'} 2023-10-18T20:43:08.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006427c5aacda711c2-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:43:08.201 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.201 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.202 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.202 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.202 DEBUG:boto:Method: GET 2023-10-18T20:43:08.203 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.203 DEBUG:boto:Data: 2023-10-18T20:43:08.204 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.204 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.204 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.205 DEBUG:boto:Params: {} 2023-10-18T20:43:08.205 DEBUG:boto:Token: None 2023-10-18T20:43:08.206 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.206 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yKW6rHXiNApBKabI7asHQWshIxw= 2023-10-18T20:43:08.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yKW6rHXiNApBKabI7asHQWshIxw=', 'Content-Length': '0'} 2023-10-18T20:43:08.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f508111075142dbe-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.209 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.210 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.210 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.211 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.211 DEBUG:boto:Method: GET 2023-10-18T20:43:08.211 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.212 DEBUG:boto:Data: 2023-10-18T20:43:08.212 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.213 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.213 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.213 DEBUG:boto:Params: {} 2023-10-18T20:43:08.214 DEBUG:boto:Token: None 2023-10-18T20:43:08.214 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:43:08.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yKW6rHXiNApBKabI7asHQWshIxw= 2023-10-18T20:43:08.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yKW6rHXiNApBKabI7asHQWshIxw=', 'Content-Length': '0'} 2023-10-18T20:43:08.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e3fc63760de1338-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.218 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.219 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.219 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.219 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.220 DEBUG:boto:Method: GET 2023-10-18T20:43:08.220 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.221 DEBUG:boto:Data: 2023-10-18T20:43:08.221 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.221 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.222 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.222 DEBUG:boto:Params: {} 2023-10-18T20:43:08.223 DEBUG:boto:Token: None 2023-10-18T20:43:08.223 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.224 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b4202cc68c2439-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.227 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.228 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.228 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.229 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.229 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.229 DEBUG:boto:Method: GET 2023-10-18T20:43:08.230 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.230 DEBUG:boto:Data: 2023-10-18T20:43:08.231 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.231 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.232 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.232 DEBUG:boto:Params: {} 2023-10-18T20:43:08.233 DEBUG:boto:Token: None 2023-10-18T20:43:08.233 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.234 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d32b1337154c1bfd-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.239 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.239 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.240 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.240 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.241 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:43:08.241 DEBUG:boto:Data: 2023-10-18T20:43:08.241 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.242 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.243 DEBUG:boto:Params: {} 2023-10-18T20:43:08.243 DEBUG:boto:Token: None 2023-10-18T20:43:08.243 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69 2023-10-18T20:43:08.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z4cXQXucw4ObCiPs1j3Av4VpSPw= 2023-10-18T20:43:08.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z4cXQXucw4ObCiPs1j3Av4VpSPw=', 'Content-Length': '0'} 2023-10-18T20:43:08.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002efc3880a5d61f9-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.247 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.248 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.248 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.248 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:43:08.249 DEBUG:boto:Data: 2023-10-18T20:43:08.249 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.250 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.250 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.250 DEBUG:boto:Params: {} 2023-10-18T20:43:08.251 DEBUG:boto:Token: None 2023-10-18T20:43:08.251 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69 2023-10-18T20:43:08.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z4cXQXucw4ObCiPs1j3Av4VpSPw= 2023-10-18T20:43:08.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z4cXQXucw4ObCiPs1j3Av4VpSPw=', 'Content-Length': '0'} 2023-10-18T20:43:08.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c90ca85397ef52b-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:43:08.255 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.256 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.256 DEBUG:boto:Method: GET 2023-10-18T20:43:08.256 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:43:08.257 DEBUG:boto:Data: 2023-10-18T20:43:08.257 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.258 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.258 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.259 DEBUG:boto:Params: {} 2023-10-18T20:43:08.259 DEBUG:boto:Token: None 2023-10-18T20:43:08.259 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69 2023-10-18T20:43:08.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yTwz8mGoiyWgYvfFidnJAUnlYvk= 2023-10-18T20:43:08.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yTwz8mGoiyWgYvfFidnJAUnlYvk=', 'Content-Length': '0'} 2023-10-18T20:43:08.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000361f3dd4a1ac57bc-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.306 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:43:08.307 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:43:08.307 DEBUG:boto:Method: GET 2023-10-18T20:43:08.308 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:43:08.308 DEBUG:boto:Data: 2023-10-18T20:43:08.309 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.309 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.309 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.310 DEBUG:boto:Params: {} 2023-10-18T20:43:08.310 DEBUG:boto:Token: None 2023-10-18T20:43:08.311 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj69 2023-10-18T20:43:08.311 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yTwz8mGoiyWgYvfFidnJAUnlYvk= 2023-10-18T20:43:08.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yTwz8mGoiyWgYvfFidnJAUnlYvk=', 'Content-Length': '0'} 2023-10-18T20:43:08.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c93dfc32e41810a-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.315 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.315 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.315 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.316 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.316 DEBUG:boto:Method: GET 2023-10-18T20:43:08.317 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.317 DEBUG:boto:Data: 2023-10-18T20:43:08.318 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.318 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.319 DEBUG:boto:Params: {} 2023-10-18T20:43:08.319 DEBUG:boto:Token: None 2023-10-18T20:43:08.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ad7565a93734a5-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.324 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.324 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.325 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.325 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.326 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.326 DEBUG:boto:Method: GET 2023-10-18T20:43:08.327 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:43:08.327 DEBUG:boto:Data: 2023-10-18T20:43:08.328 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.328 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.329 DEBUG:boto:Params: {} 2023-10-18T20:43:08.329 DEBUG:boto:Token: None 2023-10-18T20:43:08.330 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001252f6a3c0318bda-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.335 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:43:08.336 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.337 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.337 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.338 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.338 DEBUG:boto:Method: GET 2023-10-18T20:43:08.339 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.339 DEBUG:boto:Data: 2023-10-18T20:43:08.339 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.340 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.340 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.341 DEBUG:boto:Params: {} 2023-10-18T20:43:08.341 DEBUG:boto:Token: None 2023-10-18T20:43:08.342 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.342 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JFfOuwds4HKse49B5kG/jYd4Kz4= 2023-10-18T20:43:08.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JFfOuwds4HKse49B5kG/jYd4Kz4=', 'Content-Length': '0'} 2023-10-18T20:43:08.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acda9505d173e43a-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.345 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.346 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.346 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.347 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.347 DEBUG:boto:Method: GET 2023-10-18T20:43:08.347 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.348 DEBUG:boto:Data: 2023-10-18T20:43:08.348 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.349 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.349 DEBUG:boto:Params: {} 2023-10-18T20:43:08.350 DEBUG:boto:Token: None 2023-10-18T20:43:08.350 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JFfOuwds4HKse49B5kG/jYd4Kz4= 2023-10-18T20:43:08.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JFfOuwds4HKse49B5kG/jYd4Kz4=', 'Content-Length': '0'} 2023-10-18T20:43:08.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b48710d3ccaa1fa5-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.354 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.354 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.355 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.355 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.355 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.356 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.356 DEBUG:boto:Data: 2023-10-18T20:43:08.357 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.357 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.357 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.358 DEBUG:boto:Params: {} 2023-10-18T20:43:08.358 DEBUG:boto:Token: None 2023-10-18T20:43:08.359 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.359 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rQLKppT26kkG2cx3f4lu2uCIbSo= 2023-10-18T20:43:08.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rQLKppT26kkG2cx3f4lu2uCIbSo=', 'Content-Length': '0'} 2023-10-18T20:43:08.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7b8fd3d45137c83-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.362 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.362 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.363 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.363 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.364 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.364 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.364 DEBUG:boto:Data: 2023-10-18T20:43:08.365 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.365 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.366 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.366 DEBUG:boto:Params: {} 2023-10-18T20:43:08.366 DEBUG:boto:Token: None 2023-10-18T20:43:08.367 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.367 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rQLKppT26kkG2cx3f4lu2uCIbSo= 2023-10-18T20:43:08.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rQLKppT26kkG2cx3f4lu2uCIbSo=', 'Content-Length': '0'} 2023-10-18T20:43:08.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c85263d310e931f5-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:43:08.371 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.371 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.372 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.372 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.372 DEBUG:boto:Method: GET 2023-10-18T20:43:08.373 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.373 DEBUG:boto:Data: 2023-10-18T20:43:08.374 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.375 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.375 DEBUG:boto:Params: {} 2023-10-18T20:43:08.376 DEBUG:boto:Token: None 2023-10-18T20:43:08.376 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.376 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JFfOuwds4HKse49B5kG/jYd4Kz4= 2023-10-18T20:43:08.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JFfOuwds4HKse49B5kG/jYd4Kz4=', 'Content-Length': '0'} 2023-10-18T20:43:08.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaf1e93eb44c2ab1-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.380 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.380 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.380 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.381 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.381 DEBUG:boto:Method: GET 2023-10-18T20:43:08.382 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.382 DEBUG:boto:Data: 2023-10-18T20:43:08.383 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.383 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.383 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.384 DEBUG:boto:Params: {} 2023-10-18T20:43:08.384 DEBUG:boto:Token: None 2023-10-18T20:43:08.385 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:43:08.385 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JFfOuwds4HKse49B5kG/jYd4Kz4= 2023-10-18T20:43:08.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JFfOuwds4HKse49B5kG/jYd4Kz4=', 'Content-Length': '0'} 2023-10-18T20:43:08.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fb2a416139d615b-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.388 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.389 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.389 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.389 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.390 DEBUG:boto:Method: GET 2023-10-18T20:43:08.390 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.391 DEBUG:boto:Data: 2023-10-18T20:43:08.391 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.391 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.392 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.392 DEBUG:boto:Params: {} 2023-10-18T20:43:08.393 DEBUG:boto:Token: None 2023-10-18T20:43:08.393 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.394 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9c7d7f7d74acf61-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.438 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.439 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.439 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.440 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.440 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.441 DEBUG:boto:Method: GET 2023-10-18T20:43:08.441 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.442 DEBUG:boto:Data: 2023-10-18T20:43:08.442 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.442 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.443 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.443 DEBUG:boto:Params: {} 2023-10-18T20:43:08.444 DEBUG:boto:Token: None 2023-10-18T20:43:08.444 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.445 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea626aae1d4234c-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.450 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.451 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.451 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.452 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.452 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:43:08.452 DEBUG:boto:Data: 2023-10-18T20:43:08.453 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.454 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.454 DEBUG:boto:Params: {} 2023-10-18T20:43:08.455 DEBUG:boto:Token: None 2023-10-18T20:43:08.455 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76 2023-10-18T20:43:08.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YAacSp9mwx7HG/SGsy5kTIEc2+s= 2023-10-18T20:43:08.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YAacSp9mwx7HG/SGsy5kTIEc2+s=', 'Content-Length': '0'} 2023-10-18T20:43:08.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000855702ed91a8b577-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.458 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.459 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.459 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.460 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:43:08.460 DEBUG:boto:Data: 2023-10-18T20:43:08.460 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.461 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.461 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.462 DEBUG:boto:Params: {} 2023-10-18T20:43:08.462 DEBUG:boto:Token: None 2023-10-18T20:43:08.462 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76 2023-10-18T20:43:08.463 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YAacSp9mwx7HG/SGsy5kTIEc2+s= 2023-10-18T20:43:08.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YAacSp9mwx7HG/SGsy5kTIEc2+s=', 'Content-Length': '0'} 2023-10-18T20:43:08.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000608995964f52aa9f-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:43:08.466 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.467 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.467 DEBUG:boto:Method: GET 2023-10-18T20:43:08.468 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:43:08.468 DEBUG:boto:Data: 2023-10-18T20:43:08.468 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.469 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.469 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.470 DEBUG:boto:Params: {} 2023-10-18T20:43:08.470 DEBUG:boto:Token: None 2023-10-18T20:43:08.471 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76 2023-10-18T20:43:08.471 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Aq2KCpGeKTi2ZF9sibiCZMKIpPw= 2023-10-18T20:43:08.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Aq2KCpGeKTi2ZF9sibiCZMKIpPw=', 'Content-Length': '0'} 2023-10-18T20:43:08.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093f9b2cd0caef995-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.474 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:43:08.475 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:43:08.475 DEBUG:boto:Method: GET 2023-10-18T20:43:08.476 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:43:08.476 DEBUG:boto:Data: 2023-10-18T20:43:08.476 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.477 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.477 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.478 DEBUG:boto:Params: {} 2023-10-18T20:43:08.478 DEBUG:boto:Token: None 2023-10-18T20:43:08.478 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj76 2023-10-18T20:43:08.479 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Aq2KCpGeKTi2ZF9sibiCZMKIpPw= 2023-10-18T20:43:08.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Aq2KCpGeKTi2ZF9sibiCZMKIpPw=', 'Content-Length': '0'} 2023-10-18T20:43:08.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009502541643b63664-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.482 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.483 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.483 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.483 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.484 DEBUG:boto:Method: GET 2023-10-18T20:43:08.484 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.485 DEBUG:boto:Data: 2023-10-18T20:43:08.485 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.485 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.486 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.486 DEBUG:boto:Params: {} 2023-10-18T20:43:08.487 DEBUG:boto:Token: None 2023-10-18T20:43:08.487 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.488 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c37d0b797f296e9-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.491 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.491 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.492 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.492 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.493 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.493 DEBUG:boto:Method: GET 2023-10-18T20:43:08.493 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:43:08.494 DEBUG:boto:Data: 2023-10-18T20:43:08.494 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.495 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.495 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.495 DEBUG:boto:Params: {} 2023-10-18T20:43:08.496 DEBUG:boto:Token: None 2023-10-18T20:43:08.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb555b85d485daac-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.502 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:43:08.503 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.503 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.504 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.504 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.505 DEBUG:boto:Method: GET 2023-10-18T20:43:08.505 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.505 DEBUG:boto:Data: 2023-10-18T20:43:08.506 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.506 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.507 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.507 DEBUG:boto:Params: {} 2023-10-18T20:43:08.507 DEBUG:boto:Token: None 2023-10-18T20:43:08.508 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.508 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cXneGTPz0+VTVrtFZ4nwdNbaE9E= 2023-10-18T20:43:08.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cXneGTPz0+VTVrtFZ4nwdNbaE9E=', 'Content-Length': '0'} 2023-10-18T20:43:08.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc6415444c092df-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.511 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.512 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.512 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.513 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.513 DEBUG:boto:Method: GET 2023-10-18T20:43:08.513 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.514 DEBUG:boto:Data: 2023-10-18T20:43:08.514 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.515 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.515 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.515 DEBUG:boto:Params: {} 2023-10-18T20:43:08.516 DEBUG:boto:Token: None 2023-10-18T20:43:08.516 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.517 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cXneGTPz0+VTVrtFZ4nwdNbaE9E= 2023-10-18T20:43:08.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cXneGTPz0+VTVrtFZ4nwdNbaE9E=', 'Content-Length': '0'} 2023-10-18T20:43:08.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff69b4a46c023d2c-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.562 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.563 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.563 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.564 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.564 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.564 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.565 DEBUG:boto:Data: 2023-10-18T20:43:08.565 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.565 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.566 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.566 DEBUG:boto:Params: {} 2023-10-18T20:43:08.567 DEBUG:boto:Token: None 2023-10-18T20:43:08.567 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mpJwLi1ot5kQZTzHgikDdp2V00o= 2023-10-18T20:43:08.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mpJwLi1ot5kQZTzHgikDdp2V00o=', 'Content-Length': '0'} 2023-10-18T20:43:08.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000de83a69db0842b-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.571 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.571 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.571 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.572 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.572 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.572 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.573 DEBUG:boto:Data: 2023-10-18T20:43:08.573 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.573 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.574 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.575 DEBUG:boto:Params: {} 2023-10-18T20:43:08.575 DEBUG:boto:Token: None 2023-10-18T20:43:08.576 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mpJwLi1ot5kQZTzHgikDdp2V00o= 2023-10-18T20:43:08.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mpJwLi1ot5kQZTzHgikDdp2V00o=', 'Content-Length': '0'} 2023-10-18T20:43:08.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d60be9a3f803e173-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:43:08.580 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.580 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.581 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.581 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.581 DEBUG:boto:Method: GET 2023-10-18T20:43:08.582 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.582 DEBUG:boto:Data: 2023-10-18T20:43:08.583 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.584 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.584 DEBUG:boto:Params: {} 2023-10-18T20:43:08.585 DEBUG:boto:Token: None 2023-10-18T20:43:08.585 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cXneGTPz0+VTVrtFZ4nwdNbaE9E= 2023-10-18T20:43:08.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cXneGTPz0+VTVrtFZ4nwdNbaE9E=', 'Content-Length': '0'} 2023-10-18T20:43:08.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d8a829fcae2f862-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.589 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.589 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.590 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.590 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.591 DEBUG:boto:Method: GET 2023-10-18T20:43:08.591 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.591 DEBUG:boto:Data: 2023-10-18T20:43:08.592 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.592 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.593 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.593 DEBUG:boto:Params: {} 2023-10-18T20:43:08.594 DEBUG:boto:Token: None 2023-10-18T20:43:08.594 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:43:08.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cXneGTPz0+VTVrtFZ4nwdNbaE9E= 2023-10-18T20:43:08.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cXneGTPz0+VTVrtFZ4nwdNbaE9E=', 'Content-Length': '0'} 2023-10-18T20:43:08.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b37a1b975d86257-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.598 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.599 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.599 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.599 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.600 DEBUG:boto:Method: GET 2023-10-18T20:43:08.600 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.601 DEBUG:boto:Data: 2023-10-18T20:43:08.601 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.602 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.603 DEBUG:boto:Params: {} 2023-10-18T20:43:08.603 DEBUG:boto:Token: None 2023-10-18T20:43:08.604 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b90198ae4b9a38e6-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.608 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.609 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.609 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.610 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.610 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.611 DEBUG:boto:Method: GET 2023-10-18T20:43:08.611 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.612 DEBUG:boto:Data: 2023-10-18T20:43:08.612 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.612 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.613 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.614 DEBUG:boto:Params: {} 2023-10-18T20:43:08.614 DEBUG:boto:Token: None 2023-10-18T20:43:08.615 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.616 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008484d0574503119e-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.620 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.621 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.621 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.622 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.622 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:43:08.623 DEBUG:boto:Data: 2023-10-18T20:43:08.623 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.623 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.624 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.624 DEBUG:boto:Params: {} 2023-10-18T20:43:08.625 DEBUG:boto:Token: None 2023-10-18T20:43:08.625 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85 2023-10-18T20:43:08.625 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iP3AiSfVl1LuBrQ4EOAmQQjGmIw= 2023-10-18T20:43:08.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iP3AiSfVl1LuBrQ4EOAmQQjGmIw=', 'Content-Length': '0'} 2023-10-18T20:43:08.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096ddc7c219566a5e-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.628 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.629 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.629 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.629 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:43:08.630 DEBUG:boto:Data: 2023-10-18T20:43:08.630 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.631 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.631 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.631 DEBUG:boto:Params: {} 2023-10-18T20:43:08.632 DEBUG:boto:Token: None 2023-10-18T20:43:08.632 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85 2023-10-18T20:43:08.633 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iP3AiSfVl1LuBrQ4EOAmQQjGmIw= 2023-10-18T20:43:08.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iP3AiSfVl1LuBrQ4EOAmQQjGmIw=', 'Content-Length': '0'} 2023-10-18T20:43:08.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000812da9d5b51da842-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:43:08.636 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.636 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.637 DEBUG:boto:Method: GET 2023-10-18T20:43:08.637 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:43:08.637 DEBUG:boto:Data: 2023-10-18T20:43:08.638 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.638 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.639 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.639 DEBUG:boto:Params: {} 2023-10-18T20:43:08.640 DEBUG:boto:Token: None 2023-10-18T20:43:08.640 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85 2023-10-18T20:43:08.641 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HxDDW0RwwZJwLwjKX/3jcrtNTGg= 2023-10-18T20:43:08.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HxDDW0RwwZJwLwjKX/3jcrtNTGg=', 'Content-Length': '0'} 2023-10-18T20:43:08.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ee4e59a8fab1e55-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.686 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:43:08.687 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:43:08.687 DEBUG:boto:Method: GET 2023-10-18T20:43:08.688 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:43:08.688 DEBUG:boto:Data: 2023-10-18T20:43:08.689 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.690 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.690 DEBUG:boto:Params: {} 2023-10-18T20:43:08.691 DEBUG:boto:Token: None 2023-10-18T20:43:08.691 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj85 2023-10-18T20:43:08.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HxDDW0RwwZJwLwjKX/3jcrtNTGg= 2023-10-18T20:43:08.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HxDDW0RwwZJwLwjKX/3jcrtNTGg=', 'Content-Length': '0'} 2023-10-18T20:43:08.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a69785ffb184544-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.695 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.695 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.696 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.696 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.697 DEBUG:boto:Method: GET 2023-10-18T20:43:08.697 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.697 DEBUG:boto:Data: 2023-10-18T20:43:08.698 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.699 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.699 DEBUG:boto:Params: {} 2023-10-18T20:43:08.699 DEBUG:boto:Token: None 2023-10-18T20:43:08.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.700 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000006ca6fdb8dc3fdc-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.704 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.705 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.705 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.705 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.706 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.706 DEBUG:boto:Method: GET 2023-10-18T20:43:08.707 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:43:08.707 DEBUG:boto:Data: 2023-10-18T20:43:08.708 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.708 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.708 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.709 DEBUG:boto:Params: {} 2023-10-18T20:43:08.709 DEBUG:boto:Token: None 2023-10-18T20:43:08.710 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e567612836c06cd-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.714 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:43:08.715 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.716 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.716 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.716 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.717 DEBUG:boto:Method: GET 2023-10-18T20:43:08.717 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.718 DEBUG:boto:Data: 2023-10-18T20:43:08.718 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.719 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.719 DEBUG:boto:Params: {} 2023-10-18T20:43:08.720 DEBUG:boto:Token: None 2023-10-18T20:43:08.720 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mdBVtIHTbxCLCzjoMu5MIt6GCV8= 2023-10-18T20:43:08.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mdBVtIHTbxCLCzjoMu5MIt6GCV8=', 'Content-Length': '0'} 2023-10-18T20:43:08.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33ddfc3796784a6-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.724 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.725 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.725 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.725 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.726 DEBUG:boto:Method: GET 2023-10-18T20:43:08.726 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.727 DEBUG:boto:Data: 2023-10-18T20:43:08.727 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.728 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.728 DEBUG:boto:Params: {} 2023-10-18T20:43:08.729 DEBUG:boto:Token: None 2023-10-18T20:43:08.729 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.730 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mdBVtIHTbxCLCzjoMu5MIt6GCV8= 2023-10-18T20:43:08.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mdBVtIHTbxCLCzjoMu5MIt6GCV8=', 'Content-Length': '0'} 2023-10-18T20:43:08.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067cf3f3c0f45ae11-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.733 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.733 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.734 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.734 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.734 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.735 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.735 DEBUG:boto:Data: 2023-10-18T20:43:08.736 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.736 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.736 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.737 DEBUG:boto:Params: {} 2023-10-18T20:43:08.737 DEBUG:boto:Token: None 2023-10-18T20:43:08.737 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.738 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O4O9TBfylvhXVza7LD1joZz4MhU= 2023-10-18T20:43:08.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O4O9TBfylvhXVza7LD1joZz4MhU=', 'Content-Length': '0'} 2023-10-18T20:43:08.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c6f11a98e27437f-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.741 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.741 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.742 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.742 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.742 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.743 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.743 DEBUG:boto:Data: 2023-10-18T20:43:08.744 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.744 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.744 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.745 DEBUG:boto:Params: {} 2023-10-18T20:43:08.745 DEBUG:boto:Token: None 2023-10-18T20:43:08.746 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.746 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O4O9TBfylvhXVza7LD1joZz4MhU= 2023-10-18T20:43:08.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O4O9TBfylvhXVza7LD1joZz4MhU=', 'Content-Length': '0'} 2023-10-18T20:43:08.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086fbb18c509e4c77-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:43:08.749 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.750 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.750 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.751 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.751 DEBUG:boto:Method: GET 2023-10-18T20:43:08.751 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.752 DEBUG:boto:Data: 2023-10-18T20:43:08.752 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.753 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.754 DEBUG:boto:Params: {} 2023-10-18T20:43:08.754 DEBUG:boto:Token: None 2023-10-18T20:43:08.754 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mdBVtIHTbxCLCzjoMu5MIt6GCV8= 2023-10-18T20:43:08.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mdBVtIHTbxCLCzjoMu5MIt6GCV8=', 'Content-Length': '0'} 2023-10-18T20:43:08.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7fcccf7abf943aa-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.758 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.758 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.759 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.759 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.760 DEBUG:boto:Method: GET 2023-10-18T20:43:08.760 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.760 DEBUG:boto:Data: 2023-10-18T20:43:08.761 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.762 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.762 DEBUG:boto:Params: {} 2023-10-18T20:43:08.763 DEBUG:boto:Token: None 2023-10-18T20:43:08.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:43:08.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mdBVtIHTbxCLCzjoMu5MIt6GCV8= 2023-10-18T20:43:08.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mdBVtIHTbxCLCzjoMu5MIt6GCV8=', 'Content-Length': '0'} 2023-10-18T20:43:08.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa48f16c8ca30ab8-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.767 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.767 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.768 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.768 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.769 DEBUG:boto:Method: GET 2023-10-18T20:43:08.769 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.770 DEBUG:boto:Data: 2023-10-18T20:43:08.770 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.771 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.771 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.771 DEBUG:boto:Params: {} 2023-10-18T20:43:08.772 DEBUG:boto:Token: None 2023-10-18T20:43:08.772 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.773 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092fe17ea83b24b9a-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.776 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.777 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.777 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.778 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.778 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.779 DEBUG:boto:Method: GET 2023-10-18T20:43:08.779 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.779 DEBUG:boto:Data: 2023-10-18T20:43:08.780 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.780 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.780 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.781 DEBUG:boto:Params: {} 2023-10-18T20:43:08.781 DEBUG:boto:Token: None 2023-10-18T20:43:08.782 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006995fec6c6bced6c-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.786 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.787 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.787 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.788 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.788 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:43:08.788 DEBUG:boto:Data: 2023-10-18T20:43:08.789 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.790 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.790 DEBUG:boto:Params: {} 2023-10-18T20:43:08.790 DEBUG:boto:Token: None 2023-10-18T20:43:08.791 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87 2023-10-18T20:43:08.791 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Ci3TIp+9qdE7/PNtcwvBc1mETo= 2023-10-18T20:43:08.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Ci3TIp+9qdE7/PNtcwvBc1mETo=', 'Content-Length': '0'} 2023-10-18T20:43:08.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b021a87023af99-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.794 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.794 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.795 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.795 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:43:08.795 DEBUG:boto:Data: 2023-10-18T20:43:08.796 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.796 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.797 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.797 DEBUG:boto:Params: {} 2023-10-18T20:43:08.797 DEBUG:boto:Token: None 2023-10-18T20:43:08.798 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87 2023-10-18T20:43:08.798 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6Ci3TIp+9qdE7/PNtcwvBc1mETo= 2023-10-18T20:43:08.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Ci3TIp+9qdE7/PNtcwvBc1mETo=', 'Content-Length': '0'} 2023-10-18T20:43:08.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8bc33a9b4dabe6a-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:43:08.801 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.802 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.802 DEBUG:boto:Method: GET 2023-10-18T20:43:08.803 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:43:08.803 DEBUG:boto:Data: 2023-10-18T20:43:08.804 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.804 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.804 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.805 DEBUG:boto:Params: {} 2023-10-18T20:43:08.805 DEBUG:boto:Token: None 2023-10-18T20:43:08.806 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87 2023-10-18T20:43:08.806 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UXacpwjLm1d1bR9DTUKNshFOMhs= 2023-10-18T20:43:08.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UXacpwjLm1d1bR9DTUKNshFOMhs=', 'Content-Length': '0'} 2023-10-18T20:43:08.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2a5b4e3cf7a6ee0-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.854 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:43:08.855 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:43:08.855 DEBUG:boto:Method: GET 2023-10-18T20:43:08.856 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:43:08.856 DEBUG:boto:Data: 2023-10-18T20:43:08.857 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.857 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.857 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.858 DEBUG:boto:Params: {} 2023-10-18T20:43:08.858 DEBUG:boto:Token: None 2023-10-18T20:43:08.859 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj87 2023-10-18T20:43:08.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UXacpwjLm1d1bR9DTUKNshFOMhs= 2023-10-18T20:43:08.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UXacpwjLm1d1bR9DTUKNshFOMhs=', 'Content-Length': '0'} 2023-10-18T20:43:08.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000071d781aa2fa901b-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.862 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.863 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.863 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.864 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.864 DEBUG:boto:Method: GET 2023-10-18T20:43:08.864 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.865 DEBUG:boto:Data: 2023-10-18T20:43:08.865 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.866 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.866 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.867 DEBUG:boto:Params: {} 2023-10-18T20:43:08.867 DEBUG:boto:Token: None 2023-10-18T20:43:08.867 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.868 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000021c91ea3aed2f74-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.871 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.872 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.872 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.872 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.873 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.873 DEBUG:boto:Method: GET 2023-10-18T20:43:08.874 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:43:08.874 DEBUG:boto:Data: 2023-10-18T20:43:08.875 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.875 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.876 DEBUG:boto:Params: {} 2023-10-18T20:43:08.876 DEBUG:boto:Token: None 2023-10-18T20:43:08.876 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.877 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5061a11531cf25f-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.881 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:43:08.883 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:08.883 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:08.884 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.884 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.884 DEBUG:boto:Method: GET 2023-10-18T20:43:08.885 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.885 DEBUG:boto:Data: 2023-10-18T20:43:08.886 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.886 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.886 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.887 DEBUG:boto:Params: {} 2023-10-18T20:43:08.887 DEBUG:boto:Token: None 2023-10-18T20:43:08.888 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.888 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IGyTmYRNLt/LD8n+BSkmT7ClkHk= 2023-10-18T20:43:08.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IGyTmYRNLt/LD8n+BSkmT7ClkHk=', 'Content-Length': '0'} 2023-10-18T20:43:08.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b57e051df333896-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.891 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:08.892 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:08.892 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.892 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.893 DEBUG:boto:Method: GET 2023-10-18T20:43:08.893 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.894 DEBUG:boto:Data: 2023-10-18T20:43:08.894 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.895 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.895 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.895 DEBUG:boto:Params: {} 2023-10-18T20:43:08.896 DEBUG:boto:Token: None 2023-10-18T20:43:08.896 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.897 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IGyTmYRNLt/LD8n+BSkmT7ClkHk= 2023-10-18T20:43:08.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IGyTmYRNLt/LD8n+BSkmT7ClkHk=', 'Content-Length': '0'} 2023-10-18T20:43:08.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeac73b1bd10062a-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.900 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:08.900 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:08.900 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.901 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.901 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.902 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.902 DEBUG:boto:Data: 2023-10-18T20:43:08.902 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.903 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.903 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.904 DEBUG:boto:Params: {} 2023-10-18T20:43:08.904 DEBUG:boto:Token: None 2023-10-18T20:43:08.904 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.905 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2M6Ns1becZJNGkqYeh1acJdgkWE= 2023-10-18T20:43:08.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2M6Ns1becZJNGkqYeh1acJdgkWE=', 'Content-Length': '0'} 2023-10-18T20:43:08.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff4739ad3d324d3-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.907 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:08.908 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:08.908 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.909 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.909 DEBUG:boto:Method: HEAD 2023-10-18T20:43:08.909 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.910 DEBUG:boto:Data: 2023-10-18T20:43:08.910 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.910 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.911 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.911 DEBUG:boto:Params: {} 2023-10-18T20:43:08.912 DEBUG:boto:Token: None 2023-10-18T20:43:08.912 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.913 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2M6Ns1becZJNGkqYeh1acJdgkWE= 2023-10-18T20:43:08.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2M6Ns1becZJNGkqYeh1acJdgkWE=', 'Content-Length': '0'} 2023-10-18T20:43:08.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b32488b9fa8978b5-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:43:08.916 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:08.916 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:08.917 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.917 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.917 DEBUG:boto:Method: GET 2023-10-18T20:43:08.918 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.918 DEBUG:boto:Data: 2023-10-18T20:43:08.919 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.919 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.920 DEBUG:boto:Params: {} 2023-10-18T20:43:08.920 DEBUG:boto:Token: None 2023-10-18T20:43:08.921 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.921 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IGyTmYRNLt/LD8n+BSkmT7ClkHk= 2023-10-18T20:43:08.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IGyTmYRNLt/LD8n+BSkmT7ClkHk=', 'Content-Length': '0'} 2023-10-18T20:43:08.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d429661e835fac4-006530435c-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.970 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:08.971 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:08.971 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.972 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.972 DEBUG:boto:Method: GET 2023-10-18T20:43:08.972 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.973 DEBUG:boto:Data: 2023-10-18T20:43:08.973 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.974 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.974 DEBUG:boto:Params: {} 2023-10-18T20:43:08.975 DEBUG:boto:Token: None 2023-10-18T20:43:08.975 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:43:08.976 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IGyTmYRNLt/LD8n+BSkmT7ClkHk= 2023-10-18T20:43:08.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IGyTmYRNLt/LD8n+BSkmT7ClkHk=', 'Content-Length': '0'} 2023-10-18T20:43:08.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001393c2e84573a670-006530435c-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.979 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.979 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.979 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:43:08.980 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:43:08.980 DEBUG:boto:Method: GET 2023-10-18T20:43:08.981 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:43:08.981 DEBUG:boto:Data: 2023-10-18T20:43:08.981 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.982 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:08.982 DEBUG:boto:Port: 8001 2023-10-18T20:43:08.983 DEBUG:boto:Params: {} 2023-10-18T20:43:08.983 DEBUG:boto:Token: None 2023-10-18T20:43:08.984 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.984 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2cd5bd4455e141d-006530435c-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.987 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.988 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:08.988 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:08.989 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:43:08.989 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:43:08.989 DEBUG:boto:Method: GET 2023-10-18T20:43:08.990 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:43:08.990 DEBUG:boto:Data: 2023-10-18T20:43:08.991 DEBUG:boto:Headers: {} 2023-10-18T20:43:08.991 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:08.992 DEBUG:boto:Port: 8000 2023-10-18T20:43:08.992 DEBUG:boto:Params: {} 2023-10-18T20:43:08.992 DEBUG:boto:Token: None 2023-10-18T20:43:08.993 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:08 GMT /bofrnf-59/ 2023-10-18T20:43:08.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as= 2023-10-18T20:43:08.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rBW/o5kR1cJyEAe37cOr4B0+1as=', 'Content-Length': '0'} 2023-10-18T20:43:08.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a3a4f8c6b914742-006530435c-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:08.998 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:08.998 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:08.999 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:08.999 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.000 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:43:09.000 DEBUG:boto:Data: 2023-10-18T20:43:09.001 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.001 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.002 DEBUG:boto:Params: {} 2023-10-18T20:43:09.002 DEBUG:boto:Token: None 2023-10-18T20:43:09.003 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj92 2023-10-18T20:43:09.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8S4662clkdiO+Gf4sROsxSCD2Vc= 2023-10-18T20:43:09.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8S4662clkdiO+Gf4sROsxSCD2Vc=', 'Content-Length': '0'} 2023-10-18T20:43:09.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac4b64006201daa-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.006 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:09.006 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:09.007 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.007 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:43:09.007 DEBUG:boto:Data: 2023-10-18T20:43:09.008 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.008 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.009 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.009 DEBUG:boto:Params: {} 2023-10-18T20:43:09.009 DEBUG:boto:Token: None 2023-10-18T20:43:09.010 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj92 2023-10-18T20:43:09.010 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8S4662clkdiO+Gf4sROsxSCD2Vc= 2023-10-18T20:43:09.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8S4662clkdiO+Gf4sROsxSCD2Vc=', 'Content-Length': '0'} 2023-10-18T20:43:09.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d1c4bb1451e553-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:43:09.013 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:09.014 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:09.014 DEBUG:boto:Method: GET 2023-10-18T20:43:09.015 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:43:09.015 DEBUG:boto:Data: 2023-10-18T20:43:09.015 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.016 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.016 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.016 DEBUG:boto:Params: {} 2023-10-18T20:43:09.017 DEBUG:boto:Token: None 2023-10-18T20:43:09.017 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj92 2023-10-18T20:43:09.018 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QVWvhz/FCC0ateYD/sXbzRQWMmk= 2023-10-18T20:43:09.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QVWvhz/FCC0ateYD/sXbzRQWMmk=', 'Content-Length': '0'} 2023-10-18T20:43:09.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b183baea38bc88e9-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.021 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:43:09.021 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:43:09.022 DEBUG:boto:Method: GET 2023-10-18T20:43:09.022 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:43:09.022 DEBUG:boto:Data: 2023-10-18T20:43:09.023 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.023 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.024 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.024 DEBUG:boto:Params: {} 2023-10-18T20:43:09.025 DEBUG:boto:Token: None 2023-10-18T20:43:09.025 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj92 2023-10-18T20:43:09.026 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QVWvhz/FCC0ateYD/sXbzRQWMmk= 2023-10-18T20:43:09.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QVWvhz/FCC0ateYD/sXbzRQWMmk=', 'Content-Length': '0'} 2023-10-18T20:43:09.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e12af6651284a14b-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.029 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.029 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.029 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:43:09.030 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:43:09.030 DEBUG:boto:Method: GET 2023-10-18T20:43:09.031 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:43:09.031 DEBUG:boto:Data: 2023-10-18T20:43:09.031 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.032 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.032 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.033 DEBUG:boto:Params: {} 2023-10-18T20:43:09.033 DEBUG:boto:Token: None 2023-10-18T20:43:09.034 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.034 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006105e8b591cbd0ba-006530435d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.037 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.038 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.038 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.039 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:43:09.039 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:43:09.040 DEBUG:boto:Method: GET 2023-10-18T20:43:09.040 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:43:09.040 DEBUG:boto:Data: 2023-10-18T20:43:09.041 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.041 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.042 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.042 DEBUG:boto:Params: {} 2023-10-18T20:43:09.043 DEBUG:boto:Token: None 2023-10-18T20:43:09.043 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000590e02cee97e1ec7-006530435d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.048 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-18T20:43:09.049 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.049 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.050 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.050 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.051 DEBUG:boto:Method: GET 2023-10-18T20:43:09.051 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.051 DEBUG:boto:Data: 2023-10-18T20:43:09.052 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.052 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.053 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.053 DEBUG:boto:Params: {} 2023-10-18T20:43:09.054 DEBUG:boto:Token: None 2023-10-18T20:43:09.054 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.055 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kZ2AnemMLNcWVRylWpMjPK6RsSg= 2023-10-18T20:43:09.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kZ2AnemMLNcWVRylWpMjPK6RsSg=', 'Content-Length': '0'} 2023-10-18T20:43:09.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e102b3216ca972-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.058 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.058 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.059 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.059 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.059 DEBUG:boto:Method: GET 2023-10-18T20:43:09.060 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.060 DEBUG:boto:Data: 2023-10-18T20:43:09.060 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.061 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.061 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.062 DEBUG:boto:Params: {} 2023-10-18T20:43:09.062 DEBUG:boto:Token: None 2023-10-18T20:43:09.063 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.063 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kZ2AnemMLNcWVRylWpMjPK6RsSg= 2023-10-18T20:43:09.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kZ2AnemMLNcWVRylWpMjPK6RsSg=', 'Content-Length': '0'} 2023-10-18T20:43:09.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000615ef7aa3e314142-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.106 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.107 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.107 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.108 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.108 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.108 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.109 DEBUG:boto:Data: 2023-10-18T20:43:09.109 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.110 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.110 DEBUG:boto:Params: {} 2023-10-18T20:43:09.111 DEBUG:boto:Token: None 2023-10-18T20:43:09.111 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.112 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:quHy0MbT7q4feesR9bwUiIqBvfQ= 2023-10-18T20:43:09.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:quHy0MbT7q4feesR9bwUiIqBvfQ=', 'Content-Length': '0'} 2023-10-18T20:43:09.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c818fd8ea35a08-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.115 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.115 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.115 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.116 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.116 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.117 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.117 DEBUG:boto:Data: 2023-10-18T20:43:09.117 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.118 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.118 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.119 DEBUG:boto:Params: {} 2023-10-18T20:43:09.119 DEBUG:boto:Token: None 2023-10-18T20:43:09.119 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.120 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:quHy0MbT7q4feesR9bwUiIqBvfQ= 2023-10-18T20:43:09.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:quHy0MbT7q4feesR9bwUiIqBvfQ=', 'Content-Length': '0'} 2023-10-18T20:43:09.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9291c96ef86435a-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-18T20:43:09.123 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.123 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.124 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.124 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.125 DEBUG:boto:Method: GET 2023-10-18T20:43:09.125 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.125 DEBUG:boto:Data: 2023-10-18T20:43:09.126 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.126 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.127 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.127 DEBUG:boto:Params: {} 2023-10-18T20:43:09.128 DEBUG:boto:Token: None 2023-10-18T20:43:09.128 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.129 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kZ2AnemMLNcWVRylWpMjPK6RsSg= 2023-10-18T20:43:09.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kZ2AnemMLNcWVRylWpMjPK6RsSg=', 'Content-Length': '0'} 2023-10-18T20:43:09.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a653f06d09ae57-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.132 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.132 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.132 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.133 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.133 DEBUG:boto:Method: GET 2023-10-18T20:43:09.134 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.134 DEBUG:boto:Data: 2023-10-18T20:43:09.135 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.135 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.135 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.136 DEBUG:boto:Params: {} 2023-10-18T20:43:09.136 DEBUG:boto:Token: None 2023-10-18T20:43:09.137 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:43:09.137 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kZ2AnemMLNcWVRylWpMjPK6RsSg= 2023-10-18T20:43:09.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kZ2AnemMLNcWVRylWpMjPK6RsSg=', 'Content-Length': '0'} 2023-10-18T20:43:09.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c75abfad1067c94-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.140 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.141 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.141 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.141 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.142 DEBUG:boto:Method: GET 2023-10-18T20:43:09.142 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.143 DEBUG:boto:Data: 2023-10-18T20:43:09.143 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.143 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.144 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.144 DEBUG:boto:Params: {} 2023-10-18T20:43:09.144 DEBUG:boto:Token: None 2023-10-18T20:43:09.145 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.145 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033d3690b29ddb6a2-006530435d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.190 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.191 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.192 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.192 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.193 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.193 DEBUG:boto:Method: GET 2023-10-18T20:43:09.193 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.194 DEBUG:boto:Data: 2023-10-18T20:43:09.194 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.195 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.195 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.195 DEBUG:boto:Params: {} 2023-10-18T20:43:09.196 DEBUG:boto:Token: None 2023-10-18T20:43:09.196 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.197 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea8d5e8ca442fcd6-006530435d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.201 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.202 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.202 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.203 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.203 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:43:09.203 DEBUG:boto:Data: 2023-10-18T20:43:09.204 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.204 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.205 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.205 DEBUG:boto:Params: {} 2023-10-18T20:43:09.205 DEBUG:boto:Token: None 2023-10-18T20:43:09.206 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93 2023-10-18T20:43:09.206 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TLi9Tcwj3E2f+un4wCTqf123YwA= 2023-10-18T20:43:09.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TLi9Tcwj3E2f+un4wCTqf123YwA=', 'Content-Length': '0'} 2023-10-18T20:43:09.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005182cf2fffd6ce14-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.209 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.210 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.210 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.210 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:43:09.211 DEBUG:boto:Data: 2023-10-18T20:43:09.211 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.211 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.212 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.212 DEBUG:boto:Params: {} 2023-10-18T20:43:09.213 DEBUG:boto:Token: None 2023-10-18T20:43:09.213 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93 2023-10-18T20:43:09.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TLi9Tcwj3E2f+un4wCTqf123YwA= 2023-10-18T20:43:09.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TLi9Tcwj3E2f+un4wCTqf123YwA=', 'Content-Length': '0'} 2023-10-18T20:43:09.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514b296ef5ef2111-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-18T20:43:09.217 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.217 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.217 DEBUG:boto:Method: GET 2023-10-18T20:43:09.218 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:43:09.218 DEBUG:boto:Data: 2023-10-18T20:43:09.219 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.219 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.220 DEBUG:boto:Params: {} 2023-10-18T20:43:09.220 DEBUG:boto:Token: None 2023-10-18T20:43:09.221 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93 2023-10-18T20:43:09.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZCeIDFZetS05KwPCee6Hgoqxg3c= 2023-10-18T20:43:09.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZCeIDFZetS05KwPCee6Hgoqxg3c=', 'Content-Length': '0'} 2023-10-18T20:43:09.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa0bd36934b4ade-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.224 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:43:09.224 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:43:09.225 DEBUG:boto:Method: GET 2023-10-18T20:43:09.225 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:43:09.226 DEBUG:boto:Data: 2023-10-18T20:43:09.226 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.227 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.227 DEBUG:boto:Params: {} 2023-10-18T20:43:09.228 DEBUG:boto:Token: None 2023-10-18T20:43:09.228 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj93 2023-10-18T20:43:09.228 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZCeIDFZetS05KwPCee6Hgoqxg3c= 2023-10-18T20:43:09.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZCeIDFZetS05KwPCee6Hgoqxg3c=', 'Content-Length': '0'} 2023-10-18T20:43:09.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000244f3de20b9331de-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.231 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.231 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.232 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.232 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.233 DEBUG:boto:Method: GET 2023-10-18T20:43:09.233 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.233 DEBUG:boto:Data: 2023-10-18T20:43:09.234 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.234 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.235 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.235 DEBUG:boto:Params: {} 2023-10-18T20:43:09.235 DEBUG:boto:Token: None 2023-10-18T20:43:09.236 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.236 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021332680457d4659-006530435d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.239 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.240 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.241 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.241 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.241 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.242 DEBUG:boto:Method: GET 2023-10-18T20:43:09.242 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:43:09.243 DEBUG:boto:Data: 2023-10-18T20:43:09.243 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.243 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.244 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.244 DEBUG:boto:Params: {} 2023-10-18T20:43:09.245 DEBUG:boto:Token: None 2023-10-18T20:43:09.245 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.245 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005acce13d167cf803-006530435d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.290 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:43:09.291 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.292 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.292 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.292 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.293 DEBUG:boto:Method: GET 2023-10-18T20:43:09.293 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.294 DEBUG:boto:Data: 2023-10-18T20:43:09.294 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.295 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.295 DEBUG:boto:Params: {} 2023-10-18T20:43:09.296 DEBUG:boto:Token: None 2023-10-18T20:43:09.296 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.297 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:39eirOxLaN99ATboMD8Q+ifwezY= 2023-10-18T20:43:09.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:39eirOxLaN99ATboMD8Q+ifwezY=', 'Content-Length': '0'} 2023-10-18T20:43:09.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005acd393f439f6e97-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.300 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.300 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.301 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.301 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.301 DEBUG:boto:Method: GET 2023-10-18T20:43:09.302 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.302 DEBUG:boto:Data: 2023-10-18T20:43:09.303 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.303 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.304 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.304 DEBUG:boto:Params: {} 2023-10-18T20:43:09.304 DEBUG:boto:Token: None 2023-10-18T20:43:09.305 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:39eirOxLaN99ATboMD8Q+ifwezY= 2023-10-18T20:43:09.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:39eirOxLaN99ATboMD8Q+ifwezY=', 'Content-Length': '0'} 2023-10-18T20:43:09.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9cd45045fb523d-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.308 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.309 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.309 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.310 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.310 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.310 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.311 DEBUG:boto:Data: 2023-10-18T20:43:09.311 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.312 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.312 DEBUG:boto:Params: {} 2023-10-18T20:43:09.313 DEBUG:boto:Token: None 2023-10-18T20:43:09.313 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.314 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nRG5/KW7yy2Dz4WCni7i2eTUTGU= 2023-10-18T20:43:09.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nRG5/KW7yy2Dz4WCni7i2eTUTGU=', 'Content-Length': '0'} 2023-10-18T20:43:09.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343ef5f952127f82-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.317 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.317 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.317 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.318 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.318 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.319 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.319 DEBUG:boto:Data: 2023-10-18T20:43:09.319 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.320 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.320 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.320 DEBUG:boto:Params: {} 2023-10-18T20:43:09.321 DEBUG:boto:Token: None 2023-10-18T20:43:09.321 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.322 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nRG5/KW7yy2Dz4WCni7i2eTUTGU= 2023-10-18T20:43:09.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nRG5/KW7yy2Dz4WCni7i2eTUTGU=', 'Content-Length': '0'} 2023-10-18T20:43:09.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007814cfcd5c5bfd51-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:43:09.325 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.326 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.326 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.326 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.327 DEBUG:boto:Method: GET 2023-10-18T20:43:09.327 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.328 DEBUG:boto:Data: 2023-10-18T20:43:09.328 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.329 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.329 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.329 DEBUG:boto:Params: {} 2023-10-18T20:43:09.330 DEBUG:boto:Token: None 2023-10-18T20:43:09.330 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.331 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:39eirOxLaN99ATboMD8Q+ifwezY= 2023-10-18T20:43:09.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:39eirOxLaN99ATboMD8Q+ifwezY=', 'Content-Length': '0'} 2023-10-18T20:43:09.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f88a147e808d595-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.334 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.334 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.335 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.335 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.336 DEBUG:boto:Method: GET 2023-10-18T20:43:09.336 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.336 DEBUG:boto:Data: 2023-10-18T20:43:09.337 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.337 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.338 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.338 DEBUG:boto:Params: {} 2023-10-18T20:43:09.339 DEBUG:boto:Token: None 2023-10-18T20:43:09.339 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:43:09.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:39eirOxLaN99ATboMD8Q+ifwezY= 2023-10-18T20:43:09.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:39eirOxLaN99ATboMD8Q+ifwezY=', 'Content-Length': '0'} 2023-10-18T20:43:09.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010679b71427e55c9-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.343 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.343 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.344 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.344 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.345 DEBUG:boto:Method: GET 2023-10-18T20:43:09.345 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.345 DEBUG:boto:Data: 2023-10-18T20:43:09.346 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.347 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.347 DEBUG:boto:Params: {} 2023-10-18T20:43:09.348 DEBUG:boto:Token: None 2023-10-18T20:43:09.348 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.349 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa6871403427387b-006530435d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.352 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.352 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.353 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.353 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.354 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.354 DEBUG:boto:Method: GET 2023-10-18T20:43:09.355 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.355 DEBUG:boto:Data: 2023-10-18T20:43:09.355 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.356 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.356 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.357 DEBUG:boto:Params: {} 2023-10-18T20:43:09.357 DEBUG:boto:Token: None 2023-10-18T20:43:09.357 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0ad7afdd819506e-006530435d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.362 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.363 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.363 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.363 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.364 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:43:09.364 DEBUG:boto:Data: 2023-10-18T20:43:09.365 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.365 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.365 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.366 DEBUG:boto:Params: {} 2023-10-18T20:43:09.366 DEBUG:boto:Token: None 2023-10-18T20:43:09.367 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95 2023-10-18T20:43:09.367 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5ePU8Rcq0uCoxDozzvJizDba7PY= 2023-10-18T20:43:09.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5ePU8Rcq0uCoxDozzvJizDba7PY=', 'Content-Length': '0'} 2023-10-18T20:43:09.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000206959244711d4f2-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.370 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.370 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.371 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.371 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:43:09.371 DEBUG:boto:Data: 2023-10-18T20:43:09.372 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.372 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.373 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.373 DEBUG:boto:Params: {} 2023-10-18T20:43:09.373 DEBUG:boto:Token: None 2023-10-18T20:43:09.374 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95 2023-10-18T20:43:09.374 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5ePU8Rcq0uCoxDozzvJizDba7PY= 2023-10-18T20:43:09.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5ePU8Rcq0uCoxDozzvJizDba7PY=', 'Content-Length': '0'} 2023-10-18T20:43:09.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000524270ce46efa22a-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:43:09.377 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.378 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.378 DEBUG:boto:Method: GET 2023-10-18T20:43:09.379 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:43:09.379 DEBUG:boto:Data: 2023-10-18T20:43:09.380 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.380 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.381 DEBUG:boto:Params: {} 2023-10-18T20:43:09.381 DEBUG:boto:Token: None 2023-10-18T20:43:09.382 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95 2023-10-18T20:43:09.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XOZYyCNWZ32pjuAATA6Oxg8fh5U= 2023-10-18T20:43:09.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XOZYyCNWZ32pjuAATA6Oxg8fh5U=', 'Content-Length': '0'} 2023-10-18T20:43:09.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd170549b05fd8a9-006530435d-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.426 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:43:09.427 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:43:09.427 DEBUG:boto:Method: GET 2023-10-18T20:43:09.428 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:43:09.428 DEBUG:boto:Data: 2023-10-18T20:43:09.428 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.429 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.429 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.430 DEBUG:boto:Params: {} 2023-10-18T20:43:09.430 DEBUG:boto:Token: None 2023-10-18T20:43:09.431 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj95 2023-10-18T20:43:09.431 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XOZYyCNWZ32pjuAATA6Oxg8fh5U= 2023-10-18T20:43:09.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XOZYyCNWZ32pjuAATA6Oxg8fh5U=', 'Content-Length': '0'} 2023-10-18T20:43:09.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebefb6245d831832-006530435d-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.435 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.435 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.435 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.436 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.436 DEBUG:boto:Method: GET 2023-10-18T20:43:09.437 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.437 DEBUG:boto:Data: 2023-10-18T20:43:09.438 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.438 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:43:09.438 DEBUG:boto:Port: 8001 2023-10-18T20:43:09.439 DEBUG:boto:Params: {} 2023-10-18T20:43:09.439 DEBUG:boto:Token: None 2023-10-18T20:43:09.440 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc22f1fd84262cb-006530435d-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.444 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.444 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.445 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.445 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.446 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.446 DEBUG:boto:Method: GET 2023-10-18T20:43:09.447 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:43:09.447 DEBUG:boto:Data: 2023-10-18T20:43:09.447 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.448 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.448 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.449 DEBUG:boto:Params: {} 2023-10-18T20:43:09.449 DEBUG:boto:Token: None 2023-10-18T20:43:09.450 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.450 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY= 2023-10-18T20:43:09.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UNDuHy9OwsMBXorCTPykAEAkXkY=', 'Content-Length': '0'} 2023-10-18T20:43:09.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2fcb97eaffcb28f-006530435d-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.455 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:43:09.456 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:43:09.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:43:09.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:43:09.457 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:43:09.513 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:43:09.513 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:43:09.565 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:43:09.565 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-59 2023-10-18T20:43:09.576 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 30 2023-10-18T20:43:09.632 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.633 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.633 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.634 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:43:09.634 DEBUG:boto:Data: 2023-10-18T20:43:09.635 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.635 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.635 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.636 DEBUG:boto:Params: {} 2023-10-18T20:43:09.636 DEBUG:boto:Token: None 2023-10-18T20:43:09.637 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.637 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms= 2023-10-18T20:43:09.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms=', 'Content-Length': '0'} 2023-10-18T20:43:09.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b18f7945be94e57-006530435d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.642 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:43:09.642 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:43:09.643 DEBUG:boto:Method: PUT 2023-10-18T20:43:09.643 DEBUG:boto:Path: /bofrnf-59/obj37 2023-10-18T20:43:09.644 DEBUG:boto:Data: 2023-10-18T20:43:09.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:43:09.644 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.645 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.645 DEBUG:boto:Params: {} 2023-10-18T20:43:09.645 DEBUG:boto:Token: None 2023-10-18T20:43:09.646 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj37 2023-10-18T20:43:09.646 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BKSBCttI4ndQxC3WL0oExCEeSGA= 2023-10-18T20:43:09.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BKSBCttI4ndQxC3WL0oExCEeSGA='} 2023-10-18T20:43:09.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ea853a71f785d05-006530435d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.653 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.653 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.653 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.654 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:43:09.654 DEBUG:boto:Data: 2023-10-18T20:43:09.655 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.655 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.655 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.656 DEBUG:boto:Params: {} 2023-10-18T20:43:09.656 DEBUG:boto:Token: None 2023-10-18T20:43:09.657 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.657 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms= 2023-10-18T20:43:09.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms=', 'Content-Length': '0'} 2023-10-18T20:43:09.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062178fd1e96973ea-006530435d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.661 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:43:09.662 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:43:09.662 DEBUG:boto:Method: PUT 2023-10-18T20:43:09.662 DEBUG:boto:Path: /bofrnf-59/obj74 2023-10-18T20:43:09.663 DEBUG:boto:Data: 2023-10-18T20:43:09.663 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:43:09.664 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.664 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.664 DEBUG:boto:Params: {} 2023-10-18T20:43:09.665 DEBUG:boto:Token: None 2023-10-18T20:43:09.665 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj74 2023-10-18T20:43:09.666 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WW371PGLHDUl7d2WPeUKT27ateo= 2023-10-18T20:43:09.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WW371PGLHDUl7d2WPeUKT27ateo='} 2023-10-18T20:43:09.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fa0ab633c268be3-006530435d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.711 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.711 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.712 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.712 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:43:09.713 DEBUG:boto:Data: 2023-10-18T20:43:09.713 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.713 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.714 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.714 DEBUG:boto:Params: {} 2023-10-18T20:43:09.715 DEBUG:boto:Token: None 2023-10-18T20:43:09.715 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.716 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms= 2023-10-18T20:43:09.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms=', 'Content-Length': '0'} 2023-10-18T20:43:09.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011397bf4650c457e-006530435d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.720 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:43:09.721 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:43:09.721 DEBUG:boto:Method: PUT 2023-10-18T20:43:09.722 DEBUG:boto:Path: /bofrnf-59/obj111 2023-10-18T20:43:09.722 DEBUG:boto:Data: 2023-10-18T20:43:09.722 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:43:09.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.723 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.724 DEBUG:boto:Params: {} 2023-10-18T20:43:09.724 DEBUG:boto:Token: None 2023-10-18T20:43:09.725 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj111 2023-10-18T20:43:09.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Vs36jX18/LZRtFWeJhHY6k8WTcU= 2023-10-18T20:43:09.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vs36jX18/LZRtFWeJhHY6k8WTcU='} 2023-10-18T20:43:09.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007de51952a030945f-006530435d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.731 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.731 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.732 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.732 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:43:09.732 DEBUG:boto:Data: 2023-10-18T20:43:09.733 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.734 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.734 DEBUG:boto:Params: {} 2023-10-18T20:43:09.734 DEBUG:boto:Token: None 2023-10-18T20:43:09.735 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms= 2023-10-18T20:43:09.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms=', 'Content-Length': '0'} 2023-10-18T20:43:09.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a1c9d0490a0036a-006530435d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.740 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:43:09.740 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:43:09.741 DEBUG:boto:Method: PUT 2023-10-18T20:43:09.741 DEBUG:boto:Path: /bofrnf-59/obj148 2023-10-18T20:43:09.741 DEBUG:boto:Data: 2023-10-18T20:43:09.742 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:43:09.742 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.742 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.743 DEBUG:boto:Params: {} 2023-10-18T20:43:09.743 DEBUG:boto:Token: None 2023-10-18T20:43:09.744 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj148 2023-10-18T20:43:09.744 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CjLf88hD812JYtxEItXdc0TxcmI= 2023-10-18T20:43:09.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CjLf88hD812JYtxEItXdc0TxcmI='} 2023-10-18T20:43:09.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d45e626c2fa96330-006530435d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.787 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:43:09.788 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:43:09.788 DEBUG:boto:Method: HEAD 2023-10-18T20:43:09.788 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:43:09.789 DEBUG:boto:Data: 2023-10-18T20:43:09.789 DEBUG:boto:Headers: {} 2023-10-18T20:43:09.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.790 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.790 DEBUG:boto:Params: {} 2023-10-18T20:43:09.791 DEBUG:boto:Token: None 2023-10-18T20:43:09.791 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/ 2023-10-18T20:43:09.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms= 2023-10-18T20:43:09.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1JV/fRTLmg37LUVvU9RnQF8e0Ms=', 'Content-Length': '0'} 2023-10-18T20:43:09.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099735a64e29c3923-006530435d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.796 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:43:09.797 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:43:09.797 DEBUG:boto:Method: PUT 2023-10-18T20:43:09.798 DEBUG:boto:Path: /bofrnf-59/obj185 2023-10-18T20:43:09.798 DEBUG:boto:Data: 2023-10-18T20:43:09.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:43:09.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:43:09.799 DEBUG:boto:Port: 8000 2023-10-18T20:43:09.800 DEBUG:boto:Params: {} 2023-10-18T20:43:09.800 DEBUG:boto:Token: None 2023-10-18T20:43:09.800 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:43:09 GMT /bofrnf-59/obj185 2023-10-18T20:43:09.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HVkGxAplXfnZOP6+f674Js108Dw= 2023-10-18T20:43:09.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:43:09 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HVkGxAplXfnZOP6+f674Js108Dw='} 2023-10-18T20:43:09.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e3af9334a91b3a2-006530435d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:43:09.807 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-59 2023-10-18T20:43:09.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:43:09.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:43:09.808 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:43:09.864 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:43:09.865 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:43:09.900 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:09.898+0000 7fa13564d780 1 waiting to reach latest gen 6, on 5.. 2023-10-18T20:43:14.901 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:14.899+0000 7fa13564d780 1 waiting to reach latest gen 6, on 5.. 2023-10-18T20:43:19.901 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:19.899+0000 7fa13564d780 1 waiting to reach latest gen 6, on 5.. 2023-10-18T20:43:24.902 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:24.900+0000 7fa13564d780 1 waiting to reach latest gen 6, on 5.. 2023-10-18T20:43:29.905 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:29.903+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:43:29.906 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:43:29.906 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:43:34.908 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:34.906+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:43:34.909 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:43:34.909 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:43:39.910 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:39.909+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:43:39.911 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:43:39.911 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:43:44.912 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:44.911+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:43:44.913 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-18T20:43:44.913 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:43:49.915 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:49.914+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:43:49.915 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:43:49.916 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:43:54.918 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:54.916+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:43:54.918 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:43:54.918 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:43:59.920 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:43:59.919+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:43:59.921 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:43:59.921 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:04.923 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:04.921+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:04.923 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:04.923 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:09.925 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:09.923+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:09.926 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:09.926 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:14.928 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:14.926+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:14.929 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:14.929 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:19.931 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:19.928+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:19.931 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:19.931 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:24.933 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:24.931+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:24.933 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:24.934 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:29.936 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:29.934+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:29.936 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:29.936 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:34.938 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:34.936+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:34.938 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:34.939 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:39.941 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:39.939+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:39.941 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:39.941 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:44.943 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:44.942+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:44.944 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:44.944 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:49.946 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:49.944+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:49.946 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:49.946 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:54.948 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:54.947+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:54.949 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:54.949 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:44:59.951 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:44:59.949+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:44:59.951 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:44:59.952 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:45:04.954 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:04.951+0000 7fa13564d780 1 waiting for incremental sync to catch up: 2023-10-18T20:45:04.954 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , , , , , 00000000001.100.6] 2023-10-18T20:45:04.954 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:45:09.956 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:09.954+0000 7fa13564d780 1 bucket sync caught up with source: 2023-10-18T20:45:09.957 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:45:09.957 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:45:09.957 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:09.954+0000 7fa13564d780 0 bucket checkpoint complete 2023-10-18T20:45:09.974 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-59 2023-10-18T20:45:09.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:45:09.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:45:09.976 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:45:10.037 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:10.037 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:10.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:10.071+0000 7efc2a8cb780 1 waiting for incremental sync to catch up: 2023-10-18T20:45:10.073 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000004.710.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000004.268.6, 00000000007.468.6, 00000000006.270.6] 2023-10-18T20:45:10.073 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000005.714.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000004.268.6, 00000000007.468.6, 00000000006.270.6] 2023-10-18T20:45:15.075 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:15.073+0000 7efc2a8cb780 1 waiting for incremental sync to catch up: 2023-10-18T20:45:15.075 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000004.710.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000004.268.6, 00000000007.468.6, 00000000006.270.6] 2023-10-18T20:45:15.075 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000005.714.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000004.268.6, 00000000007.468.6, 00000000006.270.6] 2023-10-18T20:45:20.076 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:20.074+0000 7efc2a8cb780 1 bucket sync caught up with source: 2023-10-18T20:45:20.077 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000005.714.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000004.268.6, 00000000007.468.6, 00000000006.270.6] 2023-10-18T20:45:20.077 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000003.789.6, 00000000002.1177.6, 00000000002.1262.6, 00000000005.714.6, 00000000004.783.6, 00000000001.700.6, 00000000001.863.6, 00000000004.268.6, 00000000007.468.6, 00000000006.270.6] 2023-10-18T20:45:20.077 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:20.074+0000 7efc2a8cb780 0 bucket checkpoint complete 2023-10-18T20:45:20.093 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:45:20.094 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.094 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.095 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.095 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:45:20.096 DEBUG:boto:Data: 2023-10-18T20:45:20.096 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.096 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.097 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.097 DEBUG:boto:Params: {} 2023-10-18T20:45:20.098 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:45:20.099 DEBUG:boto:Token: None 2023-10-18T20:45:20.099 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hEJJUa9Yyr2Smd8+BT4bBiQBuwo= 2023-10-18T20:45:20.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hEJJUa9Yyr2Smd8+BT4bBiQBuwo=', 'Content-Length': '0'} 2023-10-18T20:45:20.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fea7557b3d46303-00653043e0-1af4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.105 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.105 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.106 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.106 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:45:20.107 DEBUG:boto:Data: 2023-10-18T20:45:20.107 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.107 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.108 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.108 DEBUG:boto:Params: {} 2023-10-18T20:45:20.109 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:45:20.109 DEBUG:boto:Token: None 2023-10-18T20:45:20.110 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.110 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hEJJUa9Yyr2Smd8+BT4bBiQBuwo= 2023-10-18T20:45:20.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hEJJUa9Yyr2Smd8+BT4bBiQBuwo=', 'Content-Length': '0'} 2023-10-18T20:45:20.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b245185ee378039-00653043e0-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.118 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:45:20.118 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.119 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.119 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.119 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.120 DEBUG:boto:Method: GET 2023-10-18T20:45:20.120 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.120 DEBUG:boto:Data: 2023-10-18T20:45:20.121 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.121 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.122 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.122 DEBUG:boto:Params: {} 2023-10-18T20:45:20.122 DEBUG:boto:Token: None 2023-10-18T20:45:20.123 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/?versions 2023-10-18T20:45:20.123 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lUd9N3NX6IVeFH2hyqkLMJpG8Dc= 2023-10-18T20:45:20.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lUd9N3NX6IVeFH2hyqkLMJpG8Dc=', 'Content-Length': '0'} 2023-10-18T20:45:20.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cc92cdf241e6ad1-00653043e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.166 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:45:20.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-18T20:45:20.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:45:20.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-18T20:45:20.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-18T20:45:20.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-18T20:45:20.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-18T20:45:20.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-18T20:45:20.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:45:20.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-18T20:45:20.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:45:20.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:45:20.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:45:20.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-18T20:45:20.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-18T20:45:20.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:45:20.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:45:20.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-18T20:45:20.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:45:20.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:45:20.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:45:20.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:45:20.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:45:20.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:45:20.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-18T20:45:20.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-18T20:45:20.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:45:20.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:45:20.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-18T20:45:20.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:45:20.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:45:20.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-18T20:45:20.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:45:20.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-18T20:45:20.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:45:20.182 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:45:20.183 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.183 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.184 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.184 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.184 DEBUG:boto:Method: GET 2023-10-18T20:45:20.185 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.185 DEBUG:boto:Data: 2023-10-18T20:45:20.186 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.186 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.187 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.187 DEBUG:boto:Params: {} 2023-10-18T20:45:20.187 DEBUG:boto:Token: None 2023-10-18T20:45:20.188 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/?versions 2023-10-18T20:45:20.188 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lUd9N3NX6IVeFH2hyqkLMJpG8Dc= 2023-10-18T20:45:20.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lUd9N3NX6IVeFH2hyqkLMJpG8Dc=', 'Content-Length': '0'} 2023-10-18T20:45:20.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021ac9f029144e40f-00653043e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.194 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:45:20.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-18T20:45:20.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:45:20.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-18T20:45:20.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-18T20:45:20.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-18T20:45:20.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-18T20:45:20.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-18T20:45:20.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:45:20.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-18T20:45:20.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:45:20.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:45:20.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:45:20.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-18T20:45:20.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-18T20:45:20.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:45:20.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:45:20.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-18T20:45:20.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:45:20.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:45:20.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:45:20.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:45:20.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:45:20.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:45:20.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-18T20:45:20.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-18T20:45:20.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:45:20.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:45:20.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-18T20:45:20.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:45:20.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:45:20.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-18T20:45:20.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:45:20.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-18T20:45:20.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:45:20.209 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.209 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.210 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.210 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.211 DEBUG:boto:Method: GET 2023-10-18T20:45:20.211 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.211 DEBUG:boto:Data: 2023-10-18T20:45:20.212 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.212 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.213 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.213 DEBUG:boto:Params: {} 2023-10-18T20:45:20.214 DEBUG:boto:Token: None 2023-10-18T20:45:20.214 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/?versions 2023-10-18T20:45:20.215 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lUd9N3NX6IVeFH2hyqkLMJpG8Dc= 2023-10-18T20:45:20.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lUd9N3NX6IVeFH2hyqkLMJpG8Dc=', 'Content-Length': '0'} 2023-10-18T20:45:20.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a17f8824f1b06d3-00653043e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.219 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.221 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.221 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.222 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.222 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.223 DEBUG:boto:Method: GET 2023-10-18T20:45:20.223 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:20.224 DEBUG:boto:Data: 2023-10-18T20:45:20.224 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.225 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.226 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.226 DEBUG:boto:Params: {} 2023-10-18T20:45:20.227 DEBUG:boto:Token: None 2023-10-18T20:45:20.227 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/?versions 2023-10-18T20:45:20.228 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lUd9N3NX6IVeFH2hyqkLMJpG8Dc= 2023-10-18T20:45:20.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lUd9N3NX6IVeFH2hyqkLMJpG8Dc=', 'Content-Length': '0'} 2023-10-18T20:45:20.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013aa149237b13fab-00653043e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.234 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:45:20.236 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.237 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.237 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.238 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.238 DEBUG:boto:Method: GET 2023-10-18T20:45:20.239 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.239 DEBUG:boto:Data: 2023-10-18T20:45:20.240 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.240 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.241 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.241 DEBUG:boto:Params: {} 2023-10-18T20:45:20.242 DEBUG:boto:Token: None 2023-10-18T20:45:20.242 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.243 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q/noN+pQiTSZyl04OLRujoq6v9I= 2023-10-18T20:45:20.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q/noN+pQiTSZyl04OLRujoq6v9I=', 'Content-Length': '0'} 2023-10-18T20:45:20.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002276150862e5d2f4-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.246 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.247 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.247 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.247 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.248 DEBUG:boto:Method: GET 2023-10-18T20:45:20.248 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.249 DEBUG:boto:Data: 2023-10-18T20:45:20.249 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.250 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.250 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.251 DEBUG:boto:Params: {} 2023-10-18T20:45:20.251 DEBUG:boto:Token: None 2023-10-18T20:45:20.252 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q/noN+pQiTSZyl04OLRujoq6v9I= 2023-10-18T20:45:20.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q/noN+pQiTSZyl04OLRujoq6v9I=', 'Content-Length': '0'} 2023-10-18T20:45:20.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645bebdb0c44ed7d-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.255 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.256 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.256 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.256 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.257 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.257 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.258 DEBUG:boto:Data: 2023-10-18T20:45:20.258 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.259 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.259 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.259 DEBUG:boto:Params: {} 2023-10-18T20:45:20.260 DEBUG:boto:Token: None 2023-10-18T20:45:20.260 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.261 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7VGLKKb810nT/G13Rz5G5plzac8= 2023-10-18T20:45:20.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7VGLKKb810nT/G13Rz5G5plzac8=', 'Content-Length': '0'} 2023-10-18T20:45:20.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5cb227fbf62e4c9-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.264 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.264 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.265 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.265 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.265 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.266 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.266 DEBUG:boto:Data: 2023-10-18T20:45:20.267 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.267 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.268 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.268 DEBUG:boto:Params: {} 2023-10-18T20:45:20.269 DEBUG:boto:Token: None 2023-10-18T20:45:20.269 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.269 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7VGLKKb810nT/G13Rz5G5plzac8= 2023-10-18T20:45:20.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7VGLKKb810nT/G13Rz5G5plzac8=', 'Content-Length': '0'} 2023-10-18T20:45:20.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000911d7fd80af4dcd4-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:45:20.273 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.273 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.273 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.274 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.274 DEBUG:boto:Method: GET 2023-10-18T20:45:20.275 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.275 DEBUG:boto:Data: 2023-10-18T20:45:20.276 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.276 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.276 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.277 DEBUG:boto:Params: {} 2023-10-18T20:45:20.277 DEBUG:boto:Token: None 2023-10-18T20:45:20.278 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.278 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q/noN+pQiTSZyl04OLRujoq6v9I= 2023-10-18T20:45:20.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q/noN+pQiTSZyl04OLRujoq6v9I=', 'Content-Length': '0'} 2023-10-18T20:45:20.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000895bac191d92e2a9-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.281 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.281 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.282 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.282 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.283 DEBUG:boto:Method: GET 2023-10-18T20:45:20.283 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.284 DEBUG:boto:Data: 2023-10-18T20:45:20.284 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.284 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.285 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.285 DEBUG:boto:Params: {} 2023-10-18T20:45:20.286 DEBUG:boto:Token: None 2023-10-18T20:45:20.286 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:20.287 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q/noN+pQiTSZyl04OLRujoq6v9I= 2023-10-18T20:45:20.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q/noN+pQiTSZyl04OLRujoq6v9I=', 'Content-Length': '0'} 2023-10-18T20:45:20.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d33f113456272ce-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.334 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.335 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.336 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.336 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.336 DEBUG:boto:Method: GET 2023-10-18T20:45:20.337 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.337 DEBUG:boto:Data: 2023-10-18T20:45:20.338 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.338 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.339 DEBUG:boto:Params: {} 2023-10-18T20:45:20.339 DEBUG:boto:Token: None 2023-10-18T20:45:20.340 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000653dea3f72c5e4f2-00653043e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.344 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.345 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.346 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.346 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.347 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.347 DEBUG:boto:Method: GET 2023-10-18T20:45:20.347 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.348 DEBUG:boto:Data: 2023-10-18T20:45:20.348 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.349 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.350 DEBUG:boto:Params: {} 2023-10-18T20:45:20.350 DEBUG:boto:Token: None 2023-10-18T20:45:20.351 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000417c49aa87a0d48d-00653043e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.356 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.357 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.357 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.358 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.358 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:45:20.359 DEBUG:boto:Data: 2023-10-18T20:45:20.359 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.359 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.360 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.360 DEBUG:boto:Params: {} 2023-10-18T20:45:20.361 DEBUG:boto:Token: None 2023-10-18T20:45:20.361 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11 2023-10-18T20:45:20.362 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VjYB5+S1JyIQiaRWVlwnFKq+zow= 2023-10-18T20:45:20.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VjYB5+S1JyIQiaRWVlwnFKq+zow=', 'Content-Length': '0'} 2023-10-18T20:45:20.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c0eb0eaa0c7c35c-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.365 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.365 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.366 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.366 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:45:20.366 DEBUG:boto:Data: 2023-10-18T20:45:20.367 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.368 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.368 DEBUG:boto:Params: {} 2023-10-18T20:45:20.368 DEBUG:boto:Token: None 2023-10-18T20:45:20.369 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11 2023-10-18T20:45:20.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VjYB5+S1JyIQiaRWVlwnFKq+zow= 2023-10-18T20:45:20.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VjYB5+S1JyIQiaRWVlwnFKq+zow=', 'Content-Length': '0'} 2023-10-18T20:45:20.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd4753e576b4d00-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:45:20.373 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.373 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.374 DEBUG:boto:Method: GET 2023-10-18T20:45:20.374 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:45:20.375 DEBUG:boto:Data: 2023-10-18T20:45:20.375 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.375 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.376 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.376 DEBUG:boto:Params: {} 2023-10-18T20:45:20.377 DEBUG:boto:Token: None 2023-10-18T20:45:20.377 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11 2023-10-18T20:45:20.378 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zEqPaLc4zXk1cZbt0DShoRv9mBg= 2023-10-18T20:45:20.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zEqPaLc4zXk1cZbt0DShoRv9mBg=', 'Content-Length': '0'} 2023-10-18T20:45:20.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3ba06f565590d21-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.422 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:20.423 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:20.423 DEBUG:boto:Method: GET 2023-10-18T20:45:20.424 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:45:20.424 DEBUG:boto:Data: 2023-10-18T20:45:20.425 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.426 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.426 DEBUG:boto:Params: {} 2023-10-18T20:45:20.427 DEBUG:boto:Token: None 2023-10-18T20:45:20.427 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj11 2023-10-18T20:45:20.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zEqPaLc4zXk1cZbt0DShoRv9mBg= 2023-10-18T20:45:20.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zEqPaLc4zXk1cZbt0DShoRv9mBg=', 'Content-Length': '0'} 2023-10-18T20:45:20.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a2172852b6e0a5-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.431 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.432 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.432 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.432 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.433 DEBUG:boto:Method: GET 2023-10-18T20:45:20.433 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.434 DEBUG:boto:Data: 2023-10-18T20:45:20.434 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.435 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.435 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.435 DEBUG:boto:Params: {} 2023-10-18T20:45:20.436 DEBUG:boto:Token: None 2023-10-18T20:45:20.436 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.437 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f4f7bfaf96d1262-00653043e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.441 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.441 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.442 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.442 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.443 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.443 DEBUG:boto:Method: GET 2023-10-18T20:45:20.444 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:45:20.444 DEBUG:boto:Data: 2023-10-18T20:45:20.444 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.445 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.446 DEBUG:boto:Params: {} 2023-10-18T20:45:20.446 DEBUG:boto:Token: None 2023-10-18T20:45:20.447 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.447 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000144c434a647ae981-00653043e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.452 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-18T20:45:20.453 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.454 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.454 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.454 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.455 DEBUG:boto:Method: GET 2023-10-18T20:45:20.455 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.456 DEBUG:boto:Data: 2023-10-18T20:45:20.456 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.456 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.457 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.457 DEBUG:boto:Params: {} 2023-10-18T20:45:20.458 DEBUG:boto:Token: None 2023-10-18T20:45:20.458 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.459 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5B62sAep4hpmI/rtbxWpqdIvJng= 2023-10-18T20:45:20.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5B62sAep4hpmI/rtbxWpqdIvJng=', 'Content-Length': '0'} 2023-10-18T20:45:20.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afbe1f5f9705b18f-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.462 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.462 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.463 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.463 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.464 DEBUG:boto:Method: GET 2023-10-18T20:45:20.464 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.465 DEBUG:boto:Data: 2023-10-18T20:45:20.465 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.465 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.466 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.466 DEBUG:boto:Params: {} 2023-10-18T20:45:20.467 DEBUG:boto:Token: None 2023-10-18T20:45:20.467 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.468 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5B62sAep4hpmI/rtbxWpqdIvJng= 2023-10-18T20:45:20.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5B62sAep4hpmI/rtbxWpqdIvJng=', 'Content-Length': '0'} 2023-10-18T20:45:20.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb1fd00bbeabc922-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.518 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.519 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.519 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.520 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.520 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.520 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.521 DEBUG:boto:Data: 2023-10-18T20:45:20.521 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.522 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.522 DEBUG:boto:Params: {} 2023-10-18T20:45:20.523 DEBUG:boto:Token: None 2023-10-18T20:45:20.523 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDnTwpNgP8kZ1SPjfgfATan8YQQ= 2023-10-18T20:45:20.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDnTwpNgP8kZ1SPjfgfATan8YQQ=', 'Content-Length': '0'} 2023-10-18T20:45:20.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81b928e53fb1952-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.527 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.527 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.528 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.528 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.529 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.529 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.530 DEBUG:boto:Data: 2023-10-18T20:45:20.530 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.531 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.531 DEBUG:boto:Params: {} 2023-10-18T20:45:20.532 DEBUG:boto:Token: None 2023-10-18T20:45:20.532 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eDnTwpNgP8kZ1SPjfgfATan8YQQ= 2023-10-18T20:45:20.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eDnTwpNgP8kZ1SPjfgfATan8YQQ=', 'Content-Length': '0'} 2023-10-18T20:45:20.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140cf42e6cb2b728-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-18T20:45:20.536 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.536 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.537 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.537 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.538 DEBUG:boto:Method: GET 2023-10-18T20:45:20.538 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.539 DEBUG:boto:Data: 2023-10-18T20:45:20.539 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.539 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.540 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.540 DEBUG:boto:Params: {} 2023-10-18T20:45:20.541 DEBUG:boto:Token: None 2023-10-18T20:45:20.541 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.542 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5B62sAep4hpmI/rtbxWpqdIvJng= 2023-10-18T20:45:20.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5B62sAep4hpmI/rtbxWpqdIvJng=', 'Content-Length': '0'} 2023-10-18T20:45:20.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ce15ba8cb4243b-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.545 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.545 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.546 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.546 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.547 DEBUG:boto:Method: GET 2023-10-18T20:45:20.547 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.547 DEBUG:boto:Data: 2023-10-18T20:45:20.548 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.548 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.549 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.549 DEBUG:boto:Params: {} 2023-10-18T20:45:20.550 DEBUG:boto:Token: None 2023-10-18T20:45:20.550 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:20.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5B62sAep4hpmI/rtbxWpqdIvJng= 2023-10-18T20:45:20.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5B62sAep4hpmI/rtbxWpqdIvJng=', 'Content-Length': '0'} 2023-10-18T20:45:20.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db4c44046add683b-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.554 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.554 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.555 DEBUG:boto:path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.555 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.556 DEBUG:boto:Method: GET 2023-10-18T20:45:20.556 DEBUG:boto:Path: /bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.557 DEBUG:boto:Data: 2023-10-18T20:45:20.557 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.557 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.558 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.558 DEBUG:boto:Params: {} 2023-10-18T20:45:20.559 DEBUG:boto:Token: None 2023-10-18T20:45:20.559 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085f6072f9ecdb72e-00653043e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.563 DEBUG:boto:b'bofrnf-59obj1111000falseobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.564 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.564 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.564 DEBUG:boto:path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.565 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.565 DEBUG:boto:Method: GET 2023-10-18T20:45:20.566 DEBUG:boto:Path: /bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.566 DEBUG:boto:Data: 2023-10-18T20:45:20.567 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.567 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.567 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.568 DEBUG:boto:Params: {} 2023-10-18T20:45:20.568 DEBUG:boto:Token: None 2023-10-18T20:45:20.569 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b5fbad18645e4d8-00653043e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.575 DEBUG:boto:b'bofrnf-59obj1111000falseobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.576 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.576 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.577 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.577 DEBUG:boto:Path: /bofrnf-59/obj111 2023-10-18T20:45:20.577 DEBUG:boto:Data: 2023-10-18T20:45:20.578 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.578 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.579 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.579 DEBUG:boto:Params: {} 2023-10-18T20:45:20.579 DEBUG:boto:Token: None 2023-10-18T20:45:20.580 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111 2023-10-18T20:45:20.580 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ddOJTNh4ji0N295KhI9SKk22hkg= 2023-10-18T20:45:20.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ddOJTNh4ji0N295KhI9SKk22hkg=', 'Content-Length': '0'} 2023-10-18T20:45:20.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd19fa98f4c290fa-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.583 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.584 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.584 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.585 DEBUG:boto:Path: /bofrnf-59/obj111 2023-10-18T20:45:20.585 DEBUG:boto:Data: 2023-10-18T20:45:20.585 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.586 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.586 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.587 DEBUG:boto:Params: {} 2023-10-18T20:45:20.587 DEBUG:boto:Token: None 2023-10-18T20:45:20.587 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111 2023-10-18T20:45:20.588 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ddOJTNh4ji0N295KhI9SKk22hkg= 2023-10-18T20:45:20.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ddOJTNh4ji0N295KhI9SKk22hkg=', 'Content-Length': '0'} 2023-10-18T20:45:20.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eae73d81bb54cf04-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-18T20:45:20.591 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.592 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.592 DEBUG:boto:Method: GET 2023-10-18T20:45:20.593 DEBUG:boto:Path: /bofrnf-59/obj111 2023-10-18T20:45:20.593 DEBUG:boto:Data: 2023-10-18T20:45:20.593 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.594 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.594 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.595 DEBUG:boto:Params: {} 2023-10-18T20:45:20.595 DEBUG:boto:Token: None 2023-10-18T20:45:20.596 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111 2023-10-18T20:45:20.596 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zavROsbJRckunwqSIWvmShf6mfA= 2023-10-18T20:45:20.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zavROsbJRckunwqSIWvmShf6mfA=', 'Content-Length': '0'} 2023-10-18T20:45:20.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4628837f056a4c-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.642 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:20.643 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:20.644 DEBUG:boto:Method: GET 2023-10-18T20:45:20.644 DEBUG:boto:Path: /bofrnf-59/obj111 2023-10-18T20:45:20.645 DEBUG:boto:Data: 2023-10-18T20:45:20.645 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.646 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.646 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.647 DEBUG:boto:Params: {} 2023-10-18T20:45:20.648 DEBUG:boto:Token: None 2023-10-18T20:45:20.648 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj111 2023-10-18T20:45:20.649 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zavROsbJRckunwqSIWvmShf6mfA= 2023-10-18T20:45:20.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zavROsbJRckunwqSIWvmShf6mfA=', 'Content-Length': '0'} 2023-10-18T20:45:20.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7a83249c85d8307-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.652 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.653 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.653 DEBUG:boto:path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.653 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.654 DEBUG:boto:Method: GET 2023-10-18T20:45:20.654 DEBUG:boto:Path: /bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.655 DEBUG:boto:Data: 2023-10-18T20:45:20.656 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.656 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.656 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.657 DEBUG:boto:Params: {} 2023-10-18T20:45:20.657 DEBUG:boto:Token: None 2023-10-18T20:45:20.658 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.658 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010deda736a3492d7-00653043e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.662 DEBUG:boto:b'bofrnf-59obj1111000falseobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.663 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.663 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.663 DEBUG:boto:path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.664 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.664 DEBUG:boto:Method: GET 2023-10-18T20:45:20.665 DEBUG:boto:Path: /bofrnf-59/?prefix=obj111 2023-10-18T20:45:20.665 DEBUG:boto:Data: 2023-10-18T20:45:20.666 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.666 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.667 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.667 DEBUG:boto:Params: {} 2023-10-18T20:45:20.668 DEBUG:boto:Token: None 2023-10-18T20:45:20.668 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.669 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004252206ab25528e8-00653043e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.674 DEBUG:boto:b'bofrnf-59obj1111000falseobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:45:20.678 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.678 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.679 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.679 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.680 DEBUG:boto:Method: GET 2023-10-18T20:45:20.680 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.681 DEBUG:boto:Data: 2023-10-18T20:45:20.681 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.682 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.682 DEBUG:boto:Params: {} 2023-10-18T20:45:20.683 DEBUG:boto:Token: None 2023-10-18T20:45:20.683 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.684 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HNKYx6cg0Z2ceMaIjZpAry6qHw4= 2023-10-18T20:45:20.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HNKYx6cg0Z2ceMaIjZpAry6qHw4=', 'Content-Length': '0'} 2023-10-18T20:45:20.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8bfd291d0370305-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.687 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.687 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.688 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.688 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.689 DEBUG:boto:Method: GET 2023-10-18T20:45:20.689 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.690 DEBUG:boto:Data: 2023-10-18T20:45:20.690 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.691 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.691 DEBUG:boto:Params: {} 2023-10-18T20:45:20.692 DEBUG:boto:Token: None 2023-10-18T20:45:20.692 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.693 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HNKYx6cg0Z2ceMaIjZpAry6qHw4= 2023-10-18T20:45:20.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HNKYx6cg0Z2ceMaIjZpAry6qHw4=', 'Content-Length': '0'} 2023-10-18T20:45:20.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fabcc03d10fc9c3-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.696 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.696 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.697 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.697 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.697 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.698 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.698 DEBUG:boto:Data: 2023-10-18T20:45:20.699 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.699 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.700 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.700 DEBUG:boto:Params: {} 2023-10-18T20:45:20.700 DEBUG:boto:Token: None 2023-10-18T20:45:20.701 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODqE7KWV+sHrwIRsHad97C/owiY= 2023-10-18T20:45:20.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODqE7KWV+sHrwIRsHad97C/owiY=', 'Content-Length': '0'} 2023-10-18T20:45:20.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1781b7462fe833-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.704 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.705 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.705 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.705 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.706 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.706 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.707 DEBUG:boto:Data: 2023-10-18T20:45:20.707 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.708 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.708 DEBUG:boto:Params: {} 2023-10-18T20:45:20.709 DEBUG:boto:Token: None 2023-10-18T20:45:20.709 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ODqE7KWV+sHrwIRsHad97C/owiY= 2023-10-18T20:45:20.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODqE7KWV+sHrwIRsHad97C/owiY=', 'Content-Length': '0'} 2023-10-18T20:45:20.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b8f207054faa897-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:45:20.713 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.714 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.714 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.715 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.715 DEBUG:boto:Method: GET 2023-10-18T20:45:20.715 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.716 DEBUG:boto:Data: 2023-10-18T20:45:20.716 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.717 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.717 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.717 DEBUG:boto:Params: {} 2023-10-18T20:45:20.718 DEBUG:boto:Token: None 2023-10-18T20:45:20.718 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.719 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HNKYx6cg0Z2ceMaIjZpAry6qHw4= 2023-10-18T20:45:20.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HNKYx6cg0Z2ceMaIjZpAry6qHw4=', 'Content-Length': '0'} 2023-10-18T20:45:20.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5d5e63b84a84ab-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.722 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.722 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.723 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.723 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.724 DEBUG:boto:Method: GET 2023-10-18T20:45:20.724 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.724 DEBUG:boto:Data: 2023-10-18T20:45:20.725 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.726 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.726 DEBUG:boto:Params: {} 2023-10-18T20:45:20.727 DEBUG:boto:Token: None 2023-10-18T20:45:20.727 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:20.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HNKYx6cg0Z2ceMaIjZpAry6qHw4= 2023-10-18T20:45:20.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HNKYx6cg0Z2ceMaIjZpAry6qHw4=', 'Content-Length': '0'} 2023-10-18T20:45:20.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b400f96687f4513-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.774 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.775 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.775 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.776 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.776 DEBUG:boto:Method: GET 2023-10-18T20:45:20.777 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.777 DEBUG:boto:Data: 2023-10-18T20:45:20.778 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.778 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.778 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.779 DEBUG:boto:Params: {} 2023-10-18T20:45:20.779 DEBUG:boto:Token: None 2023-10-18T20:45:20.780 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017392aee777ecde9-00653043e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.784 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.785 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.785 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.786 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.786 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.786 DEBUG:boto:Method: GET 2023-10-18T20:45:20.787 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.787 DEBUG:boto:Data: 2023-10-18T20:45:20.788 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.788 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.789 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.789 DEBUG:boto:Params: {} 2023-10-18T20:45:20.789 DEBUG:boto:Token: None 2023-10-18T20:45:20.790 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.790 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c71563637e234ac-00653043e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.795 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.796 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.797 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.797 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.797 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:45:20.798 DEBUG:boto:Data: 2023-10-18T20:45:20.798 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.799 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.799 DEBUG:boto:Params: {} 2023-10-18T20:45:20.800 DEBUG:boto:Token: None 2023-10-18T20:45:20.800 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115 2023-10-18T20:45:20.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F6RPwC/pVnrbsY8UPP9OpqpVMV0= 2023-10-18T20:45:20.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F6RPwC/pVnrbsY8UPP9OpqpVMV0=', 'Content-Length': '0'} 2023-10-18T20:45:20.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060f44e3c8aa93eae-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.804 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.804 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.805 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.805 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:45:20.805 DEBUG:boto:Data: 2023-10-18T20:45:20.806 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.806 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.807 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.807 DEBUG:boto:Params: {} 2023-10-18T20:45:20.808 DEBUG:boto:Token: None 2023-10-18T20:45:20.808 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115 2023-10-18T20:45:20.809 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F6RPwC/pVnrbsY8UPP9OpqpVMV0= 2023-10-18T20:45:20.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F6RPwC/pVnrbsY8UPP9OpqpVMV0=', 'Content-Length': '0'} 2023-10-18T20:45:20.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928a0234fb08242c-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:45:20.812 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.813 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.813 DEBUG:boto:Method: GET 2023-10-18T20:45:20.814 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:45:20.814 DEBUG:boto:Data: 2023-10-18T20:45:20.815 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.815 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.816 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.816 DEBUG:boto:Params: {} 2023-10-18T20:45:20.817 DEBUG:boto:Token: None 2023-10-18T20:45:20.817 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115 2023-10-18T20:45:20.818 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aK7hfelDqZbDQasHIJpHNos5C7c= 2023-10-18T20:45:20.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aK7hfelDqZbDQasHIJpHNos5C7c=', 'Content-Length': '0'} 2023-10-18T20:45:20.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdbf2dab2da4c5e5-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.866 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:20.867 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:20.868 DEBUG:boto:Method: GET 2023-10-18T20:45:20.868 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:45:20.869 DEBUG:boto:Data: 2023-10-18T20:45:20.869 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.870 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.870 DEBUG:boto:Params: {} 2023-10-18T20:45:20.871 DEBUG:boto:Token: None 2023-10-18T20:45:20.871 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj115 2023-10-18T20:45:20.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aK7hfelDqZbDQasHIJpHNos5C7c= 2023-10-18T20:45:20.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aK7hfelDqZbDQasHIJpHNos5C7c=', 'Content-Length': '0'} 2023-10-18T20:45:20.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cb0b55717e24c00-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.876 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.876 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.876 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.877 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.877 DEBUG:boto:Method: GET 2023-10-18T20:45:20.878 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.878 DEBUG:boto:Data: 2023-10-18T20:45:20.879 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.881 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.882 DEBUG:boto:Params: {} 2023-10-18T20:45:20.882 DEBUG:boto:Token: None 2023-10-18T20:45:20.883 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.883 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013b6def0c6f21a89-00653043e0-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.887 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.888 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:20.888 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:20.889 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.889 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.890 DEBUG:boto:Method: GET 2023-10-18T20:45:20.890 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:45:20.891 DEBUG:boto:Data: 2023-10-18T20:45:20.891 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.892 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.892 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.892 DEBUG:boto:Params: {} 2023-10-18T20:45:20.893 DEBUG:boto:Token: None 2023-10-18T20:45:20.893 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/ 2023-10-18T20:45:20.894 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY= 2023-10-18T20:45:20.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wXzQZIWhZiDXlvKV07dJLAwUZqY=', 'Content-Length': '0'} 2023-10-18T20:45:20.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9f720cf773d1a8b-00653043e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.942 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:20.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:45:20.944 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:20.944 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:20.945 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.945 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.946 DEBUG:boto:Method: GET 2023-10-18T20:45:20.946 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.946 DEBUG:boto:Data: 2023-10-18T20:45:20.947 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.947 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.948 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.948 DEBUG:boto:Params: {} 2023-10-18T20:45:20.949 DEBUG:boto:Token: None 2023-10-18T20:45:20.949 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.950 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vy2p2UEMH6Z/q0yutORWWf4ptw4= 2023-10-18T20:45:20.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vy2p2UEMH6Z/q0yutORWWf4ptw4=', 'Content-Length': '0'} 2023-10-18T20:45:20.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d609d7d4684d8cf6-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.954 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:20.954 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:20.955 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.955 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.955 DEBUG:boto:Method: GET 2023-10-18T20:45:20.956 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.956 DEBUG:boto:Data: 2023-10-18T20:45:20.957 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.958 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.958 DEBUG:boto:Params: {} 2023-10-18T20:45:20.959 DEBUG:boto:Token: None 2023-10-18T20:45:20.959 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vy2p2UEMH6Z/q0yutORWWf4ptw4= 2023-10-18T20:45:20.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vy2p2UEMH6Z/q0yutORWWf4ptw4=', 'Content-Length': '0'} 2023-10-18T20:45:20.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064f1312d19e3edca-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.963 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:20.964 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:20.964 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.965 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.965 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.965 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.966 DEBUG:boto:Data: 2023-10-18T20:45:20.966 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.967 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.967 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.967 DEBUG:boto:Params: {} 2023-10-18T20:45:20.968 DEBUG:boto:Token: None 2023-10-18T20:45:20.968 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.969 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s6A/hsLloqElLPmjThr6aXpQLyY= 2023-10-18T20:45:20.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s6A/hsLloqElLPmjThr6aXpQLyY=', 'Content-Length': '0'} 2023-10-18T20:45:20.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd04c011b1abdc57-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.972 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:20.972 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:20.973 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.973 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.973 DEBUG:boto:Method: HEAD 2023-10-18T20:45:20.974 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.974 DEBUG:boto:Data: 2023-10-18T20:45:20.975 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.975 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.976 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.976 DEBUG:boto:Params: {} 2023-10-18T20:45:20.977 DEBUG:boto:Token: None 2023-10-18T20:45:20.977 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.978 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s6A/hsLloqElLPmjThr6aXpQLyY= 2023-10-18T20:45:20.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s6A/hsLloqElLPmjThr6aXpQLyY=', 'Content-Length': '0'} 2023-10-18T20:45:20.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a56b5d87c1eb6bf-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:45:20.981 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:20.982 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:20.982 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.982 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.983 DEBUG:boto:Method: GET 2023-10-18T20:45:20.983 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.984 DEBUG:boto:Data: 2023-10-18T20:45:20.984 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.985 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:20.985 DEBUG:boto:Port: 8001 2023-10-18T20:45:20.985 DEBUG:boto:Params: {} 2023-10-18T20:45:20.986 DEBUG:boto:Token: None 2023-10-18T20:45:20.986 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.987 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vy2p2UEMH6Z/q0yutORWWf4ptw4= 2023-10-18T20:45:20.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vy2p2UEMH6Z/q0yutORWWf4ptw4=', 'Content-Length': '0'} 2023-10-18T20:45:20.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c21ee9eeeea8020b-00653043e0-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:20.990 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:20.991 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:20.991 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.992 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.992 DEBUG:boto:Method: GET 2023-10-18T20:45:20.993 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.993 DEBUG:boto:Data: 2023-10-18T20:45:20.994 DEBUG:boto:Headers: {} 2023-10-18T20:45:20.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:20.995 DEBUG:boto:Port: 8000 2023-10-18T20:45:20.995 DEBUG:boto:Params: {} 2023-10-18T20:45:20.995 DEBUG:boto:Token: None 2023-10-18T20:45:20.996 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:20 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:20.997 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vy2p2UEMH6Z/q0yutORWWf4ptw4= 2023-10-18T20:45:20.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vy2p2UEMH6Z/q0yutORWWf4ptw4=', 'Content-Length': '0'} 2023-10-18T20:45:20.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12f767be15c24b8-00653043e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.000 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.000 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.001 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.001 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.002 DEBUG:boto:Method: GET 2023-10-18T20:45:21.002 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.003 DEBUG:boto:Data: 2023-10-18T20:45:21.003 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.004 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.004 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.004 DEBUG:boto:Params: {} 2023-10-18T20:45:21.005 DEBUG:boto:Token: None 2023-10-18T20:45:21.005 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.006 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006488b53b84438510-00653043e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.010 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.010 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.011 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.011 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.012 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.012 DEBUG:boto:Method: GET 2023-10-18T20:45:21.013 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.013 DEBUG:boto:Data: 2023-10-18T20:45:21.014 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.014 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.015 DEBUG:boto:Params: {} 2023-10-18T20:45:21.015 DEBUG:boto:Token: None 2023-10-18T20:45:21.016 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.016 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000002624e08fe5cc32-00653043e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.062 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.063 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:21.064 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:21.064 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.065 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:45:21.065 DEBUG:boto:Data: 2023-10-18T20:45:21.065 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.066 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.067 DEBUG:boto:Params: {} 2023-10-18T20:45:21.067 DEBUG:boto:Token: None 2023-10-18T20:45:21.068 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj116 2023-10-18T20:45:21.068 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/Y3IfLvCh64QIGV72xLb4W7Fy0U= 2023-10-18T20:45:21.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/Y3IfLvCh64QIGV72xLb4W7Fy0U=', 'Content-Length': '0'} 2023-10-18T20:45:21.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64bf2a28f21193e-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.071 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:21.072 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:21.072 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.073 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:45:21.073 DEBUG:boto:Data: 2023-10-18T20:45:21.074 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.074 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.074 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.075 DEBUG:boto:Params: {} 2023-10-18T20:45:21.075 DEBUG:boto:Token: None 2023-10-18T20:45:21.076 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj116 2023-10-18T20:45:21.076 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/Y3IfLvCh64QIGV72xLb4W7Fy0U= 2023-10-18T20:45:21.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/Y3IfLvCh64QIGV72xLb4W7Fy0U=', 'Content-Length': '0'} 2023-10-18T20:45:21.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd5abcd2caf37b9-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:45:21.080 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:21.080 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:21.081 DEBUG:boto:Method: GET 2023-10-18T20:45:21.081 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:45:21.082 DEBUG:boto:Data: 2023-10-18T20:45:21.082 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.083 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.085 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.086 DEBUG:boto:Params: {} 2023-10-18T20:45:21.086 DEBUG:boto:Token: None 2023-10-18T20:45:21.087 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj116 2023-10-18T20:45:21.087 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:x8v5fL7Dz8lBwTxu/ONIQdJDT+c= 2023-10-18T20:45:21.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x8v5fL7Dz8lBwTxu/ONIQdJDT+c=', 'Content-Length': '0'} 2023-10-18T20:45:21.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086eb8d2589535346-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.130 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:21.131 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:21.131 DEBUG:boto:Method: GET 2023-10-18T20:45:21.132 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:45:21.132 DEBUG:boto:Data: 2023-10-18T20:45:21.133 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.133 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.133 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.134 DEBUG:boto:Params: {} 2023-10-18T20:45:21.134 DEBUG:boto:Token: None 2023-10-18T20:45:21.135 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj116 2023-10-18T20:45:21.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:x8v5fL7Dz8lBwTxu/ONIQdJDT+c= 2023-10-18T20:45:21.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x8v5fL7Dz8lBwTxu/ONIQdJDT+c=', 'Content-Length': '0'} 2023-10-18T20:45:21.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dad340a35b4d7d0-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.139 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.139 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.139 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.140 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.140 DEBUG:boto:Method: GET 2023-10-18T20:45:21.141 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.141 DEBUG:boto:Data: 2023-10-18T20:45:21.141 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.142 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.143 DEBUG:boto:Params: {} 2023-10-18T20:45:21.143 DEBUG:boto:Token: None 2023-10-18T20:45:21.144 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000689dc481792ac57c-00653043e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.147 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.148 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.149 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.149 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.150 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.150 DEBUG:boto:Method: GET 2023-10-18T20:45:21.150 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:45:21.151 DEBUG:boto:Data: 2023-10-18T20:45:21.151 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.152 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.152 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.153 DEBUG:boto:Params: {} 2023-10-18T20:45:21.153 DEBUG:boto:Token: None 2023-10-18T20:45:21.153 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.154 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005f7b78e7c894276-00653043e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.159 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-18T20:45:21.160 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.161 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.161 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.162 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.162 DEBUG:boto:Method: GET 2023-10-18T20:45:21.163 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.163 DEBUG:boto:Data: 2023-10-18T20:45:21.164 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.164 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.164 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.165 DEBUG:boto:Params: {} 2023-10-18T20:45:21.165 DEBUG:boto:Token: None 2023-10-18T20:45:21.166 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.166 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lrocE4bqko7mcChdjIdgnaVud/Y= 2023-10-18T20:45:21.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lrocE4bqko7mcChdjIdgnaVud/Y=', 'Content-Length': '0'} 2023-10-18T20:45:21.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d788400c062c1509-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.170 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.170 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.171 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.171 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.171 DEBUG:boto:Method: GET 2023-10-18T20:45:21.172 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.172 DEBUG:boto:Data: 2023-10-18T20:45:21.173 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.173 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.173 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.174 DEBUG:boto:Params: {} 2023-10-18T20:45:21.174 DEBUG:boto:Token: None 2023-10-18T20:45:21.175 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.175 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lrocE4bqko7mcChdjIdgnaVud/Y= 2023-10-18T20:45:21.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lrocE4bqko7mcChdjIdgnaVud/Y=', 'Content-Length': '0'} 2023-10-18T20:45:21.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d75d2e676fb1cdd0-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.179 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.179 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.179 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.180 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.180 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.181 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.181 DEBUG:boto:Data: 2023-10-18T20:45:21.181 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.182 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.182 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.182 DEBUG:boto:Params: {} 2023-10-18T20:45:21.183 DEBUG:boto:Token: None 2023-10-18T20:45:21.183 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.184 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LH7pev+Z/8ZoPY1FRjQik3DBGks= 2023-10-18T20:45:21.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LH7pev+Z/8ZoPY1FRjQik3DBGks=', 'Content-Length': '0'} 2023-10-18T20:45:21.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8567ea4ec027dc-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.187 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.187 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.188 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.188 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.188 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.189 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.189 DEBUG:boto:Data: 2023-10-18T20:45:21.190 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.190 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.191 DEBUG:boto:Params: {} 2023-10-18T20:45:21.191 DEBUG:boto:Token: None 2023-10-18T20:45:21.192 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LH7pev+Z/8ZoPY1FRjQik3DBGks= 2023-10-18T20:45:21.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LH7pev+Z/8ZoPY1FRjQik3DBGks=', 'Content-Length': '0'} 2023-10-18T20:45:21.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4525c09a4eb3394-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-18T20:45:21.195 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.196 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.196 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.197 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.197 DEBUG:boto:Method: GET 2023-10-18T20:45:21.198 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.198 DEBUG:boto:Data: 2023-10-18T20:45:21.198 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.199 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.200 DEBUG:boto:Params: {} 2023-10-18T20:45:21.200 DEBUG:boto:Token: None 2023-10-18T20:45:21.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lrocE4bqko7mcChdjIdgnaVud/Y= 2023-10-18T20:45:21.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lrocE4bqko7mcChdjIdgnaVud/Y=', 'Content-Length': '0'} 2023-10-18T20:45:21.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a450c922d78114-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.246 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.247 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.247 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.248 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.248 DEBUG:boto:Method: GET 2023-10-18T20:45:21.249 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.249 DEBUG:boto:Data: 2023-10-18T20:45:21.250 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.250 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.250 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.251 DEBUG:boto:Params: {} 2023-10-18T20:45:21.251 DEBUG:boto:Token: None 2023-10-18T20:45:21.252 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:21.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lrocE4bqko7mcChdjIdgnaVud/Y= 2023-10-18T20:45:21.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lrocE4bqko7mcChdjIdgnaVud/Y=', 'Content-Length': '0'} 2023-10-18T20:45:21.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000917bc74e64c728f6-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.256 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.256 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.257 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.257 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.257 DEBUG:boto:Method: GET 2023-10-18T20:45:21.258 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.258 DEBUG:boto:Data: 2023-10-18T20:45:21.259 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.259 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.259 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.260 DEBUG:boto:Params: {} 2023-10-18T20:45:21.260 DEBUG:boto:Token: None 2023-10-18T20:45:21.261 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.261 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005151f76cf34fa684-00653043e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.265 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.266 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.266 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.267 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.267 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.267 DEBUG:boto:Method: GET 2023-10-18T20:45:21.268 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.268 DEBUG:boto:Data: 2023-10-18T20:45:21.269 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.270 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.270 DEBUG:boto:Params: {} 2023-10-18T20:45:21.270 DEBUG:boto:Token: None 2023-10-18T20:45:21.271 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.271 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090325c41f3c6d087-00653043e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.318 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.319 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.320 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.320 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.321 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:45:21.321 DEBUG:boto:Data: 2023-10-18T20:45:21.321 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.322 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.323 DEBUG:boto:Params: {} 2023-10-18T20:45:21.323 DEBUG:boto:Token: None 2023-10-18T20:45:21.324 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124 2023-10-18T20:45:21.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P6MIqSj5PgBby5ZWLpHJUsQranY= 2023-10-18T20:45:21.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P6MIqSj5PgBby5ZWLpHJUsQranY=', 'Content-Length': '0'} 2023-10-18T20:45:21.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70a43cf89ae879a-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.327 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.328 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.328 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.329 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:45:21.329 DEBUG:boto:Data: 2023-10-18T20:45:21.329 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.330 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.330 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.331 DEBUG:boto:Params: {} 2023-10-18T20:45:21.331 DEBUG:boto:Token: None 2023-10-18T20:45:21.331 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124 2023-10-18T20:45:21.332 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P6MIqSj5PgBby5ZWLpHJUsQranY= 2023-10-18T20:45:21.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P6MIqSj5PgBby5ZWLpHJUsQranY=', 'Content-Length': '0'} 2023-10-18T20:45:21.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000474debf87c47af11-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-18T20:45:21.335 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.336 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.336 DEBUG:boto:Method: GET 2023-10-18T20:45:21.336 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:45:21.337 DEBUG:boto:Data: 2023-10-18T20:45:21.337 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.338 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.339 DEBUG:boto:Params: {} 2023-10-18T20:45:21.339 DEBUG:boto:Token: None 2023-10-18T20:45:21.339 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124 2023-10-18T20:45:21.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8YeAQBzunWBNIErsP0FXvMFPjh4= 2023-10-18T20:45:21.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8YeAQBzunWBNIErsP0FXvMFPjh4=', 'Content-Length': '0'} 2023-10-18T20:45:21.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b05be0440137f3-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.343 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:21.343 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:21.344 DEBUG:boto:Method: GET 2023-10-18T20:45:21.344 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:45:21.345 DEBUG:boto:Data: 2023-10-18T20:45:21.345 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.346 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.346 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.347 DEBUG:boto:Params: {} 2023-10-18T20:45:21.347 DEBUG:boto:Token: None 2023-10-18T20:45:21.348 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj124 2023-10-18T20:45:21.348 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8YeAQBzunWBNIErsP0FXvMFPjh4= 2023-10-18T20:45:21.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8YeAQBzunWBNIErsP0FXvMFPjh4=', 'Content-Length': '0'} 2023-10-18T20:45:21.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003939e0a4a8b01cb1-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.351 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.352 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.352 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.353 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.353 DEBUG:boto:Method: GET 2023-10-18T20:45:21.354 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.354 DEBUG:boto:Data: 2023-10-18T20:45:21.354 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.355 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.355 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.356 DEBUG:boto:Params: {} 2023-10-18T20:45:21.356 DEBUG:boto:Token: None 2023-10-18T20:45:21.357 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.357 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e973f74853131468-00653043e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.402 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.403 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.404 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.404 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.405 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.405 DEBUG:boto:Method: GET 2023-10-18T20:45:21.405 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:45:21.406 DEBUG:boto:Data: 2023-10-18T20:45:21.406 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.407 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.407 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.408 DEBUG:boto:Params: {} 2023-10-18T20:45:21.408 DEBUG:boto:Token: None 2023-10-18T20:45:21.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.409 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000470b3e3fca1bcc52-00653043e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.416 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:45:21.417 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.417 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.418 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.418 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.418 DEBUG:boto:Method: GET 2023-10-18T20:45:21.419 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.419 DEBUG:boto:Data: 2023-10-18T20:45:21.420 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.420 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.421 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.421 DEBUG:boto:Params: {} 2023-10-18T20:45:21.421 DEBUG:boto:Token: None 2023-10-18T20:45:21.422 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.422 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mtZY3WuGcnMiMvetyVa++HK6wNk= 2023-10-18T20:45:21.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mtZY3WuGcnMiMvetyVa++HK6wNk=', 'Content-Length': '0'} 2023-10-18T20:45:21.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000433b5a934e82b30c-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.426 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.426 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.427 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.427 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.428 DEBUG:boto:Method: GET 2023-10-18T20:45:21.428 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.428 DEBUG:boto:Data: 2023-10-18T20:45:21.429 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.429 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.430 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.430 DEBUG:boto:Params: {} 2023-10-18T20:45:21.431 DEBUG:boto:Token: None 2023-10-18T20:45:21.431 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.432 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mtZY3WuGcnMiMvetyVa++HK6wNk= 2023-10-18T20:45:21.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mtZY3WuGcnMiMvetyVa++HK6wNk=', 'Content-Length': '0'} 2023-10-18T20:45:21.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b02df90ef3bf6d3d-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.478 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.479 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.479 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.480 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.480 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.481 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.481 DEBUG:boto:Data: 2023-10-18T20:45:21.481 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.482 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.482 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.483 DEBUG:boto:Params: {} 2023-10-18T20:45:21.483 DEBUG:boto:Token: None 2023-10-18T20:45:21.484 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.484 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M+9FU2IX68NrpIPCPPqcJpLOCm8= 2023-10-18T20:45:21.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M+9FU2IX68NrpIPCPPqcJpLOCm8=', 'Content-Length': '0'} 2023-10-18T20:45:21.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a1b10ac8572da3-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.487 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.488 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.488 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.489 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.489 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.489 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.490 DEBUG:boto:Data: 2023-10-18T20:45:21.490 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.493 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.494 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.494 DEBUG:boto:Params: {} 2023-10-18T20:45:21.495 DEBUG:boto:Token: None 2023-10-18T20:45:21.495 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.496 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M+9FU2IX68NrpIPCPPqcJpLOCm8= 2023-10-18T20:45:21.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M+9FU2IX68NrpIPCPPqcJpLOCm8=', 'Content-Length': '0'} 2023-10-18T20:45:21.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeccbe2acc81b9eb-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:45:21.500 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.500 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.500 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.501 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.501 DEBUG:boto:Method: GET 2023-10-18T20:45:21.501 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.502 DEBUG:boto:Data: 2023-10-18T20:45:21.502 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.503 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.504 DEBUG:boto:Params: {} 2023-10-18T20:45:21.504 DEBUG:boto:Token: None 2023-10-18T20:45:21.505 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mtZY3WuGcnMiMvetyVa++HK6wNk= 2023-10-18T20:45:21.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mtZY3WuGcnMiMvetyVa++HK6wNk=', 'Content-Length': '0'} 2023-10-18T20:45:21.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d318b228a851a239-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.508 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.509 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.509 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.509 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.510 DEBUG:boto:Method: GET 2023-10-18T20:45:21.510 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.512 DEBUG:boto:Data: 2023-10-18T20:45:21.512 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.512 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.513 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.513 DEBUG:boto:Params: {} 2023-10-18T20:45:21.514 DEBUG:boto:Token: None 2023-10-18T20:45:21.514 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:21.515 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mtZY3WuGcnMiMvetyVa++HK6wNk= 2023-10-18T20:45:21.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mtZY3WuGcnMiMvetyVa++HK6wNk=', 'Content-Length': '0'} 2023-10-18T20:45:21.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a83d938224a31a-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.518 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.518 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.519 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.519 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.520 DEBUG:boto:Method: GET 2023-10-18T20:45:21.520 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.521 DEBUG:boto:Data: 2023-10-18T20:45:21.521 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.522 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.523 DEBUG:boto:Params: {} 2023-10-18T20:45:21.523 DEBUG:boto:Token: None 2023-10-18T20:45:21.524 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000735fbbf456dc30c3-00653043e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.528 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.528 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.529 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.529 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.530 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.530 DEBUG:boto:Method: GET 2023-10-18T20:45:21.531 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.531 DEBUG:boto:Data: 2023-10-18T20:45:21.532 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.532 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.532 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.533 DEBUG:boto:Params: {} 2023-10-18T20:45:21.533 DEBUG:boto:Token: None 2023-10-18T20:45:21.534 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.534 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007695a142d8c7a83f-00653043e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.540 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.540 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.541 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.541 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.541 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:45:21.542 DEBUG:boto:Data: 2023-10-18T20:45:21.542 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.543 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.543 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.543 DEBUG:boto:Params: {} 2023-10-18T20:45:21.544 DEBUG:boto:Token: None 2023-10-18T20:45:21.544 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145 2023-10-18T20:45:21.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ruPINSOJXxA08lW5Tkkd+YnsJ2I= 2023-10-18T20:45:21.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ruPINSOJXxA08lW5Tkkd+YnsJ2I=', 'Content-Length': '0'} 2023-10-18T20:45:21.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f07feb72fa90007-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.548 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.548 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.549 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.549 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:45:21.549 DEBUG:boto:Data: 2023-10-18T20:45:21.550 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.551 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.551 DEBUG:boto:Params: {} 2023-10-18T20:45:21.552 DEBUG:boto:Token: None 2023-10-18T20:45:21.552 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145 2023-10-18T20:45:21.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ruPINSOJXxA08lW5Tkkd+YnsJ2I= 2023-10-18T20:45:21.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ruPINSOJXxA08lW5Tkkd+YnsJ2I=', 'Content-Length': '0'} 2023-10-18T20:45:21.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bc2b06ac68f350f-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:45:21.556 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.557 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.557 DEBUG:boto:Method: GET 2023-10-18T20:45:21.557 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:45:21.558 DEBUG:boto:Data: 2023-10-18T20:45:21.558 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.559 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.559 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.560 DEBUG:boto:Params: {} 2023-10-18T20:45:21.560 DEBUG:boto:Token: None 2023-10-18T20:45:21.560 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145 2023-10-18T20:45:21.561 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utf72BgmABeMfFyd43AjpFfF56Q= 2023-10-18T20:45:21.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utf72BgmABeMfFyd43AjpFfF56Q=', 'Content-Length': '0'} 2023-10-18T20:45:21.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084ae8397a7cae3ee-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.610 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:21.611 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:21.611 DEBUG:boto:Method: GET 2023-10-18T20:45:21.612 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:45:21.612 DEBUG:boto:Data: 2023-10-18T20:45:21.613 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.613 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.613 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.614 DEBUG:boto:Params: {} 2023-10-18T20:45:21.614 DEBUG:boto:Token: None 2023-10-18T20:45:21.615 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj145 2023-10-18T20:45:21.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:utf72BgmABeMfFyd43AjpFfF56Q= 2023-10-18T20:45:21.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:utf72BgmABeMfFyd43AjpFfF56Q=', 'Content-Length': '0'} 2023-10-18T20:45:21.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097bba63ae250a53c-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.619 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.619 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.620 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.620 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.620 DEBUG:boto:Method: GET 2023-10-18T20:45:21.621 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.621 DEBUG:boto:Data: 2023-10-18T20:45:21.622 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.622 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.623 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.623 DEBUG:boto:Params: {} 2023-10-18T20:45:21.624 DEBUG:boto:Token: None 2023-10-18T20:45:21.624 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.625 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b24cd408dab8a9ae-00653043e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.629 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.629 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.630 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.630 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.631 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.631 DEBUG:boto:Method: GET 2023-10-18T20:45:21.632 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:45:21.632 DEBUG:boto:Data: 2023-10-18T20:45:21.632 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.633 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.633 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.634 DEBUG:boto:Params: {} 2023-10-18T20:45:21.634 DEBUG:boto:Token: None 2023-10-18T20:45:21.635 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.635 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d14f8fbeb7916fc0-00653043e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.641 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-18T20:45:21.642 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.642 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.643 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.643 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.644 DEBUG:boto:Method: GET 2023-10-18T20:45:21.644 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.644 DEBUG:boto:Data: 2023-10-18T20:45:21.645 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.645 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.646 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.646 DEBUG:boto:Params: {} 2023-10-18T20:45:21.647 DEBUG:boto:Token: None 2023-10-18T20:45:21.647 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.648 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wF701EUheOWQKWhcVlMLQKR6J18= 2023-10-18T20:45:21.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wF701EUheOWQKWhcVlMLQKR6J18=', 'Content-Length': '0'} 2023-10-18T20:45:21.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036ace3dbc6942cf9-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.651 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.652 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.652 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.652 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.653 DEBUG:boto:Method: GET 2023-10-18T20:45:21.653 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.654 DEBUG:boto:Data: 2023-10-18T20:45:21.654 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.655 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.655 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.656 DEBUG:boto:Params: {} 2023-10-18T20:45:21.656 DEBUG:boto:Token: None 2023-10-18T20:45:21.657 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.657 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wF701EUheOWQKWhcVlMLQKR6J18= 2023-10-18T20:45:21.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wF701EUheOWQKWhcVlMLQKR6J18=', 'Content-Length': '0'} 2023-10-18T20:45:21.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5db1447acbfb14-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.702 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.703 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.703 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.704 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.704 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.705 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.705 DEBUG:boto:Data: 2023-10-18T20:45:21.706 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.706 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.707 DEBUG:boto:Params: {} 2023-10-18T20:45:21.707 DEBUG:boto:Token: None 2023-10-18T20:45:21.708 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.708 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WDU0lEZnuqizmWQe3tgXU3/t2JE= 2023-10-18T20:45:21.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WDU0lEZnuqizmWQe3tgXU3/t2JE=', 'Content-Length': '0'} 2023-10-18T20:45:21.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059233c224bf66b80-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.712 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.712 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.713 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.713 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.713 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.714 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.714 DEBUG:boto:Data: 2023-10-18T20:45:21.715 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.715 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.716 DEBUG:boto:Params: {} 2023-10-18T20:45:21.716 DEBUG:boto:Token: None 2023-10-18T20:45:21.717 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WDU0lEZnuqizmWQe3tgXU3/t2JE= 2023-10-18T20:45:21.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WDU0lEZnuqizmWQe3tgXU3/t2JE=', 'Content-Length': '0'} 2023-10-18T20:45:21.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000728d1c305238b543-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-18T20:45:21.721 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.722 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.722 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.722 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.723 DEBUG:boto:Method: GET 2023-10-18T20:45:21.723 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.724 DEBUG:boto:Data: 2023-10-18T20:45:21.724 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.725 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.725 DEBUG:boto:Params: {} 2023-10-18T20:45:21.726 DEBUG:boto:Token: None 2023-10-18T20:45:21.726 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.727 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wF701EUheOWQKWhcVlMLQKR6J18= 2023-10-18T20:45:21.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wF701EUheOWQKWhcVlMLQKR6J18=', 'Content-Length': '0'} 2023-10-18T20:45:21.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c02616ad08a484f-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.770 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.771 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.772 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.772 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.772 DEBUG:boto:Method: GET 2023-10-18T20:45:21.773 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.773 DEBUG:boto:Data: 2023-10-18T20:45:21.774 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.774 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.775 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.775 DEBUG:boto:Params: {} 2023-10-18T20:45:21.776 DEBUG:boto:Token: None 2023-10-18T20:45:21.776 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:21.777 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wF701EUheOWQKWhcVlMLQKR6J18= 2023-10-18T20:45:21.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wF701EUheOWQKWhcVlMLQKR6J18=', 'Content-Length': '0'} 2023-10-18T20:45:21.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc6769c8610a027e-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.780 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.781 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.781 DEBUG:boto:path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.781 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.782 DEBUG:boto:Method: GET 2023-10-18T20:45:21.782 DEBUG:boto:Path: /bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.783 DEBUG:boto:Data: 2023-10-18T20:45:21.783 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.784 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.784 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.784 DEBUG:boto:Params: {} 2023-10-18T20:45:21.785 DEBUG:boto:Token: None 2023-10-18T20:45:21.785 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.786 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b341050017b8f2-00653043e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.790 DEBUG:boto:b'bofrnf-59obj1481000falseobj1482023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.790 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.791 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.791 DEBUG:boto:path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.792 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.792 DEBUG:boto:Method: GET 2023-10-18T20:45:21.793 DEBUG:boto:Path: /bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.793 DEBUG:boto:Data: 2023-10-18T20:45:21.793 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.794 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.794 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.795 DEBUG:boto:Params: {} 2023-10-18T20:45:21.795 DEBUG:boto:Token: None 2023-10-18T20:45:21.796 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.796 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000935756c448a804fa-00653043e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.801 DEBUG:boto:b'bofrnf-59obj1481000falseobj1482023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.802 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.803 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.803 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.803 DEBUG:boto:Path: /bofrnf-59/obj148 2023-10-18T20:45:21.804 DEBUG:boto:Data: 2023-10-18T20:45:21.804 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.805 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.805 DEBUG:boto:Params: {} 2023-10-18T20:45:21.806 DEBUG:boto:Token: None 2023-10-18T20:45:21.806 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148 2023-10-18T20:45:21.807 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z4rlIXIA+z4zgQGOnxlcm3GNpaY= 2023-10-18T20:45:21.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z4rlIXIA+z4zgQGOnxlcm3GNpaY=', 'Content-Length': '0'} 2023-10-18T20:45:21.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb9051f291a9952d-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.810 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.811 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.811 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.812 DEBUG:boto:Path: /bofrnf-59/obj148 2023-10-18T20:45:21.812 DEBUG:boto:Data: 2023-10-18T20:45:21.812 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.813 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.813 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.814 DEBUG:boto:Params: {} 2023-10-18T20:45:21.814 DEBUG:boto:Token: None 2023-10-18T20:45:21.815 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148 2023-10-18T20:45:21.815 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z4rlIXIA+z4zgQGOnxlcm3GNpaY= 2023-10-18T20:45:21.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z4rlIXIA+z4zgQGOnxlcm3GNpaY=', 'Content-Length': '0'} 2023-10-18T20:45:21.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd617f9ca280106c-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-18T20:45:21.818 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.819 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.819 DEBUG:boto:Method: GET 2023-10-18T20:45:21.820 DEBUG:boto:Path: /bofrnf-59/obj148 2023-10-18T20:45:21.820 DEBUG:boto:Data: 2023-10-18T20:45:21.821 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.821 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.822 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.822 DEBUG:boto:Params: {} 2023-10-18T20:45:21.823 DEBUG:boto:Token: None 2023-10-18T20:45:21.823 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148 2023-10-18T20:45:21.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lOIQz8fEwNqc7UFvWgIzqWBmlAk= 2023-10-18T20:45:21.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lOIQz8fEwNqc7UFvWgIzqWBmlAk=', 'Content-Length': '0'} 2023-10-18T20:45:21.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baff2a8e208c05b1-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.827 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:21.827 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:21.828 DEBUG:boto:Method: GET 2023-10-18T20:45:21.828 DEBUG:boto:Path: /bofrnf-59/obj148 2023-10-18T20:45:21.828 DEBUG:boto:Data: 2023-10-18T20:45:21.829 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.829 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.830 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.830 DEBUG:boto:Params: {} 2023-10-18T20:45:21.831 DEBUG:boto:Token: None 2023-10-18T20:45:21.831 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj148 2023-10-18T20:45:21.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lOIQz8fEwNqc7UFvWgIzqWBmlAk= 2023-10-18T20:45:21.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lOIQz8fEwNqc7UFvWgIzqWBmlAk=', 'Content-Length': '0'} 2023-10-18T20:45:21.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5fdd765409c9fd4-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.835 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.835 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.836 DEBUG:boto:path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.836 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.836 DEBUG:boto:Method: GET 2023-10-18T20:45:21.837 DEBUG:boto:Path: /bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.837 DEBUG:boto:Data: 2023-10-18T20:45:21.838 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.839 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.839 DEBUG:boto:Params: {} 2023-10-18T20:45:21.840 DEBUG:boto:Token: None 2023-10-18T20:45:21.840 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.841 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000889ab698282ae799-00653043e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.844 DEBUG:boto:b'bofrnf-59obj1481000falseobj1482023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.845 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.845 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.846 DEBUG:boto:path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.846 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.847 DEBUG:boto:Method: GET 2023-10-18T20:45:21.847 DEBUG:boto:Path: /bofrnf-59/?prefix=obj148 2023-10-18T20:45:21.847 DEBUG:boto:Data: 2023-10-18T20:45:21.848 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.849 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.849 DEBUG:boto:Params: {} 2023-10-18T20:45:21.850 DEBUG:boto:Token: None 2023-10-18T20:45:21.850 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.851 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d71b196f0bdc9437-00653043e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.898 DEBUG:boto:b'bofrnf-59obj1481000falseobj1482023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-18T20:45:21.900 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:21.901 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:21.901 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.901 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.902 DEBUG:boto:Method: GET 2023-10-18T20:45:21.902 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.903 DEBUG:boto:Data: 2023-10-18T20:45:21.903 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.905 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.906 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.906 DEBUG:boto:Params: {} 2023-10-18T20:45:21.907 DEBUG:boto:Token: None 2023-10-18T20:45:21.907 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.908 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pVhYHvvPkCuGDk+en4NsHFBHayw= 2023-10-18T20:45:21.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pVhYHvvPkCuGDk+en4NsHFBHayw=', 'Content-Length': '0'} 2023-10-18T20:45:21.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004510836a6a068813-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.912 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:21.912 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:21.913 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.913 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.914 DEBUG:boto:Method: GET 2023-10-18T20:45:21.914 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.914 DEBUG:boto:Data: 2023-10-18T20:45:21.915 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.916 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.916 DEBUG:boto:Params: {} 2023-10-18T20:45:21.917 DEBUG:boto:Token: None 2023-10-18T20:45:21.917 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.918 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pVhYHvvPkCuGDk+en4NsHFBHayw= 2023-10-18T20:45:21.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pVhYHvvPkCuGDk+en4NsHFBHayw=', 'Content-Length': '0'} 2023-10-18T20:45:21.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2eada38c590b221-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.921 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:21.922 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:21.922 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.922 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.923 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.923 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.924 DEBUG:boto:Data: 2023-10-18T20:45:21.924 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.924 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.925 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.925 DEBUG:boto:Params: {} 2023-10-18T20:45:21.926 DEBUG:boto:Token: None 2023-10-18T20:45:21.926 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kSkS5gfIlpBKx65bTUwaOARbUug= 2023-10-18T20:45:21.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kSkS5gfIlpBKx65bTUwaOARbUug=', 'Content-Length': '0'} 2023-10-18T20:45:21.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9a126159f16168d-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.930 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:21.930 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:21.931 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.931 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.931 DEBUG:boto:Method: HEAD 2023-10-18T20:45:21.932 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.933 DEBUG:boto:Data: 2023-10-18T20:45:21.933 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.934 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.934 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.935 DEBUG:boto:Params: {} 2023-10-18T20:45:21.935 DEBUG:boto:Token: None 2023-10-18T20:45:21.936 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kSkS5gfIlpBKx65bTUwaOARbUug= 2023-10-18T20:45:21.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kSkS5gfIlpBKx65bTUwaOARbUug=', 'Content-Length': '0'} 2023-10-18T20:45:21.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c79ada017055262a-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-18T20:45:21.940 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:21.941 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:21.941 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.942 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.942 DEBUG:boto:Method: GET 2023-10-18T20:45:21.943 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.943 DEBUG:boto:Data: 2023-10-18T20:45:21.944 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.944 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.945 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.945 DEBUG:boto:Params: {} 2023-10-18T20:45:21.945 DEBUG:boto:Token: None 2023-10-18T20:45:21.946 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.946 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pVhYHvvPkCuGDk+en4NsHFBHayw= 2023-10-18T20:45:21.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pVhYHvvPkCuGDk+en4NsHFBHayw=', 'Content-Length': '0'} 2023-10-18T20:45:21.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064bcb99eb18b72e3-00653043e1-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.950 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:21.950 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:21.951 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.951 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.951 DEBUG:boto:Method: GET 2023-10-18T20:45:21.952 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.952 DEBUG:boto:Data: 2023-10-18T20:45:21.953 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.953 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.954 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.954 DEBUG:boto:Params: {} 2023-10-18T20:45:21.955 DEBUG:boto:Token: None 2023-10-18T20:45:21.955 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:21.955 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pVhYHvvPkCuGDk+en4NsHFBHayw= 2023-10-18T20:45:21.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pVhYHvvPkCuGDk+en4NsHFBHayw=', 'Content-Length': '0'} 2023-10-18T20:45:21.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000096d8a5c47775de9-00653043e1-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.959 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.959 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.960 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:21.960 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:21.961 DEBUG:boto:Method: GET 2023-10-18T20:45:21.961 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:45:21.961 DEBUG:boto:Data: 2023-10-18T20:45:21.962 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.962 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:21.963 DEBUG:boto:Port: 8001 2023-10-18T20:45:21.963 DEBUG:boto:Params: {} 2023-10-18T20:45:21.963 DEBUG:boto:Token: None 2023-10-18T20:45:21.964 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.964 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000729c196782210c38-00653043e1-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:21.968 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:21.969 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:21.969 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:21.969 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:21.970 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:21.970 DEBUG:boto:Method: GET 2023-10-18T20:45:21.971 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:45:21.971 DEBUG:boto:Data: 2023-10-18T20:45:21.972 DEBUG:boto:Headers: {} 2023-10-18T20:45:21.972 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:21.972 DEBUG:boto:Port: 8000 2023-10-18T20:45:21.973 DEBUG:boto:Params: {} 2023-10-18T20:45:21.973 DEBUG:boto:Token: None 2023-10-18T20:45:21.974 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:21 GMT /bofrnf-59/ 2023-10-18T20:45:21.974 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs= 2023-10-18T20:45:21.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CGkPJ9gvFxBPcEJB0hUQSRq9WEs=', 'Content-Length': '0'} 2023-10-18T20:45:21.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f31c4c71414ff8d0-00653043e1-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.026 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.027 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:22.028 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:22.028 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.029 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:45:22.029 DEBUG:boto:Data: 2023-10-18T20:45:22.029 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.030 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.030 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.031 DEBUG:boto:Params: {} 2023-10-18T20:45:22.031 DEBUG:boto:Token: None 2023-10-18T20:45:22.032 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj155 2023-10-18T20:45:22.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tCbioHeLuZJK6b6vUjWR+dRPaS4= 2023-10-18T20:45:22.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tCbioHeLuZJK6b6vUjWR+dRPaS4=', 'Content-Length': '0'} 2023-10-18T20:45:22.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad2df239923e2ff2-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.036 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:22.036 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:22.036 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.037 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:45:22.037 DEBUG:boto:Data: 2023-10-18T20:45:22.037 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.038 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.038 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.039 DEBUG:boto:Params: {} 2023-10-18T20:45:22.039 DEBUG:boto:Token: None 2023-10-18T20:45:22.039 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj155 2023-10-18T20:45:22.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tCbioHeLuZJK6b6vUjWR+dRPaS4= 2023-10-18T20:45:22.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tCbioHeLuZJK6b6vUjWR+dRPaS4=', 'Content-Length': '0'} 2023-10-18T20:45:22.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ed134eef6bbd53-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-18T20:45:22.043 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:22.044 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:22.044 DEBUG:boto:Method: GET 2023-10-18T20:45:22.045 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:45:22.045 DEBUG:boto:Data: 2023-10-18T20:45:22.045 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.046 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.046 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.047 DEBUG:boto:Params: {} 2023-10-18T20:45:22.047 DEBUG:boto:Token: None 2023-10-18T20:45:22.047 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj155 2023-10-18T20:45:22.048 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fcWUebLUWkImIpq35e1F5sqpaRU= 2023-10-18T20:45:22.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fcWUebLUWkImIpq35e1F5sqpaRU=', 'Content-Length': '0'} 2023-10-18T20:45:22.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e2365961348cf3-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.094 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:22.095 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:22.095 DEBUG:boto:Method: GET 2023-10-18T20:45:22.096 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:45:22.096 DEBUG:boto:Data: 2023-10-18T20:45:22.097 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.097 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.097 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.098 DEBUG:boto:Params: {} 2023-10-18T20:45:22.098 DEBUG:boto:Token: None 2023-10-18T20:45:22.099 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj155 2023-10-18T20:45:22.099 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fcWUebLUWkImIpq35e1F5sqpaRU= 2023-10-18T20:45:22.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fcWUebLUWkImIpq35e1F5sqpaRU=', 'Content-Length': '0'} 2023-10-18T20:45:22.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b59eefd6d83d7c-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.103 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.104 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.104 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:22.105 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:22.105 DEBUG:boto:Method: GET 2023-10-18T20:45:22.106 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:45:22.106 DEBUG:boto:Data: 2023-10-18T20:45:22.107 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.107 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.107 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.108 DEBUG:boto:Params: {} 2023-10-18T20:45:22.108 DEBUG:boto:Token: None 2023-10-18T20:45:22.109 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.109 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e2d072ce3b0ccab-00653043e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.113 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.114 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.114 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.115 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:22.115 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:22.115 DEBUG:boto:Method: GET 2023-10-18T20:45:22.116 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:45:22.116 DEBUG:boto:Data: 2023-10-18T20:45:22.117 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.117 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.118 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.118 DEBUG:boto:Params: {} 2023-10-18T20:45:22.119 DEBUG:boto:Token: None 2023-10-18T20:45:22.119 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.120 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051df84f38ba66376-00653043e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.125 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:45:22.126 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.127 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.127 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.127 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.128 DEBUG:boto:Method: GET 2023-10-18T20:45:22.128 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.129 DEBUG:boto:Data: 2023-10-18T20:45:22.129 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.130 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.130 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.130 DEBUG:boto:Params: {} 2023-10-18T20:45:22.131 DEBUG:boto:Token: None 2023-10-18T20:45:22.131 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.132 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ni+Voew5JxDqqN2j22AEd0iAZvU= 2023-10-18T20:45:22.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ni+Voew5JxDqqN2j22AEd0iAZvU=', 'Content-Length': '0'} 2023-10-18T20:45:22.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae5c939bc64dfd7c-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.135 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.135 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.136 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.136 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.137 DEBUG:boto:Method: GET 2023-10-18T20:45:22.137 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.138 DEBUG:boto:Data: 2023-10-18T20:45:22.138 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.139 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.139 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.139 DEBUG:boto:Params: {} 2023-10-18T20:45:22.140 DEBUG:boto:Token: None 2023-10-18T20:45:22.140 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.141 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ni+Voew5JxDqqN2j22AEd0iAZvU= 2023-10-18T20:45:22.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ni+Voew5JxDqqN2j22AEd0iAZvU=', 'Content-Length': '0'} 2023-10-18T20:45:22.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2c0113fd464ed9-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.186 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.187 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.187 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.188 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.188 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.189 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.189 DEBUG:boto:Data: 2023-10-18T20:45:22.189 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.190 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.191 DEBUG:boto:Params: {} 2023-10-18T20:45:22.191 DEBUG:boto:Token: None 2023-10-18T20:45:22.192 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UwOu9OusJrnnyr9IPYUbV5ccrNA= 2023-10-18T20:45:22.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UwOu9OusJrnnyr9IPYUbV5ccrNA=', 'Content-Length': '0'} 2023-10-18T20:45:22.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a00a5d2b90bf8d9-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.195 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.196 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.196 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.196 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.197 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.197 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.198 DEBUG:boto:Data: 2023-10-18T20:45:22.198 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.198 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.199 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.199 DEBUG:boto:Params: {} 2023-10-18T20:45:22.199 DEBUG:boto:Token: None 2023-10-18T20:45:22.200 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.200 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UwOu9OusJrnnyr9IPYUbV5ccrNA= 2023-10-18T20:45:22.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UwOu9OusJrnnyr9IPYUbV5ccrNA=', 'Content-Length': '0'} 2023-10-18T20:45:22.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007063d5bb933c7639-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:45:22.204 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.204 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.205 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.205 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.205 DEBUG:boto:Method: GET 2023-10-18T20:45:22.206 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.206 DEBUG:boto:Data: 2023-10-18T20:45:22.207 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.207 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.208 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.208 DEBUG:boto:Params: {} 2023-10-18T20:45:22.208 DEBUG:boto:Token: None 2023-10-18T20:45:22.209 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.209 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ni+Voew5JxDqqN2j22AEd0iAZvU= 2023-10-18T20:45:22.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ni+Voew5JxDqqN2j22AEd0iAZvU=', 'Content-Length': '0'} 2023-10-18T20:45:22.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f51fa98c0ef7f79-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.213 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.213 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.214 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.214 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.214 DEBUG:boto:Method: GET 2023-10-18T20:45:22.215 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.215 DEBUG:boto:Data: 2023-10-18T20:45:22.216 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.216 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.216 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.217 DEBUG:boto:Params: {} 2023-10-18T20:45:22.217 DEBUG:boto:Token: None 2023-10-18T20:45:22.218 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:22.218 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ni+Voew5JxDqqN2j22AEd0iAZvU= 2023-10-18T20:45:22.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ni+Voew5JxDqqN2j22AEd0iAZvU=', 'Content-Length': '0'} 2023-10-18T20:45:22.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000011dd3ad95f13b72-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.221 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.222 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.222 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.223 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.223 DEBUG:boto:Method: GET 2023-10-18T20:45:22.224 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.224 DEBUG:boto:Data: 2023-10-18T20:45:22.225 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.225 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.226 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.226 DEBUG:boto:Params: {} 2023-10-18T20:45:22.226 DEBUG:boto:Token: None 2023-10-18T20:45:22.227 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.227 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af7e0965b9406909-00653043e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.231 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.232 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.232 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.232 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.233 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.233 DEBUG:boto:Method: GET 2023-10-18T20:45:22.234 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.234 DEBUG:boto:Data: 2023-10-18T20:45:22.235 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.235 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.236 DEBUG:boto:Params: {} 2023-10-18T20:45:22.236 DEBUG:boto:Token: None 2023-10-18T20:45:22.236 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ced577f00617e005-00653043e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.242 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.243 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.243 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.244 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.244 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:45:22.244 DEBUG:boto:Data: 2023-10-18T20:45:22.245 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.245 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.246 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.246 DEBUG:boto:Params: {} 2023-10-18T20:45:22.246 DEBUG:boto:Token: None 2023-10-18T20:45:22.247 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17 2023-10-18T20:45:22.247 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C2HniieMSjtXPRphbEX1vf986p8= 2023-10-18T20:45:22.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C2HniieMSjtXPRphbEX1vf986p8=', 'Content-Length': '0'} 2023-10-18T20:45:22.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006a845cb085b5628-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.251 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.251 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.251 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.252 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:45:22.252 DEBUG:boto:Data: 2023-10-18T20:45:22.252 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.253 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.253 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.253 DEBUG:boto:Params: {} 2023-10-18T20:45:22.254 DEBUG:boto:Token: None 2023-10-18T20:45:22.254 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17 2023-10-18T20:45:22.255 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C2HniieMSjtXPRphbEX1vf986p8= 2023-10-18T20:45:22.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C2HniieMSjtXPRphbEX1vf986p8=', 'Content-Length': '0'} 2023-10-18T20:45:22.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c952c1708aebdb03-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:45:22.258 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.259 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.259 DEBUG:boto:Method: GET 2023-10-18T20:45:22.259 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:45:22.260 DEBUG:boto:Data: 2023-10-18T20:45:22.260 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.261 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.261 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.261 DEBUG:boto:Params: {} 2023-10-18T20:45:22.262 DEBUG:boto:Token: None 2023-10-18T20:45:22.262 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17 2023-10-18T20:45:22.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Tl0OHAWo+bigvXv+jJQtB/Sj384= 2023-10-18T20:45:22.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Tl0OHAWo+bigvXv+jJQtB/Sj384=', 'Content-Length': '0'} 2023-10-18T20:45:22.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e305b5db8bd4cda-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.306 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:22.307 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:22.307 DEBUG:boto:Method: GET 2023-10-18T20:45:22.308 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:45:22.308 DEBUG:boto:Data: 2023-10-18T20:45:22.309 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.309 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.310 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.310 DEBUG:boto:Params: {} 2023-10-18T20:45:22.310 DEBUG:boto:Token: None 2023-10-18T20:45:22.311 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj17 2023-10-18T20:45:22.311 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Tl0OHAWo+bigvXv+jJQtB/Sj384= 2023-10-18T20:45:22.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Tl0OHAWo+bigvXv+jJQtB/Sj384=', 'Content-Length': '0'} 2023-10-18T20:45:22.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008062e81db9608042-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.315 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.315 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.316 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.316 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.316 DEBUG:boto:Method: GET 2023-10-18T20:45:22.317 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.317 DEBUG:boto:Data: 2023-10-18T20:45:22.318 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.318 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.318 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.319 DEBUG:boto:Params: {} 2023-10-18T20:45:22.319 DEBUG:boto:Token: None 2023-10-18T20:45:22.320 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae390aea0d15a589-00653043e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.324 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.324 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.325 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.325 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.326 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.326 DEBUG:boto:Method: GET 2023-10-18T20:45:22.327 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:45:22.327 DEBUG:boto:Data: 2023-10-18T20:45:22.327 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.328 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.329 DEBUG:boto:Params: {} 2023-10-18T20:45:22.329 DEBUG:boto:Token: None 2023-10-18T20:45:22.330 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b2ef5f4a3713e6b-00653043e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.335 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-18T20:45:22.337 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.337 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.337 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.338 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.338 DEBUG:boto:Method: GET 2023-10-18T20:45:22.339 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.339 DEBUG:boto:Data: 2023-10-18T20:45:22.340 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.340 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.340 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.341 DEBUG:boto:Params: {} 2023-10-18T20:45:22.341 DEBUG:boto:Token: None 2023-10-18T20:45:22.342 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.342 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:exVlhvHCClqZ8Qd0BS/RpanB5XE= 2023-10-18T20:45:22.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:exVlhvHCClqZ8Qd0BS/RpanB5XE=', 'Content-Length': '0'} 2023-10-18T20:45:22.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e39349282719117-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.346 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.346 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.347 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.347 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.348 DEBUG:boto:Method: GET 2023-10-18T20:45:22.348 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.348 DEBUG:boto:Data: 2023-10-18T20:45:22.349 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.350 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.350 DEBUG:boto:Params: {} 2023-10-18T20:45:22.350 DEBUG:boto:Token: None 2023-10-18T20:45:22.351 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:exVlhvHCClqZ8Qd0BS/RpanB5XE= 2023-10-18T20:45:22.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:exVlhvHCClqZ8Qd0BS/RpanB5XE=', 'Content-Length': '0'} 2023-10-18T20:45:22.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff978a226d816dc7-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.354 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.355 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.355 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.356 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.356 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.356 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.357 DEBUG:boto:Data: 2023-10-18T20:45:22.357 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.357 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.358 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.358 DEBUG:boto:Params: {} 2023-10-18T20:45:22.359 DEBUG:boto:Token: None 2023-10-18T20:45:22.359 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SCZWBfuGL/r1voQD/3THtYrtidw= 2023-10-18T20:45:22.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SCZWBfuGL/r1voQD/3THtYrtidw=', 'Content-Length': '0'} 2023-10-18T20:45:22.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fab32bda273f4ff3-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.363 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.363 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.363 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.364 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.364 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.365 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.365 DEBUG:boto:Data: 2023-10-18T20:45:22.365 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.366 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.366 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.367 DEBUG:boto:Params: {} 2023-10-18T20:45:22.367 DEBUG:boto:Token: None 2023-10-18T20:45:22.367 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.368 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SCZWBfuGL/r1voQD/3THtYrtidw= 2023-10-18T20:45:22.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SCZWBfuGL/r1voQD/3THtYrtidw=', 'Content-Length': '0'} 2023-10-18T20:45:22.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7c683ef94e9719b-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-18T20:45:22.372 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.372 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.373 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.373 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.373 DEBUG:boto:Method: GET 2023-10-18T20:45:22.374 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.374 DEBUG:boto:Data: 2023-10-18T20:45:22.375 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.375 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.376 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.376 DEBUG:boto:Params: {} 2023-10-18T20:45:22.376 DEBUG:boto:Token: None 2023-10-18T20:45:22.377 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.377 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:exVlhvHCClqZ8Qd0BS/RpanB5XE= 2023-10-18T20:45:22.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:exVlhvHCClqZ8Qd0BS/RpanB5XE=', 'Content-Length': '0'} 2023-10-18T20:45:22.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7db9993e52f8c2-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.380 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.381 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.381 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.382 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.382 DEBUG:boto:Method: GET 2023-10-18T20:45:22.383 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.383 DEBUG:boto:Data: 2023-10-18T20:45:22.384 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.384 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.384 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.385 DEBUG:boto:Params: {} 2023-10-18T20:45:22.385 DEBUG:boto:Token: None 2023-10-18T20:45:22.386 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:22.386 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:exVlhvHCClqZ8Qd0BS/RpanB5XE= 2023-10-18T20:45:22.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:exVlhvHCClqZ8Qd0BS/RpanB5XE=', 'Content-Length': '0'} 2023-10-18T20:45:22.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ed32a5cd7b9b0a-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.389 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.390 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.390 DEBUG:boto:path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.391 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.391 DEBUG:boto:Method: GET 2023-10-18T20:45:22.392 DEBUG:boto:Path: /bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.392 DEBUG:boto:Data: 2023-10-18T20:45:22.392 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.393 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.393 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.394 DEBUG:boto:Params: {} 2023-10-18T20:45:22.394 DEBUG:boto:Token: None 2023-10-18T20:45:22.395 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.395 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3263584d696feb-00653043e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.399 DEBUG:boto:b'bofrnf-59obj1851000falseobj1852023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.399 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.400 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.400 DEBUG:boto:path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.401 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.401 DEBUG:boto:Method: GET 2023-10-18T20:45:22.402 DEBUG:boto:Path: /bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.402 DEBUG:boto:Data: 2023-10-18T20:45:22.402 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.403 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.403 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.404 DEBUG:boto:Params: {} 2023-10-18T20:45:22.404 DEBUG:boto:Token: None 2023-10-18T20:45:22.405 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.405 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506f74b948f40339-00653043e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.411 DEBUG:boto:b'bofrnf-59obj1851000falseobj1852023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.411 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.412 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.412 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.412 DEBUG:boto:Path: /bofrnf-59/obj185 2023-10-18T20:45:22.413 DEBUG:boto:Data: 2023-10-18T20:45:22.413 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.414 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.414 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.415 DEBUG:boto:Params: {} 2023-10-18T20:45:22.415 DEBUG:boto:Token: None 2023-10-18T20:45:22.415 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185 2023-10-18T20:45:22.416 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2SYlkE0anH2RfhjxMMBWuDSsq1s= 2023-10-18T20:45:22.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2SYlkE0anH2RfhjxMMBWuDSsq1s=', 'Content-Length': '0'} 2023-10-18T20:45:22.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea41d4f96706ff0b-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.419 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.419 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.420 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.420 DEBUG:boto:Path: /bofrnf-59/obj185 2023-10-18T20:45:22.421 DEBUG:boto:Data: 2023-10-18T20:45:22.421 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.422 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.422 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.422 DEBUG:boto:Params: {} 2023-10-18T20:45:22.423 DEBUG:boto:Token: None 2023-10-18T20:45:22.424 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185 2023-10-18T20:45:22.424 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2SYlkE0anH2RfhjxMMBWuDSsq1s= 2023-10-18T20:45:22.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2SYlkE0anH2RfhjxMMBWuDSsq1s=', 'Content-Length': '0'} 2023-10-18T20:45:22.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4bebc9bf9a9b364-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-18T20:45:22.428 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.428 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.429 DEBUG:boto:Method: GET 2023-10-18T20:45:22.429 DEBUG:boto:Path: /bofrnf-59/obj185 2023-10-18T20:45:22.430 DEBUG:boto:Data: 2023-10-18T20:45:22.430 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.431 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.431 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.432 DEBUG:boto:Params: {} 2023-10-18T20:45:22.432 DEBUG:boto:Token: None 2023-10-18T20:45:22.433 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185 2023-10-18T20:45:22.433 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tWvrUKMRDHot3izlVTM9eswTgFk= 2023-10-18T20:45:22.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tWvrUKMRDHot3izlVTM9eswTgFk=', 'Content-Length': '0'} 2023-10-18T20:45:22.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cf014aed7fd3f92-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.478 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:22.479 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:22.480 DEBUG:boto:Method: GET 2023-10-18T20:45:22.480 DEBUG:boto:Path: /bofrnf-59/obj185 2023-10-18T20:45:22.481 DEBUG:boto:Data: 2023-10-18T20:45:22.481 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.482 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.482 DEBUG:boto:Params: {} 2023-10-18T20:45:22.483 DEBUG:boto:Token: None 2023-10-18T20:45:22.483 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj185 2023-10-18T20:45:22.484 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tWvrUKMRDHot3izlVTM9eswTgFk= 2023-10-18T20:45:22.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tWvrUKMRDHot3izlVTM9eswTgFk=', 'Content-Length': '0'} 2023-10-18T20:45:22.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080df61b90f963a35-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.488 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.488 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.489 DEBUG:boto:path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.489 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.489 DEBUG:boto:Method: GET 2023-10-18T20:45:22.490 DEBUG:boto:Path: /bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.490 DEBUG:boto:Data: 2023-10-18T20:45:22.491 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.491 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.492 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.492 DEBUG:boto:Params: {} 2023-10-18T20:45:22.492 DEBUG:boto:Token: None 2023-10-18T20:45:22.493 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.493 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8ecbcb118ffe909-00653043e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.497 DEBUG:boto:b'bofrnf-59obj1851000falseobj1852023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.498 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.498 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.499 DEBUG:boto:path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.499 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.500 DEBUG:boto:Method: GET 2023-10-18T20:45:22.500 DEBUG:boto:Path: /bofrnf-59/?prefix=obj185 2023-10-18T20:45:22.501 DEBUG:boto:Data: 2023-10-18T20:45:22.501 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.501 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.502 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.502 DEBUG:boto:Params: {} 2023-10-18T20:45:22.503 DEBUG:boto:Token: None 2023-10-18T20:45:22.503 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.504 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bae883cf82ac1563-00653043e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.550 DEBUG:boto:b'bofrnf-59obj1851000falseobj1852023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:45:22.552 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.553 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.553 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.554 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.554 DEBUG:boto:Method: GET 2023-10-18T20:45:22.555 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.555 DEBUG:boto:Data: 2023-10-18T20:45:22.556 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.556 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.556 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.557 DEBUG:boto:Params: {} 2023-10-18T20:45:22.558 DEBUG:boto:Token: None 2023-10-18T20:45:22.558 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.559 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JMmWOGop1BhKJ32Wczb2rFhmDL0= 2023-10-18T20:45:22.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JMmWOGop1BhKJ32Wczb2rFhmDL0=', 'Content-Length': '0'} 2023-10-18T20:45:22.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec37894cb356add-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.562 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.563 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.563 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.564 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.564 DEBUG:boto:Method: GET 2023-10-18T20:45:22.564 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.565 DEBUG:boto:Data: 2023-10-18T20:45:22.565 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.566 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.567 DEBUG:boto:Params: {} 2023-10-18T20:45:22.567 DEBUG:boto:Token: None 2023-10-18T20:45:22.568 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JMmWOGop1BhKJ32Wczb2rFhmDL0= 2023-10-18T20:45:22.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JMmWOGop1BhKJ32Wczb2rFhmDL0=', 'Content-Length': '0'} 2023-10-18T20:45:22.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f4b0a98121d8bbe-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.572 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.572 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.573 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.573 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.573 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.574 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.574 DEBUG:boto:Data: 2023-10-18T20:45:22.575 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.575 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.576 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.576 DEBUG:boto:Params: {} 2023-10-18T20:45:22.576 DEBUG:boto:Token: None 2023-10-18T20:45:22.577 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ocuDwgCvRtSFbH7R51X4GbW+ziw= 2023-10-18T20:45:22.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ocuDwgCvRtSFbH7R51X4GbW+ziw=', 'Content-Length': '0'} 2023-10-18T20:45:22.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000303d0ca8af2096fb-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.580 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.581 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.581 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.582 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.582 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.582 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.583 DEBUG:boto:Data: 2023-10-18T20:45:22.583 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.584 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.584 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.584 DEBUG:boto:Params: {} 2023-10-18T20:45:22.585 DEBUG:boto:Token: None 2023-10-18T20:45:22.585 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ocuDwgCvRtSFbH7R51X4GbW+ziw= 2023-10-18T20:45:22.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ocuDwgCvRtSFbH7R51X4GbW+ziw=', 'Content-Length': '0'} 2023-10-18T20:45:22.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000758c2496b13d9977-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:45:22.589 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.590 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.590 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.590 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.591 DEBUG:boto:Method: GET 2023-10-18T20:45:22.591 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.592 DEBUG:boto:Data: 2023-10-18T20:45:22.592 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.593 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.593 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.593 DEBUG:boto:Params: {} 2023-10-18T20:45:22.594 DEBUG:boto:Token: None 2023-10-18T20:45:22.594 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JMmWOGop1BhKJ32Wczb2rFhmDL0= 2023-10-18T20:45:22.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JMmWOGop1BhKJ32Wczb2rFhmDL0=', 'Content-Length': '0'} 2023-10-18T20:45:22.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002871db21f40c6a9c-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.598 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.598 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.599 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.599 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.600 DEBUG:boto:Method: GET 2023-10-18T20:45:22.600 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.601 DEBUG:boto:Data: 2023-10-18T20:45:22.601 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.602 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.602 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.602 DEBUG:boto:Params: {} 2023-10-18T20:45:22.603 DEBUG:boto:Token: None 2023-10-18T20:45:22.603 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:22.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JMmWOGop1BhKJ32Wczb2rFhmDL0= 2023-10-18T20:45:22.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JMmWOGop1BhKJ32Wczb2rFhmDL0=', 'Content-Length': '0'} 2023-10-18T20:45:22.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af20cb306799ece-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.607 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.608 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.608 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.608 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.609 DEBUG:boto:Method: GET 2023-10-18T20:45:22.609 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.610 DEBUG:boto:Data: 2023-10-18T20:45:22.610 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.611 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.611 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.612 DEBUG:boto:Params: {} 2023-10-18T20:45:22.612 DEBUG:boto:Token: None 2023-10-18T20:45:22.613 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000692291d0149d4a35-00653043e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.617 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.618 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.618 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.619 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.619 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.619 DEBUG:boto:Method: GET 2023-10-18T20:45:22.620 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.620 DEBUG:boto:Data: 2023-10-18T20:45:22.621 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.621 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.622 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.622 DEBUG:boto:Params: {} 2023-10-18T20:45:22.623 DEBUG:boto:Token: None 2023-10-18T20:45:22.623 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.624 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002705a58bab58e825-00653043e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.670 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.671 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.672 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.672 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.673 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:45:22.673 DEBUG:boto:Data: 2023-10-18T20:45:22.673 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.674 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.674 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.675 DEBUG:boto:Params: {} 2023-10-18T20:45:22.675 DEBUG:boto:Token: None 2023-10-18T20:45:22.675 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19 2023-10-18T20:45:22.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oLrLr7LBU3YEPSK8fRJEq16yJ8E= 2023-10-18T20:45:22.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oLrLr7LBU3YEPSK8fRJEq16yJ8E=', 'Content-Length': '0'} 2023-10-18T20:45:22.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000500551b3da5a12b5-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.679 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.680 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.680 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.680 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:45:22.681 DEBUG:boto:Data: 2023-10-18T20:45:22.681 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.682 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.682 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.683 DEBUG:boto:Params: {} 2023-10-18T20:45:22.683 DEBUG:boto:Token: None 2023-10-18T20:45:22.684 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19 2023-10-18T20:45:22.684 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oLrLr7LBU3YEPSK8fRJEq16yJ8E= 2023-10-18T20:45:22.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oLrLr7LBU3YEPSK8fRJEq16yJ8E=', 'Content-Length': '0'} 2023-10-18T20:45:22.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056d172a8c638ff80-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:45:22.687 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.688 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.688 DEBUG:boto:Method: GET 2023-10-18T20:45:22.689 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:45:22.689 DEBUG:boto:Data: 2023-10-18T20:45:22.690 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.690 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.691 DEBUG:boto:Params: {} 2023-10-18T20:45:22.691 DEBUG:boto:Token: None 2023-10-18T20:45:22.692 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19 2023-10-18T20:45:22.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OGvnLuZ5H2VdO0vYo9FKUEjjW14= 2023-10-18T20:45:22.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OGvnLuZ5H2VdO0vYo9FKUEjjW14=', 'Content-Length': '0'} 2023-10-18T20:45:22.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8364987675557fe-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.696 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:22.696 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:22.697 DEBUG:boto:Method: GET 2023-10-18T20:45:22.697 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:45:22.698 DEBUG:boto:Data: 2023-10-18T20:45:22.698 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.699 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.699 DEBUG:boto:Params: {} 2023-10-18T20:45:22.700 DEBUG:boto:Token: None 2023-10-18T20:45:22.700 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj19 2023-10-18T20:45:22.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OGvnLuZ5H2VdO0vYo9FKUEjjW14= 2023-10-18T20:45:22.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OGvnLuZ5H2VdO0vYo9FKUEjjW14=', 'Content-Length': '0'} 2023-10-18T20:45:22.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b8f34949f4f8b1e-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.704 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.705 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.705 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.705 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.706 DEBUG:boto:Method: GET 2023-10-18T20:45:22.706 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.707 DEBUG:boto:Data: 2023-10-18T20:45:22.707 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.708 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.708 DEBUG:boto:Params: {} 2023-10-18T20:45:22.709 DEBUG:boto:Token: None 2023-10-18T20:45:22.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000693d4198c44a444-00653043e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.754 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.755 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.756 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.756 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.756 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.757 DEBUG:boto:Method: GET 2023-10-18T20:45:22.757 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:45:22.758 DEBUG:boto:Data: 2023-10-18T20:45:22.758 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.759 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.759 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.760 DEBUG:boto:Params: {} 2023-10-18T20:45:22.760 DEBUG:boto:Token: None 2023-10-18T20:45:22.761 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.761 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047b5e989d1182e65-00653043e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.767 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:45:22.769 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.769 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.769 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.770 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.770 DEBUG:boto:Method: GET 2023-10-18T20:45:22.771 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.771 DEBUG:boto:Data: 2023-10-18T20:45:22.772 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.772 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.772 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.773 DEBUG:boto:Params: {} 2023-10-18T20:45:22.773 DEBUG:boto:Token: None 2023-10-18T20:45:22.774 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XFjyVvwOIWu6ndXR6hzXZLhxkxw= 2023-10-18T20:45:22.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XFjyVvwOIWu6ndXR6hzXZLhxkxw=', 'Content-Length': '0'} 2023-10-18T20:45:22.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f9a59b0de854b4b-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.778 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.778 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.779 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.779 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.780 DEBUG:boto:Method: GET 2023-10-18T20:45:22.780 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.780 DEBUG:boto:Data: 2023-10-18T20:45:22.781 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.782 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.782 DEBUG:boto:Params: {} 2023-10-18T20:45:22.783 DEBUG:boto:Token: None 2023-10-18T20:45:22.783 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XFjyVvwOIWu6ndXR6hzXZLhxkxw= 2023-10-18T20:45:22.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XFjyVvwOIWu6ndXR6hzXZLhxkxw=', 'Content-Length': '0'} 2023-10-18T20:45:22.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afca0c8db4d6cae7-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.830 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.831 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.831 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.832 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.832 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.833 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.833 DEBUG:boto:Data: 2023-10-18T20:45:22.834 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.834 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.834 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.835 DEBUG:boto:Params: {} 2023-10-18T20:45:22.835 DEBUG:boto:Token: None 2023-10-18T20:45:22.836 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.836 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iwnVT1ymQNH7sBhjkVkHm9ytWYA= 2023-10-18T20:45:22.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iwnVT1ymQNH7sBhjkVkHm9ytWYA=', 'Content-Length': '0'} 2023-10-18T20:45:22.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa15c222cca4811f-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.839 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.840 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.841 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.841 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.842 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.842 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.843 DEBUG:boto:Data: 2023-10-18T20:45:22.843 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.844 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.844 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.844 DEBUG:boto:Params: {} 2023-10-18T20:45:22.845 DEBUG:boto:Token: None 2023-10-18T20:45:22.845 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.846 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iwnVT1ymQNH7sBhjkVkHm9ytWYA= 2023-10-18T20:45:22.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iwnVT1ymQNH7sBhjkVkHm9ytWYA=', 'Content-Length': '0'} 2023-10-18T20:45:22.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a196122882e7b6-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:45:22.849 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.850 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.850 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.851 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.851 DEBUG:boto:Method: GET 2023-10-18T20:45:22.852 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.852 DEBUG:boto:Data: 2023-10-18T20:45:22.852 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.853 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.853 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.854 DEBUG:boto:Params: {} 2023-10-18T20:45:22.854 DEBUG:boto:Token: None 2023-10-18T20:45:22.855 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.855 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XFjyVvwOIWu6ndXR6hzXZLhxkxw= 2023-10-18T20:45:22.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XFjyVvwOIWu6ndXR6hzXZLhxkxw=', 'Content-Length': '0'} 2023-10-18T20:45:22.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076ba4de8a7ba8261-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.859 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.859 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.860 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.860 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.860 DEBUG:boto:Method: GET 2023-10-18T20:45:22.861 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.861 DEBUG:boto:Data: 2023-10-18T20:45:22.862 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.862 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.862 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.863 DEBUG:boto:Params: {} 2023-10-18T20:45:22.863 DEBUG:boto:Token: None 2023-10-18T20:45:22.864 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:22.864 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XFjyVvwOIWu6ndXR6hzXZLhxkxw= 2023-10-18T20:45:22.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XFjyVvwOIWu6ndXR6hzXZLhxkxw=', 'Content-Length': '0'} 2023-10-18T20:45:22.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c8f21acdfb36ad-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.868 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.868 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.869 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.869 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.870 DEBUG:boto:Method: GET 2023-10-18T20:45:22.870 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.871 DEBUG:boto:Data: 2023-10-18T20:45:22.871 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.871 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.872 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.872 DEBUG:boto:Params: {} 2023-10-18T20:45:22.873 DEBUG:boto:Token: None 2023-10-18T20:45:22.873 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.874 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077cac1d661ce8e9f-00653043e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.877 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.878 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.878 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.879 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.879 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.880 DEBUG:boto:Method: GET 2023-10-18T20:45:22.880 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.881 DEBUG:boto:Data: 2023-10-18T20:45:22.881 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.881 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.882 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.882 DEBUG:boto:Params: {} 2023-10-18T20:45:22.883 DEBUG:boto:Token: None 2023-10-18T20:45:22.883 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d80cf63e95c497aa-00653043e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.889 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.890 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.891 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.891 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.891 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:45:22.892 DEBUG:boto:Data: 2023-10-18T20:45:22.892 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.893 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.893 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.893 DEBUG:boto:Params: {} 2023-10-18T20:45:22.894 DEBUG:boto:Token: None 2023-10-18T20:45:22.894 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22 2023-10-18T20:45:22.895 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bBWicFQUCSnrX8pFdEhtlRJZxPo= 2023-10-18T20:45:22.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bBWicFQUCSnrX8pFdEhtlRJZxPo=', 'Content-Length': '0'} 2023-10-18T20:45:22.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000daef5047bf5fa6-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.898 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.898 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.899 DEBUG:boto:Method: HEAD 2023-10-18T20:45:22.899 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:45:22.900 DEBUG:boto:Data: 2023-10-18T20:45:22.900 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.901 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.901 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.901 DEBUG:boto:Params: {} 2023-10-18T20:45:22.902 DEBUG:boto:Token: None 2023-10-18T20:45:22.902 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22 2023-10-18T20:45:22.903 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bBWicFQUCSnrX8pFdEhtlRJZxPo= 2023-10-18T20:45:22.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bBWicFQUCSnrX8pFdEhtlRJZxPo=', 'Content-Length': '0'} 2023-10-18T20:45:22.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d103f35170aa70f1-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:45:22.906 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.907 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.907 DEBUG:boto:Method: GET 2023-10-18T20:45:22.908 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:45:22.908 DEBUG:boto:Data: 2023-10-18T20:45:22.909 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.909 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.909 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.910 DEBUG:boto:Params: {} 2023-10-18T20:45:22.910 DEBUG:boto:Token: None 2023-10-18T20:45:22.911 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22 2023-10-18T20:45:22.911 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m5S5ygY2w0jJiK6yHkfyvTU7LbE= 2023-10-18T20:45:22.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m5S5ygY2w0jJiK6yHkfyvTU7LbE=', 'Content-Length': '0'} 2023-10-18T20:45:22.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc37d096f8162be-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.914 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:22.915 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:22.915 DEBUG:boto:Method: GET 2023-10-18T20:45:22.916 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:45:22.916 DEBUG:boto:Data: 2023-10-18T20:45:22.917 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.917 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.918 DEBUG:boto:Params: {} 2023-10-18T20:45:22.918 DEBUG:boto:Token: None 2023-10-18T20:45:22.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj22 2023-10-18T20:45:22.919 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m5S5ygY2w0jJiK6yHkfyvTU7LbE= 2023-10-18T20:45:22.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m5S5ygY2w0jJiK6yHkfyvTU7LbE=', 'Content-Length': '0'} 2023-10-18T20:45:22.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173a8cf2ad0d70b8-00653043e2-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.962 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.963 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.964 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.964 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.964 DEBUG:boto:Method: GET 2023-10-18T20:45:22.965 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.965 DEBUG:boto:Data: 2023-10-18T20:45:22.966 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.966 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.967 DEBUG:boto:Params: {} 2023-10-18T20:45:22.967 DEBUG:boto:Token: None 2023-10-18T20:45:22.968 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4bae947636ad255-00653043e2-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.972 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.972 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:22.973 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:22.973 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.974 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.974 DEBUG:boto:Method: GET 2023-10-18T20:45:22.975 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:45:22.975 DEBUG:boto:Data: 2023-10-18T20:45:22.975 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.976 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:22.976 DEBUG:boto:Port: 8000 2023-10-18T20:45:22.977 DEBUG:boto:Params: {} 2023-10-18T20:45:22.977 DEBUG:boto:Token: None 2023-10-18T20:45:22.978 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/ 2023-10-18T20:45:22.978 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo= 2023-10-18T20:45:22.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWwZa1OxdDeB6X+MDTbKz89FHAo=', 'Content-Length': '0'} 2023-10-18T20:45:22.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b587516ab7e01f1-00653043e2-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:22.983 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:22.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:45:22.985 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:22.985 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:22.985 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:22.986 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:22.986 DEBUG:boto:Method: GET 2023-10-18T20:45:22.987 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:22.987 DEBUG:boto:Data: 2023-10-18T20:45:22.987 DEBUG:boto:Headers: {} 2023-10-18T20:45:22.988 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:22.988 DEBUG:boto:Port: 8001 2023-10-18T20:45:22.989 DEBUG:boto:Params: {} 2023-10-18T20:45:22.989 DEBUG:boto:Token: None 2023-10-18T20:45:22.990 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:22 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:22.990 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HtCjY9Vi9Sy6950fDR4nsV2FNYg= 2023-10-18T20:45:22.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:22 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HtCjY9Vi9Sy6950fDR4nsV2FNYg=', 'Content-Length': '0'} 2023-10-18T20:45:22.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009137542dca143fcb-00653043e2-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.038 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:23.039 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:23.039 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.040 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.040 DEBUG:boto:Method: GET 2023-10-18T20:45:23.041 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.041 DEBUG:boto:Data: 2023-10-18T20:45:23.042 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.042 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.043 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.043 DEBUG:boto:Params: {} 2023-10-18T20:45:23.044 DEBUG:boto:Token: None 2023-10-18T20:45:23.044 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.045 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PH+i7u0AlmCZ1Z8xKF35Bxmzmko= 2023-10-18T20:45:23.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PH+i7u0AlmCZ1Z8xKF35Bxmzmko=', 'Content-Length': '0'} 2023-10-18T20:45:23.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b87fcec6bbd75ef-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.048 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:23.048 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:23.049 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.049 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.049 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.050 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.050 DEBUG:boto:Data: 2023-10-18T20:45:23.051 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.051 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.051 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.052 DEBUG:boto:Params: {} 2023-10-18T20:45:23.052 DEBUG:boto:Token: None 2023-10-18T20:45:23.053 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.053 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q1/56IloM08LHfhI43y/YhiMlrk= 2023-10-18T20:45:23.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q1/56IloM08LHfhI43y/YhiMlrk=', 'Content-Length': '0'} 2023-10-18T20:45:23.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea174f35985c023-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.056 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:23.057 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:23.057 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.057 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.058 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.058 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.059 DEBUG:boto:Data: 2023-10-18T20:45:23.059 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.059 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.060 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.060 DEBUG:boto:Params: {} 2023-10-18T20:45:23.061 DEBUG:boto:Token: None 2023-10-18T20:45:23.061 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.062 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q1/56IloM08LHfhI43y/YhiMlrk= 2023-10-18T20:45:23.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q1/56IloM08LHfhI43y/YhiMlrk=', 'Content-Length': '0'} 2023-10-18T20:45:23.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003329132cceb049c8-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:45:23.065 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:23.066 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:23.066 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.066 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.067 DEBUG:boto:Method: GET 2023-10-18T20:45:23.067 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.068 DEBUG:boto:Data: 2023-10-18T20:45:23.068 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.069 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.069 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.070 DEBUG:boto:Params: {} 2023-10-18T20:45:23.070 DEBUG:boto:Token: None 2023-10-18T20:45:23.071 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.071 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PH+i7u0AlmCZ1Z8xKF35Bxmzmko= 2023-10-18T20:45:23.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PH+i7u0AlmCZ1Z8xKF35Bxmzmko=', 'Content-Length': '0'} 2023-10-18T20:45:23.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fc8fdeaaa091998-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.074 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:23.074 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:23.075 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.075 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.076 DEBUG:boto:Method: GET 2023-10-18T20:45:23.076 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.077 DEBUG:boto:Data: 2023-10-18T20:45:23.077 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.077 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.078 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.078 DEBUG:boto:Params: {} 2023-10-18T20:45:23.079 DEBUG:boto:Token: None 2023-10-18T20:45:23.079 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:23.080 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PH+i7u0AlmCZ1Z8xKF35Bxmzmko= 2023-10-18T20:45:23.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PH+i7u0AlmCZ1Z8xKF35Bxmzmko=', 'Content-Length': '0'} 2023-10-18T20:45:23.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013cc0eb8a48500c9-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.083 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.083 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.084 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.084 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.085 DEBUG:boto:Method: GET 2023-10-18T20:45:23.085 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.085 DEBUG:boto:Data: 2023-10-18T20:45:23.086 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.086 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.087 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.087 DEBUG:boto:Params: {} 2023-10-18T20:45:23.088 DEBUG:boto:Token: None 2023-10-18T20:45:23.088 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.089 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000491b27b7872db320-00653043e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.092 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.093 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.093 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.094 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.094 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.094 DEBUG:boto:Method: GET 2023-10-18T20:45:23.095 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.095 DEBUG:boto:Data: 2023-10-18T20:45:23.096 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.096 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.097 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.097 DEBUG:boto:Params: {} 2023-10-18T20:45:23.098 DEBUG:boto:Token: None 2023-10-18T20:45:23.098 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.099 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ff9acc4819cd3d-00653043e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.146 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.147 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:23.148 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:23.148 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.149 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:45:23.149 DEBUG:boto:Data: 2023-10-18T20:45:23.150 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.150 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.151 DEBUG:boto:Params: {} 2023-10-18T20:45:23.151 DEBUG:boto:Token: None 2023-10-18T20:45:23.152 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj23 2023-10-18T20:45:23.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xVv88tJb925RkOOtl93b2tD6uhI= 2023-10-18T20:45:23.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xVv88tJb925RkOOtl93b2tD6uhI=', 'Content-Length': '0'} 2023-10-18T20:45:23.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002302ed9d9e70ef3c-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.155 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:23.156 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:23.156 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.157 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:45:23.157 DEBUG:boto:Data: 2023-10-18T20:45:23.158 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.158 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.159 DEBUG:boto:Params: {} 2023-10-18T20:45:23.159 DEBUG:boto:Token: None 2023-10-18T20:45:23.160 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj23 2023-10-18T20:45:23.160 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xVv88tJb925RkOOtl93b2tD6uhI= 2023-10-18T20:45:23.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xVv88tJb925RkOOtl93b2tD6uhI=', 'Content-Length': '0'} 2023-10-18T20:45:23.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4bec72cdf146098-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:45:23.163 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:23.164 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:23.164 DEBUG:boto:Method: GET 2023-10-18T20:45:23.165 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:45:23.165 DEBUG:boto:Data: 2023-10-18T20:45:23.166 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.166 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.167 DEBUG:boto:Params: {} 2023-10-18T20:45:23.167 DEBUG:boto:Token: None 2023-10-18T20:45:23.168 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj23 2023-10-18T20:45:23.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uq8BuBHiFFVw1GELQshSvcsiihU= 2023-10-18T20:45:23.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uq8BuBHiFFVw1GELQshSvcsiihU=', 'Content-Length': '0'} 2023-10-18T20:45:23.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000668165c7d87e338d-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.171 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:23.172 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:23.172 DEBUG:boto:Method: GET 2023-10-18T20:45:23.173 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:45:23.173 DEBUG:boto:Data: 2023-10-18T20:45:23.174 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.174 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.175 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.175 DEBUG:boto:Params: {} 2023-10-18T20:45:23.175 DEBUG:boto:Token: None 2023-10-18T20:45:23.176 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj23 2023-10-18T20:45:23.176 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uq8BuBHiFFVw1GELQshSvcsiihU= 2023-10-18T20:45:23.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uq8BuBHiFFVw1GELQshSvcsiihU=', 'Content-Length': '0'} 2023-10-18T20:45:23.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f64d4626bd4904b-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.180 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.180 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.180 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.181 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.181 DEBUG:boto:Method: GET 2023-10-18T20:45:23.182 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.182 DEBUG:boto:Data: 2023-10-18T20:45:23.183 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.183 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.183 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.184 DEBUG:boto:Params: {} 2023-10-18T20:45:23.184 DEBUG:boto:Token: None 2023-10-18T20:45:23.185 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.185 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053be40cd1028fec0-00653043e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.230 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.231 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.232 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.232 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.233 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.233 DEBUG:boto:Method: GET 2023-10-18T20:45:23.233 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:45:23.234 DEBUG:boto:Data: 2023-10-18T20:45:23.234 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.235 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.235 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.236 DEBUG:boto:Params: {} 2023-10-18T20:45:23.236 DEBUG:boto:Token: None 2023-10-18T20:45:23.237 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c869a1767c2badd0-00653043e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.243 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:45:23.244 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.245 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.245 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.246 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.246 DEBUG:boto:Method: GET 2023-10-18T20:45:23.247 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.247 DEBUG:boto:Data: 2023-10-18T20:45:23.247 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.249 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.249 DEBUG:boto:Params: {} 2023-10-18T20:45:23.250 DEBUG:boto:Token: None 2023-10-18T20:45:23.250 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.251 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RCWCfbXeWw/bvWNiiHt2UeIt4+g= 2023-10-18T20:45:23.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RCWCfbXeWw/bvWNiiHt2UeIt4+g=', 'Content-Length': '0'} 2023-10-18T20:45:23.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e80301c69dfa51-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.255 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.256 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.256 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.257 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.257 DEBUG:boto:Method: GET 2023-10-18T20:45:23.258 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.258 DEBUG:boto:Data: 2023-10-18T20:45:23.259 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.259 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.260 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.260 DEBUG:boto:Params: {} 2023-10-18T20:45:23.261 DEBUG:boto:Token: None 2023-10-18T20:45:23.261 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.262 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RCWCfbXeWw/bvWNiiHt2UeIt4+g= 2023-10-18T20:45:23.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RCWCfbXeWw/bvWNiiHt2UeIt4+g=', 'Content-Length': '0'} 2023-10-18T20:45:23.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f57476638c42f12b-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.265 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.266 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.267 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.267 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.268 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.268 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.268 DEBUG:boto:Data: 2023-10-18T20:45:23.269 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.270 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.270 DEBUG:boto:Params: {} 2023-10-18T20:45:23.271 DEBUG:boto:Token: None 2023-10-18T20:45:23.271 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.272 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:htb5VvPYxYp5YJEAA1o2s/tNoVg= 2023-10-18T20:45:23.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:htb5VvPYxYp5YJEAA1o2s/tNoVg=', 'Content-Length': '0'} 2023-10-18T20:45:23.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a10f24cb51cbb9-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.275 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.275 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.275 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.276 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.276 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.277 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.278 DEBUG:boto:Data: 2023-10-18T20:45:23.279 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.279 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.280 DEBUG:boto:Params: {} 2023-10-18T20:45:23.280 DEBUG:boto:Token: None 2023-10-18T20:45:23.281 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.281 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:htb5VvPYxYp5YJEAA1o2s/tNoVg= 2023-10-18T20:45:23.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:htb5VvPYxYp5YJEAA1o2s/tNoVg=', 'Content-Length': '0'} 2023-10-18T20:45:23.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5faf159e42fa4b-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:45:23.285 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.285 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.285 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.286 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.287 DEBUG:boto:Method: GET 2023-10-18T20:45:23.287 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.288 DEBUG:boto:Data: 2023-10-18T20:45:23.288 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.289 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.289 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.290 DEBUG:boto:Params: {} 2023-10-18T20:45:23.291 DEBUG:boto:Token: None 2023-10-18T20:45:23.292 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.293 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RCWCfbXeWw/bvWNiiHt2UeIt4+g= 2023-10-18T20:45:23.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RCWCfbXeWw/bvWNiiHt2UeIt4+g=', 'Content-Length': '0'} 2023-10-18T20:45:23.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e16dfe2372db187b-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.297 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.298 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.299 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.300 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.300 DEBUG:boto:Method: GET 2023-10-18T20:45:23.301 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.302 DEBUG:boto:Data: 2023-10-18T20:45:23.303 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.303 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.304 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.305 DEBUG:boto:Params: {} 2023-10-18T20:45:23.305 DEBUG:boto:Token: None 2023-10-18T20:45:23.306 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:23.307 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RCWCfbXeWw/bvWNiiHt2UeIt4+g= 2023-10-18T20:45:23.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RCWCfbXeWw/bvWNiiHt2UeIt4+g=', 'Content-Length': '0'} 2023-10-18T20:45:23.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fd9fa88c036e8ec-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.311 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.312 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.312 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.313 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.313 DEBUG:boto:Method: GET 2023-10-18T20:45:23.314 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.314 DEBUG:boto:Data: 2023-10-18T20:45:23.315 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.315 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.316 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.316 DEBUG:boto:Params: {} 2023-10-18T20:45:23.317 DEBUG:boto:Token: None 2023-10-18T20:45:23.317 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.318 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc6345a7e15a7cb3-00653043e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.322 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.323 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.324 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.324 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.324 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.325 DEBUG:boto:Method: GET 2023-10-18T20:45:23.325 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.326 DEBUG:boto:Data: 2023-10-18T20:45:23.326 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.327 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.327 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.328 DEBUG:boto:Params: {} 2023-10-18T20:45:23.328 DEBUG:boto:Token: None 2023-10-18T20:45:23.329 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.329 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2fa576db7489ab-00653043e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.378 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.379 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.379 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.380 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.380 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:45:23.381 DEBUG:boto:Data: 2023-10-18T20:45:23.381 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.382 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.382 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.383 DEBUG:boto:Params: {} 2023-10-18T20:45:23.383 DEBUG:boto:Token: None 2023-10-18T20:45:23.384 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29 2023-10-18T20:45:23.384 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Tf++/uy9R3JpTr3A42+lErdgVSo= 2023-10-18T20:45:23.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Tf++/uy9R3JpTr3A42+lErdgVSo=', 'Content-Length': '0'} 2023-10-18T20:45:23.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e4593c29aba428a-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.387 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.388 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.388 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.389 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:45:23.389 DEBUG:boto:Data: 2023-10-18T20:45:23.390 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.390 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.391 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.391 DEBUG:boto:Params: {} 2023-10-18T20:45:23.391 DEBUG:boto:Token: None 2023-10-18T20:45:23.392 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29 2023-10-18T20:45:23.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Tf++/uy9R3JpTr3A42+lErdgVSo= 2023-10-18T20:45:23.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Tf++/uy9R3JpTr3A42+lErdgVSo=', 'Content-Length': '0'} 2023-10-18T20:45:23.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be85535fcc370a5d-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:45:23.396 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.397 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.397 DEBUG:boto:Method: GET 2023-10-18T20:45:23.398 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:45:23.398 DEBUG:boto:Data: 2023-10-18T20:45:23.399 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.400 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.400 DEBUG:boto:Params: {} 2023-10-18T20:45:23.401 DEBUG:boto:Token: None 2023-10-18T20:45:23.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29 2023-10-18T20:45:23.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5k3vKdtx97K02EO4Z65RaYTmyDk= 2023-10-18T20:45:23.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5k3vKdtx97K02EO4Z65RaYTmyDk=', 'Content-Length': '0'} 2023-10-18T20:45:23.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d854a9f5db8e930-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.405 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:23.406 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:23.406 DEBUG:boto:Method: GET 2023-10-18T20:45:23.407 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:45:23.407 DEBUG:boto:Data: 2023-10-18T20:45:23.407 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.408 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.409 DEBUG:boto:Params: {} 2023-10-18T20:45:23.409 DEBUG:boto:Token: None 2023-10-18T20:45:23.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj29 2023-10-18T20:45:23.410 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5k3vKdtx97K02EO4Z65RaYTmyDk= 2023-10-18T20:45:23.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5k3vKdtx97K02EO4Z65RaYTmyDk=', 'Content-Length': '0'} 2023-10-18T20:45:23.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2050cd90b0c8c1c-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.414 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.414 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.415 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.415 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.416 DEBUG:boto:Method: GET 2023-10-18T20:45:23.416 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.416 DEBUG:boto:Data: 2023-10-18T20:45:23.417 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.418 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.418 DEBUG:boto:Params: {} 2023-10-18T20:45:23.419 DEBUG:boto:Token: None 2023-10-18T20:45:23.419 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006468745f79c03157-00653043e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.423 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.424 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.424 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.425 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.425 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.425 DEBUG:boto:Method: GET 2023-10-18T20:45:23.426 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:45:23.426 DEBUG:boto:Data: 2023-10-18T20:45:23.427 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.427 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.427 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.428 DEBUG:boto:Params: {} 2023-10-18T20:45:23.428 DEBUG:boto:Token: None 2023-10-18T20:45:23.429 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.429 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000808720734cac899e-00653043e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.434 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-18T20:45:23.435 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.436 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.436 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.437 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.437 DEBUG:boto:Method: GET 2023-10-18T20:45:23.438 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.438 DEBUG:boto:Data: 2023-10-18T20:45:23.438 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.439 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.439 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.440 DEBUG:boto:Params: {} 2023-10-18T20:45:23.440 DEBUG:boto:Token: None 2023-10-18T20:45:23.441 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.441 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dPOEothrXG8OSckTOD8JqlWBQEY= 2023-10-18T20:45:23.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dPOEothrXG8OSckTOD8JqlWBQEY=', 'Content-Length': '0'} 2023-10-18T20:45:23.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085101e41b6269776-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.490 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.491 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.491 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.492 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.492 DEBUG:boto:Method: GET 2023-10-18T20:45:23.493 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.493 DEBUG:boto:Data: 2023-10-18T20:45:23.494 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.495 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.495 DEBUG:boto:Params: {} 2023-10-18T20:45:23.496 DEBUG:boto:Token: None 2023-10-18T20:45:23.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dPOEothrXG8OSckTOD8JqlWBQEY= 2023-10-18T20:45:23.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dPOEothrXG8OSckTOD8JqlWBQEY=', 'Content-Length': '0'} 2023-10-18T20:45:23.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005115204d3fa49a5-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.500 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.501 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.501 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.501 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.502 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.502 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.503 DEBUG:boto:Data: 2023-10-18T20:45:23.503 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.504 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.504 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.504 DEBUG:boto:Params: {} 2023-10-18T20:45:23.505 DEBUG:boto:Token: None 2023-10-18T20:45:23.505 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.506 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7hRrbxlIMam9oX2pOI4vtZECujQ= 2023-10-18T20:45:23.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7hRrbxlIMam9oX2pOI4vtZECujQ=', 'Content-Length': '0'} 2023-10-18T20:45:23.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a59184df531afb1-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.509 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.509 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.510 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.510 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.511 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.511 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.511 DEBUG:boto:Data: 2023-10-18T20:45:23.512 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.512 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.513 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.513 DEBUG:boto:Params: {} 2023-10-18T20:45:23.513 DEBUG:boto:Token: None 2023-10-18T20:45:23.514 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.514 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7hRrbxlIMam9oX2pOI4vtZECujQ= 2023-10-18T20:45:23.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7hRrbxlIMam9oX2pOI4vtZECujQ=', 'Content-Length': '0'} 2023-10-18T20:45:23.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b888d1f55bbd1a-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-18T20:45:23.518 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.519 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.519 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.520 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.520 DEBUG:boto:Method: GET 2023-10-18T20:45:23.521 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.521 DEBUG:boto:Data: 2023-10-18T20:45:23.521 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.523 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.523 DEBUG:boto:Params: {} 2023-10-18T20:45:23.523 DEBUG:boto:Token: None 2023-10-18T20:45:23.524 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.524 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dPOEothrXG8OSckTOD8JqlWBQEY= 2023-10-18T20:45:23.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dPOEothrXG8OSckTOD8JqlWBQEY=', 'Content-Length': '0'} 2023-10-18T20:45:23.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094f6ee1ceb3c0fb9-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.528 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.528 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.529 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.529 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.529 DEBUG:boto:Method: GET 2023-10-18T20:45:23.530 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.530 DEBUG:boto:Data: 2023-10-18T20:45:23.531 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.531 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.532 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.532 DEBUG:boto:Params: {} 2023-10-18T20:45:23.532 DEBUG:boto:Token: None 2023-10-18T20:45:23.533 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:23.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dPOEothrXG8OSckTOD8JqlWBQEY= 2023-10-18T20:45:23.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dPOEothrXG8OSckTOD8JqlWBQEY=', 'Content-Length': '0'} 2023-10-18T20:45:23.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb32f06aadc413bb-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.582 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.583 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.583 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.584 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.584 DEBUG:boto:Method: GET 2023-10-18T20:45:23.585 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.585 DEBUG:boto:Data: 2023-10-18T20:45:23.586 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.586 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.586 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.587 DEBUG:boto:Params: {} 2023-10-18T20:45:23.587 DEBUG:boto:Token: None 2023-10-18T20:45:23.588 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.588 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b95053692be9e445-00653043e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.592 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.593 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.593 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.594 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.594 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.594 DEBUG:boto:Method: GET 2023-10-18T20:45:23.595 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.595 DEBUG:boto:Data: 2023-10-18T20:45:23.596 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.596 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.597 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.597 DEBUG:boto:Params: {} 2023-10-18T20:45:23.597 DEBUG:boto:Token: None 2023-10-18T20:45:23.598 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.598 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aa9a9a0c6d92b02-00653043e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.604 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.604 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.605 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.605 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.606 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:45:23.606 DEBUG:boto:Data: 2023-10-18T20:45:23.607 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.607 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.607 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.608 DEBUG:boto:Params: {} 2023-10-18T20:45:23.608 DEBUG:boto:Token: None 2023-10-18T20:45:23.609 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31 2023-10-18T20:45:23.609 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MNBL5JOA+jtqHMygjBYwtyg/MIk= 2023-10-18T20:45:23.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MNBL5JOA+jtqHMygjBYwtyg/MIk=', 'Content-Length': '0'} 2023-10-18T20:45:23.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085ff77c4f624c2ef-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.612 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.613 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.613 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.614 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:45:23.614 DEBUG:boto:Data: 2023-10-18T20:45:23.615 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.615 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.615 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.616 DEBUG:boto:Params: {} 2023-10-18T20:45:23.616 DEBUG:boto:Token: None 2023-10-18T20:45:23.617 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31 2023-10-18T20:45:23.617 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MNBL5JOA+jtqHMygjBYwtyg/MIk= 2023-10-18T20:45:23.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MNBL5JOA+jtqHMygjBYwtyg/MIk=', 'Content-Length': '0'} 2023-10-18T20:45:23.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006cf8be0ac929907-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-18T20:45:23.621 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.621 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.622 DEBUG:boto:Method: GET 2023-10-18T20:45:23.622 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:45:23.622 DEBUG:boto:Data: 2023-10-18T20:45:23.623 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.623 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.624 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.624 DEBUG:boto:Params: {} 2023-10-18T20:45:23.625 DEBUG:boto:Token: None 2023-10-18T20:45:23.625 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31 2023-10-18T20:45:23.626 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MunqesoUpt8LEnTBijo6OG/g3fE= 2023-10-18T20:45:23.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MunqesoUpt8LEnTBijo6OG/g3fE=', 'Content-Length': '0'} 2023-10-18T20:45:23.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009248be2982589616-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.629 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:23.629 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:23.630 DEBUG:boto:Method: GET 2023-10-18T20:45:23.630 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:45:23.631 DEBUG:boto:Data: 2023-10-18T20:45:23.631 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.632 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.632 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.632 DEBUG:boto:Params: {} 2023-10-18T20:45:23.633 DEBUG:boto:Token: None 2023-10-18T20:45:23.633 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj31 2023-10-18T20:45:23.634 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MunqesoUpt8LEnTBijo6OG/g3fE= 2023-10-18T20:45:23.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MunqesoUpt8LEnTBijo6OG/g3fE=', 'Content-Length': '0'} 2023-10-18T20:45:23.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000382a7050f3bea752-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.637 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.638 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.639 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.639 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.639 DEBUG:boto:Method: GET 2023-10-18T20:45:23.640 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.640 DEBUG:boto:Data: 2023-10-18T20:45:23.641 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.641 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.642 DEBUG:boto:Params: {} 2023-10-18T20:45:23.642 DEBUG:boto:Token: None 2023-10-18T20:45:23.643 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a47e60ba3809d9ae-00653043e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.647 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.648 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.649 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.649 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.649 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.650 DEBUG:boto:Method: GET 2023-10-18T20:45:23.650 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:45:23.651 DEBUG:boto:Data: 2023-10-18T20:45:23.651 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.652 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.652 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.652 DEBUG:boto:Params: {} 2023-10-18T20:45:23.653 DEBUG:boto:Token: None 2023-10-18T20:45:23.653 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.654 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec0a9b642898db28-00653043e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.659 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:45:23.660 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.661 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.661 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.662 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.662 DEBUG:boto:Method: GET 2023-10-18T20:45:23.663 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.663 DEBUG:boto:Data: 2023-10-18T20:45:23.663 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.664 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.664 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.665 DEBUG:boto:Params: {} 2023-10-18T20:45:23.665 DEBUG:boto:Token: None 2023-10-18T20:45:23.666 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.666 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m6xzrYh4dvylER+QQFrp1S7uThc= 2023-10-18T20:45:23.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m6xzrYh4dvylER+QQFrp1S7uThc=', 'Content-Length': '0'} 2023-10-18T20:45:23.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c7250f57879c8c3-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.714 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.715 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.715 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.716 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.716 DEBUG:boto:Method: GET 2023-10-18T20:45:23.717 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.717 DEBUG:boto:Data: 2023-10-18T20:45:23.718 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.718 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.718 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.719 DEBUG:boto:Params: {} 2023-10-18T20:45:23.720 DEBUG:boto:Token: None 2023-10-18T20:45:23.720 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m6xzrYh4dvylER+QQFrp1S7uThc= 2023-10-18T20:45:23.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m6xzrYh4dvylER+QQFrp1S7uThc=', 'Content-Length': '0'} 2023-10-18T20:45:23.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793f6e2e01b927e8-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.724 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.725 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.725 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.725 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.726 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.726 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.727 DEBUG:boto:Data: 2023-10-18T20:45:23.727 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.728 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.728 DEBUG:boto:Params: {} 2023-10-18T20:45:23.729 DEBUG:boto:Token: None 2023-10-18T20:45:23.729 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.730 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iS4NZiGYYuUnBb3mwnKK0BKnKBA= 2023-10-18T20:45:23.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iS4NZiGYYuUnBb3mwnKK0BKnKBA=', 'Content-Length': '0'} 2023-10-18T20:45:23.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285d4d8169a258a2-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.733 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.733 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.734 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.734 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.734 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.735 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.735 DEBUG:boto:Data: 2023-10-18T20:45:23.736 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.736 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.736 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.737 DEBUG:boto:Params: {} 2023-10-18T20:45:23.737 DEBUG:boto:Token: None 2023-10-18T20:45:23.738 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.738 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iS4NZiGYYuUnBb3mwnKK0BKnKBA= 2023-10-18T20:45:23.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iS4NZiGYYuUnBb3mwnKK0BKnKBA=', 'Content-Length': '0'} 2023-10-18T20:45:23.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022ff013e1b59bd1d-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:45:23.742 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.742 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.743 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.743 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.743 DEBUG:boto:Method: GET 2023-10-18T20:45:23.744 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.744 DEBUG:boto:Data: 2023-10-18T20:45:23.745 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.745 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.745 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.746 DEBUG:boto:Params: {} 2023-10-18T20:45:23.747 DEBUG:boto:Token: None 2023-10-18T20:45:23.747 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m6xzrYh4dvylER+QQFrp1S7uThc= 2023-10-18T20:45:23.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m6xzrYh4dvylER+QQFrp1S7uThc=', 'Content-Length': '0'} 2023-10-18T20:45:23.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000812439675dda9cc7-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.751 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.751 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.752 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.752 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.752 DEBUG:boto:Method: GET 2023-10-18T20:45:23.753 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.753 DEBUG:boto:Data: 2023-10-18T20:45:23.754 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.755 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.755 DEBUG:boto:Params: {} 2023-10-18T20:45:23.755 DEBUG:boto:Token: None 2023-10-18T20:45:23.756 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:23.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m6xzrYh4dvylER+QQFrp1S7uThc= 2023-10-18T20:45:23.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m6xzrYh4dvylER+QQFrp1S7uThc=', 'Content-Length': '0'} 2023-10-18T20:45:23.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deee70a274932ba1-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.806 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.807 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.807 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.808 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.808 DEBUG:boto:Method: GET 2023-10-18T20:45:23.809 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.809 DEBUG:boto:Data: 2023-10-18T20:45:23.810 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.810 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.810 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.811 DEBUG:boto:Params: {} 2023-10-18T20:45:23.812 DEBUG:boto:Token: None 2023-10-18T20:45:23.812 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.813 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032aae9901aeceae4-00653043e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.816 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.817 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.818 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.818 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.818 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.819 DEBUG:boto:Method: GET 2023-10-18T20:45:23.819 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.820 DEBUG:boto:Data: 2023-10-18T20:45:23.820 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.821 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.821 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.821 DEBUG:boto:Params: {} 2023-10-18T20:45:23.822 DEBUG:boto:Token: None 2023-10-18T20:45:23.822 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.823 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cdca83521de1a59-00653043e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.828 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.829 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.830 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.830 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.830 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:45:23.831 DEBUG:boto:Data: 2023-10-18T20:45:23.831 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.832 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.832 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.832 DEBUG:boto:Params: {} 2023-10-18T20:45:23.833 DEBUG:boto:Token: None 2023-10-18T20:45:23.833 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33 2023-10-18T20:45:23.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0eE7j27600M+lWGlJzsz/TnmKQ8= 2023-10-18T20:45:23.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0eE7j27600M+lWGlJzsz/TnmKQ8=', 'Content-Length': '0'} 2023-10-18T20:45:23.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f35237f7687457c-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.837 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.837 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.838 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.838 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:45:23.839 DEBUG:boto:Data: 2023-10-18T20:45:23.839 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.840 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.840 DEBUG:boto:Params: {} 2023-10-18T20:45:23.841 DEBUG:boto:Token: None 2023-10-18T20:45:23.841 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33 2023-10-18T20:45:23.842 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0eE7j27600M+lWGlJzsz/TnmKQ8= 2023-10-18T20:45:23.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0eE7j27600M+lWGlJzsz/TnmKQ8=', 'Content-Length': '0'} 2023-10-18T20:45:23.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c61d617a4d637a4-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:45:23.845 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.845 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.846 DEBUG:boto:Method: GET 2023-10-18T20:45:23.846 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:45:23.847 DEBUG:boto:Data: 2023-10-18T20:45:23.847 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.848 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.848 DEBUG:boto:Params: {} 2023-10-18T20:45:23.849 DEBUG:boto:Token: None 2023-10-18T20:45:23.849 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33 2023-10-18T20:45:23.850 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RAW3HRrvhm8f9q4XgOGHkGWOqeY= 2023-10-18T20:45:23.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RAW3HRrvhm8f9q4XgOGHkGWOqeY=', 'Content-Length': '0'} 2023-10-18T20:45:23.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f96898a69e76ca6-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.853 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:23.853 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:23.854 DEBUG:boto:Method: GET 2023-10-18T20:45:23.854 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:45:23.854 DEBUG:boto:Data: 2023-10-18T20:45:23.855 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.856 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.856 DEBUG:boto:Params: {} 2023-10-18T20:45:23.857 DEBUG:boto:Token: None 2023-10-18T20:45:23.857 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj33 2023-10-18T20:45:23.858 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RAW3HRrvhm8f9q4XgOGHkGWOqeY= 2023-10-18T20:45:23.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RAW3HRrvhm8f9q4XgOGHkGWOqeY=', 'Content-Length': '0'} 2023-10-18T20:45:23.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6bc1a0e73c0a59-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.861 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.861 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.862 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.862 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.863 DEBUG:boto:Method: GET 2023-10-18T20:45:23.863 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.864 DEBUG:boto:Data: 2023-10-18T20:45:23.864 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.864 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.865 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.865 DEBUG:boto:Params: {} 2023-10-18T20:45:23.866 DEBUG:boto:Token: None 2023-10-18T20:45:23.866 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc214650889ce6a4-00653043e3-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.870 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.871 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:23.871 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:23.872 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.872 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.873 DEBUG:boto:Method: GET 2023-10-18T20:45:23.873 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:45:23.874 DEBUG:boto:Data: 2023-10-18T20:45:23.874 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.874 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.875 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.875 DEBUG:boto:Params: {} 2023-10-18T20:45:23.876 DEBUG:boto:Token: None 2023-10-18T20:45:23.876 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/ 2023-10-18T20:45:23.877 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU= 2023-10-18T20:45:23.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YRPp60kTQImyrmYt7qWNgFr06OU=', 'Content-Length': '0'} 2023-10-18T20:45:23.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8dbe188f3fb1768-00653043e3-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.882 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:23.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:45:23.883 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:23.884 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:23.884 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.884 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.885 DEBUG:boto:Method: GET 2023-10-18T20:45:23.885 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.886 DEBUG:boto:Data: 2023-10-18T20:45:23.886 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.887 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.887 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.887 DEBUG:boto:Params: {} 2023-10-18T20:45:23.888 DEBUG:boto:Token: None 2023-10-18T20:45:23.888 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.889 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ayfCf5g9sTqVRbKrtPdwFemusF4= 2023-10-18T20:45:23.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ayfCf5g9sTqVRbKrtPdwFemusF4=', 'Content-Length': '0'} 2023-10-18T20:45:23.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000116441e5d4b411f9-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.892 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:23.892 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:23.893 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.893 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.894 DEBUG:boto:Method: GET 2023-10-18T20:45:23.894 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.895 DEBUG:boto:Data: 2023-10-18T20:45:23.895 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.895 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.896 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.896 DEBUG:boto:Params: {} 2023-10-18T20:45:23.897 DEBUG:boto:Token: None 2023-10-18T20:45:23.897 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.898 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ayfCf5g9sTqVRbKrtPdwFemusF4= 2023-10-18T20:45:23.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ayfCf5g9sTqVRbKrtPdwFemusF4=', 'Content-Length': '0'} 2023-10-18T20:45:23.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000990c452284ef0c00-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.942 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:23.943 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:23.943 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.944 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.944 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.945 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.945 DEBUG:boto:Data: 2023-10-18T20:45:23.946 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.946 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.946 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.947 DEBUG:boto:Params: {} 2023-10-18T20:45:23.947 DEBUG:boto:Token: None 2023-10-18T20:45:23.948 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OKjRaARPGg0A7XBo38LgpbCIZPU= 2023-10-18T20:45:23.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OKjRaARPGg0A7XBo38LgpbCIZPU=', 'Content-Length': '0'} 2023-10-18T20:45:23.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a9b71c35c27a29b-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.951 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:23.952 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:23.952 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.953 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.953 DEBUG:boto:Method: HEAD 2023-10-18T20:45:23.953 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.954 DEBUG:boto:Data: 2023-10-18T20:45:23.954 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.955 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:23.955 DEBUG:boto:Port: 8000 2023-10-18T20:45:23.955 DEBUG:boto:Params: {} 2023-10-18T20:45:23.956 DEBUG:boto:Token: None 2023-10-18T20:45:23.956 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.957 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OKjRaARPGg0A7XBo38LgpbCIZPU= 2023-10-18T20:45:23.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OKjRaARPGg0A7XBo38LgpbCIZPU=', 'Content-Length': '0'} 2023-10-18T20:45:23.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f8e62b99bc893f-00653043e3-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:23.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:45:23.960 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:23.961 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:23.961 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.962 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.962 DEBUG:boto:Method: GET 2023-10-18T20:45:23.963 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.963 DEBUG:boto:Data: 2023-10-18T20:45:23.966 DEBUG:boto:Headers: {} 2023-10-18T20:45:23.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:23.966 DEBUG:boto:Port: 8001 2023-10-18T20:45:23.967 DEBUG:boto:Params: {} 2023-10-18T20:45:23.967 DEBUG:boto:Token: None 2023-10-18T20:45:23.968 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:23 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:23.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ayfCf5g9sTqVRbKrtPdwFemusF4= 2023-10-18T20:45:23.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:23 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ayfCf5g9sTqVRbKrtPdwFemusF4=', 'Content-Length': '0'} 2023-10-18T20:45:23.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0937b1e62a4a291-00653043e3-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.014 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:24.015 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:24.015 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:24.016 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:24.016 DEBUG:boto:Method: GET 2023-10-18T20:45:24.017 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:24.017 DEBUG:boto:Data: 2023-10-18T20:45:24.018 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.018 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.018 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.019 DEBUG:boto:Params: {} 2023-10-18T20:45:24.019 DEBUG:boto:Token: None 2023-10-18T20:45:24.020 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:24.020 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DjWlhz6SixQbNaf6t4B3Q3bHLrg= 2023-10-18T20:45:24.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DjWlhz6SixQbNaf6t4B3Q3bHLrg=', 'Content-Length': '0'} 2023-10-18T20:45:24.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453bdeb9a6489bbb-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.024 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.024 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.025 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.025 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.026 DEBUG:boto:Method: GET 2023-10-18T20:45:24.026 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.027 DEBUG:boto:Data: 2023-10-18T20:45:24.027 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.028 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.028 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.028 DEBUG:boto:Params: {} 2023-10-18T20:45:24.029 DEBUG:boto:Token: None 2023-10-18T20:45:24.029 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000093b7e041caec446-00653043e4-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.033 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.034 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.035 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.035 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.036 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.036 DEBUG:boto:Method: GET 2023-10-18T20:45:24.036 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.037 DEBUG:boto:Data: 2023-10-18T20:45:24.037 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.038 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.038 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.039 DEBUG:boto:Params: {} 2023-10-18T20:45:24.039 DEBUG:boto:Token: None 2023-10-18T20:45:24.040 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047255f26b07f8433-00653043e4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.045 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.046 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:24.047 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:24.047 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.047 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:45:24.048 DEBUG:boto:Data: 2023-10-18T20:45:24.048 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.049 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.049 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.049 DEBUG:boto:Params: {} 2023-10-18T20:45:24.050 DEBUG:boto:Token: None 2023-10-18T20:45:24.050 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj34 2023-10-18T20:45:24.051 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1d8fGx1CCSrPEhL4/dh6yGYI2/0= 2023-10-18T20:45:24.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1d8fGx1CCSrPEhL4/dh6yGYI2/0=', 'Content-Length': '0'} 2023-10-18T20:45:24.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c8c324e5e4504b6-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.054 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:24.054 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:24.055 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.055 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:45:24.056 DEBUG:boto:Data: 2023-10-18T20:45:24.056 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.057 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.058 DEBUG:boto:Params: {} 2023-10-18T20:45:24.058 DEBUG:boto:Token: None 2023-10-18T20:45:24.058 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj34 2023-10-18T20:45:24.059 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1d8fGx1CCSrPEhL4/dh6yGYI2/0= 2023-10-18T20:45:24.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1d8fGx1CCSrPEhL4/dh6yGYI2/0=', 'Content-Length': '0'} 2023-10-18T20:45:24.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4103d78dc67c66-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:45:24.062 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:24.063 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:24.063 DEBUG:boto:Method: GET 2023-10-18T20:45:24.064 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:45:24.064 DEBUG:boto:Data: 2023-10-18T20:45:24.064 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.065 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.065 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.066 DEBUG:boto:Params: {} 2023-10-18T20:45:24.066 DEBUG:boto:Token: None 2023-10-18T20:45:24.067 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj34 2023-10-18T20:45:24.067 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PUugAXADdlXxld0Q2b9Qj7o62ro= 2023-10-18T20:45:24.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PUugAXADdlXxld0Q2b9Qj7o62ro=', 'Content-Length': '0'} 2023-10-18T20:45:24.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aa65786f2e01111-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.070 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:24.071 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:24.071 DEBUG:boto:Method: GET 2023-10-18T20:45:24.072 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:45:24.072 DEBUG:boto:Data: 2023-10-18T20:45:24.072 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.073 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.074 DEBUG:boto:Params: {} 2023-10-18T20:45:24.074 DEBUG:boto:Token: None 2023-10-18T20:45:24.075 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj34 2023-10-18T20:45:24.075 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PUugAXADdlXxld0Q2b9Qj7o62ro= 2023-10-18T20:45:24.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PUugAXADdlXxld0Q2b9Qj7o62ro=', 'Content-Length': '0'} 2023-10-18T20:45:24.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000879260cb97da73ed-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.122 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.123 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.124 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.124 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.124 DEBUG:boto:Method: GET 2023-10-18T20:45:24.125 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.125 DEBUG:boto:Data: 2023-10-18T20:45:24.126 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.126 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.127 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.127 DEBUG:boto:Params: {} 2023-10-18T20:45:24.127 DEBUG:boto:Token: None 2023-10-18T20:45:24.128 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.129 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090fcaa2db420d9e9-00653043e4-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.132 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.133 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.133 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.134 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.134 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.135 DEBUG:boto:Method: GET 2023-10-18T20:45:24.135 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:45:24.135 DEBUG:boto:Data: 2023-10-18T20:45:24.136 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.136 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.137 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.137 DEBUG:boto:Params: {} 2023-10-18T20:45:24.138 DEBUG:boto:Token: None 2023-10-18T20:45:24.138 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.139 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b03438b2a81e7d06-00653043e4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.144 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-18T20:45:24.145 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.145 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.146 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.146 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.147 DEBUG:boto:Method: GET 2023-10-18T20:45:24.147 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.148 DEBUG:boto:Data: 2023-10-18T20:45:24.148 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.148 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.149 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.149 DEBUG:boto:Params: {} 2023-10-18T20:45:24.150 DEBUG:boto:Token: None 2023-10-18T20:45:24.150 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.151 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IMFo6eOJYPnGjqTim064Az0nbHU= 2023-10-18T20:45:24.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IMFo6eOJYPnGjqTim064Az0nbHU=', 'Content-Length': '0'} 2023-10-18T20:45:24.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e93f3ccac866f36b-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.154 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.155 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.155 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.155 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.156 DEBUG:boto:Method: GET 2023-10-18T20:45:24.156 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.157 DEBUG:boto:Data: 2023-10-18T20:45:24.157 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.157 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.158 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.158 DEBUG:boto:Params: {} 2023-10-18T20:45:24.159 DEBUG:boto:Token: None 2023-10-18T20:45:24.159 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.160 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IMFo6eOJYPnGjqTim064Az0nbHU= 2023-10-18T20:45:24.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IMFo6eOJYPnGjqTim064Az0nbHU=', 'Content-Length': '0'} 2023-10-18T20:45:24.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000366c12d908e338d8-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.163 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.163 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.164 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.164 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.165 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.165 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.166 DEBUG:boto:Data: 2023-10-18T20:45:24.166 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.167 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.170 DEBUG:boto:Params: {} 2023-10-18T20:45:24.171 DEBUG:boto:Token: None 2023-10-18T20:45:24.171 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.172 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2Jg4aHaqt2EH5r3Q3cRA92LA4dE= 2023-10-18T20:45:24.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2Jg4aHaqt2EH5r3Q3cRA92LA4dE=', 'Content-Length': '0'} 2023-10-18T20:45:24.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b002dc3859671f-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.175 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.175 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.176 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.176 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.176 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.177 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.177 DEBUG:boto:Data: 2023-10-18T20:45:24.178 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.178 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.179 DEBUG:boto:Params: {} 2023-10-18T20:45:24.179 DEBUG:boto:Token: None 2023-10-18T20:45:24.180 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.180 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2Jg4aHaqt2EH5r3Q3cRA92LA4dE= 2023-10-18T20:45:24.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2Jg4aHaqt2EH5r3Q3cRA92LA4dE=', 'Content-Length': '0'} 2023-10-18T20:45:24.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f2c6aa67d6acbb9-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-18T20:45:24.184 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.184 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.185 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.185 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.186 DEBUG:boto:Method: GET 2023-10-18T20:45:24.186 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.186 DEBUG:boto:Data: 2023-10-18T20:45:24.187 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.187 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.188 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.188 DEBUG:boto:Params: {} 2023-10-18T20:45:24.189 DEBUG:boto:Token: None 2023-10-18T20:45:24.189 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.190 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IMFo6eOJYPnGjqTim064Az0nbHU= 2023-10-18T20:45:24.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IMFo6eOJYPnGjqTim064Az0nbHU=', 'Content-Length': '0'} 2023-10-18T20:45:24.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a09790cffb059c4-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.193 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.194 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.194 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.194 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.195 DEBUG:boto:Method: GET 2023-10-18T20:45:24.195 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.196 DEBUG:boto:Data: 2023-10-18T20:45:24.196 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.197 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.197 DEBUG:boto:Params: {} 2023-10-18T20:45:24.198 DEBUG:boto:Token: None 2023-10-18T20:45:24.198 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:24.199 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IMFo6eOJYPnGjqTim064Az0nbHU= 2023-10-18T20:45:24.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IMFo6eOJYPnGjqTim064Az0nbHU=', 'Content-Length': '0'} 2023-10-18T20:45:24.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000712f9ddb158c6ea5-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.246 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.247 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.248 DEBUG:boto:path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.248 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.249 DEBUG:boto:Method: GET 2023-10-18T20:45:24.249 DEBUG:boto:Path: /bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.250 DEBUG:boto:Data: 2023-10-18T20:45:24.250 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.252 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.252 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.253 DEBUG:boto:Params: {} 2023-10-18T20:45:24.254 DEBUG:boto:Token: None 2023-10-18T20:45:24.255 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.256 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003209ab1ee95d0a95-00653043e4-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.261 DEBUG:boto:b'bofrnf-59obj371000falseobj372023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.262 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.263 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.264 DEBUG:boto:path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.265 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.266 DEBUG:boto:Method: GET 2023-10-18T20:45:24.266 DEBUG:boto:Path: /bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.267 DEBUG:boto:Data: 2023-10-18T20:45:24.268 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.270 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.270 DEBUG:boto:Params: {} 2023-10-18T20:45:24.271 DEBUG:boto:Token: None 2023-10-18T20:45:24.272 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.273 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a23af98392bd1e1b-00653043e4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.280 DEBUG:boto:b'bofrnf-59obj371000falseobj372023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.281 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.282 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.282 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.283 DEBUG:boto:Path: /bofrnf-59/obj37 2023-10-18T20:45:24.284 DEBUG:boto:Data: 2023-10-18T20:45:24.284 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.285 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.285 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.285 DEBUG:boto:Params: {} 2023-10-18T20:45:24.286 DEBUG:boto:Token: None 2023-10-18T20:45:24.286 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37 2023-10-18T20:45:24.287 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bAx182ccT46H4kbjwGg30SVu0MA= 2023-10-18T20:45:24.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bAx182ccT46H4kbjwGg30SVu0MA=', 'Content-Length': '0'} 2023-10-18T20:45:24.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000368183d907e8e2ee-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.290 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.290 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.291 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.292 DEBUG:boto:Path: /bofrnf-59/obj37 2023-10-18T20:45:24.292 DEBUG:boto:Data: 2023-10-18T20:45:24.292 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.293 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.293 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.294 DEBUG:boto:Params: {} 2023-10-18T20:45:24.294 DEBUG:boto:Token: None 2023-10-18T20:45:24.294 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37 2023-10-18T20:45:24.295 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bAx182ccT46H4kbjwGg30SVu0MA= 2023-10-18T20:45:24.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bAx182ccT46H4kbjwGg30SVu0MA=', 'Content-Length': '0'} 2023-10-18T20:45:24.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000783541a0fffc1f15-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-18T20:45:24.299 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.299 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.299 DEBUG:boto:Method: GET 2023-10-18T20:45:24.300 DEBUG:boto:Path: /bofrnf-59/obj37 2023-10-18T20:45:24.301 DEBUG:boto:Data: 2023-10-18T20:45:24.301 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.303 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.304 DEBUG:boto:Params: {} 2023-10-18T20:45:24.305 DEBUG:boto:Token: None 2023-10-18T20:45:24.305 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37 2023-10-18T20:45:24.306 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1jLT+/o8TPsX7y3axiWPu7BMikk= 2023-10-18T20:45:24.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1jLT+/o8TPsX7y3axiWPu7BMikk=', 'Content-Length': '0'} 2023-10-18T20:45:24.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c410c94e43e07425-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.350 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:24.351 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:24.352 DEBUG:boto:Method: GET 2023-10-18T20:45:24.352 DEBUG:boto:Path: /bofrnf-59/obj37 2023-10-18T20:45:24.353 DEBUG:boto:Data: 2023-10-18T20:45:24.353 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.354 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.355 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.355 DEBUG:boto:Params: {} 2023-10-18T20:45:24.356 DEBUG:boto:Token: None 2023-10-18T20:45:24.357 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj37 2023-10-18T20:45:24.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1jLT+/o8TPsX7y3axiWPu7BMikk= 2023-10-18T20:45:24.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1jLT+/o8TPsX7y3axiWPu7BMikk=', 'Content-Length': '0'} 2023-10-18T20:45:24.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abfc4d92d2a36318-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.362 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.362 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.363 DEBUG:boto:path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.363 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.364 DEBUG:boto:Method: GET 2023-10-18T20:45:24.364 DEBUG:boto:Path: /bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.365 DEBUG:boto:Data: 2023-10-18T20:45:24.365 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.366 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.366 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.366 DEBUG:boto:Params: {} 2023-10-18T20:45:24.367 DEBUG:boto:Token: None 2023-10-18T20:45:24.367 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.368 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002066a38be35c4d5a-00653043e4-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.371 DEBUG:boto:b'bofrnf-59obj371000falseobj372023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.373 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.374 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.375 DEBUG:boto:path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.375 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.375 DEBUG:boto:Method: GET 2023-10-18T20:45:24.376 DEBUG:boto:Path: /bofrnf-59/?prefix=obj37 2023-10-18T20:45:24.376 DEBUG:boto:Data: 2023-10-18T20:45:24.377 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.378 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.378 DEBUG:boto:Params: {} 2023-10-18T20:45:24.379 DEBUG:boto:Token: None 2023-10-18T20:45:24.379 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.379 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e0a782d4a8db51-00653043e4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.385 DEBUG:boto:b'bofrnf-59obj371000falseobj372023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:45:24.387 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.387 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.388 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.389 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.390 DEBUG:boto:Method: GET 2023-10-18T20:45:24.391 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.391 DEBUG:boto:Data: 2023-10-18T20:45:24.392 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.392 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.392 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.393 DEBUG:boto:Params: {} 2023-10-18T20:45:24.393 DEBUG:boto:Token: None 2023-10-18T20:45:24.394 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.395 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KIf4OS9EB/3h01SL57KB00uU/dY= 2023-10-18T20:45:24.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KIf4OS9EB/3h01SL57KB00uU/dY=', 'Content-Length': '0'} 2023-10-18T20:45:24.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000298f0acd35e0b42e-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.398 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.398 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.399 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.399 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.400 DEBUG:boto:Method: GET 2023-10-18T20:45:24.400 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.400 DEBUG:boto:Data: 2023-10-18T20:45:24.401 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.402 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.402 DEBUG:boto:Params: {} 2023-10-18T20:45:24.403 DEBUG:boto:Token: None 2023-10-18T20:45:24.403 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KIf4OS9EB/3h01SL57KB00uU/dY= 2023-10-18T20:45:24.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KIf4OS9EB/3h01SL57KB00uU/dY=', 'Content-Length': '0'} 2023-10-18T20:45:24.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b60f9d7f9e1e8d4a-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.450 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.451 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.452 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.452 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.452 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.453 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.453 DEBUG:boto:Data: 2023-10-18T20:45:24.453 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.454 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.455 DEBUG:boto:Params: {} 2023-10-18T20:45:24.455 DEBUG:boto:Token: None 2023-10-18T20:45:24.456 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6v7r4dyhud+WsSf7xggyVMMiwL0= 2023-10-18T20:45:24.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6v7r4dyhud+WsSf7xggyVMMiwL0=', 'Content-Length': '0'} 2023-10-18T20:45:24.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009615bc17018a8923-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.460 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.460 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.461 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.461 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.462 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.462 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.463 DEBUG:boto:Data: 2023-10-18T20:45:24.464 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.464 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.465 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.465 DEBUG:boto:Params: {} 2023-10-18T20:45:24.466 DEBUG:boto:Token: None 2023-10-18T20:45:24.466 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.466 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6v7r4dyhud+WsSf7xggyVMMiwL0= 2023-10-18T20:45:24.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6v7r4dyhud+WsSf7xggyVMMiwL0=', 'Content-Length': '0'} 2023-10-18T20:45:24.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084b9760045006fae-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:45:24.470 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.471 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.471 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.472 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.472 DEBUG:boto:Method: GET 2023-10-18T20:45:24.473 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.473 DEBUG:boto:Data: 2023-10-18T20:45:24.473 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.474 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.475 DEBUG:boto:Params: {} 2023-10-18T20:45:24.475 DEBUG:boto:Token: None 2023-10-18T20:45:24.476 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KIf4OS9EB/3h01SL57KB00uU/dY= 2023-10-18T20:45:24.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KIf4OS9EB/3h01SL57KB00uU/dY=', 'Content-Length': '0'} 2023-10-18T20:45:24.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000840de0a345f8ab1c-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.526 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.527 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.528 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.528 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.529 DEBUG:boto:Method: GET 2023-10-18T20:45:24.529 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.529 DEBUG:boto:Data: 2023-10-18T20:45:24.530 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.531 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.531 DEBUG:boto:Params: {} 2023-10-18T20:45:24.532 DEBUG:boto:Token: None 2023-10-18T20:45:24.533 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:24.534 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KIf4OS9EB/3h01SL57KB00uU/dY= 2023-10-18T20:45:24.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KIf4OS9EB/3h01SL57KB00uU/dY=', 'Content-Length': '0'} 2023-10-18T20:45:24.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010059c088077c60a-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.537 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.538 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.538 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.539 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.539 DEBUG:boto:Method: GET 2023-10-18T20:45:24.540 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.540 DEBUG:boto:Data: 2023-10-18T20:45:24.541 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.541 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.542 DEBUG:boto:Params: {} 2023-10-18T20:45:24.542 DEBUG:boto:Token: None 2023-10-18T20:45:24.543 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000195206014ac8dc8e-00653043e4-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.548 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.548 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.549 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.549 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.550 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.550 DEBUG:boto:Method: GET 2023-10-18T20:45:24.551 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.551 DEBUG:boto:Data: 2023-10-18T20:45:24.552 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.552 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.553 DEBUG:boto:Params: {} 2023-10-18T20:45:24.553 DEBUG:boto:Token: None 2023-10-18T20:45:24.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7df657fd7864ebc-00653043e4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.561 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.562 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.563 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.564 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.564 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:45:24.565 DEBUG:boto:Data: 2023-10-18T20:45:24.566 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.567 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.567 DEBUG:boto:Params: {} 2023-10-18T20:45:24.568 DEBUG:boto:Token: None 2023-10-18T20:45:24.569 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38 2023-10-18T20:45:24.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j6MhBtiw9y5qZlVcmRXL3L3HwFQ= 2023-10-18T20:45:24.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j6MhBtiw9y5qZlVcmRXL3L3HwFQ=', 'Content-Length': '0'} 2023-10-18T20:45:24.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be23d967ce9fe049-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.572 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.573 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.573 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.574 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:45:24.574 DEBUG:boto:Data: 2023-10-18T20:45:24.575 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.575 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.576 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.576 DEBUG:boto:Params: {} 2023-10-18T20:45:24.576 DEBUG:boto:Token: None 2023-10-18T20:45:24.577 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38 2023-10-18T20:45:24.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:j6MhBtiw9y5qZlVcmRXL3L3HwFQ= 2023-10-18T20:45:24.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:j6MhBtiw9y5qZlVcmRXL3L3HwFQ=', 'Content-Length': '0'} 2023-10-18T20:45:24.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad1bceaec4cfc8b1-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:45:24.581 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.582 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.582 DEBUG:boto:Method: GET 2023-10-18T20:45:24.583 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:45:24.583 DEBUG:boto:Data: 2023-10-18T20:45:24.584 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.584 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.584 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.585 DEBUG:boto:Params: {} 2023-10-18T20:45:24.585 DEBUG:boto:Token: None 2023-10-18T20:45:24.586 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38 2023-10-18T20:45:24.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9+rh8U/ySXRX3zWAw33zgEV2p1g= 2023-10-18T20:45:24.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9+rh8U/ySXRX3zWAw33zgEV2p1g=', 'Content-Length': '0'} 2023-10-18T20:45:24.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000607b486779b14dfa-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.590 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:24.590 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:24.591 DEBUG:boto:Method: GET 2023-10-18T20:45:24.591 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:45:24.592 DEBUG:boto:Data: 2023-10-18T20:45:24.592 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.593 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.593 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.594 DEBUG:boto:Params: {} 2023-10-18T20:45:24.594 DEBUG:boto:Token: None 2023-10-18T20:45:24.595 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj38 2023-10-18T20:45:24.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9+rh8U/ySXRX3zWAw33zgEV2p1g= 2023-10-18T20:45:24.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9+rh8U/ySXRX3zWAw33zgEV2p1g=', 'Content-Length': '0'} 2023-10-18T20:45:24.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e22982f19e05286d-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.600 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.600 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.601 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.602 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.602 DEBUG:boto:Method: GET 2023-10-18T20:45:24.603 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.604 DEBUG:boto:Data: 2023-10-18T20:45:24.604 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.605 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.605 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.606 DEBUG:boto:Params: {} 2023-10-18T20:45:24.606 DEBUG:boto:Token: None 2023-10-18T20:45:24.607 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.607 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b053cad3d921915-00653043e4-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.611 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.612 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.613 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.614 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.614 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.615 DEBUG:boto:Method: GET 2023-10-18T20:45:24.616 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:45:24.616 DEBUG:boto:Data: 2023-10-18T20:45:24.617 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.617 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.618 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.619 DEBUG:boto:Params: {} 2023-10-18T20:45:24.619 DEBUG:boto:Token: None 2023-10-18T20:45:24.620 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.620 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fceffa49ea9c429-00653043e4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.666 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:45:24.668 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.668 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.669 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.669 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.670 DEBUG:boto:Method: GET 2023-10-18T20:45:24.670 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.671 DEBUG:boto:Data: 2023-10-18T20:45:24.671 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.672 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.672 DEBUG:boto:Params: {} 2023-10-18T20:45:24.673 DEBUG:boto:Token: None 2023-10-18T20:45:24.673 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.674 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ntN45sioi/+XDFLnaW19obThM9E= 2023-10-18T20:45:24.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ntN45sioi/+XDFLnaW19obThM9E=', 'Content-Length': '0'} 2023-10-18T20:45:24.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000042e806301240b8e-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.679 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.680 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.680 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.681 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.681 DEBUG:boto:Method: GET 2023-10-18T20:45:24.682 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.682 DEBUG:boto:Data: 2023-10-18T20:45:24.683 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.683 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.684 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.684 DEBUG:boto:Params: {} 2023-10-18T20:45:24.684 DEBUG:boto:Token: None 2023-10-18T20:45:24.685 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.685 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ntN45sioi/+XDFLnaW19obThM9E= 2023-10-18T20:45:24.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ntN45sioi/+XDFLnaW19obThM9E=', 'Content-Length': '0'} 2023-10-18T20:45:24.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057fcdf70557088de-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.689 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.689 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.690 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.690 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.690 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.691 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.691 DEBUG:boto:Data: 2023-10-18T20:45:24.692 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.692 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.693 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.693 DEBUG:boto:Params: {} 2023-10-18T20:45:24.693 DEBUG:boto:Token: None 2023-10-18T20:45:24.694 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.694 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gaggxb4vFpoDbStqgEWt4aG7O2I= 2023-10-18T20:45:24.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gaggxb4vFpoDbStqgEWt4aG7O2I=', 'Content-Length': '0'} 2023-10-18T20:45:24.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87611377cf3cf10-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.698 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.699 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.699 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.700 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.701 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.702 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.703 DEBUG:boto:Data: 2023-10-18T20:45:24.704 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.705 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.706 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.706 DEBUG:boto:Params: {} 2023-10-18T20:45:24.707 DEBUG:boto:Token: None 2023-10-18T20:45:24.708 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gaggxb4vFpoDbStqgEWt4aG7O2I= 2023-10-18T20:45:24.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gaggxb4vFpoDbStqgEWt4aG7O2I=', 'Content-Length': '0'} 2023-10-18T20:45:24.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c4a5f6c1c5c5c3-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:45:24.714 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.715 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.716 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.716 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.717 DEBUG:boto:Method: GET 2023-10-18T20:45:24.718 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.719 DEBUG:boto:Data: 2023-10-18T20:45:24.721 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.721 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.722 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.723 DEBUG:boto:Params: {} 2023-10-18T20:45:24.724 DEBUG:boto:Token: None 2023-10-18T20:45:24.725 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.726 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ntN45sioi/+XDFLnaW19obThM9E= 2023-10-18T20:45:24.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ntN45sioi/+XDFLnaW19obThM9E=', 'Content-Length': '0'} 2023-10-18T20:45:24.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5376fbbe036997a-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.731 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.732 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.733 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.734 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.735 DEBUG:boto:Method: GET 2023-10-18T20:45:24.737 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.738 DEBUG:boto:Data: 2023-10-18T20:45:24.739 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.741 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.742 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.743 DEBUG:boto:Params: {} 2023-10-18T20:45:24.744 DEBUG:boto:Token: None 2023-10-18T20:45:24.744 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:24.745 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ntN45sioi/+XDFLnaW19obThM9E= 2023-10-18T20:45:24.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ntN45sioi/+XDFLnaW19obThM9E=', 'Content-Length': '0'} 2023-10-18T20:45:24.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e69223f76ecc279-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.749 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.750 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.751 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.752 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.752 DEBUG:boto:Method: GET 2023-10-18T20:45:24.753 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.754 DEBUG:boto:Data: 2023-10-18T20:45:24.755 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.755 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.756 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.757 DEBUG:boto:Params: {} 2023-10-18T20:45:24.757 DEBUG:boto:Token: None 2023-10-18T20:45:24.758 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.759 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f30ece1ed5c708f-00653043e4-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.763 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.764 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.765 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.766 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.767 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.768 DEBUG:boto:Method: GET 2023-10-18T20:45:24.768 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.769 DEBUG:boto:Data: 2023-10-18T20:45:24.770 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.771 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.772 DEBUG:boto:Params: {} 2023-10-18T20:45:24.772 DEBUG:boto:Token: None 2023-10-18T20:45:24.773 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065ee76ee7dcc1d3d-00653043e4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.780 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.781 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.782 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.785 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.787 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:45:24.787 DEBUG:boto:Data: 2023-10-18T20:45:24.788 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.789 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.790 DEBUG:boto:Params: {} 2023-10-18T20:45:24.790 DEBUG:boto:Token: None 2023-10-18T20:45:24.791 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44 2023-10-18T20:45:24.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tWEYP4vfG/oauaj97Uf04lpNjyQ= 2023-10-18T20:45:24.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tWEYP4vfG/oauaj97Uf04lpNjyQ=', 'Content-Length': '0'} 2023-10-18T20:45:24.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d67fe1917179438-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.796 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.796 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.797 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.798 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:45:24.798 DEBUG:boto:Data: 2023-10-18T20:45:24.799 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.800 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.801 DEBUG:boto:Params: {} 2023-10-18T20:45:24.801 DEBUG:boto:Token: None 2023-10-18T20:45:24.802 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44 2023-10-18T20:45:24.803 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tWEYP4vfG/oauaj97Uf04lpNjyQ= 2023-10-18T20:45:24.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tWEYP4vfG/oauaj97Uf04lpNjyQ=', 'Content-Length': '0'} 2023-10-18T20:45:24.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b682881d8d65396c-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:45:24.807 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.808 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.808 DEBUG:boto:Method: GET 2023-10-18T20:45:24.809 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:45:24.810 DEBUG:boto:Data: 2023-10-18T20:45:24.810 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.811 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.811 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.812 DEBUG:boto:Params: {} 2023-10-18T20:45:24.813 DEBUG:boto:Token: None 2023-10-18T20:45:24.813 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44 2023-10-18T20:45:24.814 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p3Fihw2GzNN69XvuXBWil5pfTzo= 2023-10-18T20:45:24.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p3Fihw2GzNN69XvuXBWil5pfTzo=', 'Content-Length': '0'} 2023-10-18T20:45:24.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000964b3f447322bd01-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.858 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:24.859 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:24.860 DEBUG:boto:Method: GET 2023-10-18T20:45:24.860 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:45:24.861 DEBUG:boto:Data: 2023-10-18T20:45:24.861 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.861 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.862 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.862 DEBUG:boto:Params: {} 2023-10-18T20:45:24.863 DEBUG:boto:Token: None 2023-10-18T20:45:24.863 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj44 2023-10-18T20:45:24.864 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p3Fihw2GzNN69XvuXBWil5pfTzo= 2023-10-18T20:45:24.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p3Fihw2GzNN69XvuXBWil5pfTzo=', 'Content-Length': '0'} 2023-10-18T20:45:24.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923ed4576cd48958-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.868 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.869 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.869 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.869 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.870 DEBUG:boto:Method: GET 2023-10-18T20:45:24.870 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.871 DEBUG:boto:Data: 2023-10-18T20:45:24.871 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.872 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.872 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.873 DEBUG:boto:Params: {} 2023-10-18T20:45:24.873 DEBUG:boto:Token: None 2023-10-18T20:45:24.874 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.874 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e41738675096bede-00653043e4-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.879 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.880 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:24.880 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:24.880 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.881 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.881 DEBUG:boto:Method: GET 2023-10-18T20:45:24.882 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:45:24.882 DEBUG:boto:Data: 2023-10-18T20:45:24.883 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.883 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.884 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.884 DEBUG:boto:Params: {} 2023-10-18T20:45:24.885 DEBUG:boto:Token: None 2023-10-18T20:45:24.885 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/ 2023-10-18T20:45:24.886 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk= 2023-10-18T20:45:24.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p+pvmAVmlpLYSwsbo5aQhQYGEsk=', 'Content-Length': '0'} 2023-10-18T20:45:24.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4b2f34e6e3eda71-00653043e4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.938 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:24.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:45:24.940 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:24.941 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:24.942 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.943 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.944 DEBUG:boto:Method: GET 2023-10-18T20:45:24.944 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.945 DEBUG:boto:Data: 2023-10-18T20:45:24.945 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.946 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.947 DEBUG:boto:Params: {} 2023-10-18T20:45:24.947 DEBUG:boto:Token: None 2023-10-18T20:45:24.948 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O4/pwQpGBtVjh9zTqfS4B8msb2Y= 2023-10-18T20:45:24.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O4/pwQpGBtVjh9zTqfS4B8msb2Y=', 'Content-Length': '0'} 2023-10-18T20:45:24.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c44bd3fb8ec5b4-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.951 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:24.952 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:24.952 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.953 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.953 DEBUG:boto:Method: GET 2023-10-18T20:45:24.954 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.954 DEBUG:boto:Data: 2023-10-18T20:45:24.955 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.955 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.956 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.956 DEBUG:boto:Params: {} 2023-10-18T20:45:24.957 DEBUG:boto:Token: None 2023-10-18T20:45:24.957 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.958 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O4/pwQpGBtVjh9zTqfS4B8msb2Y= 2023-10-18T20:45:24.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O4/pwQpGBtVjh9zTqfS4B8msb2Y=', 'Content-Length': '0'} 2023-10-18T20:45:24.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000291aab0e073e4981-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.961 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:24.962 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:24.962 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.962 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.963 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.963 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.964 DEBUG:boto:Data: 2023-10-18T20:45:24.964 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.965 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.965 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.966 DEBUG:boto:Params: {} 2023-10-18T20:45:24.966 DEBUG:boto:Token: None 2023-10-18T20:45:24.967 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.967 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iEiezrXD/0NqbfXT3oW+M05M0WE= 2023-10-18T20:45:24.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iEiezrXD/0NqbfXT3oW+M05M0WE=', 'Content-Length': '0'} 2023-10-18T20:45:24.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2724b2c91844497-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.970 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:24.971 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:24.971 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.972 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.972 DEBUG:boto:Method: HEAD 2023-10-18T20:45:24.973 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.973 DEBUG:boto:Data: 2023-10-18T20:45:24.974 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.975 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.975 DEBUG:boto:Params: {} 2023-10-18T20:45:24.976 DEBUG:boto:Token: None 2023-10-18T20:45:24.976 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.977 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iEiezrXD/0NqbfXT3oW+M05M0WE= 2023-10-18T20:45:24.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iEiezrXD/0NqbfXT3oW+M05M0WE=', 'Content-Length': '0'} 2023-10-18T20:45:24.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fb77070e991ac07-00653043e4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:45:24.981 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:24.982 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:24.982 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.983 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.983 DEBUG:boto:Method: GET 2023-10-18T20:45:24.984 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.984 DEBUG:boto:Data: 2023-10-18T20:45:24.985 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.985 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:24.986 DEBUG:boto:Port: 8001 2023-10-18T20:45:24.987 DEBUG:boto:Params: {} 2023-10-18T20:45:24.987 DEBUG:boto:Token: None 2023-10-18T20:45:24.988 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:24 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.989 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O4/pwQpGBtVjh9zTqfS4B8msb2Y= 2023-10-18T20:45:24.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O4/pwQpGBtVjh9zTqfS4B8msb2Y=', 'Content-Length': '0'} 2023-10-18T20:45:24.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da82521fd9b58ddf-00653043e4-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:24.994 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:24.995 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:24.995 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.996 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.996 DEBUG:boto:Method: GET 2023-10-18T20:45:24.997 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:24.997 DEBUG:boto:Data: 2023-10-18T20:45:24.997 DEBUG:boto:Headers: {} 2023-10-18T20:45:24.998 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:24.998 DEBUG:boto:Port: 8000 2023-10-18T20:45:24.999 DEBUG:boto:Params: {} 2023-10-18T20:45:24.999 DEBUG:boto:Token: None 2023-10-18T20:45:25.000 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:25.000 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q6qFi7/BCMl9Fc8RyPvsMRJOzwI= 2023-10-18T20:45:25.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q6qFi7/BCMl9Fc8RyPvsMRJOzwI=', 'Content-Length': '0'} 2023-10-18T20:45:25.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba1c4a644677d67-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.003 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.004 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.004 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.005 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.005 DEBUG:boto:Method: GET 2023-10-18T20:45:25.006 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.006 DEBUG:boto:Data: 2023-10-18T20:45:25.007 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.008 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.008 DEBUG:boto:Params: {} 2023-10-18T20:45:25.008 DEBUG:boto:Token: None 2023-10-18T20:45:25.009 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d8f896e5ed19500-00653043e5-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.013 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.014 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.014 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.015 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.015 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.016 DEBUG:boto:Method: GET 2023-10-18T20:45:25.016 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.017 DEBUG:boto:Data: 2023-10-18T20:45:25.017 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.017 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.018 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.018 DEBUG:boto:Params: {} 2023-10-18T20:45:25.019 DEBUG:boto:Token: None 2023-10-18T20:45:25.019 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.020 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044317e1e09d91b37-00653043e5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.066 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.068 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:25.068 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:25.069 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.069 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:45:25.070 DEBUG:boto:Data: 2023-10-18T20:45:25.070 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.071 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.071 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.072 DEBUG:boto:Params: {} 2023-10-18T20:45:25.072 DEBUG:boto:Token: None 2023-10-18T20:45:25.073 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj46 2023-10-18T20:45:25.073 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qbVbhW91WX4z99Vl7cvPeFSZt7Y= 2023-10-18T20:45:25.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qbVbhW91WX4z99Vl7cvPeFSZt7Y=', 'Content-Length': '0'} 2023-10-18T20:45:25.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebdd622bf73ea8ca-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.076 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:25.077 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:25.078 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.078 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:45:25.078 DEBUG:boto:Data: 2023-10-18T20:45:25.079 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.079 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.080 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.080 DEBUG:boto:Params: {} 2023-10-18T20:45:25.080 DEBUG:boto:Token: None 2023-10-18T20:45:25.081 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj46 2023-10-18T20:45:25.081 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qbVbhW91WX4z99Vl7cvPeFSZt7Y= 2023-10-18T20:45:25.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qbVbhW91WX4z99Vl7cvPeFSZt7Y=', 'Content-Length': '0'} 2023-10-18T20:45:25.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db16bb6b767da0ca-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:45:25.085 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:25.085 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:25.086 DEBUG:boto:Method: GET 2023-10-18T20:45:25.086 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:45:25.087 DEBUG:boto:Data: 2023-10-18T20:45:25.087 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.088 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.088 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.089 DEBUG:boto:Params: {} 2023-10-18T20:45:25.089 DEBUG:boto:Token: None 2023-10-18T20:45:25.089 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj46 2023-10-18T20:45:25.090 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tBQzyCOf5diU4YfZ7EMydcEoJIk= 2023-10-18T20:45:25.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tBQzyCOf5diU4YfZ7EMydcEoJIk=', 'Content-Length': '0'} 2023-10-18T20:45:25.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de248525e6b46355-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.093 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:25.094 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:25.094 DEBUG:boto:Method: GET 2023-10-18T20:45:25.095 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:45:25.095 DEBUG:boto:Data: 2023-10-18T20:45:25.096 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.096 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.097 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.097 DEBUG:boto:Params: {} 2023-10-18T20:45:25.098 DEBUG:boto:Token: None 2023-10-18T20:45:25.098 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj46 2023-10-18T20:45:25.099 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tBQzyCOf5diU4YfZ7EMydcEoJIk= 2023-10-18T20:45:25.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tBQzyCOf5diU4YfZ7EMydcEoJIk=', 'Content-Length': '0'} 2023-10-18T20:45:25.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a9178d95ee4fabe-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.102 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.102 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.103 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.103 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.104 DEBUG:boto:Method: GET 2023-10-18T20:45:25.104 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.104 DEBUG:boto:Data: 2023-10-18T20:45:25.105 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.105 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.106 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.107 DEBUG:boto:Params: {} 2023-10-18T20:45:25.107 DEBUG:boto:Token: None 2023-10-18T20:45:25.108 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a983d6d4770d91fe-00653043e5-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.112 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.112 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.113 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.113 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.114 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.114 DEBUG:boto:Method: GET 2023-10-18T20:45:25.114 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:45:25.115 DEBUG:boto:Data: 2023-10-18T20:45:25.115 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.116 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.116 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.117 DEBUG:boto:Params: {} 2023-10-18T20:45:25.117 DEBUG:boto:Token: None 2023-10-18T20:45:25.118 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.118 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0e428e162e1f851-00653043e5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.124 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:45:25.125 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.126 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.126 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.127 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.127 DEBUG:boto:Method: GET 2023-10-18T20:45:25.128 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.128 DEBUG:boto:Data: 2023-10-18T20:45:25.129 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.129 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.129 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.130 DEBUG:boto:Params: {} 2023-10-18T20:45:25.131 DEBUG:boto:Token: None 2023-10-18T20:45:25.131 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.132 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SFU2mOXiuis361i+0+3TXGmWZKM= 2023-10-18T20:45:25.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SFU2mOXiuis361i+0+3TXGmWZKM=', 'Content-Length': '0'} 2023-10-18T20:45:25.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044ca41dd622c6e09-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.182 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.183 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.183 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.184 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.184 DEBUG:boto:Method: GET 2023-10-18T20:45:25.185 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.185 DEBUG:boto:Data: 2023-10-18T20:45:25.186 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.186 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.186 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.187 DEBUG:boto:Params: {} 2023-10-18T20:45:25.187 DEBUG:boto:Token: None 2023-10-18T20:45:25.188 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.188 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SFU2mOXiuis361i+0+3TXGmWZKM= 2023-10-18T20:45:25.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SFU2mOXiuis361i+0+3TXGmWZKM=', 'Content-Length': '0'} 2023-10-18T20:45:25.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcfada773e51c033-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.194 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.194 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.195 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.195 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.196 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.196 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.197 DEBUG:boto:Data: 2023-10-18T20:45:25.197 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.198 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.198 DEBUG:boto:Params: {} 2023-10-18T20:45:25.199 DEBUG:boto:Token: None 2023-10-18T20:45:25.199 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.200 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3+sG5ubiBxyDLJ8M2AVgyqbSIT0= 2023-10-18T20:45:25.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3+sG5ubiBxyDLJ8M2AVgyqbSIT0=', 'Content-Length': '0'} 2023-10-18T20:45:25.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dabe91a6bfa40d5-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.203 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.204 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.204 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.205 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.206 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.207 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.207 DEBUG:boto:Data: 2023-10-18T20:45:25.208 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.209 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.209 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.210 DEBUG:boto:Params: {} 2023-10-18T20:45:25.211 DEBUG:boto:Token: None 2023-10-18T20:45:25.211 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.212 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3+sG5ubiBxyDLJ8M2AVgyqbSIT0= 2023-10-18T20:45:25.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3+sG5ubiBxyDLJ8M2AVgyqbSIT0=', 'Content-Length': '0'} 2023-10-18T20:45:25.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d732a44c1eb9cf31-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:45:25.215 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.216 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.216 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.216 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.217 DEBUG:boto:Method: GET 2023-10-18T20:45:25.217 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.218 DEBUG:boto:Data: 2023-10-18T20:45:25.218 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.219 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.220 DEBUG:boto:Params: {} 2023-10-18T20:45:25.220 DEBUG:boto:Token: None 2023-10-18T20:45:25.221 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SFU2mOXiuis361i+0+3TXGmWZKM= 2023-10-18T20:45:25.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SFU2mOXiuis361i+0+3TXGmWZKM=', 'Content-Length': '0'} 2023-10-18T20:45:25.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d193cd8fb6ed4170-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.225 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.225 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.226 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.226 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.227 DEBUG:boto:Method: GET 2023-10-18T20:45:25.227 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.228 DEBUG:boto:Data: 2023-10-18T20:45:25.228 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.228 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.229 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.229 DEBUG:boto:Params: {} 2023-10-18T20:45:25.230 DEBUG:boto:Token: None 2023-10-18T20:45:25.230 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:25.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SFU2mOXiuis361i+0+3TXGmWZKM= 2023-10-18T20:45:25.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SFU2mOXiuis361i+0+3TXGmWZKM=', 'Content-Length': '0'} 2023-10-18T20:45:25.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e3a37e4076f7ccd-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.234 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.234 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.235 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.235 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.236 DEBUG:boto:Method: GET 2023-10-18T20:45:25.236 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.236 DEBUG:boto:Data: 2023-10-18T20:45:25.237 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.237 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.238 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.238 DEBUG:boto:Params: {} 2023-10-18T20:45:25.239 DEBUG:boto:Token: None 2023-10-18T20:45:25.239 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001423075e3ee5d31f-00653043e5-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.243 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.244 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.244 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.245 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.245 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.246 DEBUG:boto:Method: GET 2023-10-18T20:45:25.246 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.247 DEBUG:boto:Data: 2023-10-18T20:45:25.247 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.247 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.248 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.248 DEBUG:boto:Params: {} 2023-10-18T20:45:25.249 DEBUG:boto:Token: None 2023-10-18T20:45:25.249 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b15e0d426612f5d-00653043e5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.255 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.256 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.256 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.257 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.257 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:45:25.258 DEBUG:boto:Data: 2023-10-18T20:45:25.258 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.259 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.259 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.259 DEBUG:boto:Params: {} 2023-10-18T20:45:25.260 DEBUG:boto:Token: None 2023-10-18T20:45:25.260 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51 2023-10-18T20:45:25.261 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hTeyn5vQ1x8Qb3OypXpsqCH9InM= 2023-10-18T20:45:25.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hTeyn5vQ1x8Qb3OypXpsqCH9InM=', 'Content-Length': '0'} 2023-10-18T20:45:25.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a84116f0f190ae70-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.264 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.264 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.265 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.265 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:45:25.265 DEBUG:boto:Data: 2023-10-18T20:45:25.266 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.266 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.267 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.267 DEBUG:boto:Params: {} 2023-10-18T20:45:25.268 DEBUG:boto:Token: None 2023-10-18T20:45:25.268 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51 2023-10-18T20:45:25.269 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hTeyn5vQ1x8Qb3OypXpsqCH9InM= 2023-10-18T20:45:25.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hTeyn5vQ1x8Qb3OypXpsqCH9InM=', 'Content-Length': '0'} 2023-10-18T20:45:25.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc1f942409f20c8f-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:45:25.272 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.273 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.274 DEBUG:boto:Method: GET 2023-10-18T20:45:25.274 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:45:25.275 DEBUG:boto:Data: 2023-10-18T20:45:25.276 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.277 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.277 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.278 DEBUG:boto:Params: {} 2023-10-18T20:45:25.279 DEBUG:boto:Token: None 2023-10-18T20:45:25.279 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51 2023-10-18T20:45:25.280 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YdGIi2XfXVZPcH4dqv7Jq6qRvXk= 2023-10-18T20:45:25.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YdGIi2XfXVZPcH4dqv7Jq6qRvXk=', 'Content-Length': '0'} 2023-10-18T20:45:25.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee8d501b44e0a58-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.326 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:25.327 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:25.327 DEBUG:boto:Method: GET 2023-10-18T20:45:25.328 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:45:25.328 DEBUG:boto:Data: 2023-10-18T20:45:25.329 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.329 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.330 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.330 DEBUG:boto:Params: {} 2023-10-18T20:45:25.330 DEBUG:boto:Token: None 2023-10-18T20:45:25.331 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj51 2023-10-18T20:45:25.331 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YdGIi2XfXVZPcH4dqv7Jq6qRvXk= 2023-10-18T20:45:25.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YdGIi2XfXVZPcH4dqv7Jq6qRvXk=', 'Content-Length': '0'} 2023-10-18T20:45:25.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a99d6a02950826-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.335 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.335 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.336 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.336 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.336 DEBUG:boto:Method: GET 2023-10-18T20:45:25.337 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.337 DEBUG:boto:Data: 2023-10-18T20:45:25.338 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.339 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.339 DEBUG:boto:Params: {} 2023-10-18T20:45:25.339 DEBUG:boto:Token: None 2023-10-18T20:45:25.340 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce403f09fed7baf0-00653043e5-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.344 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.345 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.345 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.345 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.346 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.346 DEBUG:boto:Method: GET 2023-10-18T20:45:25.347 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:45:25.347 DEBUG:boto:Data: 2023-10-18T20:45:25.348 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.348 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.348 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.349 DEBUG:boto:Params: {} 2023-10-18T20:45:25.349 DEBUG:boto:Token: None 2023-10-18T20:45:25.350 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.350 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000137697534fb9db7-00653043e5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.356 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:45:25.357 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.358 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.358 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.359 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.359 DEBUG:boto:Method: GET 2023-10-18T20:45:25.359 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.360 DEBUG:boto:Data: 2023-10-18T20:45:25.360 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.361 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.361 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.362 DEBUG:boto:Params: {} 2023-10-18T20:45:25.362 DEBUG:boto:Token: None 2023-10-18T20:45:25.363 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.363 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8GONdC7GYaSf9iLO6Q2WGI5qCFA= 2023-10-18T20:45:25.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8GONdC7GYaSf9iLO6Q2WGI5qCFA=', 'Content-Length': '0'} 2023-10-18T20:45:25.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d88f4742eae550fd-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.366 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.367 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.368 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.368 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.368 DEBUG:boto:Method: GET 2023-10-18T20:45:25.369 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.369 DEBUG:boto:Data: 2023-10-18T20:45:25.370 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.370 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.371 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.371 DEBUG:boto:Params: {} 2023-10-18T20:45:25.372 DEBUG:boto:Token: None 2023-10-18T20:45:25.372 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8GONdC7GYaSf9iLO6Q2WGI5qCFA= 2023-10-18T20:45:25.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8GONdC7GYaSf9iLO6Q2WGI5qCFA=', 'Content-Length': '0'} 2023-10-18T20:45:25.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d2fcbbd65133cdd-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.376 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.376 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.377 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.377 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.377 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.378 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.378 DEBUG:boto:Data: 2023-10-18T20:45:25.379 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.379 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.380 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.380 DEBUG:boto:Params: {} 2023-10-18T20:45:25.380 DEBUG:boto:Token: None 2023-10-18T20:45:25.381 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.381 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A8jx/0OzFfseIG5oAvm9kXrfOTg= 2023-10-18T20:45:25.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A8jx/0OzFfseIG5oAvm9kXrfOTg=', 'Content-Length': '0'} 2023-10-18T20:45:25.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e9f7b41962cd88d-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.384 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.385 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.385 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.386 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.386 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.387 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.387 DEBUG:boto:Data: 2023-10-18T20:45:25.388 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.388 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.389 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.389 DEBUG:boto:Params: {} 2023-10-18T20:45:25.390 DEBUG:boto:Token: None 2023-10-18T20:45:25.390 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.391 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A8jx/0OzFfseIG5oAvm9kXrfOTg= 2023-10-18T20:45:25.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A8jx/0OzFfseIG5oAvm9kXrfOTg=', 'Content-Length': '0'} 2023-10-18T20:45:25.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096719e1b20c1eaa9-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:45:25.394 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.395 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.395 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.396 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.396 DEBUG:boto:Method: GET 2023-10-18T20:45:25.396 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.397 DEBUG:boto:Data: 2023-10-18T20:45:25.397 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.398 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.398 DEBUG:boto:Params: {} 2023-10-18T20:45:25.399 DEBUG:boto:Token: None 2023-10-18T20:45:25.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8GONdC7GYaSf9iLO6Q2WGI5qCFA= 2023-10-18T20:45:25.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8GONdC7GYaSf9iLO6Q2WGI5qCFA=', 'Content-Length': '0'} 2023-10-18T20:45:25.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083d3638f470f4718-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.406 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.407 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.408 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.408 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.409 DEBUG:boto:Method: GET 2023-10-18T20:45:25.410 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.411 DEBUG:boto:Data: 2023-10-18T20:45:25.412 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.412 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.413 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.414 DEBUG:boto:Params: {} 2023-10-18T20:45:25.415 DEBUG:boto:Token: None 2023-10-18T20:45:25.415 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:25.416 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8GONdC7GYaSf9iLO6Q2WGI5qCFA= 2023-10-18T20:45:25.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8GONdC7GYaSf9iLO6Q2WGI5qCFA=', 'Content-Length': '0'} 2023-10-18T20:45:25.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045962b12f5747a77-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.420 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.421 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.421 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.422 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.422 DEBUG:boto:Method: GET 2023-10-18T20:45:25.423 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.423 DEBUG:boto:Data: 2023-10-18T20:45:25.423 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.424 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.425 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.425 DEBUG:boto:Params: {} 2023-10-18T20:45:25.426 DEBUG:boto:Token: None 2023-10-18T20:45:25.427 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dc8250aa6cfc6ca-00653043e5-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.433 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.433 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.434 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.434 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.435 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.435 DEBUG:boto:Method: GET 2023-10-18T20:45:25.436 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.436 DEBUG:boto:Data: 2023-10-18T20:45:25.436 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.437 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.437 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.438 DEBUG:boto:Params: {} 2023-10-18T20:45:25.438 DEBUG:boto:Token: None 2023-10-18T20:45:25.439 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.439 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008595d64249f751e-00653043e5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.444 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.445 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.445 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.446 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.446 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:45:25.447 DEBUG:boto:Data: 2023-10-18T20:45:25.447 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.447 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.448 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.448 DEBUG:boto:Params: {} 2023-10-18T20:45:25.449 DEBUG:boto:Token: None 2023-10-18T20:45:25.449 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55 2023-10-18T20:45:25.449 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vB/w2azEAcg4FpvU/MzOxNcFeps= 2023-10-18T20:45:25.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vB/w2azEAcg4FpvU/MzOxNcFeps=', 'Content-Length': '0'} 2023-10-18T20:45:25.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027bfa1968d156512-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.453 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.453 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.454 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.454 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:45:25.454 DEBUG:boto:Data: 2023-10-18T20:45:25.455 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.455 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.456 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.456 DEBUG:boto:Params: {} 2023-10-18T20:45:25.456 DEBUG:boto:Token: None 2023-10-18T20:45:25.457 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55 2023-10-18T20:45:25.457 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vB/w2azEAcg4FpvU/MzOxNcFeps= 2023-10-18T20:45:25.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vB/w2azEAcg4FpvU/MzOxNcFeps=', 'Content-Length': '0'} 2023-10-18T20:45:25.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6f452de090aeabc-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:45:25.461 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.461 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.462 DEBUG:boto:Method: GET 2023-10-18T20:45:25.462 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:45:25.463 DEBUG:boto:Data: 2023-10-18T20:45:25.463 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.463 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.464 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.464 DEBUG:boto:Params: {} 2023-10-18T20:45:25.465 DEBUG:boto:Token: None 2023-10-18T20:45:25.466 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55 2023-10-18T20:45:25.466 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:66EL2Q/2//ZW9xg/gApensDeCpI= 2023-10-18T20:45:25.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:66EL2Q/2//ZW9xg/gApensDeCpI=', 'Content-Length': '0'} 2023-10-18T20:45:25.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a2d7c0ed7b7b1f-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.514 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:25.515 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:25.515 DEBUG:boto:Method: GET 2023-10-18T20:45:25.516 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:45:25.516 DEBUG:boto:Data: 2023-10-18T20:45:25.517 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.517 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.518 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.518 DEBUG:boto:Params: {} 2023-10-18T20:45:25.519 DEBUG:boto:Token: None 2023-10-18T20:45:25.519 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj55 2023-10-18T20:45:25.520 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:66EL2Q/2//ZW9xg/gApensDeCpI= 2023-10-18T20:45:25.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:66EL2Q/2//ZW9xg/gApensDeCpI=', 'Content-Length': '0'} 2023-10-18T20:45:25.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c29ac6ed748f336-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.523 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.524 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.524 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.525 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.525 DEBUG:boto:Method: GET 2023-10-18T20:45:25.525 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.526 DEBUG:boto:Data: 2023-10-18T20:45:25.526 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.527 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.527 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.528 DEBUG:boto:Params: {} 2023-10-18T20:45:25.528 DEBUG:boto:Token: None 2023-10-18T20:45:25.529 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.529 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003ac2629a74a89e9-00653043e5-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.533 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.534 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.535 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.535 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.536 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.537 DEBUG:boto:Method: GET 2023-10-18T20:45:25.538 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:45:25.538 DEBUG:boto:Data: 2023-10-18T20:45:25.539 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.541 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.541 DEBUG:boto:Params: {} 2023-10-18T20:45:25.542 DEBUG:boto:Token: None 2023-10-18T20:45:25.542 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088dd4e234ef6d70b-00653043e5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.548 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:45:25.549 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.550 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.550 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.551 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.551 DEBUG:boto:Method: GET 2023-10-18T20:45:25.552 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.552 DEBUG:boto:Data: 2023-10-18T20:45:25.553 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.553 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.553 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.554 DEBUG:boto:Params: {} 2023-10-18T20:45:25.554 DEBUG:boto:Token: None 2023-10-18T20:45:25.555 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VDuKsmXqqwpthSiEWqN0pjoz7As= 2023-10-18T20:45:25.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VDuKsmXqqwpthSiEWqN0pjoz7As=', 'Content-Length': '0'} 2023-10-18T20:45:25.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004faee36386eafd97-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.558 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.559 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.559 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.560 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.560 DEBUG:boto:Method: GET 2023-10-18T20:45:25.561 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.561 DEBUG:boto:Data: 2023-10-18T20:45:25.561 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.562 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.563 DEBUG:boto:Params: {} 2023-10-18T20:45:25.563 DEBUG:boto:Token: None 2023-10-18T20:45:25.564 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VDuKsmXqqwpthSiEWqN0pjoz7As= 2023-10-18T20:45:25.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VDuKsmXqqwpthSiEWqN0pjoz7As=', 'Content-Length': '0'} 2023-10-18T20:45:25.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7bf66d16ef3a650-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.567 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.568 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.568 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.569 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.569 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.569 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.570 DEBUG:boto:Data: 2023-10-18T20:45:25.570 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.571 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.571 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.572 DEBUG:boto:Params: {} 2023-10-18T20:45:25.572 DEBUG:boto:Token: None 2023-10-18T20:45:25.572 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.573 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ocd/Z0E22Sfgwlkpqpz8ZcWmqkM= 2023-10-18T20:45:25.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ocd/Z0E22Sfgwlkpqpz8ZcWmqkM=', 'Content-Length': '0'} 2023-10-18T20:45:25.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000348fe65bd47a63ce-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.576 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.576 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.577 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.577 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.577 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.578 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.578 DEBUG:boto:Data: 2023-10-18T20:45:25.579 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.579 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.580 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.580 DEBUG:boto:Params: {} 2023-10-18T20:45:25.580 DEBUG:boto:Token: None 2023-10-18T20:45:25.581 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.581 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ocd/Z0E22Sfgwlkpqpz8ZcWmqkM= 2023-10-18T20:45:25.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ocd/Z0E22Sfgwlkpqpz8ZcWmqkM=', 'Content-Length': '0'} 2023-10-18T20:45:25.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099b2cf276db3811d-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:45:25.584 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.585 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.585 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.586 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.586 DEBUG:boto:Method: GET 2023-10-18T20:45:25.587 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.587 DEBUG:boto:Data: 2023-10-18T20:45:25.587 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.588 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.588 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.589 DEBUG:boto:Params: {} 2023-10-18T20:45:25.589 DEBUG:boto:Token: None 2023-10-18T20:45:25.590 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.590 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VDuKsmXqqwpthSiEWqN0pjoz7As= 2023-10-18T20:45:25.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VDuKsmXqqwpthSiEWqN0pjoz7As=', 'Content-Length': '0'} 2023-10-18T20:45:25.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a987acf16ebb95a-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.593 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.594 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.594 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.595 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.595 DEBUG:boto:Method: GET 2023-10-18T20:45:25.596 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.596 DEBUG:boto:Data: 2023-10-18T20:45:25.597 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.598 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.599 DEBUG:boto:Params: {} 2023-10-18T20:45:25.601 DEBUG:boto:Token: None 2023-10-18T20:45:25.602 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:25.603 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VDuKsmXqqwpthSiEWqN0pjoz7As= 2023-10-18T20:45:25.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VDuKsmXqqwpthSiEWqN0pjoz7As=', 'Content-Length': '0'} 2023-10-18T20:45:25.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9e141c11fc6245a-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.650 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.651 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.652 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.652 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.652 DEBUG:boto:Method: GET 2023-10-18T20:45:25.653 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.654 DEBUG:boto:Data: 2023-10-18T20:45:25.654 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.655 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.655 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.655 DEBUG:boto:Params: {} 2023-10-18T20:45:25.656 DEBUG:boto:Token: None 2023-10-18T20:45:25.656 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.657 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000816042df280dc1b5-00653043e5-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.661 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.661 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.662 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.662 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.663 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.663 DEBUG:boto:Method: GET 2023-10-18T20:45:25.663 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.664 DEBUG:boto:Data: 2023-10-18T20:45:25.664 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.665 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.665 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.666 DEBUG:boto:Params: {} 2023-10-18T20:45:25.666 DEBUG:boto:Token: None 2023-10-18T20:45:25.666 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.667 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dba4356484dcf2e-00653043e5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.672 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.673 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.674 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.674 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.675 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:45:25.676 DEBUG:boto:Data: 2023-10-18T20:45:25.676 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.677 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.678 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.679 DEBUG:boto:Params: {} 2023-10-18T20:45:25.679 DEBUG:boto:Token: None 2023-10-18T20:45:25.680 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57 2023-10-18T20:45:25.681 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dWtWr/n8tuQMYVwXVQl3VGlgMwo= 2023-10-18T20:45:25.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dWtWr/n8tuQMYVwXVQl3VGlgMwo=', 'Content-Length': '0'} 2023-10-18T20:45:25.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097673b7e174b1343-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.686 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.686 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.687 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.688 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:45:25.688 DEBUG:boto:Data: 2023-10-18T20:45:25.689 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.691 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.691 DEBUG:boto:Params: {} 2023-10-18T20:45:25.692 DEBUG:boto:Token: None 2023-10-18T20:45:25.693 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57 2023-10-18T20:45:25.694 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dWtWr/n8tuQMYVwXVQl3VGlgMwo= 2023-10-18T20:45:25.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dWtWr/n8tuQMYVwXVQl3VGlgMwo=', 'Content-Length': '0'} 2023-10-18T20:45:25.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900d0eb5d24fc1e9-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:45:25.697 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.698 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.698 DEBUG:boto:Method: GET 2023-10-18T20:45:25.699 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:45:25.699 DEBUG:boto:Data: 2023-10-18T20:45:25.700 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.700 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.700 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.701 DEBUG:boto:Params: {} 2023-10-18T20:45:25.701 DEBUG:boto:Token: None 2023-10-18T20:45:25.702 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57 2023-10-18T20:45:25.702 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8e8TZSH+NSSQmVoaXV2Ptchglsg= 2023-10-18T20:45:25.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8e8TZSH+NSSQmVoaXV2Ptchglsg=', 'Content-Length': '0'} 2023-10-18T20:45:25.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72f41dae25f8784-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.706 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:25.706 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:25.706 DEBUG:boto:Method: GET 2023-10-18T20:45:25.707 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:45:25.708 DEBUG:boto:Data: 2023-10-18T20:45:25.708 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.709 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.710 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.710 DEBUG:boto:Params: {} 2023-10-18T20:45:25.711 DEBUG:boto:Token: None 2023-10-18T20:45:25.712 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj57 2023-10-18T20:45:25.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8e8TZSH+NSSQmVoaXV2Ptchglsg= 2023-10-18T20:45:25.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8e8TZSH+NSSQmVoaXV2Ptchglsg=', 'Content-Length': '0'} 2023-10-18T20:45:25.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c50b0d8d9d27802-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.716 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.717 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.718 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.719 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.719 DEBUG:boto:Method: GET 2023-10-18T20:45:25.720 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.721 DEBUG:boto:Data: 2023-10-18T20:45:25.721 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.722 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.722 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.723 DEBUG:boto:Params: {} 2023-10-18T20:45:25.724 DEBUG:boto:Token: None 2023-10-18T20:45:25.724 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2da9bd7f0f30706-00653043e5-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.728 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.729 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.729 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.730 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.730 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.731 DEBUG:boto:Method: GET 2023-10-18T20:45:25.731 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:45:25.732 DEBUG:boto:Data: 2023-10-18T20:45:25.732 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.732 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.733 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.733 DEBUG:boto:Params: {} 2023-10-18T20:45:25.734 DEBUG:boto:Token: None 2023-10-18T20:45:25.734 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f050cb0b4f455c76-00653043e5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.782 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:45:25.784 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.784 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.784 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.785 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.785 DEBUG:boto:Method: GET 2023-10-18T20:45:25.786 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.786 DEBUG:boto:Data: 2023-10-18T20:45:25.787 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.787 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.788 DEBUG:boto:Params: {} 2023-10-18T20:45:25.788 DEBUG:boto:Token: None 2023-10-18T20:45:25.789 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JQkKOf52QxdRp6RDANMifIm37Jk= 2023-10-18T20:45:25.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JQkKOf52QxdRp6RDANMifIm37Jk=', 'Content-Length': '0'} 2023-10-18T20:45:25.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017bf1be42884578f-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.793 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.793 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.794 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.794 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.795 DEBUG:boto:Method: GET 2023-10-18T20:45:25.795 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.796 DEBUG:boto:Data: 2023-10-18T20:45:25.796 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.796 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.797 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.797 DEBUG:boto:Params: {} 2023-10-18T20:45:25.798 DEBUG:boto:Token: None 2023-10-18T20:45:25.798 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JQkKOf52QxdRp6RDANMifIm37Jk= 2023-10-18T20:45:25.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JQkKOf52QxdRp6RDANMifIm37Jk=', 'Content-Length': '0'} 2023-10-18T20:45:25.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e6a7bea485b407-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.802 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.803 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.803 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.803 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.804 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.805 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.805 DEBUG:boto:Data: 2023-10-18T20:45:25.806 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.810 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.811 DEBUG:boto:Params: {} 2023-10-18T20:45:25.812 DEBUG:boto:Token: None 2023-10-18T20:45:25.813 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.814 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JkMh3bazI64vsDuH8ErDoc5EtO0= 2023-10-18T20:45:25.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JkMh3bazI64vsDuH8ErDoc5EtO0=', 'Content-Length': '0'} 2023-10-18T20:45:25.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041d4c5b8e8afdaf3-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.819 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.819 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.820 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.821 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.822 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.822 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.823 DEBUG:boto:Data: 2023-10-18T20:45:25.823 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.824 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.825 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.825 DEBUG:boto:Params: {} 2023-10-18T20:45:25.826 DEBUG:boto:Token: None 2023-10-18T20:45:25.827 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.828 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JkMh3bazI64vsDuH8ErDoc5EtO0= 2023-10-18T20:45:25.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JkMh3bazI64vsDuH8ErDoc5EtO0=', 'Content-Length': '0'} 2023-10-18T20:45:25.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000302536c0f5c74b78-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:45:25.832 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.833 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.834 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.834 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.835 DEBUG:boto:Method: GET 2023-10-18T20:45:25.836 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.836 DEBUG:boto:Data: 2023-10-18T20:45:25.837 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.838 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.838 DEBUG:boto:Params: {} 2023-10-18T20:45:25.839 DEBUG:boto:Token: None 2023-10-18T20:45:25.839 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JQkKOf52QxdRp6RDANMifIm37Jk= 2023-10-18T20:45:25.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JQkKOf52QxdRp6RDANMifIm37Jk=', 'Content-Length': '0'} 2023-10-18T20:45:25.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f82086d65e6a8e27-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.843 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.844 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.844 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.845 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.845 DEBUG:boto:Method: GET 2023-10-18T20:45:25.845 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.846 DEBUG:boto:Data: 2023-10-18T20:45:25.846 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.847 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.848 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.848 DEBUG:boto:Params: {} 2023-10-18T20:45:25.849 DEBUG:boto:Token: None 2023-10-18T20:45:25.850 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:25.850 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JQkKOf52QxdRp6RDANMifIm37Jk= 2023-10-18T20:45:25.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JQkKOf52QxdRp6RDANMifIm37Jk=', 'Content-Length': '0'} 2023-10-18T20:45:25.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2fc7fa6525292f2-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.855 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.856 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.857 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:25.857 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:25.858 DEBUG:boto:Method: GET 2023-10-18T20:45:25.859 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:45:25.859 DEBUG:boto:Data: 2023-10-18T20:45:25.859 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.860 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.860 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.861 DEBUG:boto:Params: {} 2023-10-18T20:45:25.861 DEBUG:boto:Token: None 2023-10-18T20:45:25.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000560f4466306b9800-00653043e5-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.910 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.912 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:25.913 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:25.913 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:25.914 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:25.915 DEBUG:boto:Method: GET 2023-10-18T20:45:25.916 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:45:25.916 DEBUG:boto:Data: 2023-10-18T20:45:25.917 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.918 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.918 DEBUG:boto:Params: {} 2023-10-18T20:45:25.919 DEBUG:boto:Token: None 2023-10-18T20:45:25.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/ 2023-10-18T20:45:25.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E= 2023-10-18T20:45:25.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GxemDeUwWeY7SpAJ8fB0WGJMe/E=', 'Content-Length': '0'} 2023-10-18T20:45:25.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000140e002bf622e6f0-00653043e5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.925 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:25.926 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.926 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.927 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.927 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:45:25.927 DEBUG:boto:Data: 2023-10-18T20:45:25.928 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.928 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.929 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.929 DEBUG:boto:Params: {} 2023-10-18T20:45:25.930 DEBUG:boto:Token: None 2023-10-18T20:45:25.930 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58 2023-10-18T20:45:25.931 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ifiNNN2XUR3lfJIxBIK/+qyl628= 2023-10-18T20:45:25.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ifiNNN2XUR3lfJIxBIK/+qyl628=', 'Content-Length': '0'} 2023-10-18T20:45:25.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4d2a2b5ce299566-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.934 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.934 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.935 DEBUG:boto:Method: HEAD 2023-10-18T20:45:25.935 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:45:25.936 DEBUG:boto:Data: 2023-10-18T20:45:25.936 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.937 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.937 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.937 DEBUG:boto:Params: {} 2023-10-18T20:45:25.938 DEBUG:boto:Token: None 2023-10-18T20:45:25.938 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58 2023-10-18T20:45:25.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ifiNNN2XUR3lfJIxBIK/+qyl628= 2023-10-18T20:45:25.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ifiNNN2XUR3lfJIxBIK/+qyl628=', 'Content-Length': '0'} 2023-10-18T20:45:25.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068fb739884d82782-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:45:25.943 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.943 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.943 DEBUG:boto:Method: GET 2023-10-18T20:45:25.944 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:45:25.944 DEBUG:boto:Data: 2023-10-18T20:45:25.945 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:25.946 DEBUG:boto:Port: 8001 2023-10-18T20:45:25.946 DEBUG:boto:Params: {} 2023-10-18T20:45:25.947 DEBUG:boto:Token: None 2023-10-18T20:45:25.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58 2023-10-18T20:45:25.948 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0A42HWMEM6fR0cZvkWdVDgb1Gl0= 2023-10-18T20:45:25.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0A42HWMEM6fR0cZvkWdVDgb1Gl0=', 'Content-Length': '0'} 2023-10-18T20:45:25.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029e71efd61ab5c0e-00653043e5-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:25.951 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:25.951 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:25.952 DEBUG:boto:Method: GET 2023-10-18T20:45:25.952 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:45:25.952 DEBUG:boto:Data: 2023-10-18T20:45:25.953 DEBUG:boto:Headers: {} 2023-10-18T20:45:25.953 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:25.954 DEBUG:boto:Port: 8000 2023-10-18T20:45:25.954 DEBUG:boto:Params: {} 2023-10-18T20:45:25.955 DEBUG:boto:Token: None 2023-10-18T20:45:25.955 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:25 GMT /bofrnf-59/obj58 2023-10-18T20:45:25.956 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0A42HWMEM6fR0cZvkWdVDgb1Gl0= 2023-10-18T20:45:25.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0A42HWMEM6fR0cZvkWdVDgb1Gl0=', 'Content-Length': '0'} 2023-10-18T20:45:25.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f475711442d3fa50-00653043e5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.002 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.004 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.004 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:26.005 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:26.006 DEBUG:boto:Method: GET 2023-10-18T20:45:26.010 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:45:26.011 DEBUG:boto:Data: 2023-10-18T20:45:26.011 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.012 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.012 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.013 DEBUG:boto:Params: {} 2023-10-18T20:45:26.013 DEBUG:boto:Token: None 2023-10-18T20:45:26.014 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.014 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034e2e04262579444-00653043e6-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.018 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.019 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.020 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.020 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:26.020 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:26.021 DEBUG:boto:Method: GET 2023-10-18T20:45:26.021 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:45:26.022 DEBUG:boto:Data: 2023-10-18T20:45:26.022 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.023 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.023 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.024 DEBUG:boto:Params: {} 2023-10-18T20:45:26.024 DEBUG:boto:Token: None 2023-10-18T20:45:26.025 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.025 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa01cd13307a14b7-00653043e6-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.030 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-18T20:45:26.031 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.032 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.032 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.033 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.033 DEBUG:boto:Method: GET 2023-10-18T20:45:26.034 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.034 DEBUG:boto:Data: 2023-10-18T20:45:26.035 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.035 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.036 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.036 DEBUG:boto:Params: {} 2023-10-18T20:45:26.036 DEBUG:boto:Token: None 2023-10-18T20:45:26.037 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.037 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t9ag6wvVJT8L/NGlw85w4waJ2TM= 2023-10-18T20:45:26.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t9ag6wvVJT8L/NGlw85w4waJ2TM=', 'Content-Length': '0'} 2023-10-18T20:45:26.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e19982a4c43daa97-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.041 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.041 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.042 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.042 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.043 DEBUG:boto:Method: GET 2023-10-18T20:45:26.043 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.043 DEBUG:boto:Data: 2023-10-18T20:45:26.044 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.044 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.045 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.045 DEBUG:boto:Params: {} 2023-10-18T20:45:26.046 DEBUG:boto:Token: None 2023-10-18T20:45:26.046 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.047 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t9ag6wvVJT8L/NGlw85w4waJ2TM= 2023-10-18T20:45:26.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t9ag6wvVJT8L/NGlw85w4waJ2TM=', 'Content-Length': '0'} 2023-10-18T20:45:26.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b8d94820a05451-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.050 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.051 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.051 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.052 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.052 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.052 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.053 DEBUG:boto:Data: 2023-10-18T20:45:26.053 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.054 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.054 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.055 DEBUG:boto:Params: {} 2023-10-18T20:45:26.055 DEBUG:boto:Token: None 2023-10-18T20:45:26.056 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.056 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B4vfsyvCFVjd4rKP7ozvXQ3917o= 2023-10-18T20:45:26.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B4vfsyvCFVjd4rKP7ozvXQ3917o=', 'Content-Length': '0'} 2023-10-18T20:45:26.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000308128869d133f6d-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.059 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.060 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.060 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.061 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.061 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.062 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.062 DEBUG:boto:Data: 2023-10-18T20:45:26.062 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.063 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.063 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.064 DEBUG:boto:Params: {} 2023-10-18T20:45:26.064 DEBUG:boto:Token: None 2023-10-18T20:45:26.065 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.065 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B4vfsyvCFVjd4rKP7ozvXQ3917o= 2023-10-18T20:45:26.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B4vfsyvCFVjd4rKP7ozvXQ3917o=', 'Content-Length': '0'} 2023-10-18T20:45:26.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc844274cc65647-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-18T20:45:26.069 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.069 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.069 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.070 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.070 DEBUG:boto:Method: GET 2023-10-18T20:45:26.071 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.071 DEBUG:boto:Data: 2023-10-18T20:45:26.072 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.072 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.073 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.073 DEBUG:boto:Params: {} 2023-10-18T20:45:26.074 DEBUG:boto:Token: None 2023-10-18T20:45:26.074 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.074 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t9ag6wvVJT8L/NGlw85w4waJ2TM= 2023-10-18T20:45:26.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t9ag6wvVJT8L/NGlw85w4waJ2TM=', 'Content-Length': '0'} 2023-10-18T20:45:26.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a6cc3b46b97a515-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.078 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.078 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.079 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.079 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.080 DEBUG:boto:Method: GET 2023-10-18T20:45:26.080 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.081 DEBUG:boto:Data: 2023-10-18T20:45:26.081 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.082 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.083 DEBUG:boto:Params: {} 2023-10-18T20:45:26.084 DEBUG:boto:Token: None 2023-10-18T20:45:26.084 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:26.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t9ag6wvVJT8L/NGlw85w4waJ2TM= 2023-10-18T20:45:26.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t9ag6wvVJT8L/NGlw85w4waJ2TM=', 'Content-Length': '0'} 2023-10-18T20:45:26.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069ff38233d81d815-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.088 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.088 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.089 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.089 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.090 DEBUG:boto:Method: GET 2023-10-18T20:45:26.090 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.090 DEBUG:boto:Data: 2023-10-18T20:45:26.091 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.091 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.092 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.092 DEBUG:boto:Params: {} 2023-10-18T20:45:26.093 DEBUG:boto:Token: None 2023-10-18T20:45:26.093 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.094 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000803334cc42d3f602-00653043e6-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.097 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.098 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.098 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.099 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.099 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.100 DEBUG:boto:Method: GET 2023-10-18T20:45:26.100 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.100 DEBUG:boto:Data: 2023-10-18T20:45:26.101 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.101 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.102 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.102 DEBUG:boto:Params: {} 2023-10-18T20:45:26.103 DEBUG:boto:Token: None 2023-10-18T20:45:26.103 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.103 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6e67b31afadafc9-00653043e6-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.109 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.109 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.110 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.110 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.111 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:45:26.111 DEBUG:boto:Data: 2023-10-18T20:45:26.111 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.112 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.112 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.113 DEBUG:boto:Params: {} 2023-10-18T20:45:26.113 DEBUG:boto:Token: None 2023-10-18T20:45:26.114 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62 2023-10-18T20:45:26.114 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZS2uFs8DSVKeY5kUf5c54NBtsKs= 2023-10-18T20:45:26.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZS2uFs8DSVKeY5kUf5c54NBtsKs=', 'Content-Length': '0'} 2023-10-18T20:45:26.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7b584b4dd64797b-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.117 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.118 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.118 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.118 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:45:26.119 DEBUG:boto:Data: 2023-10-18T20:45:26.119 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.120 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.120 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.120 DEBUG:boto:Params: {} 2023-10-18T20:45:26.121 DEBUG:boto:Token: None 2023-10-18T20:45:26.121 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62 2023-10-18T20:45:26.122 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZS2uFs8DSVKeY5kUf5c54NBtsKs= 2023-10-18T20:45:26.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZS2uFs8DSVKeY5kUf5c54NBtsKs=', 'Content-Length': '0'} 2023-10-18T20:45:26.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f1669c1f49b46cd-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-18T20:45:26.125 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.126 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.126 DEBUG:boto:Method: GET 2023-10-18T20:45:26.127 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:45:26.127 DEBUG:boto:Data: 2023-10-18T20:45:26.128 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.128 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.129 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.129 DEBUG:boto:Params: {} 2023-10-18T20:45:26.130 DEBUG:boto:Token: None 2023-10-18T20:45:26.130 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62 2023-10-18T20:45:26.131 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eaDMZcITDhJatM2AtZaPeEsFUMo= 2023-10-18T20:45:26.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eaDMZcITDhJatM2AtZaPeEsFUMo=', 'Content-Length': '0'} 2023-10-18T20:45:26.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2abc01ae912422-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.134 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:26.134 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:26.135 DEBUG:boto:Method: GET 2023-10-18T20:45:26.135 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:45:26.136 DEBUG:boto:Data: 2023-10-18T20:45:26.136 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.137 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.137 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.138 DEBUG:boto:Params: {} 2023-10-18T20:45:26.138 DEBUG:boto:Token: None 2023-10-18T20:45:26.139 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj62 2023-10-18T20:45:26.139 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eaDMZcITDhJatM2AtZaPeEsFUMo= 2023-10-18T20:45:26.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eaDMZcITDhJatM2AtZaPeEsFUMo=', 'Content-Length': '0'} 2023-10-18T20:45:26.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd142800f84ee26c-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.182 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.183 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.183 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.184 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.184 DEBUG:boto:Method: GET 2023-10-18T20:45:26.184 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.185 DEBUG:boto:Data: 2023-10-18T20:45:26.185 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.186 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.186 DEBUG:boto:Params: {} 2023-10-18T20:45:26.187 DEBUG:boto:Token: None 2023-10-18T20:45:26.188 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.188 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000729fbb78df1f28a-00653043e6-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.192 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.193 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.193 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.194 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.194 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.194 DEBUG:boto:Method: GET 2023-10-18T20:45:26.195 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:45:26.195 DEBUG:boto:Data: 2023-10-18T20:45:26.196 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.196 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.197 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.197 DEBUG:boto:Params: {} 2023-10-18T20:45:26.198 DEBUG:boto:Token: None 2023-10-18T20:45:26.198 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.199 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adeb5527b29c9380-00653043e6-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.206 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:45:26.207 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.208 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.208 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.209 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.209 DEBUG:boto:Method: GET 2023-10-18T20:45:26.210 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.210 DEBUG:boto:Data: 2023-10-18T20:45:26.210 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.211 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.211 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.212 DEBUG:boto:Params: {} 2023-10-18T20:45:26.212 DEBUG:boto:Token: None 2023-10-18T20:45:26.213 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.213 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JnUit/5w904GkSeFE9PpHRf8vlw= 2023-10-18T20:45:26.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JnUit/5w904GkSeFE9PpHRf8vlw=', 'Content-Length': '0'} 2023-10-18T20:45:26.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000934c08cefc7edb77-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.262 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.263 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.263 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.264 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.264 DEBUG:boto:Method: GET 2023-10-18T20:45:26.265 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.265 DEBUG:boto:Data: 2023-10-18T20:45:26.266 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.266 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.267 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.267 DEBUG:boto:Params: {} 2023-10-18T20:45:26.268 DEBUG:boto:Token: None 2023-10-18T20:45:26.268 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.269 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JnUit/5w904GkSeFE9PpHRf8vlw= 2023-10-18T20:45:26.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JnUit/5w904GkSeFE9PpHRf8vlw=', 'Content-Length': '0'} 2023-10-18T20:45:26.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000590a5f998bec90cc-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.272 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.272 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.273 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.273 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.274 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.274 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.274 DEBUG:boto:Data: 2023-10-18T20:45:26.275 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.276 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.277 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.278 DEBUG:boto:Params: {} 2023-10-18T20:45:26.278 DEBUG:boto:Token: None 2023-10-18T20:45:26.279 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.280 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+fnyJKj31UqwkWtecpGbIUfF0xE= 2023-10-18T20:45:26.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+fnyJKj31UqwkWtecpGbIUfF0xE=', 'Content-Length': '0'} 2023-10-18T20:45:26.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032888eeb7c9c89b4-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.284 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.284 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.285 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.285 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.286 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.287 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.287 DEBUG:boto:Data: 2023-10-18T20:45:26.288 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.288 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.289 DEBUG:boto:Params: {} 2023-10-18T20:45:26.289 DEBUG:boto:Token: None 2023-10-18T20:45:26.290 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.290 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+fnyJKj31UqwkWtecpGbIUfF0xE= 2023-10-18T20:45:26.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+fnyJKj31UqwkWtecpGbIUfF0xE=', 'Content-Length': '0'} 2023-10-18T20:45:26.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e3c15c57c83c2d8-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:45:26.294 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.294 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.295 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.295 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.296 DEBUG:boto:Method: GET 2023-10-18T20:45:26.296 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.297 DEBUG:boto:Data: 2023-10-18T20:45:26.297 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.298 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.298 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.299 DEBUG:boto:Params: {} 2023-10-18T20:45:26.300 DEBUG:boto:Token: None 2023-10-18T20:45:26.300 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.301 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JnUit/5w904GkSeFE9PpHRf8vlw= 2023-10-18T20:45:26.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JnUit/5w904GkSeFE9PpHRf8vlw=', 'Content-Length': '0'} 2023-10-18T20:45:26.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d11133723ebcb4-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.304 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.305 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.305 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.305 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.306 DEBUG:boto:Method: GET 2023-10-18T20:45:26.306 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.307 DEBUG:boto:Data: 2023-10-18T20:45:26.307 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.308 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.308 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.308 DEBUG:boto:Params: {} 2023-10-18T20:45:26.309 DEBUG:boto:Token: None 2023-10-18T20:45:26.311 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:26.312 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JnUit/5w904GkSeFE9PpHRf8vlw= 2023-10-18T20:45:26.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JnUit/5w904GkSeFE9PpHRf8vlw=', 'Content-Length': '0'} 2023-10-18T20:45:26.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084b1bc58cec40ebb-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.315 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.315 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.316 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.316 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.317 DEBUG:boto:Method: GET 2023-10-18T20:45:26.317 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.318 DEBUG:boto:Data: 2023-10-18T20:45:26.318 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.319 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.319 DEBUG:boto:Params: {} 2023-10-18T20:45:26.320 DEBUG:boto:Token: None 2023-10-18T20:45:26.321 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c60d756423beeb-00653043e6-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.366 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.367 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.368 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.368 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.369 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.369 DEBUG:boto:Method: GET 2023-10-18T20:45:26.369 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.370 DEBUG:boto:Data: 2023-10-18T20:45:26.370 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.371 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.371 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.372 DEBUG:boto:Params: {} 2023-10-18T20:45:26.372 DEBUG:boto:Token: None 2023-10-18T20:45:26.373 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.373 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3bdfdf8ce0e35f-00653043e6-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.379 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.380 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.380 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.381 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.381 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:45:26.381 DEBUG:boto:Data: 2023-10-18T20:45:26.382 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.383 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.383 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.383 DEBUG:boto:Params: {} 2023-10-18T20:45:26.384 DEBUG:boto:Token: None 2023-10-18T20:45:26.384 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68 2023-10-18T20:45:26.385 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZjygRDSTT0J5LcOYKe5iV0lQCXU= 2023-10-18T20:45:26.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZjygRDSTT0J5LcOYKe5iV0lQCXU=', 'Content-Length': '0'} 2023-10-18T20:45:26.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004536c85a8a5d8740-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.389 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.390 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.391 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.391 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:45:26.392 DEBUG:boto:Data: 2023-10-18T20:45:26.392 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.392 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.393 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.393 DEBUG:boto:Params: {} 2023-10-18T20:45:26.394 DEBUG:boto:Token: None 2023-10-18T20:45:26.394 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68 2023-10-18T20:45:26.395 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZjygRDSTT0J5LcOYKe5iV0lQCXU= 2023-10-18T20:45:26.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZjygRDSTT0J5LcOYKe5iV0lQCXU=', 'Content-Length': '0'} 2023-10-18T20:45:26.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1036d592c17673-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:45:26.398 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.399 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.399 DEBUG:boto:Method: GET 2023-10-18T20:45:26.400 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:45:26.400 DEBUG:boto:Data: 2023-10-18T20:45:26.401 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.401 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.402 DEBUG:boto:Params: {} 2023-10-18T20:45:26.402 DEBUG:boto:Token: None 2023-10-18T20:45:26.403 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68 2023-10-18T20:45:26.403 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9pwrObtobZZUlkfVNytiGv8ctls= 2023-10-18T20:45:26.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9pwrObtobZZUlkfVNytiGv8ctls=', 'Content-Length': '0'} 2023-10-18T20:45:26.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000979a2fabb8f932c4-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.406 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:26.407 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:26.407 DEBUG:boto:Method: GET 2023-10-18T20:45:26.408 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:45:26.408 DEBUG:boto:Data: 2023-10-18T20:45:26.409 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.409 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.410 DEBUG:boto:Params: {} 2023-10-18T20:45:26.410 DEBUG:boto:Token: None 2023-10-18T20:45:26.411 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj68 2023-10-18T20:45:26.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9pwrObtobZZUlkfVNytiGv8ctls= 2023-10-18T20:45:26.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9pwrObtobZZUlkfVNytiGv8ctls=', 'Content-Length': '0'} 2023-10-18T20:45:26.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c98e8d5c380089-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.454 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.455 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.456 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.456 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.456 DEBUG:boto:Method: GET 2023-10-18T20:45:26.457 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.457 DEBUG:boto:Data: 2023-10-18T20:45:26.458 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.458 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.459 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.459 DEBUG:boto:Params: {} 2023-10-18T20:45:26.459 DEBUG:boto:Token: None 2023-10-18T20:45:26.460 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b77a4fa8a0210998-00653043e6-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.464 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.465 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.465 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.466 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.466 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.467 DEBUG:boto:Method: GET 2023-10-18T20:45:26.467 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:45:26.468 DEBUG:boto:Data: 2023-10-18T20:45:26.468 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.468 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.469 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.469 DEBUG:boto:Params: {} 2023-10-18T20:45:26.470 DEBUG:boto:Token: None 2023-10-18T20:45:26.470 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.471 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063d27ea591a8faf9-00653043e6-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.476 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:45:26.477 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.477 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.478 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.478 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.479 DEBUG:boto:Method: GET 2023-10-18T20:45:26.479 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.479 DEBUG:boto:Data: 2023-10-18T20:45:26.480 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.480 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.481 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.481 DEBUG:boto:Params: {} 2023-10-18T20:45:26.481 DEBUG:boto:Token: None 2023-10-18T20:45:26.482 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.482 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CrFsydk2VlY5pXLf1U1Yc8NhwJk= 2023-10-18T20:45:26.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CrFsydk2VlY5pXLf1U1Yc8NhwJk=', 'Content-Length': '0'} 2023-10-18T20:45:26.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea8ff8c2f9ed741a-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.486 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.486 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.486 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.487 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.487 DEBUG:boto:Method: GET 2023-10-18T20:45:26.488 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.488 DEBUG:boto:Data: 2023-10-18T20:45:26.489 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.489 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.489 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.490 DEBUG:boto:Params: {} 2023-10-18T20:45:26.490 DEBUG:boto:Token: None 2023-10-18T20:45:26.491 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.491 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CrFsydk2VlY5pXLf1U1Yc8NhwJk= 2023-10-18T20:45:26.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CrFsydk2VlY5pXLf1U1Yc8NhwJk=', 'Content-Length': '0'} 2023-10-18T20:45:26.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079da703e0095f127-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.495 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.495 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.495 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.496 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.496 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.497 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.497 DEBUG:boto:Data: 2023-10-18T20:45:26.498 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.498 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.499 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.499 DEBUG:boto:Params: {} 2023-10-18T20:45:26.499 DEBUG:boto:Token: None 2023-10-18T20:45:26.500 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.500 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EeXI9xctt5fjp8J1fVUXYqOajcY= 2023-10-18T20:45:26.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EeXI9xctt5fjp8J1fVUXYqOajcY=', 'Content-Length': '0'} 2023-10-18T20:45:26.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069cedade6fccd497-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.503 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.503 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.504 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.504 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.504 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.505 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.505 DEBUG:boto:Data: 2023-10-18T20:45:26.506 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.506 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.506 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.507 DEBUG:boto:Params: {} 2023-10-18T20:45:26.507 DEBUG:boto:Token: None 2023-10-18T20:45:26.508 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.508 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EeXI9xctt5fjp8J1fVUXYqOajcY= 2023-10-18T20:45:26.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EeXI9xctt5fjp8J1fVUXYqOajcY=', 'Content-Length': '0'} 2023-10-18T20:45:26.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208f341bb10fc724-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:45:26.511 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.512 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.512 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.513 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.513 DEBUG:boto:Method: GET 2023-10-18T20:45:26.513 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.514 DEBUG:boto:Data: 2023-10-18T20:45:26.514 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.515 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.515 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.516 DEBUG:boto:Params: {} 2023-10-18T20:45:26.516 DEBUG:boto:Token: None 2023-10-18T20:45:26.517 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.517 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CrFsydk2VlY5pXLf1U1Yc8NhwJk= 2023-10-18T20:45:26.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CrFsydk2VlY5pXLf1U1Yc8NhwJk=', 'Content-Length': '0'} 2023-10-18T20:45:26.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de5c34d875b584a7-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.562 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.563 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.564 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.564 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.564 DEBUG:boto:Method: GET 2023-10-18T20:45:26.565 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.565 DEBUG:boto:Data: 2023-10-18T20:45:26.566 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.567 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.567 DEBUG:boto:Params: {} 2023-10-18T20:45:26.568 DEBUG:boto:Token: None 2023-10-18T20:45:26.568 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:26.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CrFsydk2VlY5pXLf1U1Yc8NhwJk= 2023-10-18T20:45:26.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CrFsydk2VlY5pXLf1U1Yc8NhwJk=', 'Content-Length': '0'} 2023-10-18T20:45:26.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fed840ba53c6cff-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.572 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.573 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.573 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.574 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.574 DEBUG:boto:Method: GET 2023-10-18T20:45:26.574 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.575 DEBUG:boto:Data: 2023-10-18T20:45:26.575 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.576 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.576 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.576 DEBUG:boto:Params: {} 2023-10-18T20:45:26.577 DEBUG:boto:Token: None 2023-10-18T20:45:26.577 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.578 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004178431177165fa9-00653043e6-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.581 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.582 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.583 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.583 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.583 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.584 DEBUG:boto:Method: GET 2023-10-18T20:45:26.584 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.585 DEBUG:boto:Data: 2023-10-18T20:45:26.585 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.586 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.586 DEBUG:boto:Params: {} 2023-10-18T20:45:26.587 DEBUG:boto:Token: None 2023-10-18T20:45:26.587 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.588 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed21d70fdcd91fff-00653043e6-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.634 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.635 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.636 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.636 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.637 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:45:26.637 DEBUG:boto:Data: 2023-10-18T20:45:26.637 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.638 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.638 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.639 DEBUG:boto:Params: {} 2023-10-18T20:45:26.639 DEBUG:boto:Token: None 2023-10-18T20:45:26.640 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69 2023-10-18T20:45:26.640 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VSBszRxMDZxUFP2i8WCSd/+D9MM= 2023-10-18T20:45:26.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VSBszRxMDZxUFP2i8WCSd/+D9MM=', 'Content-Length': '0'} 2023-10-18T20:45:26.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007370a0c880d6d21c-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.643 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.644 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.644 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.645 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:45:26.645 DEBUG:boto:Data: 2023-10-18T20:45:26.645 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.646 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.646 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.647 DEBUG:boto:Params: {} 2023-10-18T20:45:26.647 DEBUG:boto:Token: None 2023-10-18T20:45:26.648 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69 2023-10-18T20:45:26.648 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VSBszRxMDZxUFP2i8WCSd/+D9MM= 2023-10-18T20:45:26.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VSBszRxMDZxUFP2i8WCSd/+D9MM=', 'Content-Length': '0'} 2023-10-18T20:45:26.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000131ad9be432fc1d0-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:45:26.652 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.652 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.652 DEBUG:boto:Method: GET 2023-10-18T20:45:26.653 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:45:26.653 DEBUG:boto:Data: 2023-10-18T20:45:26.654 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.655 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.655 DEBUG:boto:Params: {} 2023-10-18T20:45:26.656 DEBUG:boto:Token: None 2023-10-18T20:45:26.656 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69 2023-10-18T20:45:26.657 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HXRP9wgeJEnENY9sWXkR1s03vgw= 2023-10-18T20:45:26.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HXRP9wgeJEnENY9sWXkR1s03vgw=', 'Content-Length': '0'} 2023-10-18T20:45:26.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010fd8111a25396a2-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.660 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:26.660 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:26.660 DEBUG:boto:Method: GET 2023-10-18T20:45:26.661 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:45:26.661 DEBUG:boto:Data: 2023-10-18T20:45:26.662 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.663 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.663 DEBUG:boto:Params: {} 2023-10-18T20:45:26.663 DEBUG:boto:Token: None 2023-10-18T20:45:26.664 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj69 2023-10-18T20:45:26.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HXRP9wgeJEnENY9sWXkR1s03vgw= 2023-10-18T20:45:26.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HXRP9wgeJEnENY9sWXkR1s03vgw=', 'Content-Length': '0'} 2023-10-18T20:45:26.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5d37c4ec7c12388-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.667 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.668 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.668 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.669 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.669 DEBUG:boto:Method: GET 2023-10-18T20:45:26.670 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.670 DEBUG:boto:Data: 2023-10-18T20:45:26.671 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.671 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.672 DEBUG:boto:Params: {} 2023-10-18T20:45:26.672 DEBUG:boto:Token: None 2023-10-18T20:45:26.673 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9fe041d1f8bc988-00653043e6-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.718 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.719 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.720 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.720 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.720 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.721 DEBUG:boto:Method: GET 2023-10-18T20:45:26.721 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:45:26.722 DEBUG:boto:Data: 2023-10-18T20:45:26.722 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.723 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.723 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.724 DEBUG:boto:Params: {} 2023-10-18T20:45:26.724 DEBUG:boto:Token: None 2023-10-18T20:45:26.725 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cabc6929045b316-00653043e6-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.731 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-18T20:45:26.732 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.733 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.733 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.734 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.734 DEBUG:boto:Method: GET 2023-10-18T20:45:26.735 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.735 DEBUG:boto:Data: 2023-10-18T20:45:26.736 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.736 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.736 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.737 DEBUG:boto:Params: {} 2023-10-18T20:45:26.737 DEBUG:boto:Token: None 2023-10-18T20:45:26.738 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.738 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QFFJjOAfCmo5F5RQG2sfsGnz2Y8= 2023-10-18T20:45:26.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QFFJjOAfCmo5F5RQG2sfsGnz2Y8=', 'Content-Length': '0'} 2023-10-18T20:45:26.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caffb9b909f8bb30-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.742 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.742 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.743 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.743 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.744 DEBUG:boto:Method: GET 2023-10-18T20:45:26.744 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.744 DEBUG:boto:Data: 2023-10-18T20:45:26.745 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.745 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.746 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.746 DEBUG:boto:Params: {} 2023-10-18T20:45:26.747 DEBUG:boto:Token: None 2023-10-18T20:45:26.747 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QFFJjOAfCmo5F5RQG2sfsGnz2Y8= 2023-10-18T20:45:26.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QFFJjOAfCmo5F5RQG2sfsGnz2Y8=', 'Content-Length': '0'} 2023-10-18T20:45:26.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c529057dfbae6b25-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.794 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.795 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.796 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.796 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.796 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.797 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.797 DEBUG:boto:Data: 2023-10-18T20:45:26.798 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.798 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.798 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.799 DEBUG:boto:Params: {} 2023-10-18T20:45:26.799 DEBUG:boto:Token: None 2023-10-18T20:45:26.800 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lZH4VhWp+SApdg0pRUXQFChMgdY= 2023-10-18T20:45:26.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lZH4VhWp+SApdg0pRUXQFChMgdY=', 'Content-Length': '0'} 2023-10-18T20:45:26.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad689904a68c47dc-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.804 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.804 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.805 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.805 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.806 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.806 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.807 DEBUG:boto:Data: 2023-10-18T20:45:26.807 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.808 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.809 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.809 DEBUG:boto:Params: {} 2023-10-18T20:45:26.810 DEBUG:boto:Token: None 2023-10-18T20:45:26.811 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lZH4VhWp+SApdg0pRUXQFChMgdY= 2023-10-18T20:45:26.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lZH4VhWp+SApdg0pRUXQFChMgdY=', 'Content-Length': '0'} 2023-10-18T20:45:26.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1b25986c1610b8c-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-18T20:45:26.817 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.817 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.818 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.818 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.819 DEBUG:boto:Method: GET 2023-10-18T20:45:26.819 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.820 DEBUG:boto:Data: 2023-10-18T20:45:26.820 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.821 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.821 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.821 DEBUG:boto:Params: {} 2023-10-18T20:45:26.822 DEBUG:boto:Token: None 2023-10-18T20:45:26.823 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.823 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QFFJjOAfCmo5F5RQG2sfsGnz2Y8= 2023-10-18T20:45:26.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QFFJjOAfCmo5F5RQG2sfsGnz2Y8=', 'Content-Length': '0'} 2023-10-18T20:45:26.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000732c609e8046c7b2-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.826 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.827 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.827 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.829 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.830 DEBUG:boto:Method: GET 2023-10-18T20:45:26.830 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.830 DEBUG:boto:Data: 2023-10-18T20:45:26.831 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.831 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.832 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.832 DEBUG:boto:Params: {} 2023-10-18T20:45:26.833 DEBUG:boto:Token: None 2023-10-18T20:45:26.833 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:26.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QFFJjOAfCmo5F5RQG2sfsGnz2Y8= 2023-10-18T20:45:26.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QFFJjOAfCmo5F5RQG2sfsGnz2Y8=', 'Content-Length': '0'} 2023-10-18T20:45:26.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f8781a2f482441b-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.837 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.838 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.838 DEBUG:boto:path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.839 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.839 DEBUG:boto:Method: GET 2023-10-18T20:45:26.839 DEBUG:boto:Path: /bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.840 DEBUG:boto:Data: 2023-10-18T20:45:26.841 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.841 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.841 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.842 DEBUG:boto:Params: {} 2023-10-18T20:45:26.843 DEBUG:boto:Token: None 2023-10-18T20:45:26.843 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.844 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b34c0857bc42368a-00653043e6-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.848 DEBUG:boto:b'bofrnf-59obj741000falseobj742023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.849 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.849 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.849 DEBUG:boto:path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.850 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.850 DEBUG:boto:Method: GET 2023-10-18T20:45:26.851 DEBUG:boto:Path: /bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.851 DEBUG:boto:Data: 2023-10-18T20:45:26.852 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.852 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.852 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.853 DEBUG:boto:Params: {} 2023-10-18T20:45:26.853 DEBUG:boto:Token: None 2023-10-18T20:45:26.854 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.854 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005edc63453282aa7d-00653043e6-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.860 DEBUG:boto:b'bofrnf-59obj741000falseobj742023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.860 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.861 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.861 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.862 DEBUG:boto:Path: /bofrnf-59/obj74 2023-10-18T20:45:26.862 DEBUG:boto:Data: 2023-10-18T20:45:26.862 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.863 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.863 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.864 DEBUG:boto:Params: {} 2023-10-18T20:45:26.864 DEBUG:boto:Token: None 2023-10-18T20:45:26.865 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74 2023-10-18T20:45:26.865 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T3/FZwZwtpBZWMbfAGId+pUJxqU= 2023-10-18T20:45:26.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T3/FZwZwtpBZWMbfAGId+pUJxqU=', 'Content-Length': '0'} 2023-10-18T20:45:26.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010038599ee62be97-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.868 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.868 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.869 DEBUG:boto:Method: HEAD 2023-10-18T20:45:26.869 DEBUG:boto:Path: /bofrnf-59/obj74 2023-10-18T20:45:26.870 DEBUG:boto:Data: 2023-10-18T20:45:26.870 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.870 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.871 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.871 DEBUG:boto:Params: {} 2023-10-18T20:45:26.872 DEBUG:boto:Token: None 2023-10-18T20:45:26.872 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74 2023-10-18T20:45:26.873 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T3/FZwZwtpBZWMbfAGId+pUJxqU= 2023-10-18T20:45:26.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T3/FZwZwtpBZWMbfAGId+pUJxqU=', 'Content-Length': '0'} 2023-10-18T20:45:26.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017e5e48a0c67080d-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-18T20:45:26.876 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.877 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.877 DEBUG:boto:Method: GET 2023-10-18T20:45:26.877 DEBUG:boto:Path: /bofrnf-59/obj74 2023-10-18T20:45:26.878 DEBUG:boto:Data: 2023-10-18T20:45:26.878 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.879 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.879 DEBUG:boto:Params: {} 2023-10-18T20:45:26.880 DEBUG:boto:Token: None 2023-10-18T20:45:26.880 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74 2023-10-18T20:45:26.881 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QF4/hnglpfu75L+nbevndnS+4Ms= 2023-10-18T20:45:26.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QF4/hnglpfu75L+nbevndnS+4Ms=', 'Content-Length': '0'} 2023-10-18T20:45:26.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006869a8abbfdc376a-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.926 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:26.927 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:26.928 DEBUG:boto:Method: GET 2023-10-18T20:45:26.928 DEBUG:boto:Path: /bofrnf-59/obj74 2023-10-18T20:45:26.928 DEBUG:boto:Data: 2023-10-18T20:45:26.929 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.930 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.930 DEBUG:boto:Params: {} 2023-10-18T20:45:26.931 DEBUG:boto:Token: None 2023-10-18T20:45:26.931 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj74 2023-10-18T20:45:26.931 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QF4/hnglpfu75L+nbevndnS+4Ms= 2023-10-18T20:45:26.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QF4/hnglpfu75L+nbevndnS+4Ms=', 'Content-Length': '0'} 2023-10-18T20:45:26.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cfbea53a9bcd2ca-00653043e6-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.935 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.935 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.936 DEBUG:boto:path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.936 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.936 DEBUG:boto:Method: GET 2023-10-18T20:45:26.937 DEBUG:boto:Path: /bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.937 DEBUG:boto:Data: 2023-10-18T20:45:26.938 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.938 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.939 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.939 DEBUG:boto:Params: {} 2023-10-18T20:45:26.940 DEBUG:boto:Token: None 2023-10-18T20:45:26.940 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.940 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f55b6e7835f96624-00653043e6-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.944 DEBUG:boto:b'bofrnf-59obj741000falseobj742023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.945 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:26.946 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:26.947 DEBUG:boto:path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.948 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.949 DEBUG:boto:Method: GET 2023-10-18T20:45:26.950 DEBUG:boto:Path: /bofrnf-59/?prefix=obj74 2023-10-18T20:45:26.951 DEBUG:boto:Data: 2023-10-18T20:45:26.952 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.953 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:26.954 DEBUG:boto:Port: 8000 2023-10-18T20:45:26.955 DEBUG:boto:Params: {} 2023-10-18T20:45:26.956 DEBUG:boto:Token: None 2023-10-18T20:45:26.957 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/ 2023-10-18T20:45:26.958 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o= 2023-10-18T20:45:26.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2/4bnyjn0yZXHgTGOYoSHzPUM3o=', 'Content-Length': '0'} 2023-10-18T20:45:26.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b6e6ab4ce97767-00653043e6-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.965 DEBUG:boto:b'bofrnf-59obj741000falseobj742023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:26.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:45:26.967 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:26.969 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:26.970 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:26.972 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:26.973 DEBUG:boto:Method: GET 2023-10-18T20:45:26.974 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:26.975 DEBUG:boto:Data: 2023-10-18T20:45:26.976 DEBUG:boto:Headers: {} 2023-10-18T20:45:26.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:26.978 DEBUG:boto:Port: 8001 2023-10-18T20:45:26.979 DEBUG:boto:Params: {} 2023-10-18T20:45:26.981 DEBUG:boto:Token: None 2023-10-18T20:45:26.982 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:26 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:26.983 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hfP2jB8miz3M1GXZIf9X1C3HKdM= 2023-10-18T20:45:26.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hfP2jB8miz3M1GXZIf9X1C3HKdM=', 'Content-Length': '0'} 2023-10-18T20:45:26.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d531542b5e58023e-00653043e6-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:26.989 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:26.990 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:26.992 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:26.994 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:26.996 DEBUG:boto:Method: GET 2023-10-18T20:45:26.998 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:26.999 DEBUG:boto:Data: 2023-10-18T20:45:26.999 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.000 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.001 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.002 DEBUG:boto:Params: {} 2023-10-18T20:45:27.003 DEBUG:boto:Token: None 2023-10-18T20:45:27.004 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AraWOWcD5cQzNKHbn6vKXRfeSLQ= 2023-10-18T20:45:27.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AraWOWcD5cQzNKHbn6vKXRfeSLQ=', 'Content-Length': '0'} 2023-10-18T20:45:27.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c7988f60c39309e-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.009 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:27.010 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:27.011 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.012 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.013 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.014 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.015 DEBUG:boto:Data: 2023-10-18T20:45:27.016 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.017 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.017 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.018 DEBUG:boto:Params: {} 2023-10-18T20:45:27.019 DEBUG:boto:Token: None 2023-10-18T20:45:27.020 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.021 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8Nxa8femWiIDCF9nw7wVHCeqiFg= 2023-10-18T20:45:27.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8Nxa8femWiIDCF9nw7wVHCeqiFg=', 'Content-Length': '0'} 2023-10-18T20:45:27.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e357063c3f5c3cab-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.025 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:27.025 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:27.026 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.027 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.028 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.029 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.029 DEBUG:boto:Data: 2023-10-18T20:45:27.030 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.031 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.032 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.033 DEBUG:boto:Params: {} 2023-10-18T20:45:27.034 DEBUG:boto:Token: None 2023-10-18T20:45:27.034 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.035 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8Nxa8femWiIDCF9nw7wVHCeqiFg= 2023-10-18T20:45:27.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8Nxa8femWiIDCF9nw7wVHCeqiFg=', 'Content-Length': '0'} 2023-10-18T20:45:27.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d892b343adb173-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:45:27.042 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:27.043 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:27.044 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.045 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.046 DEBUG:boto:Method: GET 2023-10-18T20:45:27.047 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.047 DEBUG:boto:Data: 2023-10-18T20:45:27.048 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.049 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.049 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.050 DEBUG:boto:Params: {} 2023-10-18T20:45:27.051 DEBUG:boto:Token: None 2023-10-18T20:45:27.052 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AraWOWcD5cQzNKHbn6vKXRfeSLQ= 2023-10-18T20:45:27.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AraWOWcD5cQzNKHbn6vKXRfeSLQ=', 'Content-Length': '0'} 2023-10-18T20:45:27.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000661789db8e5adce9-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.058 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:27.059 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:27.060 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.061 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.062 DEBUG:boto:Method: GET 2023-10-18T20:45:27.063 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.064 DEBUG:boto:Data: 2023-10-18T20:45:27.065 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.068 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.069 DEBUG:boto:Params: {} 2023-10-18T20:45:27.070 DEBUG:boto:Token: None 2023-10-18T20:45:27.071 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:27.071 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AraWOWcD5cQzNKHbn6vKXRfeSLQ= 2023-10-18T20:45:27.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AraWOWcD5cQzNKHbn6vKXRfeSLQ=', 'Content-Length': '0'} 2023-10-18T20:45:27.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e423ddb04cd4294-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.076 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.077 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.078 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.079 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.080 DEBUG:boto:Method: GET 2023-10-18T20:45:27.080 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.081 DEBUG:boto:Data: 2023-10-18T20:45:27.082 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.083 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.084 DEBUG:boto:Params: {} 2023-10-18T20:45:27.085 DEBUG:boto:Token: None 2023-10-18T20:45:27.085 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.086 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9dbe71220d2c337-00653043e7-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.090 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.091 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.092 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.093 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.093 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.094 DEBUG:boto:Method: GET 2023-10-18T20:45:27.095 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.095 DEBUG:boto:Data: 2023-10-18T20:45:27.096 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.097 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.097 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.098 DEBUG:boto:Params: {} 2023-10-18T20:45:27.098 DEBUG:boto:Token: None 2023-10-18T20:45:27.099 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.100 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0c02a6c6d4591f2-00653043e7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.106 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.107 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:27.107 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:27.107 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.108 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:45:27.108 DEBUG:boto:Data: 2023-10-18T20:45:27.109 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.110 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.110 DEBUG:boto:Params: {} 2023-10-18T20:45:27.110 DEBUG:boto:Token: None 2023-10-18T20:45:27.111 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj76 2023-10-18T20:45:27.111 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nyfw7fUC4nTnv/WC257p1skiafc= 2023-10-18T20:45:27.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nyfw7fUC4nTnv/WC257p1skiafc=', 'Content-Length': '0'} 2023-10-18T20:45:27.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f68d530596599f1f-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.115 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:27.115 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:27.115 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.116 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:45:27.116 DEBUG:boto:Data: 2023-10-18T20:45:27.117 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.117 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.117 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.118 DEBUG:boto:Params: {} 2023-10-18T20:45:27.118 DEBUG:boto:Token: None 2023-10-18T20:45:27.119 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj76 2023-10-18T20:45:27.119 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nyfw7fUC4nTnv/WC257p1skiafc= 2023-10-18T20:45:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nyfw7fUC4nTnv/WC257p1skiafc=', 'Content-Length': '0'} 2023-10-18T20:45:27.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d66fa0469b4bd452-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:45:27.123 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:27.123 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:27.124 DEBUG:boto:Method: GET 2023-10-18T20:45:27.124 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:45:27.125 DEBUG:boto:Data: 2023-10-18T20:45:27.125 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.126 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.126 DEBUG:boto:Params: {} 2023-10-18T20:45:27.127 DEBUG:boto:Token: None 2023-10-18T20:45:27.127 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj76 2023-10-18T20:45:27.128 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3tGGlwx2rttsg9Wo7IKS1mC+0MY= 2023-10-18T20:45:27.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3tGGlwx2rttsg9Wo7IKS1mC+0MY=', 'Content-Length': '0'} 2023-10-18T20:45:27.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f655ccfd1e0197-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.131 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:27.132 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:27.132 DEBUG:boto:Method: GET 2023-10-18T20:45:27.132 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:45:27.133 DEBUG:boto:Data: 2023-10-18T20:45:27.133 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.134 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.134 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.135 DEBUG:boto:Params: {} 2023-10-18T20:45:27.135 DEBUG:boto:Token: None 2023-10-18T20:45:27.136 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj76 2023-10-18T20:45:27.136 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3tGGlwx2rttsg9Wo7IKS1mC+0MY= 2023-10-18T20:45:27.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3tGGlwx2rttsg9Wo7IKS1mC+0MY=', 'Content-Length': '0'} 2023-10-18T20:45:27.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005376fa3a8271a09e-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.182 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.183 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.184 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.184 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.184 DEBUG:boto:Method: GET 2023-10-18T20:45:27.185 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.185 DEBUG:boto:Data: 2023-10-18T20:45:27.186 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.186 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.187 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.187 DEBUG:boto:Params: {} 2023-10-18T20:45:27.188 DEBUG:boto:Token: None 2023-10-18T20:45:27.188 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.189 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ef9c45f03b12af2-00653043e7-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.193 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.193 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.194 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.194 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.195 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.195 DEBUG:boto:Method: GET 2023-10-18T20:45:27.196 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:45:27.196 DEBUG:boto:Data: 2023-10-18T20:45:27.196 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.197 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.198 DEBUG:boto:Params: {} 2023-10-18T20:45:27.198 DEBUG:boto:Token: None 2023-10-18T20:45:27.199 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.199 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000147eba43c5f2fa54-00653043e7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.204 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:45:27.205 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.206 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.206 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.207 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.207 DEBUG:boto:Method: GET 2023-10-18T20:45:27.207 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.208 DEBUG:boto:Data: 2023-10-18T20:45:27.208 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.209 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.209 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.210 DEBUG:boto:Params: {} 2023-10-18T20:45:27.210 DEBUG:boto:Token: None 2023-10-18T20:45:27.211 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OgEMlVaTV1VnESuVBvrh833WjMQ= 2023-10-18T20:45:27.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OgEMlVaTV1VnESuVBvrh833WjMQ=', 'Content-Length': '0'} 2023-10-18T20:45:27.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c464f6166f569f-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.214 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.215 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.215 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.215 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.216 DEBUG:boto:Method: GET 2023-10-18T20:45:27.216 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.217 DEBUG:boto:Data: 2023-10-18T20:45:27.217 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.218 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.218 DEBUG:boto:Params: {} 2023-10-18T20:45:27.219 DEBUG:boto:Token: None 2023-10-18T20:45:27.219 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OgEMlVaTV1VnESuVBvrh833WjMQ= 2023-10-18T20:45:27.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OgEMlVaTV1VnESuVBvrh833WjMQ=', 'Content-Length': '0'} 2023-10-18T20:45:27.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6d4b56fd257dd2a-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.223 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.224 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.224 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.224 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.225 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.225 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.226 DEBUG:boto:Data: 2023-10-18T20:45:27.226 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.227 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.227 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.227 DEBUG:boto:Params: {} 2023-10-18T20:45:27.228 DEBUG:boto:Token: None 2023-10-18T20:45:27.228 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.229 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tXdrYae5nRUZDxso1nsRqMLxi60= 2023-10-18T20:45:27.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tXdrYae5nRUZDxso1nsRqMLxi60=', 'Content-Length': '0'} 2023-10-18T20:45:27.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a55d8767a955d13a-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.232 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.233 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.233 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.233 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.234 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.234 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.235 DEBUG:boto:Data: 2023-10-18T20:45:27.237 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.238 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.238 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.239 DEBUG:boto:Params: {} 2023-10-18T20:45:27.239 DEBUG:boto:Token: None 2023-10-18T20:45:27.240 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tXdrYae5nRUZDxso1nsRqMLxi60= 2023-10-18T20:45:27.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tXdrYae5nRUZDxso1nsRqMLxi60=', 'Content-Length': '0'} 2023-10-18T20:45:27.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015ce61a6e9a1074a-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:45:27.244 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.245 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.245 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.245 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.246 DEBUG:boto:Method: GET 2023-10-18T20:45:27.246 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.247 DEBUG:boto:Data: 2023-10-18T20:45:27.247 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.248 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.249 DEBUG:boto:Params: {} 2023-10-18T20:45:27.249 DEBUG:boto:Token: None 2023-10-18T20:45:27.250 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OgEMlVaTV1VnESuVBvrh833WjMQ= 2023-10-18T20:45:27.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OgEMlVaTV1VnESuVBvrh833WjMQ=', 'Content-Length': '0'} 2023-10-18T20:45:27.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f27fb65ff341a2a0-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.298 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.299 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.300 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.300 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.300 DEBUG:boto:Method: GET 2023-10-18T20:45:27.301 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.301 DEBUG:boto:Data: 2023-10-18T20:45:27.302 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.303 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.303 DEBUG:boto:Params: {} 2023-10-18T20:45:27.304 DEBUG:boto:Token: None 2023-10-18T20:45:27.304 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:27.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OgEMlVaTV1VnESuVBvrh833WjMQ= 2023-10-18T20:45:27.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OgEMlVaTV1VnESuVBvrh833WjMQ=', 'Content-Length': '0'} 2023-10-18T20:45:27.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c566393a5ee45f8-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.308 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.308 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.309 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.309 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.310 DEBUG:boto:Method: GET 2023-10-18T20:45:27.310 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.310 DEBUG:boto:Data: 2023-10-18T20:45:27.311 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.311 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.312 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.312 DEBUG:boto:Params: {} 2023-10-18T20:45:27.313 DEBUG:boto:Token: None 2023-10-18T20:45:27.313 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.314 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bd92782cbfeea16-00653043e7-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.317 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.318 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.319 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.319 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.320 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.320 DEBUG:boto:Method: GET 2023-10-18T20:45:27.320 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.321 DEBUG:boto:Data: 2023-10-18T20:45:27.321 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.322 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.323 DEBUG:boto:Params: {} 2023-10-18T20:45:27.323 DEBUG:boto:Token: None 2023-10-18T20:45:27.324 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b1d1109b6206ef2-00653043e7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.330 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.330 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.331 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.331 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.332 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:45:27.332 DEBUG:boto:Data: 2023-10-18T20:45:27.333 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.333 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.333 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.334 DEBUG:boto:Params: {} 2023-10-18T20:45:27.334 DEBUG:boto:Token: None 2023-10-18T20:45:27.335 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85 2023-10-18T20:45:27.335 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uYx97HWHkft8OU9Ovu2yKvbuE6s= 2023-10-18T20:45:27.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uYx97HWHkft8OU9Ovu2yKvbuE6s=', 'Content-Length': '0'} 2023-10-18T20:45:27.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a2240ac751657c-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.338 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.339 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.339 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.340 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:45:27.340 DEBUG:boto:Data: 2023-10-18T20:45:27.341 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.341 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.341 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.342 DEBUG:boto:Params: {} 2023-10-18T20:45:27.342 DEBUG:boto:Token: None 2023-10-18T20:45:27.343 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85 2023-10-18T20:45:27.343 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uYx97HWHkft8OU9Ovu2yKvbuE6s= 2023-10-18T20:45:27.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uYx97HWHkft8OU9Ovu2yKvbuE6s=', 'Content-Length': '0'} 2023-10-18T20:45:27.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab18628b6f7c9b83-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:45:27.347 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.347 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.347 DEBUG:boto:Method: GET 2023-10-18T20:45:27.348 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:45:27.348 DEBUG:boto:Data: 2023-10-18T20:45:27.349 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.350 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.350 DEBUG:boto:Params: {} 2023-10-18T20:45:27.351 DEBUG:boto:Token: None 2023-10-18T20:45:27.351 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85 2023-10-18T20:45:27.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:peaFDQH2uwIB7tLW+LJ3PKcgRck= 2023-10-18T20:45:27.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:peaFDQH2uwIB7tLW+LJ3PKcgRck=', 'Content-Length': '0'} 2023-10-18T20:45:27.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a60e9525769581d-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.355 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:27.355 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:27.356 DEBUG:boto:Method: GET 2023-10-18T20:45:27.356 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:45:27.357 DEBUG:boto:Data: 2023-10-18T20:45:27.357 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.357 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.358 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.358 DEBUG:boto:Params: {} 2023-10-18T20:45:27.359 DEBUG:boto:Token: None 2023-10-18T20:45:27.359 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj85 2023-10-18T20:45:27.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:peaFDQH2uwIB7tLW+LJ3PKcgRck= 2023-10-18T20:45:27.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:peaFDQH2uwIB7tLW+LJ3PKcgRck=', 'Content-Length': '0'} 2023-10-18T20:45:27.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005017740e272c0ecb-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.363 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.363 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.364 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.364 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.365 DEBUG:boto:Method: GET 2023-10-18T20:45:27.365 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.366 DEBUG:boto:Data: 2023-10-18T20:45:27.366 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.367 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.367 DEBUG:boto:Params: {} 2023-10-18T20:45:27.368 DEBUG:boto:Token: None 2023-10-18T20:45:27.368 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008240eeaf60eadcbb-00653043e7-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.373 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.373 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.374 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.374 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.375 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.375 DEBUG:boto:Method: GET 2023-10-18T20:45:27.375 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:45:27.376 DEBUG:boto:Data: 2023-10-18T20:45:27.376 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.377 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.378 DEBUG:boto:Params: {} 2023-10-18T20:45:27.378 DEBUG:boto:Token: None 2023-10-18T20:45:27.379 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.379 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bb45f37cede92d9-00653043e7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.426 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:45:27.428 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.428 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.429 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.429 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.429 DEBUG:boto:Method: GET 2023-10-18T20:45:27.430 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.430 DEBUG:boto:Data: 2023-10-18T20:45:27.431 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.431 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.432 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.432 DEBUG:boto:Params: {} 2023-10-18T20:45:27.432 DEBUG:boto:Token: None 2023-10-18T20:45:27.433 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.433 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MYuFp33SGTzuf0FRuoJHTlnwYx4= 2023-10-18T20:45:27.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MYuFp33SGTzuf0FRuoJHTlnwYx4=', 'Content-Length': '0'} 2023-10-18T20:45:27.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115f18022ec64a0b-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.437 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.438 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.438 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.439 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.441 DEBUG:boto:Method: GET 2023-10-18T20:45:27.442 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.442 DEBUG:boto:Data: 2023-10-18T20:45:27.442 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.443 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.443 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.444 DEBUG:boto:Params: {} 2023-10-18T20:45:27.444 DEBUG:boto:Token: None 2023-10-18T20:45:27.445 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.445 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MYuFp33SGTzuf0FRuoJHTlnwYx4= 2023-10-18T20:45:27.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MYuFp33SGTzuf0FRuoJHTlnwYx4=', 'Content-Length': '0'} 2023-10-18T20:45:27.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f36fd0c696455d8-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.449 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.449 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.449 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.450 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.450 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.451 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.451 DEBUG:boto:Data: 2023-10-18T20:45:27.451 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.452 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.453 DEBUG:boto:Params: {} 2023-10-18T20:45:27.453 DEBUG:boto:Token: None 2023-10-18T20:45:27.454 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.454 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o3WjGbp7Ts0k/5Aw5Wnr147fgSE= 2023-10-18T20:45:27.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o3WjGbp7Ts0k/5Aw5Wnr147fgSE=', 'Content-Length': '0'} 2023-10-18T20:45:27.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064ed23fa05f70408-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.457 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.458 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.458 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.458 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.459 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.459 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.460 DEBUG:boto:Data: 2023-10-18T20:45:27.460 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.460 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.461 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.461 DEBUG:boto:Params: {} 2023-10-18T20:45:27.462 DEBUG:boto:Token: None 2023-10-18T20:45:27.462 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.463 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o3WjGbp7Ts0k/5Aw5Wnr147fgSE= 2023-10-18T20:45:27.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o3WjGbp7Ts0k/5Aw5Wnr147fgSE=', 'Content-Length': '0'} 2023-10-18T20:45:27.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000156619ac6ea708f6-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:45:27.466 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.467 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.467 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.468 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.468 DEBUG:boto:Method: GET 2023-10-18T20:45:27.469 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.469 DEBUG:boto:Data: 2023-10-18T20:45:27.469 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.470 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.470 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.471 DEBUG:boto:Params: {} 2023-10-18T20:45:27.471 DEBUG:boto:Token: None 2023-10-18T20:45:27.472 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.472 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MYuFp33SGTzuf0FRuoJHTlnwYx4= 2023-10-18T20:45:27.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MYuFp33SGTzuf0FRuoJHTlnwYx4=', 'Content-Length': '0'} 2023-10-18T20:45:27.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6bd5ce8194b51e0-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.476 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.476 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.476 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.477 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.477 DEBUG:boto:Method: GET 2023-10-18T20:45:27.478 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.478 DEBUG:boto:Data: 2023-10-18T20:45:27.479 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.479 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.480 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.480 DEBUG:boto:Params: {} 2023-10-18T20:45:27.480 DEBUG:boto:Token: None 2023-10-18T20:45:27.481 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:27.481 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MYuFp33SGTzuf0FRuoJHTlnwYx4= 2023-10-18T20:45:27.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MYuFp33SGTzuf0FRuoJHTlnwYx4=', 'Content-Length': '0'} 2023-10-18T20:45:27.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e86e8c85c5a87c-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.485 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.485 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.486 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.486 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.487 DEBUG:boto:Method: GET 2023-10-18T20:45:27.487 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.487 DEBUG:boto:Data: 2023-10-18T20:45:27.488 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.488 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.489 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.489 DEBUG:boto:Params: {} 2023-10-18T20:45:27.490 DEBUG:boto:Token: None 2023-10-18T20:45:27.490 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.491 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000160e8e8b5326ef19-00653043e7-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.534 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.535 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.536 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.536 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.537 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.537 DEBUG:boto:Method: GET 2023-10-18T20:45:27.537 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.538 DEBUG:boto:Data: 2023-10-18T20:45:27.538 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.539 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.539 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.540 DEBUG:boto:Params: {} 2023-10-18T20:45:27.540 DEBUG:boto:Token: None 2023-10-18T20:45:27.541 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.541 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fe84f80f9d756d9-00653043e7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.546 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.547 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.548 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.548 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.548 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:45:27.549 DEBUG:boto:Data: 2023-10-18T20:45:27.549 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.550 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.551 DEBUG:boto:Params: {} 2023-10-18T20:45:27.551 DEBUG:boto:Token: None 2023-10-18T20:45:27.552 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87 2023-10-18T20:45:27.552 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RURicmew3pe3lVRdlQUqync2MD4= 2023-10-18T20:45:27.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RURicmew3pe3lVRdlQUqync2MD4=', 'Content-Length': '0'} 2023-10-18T20:45:27.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ffa0b9dd239ea2e-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.555 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.556 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.556 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.557 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:45:27.557 DEBUG:boto:Data: 2023-10-18T20:45:27.557 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.558 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.559 DEBUG:boto:Params: {} 2023-10-18T20:45:27.559 DEBUG:boto:Token: None 2023-10-18T20:45:27.560 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87 2023-10-18T20:45:27.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RURicmew3pe3lVRdlQUqync2MD4= 2023-10-18T20:45:27.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RURicmew3pe3lVRdlQUqync2MD4=', 'Content-Length': '0'} 2023-10-18T20:45:27.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067cbfecc645bff53-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:45:27.564 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.564 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.565 DEBUG:boto:Method: GET 2023-10-18T20:45:27.565 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:45:27.566 DEBUG:boto:Data: 2023-10-18T20:45:27.566 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.567 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.567 DEBUG:boto:Params: {} 2023-10-18T20:45:27.568 DEBUG:boto:Token: None 2023-10-18T20:45:27.568 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87 2023-10-18T20:45:27.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U1kaOoyF0UN/hpQ3+2Qkmxey774= 2023-10-18T20:45:27.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U1kaOoyF0UN/hpQ3+2Qkmxey774=', 'Content-Length': '0'} 2023-10-18T20:45:27.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000603e60513d52f560-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.572 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:27.572 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:27.573 DEBUG:boto:Method: GET 2023-10-18T20:45:27.573 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:45:27.574 DEBUG:boto:Data: 2023-10-18T20:45:27.574 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.575 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.575 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.575 DEBUG:boto:Params: {} 2023-10-18T20:45:27.576 DEBUG:boto:Token: None 2023-10-18T20:45:27.576 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj87 2023-10-18T20:45:27.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U1kaOoyF0UN/hpQ3+2Qkmxey774= 2023-10-18T20:45:27.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U1kaOoyF0UN/hpQ3+2Qkmxey774=', 'Content-Length': '0'} 2023-10-18T20:45:27.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682160c42e2bf7b5-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.580 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.580 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.581 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.581 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.581 DEBUG:boto:Method: GET 2023-10-18T20:45:27.582 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.582 DEBUG:boto:Data: 2023-10-18T20:45:27.583 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.584 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.584 DEBUG:boto:Params: {} 2023-10-18T20:45:27.584 DEBUG:boto:Token: None 2023-10-18T20:45:27.585 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000696c3b945bc7eace-00653043e7-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.630 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.632 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.632 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.633 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.633 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.634 DEBUG:boto:Method: GET 2023-10-18T20:45:27.634 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:45:27.634 DEBUG:boto:Data: 2023-10-18T20:45:27.635 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.635 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.636 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.636 DEBUG:boto:Params: {} 2023-10-18T20:45:27.637 DEBUG:boto:Token: None 2023-10-18T20:45:27.637 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.638 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaf5818c8c4514a9-00653043e7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.643 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:45:27.644 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.644 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.645 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.645 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.646 DEBUG:boto:Method: GET 2023-10-18T20:45:27.646 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.650 DEBUG:boto:Data: 2023-10-18T20:45:27.650 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.651 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.651 DEBUG:boto:Params: {} 2023-10-18T20:45:27.652 DEBUG:boto:Token: None 2023-10-18T20:45:27.652 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rHv25YxpUT/3NbVIByMMuW5AHWk= 2023-10-18T20:45:27.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rHv25YxpUT/3NbVIByMMuW5AHWk=', 'Content-Length': '0'} 2023-10-18T20:45:27.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a081baacc889ea-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.657 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.657 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.657 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.658 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.658 DEBUG:boto:Method: GET 2023-10-18T20:45:27.659 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.659 DEBUG:boto:Data: 2023-10-18T20:45:27.660 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.660 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.660 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.661 DEBUG:boto:Params: {} 2023-10-18T20:45:27.661 DEBUG:boto:Token: None 2023-10-18T20:45:27.662 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rHv25YxpUT/3NbVIByMMuW5AHWk= 2023-10-18T20:45:27.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rHv25YxpUT/3NbVIByMMuW5AHWk=', 'Content-Length': '0'} 2023-10-18T20:45:27.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2fea7bacd7495f8-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.706 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.707 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.707 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.708 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.708 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.709 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.709 DEBUG:boto:Data: 2023-10-18T20:45:27.709 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.710 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.710 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.711 DEBUG:boto:Params: {} 2023-10-18T20:45:27.711 DEBUG:boto:Token: None 2023-10-18T20:45:27.712 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4zgF4RwL4zUeG7R5QR98G55m59E= 2023-10-18T20:45:27.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4zgF4RwL4zUeG7R5QR98G55m59E=', 'Content-Length': '0'} 2023-10-18T20:45:27.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097008f026f5a04a7-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.716 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.716 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.717 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.717 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.717 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.718 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.718 DEBUG:boto:Data: 2023-10-18T20:45:27.719 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.719 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.720 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.720 DEBUG:boto:Params: {} 2023-10-18T20:45:27.720 DEBUG:boto:Token: None 2023-10-18T20:45:27.721 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.721 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4zgF4RwL4zUeG7R5QR98G55m59E= 2023-10-18T20:45:27.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4zgF4RwL4zUeG7R5QR98G55m59E=', 'Content-Length': '0'} 2023-10-18T20:45:27.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3b4fd7030284f1-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:45:27.725 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.725 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.726 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.726 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.727 DEBUG:boto:Method: GET 2023-10-18T20:45:27.728 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.728 DEBUG:boto:Data: 2023-10-18T20:45:27.729 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.729 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.729 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.730 DEBUG:boto:Params: {} 2023-10-18T20:45:27.730 DEBUG:boto:Token: None 2023-10-18T20:45:27.731 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.731 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rHv25YxpUT/3NbVIByMMuW5AHWk= 2023-10-18T20:45:27.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rHv25YxpUT/3NbVIByMMuW5AHWk=', 'Content-Length': '0'} 2023-10-18T20:45:27.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a770594164290a5f-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.735 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.735 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.736 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.736 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.737 DEBUG:boto:Method: GET 2023-10-18T20:45:27.737 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.737 DEBUG:boto:Data: 2023-10-18T20:45:27.738 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.739 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.739 DEBUG:boto:Params: {} 2023-10-18T20:45:27.740 DEBUG:boto:Token: None 2023-10-18T20:45:27.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:27.741 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rHv25YxpUT/3NbVIByMMuW5AHWk= 2023-10-18T20:45:27.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rHv25YxpUT/3NbVIByMMuW5AHWk=', 'Content-Length': '0'} 2023-10-18T20:45:27.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a5b1121f8191f93-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.744 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.744 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.745 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.745 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.746 DEBUG:boto:Method: GET 2023-10-18T20:45:27.746 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.747 DEBUG:boto:Data: 2023-10-18T20:45:27.747 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.748 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.748 DEBUG:boto:Params: {} 2023-10-18T20:45:27.749 DEBUG:boto:Token: None 2023-10-18T20:45:27.749 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.750 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9eff0b2478a5b22-00653043e7-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.754 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.754 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.755 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.755 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.756 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.756 DEBUG:boto:Method: GET 2023-10-18T20:45:27.757 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.757 DEBUG:boto:Data: 2023-10-18T20:45:27.757 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.758 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.759 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.759 DEBUG:boto:Params: {} 2023-10-18T20:45:27.759 DEBUG:boto:Token: None 2023-10-18T20:45:27.760 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.760 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000158fea6b608f1d57-00653043e7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.766 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.767 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.767 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.768 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.768 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:45:27.768 DEBUG:boto:Data: 2023-10-18T20:45:27.769 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.769 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.770 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.770 DEBUG:boto:Params: {} 2023-10-18T20:45:27.771 DEBUG:boto:Token: None 2023-10-18T20:45:27.771 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92 2023-10-18T20:45:27.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bqCPVM8PrRujliUhBqNbIqyVVwo= 2023-10-18T20:45:27.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bqCPVM8PrRujliUhBqNbIqyVVwo=', 'Content-Length': '0'} 2023-10-18T20:45:27.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cf428934f59a1ac-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.775 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.776 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.776 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.777 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:45:27.777 DEBUG:boto:Data: 2023-10-18T20:45:27.778 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.778 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.778 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.779 DEBUG:boto:Params: {} 2023-10-18T20:45:27.779 DEBUG:boto:Token: None 2023-10-18T20:45:27.780 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92 2023-10-18T20:45:27.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bqCPVM8PrRujliUhBqNbIqyVVwo= 2023-10-18T20:45:27.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bqCPVM8PrRujliUhBqNbIqyVVwo=', 'Content-Length': '0'} 2023-10-18T20:45:27.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56585594297cd91-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:45:27.784 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.784 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.785 DEBUG:boto:Method: GET 2023-10-18T20:45:27.786 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:45:27.786 DEBUG:boto:Data: 2023-10-18T20:45:27.786 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.787 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.788 DEBUG:boto:Params: {} 2023-10-18T20:45:27.788 DEBUG:boto:Token: None 2023-10-18T20:45:27.789 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92 2023-10-18T20:45:27.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q2NYpeeGg4eFH6ZFu0cI91lu7EE= 2023-10-18T20:45:27.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q2NYpeeGg4eFH6ZFu0cI91lu7EE=', 'Content-Length': '0'} 2023-10-18T20:45:27.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a6c6eb82001ce2e-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.834 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:27.835 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:27.836 DEBUG:boto:Method: GET 2023-10-18T20:45:27.836 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:45:27.836 DEBUG:boto:Data: 2023-10-18T20:45:27.837 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.838 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.838 DEBUG:boto:Params: {} 2023-10-18T20:45:27.839 DEBUG:boto:Token: None 2023-10-18T20:45:27.839 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj92 2023-10-18T20:45:27.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q2NYpeeGg4eFH6ZFu0cI91lu7EE= 2023-10-18T20:45:27.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q2NYpeeGg4eFH6ZFu0cI91lu7EE=', 'Content-Length': '0'} 2023-10-18T20:45:27.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f833d0f445b127-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.844 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.844 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.845 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.845 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.846 DEBUG:boto:Method: GET 2023-10-18T20:45:27.846 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.847 DEBUG:boto:Data: 2023-10-18T20:45:27.847 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.848 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.849 DEBUG:boto:Params: {} 2023-10-18T20:45:27.849 DEBUG:boto:Token: None 2023-10-18T20:45:27.850 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.850 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c10eb0c7d2b886ae-00653043e7-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.854 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.855 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.855 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.856 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.856 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.856 DEBUG:boto:Method: GET 2023-10-18T20:45:27.857 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:45:27.857 DEBUG:boto:Data: 2023-10-18T20:45:27.858 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.859 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.859 DEBUG:boto:Params: {} 2023-10-18T20:45:27.860 DEBUG:boto:Token: None 2023-10-18T20:45:27.860 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1b8ca7f4f0c38c8-00653043e7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.865 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-18T20:45:27.867 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:27.867 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:27.868 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.868 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.868 DEBUG:boto:Method: GET 2023-10-18T20:45:27.869 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.869 DEBUG:boto:Data: 2023-10-18T20:45:27.870 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.870 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.871 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.871 DEBUG:boto:Params: {} 2023-10-18T20:45:27.871 DEBUG:boto:Token: None 2023-10-18T20:45:27.872 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qzfP/839cJhijfExJJfVhs7UkpU= 2023-10-18T20:45:27.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qzfP/839cJhijfExJJfVhs7UkpU=', 'Content-Length': '0'} 2023-10-18T20:45:27.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b5444f805a9428e-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.875 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:27.876 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:27.876 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.877 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.877 DEBUG:boto:Method: GET 2023-10-18T20:45:27.878 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.878 DEBUG:boto:Data: 2023-10-18T20:45:27.879 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.879 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.880 DEBUG:boto:Params: {} 2023-10-18T20:45:27.880 DEBUG:boto:Token: None 2023-10-18T20:45:27.881 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.881 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qzfP/839cJhijfExJJfVhs7UkpU= 2023-10-18T20:45:27.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qzfP/839cJhijfExJJfVhs7UkpU=', 'Content-Length': '0'} 2023-10-18T20:45:27.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039c686f0ca711339-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.885 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:27.885 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:27.885 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.886 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.886 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.887 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.887 DEBUG:boto:Data: 2023-10-18T20:45:27.888 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.888 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.888 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.889 DEBUG:boto:Params: {} 2023-10-18T20:45:27.889 DEBUG:boto:Token: None 2023-10-18T20:45:27.890 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.890 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:md6Zi1yg18l/hssIPDVZE67Pmck= 2023-10-18T20:45:27.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:md6Zi1yg18l/hssIPDVZE67Pmck=', 'Content-Length': '0'} 2023-10-18T20:45:27.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e327f3951b356335-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.893 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:27.894 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:27.894 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.894 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.895 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.895 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.896 DEBUG:boto:Data: 2023-10-18T20:45:27.896 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.896 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.897 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.897 DEBUG:boto:Params: {} 2023-10-18T20:45:27.898 DEBUG:boto:Token: None 2023-10-18T20:45:27.898 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.899 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:md6Zi1yg18l/hssIPDVZE67Pmck= 2023-10-18T20:45:27.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:md6Zi1yg18l/hssIPDVZE67Pmck=', 'Content-Length': '0'} 2023-10-18T20:45:27.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084f98fe524c81b22-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-18T20:45:27.902 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:27.903 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:27.903 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.904 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.904 DEBUG:boto:Method: GET 2023-10-18T20:45:27.904 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.905 DEBUG:boto:Data: 2023-10-18T20:45:27.905 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.906 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.906 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.907 DEBUG:boto:Params: {} 2023-10-18T20:45:27.907 DEBUG:boto:Token: None 2023-10-18T20:45:27.908 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.908 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qzfP/839cJhijfExJJfVhs7UkpU= 2023-10-18T20:45:27.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qzfP/839cJhijfExJJfVhs7UkpU=', 'Content-Length': '0'} 2023-10-18T20:45:27.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae06d11bcb02ad5-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.911 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:27.912 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:27.912 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.913 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.913 DEBUG:boto:Method: GET 2023-10-18T20:45:27.914 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.914 DEBUG:boto:Data: 2023-10-18T20:45:27.915 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.916 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.916 DEBUG:boto:Params: {} 2023-10-18T20:45:27.916 DEBUG:boto:Token: None 2023-10-18T20:45:27.917 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:27.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qzfP/839cJhijfExJJfVhs7UkpU= 2023-10-18T20:45:27.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qzfP/839cJhijfExJJfVhs7UkpU=', 'Content-Length': '0'} 2023-10-18T20:45:27.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbe486be41064314-00653043e7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.921 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.921 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.922 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:27.922 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:27.922 DEBUG:boto:Method: GET 2023-10-18T20:45:27.923 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:45:27.924 DEBUG:boto:Data: 2023-10-18T20:45:27.924 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.924 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.925 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.925 DEBUG:boto:Params: {} 2023-10-18T20:45:27.926 DEBUG:boto:Token: None 2023-10-18T20:45:27.926 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.927 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b46f08eef9c7d07-00653043e7-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.974 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.975 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:27.976 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:27.976 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:27.977 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:27.977 DEBUG:boto:Method: GET 2023-10-18T20:45:27.978 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:45:27.978 DEBUG:boto:Data: 2023-10-18T20:45:27.979 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.979 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.979 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.980 DEBUG:boto:Params: {} 2023-10-18T20:45:27.980 DEBUG:boto:Token: None 2023-10-18T20:45:27.981 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/ 2023-10-18T20:45:27.981 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU= 2023-10-18T20:45:27.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bscbQBtl4q776Kl9KsRVMMvGoRU=', 'Content-Length': '0'} 2023-10-18T20:45:27.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030badfdd36938229-00653043e7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.987 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:27.988 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:27.988 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:27.989 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.989 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:45:27.990 DEBUG:boto:Data: 2023-10-18T20:45:27.990 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.990 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:27.991 DEBUG:boto:Port: 8001 2023-10-18T20:45:27.991 DEBUG:boto:Params: {} 2023-10-18T20:45:27.992 DEBUG:boto:Token: None 2023-10-18T20:45:27.992 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:27 GMT /bofrnf-59/obj93 2023-10-18T20:45:27.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U1hssDtH1rgni/yquWqLqkU5qms= 2023-10-18T20:45:27.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U1hssDtH1rgni/yquWqLqkU5qms=', 'Content-Length': '0'} 2023-10-18T20:45:27.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a0110506fa834ae-00653043e7-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:27.996 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:27.996 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:27.997 DEBUG:boto:Method: HEAD 2023-10-18T20:45:27.997 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:45:27.998 DEBUG:boto:Data: 2023-10-18T20:45:27.998 DEBUG:boto:Headers: {} 2023-10-18T20:45:27.998 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:27.999 DEBUG:boto:Port: 8000 2023-10-18T20:45:27.999 DEBUG:boto:Params: {} 2023-10-18T20:45:28.000 DEBUG:boto:Token: None 2023-10-18T20:45:28.000 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj93 2023-10-18T20:45:28.001 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Qo3D22ByVLGw0q8sLyb1J160jak= 2023-10-18T20:45:28.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Qo3D22ByVLGw0q8sLyb1J160jak=', 'Content-Length': '0'} 2023-10-18T20:45:28.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c00bafb97e67639-00653043e8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-18T20:45:28.004 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:28.005 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:28.005 DEBUG:boto:Method: GET 2023-10-18T20:45:28.006 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:45:28.006 DEBUG:boto:Data: 2023-10-18T20:45:28.007 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:28.008 DEBUG:boto:Port: 8001 2023-10-18T20:45:28.008 DEBUG:boto:Params: {} 2023-10-18T20:45:28.008 DEBUG:boto:Token: None 2023-10-18T20:45:28.009 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj93 2023-10-18T20:45:28.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jkPmGG0ME/zAL2ZBRKku9RXIez8= 2023-10-18T20:45:28.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jkPmGG0ME/zAL2ZBRKku9RXIez8=', 'Content-Length': '0'} 2023-10-18T20:45:28.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aba552c537323e6-00653043e8-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.013 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:28.013 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:28.014 DEBUG:boto:Method: GET 2023-10-18T20:45:28.014 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:45:28.015 DEBUG:boto:Data: 2023-10-18T20:45:28.015 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.016 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:28.016 DEBUG:boto:Port: 8000 2023-10-18T20:45:28.016 DEBUG:boto:Params: {} 2023-10-18T20:45:28.017 DEBUG:boto:Token: None 2023-10-18T20:45:28.017 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj93 2023-10-18T20:45:28.018 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jkPmGG0ME/zAL2ZBRKku9RXIez8= 2023-10-18T20:45:28.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jkPmGG0ME/zAL2ZBRKku9RXIez8=', 'Content-Length': '0'} 2023-10-18T20:45:28.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae16072b98127e0-00653043e8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.062 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:28.063 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:28.064 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:28.064 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:28.064 DEBUG:boto:Method: GET 2023-10-18T20:45:28.065 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:45:28.065 DEBUG:boto:Data: 2023-10-18T20:45:28.066 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:28.067 DEBUG:boto:Port: 8001 2023-10-18T20:45:28.067 DEBUG:boto:Params: {} 2023-10-18T20:45:28.068 DEBUG:boto:Token: None 2023-10-18T20:45:28.068 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/ 2023-10-18T20:45:28.069 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw= 2023-10-18T20:45:28.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw=', 'Content-Length': '0'} 2023-10-18T20:45:28.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9b4fe77cc8bc25d-00653043e8-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.073 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:28.074 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:28.074 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:28.075 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:28.075 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:28.075 DEBUG:boto:Method: GET 2023-10-18T20:45:28.076 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:45:28.076 DEBUG:boto:Data: 2023-10-18T20:45:28.077 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.077 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:28.077 DEBUG:boto:Port: 8000 2023-10-18T20:45:28.078 DEBUG:boto:Params: {} 2023-10-18T20:45:28.078 DEBUG:boto:Token: None 2023-10-18T20:45:28.079 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/ 2023-10-18T20:45:28.079 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw= 2023-10-18T20:45:28.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw=', 'Content-Length': '0'} 2023-10-18T20:45:28.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd45e3f379dd54d0-00653043e8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.084 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:28.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:45:28.086 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.086 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.087 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.087 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.087 DEBUG:boto:Method: GET 2023-10-18T20:45:28.088 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.088 DEBUG:boto:Data: 2023-10-18T20:45:28.089 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.089 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:28.090 DEBUG:boto:Port: 8001 2023-10-18T20:45:28.090 DEBUG:boto:Params: {} 2023-10-18T20:45:28.090 DEBUG:boto:Token: None 2023-10-18T20:45:28.091 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.092 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hDjEIh4u1KxJhmzIE1puuxafCv4= 2023-10-18T20:45:28.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hDjEIh4u1KxJhmzIE1puuxafCv4=', 'Content-Length': '0'} 2023-10-18T20:45:28.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fddbb3f84e9e05dc-00653043e8-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.134 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.135 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.136 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.136 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.136 DEBUG:boto:Method: GET 2023-10-18T20:45:28.137 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.137 DEBUG:boto:Data: 2023-10-18T20:45:28.138 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.138 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:28.139 DEBUG:boto:Port: 8000 2023-10-18T20:45:28.139 DEBUG:boto:Params: {} 2023-10-18T20:45:28.140 DEBUG:boto:Token: None 2023-10-18T20:45:28.140 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.141 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hDjEIh4u1KxJhmzIE1puuxafCv4= 2023-10-18T20:45:28.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hDjEIh4u1KxJhmzIE1puuxafCv4=', 'Content-Length': '0'} 2023-10-18T20:45:28.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000210346aeb81e5f19-00653043e8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.144 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.145 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.145 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.146 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.146 DEBUG:boto:Method: HEAD 2023-10-18T20:45:28.146 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.147 DEBUG:boto:Data: 2023-10-18T20:45:28.147 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.148 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:28.148 DEBUG:boto:Port: 8001 2023-10-18T20:45:28.149 DEBUG:boto:Params: {} 2023-10-18T20:45:28.149 DEBUG:boto:Token: None 2023-10-18T20:45:28.149 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.150 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lm/xjDGsihFPZxP/nhqxUnacI0w= 2023-10-18T20:45:28.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lm/xjDGsihFPZxP/nhqxUnacI0w=', 'Content-Length': '0'} 2023-10-18T20:45:28.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7eaf1bda3a51ff-00653043e8-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.153 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.153 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.154 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.154 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.155 DEBUG:boto:Method: HEAD 2023-10-18T20:45:28.155 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.156 DEBUG:boto:Data: 2023-10-18T20:45:28.156 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.157 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:28.157 DEBUG:boto:Port: 8000 2023-10-18T20:45:28.157 DEBUG:boto:Params: {} 2023-10-18T20:45:28.158 DEBUG:boto:Token: None 2023-10-18T20:45:28.158 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.159 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lm/xjDGsihFPZxP/nhqxUnacI0w= 2023-10-18T20:45:28.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lm/xjDGsihFPZxP/nhqxUnacI0w=', 'Content-Length': '0'} 2023-10-18T20:45:28.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d6282d9476f2d1a-00653043e8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:45:28.163 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.163 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.163 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.164 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.165 DEBUG:boto:Method: GET 2023-10-18T20:45:28.165 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.165 DEBUG:boto:Data: 2023-10-18T20:45:28.166 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:28.167 DEBUG:boto:Port: 8001 2023-10-18T20:45:28.167 DEBUG:boto:Params: {} 2023-10-18T20:45:28.168 DEBUG:boto:Token: None 2023-10-18T20:45:28.168 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.169 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hDjEIh4u1KxJhmzIE1puuxafCv4= 2023-10-18T20:45:28.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hDjEIh4u1KxJhmzIE1puuxafCv4=', 'Content-Length': '0'} 2023-10-18T20:45:28.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063dc5b1aa05293d5-00653043e8-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.172 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.173 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.173 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.173 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.174 DEBUG:boto:Method: GET 2023-10-18T20:45:28.174 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.175 DEBUG:boto:Data: 2023-10-18T20:45:28.175 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.176 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:28.176 DEBUG:boto:Port: 8000 2023-10-18T20:45:28.176 DEBUG:boto:Params: {} 2023-10-18T20:45:28.177 DEBUG:boto:Token: None 2023-10-18T20:45:28.177 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:28.178 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hDjEIh4u1KxJhmzIE1puuxafCv4= 2023-10-18T20:45:28.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hDjEIh4u1KxJhmzIE1puuxafCv4=', 'Content-Length': '0'} 2023-10-18T20:45:28.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000981e97bb207f3647-00653043e8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.181 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:28.182 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:28.182 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.182 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.183 DEBUG:boto:Method: GET 2023-10-18T20:45:28.183 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.184 DEBUG:boto:Data: 2023-10-18T20:45:28.184 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.185 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:28.185 DEBUG:boto:Port: 8001 2023-10-18T20:45:28.185 DEBUG:boto:Params: {} 2023-10-18T20:45:28.186 DEBUG:boto:Token: None 2023-10-18T20:45:28.186 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/ 2023-10-18T20:45:28.187 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw= 2023-10-18T20:45:28.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw=', 'Content-Length': '0'} 2023-10-18T20:45:28.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007efccf9d13536687-00653043e8-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.191 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:28.191 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:28.192 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:28.192 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.193 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.193 DEBUG:boto:Method: GET 2023-10-18T20:45:28.194 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.194 DEBUG:boto:Data: 2023-10-18T20:45:28.194 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.195 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:28.195 DEBUG:boto:Port: 8000 2023-10-18T20:45:28.196 DEBUG:boto:Params: {} 2023-10-18T20:45:28.196 DEBUG:boto:Token: None 2023-10-18T20:45:28.197 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/ 2023-10-18T20:45:28.197 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw= 2023-10-18T20:45:28.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw=', 'Content-Length': '0'} 2023-10-18T20:45:28.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077e49cee24652baa-00653043e8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.203 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:28.203 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.204 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.204 DEBUG:boto:Method: HEAD 2023-10-18T20:45:28.205 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:45:28.205 DEBUG:boto:Data: 2023-10-18T20:45:28.205 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.206 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:28.206 DEBUG:boto:Port: 8001 2023-10-18T20:45:28.207 DEBUG:boto:Params: {} 2023-10-18T20:45:28.207 DEBUG:boto:Token: None 2023-10-18T20:45:28.208 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95 2023-10-18T20:45:28.208 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i9PSItE2bxm7iFb7cZAuM07DmwU= 2023-10-18T20:45:28.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i9PSItE2bxm7iFb7cZAuM07DmwU=', 'Content-Length': '0'} 2023-10-18T20:45:28.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d531c8981a30929-00653043e8-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.211 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.212 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.212 DEBUG:boto:Method: HEAD 2023-10-18T20:45:28.212 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:45:28.213 DEBUG:boto:Data: 2023-10-18T20:45:28.213 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.214 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:28.214 DEBUG:boto:Port: 8000 2023-10-18T20:45:28.215 DEBUG:boto:Params: {} 2023-10-18T20:45:28.215 DEBUG:boto:Token: None 2023-10-18T20:45:28.216 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95 2023-10-18T20:45:28.216 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i9PSItE2bxm7iFb7cZAuM07DmwU= 2023-10-18T20:45:28.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i9PSItE2bxm7iFb7cZAuM07DmwU=', 'Content-Length': '0'} 2023-10-18T20:45:28.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50b2724aa4458a7-00653043e8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:45:28.221 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.222 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.223 DEBUG:boto:Method: GET 2023-10-18T20:45:28.223 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:45:28.223 DEBUG:boto:Data: 2023-10-18T20:45:28.224 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.224 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:28.225 DEBUG:boto:Port: 8001 2023-10-18T20:45:28.225 DEBUG:boto:Params: {} 2023-10-18T20:45:28.226 DEBUG:boto:Token: None 2023-10-18T20:45:28.226 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95 2023-10-18T20:45:28.227 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:13YZjBRoo1HefhqtGHMhzmvtBPI= 2023-10-18T20:45:28.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:13YZjBRoo1HefhqtGHMhzmvtBPI=', 'Content-Length': '0'} 2023-10-18T20:45:28.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d66f3d2ee883645-00653043e8-1af4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.274 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:28.275 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:28.275 DEBUG:boto:Method: GET 2023-10-18T20:45:28.276 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:45:28.276 DEBUG:boto:Data: 2023-10-18T20:45:28.277 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.277 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:28.278 DEBUG:boto:Port: 8000 2023-10-18T20:45:28.278 DEBUG:boto:Params: {} 2023-10-18T20:45:28.279 DEBUG:boto:Token: None 2023-10-18T20:45:28.279 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/obj95 2023-10-18T20:45:28.279 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:13YZjBRoo1HefhqtGHMhzmvtBPI= 2023-10-18T20:45:28.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:13YZjBRoo1HefhqtGHMhzmvtBPI=', 'Content-Length': '0'} 2023-10-18T20:45:28.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000450aa122e58888e2-00653043e8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.283 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:28.283 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:28.284 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.284 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.284 DEBUG:boto:Method: GET 2023-10-18T20:45:28.285 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.285 DEBUG:boto:Data: 2023-10-18T20:45:28.286 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.286 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:28.287 DEBUG:boto:Port: 8001 2023-10-18T20:45:28.287 DEBUG:boto:Params: {} 2023-10-18T20:45:28.287 DEBUG:boto:Token: None 2023-10-18T20:45:28.288 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/ 2023-10-18T20:45:28.288 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw= 2023-10-18T20:45:28.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw=', 'Content-Length': '0'} 2023-10-18T20:45:28.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc1a0638b0b7c59c-00653043e8-1af4-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.292 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:28.292 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:28.293 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:28.293 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.294 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.294 DEBUG:boto:Method: GET 2023-10-18T20:45:28.295 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:45:28.295 DEBUG:boto:Data: 2023-10-18T20:45:28.296 DEBUG:boto:Headers: {} 2023-10-18T20:45:28.296 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:28.297 DEBUG:boto:Port: 8000 2023-10-18T20:45:28.297 DEBUG:boto:Params: {} 2023-10-18T20:45:28.297 DEBUG:boto:Token: None 2023-10-18T20:45:28.298 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:28 GMT /bofrnf-59/ 2023-10-18T20:45:28.299 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw= 2023-10-18T20:45:28.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FOWIUkXaUT6jvv2PzIYAtiH6Btw=', 'Content-Length': '0'} 2023-10-18T20:45:28.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000333ae9dff4b5f298-00653043e8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:28.350 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:28.351 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:45:28.352 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-18T20:45:28.353 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T20:45:28.354 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:45:28.352+0000 7fb8d4387700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 159870) UID: 0 2023-10-18T20:45:28.354 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:45:28.352+0000 7fb8d85f07c0 -1 shutting down 2023-10-18T20:45:32.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T20:45:34.452 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-18T20:45:34.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:34.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:34.454 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:45:34.514 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:34.515 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:34.550 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:45:34.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.550 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj11", 2023-10-18T20:45:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:34.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.552 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.552 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.552 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.552 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.624897Z", 2023-10-18T20:45:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.553 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.554 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.554 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.554 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.554 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_XgwpKhUeXgMILhInrtsddtAyewYgnJu", 2023-10-18T20:45:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.555 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj111", 2023-10-18T20:45:34.556 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:45:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.557 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:34.558 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.726998Z", 2023-10-18T20:45:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.559 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_TtMyEBr4AiefJHzRwQdLu0pS92FvI5I", 2023-10-18T20:45:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj115", 2023-10-18T20:45:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.562 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.563 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:34.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.563 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.563 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.564 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.564 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.564 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.564 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.234025Z", 2023-10-18T20:45:34.564 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.565 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_3p_oSe6C72gI-FZPNxmRyqzC9VvyBbD", 2023-10-18T20:45:34.566 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.567 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.567 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj116", 2023-10-18T20:45:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.568 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.992319Z", 2023-10-18T20:45:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.570 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.571 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.571 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.571 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.571 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_ZbR7zhjmUSQjJDNcImW4Y1b9SVm45Yx", 2023-10-18T20:45:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.572 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.573 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.573 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj124", 2023-10-18T20:45:34.573 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.574 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.574 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.574 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:34.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.575 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.828946Z", 2023-10-18T20:45:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.576 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.578 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_Zfh3Xmct4pNc5ronCxihH7AA4kE-1QO", 2023-10-18T20:45:34.578 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.578 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.578 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj145", 2023-10-18T20:45:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.579 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:45:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.580 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:47.012959Z", 2023-10-18T20:45:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_WDk_byaHQaT4mW00CmUdmJ7lgYoIYr3", 2023-10-18T20:45:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj148", 2023-10-18T20:45:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.746370Z", 2023-10-18T20:45:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_i8dhfKoRlrn36-IedCW0f_xErGo8l59", 2023-10-18T20:45:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj155", 2023-10-18T20:45:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:34.593 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.890428Z", 2023-10-18T20:45:34.593 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.593 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.593 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.594 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_WHNTIlJY1uji9aY8ndOSO1LSfJjwgEq", 2023-10-18T20:45:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.595 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj17", 2023-10-18T20:45:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.596 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.597 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.597 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:34.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.597 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.598 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.598 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.598 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.598 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.598 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.330763Z", 2023-10-18T20:45:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.599 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.600 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_vmhsCNtGXoV5ruUyn6kWPVTMd_K8fq5", 2023-10-18T20:45:34.601 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.601 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.601 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj185", 2023-10-18T20:45:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.602 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.603 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:34.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.603 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.603 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.604 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.604 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.604 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:34.604 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.802901Z", 2023-10-18T20:45:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.605 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.606 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.606 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.606 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.606 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_trPmvLk-BZpmMIak66Eu1TeTSYyAH_k", 2023-10-18T20:45:34.607 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.607 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.607 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj19", 2023-10-18T20:45:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.608 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.609 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.373675Z", 2023-10-18T20:45:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.610 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.611 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.611 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.611 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.611 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_mmX_6Xw4BYP-oAdJO0mvlSEv-vQm_zh", 2023-10-18T20:45:34.612 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.613 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj22", 2023-10-18T20:45:34.614 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.614 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.614 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.615 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.616 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.616 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:34.616 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.645343Z", 2023-10-18T20:45:34.616 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.617 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.618 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.618 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.618 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_22cn_dMLB3gdM3A9s910HtSNEC5vocJ", 2023-10-18T20:45:34.618 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.619 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj23", 2023-10-18T20:45:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:45:34.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.621 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.621 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.621 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.621 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.071727Z", 2023-10-18T20:45:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.622 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.623 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_75fnW8IHTOcf97q2hQuwRCkBrb9Wt7o", 2023-10-18T20:45:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.624 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.625 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj29", 2023-10-18T20:45:34.625 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:45:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.626 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.627 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.893544Z", 2023-10-18T20:45:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.628 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.629 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_5GoHjuOUzaSPdOzfopOw2vLPOmICI_y", 2023-10-18T20:45:34.630 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.630 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.630 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj31", 2023-10-18T20:45:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.631 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:45:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.632 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.732307Z", 2023-10-18T20:45:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.633 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.634 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.634 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.634 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.634 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_gFQN7sXaIaEPlzK_ACrS1mHsdsEMskh", 2023-10-18T20:45:34.635 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.636 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj33", 2023-10-18T20:45:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:34.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.638 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.638 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.638 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.638 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.664873Z", 2023-10-18T20:45:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.639 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.640 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_OmKcMjYu3rHagbiJPAYIqUFjsJlb-p2", 2023-10-18T20:45:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.641 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.642 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj34", 2023-10-18T20:45:34.642 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:45:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.643 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:34.644 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.349351Z", 2023-10-18T20:45:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.645 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.646 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_o6EuUvg_jxg01MWk-1hHkCfScNchThP", 2023-10-18T20:45:34.647 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.647 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.647 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj37", 2023-10-18T20:45:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.648 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:45:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.649 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.650 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.650 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:34.650 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.648488Z", 2023-10-18T20:45:34.650 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.651 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_Oi8LqlkhibZUGpDMp5IEzXOY2L0fpqe", 2023-10-18T20:45:34.652 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj38", 2023-10-18T20:45:34.654 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.654 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.654 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.654 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.655 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.394123Z", 2023-10-18T20:45:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.656 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.657 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.658 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_srRkDWlI8ZwWYR7vXt9E60oPb9auw5v", 2023-10-18T20:45:34.658 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.658 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj44", 2023-10-18T20:45:34.659 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.660 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.660 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.660 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 13 2023-10-18T20:45:34.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.660 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.661 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.661 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.661 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.661 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.692260Z", 2023-10-18T20:45:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.662 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.663 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.663 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.663 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.663 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.664 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_QWbRmo7lX1DzYE7wzlT15b7XIUIUZBB", 2023-10-18T20:45:34.664 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.664 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.664 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.665 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj46", 2023-10-18T20:45:34.665 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.666 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.667 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.132203Z", 2023-10-18T20:45:34.668 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.668 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.668 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.668 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.669 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.669 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.669 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.669 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_V633SNmsbI0JHmCXTnTrlWdqWfhKE0G", 2023-10-18T20:45:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.670 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj51", 2023-10-18T20:45:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.671 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.672 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.672 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:45:34.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.672 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:34.673 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.411923Z", 2023-10-18T20:45:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.675 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.675 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.675 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.676 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_K0wxgGWIyx52OTEu3nPfkrCzjE3u4Uv", 2023-10-18T20:45:34.676 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.676 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.676 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj55", 2023-10-18T20:45:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.677 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.678 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.679 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.679 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:34.679 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.750824Z", 2023-10-18T20:45:34.679 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.680 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_wM4tbHG6MjT8FOws4Mn5O4RLz7pvwHK", 2023-10-18T20:45:34.681 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.682 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj57", 2023-10-18T20:45:34.683 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.683 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.683 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.683 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 17 2023-10-18T20:45:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.684 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.453381Z", 2023-10-18T20:45:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.685 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.686 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.686 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.686 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.686 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.686 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_VsiV11Wan4dxVNMXanaywTlZznuJDZZ", 2023-10-18T20:45:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.687 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj58", 2023-10-18T20:45:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.688 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.689 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.689 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:45:34.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.689 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.690 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.913247Z", 2023-10-18T20:45:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.691 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.692 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.692 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.692 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_H8vZz6SvIaQ8ylrOyuupMJ3rTQLoP_g", 2023-10-18T20:45:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj62", 2023-10-18T20:45:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.694 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.695 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:45:34.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.695 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.695 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.695 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.696 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.696 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.696 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.751922Z", 2023-10-18T20:45:34.696 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.697 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_XLF-jyoIhvua8gp-olkqFe6pRzKv-dk", 2023-10-18T20:45:34.698 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.699 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.699 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj68", 2023-10-18T20:45:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.700 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:34.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.701 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.701 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.702 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.702 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.702 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:34.702 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.431067Z", 2023-10-18T20:45:34.702 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.703 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.703 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.703 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.703 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.704 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_oD_ddTOp0Fez2f8-cytQjbwXBVa1wEE", 2023-10-18T20:45:34.705 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.705 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.705 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj69", 2023-10-18T20:45:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.706 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.707 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:34.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.707 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.707 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.150793Z", 2023-10-18T20:45:34.708 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.709 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.710 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.710 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.710 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_U9C3Ta5qyCglO03gVL4XQzypQSG3vrl", 2023-10-18T20:45:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj74", 2023-10-18T20:45:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.712 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:45:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.667632Z", 2023-10-18T20:45:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.714 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.715 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.715 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.715 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.715 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_swM_VRrzoja5BtwTEATvYLPIMU_hXD4", 2023-10-18T20:45:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.716 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj76", 2023-10-18T20:45:34.717 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:45:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.718 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.719 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.719 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.719 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.719 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.470726Z", 2023-10-18T20:45:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.720 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.721 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.721 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.721 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.721 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_BH-eLFj67GV70qlUZLSrF8qHaAKgtNp", 2023-10-18T20:45:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj85", 2023-10-18T20:45:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.723 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.724 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:45:34.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.724 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.724 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.491392Z", 2023-10-18T20:45:34.725 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.726 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.727 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.727 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.727 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_hK37L2qpYkSljs7tGLmEta6gOLN6xh2", 2023-10-18T20:45:34.728 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.728 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.728 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj87", 2023-10-18T20:45:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.729 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:34.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.730 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.730 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.731 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.932864Z", 2023-10-18T20:45:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.732 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.733 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.733 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.733 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.733 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_4cRr1aIOumfx2rDegG0cQjUeGPFa9au", 2023-10-18T20:45:34.734 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.735 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj92", 2023-10-18T20:45:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:34.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.737 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.737 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.737 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.737 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.738 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:34.738 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.214384Z", 2023-10-18T20:45:34.738 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.738 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.738 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.739 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.739 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.739 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.739 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_20YHlV2Tc4vQU8sd8pMO7iXFbvfjwQB", 2023-10-18T20:45:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.740 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj93", 2023-10-18T20:45:34.741 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 15 2023-10-18T20:45:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.743 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.743 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.743 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.743 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.810194Z", 2023-10-18T20:45:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.744 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_7hSFL7NbKwtVGq-k2BvXWypWRBswjLZ", 2023-10-18T20:45:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:34.747 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj95", 2023-10-18T20:45:34.747 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:34.747 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:45:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:34.748 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.530014Z", 2023-10-18T20:45:34.749 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:34.750 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:34.751 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:34.751 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:34.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:34.751 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_qrKYPaQYoJn8jh5EZnNjOY3I6JJrPCw", 2023-10-18T20:45:34.752 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:34.752 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:34.752 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:34.752 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:34.752 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:45:34.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:34.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:34.754 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:34.808 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:34.809 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:34.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:34.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:34.895 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:34.946 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:34.946 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:35.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.029 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:35.080 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:35.081 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:35.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.163 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:35.214 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:35.214 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:35.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.299 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:35.349 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:35.350 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:35.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.437 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:35.487 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:35.488 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:35.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.571 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:35.623 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:35.624 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:35.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.708 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:35.760 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:35.761 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:35.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.844 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:35.895 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:35.895 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:35.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:35.979 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:36.030 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:36.030 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:36.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.116 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:36.168 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:36.168 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:36.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.253 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:36.307 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:36.308 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:36.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.393 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:36.443 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:36.444 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:36.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.529 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:36.580 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:36.581 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:36.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.665 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:36.716 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:36.717 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:36.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.803 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:36.854 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:36.855 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:36.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:36.939 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:36.989 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:36.990 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:37.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.075 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:37.126 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:37.126 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:37.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.212 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:37.263 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:37.264 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:37.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.347 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:37.399 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:37.399 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:37.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.486 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:37.538 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:37.538 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:37.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.622 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:37.673 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:37.674 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:37.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.758 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:37.809 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:37.810 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:37.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:37.894 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:37.946 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:37.946 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:37.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-18T20:45:38.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.032 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:38.083 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:38.083 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:38.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.172 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:38.223 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:38.224 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:38.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.308 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:38.359 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:38.360 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:38.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.445 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:38.497 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:38.498 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:38.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.586 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:38.637 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:38.637 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:38.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.721 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:38.773 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:38.773 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:38.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.862 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:38.913 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:38.914 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:38.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:38.997 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:39.049 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:39.049 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:39.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.134 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:39.186 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:39.186 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:39.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.274 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:39.325 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:39.325 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:39.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.412 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:39.464 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:39.465 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:39.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.552 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:39.601 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:39.602 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:39.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.652 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:39.704 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:39.704 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:39.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:45:39.749 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:45:39.802 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:39.802 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:39.867 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.866+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2023-10-18T20:45:39.872 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.866+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: waiting on bucket lease 2023-10-18T20:45:39.873 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.871+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: status after acquiring the lock is: init 2023-10-18T20:45:39.885 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.884+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: listing bucket for full sync 2023-10-18T20:45:39.933 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj11 2023-10-18T20:45:39.934 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj11 op=0 op_state=1 2023-10-18T20:45:39.934 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj11[0] 2023-10-18T20:45:39.934 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj111 2023-10-18T20:45:39.934 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj111 op=0 op_state=1 2023-10-18T20:45:39.934 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-10-18T20:45:39.935 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:39.935 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj111 2023-10-18T20:45:39.935 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj115 2023-10-18T20:45:39.935 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj115 op=0 op_state=1 2023-10-18T20:45:39.936 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj115[0] 2023-10-18T20:45:39.936 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj116 2023-10-18T20:45:39.936 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj116 op=0 op_state=1 2023-10-18T20:45:39.936 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj116[0] 2023-10-18T20:45:39.936 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj124 2023-10-18T20:45:39.937 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj124 op=0 op_state=1 2023-10-18T20:45:39.937 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj124[0] 2023-10-18T20:45:39.937 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj145 2023-10-18T20:45:39.937 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj145 op=0 op_state=1 2023-10-18T20:45:39.938 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj145[0] 2023-10-18T20:45:39.938 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj148 2023-10-18T20:45:39.938 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj148 op=0 op_state=1 2023-10-18T20:45:39.938 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj148[0] 2023-10-18T20:45:39.939 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj155 2023-10-18T20:45:39.939 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj155 op=0 op_state=1 2023-10-18T20:45:39.939 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-10-18T20:45:39.939 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:39.939 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj155 2023-10-18T20:45:39.940 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj17 2023-10-18T20:45:39.940 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj17 op=0 op_state=1 2023-10-18T20:45:39.940 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.930+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj17[0] 2023-10-18T20:45:39.940 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj185 2023-10-18T20:45:39.940 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj185 op=0 op_state=1 2023-10-18T20:45:39.941 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj185[0] 2023-10-18T20:45:39.941 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj19 2023-10-18T20:45:39.941 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj19 op=0 op_state=1 2023-10-18T20:45:39.941 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj19[0] 2023-10-18T20:45:39.942 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj22 2023-10-18T20:45:39.942 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj22 op=0 op_state=1 2023-10-18T20:45:39.942 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: : injecting data sync error on key=obj22 2023-10-18T20:45:39.942 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:39.943 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj22 2023-10-18T20:45:39.943 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj23 2023-10-18T20:45:39.943 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj23 op=0 op_state=1 2023-10-18T20:45:39.943 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj23[0] 2023-10-18T20:45:39.943 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj29 2023-10-18T20:45:39.944 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj29 op=0 op_state=1 2023-10-18T20:45:39.944 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-10-18T20:45:39.944 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:39.944 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj29 2023-10-18T20:45:39.944 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj31 2023-10-18T20:45:39.945 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj31 op=0 op_state=1 2023-10-18T20:45:39.945 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj31[0] 2023-10-18T20:45:39.945 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj33 2023-10-18T20:45:39.945 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj33 op=0 op_state=1 2023-10-18T20:45:39.946 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj33[0] 2023-10-18T20:45:39.946 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj34 2023-10-18T20:45:39.946 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj34 op=0 op_state=1 2023-10-18T20:45:39.946 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: : injecting data sync error on key=obj34 2023-10-18T20:45:39.946 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:39.947 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj34 2023-10-18T20:45:39.947 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj37 2023-10-18T20:45:39.947 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj37 op=0 op_state=1 2023-10-18T20:45:39.947 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj37[0] 2023-10-18T20:45:39.948 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj38 2023-10-18T20:45:39.948 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj38 op=0 op_state=1 2023-10-18T20:45:39.948 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj38[0] 2023-10-18T20:45:39.948 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj44 2023-10-18T20:45:39.948 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj44 op=0 op_state=1 2023-10-18T20:45:39.949 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj44[0] 2023-10-18T20:45:39.949 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj46 2023-10-18T20:45:39.949 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj46 op=0 op_state=1 2023-10-18T20:45:39.949 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.931+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj46[0] 2023-10-18T20:45:39.951 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:39.951 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj51 2023-10-18T20:45:39.951 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj51 op=0 op_state=1 2023-10-18T20:45:39.951 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj51[0] 2023-10-18T20:45:39.952 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:39.952 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj55 2023-10-18T20:45:39.952 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj55 op=0 op_state=1 2023-10-18T20:45:39.952 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj55[0] 2023-10-18T20:45:39.952 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:39.953 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj57 2023-10-18T20:45:39.953 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj57 op=0 op_state=1 2023-10-18T20:45:39.953 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj57[0] 2023-10-18T20:45:39.953 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:39.954 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.935+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj58 2023-10-18T20:45:39.954 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.936+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj58 op=0 op_state=1 2023-10-18T20:45:39.954 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.936+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj58[0] 2023-10-18T20:45:39.954 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.936+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:39.955 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.936+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj62 2023-10-18T20:45:39.955 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.936+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj62 op=0 op_state=1 2023-10-18T20:45:39.955 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.936+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj62[0] 2023-10-18T20:45:39.956 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: success 2023-10-18T20:45:39.956 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj68 2023-10-18T20:45:39.956 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj68 op=0 op_state=1 2023-10-18T20:45:39.957 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj68[0] 2023-10-18T20:45:39.957 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: success 2023-10-18T20:45:39.957 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj69 2023-10-18T20:45:39.957 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj69 op=0 op_state=1 2023-10-18T20:45:39.958 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj69[0] 2023-10-18T20:45:39.958 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: success 2023-10-18T20:45:39.958 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj74 2023-10-18T20:45:39.958 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj74 op=0 op_state=1 2023-10-18T20:45:39.959 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj74[0] 2023-10-18T20:45:39.959 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: success 2023-10-18T20:45:39.959 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: success 2023-10-18T20:45:39.959 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj76 2023-10-18T20:45:39.959 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj76 op=0 op_state=1 2023-10-18T20:45:39.961 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj76[0] 2023-10-18T20:45:39.961 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: success 2023-10-18T20:45:39.961 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj85 2023-10-18T20:45:39.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj85 op=0 op_state=1 2023-10-18T20:45:39.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj85[0] 2023-10-18T20:45:39.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj87 2023-10-18T20:45:39.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj87 op=0 op_state=1 2023-10-18T20:45:39.963 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.954+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj87[0] 2023-10-18T20:45:39.964 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.956+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: success 2023-10-18T20:45:39.964 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.956+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj92 2023-10-18T20:45:39.965 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.956+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj92 op=0 op_state=1 2023-10-18T20:45:39.965 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.956+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj92[0] 2023-10-18T20:45:39.965 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: success 2023-10-18T20:45:39.965 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: success 2023-10-18T20:45:39.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj93 2023-10-18T20:45:39.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj93 op=0 op_state=1 2023-10-18T20:45:39.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj93[0] 2023-10-18T20:45:39.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj95 2023-10-18T20:45:39.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj95 op=0 op_state=1 2023-10-18T20:45:39.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj95[0] 2023-10-18T20:45:39.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: success 2023-10-18T20:45:39.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.958+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: success 2023-10-18T20:45:39.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.959+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: success 2023-10-18T20:45:39.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.959+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: success 2023-10-18T20:45:39.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.962+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: success 2023-10-18T20:45:39.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.964+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: success 2023-10-18T20:45:39.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.965+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: success 2023-10-18T20:45:39.983 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.981+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: success 2023-10-18T20:45:39.984 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.983+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: success 2023-10-18T20:45:39.986 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.985+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: success 2023-10-18T20:45:39.987 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.986+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: success 2023-10-18T20:45:39.988 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.987+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: success 2023-10-18T20:45:39.988 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.987+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: success 2023-10-18T20:45:39.989 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.987+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: success 2023-10-18T20:45:39.990 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:39.989+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: success 2023-10-18T20:45:40.003 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.002+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: success 2023-10-18T20:45:40.004 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.003+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: success 2023-10-18T20:45:40.005 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.003+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: success 2023-10-18T20:45:40.005 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.003+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: success 2023-10-18T20:45:40.005 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.003+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: success 2023-10-18T20:45:40.006 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.005+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: success 2023-10-18T20:45:40.006 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.005+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj11 2023-10-18T20:45:40.007 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.006+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: backing out with sync_status=-5 2023-10-18T20:45:40.007 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.006+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: ERROR: full sync failed. error: -5 2023-10-18T20:45:40.008 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.007+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.009 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.007+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: waiting on bucket lease 2023-10-18T20:45:40.013 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.012+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: status after acquiring the lock is: full 2023-10-18T20:45:40.027 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.025+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: listing bucket for full sync 2023-10-18T20:45:40.074 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj11 2023-10-18T20:45:40.074 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj11 op=0 op_state=1 2023-10-18T20:45:40.075 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj11[0] 2023-10-18T20:45:40.075 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj111 2023-10-18T20:45:40.075 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj111 op=0 op_state=1 2023-10-18T20:45:40.075 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj111[0] 2023-10-18T20:45:40.075 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj115 2023-10-18T20:45:40.076 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj115 op=0 op_state=1 2023-10-18T20:45:40.076 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj115[0] 2023-10-18T20:45:40.076 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj116 2023-10-18T20:45:40.076 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj116 op=0 op_state=1 2023-10-18T20:45:40.077 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj116[0] 2023-10-18T20:45:40.077 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj124 2023-10-18T20:45:40.077 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj124 op=0 op_state=1 2023-10-18T20:45:40.077 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj124[0] 2023-10-18T20:45:40.078 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj145 2023-10-18T20:45:40.078 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj145 op=0 op_state=1 2023-10-18T20:45:40.078 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj145[0] 2023-10-18T20:45:40.078 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj148 2023-10-18T20:45:40.079 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj148 op=0 op_state=1 2023-10-18T20:45:40.079 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj148[0] 2023-10-18T20:45:40.079 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj155 2023-10-18T20:45:40.079 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj155 op=0 op_state=1 2023-10-18T20:45:40.080 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj155[0] 2023-10-18T20:45:40.080 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj17 2023-10-18T20:45:40.080 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj17 op=0 op_state=1 2023-10-18T20:45:40.080 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj17[0] 2023-10-18T20:45:40.080 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj185 2023-10-18T20:45:40.081 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj185 op=0 op_state=1 2023-10-18T20:45:40.081 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj185[0] 2023-10-18T20:45:40.081 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj19 2023-10-18T20:45:40.081 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj19 op=0 op_state=1 2023-10-18T20:45:40.082 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj19[0] 2023-10-18T20:45:40.082 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj22 2023-10-18T20:45:40.082 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj22 op=0 op_state=1 2023-10-18T20:45:40.082 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj22[0] 2023-10-18T20:45:40.082 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj23 2023-10-18T20:45:40.083 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj23 op=0 op_state=1 2023-10-18T20:45:40.083 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj23[0] 2023-10-18T20:45:40.083 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj29 2023-10-18T20:45:40.083 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj29 op=0 op_state=1 2023-10-18T20:45:40.084 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj29[0] 2023-10-18T20:45:40.084 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj31 2023-10-18T20:45:40.084 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj31 op=0 op_state=1 2023-10-18T20:45:40.084 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj31[0] 2023-10-18T20:45:40.084 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj33 2023-10-18T20:45:40.085 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj33 op=0 op_state=1 2023-10-18T20:45:40.085 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj33[0] 2023-10-18T20:45:40.085 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj34 2023-10-18T20:45:40.085 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj34 op=0 op_state=1 2023-10-18T20:45:40.086 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj34[0] 2023-10-18T20:45:40.086 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj37 2023-10-18T20:45:40.086 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj37 op=0 op_state=1 2023-10-18T20:45:40.086 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.071+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj37[0] 2023-10-18T20:45:40.086 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.072+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj38 2023-10-18T20:45:40.087 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.072+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj38 op=0 op_state=1 2023-10-18T20:45:40.087 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.072+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj38[0] 2023-10-18T20:45:40.087 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.072+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj44 2023-10-18T20:45:40.087 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.072+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj44 op=0 op_state=1 2023-10-18T20:45:40.088 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.072+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj44[0] 2023-10-18T20:45:40.088 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.072+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj46 2023-10-18T20:45:40.088 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.072+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj46 op=0 op_state=1 2023-10-18T20:45:40.088 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.072+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj46[0] 2023-10-18T20:45:40.091 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: success 2023-10-18T20:45:40.092 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj51 2023-10-18T20:45:40.092 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj51 op=0 op_state=1 2023-10-18T20:45:40.092 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj51[0] 2023-10-18T20:45:40.092 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: success 2023-10-18T20:45:40.093 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: success 2023-10-18T20:45:40.093 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: success 2023-10-18T20:45:40.093 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj55 2023-10-18T20:45:40.093 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj55 op=0 op_state=1 2023-10-18T20:45:40.093 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj55[0] 2023-10-18T20:45:40.094 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj57 2023-10-18T20:45:40.094 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj57 op=0 op_state=1 2023-10-18T20:45:40.094 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj57[0] 2023-10-18T20:45:40.094 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: success 2023-10-18T20:45:40.095 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj58 2023-10-18T20:45:40.095 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj58 op=0 op_state=1 2023-10-18T20:45:40.095 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj58[0] 2023-10-18T20:45:40.095 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj62 2023-10-18T20:45:40.096 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj62 op=0 op_state=1 2023-10-18T20:45:40.096 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: success 2023-10-18T20:45:40.096 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj62[0] 2023-10-18T20:45:40.096 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: success 2023-10-18T20:45:40.096 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj68 2023-10-18T20:45:40.097 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj68 op=0 op_state=1 2023-10-18T20:45:40.097 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: : injecting data sync error on key=obj68 2023-10-18T20:45:40.097 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.097 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj68 2023-10-18T20:45:40.098 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj69 2023-10-18T20:45:40.098 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj69 op=0 op_state=1 2023-10-18T20:45:40.098 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj69[0] 2023-10-18T20:45:40.098 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: success 2023-10-18T20:45:40.099 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj74 2023-10-18T20:45:40.099 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj74 op=0 op_state=1 2023-10-18T20:45:40.099 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.076+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj74[0] 2023-10-18T20:45:40.099 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: success 2023-10-18T20:45:40.099 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: success 2023-10-18T20:45:40.100 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: success 2023-10-18T20:45:40.100 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj76 2023-10-18T20:45:40.100 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj76 op=0 op_state=1 2023-10-18T20:45:40.100 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: : injecting data sync error on key=obj76 2023-10-18T20:45:40.101 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.101 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj76 2023-10-18T20:45:40.101 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj85 2023-10-18T20:45:40.101 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj85 op=0 op_state=1 2023-10-18T20:45:40.101 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj85[0] 2023-10-18T20:45:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj87 2023-10-18T20:45:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj87 op=0 op_state=1 2023-10-18T20:45:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-10-18T20:45:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.103 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.077+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj87 2023-10-18T20:45:40.103 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.079+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: success 2023-10-18T20:45:40.103 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.079+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj92 2023-10-18T20:45:40.103 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.079+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj92 op=0 op_state=1 2023-10-18T20:45:40.104 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.079+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj92[0] 2023-10-18T20:45:40.104 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.080+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: success 2023-10-18T20:45:40.104 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.080+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj93 2023-10-18T20:45:40.104 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.080+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj93 op=0 op_state=1 2023-10-18T20:45:40.104 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.080+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj93[0] 2023-10-18T20:45:40.105 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.080+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: success 2023-10-18T20:45:40.105 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.080+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: success 2023-10-18T20:45:40.105 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.080+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj95 2023-10-18T20:45:40.105 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.080+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj95 op=0 op_state=1 2023-10-18T20:45:40.105 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.080+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj95[0] 2023-10-18T20:45:40.106 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.081+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: success 2023-10-18T20:45:40.106 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.081+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.106 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.081+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.106 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.082+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: success 2023-10-18T20:45:40.107 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.082+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: success 2023-10-18T20:45:40.107 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.083+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: success 2023-10-18T20:45:40.107 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.083+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: success 2023-10-18T20:45:40.107 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.083+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: success 2023-10-18T20:45:40.107 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.083+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: success 2023-10-18T20:45:40.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.083+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: success 2023-10-18T20:45:40.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.083+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.084+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: success 2023-10-18T20:45:40.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.084+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: success 2023-10-18T20:45:40.108 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.084+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: success 2023-10-18T20:45:40.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.084+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: success 2023-10-18T20:45:40.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.084+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: success 2023-10-18T20:45:40.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.085+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: success 2023-10-18T20:45:40.109 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.085+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj19 2023-10-18T20:45:40.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.086+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: success 2023-10-18T20:45:40.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.087+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: success 2023-10-18T20:45:40.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.088+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: success 2023-10-18T20:45:40.110 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.088+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj62 2023-10-18T20:45:40.111 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.089+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: backing out with sync_status=-5 2023-10-18T20:45:40.111 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.089+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: ERROR: full sync failed. error: -5 2023-10-18T20:45:40.111 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.091+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.112 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.091+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: waiting on bucket lease 2023-10-18T20:45:40.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.135+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: status after acquiring the lock is: full 2023-10-18T20:45:40.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.146+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: listing bucket for full sync 2023-10-18T20:45:40.194 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj11 2023-10-18T20:45:40.194 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj11 op=0 op_state=1 2023-10-18T20:45:40.194 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj11[0] 2023-10-18T20:45:40.195 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj111 2023-10-18T20:45:40.195 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj111 op=0 op_state=1 2023-10-18T20:45:40.195 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj111[0] 2023-10-18T20:45:40.196 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj115 2023-10-18T20:45:40.196 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj115 op=0 op_state=1 2023-10-18T20:45:40.196 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj115[0] 2023-10-18T20:45:40.196 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj116 2023-10-18T20:45:40.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj116 op=0 op_state=1 2023-10-18T20:45:40.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: : injecting data sync error on key=obj116 2023-10-18T20:45:40.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj116 2023-10-18T20:45:40.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj124 2023-10-18T20:45:40.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj124 op=0 op_state=1 2023-10-18T20:45:40.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj124[0] 2023-10-18T20:45:40.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj145 2023-10-18T20:45:40.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj145 op=0 op_state=1 2023-10-18T20:45:40.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj145[0] 2023-10-18T20:45:40.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj148 2023-10-18T20:45:40.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj148 op=0 op_state=1 2023-10-18T20:45:40.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj148[0] 2023-10-18T20:45:40.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj155 2023-10-18T20:45:40.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj155 op=0 op_state=1 2023-10-18T20:45:40.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-10-18T20:45:40.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj155 2023-10-18T20:45:40.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj17 2023-10-18T20:45:40.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj17 op=0 op_state=1 2023-10-18T20:45:40.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj17[0] 2023-10-18T20:45:40.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj185 2023-10-18T20:45:40.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj185 op=0 op_state=1 2023-10-18T20:45:40.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj185[0] 2023-10-18T20:45:40.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj19 2023-10-18T20:45:40.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj19 op=0 op_state=1 2023-10-18T20:45:40.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj19[0] 2023-10-18T20:45:40.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj22 2023-10-18T20:45:40.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj22 op=0 op_state=1 2023-10-18T20:45:40.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj22[0] 2023-10-18T20:45:40.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj23 2023-10-18T20:45:40.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj23 op=0 op_state=1 2023-10-18T20:45:40.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj23[0] 2023-10-18T20:45:40.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj29 2023-10-18T20:45:40.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj29 op=0 op_state=1 2023-10-18T20:45:40.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.191+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj29[0] 2023-10-18T20:45:40.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj31 2023-10-18T20:45:40.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj31 op=0 op_state=1 2023-10-18T20:45:40.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj31[0] 2023-10-18T20:45:40.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj33 2023-10-18T20:45:40.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj33 op=0 op_state=1 2023-10-18T20:45:40.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj33[0] 2023-10-18T20:45:40.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj34 2023-10-18T20:45:40.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj34 op=0 op_state=1 2023-10-18T20:45:40.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj34[0] 2023-10-18T20:45:40.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj37 2023-10-18T20:45:40.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj37 op=0 op_state=1 2023-10-18T20:45:40.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj37[0] 2023-10-18T20:45:40.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj38 2023-10-18T20:45:40.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj38 op=0 op_state=1 2023-10-18T20:45:40.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj38[0] 2023-10-18T20:45:40.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj44 2023-10-18T20:45:40.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj44 op=0 op_state=1 2023-10-18T20:45:40.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj44[0] 2023-10-18T20:45:40.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj46 2023-10-18T20:45:40.210 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj46 op=0 op_state=1 2023-10-18T20:45:40.210 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.192+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj46[0] 2023-10-18T20:45:40.214 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.196+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: success 2023-10-18T20:45:40.214 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: success 2023-10-18T20:45:40.214 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: success 2023-10-18T20:45:40.215 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: success 2023-10-18T20:45:40.215 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj51 2023-10-18T20:45:40.215 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj51 op=0 op_state=1 2023-10-18T20:45:40.215 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj51[0] 2023-10-18T20:45:40.216 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: success 2023-10-18T20:45:40.216 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: success 2023-10-18T20:45:40.216 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: success 2023-10-18T20:45:40.216 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: success 2023-10-18T20:45:40.217 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: success 2023-10-18T20:45:40.217 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj55 2023-10-18T20:45:40.217 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj55 op=0 op_state=1 2023-10-18T20:45:40.217 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj55[0] 2023-10-18T20:45:40.217 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj57 2023-10-18T20:45:40.218 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj57 op=0 op_state=1 2023-10-18T20:45:40.218 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj57[0] 2023-10-18T20:45:40.218 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj58 2023-10-18T20:45:40.219 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj58 op=0 op_state=1 2023-10-18T20:45:40.219 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj58[0] 2023-10-18T20:45:40.219 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: success 2023-10-18T20:45:40.219 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: success 2023-10-18T20:45:40.220 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: success 2023-10-18T20:45:40.220 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: success 2023-10-18T20:45:40.220 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj62 2023-10-18T20:45:40.220 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj62 op=0 op_state=1 2023-10-18T20:45:40.221 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: : injecting data sync error on key=obj62 2023-10-18T20:45:40.221 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: success 2023-10-18T20:45:40.221 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.221 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj62 2023-10-18T20:45:40.221 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj68 2023-10-18T20:45:40.222 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj68 op=0 op_state=1 2023-10-18T20:45:40.222 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj68[0] 2023-10-18T20:45:40.222 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj69 2023-10-18T20:45:40.222 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj69 op=0 op_state=1 2023-10-18T20:45:40.223 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj69[0] 2023-10-18T20:45:40.223 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj74 2023-10-18T20:45:40.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj74 op=0 op_state=1 2023-10-18T20:45:40.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj74[0] 2023-10-18T20:45:40.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj76 2023-10-18T20:45:40.224 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj76 op=0 op_state=1 2023-10-18T20:45:40.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj76[0] 2023-10-18T20:45:40.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.225 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj85 2023-10-18T20:45:40.226 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj85 op=0 op_state=1 2023-10-18T20:45:40.226 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-10-18T20:45:40.226 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.226 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj85 2023-10-18T20:45:40.226 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj87 2023-10-18T20:45:40.227 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj87 op=0 op_state=1 2023-10-18T20:45:40.227 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-10-18T20:45:40.227 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.227 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj87 2023-10-18T20:45:40.228 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj92 2023-10-18T20:45:40.228 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj92 op=0 op_state=1 2023-10-18T20:45:40.228 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj92[0] 2023-10-18T20:45:40.229 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj93 2023-10-18T20:45:40.229 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj93 op=0 op_state=1 2023-10-18T20:45:40.229 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: : injecting data sync error on key=obj93 2023-10-18T20:45:40.229 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.230 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj93 2023-10-18T20:45:40.230 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj95 2023-10-18T20:45:40.230 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj95 op=0 op_state=1 2023-10-18T20:45:40.230 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.197+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj95[0] 2023-10-18T20:45:40.230 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.198+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: success 2023-10-18T20:45:40.231 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.198+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: success 2023-10-18T20:45:40.231 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.198+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: success 2023-10-18T20:45:40.231 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.200+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: success 2023-10-18T20:45:40.231 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.200+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: success 2023-10-18T20:45:40.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.200+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: success 2023-10-18T20:45:40.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.200+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.200+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.200+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.200+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.201+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: success 2023-10-18T20:45:40.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.201+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: success 2023-10-18T20:45:40.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.202+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: success 2023-10-18T20:45:40.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.202+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: success 2023-10-18T20:45:40.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.202+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: success 2023-10-18T20:45:40.234 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.202+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: success 2023-10-18T20:45:40.234 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.203+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: success 2023-10-18T20:45:40.234 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.203+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: success 2023-10-18T20:45:40.234 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.203+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: success 2023-10-18T20:45:40.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.203+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj115 2023-10-18T20:45:40.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.204+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: backing out with sync_status=-5 2023-10-18T20:45:40.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.204+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: ERROR: full sync failed. error: -5 2023-10-18T20:45:40.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.205+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.205+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: waiting on bucket lease 2023-10-18T20:45:40.252 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.250+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: status after acquiring the lock is: full 2023-10-18T20:45:40.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.261+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: listing bucket for full sync 2023-10-18T20:45:40.309 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj11 2023-10-18T20:45:40.310 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj11 op=0 op_state=1 2023-10-18T20:45:40.310 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj11[0] 2023-10-18T20:45:40.310 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj111 2023-10-18T20:45:40.310 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj111 op=0 op_state=1 2023-10-18T20:45:40.311 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-10-18T20:45:40.311 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.311 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj111 2023-10-18T20:45:40.312 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj115 2023-10-18T20:45:40.312 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj115 op=0 op_state=1 2023-10-18T20:45:40.312 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj115[0] 2023-10-18T20:45:40.312 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj116 2023-10-18T20:45:40.313 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj116 op=0 op_state=1 2023-10-18T20:45:40.313 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj116[0] 2023-10-18T20:45:40.313 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj124 2023-10-18T20:45:40.313 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj124 op=0 op_state=1 2023-10-18T20:45:40.314 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: : injecting data sync error on key=obj124 2023-10-18T20:45:40.314 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.314 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj124 2023-10-18T20:45:40.314 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj145 2023-10-18T20:45:40.314 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj145 op=0 op_state=1 2023-10-18T20:45:40.315 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj145[0] 2023-10-18T20:45:40.315 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj148 2023-10-18T20:45:40.315 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj148 op=0 op_state=1 2023-10-18T20:45:40.315 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: : injecting data sync error on key=obj148 2023-10-18T20:45:40.317 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.317 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj148 2023-10-18T20:45:40.317 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj155 2023-10-18T20:45:40.318 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj155 op=0 op_state=1 2023-10-18T20:45:40.318 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj155[0] 2023-10-18T20:45:40.318 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj17 2023-10-18T20:45:40.318 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj17 op=0 op_state=1 2023-10-18T20:45:40.319 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: : injecting data sync error on key=obj17 2023-10-18T20:45:40.319 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.319 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj17 2023-10-18T20:45:40.319 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj185 2023-10-18T20:45:40.320 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj185 op=0 op_state=1 2023-10-18T20:45:40.320 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj185[0] 2023-10-18T20:45:40.320 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.306+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj19 2023-10-18T20:45:40.320 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj19 op=0 op_state=1 2023-10-18T20:45:40.321 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj19[0] 2023-10-18T20:45:40.321 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj22 2023-10-18T20:45:40.321 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj22 op=0 op_state=1 2023-10-18T20:45:40.321 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj22[0] 2023-10-18T20:45:40.321 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj23 2023-10-18T20:45:40.322 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj23 op=0 op_state=1 2023-10-18T20:45:40.322 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj23[0] 2023-10-18T20:45:40.322 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj29 2023-10-18T20:45:40.322 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj29 op=0 op_state=1 2023-10-18T20:45:40.323 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj29[0] 2023-10-18T20:45:40.323 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj31 2023-10-18T20:45:40.323 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj31 op=0 op_state=1 2023-10-18T20:45:40.323 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj31[0] 2023-10-18T20:45:40.323 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj33 2023-10-18T20:45:40.324 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj33 op=0 op_state=1 2023-10-18T20:45:40.324 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj33[0] 2023-10-18T20:45:40.324 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj34 2023-10-18T20:45:40.324 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj34 op=0 op_state=1 2023-10-18T20:45:40.325 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj34[0] 2023-10-18T20:45:40.325 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj37 2023-10-18T20:45:40.325 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj37 op=0 op_state=1 2023-10-18T20:45:40.325 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj37[0] 2023-10-18T20:45:40.325 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj38 2023-10-18T20:45:40.326 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj38 op=0 op_state=1 2023-10-18T20:45:40.326 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj38[0] 2023-10-18T20:45:40.326 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj44 2023-10-18T20:45:40.326 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj44 op=0 op_state=1 2023-10-18T20:45:40.327 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj44[0] 2023-10-18T20:45:40.327 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj46 2023-10-18T20:45:40.327 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj46 op=0 op_state=1 2023-10-18T20:45:40.327 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.307+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj46[0] 2023-10-18T20:45:40.331 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.310+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: success 2023-10-18T20:45:40.331 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.310+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj51 2023-10-18T20:45:40.331 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.311+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj51 op=0 op_state=1 2023-10-18T20:45:40.331 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.311+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj51[0] 2023-10-18T20:45:40.332 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.311+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.332 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.311+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj55 2023-10-18T20:45:40.332 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.311+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj55 op=0 op_state=1 2023-10-18T20:45:40.332 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.311+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: : injecting data sync error on key=obj55 2023-10-18T20:45:40.333 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.311+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.333 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.311+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj55 2023-10-18T20:45:40.333 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.333 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj57 2023-10-18T20:45:40.333 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj57 op=0 op_state=1 2023-10-18T20:45:40.334 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: success 2023-10-18T20:45:40.334 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: success 2023-10-18T20:45:40.334 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: success 2023-10-18T20:45:40.334 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: success 2023-10-18T20:45:40.335 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: success 2023-10-18T20:45:40.335 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: success 2023-10-18T20:45:40.335 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: success 2023-10-18T20:45:40.335 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: success 2023-10-18T20:45:40.336 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: success 2023-10-18T20:45:40.336 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: success 2023-10-18T20:45:40.336 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: : injecting data sync error on key=obj57 2023-10-18T20:45:40.336 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.336 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj57 2023-10-18T20:45:40.337 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.337 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj58 2023-10-18T20:45:40.337 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj58 op=0 op_state=1 2023-10-18T20:45:40.337 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj58[0] 2023-10-18T20:45:40.337 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: success 2023-10-18T20:45:40.338 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: success 2023-10-18T20:45:40.338 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: success 2023-10-18T20:45:40.338 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj62 2023-10-18T20:45:40.338 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj62 op=0 op_state=1 2023-10-18T20:45:40.339 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.312+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: success 2023-10-18T20:45:40.339 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj62[0] 2023-10-18T20:45:40.339 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj68 2023-10-18T20:45:40.339 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj68 op=0 op_state=1 2023-10-18T20:45:40.339 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: success 2023-10-18T20:45:40.340 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj68[0] 2023-10-18T20:45:40.340 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj69 2023-10-18T20:45:40.340 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj69 op=0 op_state=1 2023-10-18T20:45:40.340 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj69[0] 2023-10-18T20:45:40.341 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj74 2023-10-18T20:45:40.341 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj74 op=0 op_state=1 2023-10-18T20:45:40.341 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj74[0] 2023-10-18T20:45:40.341 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj76 2023-10-18T20:45:40.342 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj76 op=0 op_state=1 2023-10-18T20:45:40.342 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj76[0] 2023-10-18T20:45:40.342 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj85 2023-10-18T20:45:40.342 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj85 op=0 op_state=1 2023-10-18T20:45:40.343 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj85[0] 2023-10-18T20:45:40.343 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj87 2023-10-18T20:45:40.343 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj87 op=0 op_state=1 2023-10-18T20:45:40.343 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj87[0] 2023-10-18T20:45:40.344 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj92 2023-10-18T20:45:40.344 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj92 op=0 op_state=1 2023-10-18T20:45:40.344 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj92[0] 2023-10-18T20:45:40.344 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj93 2023-10-18T20:45:40.344 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj93 op=0 op_state=1 2023-10-18T20:45:40.345 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj93[0] 2023-10-18T20:45:40.345 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj95 2023-10-18T20:45:40.345 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj95 op=0 op_state=1 2023-10-18T20:45:40.345 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj95[0] 2023-10-18T20:45:40.346 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.313+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.346 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.314+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.346 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.314+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: success 2023-10-18T20:45:40.346 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.316+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: success 2023-10-18T20:45:40.347 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.316+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.347 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: success 2023-10-18T20:45:40.347 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: success 2023-10-18T20:45:40.347 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: success 2023-10-18T20:45:40.347 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: success 2023-10-18T20:45:40.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: success 2023-10-18T20:45:40.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: success 2023-10-18T20:45:40.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: success 2023-10-18T20:45:40.348 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: success 2023-10-18T20:45:40.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: success 2023-10-18T20:45:40.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: success 2023-10-18T20:45:40.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: success 2023-10-18T20:45:40.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.317+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj11 2023-10-18T20:45:40.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.318+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: backing out with sync_status=-5 2023-10-18T20:45:40.350 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.318+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: ERROR: full sync failed. error: -5 2023-10-18T20:45:40.350 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.319+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.350 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.319+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: waiting on bucket lease 2023-10-18T20:45:40.365 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.364+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: status after acquiring the lock is: full 2023-10-18T20:45:40.376 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.375+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: listing bucket for full sync 2023-10-18T20:45:40.423 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj11 2023-10-18T20:45:40.423 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj11 op=0 op_state=1 2023-10-18T20:45:40.423 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj11[0] 2023-10-18T20:45:40.423 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj111 2023-10-18T20:45:40.424 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj111 op=0 op_state=1 2023-10-18T20:45:40.424 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj111[0] 2023-10-18T20:45:40.424 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj115 2023-10-18T20:45:40.424 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj115 op=0 op_state=1 2023-10-18T20:45:40.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj115[0] 2023-10-18T20:45:40.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj116 2023-10-18T20:45:40.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj116 op=0 op_state=1 2023-10-18T20:45:40.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj116[0] 2023-10-18T20:45:40.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj124 2023-10-18T20:45:40.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj124 op=0 op_state=1 2023-10-18T20:45:40.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj124[0] 2023-10-18T20:45:40.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj145 2023-10-18T20:45:40.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj145 op=0 op_state=1 2023-10-18T20:45:40.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: : injecting data sync error on key=obj145 2023-10-18T20:45:40.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj145 2023-10-18T20:45:40.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj148 2023-10-18T20:45:40.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj148 op=0 op_state=1 2023-10-18T20:45:40.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj148[0] 2023-10-18T20:45:40.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj155 2023-10-18T20:45:40.429 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj155 op=0 op_state=1 2023-10-18T20:45:40.430 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj155[0] 2023-10-18T20:45:40.430 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj17 2023-10-18T20:45:40.431 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj17 op=0 op_state=1 2023-10-18T20:45:40.431 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj17[0] 2023-10-18T20:45:40.431 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj185 2023-10-18T20:45:40.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj185 op=0 op_state=1 2023-10-18T20:45:40.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj185[0] 2023-10-18T20:45:40.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj19 2023-10-18T20:45:40.433 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj19 op=0 op_state=1 2023-10-18T20:45:40.433 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj19[0] 2023-10-18T20:45:40.433 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.420+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj22 2023-10-18T20:45:40.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj22 op=0 op_state=1 2023-10-18T20:45:40.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj22[0] 2023-10-18T20:45:40.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj23 2023-10-18T20:45:40.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj23 op=0 op_state=1 2023-10-18T20:45:40.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj23[0] 2023-10-18T20:45:40.436 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj29 2023-10-18T20:45:40.436 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj29 op=0 op_state=1 2023-10-18T20:45:40.436 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-10-18T20:45:40.437 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.437 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj29 2023-10-18T20:45:40.437 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj31 2023-10-18T20:45:40.437 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj31 op=0 op_state=1 2023-10-18T20:45:40.438 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj31[0] 2023-10-18T20:45:40.438 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj33 2023-10-18T20:45:40.438 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj33 op=0 op_state=1 2023-10-18T20:45:40.439 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj33[0] 2023-10-18T20:45:40.439 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj34 2023-10-18T20:45:40.439 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj34 op=0 op_state=1 2023-10-18T20:45:40.439 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj34[0] 2023-10-18T20:45:40.439 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj37 2023-10-18T20:45:40.440 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj37 op=0 op_state=1 2023-10-18T20:45:40.440 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj37[0] 2023-10-18T20:45:40.440 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj38 2023-10-18T20:45:40.440 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj38 op=0 op_state=1 2023-10-18T20:45:40.441 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj38[0] 2023-10-18T20:45:40.441 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj44 2023-10-18T20:45:40.441 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj44 op=0 op_state=1 2023-10-18T20:45:40.441 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj44[0] 2023-10-18T20:45:40.441 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj46 2023-10-18T20:45:40.442 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj46 op=0 op_state=1 2023-10-18T20:45:40.442 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.421+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj46[0] 2023-10-18T20:45:40.444 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.425+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: success 2023-10-18T20:45:40.444 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.425+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: success 2023-10-18T20:45:40.445 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.425+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj51 2023-10-18T20:45:40.445 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.425+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj51 op=0 op_state=1 2023-10-18T20:45:40.445 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.425+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj51[0] 2023-10-18T20:45:40.445 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.425+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj55 2023-10-18T20:45:40.446 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.425+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj55 op=0 op_state=1 2023-10-18T20:45:40.446 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.425+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj55[0] 2023-10-18T20:45:40.446 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.446 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj57 2023-10-18T20:45:40.447 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj57 op=0 op_state=1 2023-10-18T20:45:40.447 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: success 2023-10-18T20:45:40.447 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: success 2023-10-18T20:45:40.447 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: success 2023-10-18T20:45:40.447 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: success 2023-10-18T20:45:40.448 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: success 2023-10-18T20:45:40.448 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: success 2023-10-18T20:45:40.448 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: success 2023-10-18T20:45:40.448 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.426+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj57[0] 2023-10-18T20:45:40.449 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: success 2023-10-18T20:45:40.449 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj111 2023-10-18T20:45:40.449 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: success 2023-10-18T20:45:40.449 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: success 2023-10-18T20:45:40.449 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: success 2023-10-18T20:45:40.450 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: success 2023-10-18T20:45:40.450 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: success 2023-10-18T20:45:40.450 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj58 2023-10-18T20:45:40.450 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj58 op=0 op_state=1 2023-10-18T20:45:40.451 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj58[0] 2023-10-18T20:45:40.451 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj62 2023-10-18T20:45:40.451 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj62 op=0 op_state=1 2023-10-18T20:45:40.452 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj62[0] 2023-10-18T20:45:40.452 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj68 2023-10-18T20:45:40.453 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj68 op=0 op_state=1 2023-10-18T20:45:40.453 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj68[0] 2023-10-18T20:45:40.453 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj69 2023-10-18T20:45:40.454 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj69 op=0 op_state=1 2023-10-18T20:45:40.454 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj69[0] 2023-10-18T20:45:40.454 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj74 2023-10-18T20:45:40.455 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj74 op=0 op_state=1 2023-10-18T20:45:40.455 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj74[0] 2023-10-18T20:45:40.455 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj76 2023-10-18T20:45:40.456 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj76 op=0 op_state=1 2023-10-18T20:45:40.456 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: success 2023-10-18T20:45:40.456 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj76[0] 2023-10-18T20:45:40.457 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj85 2023-10-18T20:45:40.457 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj85 op=0 op_state=1 2023-10-18T20:45:40.458 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: success 2023-10-18T20:45:40.458 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-10-18T20:45:40.458 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-10-18T20:45:40.458 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 0 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: ERROR: failed to sync object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj85 2023-10-18T20:45:40.459 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.459 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj87 2023-10-18T20:45:40.459 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj87 op=0 op_state=1 2023-10-18T20:45:40.459 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj87[0] 2023-10-18T20:45:40.459 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: success 2023-10-18T20:45:40.460 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj92 2023-10-18T20:45:40.460 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj92 op=0 op_state=1 2023-10-18T20:45:40.460 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj92[0] 2023-10-18T20:45:40.460 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj93 2023-10-18T20:45:40.461 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj93 op=0 op_state=1 2023-10-18T20:45:40.461 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj93[0] 2023-10-18T20:45:40.461 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj95 2023-10-18T20:45:40.461 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj95 op=0 op_state=1 2023-10-18T20:45:40.461 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.427+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj95[0] 2023-10-18T20:45:40.462 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.428+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: success 2023-10-18T20:45:40.462 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.429+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: success 2023-10-18T20:45:40.462 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.429+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: success 2023-10-18T20:45:40.462 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.430+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: a sync operation returned error 2023-10-18T20:45:40.463 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.431+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: success 2023-10-18T20:45:40.463 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.431+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: success 2023-10-18T20:45:40.463 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.431+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: success 2023-10-18T20:45:40.463 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.431+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: success 2023-10-18T20:45:40.464 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.431+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: success 2023-10-18T20:45:40.464 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.431+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: success 2023-10-18T20:45:40.464 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.431+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: success 2023-10-18T20:45:40.464 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.431+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: success 2023-10-18T20:45:40.465 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.432+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: success 2023-10-18T20:45:40.465 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.432+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: success 2023-10-18T20:45:40.465 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.432+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: success 2023-10-18T20:45:40.466 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.432+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj124 2023-10-18T20:45:40.466 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.433+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: backing out with sync_status=-5 2023-10-18T20:45:40.466 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.433+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: ERROR: full sync failed. error: -5 2023-10-18T20:45:40.467 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.434+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.467 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.434+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: waiting on bucket lease 2023-10-18T20:45:40.480 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.479+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: status after acquiring the lock is: full 2023-10-18T20:45:40.491 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.490+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: listing bucket for full sync 2023-10-18T20:45:40.538 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj11 2023-10-18T20:45:40.539 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj11 op=0 op_state=1 2023-10-18T20:45:40.539 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj11[0] 2023-10-18T20:45:40.539 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj111 2023-10-18T20:45:40.539 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj111 op=0 op_state=1 2023-10-18T20:45:40.540 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj111[0] 2023-10-18T20:45:40.540 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj115 2023-10-18T20:45:40.540 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj115 op=0 op_state=1 2023-10-18T20:45:40.541 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj115[0] 2023-10-18T20:45:40.541 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj116 2023-10-18T20:45:40.541 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj116 op=0 op_state=1 2023-10-18T20:45:40.542 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj116[0] 2023-10-18T20:45:40.542 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj124 2023-10-18T20:45:40.543 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj124 op=0 op_state=1 2023-10-18T20:45:40.543 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj124[0] 2023-10-18T20:45:40.543 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj145 2023-10-18T20:45:40.544 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj145 op=0 op_state=1 2023-10-18T20:45:40.544 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj145[0] 2023-10-18T20:45:40.544 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj148 2023-10-18T20:45:40.545 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj148 op=0 op_state=1 2023-10-18T20:45:40.545 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj148[0] 2023-10-18T20:45:40.546 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj155 2023-10-18T20:45:40.546 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj155 op=0 op_state=1 2023-10-18T20:45:40.547 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj155[0] 2023-10-18T20:45:40.547 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj17 2023-10-18T20:45:40.547 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj17 op=0 op_state=1 2023-10-18T20:45:40.548 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj17[0] 2023-10-18T20:45:40.548 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj185 2023-10-18T20:45:40.548 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj185 op=0 op_state=1 2023-10-18T20:45:40.549 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj185[0] 2023-10-18T20:45:40.549 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj19 2023-10-18T20:45:40.550 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj19 op=0 op_state=1 2023-10-18T20:45:40.550 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj19[0] 2023-10-18T20:45:40.550 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj22 2023-10-18T20:45:40.551 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj22 op=0 op_state=1 2023-10-18T20:45:40.551 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj22[0] 2023-10-18T20:45:40.552 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj23 2023-10-18T20:45:40.552 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj23 op=0 op_state=1 2023-10-18T20:45:40.552 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj23[0] 2023-10-18T20:45:40.552 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.535+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj29 2023-10-18T20:45:40.552 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj29 op=0 op_state=1 2023-10-18T20:45:40.553 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj29[0] 2023-10-18T20:45:40.553 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj31 2023-10-18T20:45:40.553 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj31 op=0 op_state=1 2023-10-18T20:45:40.553 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj31[0] 2023-10-18T20:45:40.554 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj33 2023-10-18T20:45:40.554 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj33 op=0 op_state=1 2023-10-18T20:45:40.554 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj33[0] 2023-10-18T20:45:40.554 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj34 2023-10-18T20:45:40.555 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj34 op=0 op_state=1 2023-10-18T20:45:40.555 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj34[0] 2023-10-18T20:45:40.555 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj37 2023-10-18T20:45:40.555 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj37 op=0 op_state=1 2023-10-18T20:45:40.556 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj37[0] 2023-10-18T20:45:40.556 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj38 2023-10-18T20:45:40.556 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj38 op=0 op_state=1 2023-10-18T20:45:40.556 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj38[0] 2023-10-18T20:45:40.556 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj44 2023-10-18T20:45:40.557 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj44 op=0 op_state=1 2023-10-18T20:45:40.557 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj44[0] 2023-10-18T20:45:40.557 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj46 2023-10-18T20:45:40.557 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj46 op=0 op_state=1 2023-10-18T20:45:40.558 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.536+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj46[0] 2023-10-18T20:45:40.569 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.539+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj33]: success 2023-10-18T20:45:40.569 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.539+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj51 2023-10-18T20:45:40.569 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.539+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj51 op=0 op_state=1 2023-10-18T20:45:40.570 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.539+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj51[0] 2023-10-18T20:45:40.570 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.539+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj11]: success 2023-10-18T20:45:40.571 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.539+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj55 2023-10-18T20:45:40.571 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.539+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj55 op=0 op_state=1 2023-10-18T20:45:40.571 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.539+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj55[0] 2023-10-18T20:45:40.572 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.540+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj111]: success 2023-10-18T20:45:40.572 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.540+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj115]: success 2023-10-18T20:45:40.573 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.540+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj57 2023-10-18T20:45:40.573 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.540+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj57 op=0 op_state=1 2023-10-18T20:45:40.573 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.540+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj57[0] 2023-10-18T20:45:40.574 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.540+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj58 2023-10-18T20:45:40.574 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.540+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj58 op=0 op_state=1 2023-10-18T20:45:40.575 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.540+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj58[0] 2023-10-18T20:45:40.575 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj31]: success 2023-10-18T20:45:40.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj62 2023-10-18T20:45:40.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj62 op=0 op_state=1 2023-10-18T20:45:40.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj62[0] 2023-10-18T20:45:40.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj46]: success 2023-10-18T20:45:40.579 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj68 2023-10-18T20:45:40.579 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj68 op=0 op_state=1 2023-10-18T20:45:40.580 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj38]: success 2023-10-18T20:45:40.580 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj68[0] 2023-10-18T20:45:40.580 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj185]: success 2023-10-18T20:45:40.581 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj69 2023-10-18T20:45:40.581 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj69 op=0 op_state=1 2023-10-18T20:45:40.582 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj69[0] 2023-10-18T20:45:40.582 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj74 2023-10-18T20:45:40.582 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj74 op=0 op_state=1 2023-10-18T20:45:40.583 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj74[0] 2023-10-18T20:45:40.583 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj124]: success 2023-10-18T20:45:40.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj23]: success 2023-10-18T20:45:40.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj76 2023-10-18T20:45:40.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj76 op=0 op_state=1 2023-10-18T20:45:40.585 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj34]: success 2023-10-18T20:45:40.585 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj22]: success 2023-10-18T20:45:40.586 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj145]: success 2023-10-18T20:45:40.586 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj116]: success 2023-10-18T20:45:40.586 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj145 2023-10-18T20:45:40.587 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj19]: success 2023-10-18T20:45:40.587 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj76[0] 2023-10-18T20:45:40.588 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj85 2023-10-18T20:45:40.588 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj85 op=0 op_state=1 2023-10-18T20:45:40.589 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj85[0] 2023-10-18T20:45:40.589 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj87 2023-10-18T20:45:40.590 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj87 op=0 op_state=1 2023-10-18T20:45:40.590 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj87[0] 2023-10-18T20:45:40.590 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj92 2023-10-18T20:45:40.591 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj92 op=0 op_state=1 2023-10-18T20:45:40.591 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj92[0] 2023-10-18T20:45:40.592 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj93 2023-10-18T20:45:40.592 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj93 op=0 op_state=1 2023-10-18T20:45:40.593 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.541+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj93[0] 2023-10-18T20:45:40.593 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.542+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: [full sync] syncing object: bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0/obj95 2023-10-18T20:45:40.593 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.542+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b=bucket sync single entry (source_zone=403a921b-fcfc-4ab2-b434-bf29d35a51b6) b= log_entry=obj95 op=0 op_state=1 2023-10-18T20:45:40.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.542+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: bucket sync: sync obj: 403a921b-fcfc-4ab2-b434-bf29d35a51b6/:bofrnf-59[403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28])/obj95[0] 2023-10-18T20:45:40.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.544+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj29]: success 2023-10-18T20:45:40.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.544+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj44]: success 2023-10-18T20:45:40.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.544+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj155]: success 2023-10-18T20:45:40.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.544+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj37]: success 2023-10-18T20:45:40.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.544+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj17]: success 2023-10-18T20:45:40.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.544+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj148]: success 2023-10-18T20:45:40.596 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.544+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj51]: success 2023-10-18T20:45:40.597 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.544+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj55]: success 2023-10-18T20:45:40.597 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.545+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj57]: success 2023-10-18T20:45:40.597 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.545+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj57 2023-10-18T20:45:40.598 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.545+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj58]: success 2023-10-18T20:45:40.598 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.545+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj68]: success 2023-10-18T20:45:40.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.545+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj62]: success 2023-10-18T20:45:40.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.545+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj69]: success 2023-10-18T20:45:40.599 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.546+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj92]: success 2023-10-18T20:45:40.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.546+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj76]: success 2023-10-18T20:45:40.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.546+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj95]: success 2023-10-18T20:45:40.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.546+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj85]: success 2023-10-18T20:45:40.600 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.546+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj87]: success 2023-10-18T20:45:40.601 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.546+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj74]: success 2023-10-18T20:45:40.601 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.546+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj92 2023-10-18T20:45:40.601 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.546+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:entry[obj93]: success 2023-10-18T20:45:40.601 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.546+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: updating marker oid=bucket.full-sync-status.403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28 marker=obj95 2023-10-18T20:45:40.601 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.548+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: set bucket state=incremental 2023-10-18T20:45:40.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.549+0000 7f1ae13e2780 5 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:full_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: bucket status objv={r=_sB9d4MJUdK41DhXjoPqrpsf:155,w=:0} 2023-10-18T20:45:40.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.550+0000 7f1ae13e2780 10 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: WARNING: requested sync of past generation 0 < 6, returning success 2023-10-18T20:45:40.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.602 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:1]: sync status for source bucket shard: 2 2023-10-18T20:45:40.603 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.604 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:1]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:1]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.604 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: sync status for source bucket shard: 2 2023-10-18T20:45:40.604 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:2]: sync status for source bucket shard: 2 2023-10-18T20:45:40.604 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:0]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.604 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.552+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:2]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:2]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.553+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.553+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:3]: sync status for source bucket shard: 2 2023-10-18T20:45:40.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.553+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:3]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:3]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.605 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.553+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.606 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.553+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.606 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.553+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.606 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.553+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:4]: sync status for source bucket shard: 2 2023-10-18T20:45:40.606 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.553+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:4]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:4]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.607 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:9]: sync status for source bucket shard: 2 2023-10-18T20:45:40.607 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:9]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:9]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.607 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:6]: sync status for source bucket shard: 2 2023-10-18T20:45:40.607 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:6]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:6]: listing bilog for incremental sync; position=00000000002.78.6 2023-10-18T20:45:40.607 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.608 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.608 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.608 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:5]: sync status for source bucket shard: 2 2023-10-18T20:45:40.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:7]: sync status for source bucket shard: 2 2023-10-18T20:45:40.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:5]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:5]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.554+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:7]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:7]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.555+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:14]: sync status for source bucket shard: 2 2023-10-18T20:45:40.611 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:10]: sync status for source bucket shard: 2 2023-10-18T20:45:40.611 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:14]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:14]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.611 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:10]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:10]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.611 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:12]: sync status for source bucket shard: 2 2023-10-18T20:45:40.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:12]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:12]: listing bilog for incremental sync; position=00000000002.65.6 2023-10-18T20:45:40.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.612 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.613 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.613 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.613 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.556+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.613 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:11]: sync status for source bucket shard: 2 2023-10-18T20:45:40.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:11]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:11]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.614 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.615 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:17]: sync status for source bucket shard: 2 2023-10-18T20:45:40.615 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:19]: sync status for source bucket shard: 2 2023-10-18T20:45:40.615 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:18]: sync status for source bucket shard: 2 2023-10-18T20:45:40.615 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:15]: sync status for source bucket shard: 2 2023-10-18T20:45:40.616 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:17]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:17]: listing bilog for incremental sync; position=00000000002.79.6 2023-10-18T20:45:40.616 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:20]: sync status for source bucket shard: 2 2023-10-18T20:45:40.616 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:19]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:19]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.616 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:18]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:18]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.616 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.557+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:15]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:15]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.617 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:20]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:20]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.617 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:16]: sync status for source bucket shard: 2 2023-10-18T20:45:40.617 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.618 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:16]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:16]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.618 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:22]: sync status for source bucket shard: 2 2023-10-18T20:45:40.618 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:22]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:22]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.619 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.619 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.619 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.620 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:23]: sync status for source bucket shard: 2 2023-10-18T20:45:40.620 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:23]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:23]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.620 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.558+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.621 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:25]: sync status for source bucket shard: 2 2023-10-18T20:45:40.621 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.621 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:24]: sync status for source bucket shard: 2 2023-10-18T20:45:40.622 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:25]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:25]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.622 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:24]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:24]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:28]: sync status for source bucket shard: 2 2023-10-18T20:45:40.623 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:28]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:28]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.624 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:26]: sync status for source bucket shard: 2 2023-10-18T20:45:40.624 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:26]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:26]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.624 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:21]: sync status for source bucket shard: 2 2023-10-18T20:45:40.625 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:21]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:21]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.625 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:30]: sync status for source bucket shard: 2 2023-10-18T20:45:40.626 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.559+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:30]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:30]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.626 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.626 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.627 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.627 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:8]: sync status for source bucket shard: 2 2023-10-18T20:45:40.628 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-18T20:45:40.628 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:8]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:8]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.628 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:13]: sync status for source bucket shard: 2 2023-10-18T20:45:40.629 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:29]: sync status for source bucket shard: 2 2023-10-18T20:45:40.629 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:13]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:13]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.629 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.560+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:29]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:29]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.561+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:34]: sync status for source bucket shard: 2 2023-10-18T20:45:40.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.561+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:34]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:34]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:27]: sync status for source bucket shard: 2 2023-10-18T20:45:40.630 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:27]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:27]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.631 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:31]: sync status for source bucket shard: 2 2023-10-18T20:45:40.631 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:31]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:31]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.631 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:32]: sync status for source bucket shard: 2 2023-10-18T20:45:40.631 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:35]: sync status for source bucket shard: 2 2023-10-18T20:45:40.632 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:36]: sync status for source bucket shard: 2 2023-10-18T20:45:40.632 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:33]: sync status for source bucket shard: 2 2023-10-18T20:45:40.632 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:32]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:32]: listing bilog for incremental sync; position=00000000001.83.6 2023-10-18T20:45:40.632 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:35]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:35]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.632 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:36]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:36]: listing bilog for incremental sync; position=00000000001.100.6 2023-10-18T20:45:40.633 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:40.563+0000 7f1ae13e2780 20 RGW-SYNC:bucket[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28<-bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:33]:inc_sync[bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28:33]: listing bilog for incremental sync; position= 2023-10-18T20:45:40.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:40.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:40.635 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:45:40.726 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:40.727 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:40.760 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:45:40.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.761 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj11", 2023-10-18T20:45:40.761 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.762 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.762 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.763 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 14 2023-10-18T20:45:40.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.764 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.764 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.764 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.765 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.765 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:40.766 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.624897Z", 2023-10-18T20:45:40.766 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.766 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.767 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.767 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.768 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.768 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.768 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.769 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.770 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.1140052159850751836", 2023-10-18T20:45:40.770 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.770 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.771 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.772 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj111", 2023-10-18T20:45:40.772 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.773 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.773 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.774 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 19 2023-10-18T20:45:40.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.774 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.775 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.775 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.776 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.776 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:40.776 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.726998Z", 2023-10-18T20:45:40.776 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.777 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.777 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.777 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.777 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.777 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.778 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.778 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.778 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.9071684867073283167", 2023-10-18T20:45:40.778 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.779 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.779 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.780 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj115", 2023-10-18T20:45:40.780 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.780 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.780 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.780 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:45:40.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.781 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.781 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.781 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.781 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.782 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:40.782 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.234025Z", 2023-10-18T20:45:40.782 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.782 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.782 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.783 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.783 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.783 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.783 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.783 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.784 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.10201403020666027805", 2023-10-18T20:45:40.784 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.784 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.785 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.785 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj116", 2023-10-18T20:45:40.785 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.786 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.786 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.786 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:45:40.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.786 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.787 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.787 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.787 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.787 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.787 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.992319Z", 2023-10-18T20:45:40.788 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.788 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.788 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.788 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.788 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.789 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.789 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.789 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.790 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.6534745884138597207", 2023-10-18T20:45:40.790 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.790 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.790 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.791 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj124", 2023-10-18T20:45:40.791 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.791 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.791 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.792 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:40.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.792 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.792 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.792 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.793 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.793 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.793 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.828946Z", 2023-10-18T20:45:40.793 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.793 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.794 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.794 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.794 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.794 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.795 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.795 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.795 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.1944940932513142517", 2023-10-18T20:45:40.795 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.796 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.796 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.796 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj145", 2023-10-18T20:45:40.797 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.797 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.797 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.797 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:45:40.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.798 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.798 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.798 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.798 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.798 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.799 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:47.012959Z", 2023-10-18T20:45:40.799 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.799 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.799 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.799 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.800 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.800 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.800 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.800 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.801 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.14453362137992110074", 2023-10-18T20:45:40.801 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.801 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.801 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.802 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj148", 2023-10-18T20:45:40.802 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.802 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.802 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.803 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:40.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.803 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.803 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.804 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.804 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.804 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:40.804 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.746370Z", 2023-10-18T20:45:40.804 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.805 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.805 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.805 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.805 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.805 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.806 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.806 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.806 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.15302776800647144240", 2023-10-18T20:45:40.806 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.807 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.807 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.807 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj155", 2023-10-18T20:45:40.808 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.808 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.808 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.808 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:40.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.809 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.809 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.809 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.809 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.810 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:40.810 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.890428Z", 2023-10-18T20:45:40.810 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.810 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.810 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.811 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.811 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.811 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.811 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.811 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.812 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.14571522328834391442", 2023-10-18T20:45:40.812 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.812 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.812 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.813 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj17", 2023-10-18T20:45:40.813 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.813 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.814 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.814 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 13 2023-10-18T20:45:40.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.814 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.815 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.815 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.815 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.815 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:40.815 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.330763Z", 2023-10-18T20:45:40.816 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.816 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.816 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.816 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.816 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.817 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.817 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.817 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.817 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.8389117368031193670", 2023-10-18T20:45:40.818 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.818 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.818 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.819 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj185", 2023-10-18T20:45:40.819 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.819 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.819 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.819 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:45:40.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.820 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.820 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.820 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.820 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.821 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:40.821 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.802901Z", 2023-10-18T20:45:40.821 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.821 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.821 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.822 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.822 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.822 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.822 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.822 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.823 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.1025405863869920162", 2023-10-18T20:45:40.823 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.823 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.823 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.824 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj19", 2023-10-18T20:45:40.824 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.825 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.825 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.825 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 14 2023-10-18T20:45:40.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.825 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.826 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.826 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.826 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.826 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:40.826 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.373675Z", 2023-10-18T20:45:40.827 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.827 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.827 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.827 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.827 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.828 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.828 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.828 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.828 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.10703867280330573910", 2023-10-18T20:45:40.829 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.829 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.829 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.830 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj22", 2023-10-18T20:45:40.830 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.830 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.830 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.830 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:45:40.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.831 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.831 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.831 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.831 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.832 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.832 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.645343Z", 2023-10-18T20:45:40.832 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.832 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.832 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.833 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.833 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.833 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.833 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.834 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.834 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.5407524075267606856", 2023-10-18T20:45:40.834 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.834 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.835 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.835 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj23", 2023-10-18T20:45:40.835 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.836 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.836 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.836 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:45:40.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.836 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.837 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.837 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.837 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.837 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:40.837 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.071727Z", 2023-10-18T20:45:40.838 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.838 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.838 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.838 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.838 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.839 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.839 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.839 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.839 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.8918295964245086718", 2023-10-18T20:45:40.840 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.840 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.840 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.841 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj29", 2023-10-18T20:45:40.841 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.841 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.841 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.842 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 17 2023-10-18T20:45:40.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.842 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.842 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.842 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.843 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.843 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:40.843 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.893544Z", 2023-10-18T20:45:40.843 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.843 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.844 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.844 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.844 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.844 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.844 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.845 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.845 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.16041805063037676825", 2023-10-18T20:45:40.845 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.845 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.846 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.846 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj31", 2023-10-18T20:45:40.846 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.847 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.847 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.847 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:45:40.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.847 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.848 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.848 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.848 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.848 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.848 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.732307Z", 2023-10-18T20:45:40.849 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.849 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.849 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.849 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.849 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.850 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.850 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.850 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.851 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.2484906775693184291", 2023-10-18T20:45:40.851 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.851 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.851 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.852 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj33", 2023-10-18T20:45:40.852 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.852 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.852 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.853 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:45:40.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.853 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.853 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.853 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.854 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.854 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:40.854 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.664873Z", 2023-10-18T20:45:40.854 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.854 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.855 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.855 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.855 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.855 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.855 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.856 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.856 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.7872964521715796930", 2023-10-18T20:45:40.856 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.856 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.857 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.857 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj34", 2023-10-18T20:45:40.857 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.858 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.858 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.858 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 18 2023-10-18T20:45:40.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.859 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.859 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.859 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.859 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.859 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:40.860 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.349351Z", 2023-10-18T20:45:40.860 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.860 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.860 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.860 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.861 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.861 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.861 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.861 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.862 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.3256280687761285531", 2023-10-18T20:45:40.862 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.862 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.862 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.863 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj37", 2023-10-18T20:45:40.863 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.863 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.864 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.864 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12 2023-10-18T20:45:40.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.864 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.864 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.865 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.865 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.865 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:40.865 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.648488Z", 2023-10-18T20:45:40.865 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.866 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.866 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.866 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.866 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.866 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.867 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.867 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.867 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.7973715117512088837", 2023-10-18T20:45:40.867 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.868 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.868 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.868 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj38", 2023-10-18T20:45:40.869 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.869 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.869 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.869 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:45:40.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.870 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.870 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.870 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.870 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.870 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:40.871 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.394123Z", 2023-10-18T20:45:40.871 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.871 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.871 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.871 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.872 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.872 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.872 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.872 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.873 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.14057616850333212268", 2023-10-18T20:45:40.873 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.873 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.873 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.874 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj44", 2023-10-18T20:45:40.874 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.874 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.875 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.875 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 20 2023-10-18T20:45:40.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.875 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.875 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.876 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.876 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.876 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.876 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.692260Z", 2023-10-18T20:45:40.876 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.877 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.877 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.877 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.877 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.877 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.878 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.878 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.878 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.827730044899490068", 2023-10-18T20:45:40.878 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.879 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.879 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.879 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj46", 2023-10-18T20:45:40.880 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.880 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.880 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.880 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12 2023-10-18T20:45:40.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.881 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.881 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.881 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.881 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.882 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:40.882 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.132203Z", 2023-10-18T20:45:40.882 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.882 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.882 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.883 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.883 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.883 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.883 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.884 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.884 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.11063890755539523781", 2023-10-18T20:45:40.884 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.884 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.885 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.885 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj51", 2023-10-18T20:45:40.885 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.886 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.886 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.886 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 15 2023-10-18T20:45:40.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.887 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.887 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.887 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.887 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.887 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:40.888 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.411923Z", 2023-10-18T20:45:40.888 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.888 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.888 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.888 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.889 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.889 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.889 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.889 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.890 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.9238979778088210590", 2023-10-18T20:45:40.890 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.890 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.890 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.891 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj55", 2023-10-18T20:45:40.891 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.891 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.892 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.892 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10 2023-10-18T20:45:40.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.892 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.892 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.893 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.893 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.893 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:40.893 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.750824Z", 2023-10-18T20:45:40.894 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.894 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.894 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.894 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.894 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.895 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.895 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.895 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.895 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.9777162968257113510", 2023-10-18T20:45:40.896 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.896 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.896 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.897 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj57", 2023-10-18T20:45:40.897 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.897 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.897 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.898 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:45:40.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.898 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.898 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.898 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.899 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.899 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.899 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.453381Z", 2023-10-18T20:45:40.899 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.899 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.900 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.900 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.900 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.900 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.900 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.901 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.901 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.16238409732404571472", 2023-10-18T20:45:40.901 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.901 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.902 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.902 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj58", 2023-10-18T20:45:40.903 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.903 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.903 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.903 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 1 2023-10-18T20:45:40.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.904 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.904 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.904 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.904 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.904 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.905 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.913247Z", 2023-10-18T20:45:40.905 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.905 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.905 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.905 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.906 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.906 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.906 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.906 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.907 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.37819411435056022", 2023-10-18T20:45:40.907 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.907 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.907 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.908 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj62", 2023-10-18T20:45:40.908 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.908 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.909 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.909 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:45:40.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.909 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.909 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.910 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.910 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.910 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.910 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.751922Z", 2023-10-18T20:45:40.910 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.911 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.911 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.911 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.911 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.911 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.912 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.912 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.912 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.1870861414017055515", 2023-10-18T20:45:40.912 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.913 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.913 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.913 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj68", 2023-10-18T20:45:40.914 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.914 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.914 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.914 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:40.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.915 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.915 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.915 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.915 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.916 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:40.916 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.431067Z", 2023-10-18T20:45:40.916 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.916 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.916 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.917 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.917 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.917 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.917 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.917 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.918 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.2707213890133407322", 2023-10-18T20:45:40.918 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.918 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.918 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.919 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj69", 2023-10-18T20:45:40.919 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.919 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.920 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.920 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 15 2023-10-18T20:45:40.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.920 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.920 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.921 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.921 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.921 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.921 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.150793Z", 2023-10-18T20:45:40.921 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.922 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.922 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.922 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.922 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.923 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.923 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.923 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.923 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.14678997603254626793", 2023-10-18T20:45:40.924 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.924 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.924 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.925 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj74", 2023-10-18T20:45:40.925 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.925 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.925 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.925 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:40.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.926 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.926 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.926 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.926 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.927 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.927 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.667632Z", 2023-10-18T20:45:40.927 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.927 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.928 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.928 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.928 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.928 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.929 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.929 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.929 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.5737774367969283043", 2023-10-18T20:45:40.929 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.930 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.930 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.931 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj76", 2023-10-18T20:45:40.931 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.931 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.931 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.931 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 2 2023-10-18T20:45:40.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.932 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.932 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.932 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.932 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.933 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:40.933 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.470726Z", 2023-10-18T20:45:40.933 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.933 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.933 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.934 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.934 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.934 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.934 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.934 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.935 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.2125585296907349202", 2023-10-18T20:45:40.935 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.935 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.935 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.936 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj85", 2023-10-18T20:45:40.936 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.936 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.937 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.938 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:40.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.938 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.939 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.939 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.939 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.939 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:40.939 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.491392Z", 2023-10-18T20:45:40.940 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.940 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.940 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.940 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.940 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.941 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.941 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.941 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.941 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.398036110144781827", 2023-10-18T20:45:40.942 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.942 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.942 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.943 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj87", 2023-10-18T20:45:40.943 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.943 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.943 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.944 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 16 2023-10-18T20:45:40.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.944 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.944 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.944 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.945 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.945 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:40.945 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.932864Z", 2023-10-18T20:45:40.945 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.945 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.946 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.946 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.946 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.946 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.946 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.947 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.947 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.1720486137229746986", 2023-10-18T20:45:40.947 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.947 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.948 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.948 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj92", 2023-10-18T20:45:40.948 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.949 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.949 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.949 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12 2023-10-18T20:45:40.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.950 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.950 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.950 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.950 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.950 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.951 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.214384Z", 2023-10-18T20:45:40.951 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.951 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.951 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.951 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.952 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.952 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.952 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.952 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.953 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.10893079484332643427", 2023-10-18T20:45:40.953 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.953 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.953 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.954 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj93", 2023-10-18T20:45:40.954 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.954 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.954 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.955 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:40.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.955 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.955 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.955 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.956 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.956 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.956 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.810194Z", 2023-10-18T20:45:40.956 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.956 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.957 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.957 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.957 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.957 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.958 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.958 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.958 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.15865523816116590937", 2023-10-18T20:45:40.958 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.959 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.959 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:40.959 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj95", 2023-10-18T20:45:40.960 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:40.960 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:40.960 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 9, 2023-10-18T20:45:40.960 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 3 2023-10-18T20:45:40.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.961 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:40.961 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:40.961 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:40.961 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:40.961 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:40.962 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.530014Z", 2023-10-18T20:45:40.962 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:40.962 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:40.962 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:40.963 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:40.963 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:40.963 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:40.963 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:40.963 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:40.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:40.964 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.4023772009205146513", 2023-10-18T20:45:40.964 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:40.964 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:40.965 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:40.965 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:40.965 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:45:40.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:40.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:40.967 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:45:41.021 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:41.021 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:41.054 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:45:41.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.054 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj11", 2023-10-18T20:45:41.055 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.055 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.055 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.055 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10 2023-10-18T20:45:41.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.056 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.056 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.056 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.056 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.057 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.057 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.624897Z", 2023-10-18T20:45:41.057 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.057 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.057 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.058 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.058 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.058 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.058 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.058 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.059 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_rO1AZIC3EmpqZd_5qoVljmHP9I1y0mw", 2023-10-18T20:45:41.059 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.059 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.060 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.060 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj111", 2023-10-18T20:45:41.060 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.061 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.061 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.061 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:45:41.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.061 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.062 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.062 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.062 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.062 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:41.062 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.726998Z", 2023-10-18T20:45:41.063 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.063 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.063 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.063 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.063 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.064 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.064 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.064 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_Fc7n6MGqAM9mL0kyuASnf2NFx_c4zUF", 2023-10-18T20:45:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.066 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj115", 2023-10-18T20:45:41.066 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.066 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.066 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:45:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.234025Z", 2023-10-18T20:45:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.070 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.070 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_CIoysR5fhfxowLl7VhhFpsUvMbBQ2B0", 2023-10-18T20:45:41.070 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj116", 2023-10-18T20:45:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:45:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.074 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.992319Z", 2023-10-18T20:45:41.074 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.074 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.074 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_e6COii4wwpaGmuXiVHhHUw0l1xmom7o", 2023-10-18T20:45:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.077 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj124", 2023-10-18T20:45:41.077 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.077 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.078 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.078 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:45:41.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.078 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.078 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.079 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.079 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.079 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:41.079 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.828946Z", 2023-10-18T20:45:41.080 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.080 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.080 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.080 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.080 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.081 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.081 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.081 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.081 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_9O9qm_akU4D38XcI7P-YZLXdAEQmRvq", 2023-10-18T20:45:41.082 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.082 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.082 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.083 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj145", 2023-10-18T20:45:41.083 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.083 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.083 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.083 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:41.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.084 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.084 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.084 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.085 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.085 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:41.085 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:47.012959Z", 2023-10-18T20:45:41.085 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.085 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.086 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.086 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.086 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.086 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.086 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.087 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.087 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_bVATeo_j2PPVLW-bt-mM8oLf32K9Gil", 2023-10-18T20:45:41.087 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.088 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.088 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.088 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj148", 2023-10-18T20:45:41.089 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.089 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.089 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.089 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:41.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.090 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.090 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.090 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.090 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.090 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.091 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.746370Z", 2023-10-18T20:45:41.091 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.091 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.091 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.091 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_2IXHH1YdQkZrElrp78nItjguupY4gvW", 2023-10-18T20:45:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj155", 2023-10-18T20:45:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.890428Z", 2023-10-18T20:45:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_KB9pJa31HLVmiNXX9O6GTylduvgw-15", 2023-10-18T20:45:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj17", 2023-10-18T20:45:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:45:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.102 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:41.102 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.330763Z", 2023-10-18T20:45:41.102 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.102 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_XWq-Q0D-8FnEEcb1yYk8rFpTsHo5NXl", 2023-10-18T20:45:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj185", 2023-10-18T20:45:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:45:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.107 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.107 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.107 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.107 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.107 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.802901Z", 2023-10-18T20:45:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_rTq3pRwN4cqfC-XxUfGOdmXqABpPA6c", 2023-10-18T20:45:41.110 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.110 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.110 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.111 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj19", 2023-10-18T20:45:41.111 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.111 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.111 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.112 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10 2023-10-18T20:45:41.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.112 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.112 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.112 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.113 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.113 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:41.113 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.373675Z", 2023-10-18T20:45:41.113 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.113 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.114 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.114 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.114 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.114 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.115 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.115 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.115 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_SMIHqHDAEyb3ZCb-kbO_aXDBcRy7clh", 2023-10-18T20:45:41.115 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.116 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.116 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.116 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj22", 2023-10-18T20:45:41.117 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.117 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.117 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.117 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:41.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.118 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.118 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.118 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.118 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.119 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.119 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.645343Z", 2023-10-18T20:45:41.119 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.119 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.119 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.120 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.120 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.120 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.120 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.120 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.121 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_VTVAQdeGrBLwzky9okq7yQf9Egzes1J", 2023-10-18T20:45:41.121 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.121 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.121 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.122 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj23", 2023-10-18T20:45:41.122 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.122 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.123 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.123 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 13 2023-10-18T20:45:41.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.123 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.124 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.124 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.124 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.124 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.124 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.071727Z", 2023-10-18T20:45:41.125 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.125 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.125 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.125 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.125 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.126 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.126 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.126 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.126 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_4-JC9QcwuuOG4_ByzgAoy-IVhDmf6jF", 2023-10-18T20:45:41.127 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.127 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.127 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.128 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj29", 2023-10-18T20:45:41.128 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.128 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.128 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.129 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:45:41.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.129 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.129 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.129 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.130 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.130 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.130 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.893544Z", 2023-10-18T20:45:41.130 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.130 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.131 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.131 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.131 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.131 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.131 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.132 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.132 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_vhB1Uq7yDXslP07nXzCGLkGbNRQ8R-x", 2023-10-18T20:45:41.132 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.132 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.133 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.133 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj31", 2023-10-18T20:45:41.134 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.134 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.134 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.134 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 4 2023-10-18T20:45:41.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.135 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.135 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.135 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.135 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.135 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:41.136 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.732307Z", 2023-10-18T20:45:41.136 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.136 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.136 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.136 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.137 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.137 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.137 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.137 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.138 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_HHqOJ2MoQG-hIpVCvX76zDRBxeDT-W4", 2023-10-18T20:45:41.138 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.138 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.138 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.139 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj33", 2023-10-18T20:45:41.139 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.139 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.139 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.140 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 10 2023-10-18T20:45:41.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.140 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.140 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.140 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.141 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.141 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:41.141 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.664873Z", 2023-10-18T20:45:41.141 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.141 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.142 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.142 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.142 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.142 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.142 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.143 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.143 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_SvYpbm7668yfql6nF83wuMb2cIJt_LW", 2023-10-18T20:45:41.143 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.143 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.144 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.144 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj34", 2023-10-18T20:45:41.145 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.145 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.145 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.145 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12 2023-10-18T20:45:41.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.146 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.146 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.146 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.146 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.146 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.147 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.349351Z", 2023-10-18T20:45:41.147 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.147 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.147 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.147 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.148 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.148 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.148 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.148 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.149 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_uNkepYeXUBJ5jvM9BYNbFyJ6n4yREtq", 2023-10-18T20:45:41.149 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.149 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.149 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.150 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj37", 2023-10-18T20:45:41.150 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.150 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.151 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.151 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:45:41.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.151 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.151 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.152 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.152 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.152 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.152 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.648488Z", 2023-10-18T20:45:41.152 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.153 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.153 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.153 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.153 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.153 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.154 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.154 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.154 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_NcN_QtEsRUkDOAVqcrWKpNDP4vcihEp", 2023-10-18T20:45:41.154 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.155 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.155 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.155 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj38", 2023-10-18T20:45:41.156 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.156 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.156 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.156 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:45:41.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.157 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.157 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.157 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.157 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.158 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:41.158 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.394123Z", 2023-10-18T20:45:41.158 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.158 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.158 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.159 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.159 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.159 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.159 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.159 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.160 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_W4u-u97nFN0b0JXIQVjbywDYkAj7y3n", 2023-10-18T20:45:41.160 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.160 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.160 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.161 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj44", 2023-10-18T20:45:41.161 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.161 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.162 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.162 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 15 2023-10-18T20:45:41.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.162 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.163 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.163 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.163 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.163 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:41.163 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.692260Z", 2023-10-18T20:45:41.164 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.164 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.164 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.164 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.164 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.165 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.165 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.165 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.165 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_osorMu1bad2EFZADOmnVUOYQu0nw-dQ", 2023-10-18T20:45:41.166 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.166 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.167 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj46", 2023-10-18T20:45:41.167 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.167 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.167 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.168 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:45:41.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.168 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.168 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.169 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.169 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.169 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 3, 2023-10-18T20:45:41.169 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.132203Z", 2023-10-18T20:45:41.169 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.170 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.170 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.170 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.170 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.171 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.171 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.171 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.171 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_hZyRV9EIx8WnpORXeULFhLKNI5TWr_o", 2023-10-18T20:45:41.172 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.172 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.172 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.173 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj51", 2023-10-18T20:45:41.173 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.173 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.173 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.174 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 11 2023-10-18T20:45:41.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.174 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.174 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.174 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.175 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.175 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:41.175 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.411923Z", 2023-10-18T20:45:41.175 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.175 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.176 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.176 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.176 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.176 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.177 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.177 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.177 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_RQwvTUyXAWv-ee0HYdYfU6-K9T3Xtbd", 2023-10-18T20:45:41.177 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.178 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.178 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.178 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj55", 2023-10-18T20:45:41.179 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.179 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.179 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.179 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:45:41.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.180 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.180 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.180 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.180 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.181 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.181 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:32:33.750824Z", 2023-10-18T20:45:41.181 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.181 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.181 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.182 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.182 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.182 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.182 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.182 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.183 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_l_6n_EBdiH-zF40f_CCWVIflgFtFScF", 2023-10-18T20:45:41.183 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.183 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.183 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.184 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj57", 2023-10-18T20:45:41.184 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.185 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.185 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.185 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 19 2023-10-18T20:45:41.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.185 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.186 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.186 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.186 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.186 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:41.187 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.453381Z", 2023-10-18T20:45:41.187 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.187 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.187 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.187 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.188 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.188 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.188 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.188 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.189 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_NM0Nhri_a1ukctx68Zr8lQq24k9dJvd", 2023-10-18T20:45:41.189 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.189 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.189 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.190 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj58", 2023-10-18T20:45:41.190 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.190 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.190 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.191 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:41.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.191 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.193 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.193 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.193 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.194 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:41.194 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.913247Z", 2023-10-18T20:45:41.194 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.194 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.195 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.195 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.195 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.195 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.195 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.196 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.196 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_VKHOTDr7SUVaNpnDb0esy29jW8UAsTF", 2023-10-18T20:45:41.196 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.197 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.197 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.197 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj62", 2023-10-18T20:45:41.198 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.198 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.198 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.198 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:41.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.199 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.199 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.199 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.199 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.199 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:41.200 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.751922Z", 2023-10-18T20:45:41.200 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.200 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.200 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.200 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.201 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.201 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.201 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.201 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.202 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_yuZdQjX7HNX91SrJN5dpgT8BrO7eLw3", 2023-10-18T20:45:41.202 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.202 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.202 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.203 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj68", 2023-10-18T20:45:41.203 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.203 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.204 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.204 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:41.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.204 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.204 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.205 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.205 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.205 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.205 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.431067Z", 2023-10-18T20:45:41.205 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.206 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.206 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.206 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.206 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.206 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_ylRYHBZTlCrayhrb44C35e1vK7m_BE4", 2023-10-18T20:45:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.208 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.208 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj69", 2023-10-18T20:45:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 12 2023-10-18T20:45:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.150793Z", 2023-10-18T20:45:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_az2Y2n-5PMOhzECGCYDWawLHio2Vikf", 2023-10-18T20:45:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj74", 2023-10-18T20:45:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 5 2023-10-18T20:45:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:43:09.667632Z", 2023-10-18T20:45:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.218 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.218 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.218 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_5YFLNl6J7bvgLUJ_a64t1GjT_OKtDQf", 2023-10-18T20:45:41.219 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.219 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.219 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.219 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj76", 2023-10-18T20:45:41.220 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.220 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.220 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.220 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 6 2023-10-18T20:45:41.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.221 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.221 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.221 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.221 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.222 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 5, 2023-10-18T20:45:41.222 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.470726Z", 2023-10-18T20:45:41.222 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.222 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.222 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.223 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.223 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.223 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.223 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.224 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.224 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_TS-zuH9h-fk3rUE47khlq1Ivu0z8OKq", 2023-10-18T20:45:41.224 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.224 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.225 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.225 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj85", 2023-10-18T20:45:41.225 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.226 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.226 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.226 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 8 2023-10-18T20:45:41.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.226 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.227 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.227 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.227 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.227 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.227 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:33:05.491392Z", 2023-10-18T20:45:41.228 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.228 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.228 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.228 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.228 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.229 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.229 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.229 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.229 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_OV4laIUEA18koOqVJQyMMD8CME7YyD3", 2023-10-18T20:45:41.230 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.230 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.230 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.231 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj87", 2023-10-18T20:45:41.231 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.231 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.231 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.232 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:45:41.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.232 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.233 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.234 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.234 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.234 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.234 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:39:46.932864Z", 2023-10-18T20:45:41.235 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.235 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.235 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.235 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.235 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.236 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.236 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.236 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.236 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_RBBqcgKtAA5UwLepybeSaxThezY3l04", 2023-10-18T20:45:41.237 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.237 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.237 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.238 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj92", 2023-10-18T20:45:41.238 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.238 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.238 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.239 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 9 2023-10-18T20:45:41.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.239 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.239 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.239 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.240 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.240 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:41.240 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:38:22.214384Z", 2023-10-18T20:45:41.240 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.240 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.241 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.241 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.241 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.241 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.241 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.242 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.242 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_ULHAxGdZduREfzsAUrdhiZC-RZ-y7pL", 2023-10-18T20:45:41.242 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.242 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.243 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.243 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj93", 2023-10-18T20:45:41.243 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.244 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.244 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.244 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 17 2023-10-18T20:45:41.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.244 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.245 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.245 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.245 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.245 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 16, 2023-10-18T20:45:41.245 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:41:07.810194Z", 2023-10-18T20:45:41.246 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.246 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.246 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.246 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.247 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.247 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.247 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.247 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.248 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_EW2zBc6iY-yVk8kXKncyupFv2SoHNLJ", 2023-10-18T20:45:41.248 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.248 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.248 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:41.249 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "obj95", 2023-10-18T20:45:41.249 INFO:teuthology.orchestra.run.smithi082.stdout: "instance": "", 2023-10-18T20:45:41.249 INFO:teuthology.orchestra.run.smithi082.stdout: "ver": { 2023-10-18T20:45:41.249 INFO:teuthology.orchestra.run.smithi082.stdout: "pool": 13, 2023-10-18T20:45:41.250 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 7 2023-10-18T20:45:41.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.250 INFO:teuthology.orchestra.run.smithi082.stdout: "locator": "", 2023-10-18T20:45:41.250 INFO:teuthology.orchestra.run.smithi082.stdout: "exists": true, 2023-10-18T20:45:41.250 INFO:teuthology.orchestra.run.smithi082.stdout: "meta": { 2023-10-18T20:45:41.251 INFO:teuthology.orchestra.run.smithi082.stdout: "category": 1, 2023-10-18T20:45:41.251 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 6, 2023-10-18T20:45:41.251 INFO:teuthology.orchestra.run.smithi082.stdout: "mtime": "2023-10-18T20:34:23.530014Z", 2023-10-18T20:45:41.251 INFO:teuthology.orchestra.run.smithi082.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-18T20:45:41.251 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_class": "", 2023-10-18T20:45:41.252 INFO:teuthology.orchestra.run.smithi082.stdout: "owner": "rgw-multisite-test-user", 2023-10-18T20:45:41.252 INFO:teuthology.orchestra.run.smithi082.stdout: "owner_display_name": "Multisite Test User", 2023-10-18T20:45:41.252 INFO:teuthology.orchestra.run.smithi082.stdout: "content_type": "application/octet-stream", 2023-10-18T20:45:41.252 INFO:teuthology.orchestra.run.smithi082.stdout: "accounted_size": 3, 2023-10-18T20:45:41.253 INFO:teuthology.orchestra.run.smithi082.stdout: "user_data": "", 2023-10-18T20:45:41.253 INFO:teuthology.orchestra.run.smithi082.stdout: "appendable": false 2023-10-18T20:45:41.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:41.253 INFO:teuthology.orchestra.run.smithi082.stdout: "tag": "_FfWP3PzcxiSOWHWHqPV2355Ew3cQCYo", 2023-10-18T20:45:41.253 INFO:teuthology.orchestra.run.smithi082.stdout: "flags": 0, 2023-10-18T20:45:41.254 INFO:teuthology.orchestra.run.smithi082.stdout: "pending_map": [], 2023-10-18T20:45:41.254 INFO:teuthology.orchestra.run.smithi082.stdout: "versioned_epoch": 0 2023-10-18T20:45:41.254 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:41.254 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:45:41.255 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-18T20:45:41.256 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T20:45:41.258 INFO:tasks.rgw.c1.client.1:Started 2023-10-18T20:45:41.259 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T20:45:41.282 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:45:41.282 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:45:41.282 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T20:45:42.283 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:42.306 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:45:42.306 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:45:42.307 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T20:45:42.308 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T20:45:42.308 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-59 2023-10-18T20:45:42.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:45:42.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:45:42.310 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:45:42.404 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:42.405 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:42.444 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:42.443+0000 7f7006d24780 1 bucket sync caught up with source: 2023-10-18T20:45:42.444 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:45:42.444 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.78.6, , , , , , 00000000002.65.6, , , , , 00000000002.79.6, , , , , , , , , , , , , , , 00000000001.83.6, , , , 00000000001.100.6] 2023-10-18T20:45:42.445 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:42.443+0000 7f7006d24780 0 bucket checkpoint complete 2023-10-18T20:45:42.457 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-59 2023-10-18T20:45:42.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:45:42.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:45:42.459 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:45:42.515 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:42.515 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:42.550 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:42.549+0000 7f6d61fa9780 1 bucket sync caught up with source: 2023-10-18T20:45:42.551 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, 00000000009.825.6, 00000000006.1194.6, 00000000006.1274.6, 00000000015.736.6, 00000000012.823.6, 00000000003.710.6, 00000000003.873.6, 00000000012.310.6, 00000000021.498.6, 00000000018.314.6] 2023-10-18T20:45:42.551 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, 00000000009.825.6, 00000000006.1194.6, 00000000006.1274.6, 00000000015.736.6, 00000000012.823.6, 00000000003.710.6, 00000000003.873.6, 00000000012.310.6, 00000000021.498.6, 00000000018.314.6] 2023-10-18T20:45:42.551 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:42.549+0000 7f6d61fa9780 0 bucket checkpoint complete 2023-10-18T20:45:42.564 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:45:42.564 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:42.565 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:42.565 DEBUG:boto:Method: HEAD 2023-10-18T20:45:42.566 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:45:42.566 DEBUG:boto:Data: 2023-10-18T20:45:42.566 DEBUG:boto:Headers: {} 2023-10-18T20:45:42.567 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:42.567 DEBUG:boto:Port: 8001 2023-10-18T20:45:42.568 DEBUG:boto:Params: {} 2023-10-18T20:45:42.568 DEBUG:boto:Token: None 2023-10-18T20:45:42.569 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:42 GMT /bofrnf-59/ 2023-10-18T20:45:42.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0V/lR0+BNwh/lLlH7AJg6qY5Ols= 2023-10-18T20:45:42.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0V/lR0+BNwh/lLlH7AJg6qY5Ols=', 'Content-Length': '0'} 2023-10-18T20:45:42.571 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-10-18T20:45:42.571 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:45:43.545 DEBUG:boto:Token: None 2023-10-18T20:45:43.546 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:42 GMT /bofrnf-59/ 2023-10-18T20:45:43.547 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0V/lR0+BNwh/lLlH7AJg6qY5Ols= 2023-10-18T20:45:43.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0V/lR0+BNwh/lLlH7AJg6qY5Ols=', 'Content-Length': '0'} 2023-10-18T20:45:43.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004feed87e53d4a8a7-00653043f7-1d9c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.554 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.555 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.555 DEBUG:boto:Method: HEAD 2023-10-18T20:45:43.555 DEBUG:boto:Path: /bofrnf-59/ 2023-10-18T20:45:43.556 DEBUG:boto:Data: 2023-10-18T20:45:43.556 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.557 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.557 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.557 DEBUG:boto:Params: {} 2023-10-18T20:45:43.558 DEBUG:boto:Token: None 2023-10-18T20:45:43.558 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/ 2023-10-18T20:45:43.559 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xJjBWy/BkpN2XPL34H+fUd3Cfts= 2023-10-18T20:45:43.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xJjBWy/BkpN2XPL34H+fUd3Cfts=', 'Content-Length': '0'} 2023-10-18T20:45:43.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011f3939a54f10e39-00653043f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.565 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:45:43.565 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.565 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.566 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.566 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.567 DEBUG:boto:Method: GET 2023-10-18T20:45:43.567 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.567 DEBUG:boto:Data: 2023-10-18T20:45:43.568 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.568 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.569 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.569 DEBUG:boto:Params: {} 2023-10-18T20:45:43.570 DEBUG:boto:Token: None 2023-10-18T20:45:43.570 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/?versions 2023-10-18T20:45:43.571 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ijvd4Y/GVbtlapktk6RamhZJ06M= 2023-10-18T20:45:43.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ijvd4Y/GVbtlapktk6RamhZJ06M=', 'Content-Length': '0'} 2023-10-18T20:45:43.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000219c6b7cc239b58-00653043f7-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.575 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:45:43.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-18T20:45:43.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:45:43.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-18T20:45:43.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-18T20:45:43.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-18T20:45:43.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-18T20:45:43.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-18T20:45:43.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:45:43.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-18T20:45:43.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:45:43.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:45:43.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:45:43.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-18T20:45:43.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-18T20:45:43.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:45:43.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:45:43.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-18T20:45:43.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:45:43.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:45:43.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:45:43.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:45:43.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:45:43.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:45:43.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-18T20:45:43.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-18T20:45:43.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:45:43.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:45:43.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-18T20:45:43.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:45:43.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:45:43.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-18T20:45:43.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:45:43.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-18T20:45:43.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:45:43.590 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:45:43.590 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.591 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.591 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.592 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.592 DEBUG:boto:Method: GET 2023-10-18T20:45:43.592 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.593 DEBUG:boto:Data: 2023-10-18T20:45:43.593 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.594 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.594 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.594 DEBUG:boto:Params: {} 2023-10-18T20:45:43.595 DEBUG:boto:Token: None 2023-10-18T20:45:43.595 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/?versions 2023-10-18T20:45:43.596 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ijvd4Y/GVbtlapktk6RamhZJ06M= 2023-10-18T20:45:43.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ijvd4Y/GVbtlapktk6RamhZJ06M=', 'Content-Length': '0'} 2023-10-18T20:45:43.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000084ad7e0b21e29a-00653043f7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.601 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:45:43.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-18T20:45:43.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-18T20:45:43.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-18T20:45:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-18T20:45:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-18T20:45:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-18T20:45:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-18T20:45:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-18T20:45:43.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-18T20:45:43.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-18T20:45:43.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:45:43.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-18T20:45:43.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-18T20:45:43.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-18T20:45:43.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:45:43.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-18T20:45:43.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-18T20:45:43.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-18T20:45:43.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:45:43.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-18T20:45:43.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-18T20:45:43.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:45:43.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-18T20:45:43.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-18T20:45:43.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-18T20:45:43.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-18T20:45:43.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-18T20:45:43.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-18T20:45:43.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-18T20:45:43.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-18T20:45:43.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-18T20:45:43.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-18T20:45:43.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-18T20:45:43.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-18T20:45:43.616 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.617 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.617 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.618 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.618 DEBUG:boto:Method: GET 2023-10-18T20:45:43.618 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.619 DEBUG:boto:Data: 2023-10-18T20:45:43.619 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.620 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.620 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.620 DEBUG:boto:Params: {} 2023-10-18T20:45:43.621 DEBUG:boto:Token: None 2023-10-18T20:45:43.621 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/?versions 2023-10-18T20:45:43.622 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ijvd4Y/GVbtlapktk6RamhZJ06M= 2023-10-18T20:45:43.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ijvd4Y/GVbtlapktk6RamhZJ06M=', 'Content-Length': '0'} 2023-10-18T20:45:43.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f500dbf042e6038-00653043f7-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.626 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.628 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.629 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.629 DEBUG:boto:path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.629 DEBUG:boto:auth_path=/bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.630 DEBUG:boto:Method: GET 2023-10-18T20:45:43.630 DEBUG:boto:Path: /bofrnf-59/?versions&max-keys=999 2023-10-18T20:45:43.631 DEBUG:boto:Data: 2023-10-18T20:45:43.631 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.632 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.632 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.632 DEBUG:boto:Params: {} 2023-10-18T20:45:43.633 DEBUG:boto:Token: None 2023-10-18T20:45:43.633 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/?versions 2023-10-18T20:45:43.634 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ijvd4Y/GVbtlapktk6RamhZJ06M= 2023-10-18T20:45:43.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ijvd4Y/GVbtlapktk6RamhZJ06M=', 'Content-Length': '0'} 2023-10-18T20:45:43.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d542fe99eda9d1c1-00653043f7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.639 DEBUG:boto:b'bofrnf-59999falseobj11nulltrue2023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:45:43.642 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.642 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.643 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.643 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.644 DEBUG:boto:Method: GET 2023-10-18T20:45:43.644 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.645 DEBUG:boto:Data: 2023-10-18T20:45:43.645 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.646 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.646 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.647 DEBUG:boto:Params: {} 2023-10-18T20:45:43.647 DEBUG:boto:Token: None 2023-10-18T20:45:43.648 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.648 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYP0pN3xl9/1bPIpr1Jxl+Px9NU= 2023-10-18T20:45:43.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYP0pN3xl9/1bPIpr1Jxl+Px9NU=', 'Content-Length': '0'} 2023-10-18T20:45:43.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b1e2fd7a9377508-00653043f7-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.653 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.653 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.654 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.654 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.654 DEBUG:boto:Method: GET 2023-10-18T20:45:43.655 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.655 DEBUG:boto:Data: 2023-10-18T20:45:43.656 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.656 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.657 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.657 DEBUG:boto:Params: {} 2023-10-18T20:45:43.658 DEBUG:boto:Token: None 2023-10-18T20:45:43.658 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.659 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYP0pN3xl9/1bPIpr1Jxl+Px9NU= 2023-10-18T20:45:43.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYP0pN3xl9/1bPIpr1Jxl+Px9NU=', 'Content-Length': '0'} 2023-10-18T20:45:43.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c51d73e88cd6d562-00653043f7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.702 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.703 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.704 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.704 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.704 DEBUG:boto:Method: HEAD 2023-10-18T20:45:43.705 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.705 DEBUG:boto:Data: 2023-10-18T20:45:43.706 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.707 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.707 DEBUG:boto:Params: {} 2023-10-18T20:45:43.708 DEBUG:boto:Token: None 2023-10-18T20:45:43.708 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.709 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XC4KcVRmYkr+FcGkjcx/yOexHWA= 2023-10-18T20:45:43.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XC4KcVRmYkr+FcGkjcx/yOexHWA=', 'Content-Length': '0'} 2023-10-18T20:45:43.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ceac3887e05e66-00653043f7-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.712 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.712 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.713 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.713 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.713 DEBUG:boto:Method: HEAD 2023-10-18T20:45:43.714 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.714 DEBUG:boto:Data: 2023-10-18T20:45:43.715 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.716 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.716 DEBUG:boto:Params: {} 2023-10-18T20:45:43.717 DEBUG:boto:Token: None 2023-10-18T20:45:43.717 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.718 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XC4KcVRmYkr+FcGkjcx/yOexHWA= 2023-10-18T20:45:43.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XC4KcVRmYkr+FcGkjcx/yOexHWA=', 'Content-Length': '0'} 2023-10-18T20:45:43.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1ad13c78d6354e-00653043f7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:45:43.721 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.722 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.723 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.723 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.724 DEBUG:boto:Method: GET 2023-10-18T20:45:43.724 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.725 DEBUG:boto:Data: 2023-10-18T20:45:43.725 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.726 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.726 DEBUG:boto:Params: {} 2023-10-18T20:45:43.727 DEBUG:boto:Token: None 2023-10-18T20:45:43.728 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.729 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYP0pN3xl9/1bPIpr1Jxl+Px9NU= 2023-10-18T20:45:43.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYP0pN3xl9/1bPIpr1Jxl+Px9NU=', 'Content-Length': '0'} 2023-10-18T20:45:43.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc68f26a6a6604db-00653043f7-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.733 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.733 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.734 DEBUG:boto:path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.734 DEBUG:boto:auth_path=/bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.734 DEBUG:boto:Method: GET 2023-10-18T20:45:43.735 DEBUG:boto:Path: /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.735 DEBUG:boto:Data: 2023-10-18T20:45:43.736 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.736 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.737 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.737 DEBUG:boto:Params: {} 2023-10-18T20:45:43.738 DEBUG:boto:Token: None 2023-10-18T20:45:43.739 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11?versionId=null 2023-10-18T20:45:43.739 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gYP0pN3xl9/1bPIpr1Jxl+Px9NU= 2023-10-18T20:45:43.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gYP0pN3xl9/1bPIpr1Jxl+Px9NU=', 'Content-Length': '0'} 2023-10-18T20:45:43.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd00d3df8eabadcf-00653043f7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.743 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.743 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.744 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.744 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.745 DEBUG:boto:Method: GET 2023-10-18T20:45:43.745 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.746 DEBUG:boto:Data: 2023-10-18T20:45:43.746 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.747 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.748 DEBUG:boto:Params: {} 2023-10-18T20:45:43.748 DEBUG:boto:Token: None 2023-10-18T20:45:43.749 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/ 2023-10-18T20:45:43.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0= 2023-10-18T20:45:43.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0=', 'Content-Length': '0'} 2023-10-18T20:45:43.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dab6b3c036cee7fa-00653043f7-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.753 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.754 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.755 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.756 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.756 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.756 DEBUG:boto:Method: GET 2023-10-18T20:45:43.757 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.757 DEBUG:boto:Data: 2023-10-18T20:45:43.758 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.758 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.759 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.759 DEBUG:boto:Params: {} 2023-10-18T20:45:43.759 DEBUG:boto:Token: None 2023-10-18T20:45:43.760 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/ 2023-10-18T20:45:43.760 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0= 2023-10-18T20:45:43.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0=', 'Content-Length': '0'} 2023-10-18T20:45:43.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f0cb3a03997d1dd-00653043f7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.766 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.767 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.767 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.768 DEBUG:boto:Method: HEAD 2023-10-18T20:45:43.768 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:45:43.769 DEBUG:boto:Data: 2023-10-18T20:45:43.769 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.769 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.770 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.770 DEBUG:boto:Params: {} 2023-10-18T20:45:43.771 DEBUG:boto:Token: None 2023-10-18T20:45:43.771 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11 2023-10-18T20:45:43.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AKYrpqZSuXgLz/A/juvcObkhX/k= 2023-10-18T20:45:43.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AKYrpqZSuXgLz/A/juvcObkhX/k=', 'Content-Length': '0'} 2023-10-18T20:45:43.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000291b96e300307414-00653043f7-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.775 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.775 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.776 DEBUG:boto:Method: HEAD 2023-10-18T20:45:43.776 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:45:43.777 DEBUG:boto:Data: 2023-10-18T20:45:43.777 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.777 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.778 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.778 DEBUG:boto:Params: {} 2023-10-18T20:45:43.779 DEBUG:boto:Token: None 2023-10-18T20:45:43.779 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11 2023-10-18T20:45:43.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AKYrpqZSuXgLz/A/juvcObkhX/k= 2023-10-18T20:45:43.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AKYrpqZSuXgLz/A/juvcObkhX/k=', 'Content-Length': '0'} 2023-10-18T20:45:43.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808f209f67131a5e-00653043f7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:45:43.783 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.784 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.784 DEBUG:boto:Method: GET 2023-10-18T20:45:43.784 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:45:43.785 DEBUG:boto:Data: 2023-10-18T20:45:43.785 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.786 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.786 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.787 DEBUG:boto:Params: {} 2023-10-18T20:45:43.787 DEBUG:boto:Token: None 2023-10-18T20:45:43.788 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11 2023-10-18T20:45:43.788 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r8spI/PSoGt51raiaFIHskZ4JhQ= 2023-10-18T20:45:43.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r8spI/PSoGt51raiaFIHskZ4JhQ=', 'Content-Length': '0'} 2023-10-18T20:45:43.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f989c977a3f076c9-00653043f7-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.834 DEBUG:boto:path=/bofrnf-59/obj11 2023-10-18T20:45:43.835 DEBUG:boto:auth_path=/bofrnf-59/obj11 2023-10-18T20:45:43.836 DEBUG:boto:Method: GET 2023-10-18T20:45:43.836 DEBUG:boto:Path: /bofrnf-59/obj11 2023-10-18T20:45:43.837 DEBUG:boto:Data: 2023-10-18T20:45:43.837 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.838 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.839 DEBUG:boto:Params: {} 2023-10-18T20:45:43.839 DEBUG:boto:Token: None 2023-10-18T20:45:43.839 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj11 2023-10-18T20:45:43.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:r8spI/PSoGt51raiaFIHskZ4JhQ= 2023-10-18T20:45:43.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r8spI/PSoGt51raiaFIHskZ4JhQ=', 'Content-Length': '0'} 2023-10-18T20:45:43.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f3c4bb0e243bc5-00653043f7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.844 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.844 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.845 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.845 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.846 DEBUG:boto:Method: GET 2023-10-18T20:45:43.846 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.847 DEBUG:boto:Data: 2023-10-18T20:45:43.847 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.848 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.849 DEBUG:boto:Params: {} 2023-10-18T20:45:43.849 DEBUG:boto:Token: None 2023-10-18T20:45:43.850 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/ 2023-10-18T20:45:43.850 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0= 2023-10-18T20:45:43.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0=', 'Content-Length': '0'} 2023-10-18T20:45:43.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074e7298a9515a1b0-00653043f7-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.854 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.855 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.856 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.856 DEBUG:boto:path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.856 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.857 DEBUG:boto:Method: GET 2023-10-18T20:45:43.858 DEBUG:boto:Path: /bofrnf-59/?prefix=obj11 2023-10-18T20:45:43.858 DEBUG:boto:Data: 2023-10-18T20:45:43.858 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.859 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.859 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.860 DEBUG:boto:Params: {} 2023-10-18T20:45:43.860 DEBUG:boto:Token: None 2023-10-18T20:45:43.861 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/ 2023-10-18T20:45:43.861 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0= 2023-10-18T20:45:43.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0=', 'Content-Length': '0'} 2023-10-18T20:45:43.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8cab38ce47c7c9e-00653043f7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.867 DEBUG:boto:b'bofrnf-59obj111000falseobj112023-10-18T20:32:33.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-18T20:45:43.869 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:43.869 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:43.870 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.870 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.871 DEBUG:boto:Method: GET 2023-10-18T20:45:43.871 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.872 DEBUG:boto:Data: 2023-10-18T20:45:43.872 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.873 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.873 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.874 DEBUG:boto:Params: {} 2023-10-18T20:45:43.874 DEBUG:boto:Token: None 2023-10-18T20:45:43.875 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.875 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jK12rWiL3EF/AUfHhuzItCxl9kk= 2023-10-18T20:45:43.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jK12rWiL3EF/AUfHhuzItCxl9kk=', 'Content-Length': '0'} 2023-10-18T20:45:43.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000308da80e85f2b4ca-00653043f7-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.879 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:43.879 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:43.880 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.880 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.881 DEBUG:boto:Method: GET 2023-10-18T20:45:43.881 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.882 DEBUG:boto:Data: 2023-10-18T20:45:43.882 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.883 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.883 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.884 DEBUG:boto:Params: {} 2023-10-18T20:45:43.884 DEBUG:boto:Token: None 2023-10-18T20:45:43.884 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.885 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jK12rWiL3EF/AUfHhuzItCxl9kk= 2023-10-18T20:45:43.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jK12rWiL3EF/AUfHhuzItCxl9kk=', 'Content-Length': '0'} 2023-10-18T20:45:43.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c359246d1575e94-00653043f7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.888 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:43.889 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:43.889 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.890 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.890 DEBUG:boto:Method: HEAD 2023-10-18T20:45:43.890 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.891 DEBUG:boto:Data: 2023-10-18T20:45:43.891 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.892 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.892 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.892 DEBUG:boto:Params: {} 2023-10-18T20:45:43.893 DEBUG:boto:Token: None 2023-10-18T20:45:43.893 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.894 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fOgmiqfMrXaBKiZYKHHOH11K6kY= 2023-10-18T20:45:43.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fOgmiqfMrXaBKiZYKHHOH11K6kY=', 'Content-Length': '0'} 2023-10-18T20:45:43.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d3f7865ef223f9-00653043f7-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.897 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:43.897 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:43.898 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.898 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.899 DEBUG:boto:Method: HEAD 2023-10-18T20:45:43.899 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.900 DEBUG:boto:Data: 2023-10-18T20:45:43.900 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.901 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.901 DEBUG:boto:Params: {} 2023-10-18T20:45:43.902 DEBUG:boto:Token: None 2023-10-18T20:45:43.902 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.902 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fOgmiqfMrXaBKiZYKHHOH11K6kY= 2023-10-18T20:45:43.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fOgmiqfMrXaBKiZYKHHOH11K6kY=', 'Content-Length': '0'} 2023-10-18T20:45:43.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f9a6c87c869ad2-00653043f7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-18T20:45:43.906 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:43.906 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:43.907 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.907 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.908 DEBUG:boto:Method: GET 2023-10-18T20:45:43.908 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.909 DEBUG:boto:Data: 2023-10-18T20:45:43.909 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.910 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.910 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.911 DEBUG:boto:Params: {} 2023-10-18T20:45:43.911 DEBUG:boto:Token: None 2023-10-18T20:45:43.912 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.912 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jK12rWiL3EF/AUfHhuzItCxl9kk= 2023-10-18T20:45:43.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jK12rWiL3EF/AUfHhuzItCxl9kk=', 'Content-Length': '0'} 2023-10-18T20:45:43.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016223bcd6a7fca41-00653043f7-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.915 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:43.916 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:43.916 DEBUG:boto:path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.916 DEBUG:boto:auth_path=/bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.917 DEBUG:boto:Method: GET 2023-10-18T20:45:43.917 DEBUG:boto:Path: /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.918 DEBUG:boto:Data: 2023-10-18T20:45:43.918 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.919 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.920 DEBUG:boto:Params: {} 2023-10-18T20:45:43.920 DEBUG:boto:Token: None 2023-10-18T20:45:43.921 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj111?versionId=null 2023-10-18T20:45:43.921 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jK12rWiL3EF/AUfHhuzItCxl9kk= 2023-10-18T20:45:43.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jK12rWiL3EF/AUfHhuzItCxl9kk=', 'Content-Length': '0'} 2023-10-18T20:45:43.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9ff2903d5ee104-00653043f7-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.924 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.925 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.925 DEBUG:boto:path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:43.926 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:43.926 DEBUG:boto:Method: GET 2023-10-18T20:45:43.927 DEBUG:boto:Path: /bofrnf-59/?prefix=obj111 2023-10-18T20:45:43.927 DEBUG:boto:Data: 2023-10-18T20:45:43.927 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.928 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.928 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.929 DEBUG:boto:Params: {} 2023-10-18T20:45:43.929 DEBUG:boto:Token: None 2023-10-18T20:45:43.930 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/ 2023-10-18T20:45:43.931 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0= 2023-10-18T20:45:43.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0=', 'Content-Length': '0'} 2023-10-18T20:45:43.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000393e28915abaec92-00653043f7-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.978 DEBUG:boto:b'bofrnf-59obj1111000falseobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.979 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:43.980 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:43.980 DEBUG:boto:path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:43.980 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:43.981 DEBUG:boto:Method: GET 2023-10-18T20:45:43.981 DEBUG:boto:Path: /bofrnf-59/?prefix=obj111 2023-10-18T20:45:43.982 DEBUG:boto:Data: 2023-10-18T20:45:43.983 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.983 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:43.983 DEBUG:boto:Port: 8000 2023-10-18T20:45:43.984 DEBUG:boto:Params: {} 2023-10-18T20:45:43.984 DEBUG:boto:Token: None 2023-10-18T20:45:43.985 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/ 2023-10-18T20:45:43.985 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0= 2023-10-18T20:45:43.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CxfeCC4+R1PU28Xwmq0kZInRmv0=', 'Content-Length': '0'} 2023-10-18T20:45:43.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7a2baaada934693-00653043f7-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:43.990 DEBUG:boto:b'bofrnf-59obj1111000falseobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:43.991 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:43.991 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:43.992 DEBUG:boto:Method: HEAD 2023-10-18T20:45:43.992 DEBUG:boto:Path: /bofrnf-59/obj111 2023-10-18T20:45:43.993 DEBUG:boto:Data: 2023-10-18T20:45:43.993 DEBUG:boto:Headers: {} 2023-10-18T20:45:43.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:43.994 DEBUG:boto:Port: 8001 2023-10-18T20:45:43.994 DEBUG:boto:Params: {} 2023-10-18T20:45:43.995 DEBUG:boto:Token: None 2023-10-18T20:45:43.995 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:43 GMT /bofrnf-59/obj111 2023-10-18T20:45:43.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6W69eAXy/HILpqvtX3X5ogKZFWg= 2023-10-18T20:45:43.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6W69eAXy/HILpqvtX3X5ogKZFWg=', 'Content-Length': '0'} 2023-10-18T20:45:43.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000025c53d5ed97c6e-00653043f7-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.000 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:44.001 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:44.002 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.002 DEBUG:boto:Path: /bofrnf-59/obj111 2023-10-18T20:45:44.003 DEBUG:boto:Data: 2023-10-18T20:45:44.004 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.004 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.005 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.005 DEBUG:boto:Params: {} 2023-10-18T20:45:44.006 DEBUG:boto:Token: None 2023-10-18T20:45:44.006 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj111 2023-10-18T20:45:44.007 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M/0RvyS69MCw4nZoDQANnUEtobc= 2023-10-18T20:45:44.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M/0RvyS69MCw4nZoDQANnUEtobc=', 'Content-Length': '0'} 2023-10-18T20:45:44.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d3bdddd95d71bcf-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-18T20:45:44.011 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:44.011 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:44.012 DEBUG:boto:Method: GET 2023-10-18T20:45:44.013 DEBUG:boto:Path: /bofrnf-59/obj111 2023-10-18T20:45:44.013 DEBUG:boto:Data: 2023-10-18T20:45:44.014 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.015 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.015 DEBUG:boto:Params: {} 2023-10-18T20:45:44.016 DEBUG:boto:Token: None 2023-10-18T20:45:44.016 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj111 2023-10-18T20:45:44.017 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:srQgqb88615TOo+xTyEi1by/T68= 2023-10-18T20:45:44.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:srQgqb88615TOo+xTyEi1by/T68=', 'Content-Length': '0'} 2023-10-18T20:45:44.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d748c6dd78d041d7-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.020 DEBUG:boto:path=/bofrnf-59/obj111 2023-10-18T20:45:44.020 DEBUG:boto:auth_path=/bofrnf-59/obj111 2023-10-18T20:45:44.021 DEBUG:boto:Method: GET 2023-10-18T20:45:44.021 DEBUG:boto:Path: /bofrnf-59/obj111 2023-10-18T20:45:44.022 DEBUG:boto:Data: 2023-10-18T20:45:44.022 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.023 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.023 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.024 DEBUG:boto:Params: {} 2023-10-18T20:45:44.024 DEBUG:boto:Token: None 2023-10-18T20:45:44.025 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj111 2023-10-18T20:45:44.025 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:srQgqb88615TOo+xTyEi1by/T68= 2023-10-18T20:45:44.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:srQgqb88615TOo+xTyEi1by/T68=', 'Content-Length': '0'} 2023-10-18T20:45:44.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c3b59904802340-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.028 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.029 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.029 DEBUG:boto:path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:44.030 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:44.030 DEBUG:boto:Method: GET 2023-10-18T20:45:44.031 DEBUG:boto:Path: /bofrnf-59/?prefix=obj111 2023-10-18T20:45:44.031 DEBUG:boto:Data: 2023-10-18T20:45:44.032 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.032 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.033 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.033 DEBUG:boto:Params: {} 2023-10-18T20:45:44.033 DEBUG:boto:Token: None 2023-10-18T20:45:44.034 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.035 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027b2aead6909f2ab-00653043f8-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.038 DEBUG:boto:b'bofrnf-59obj1111000falseobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.039 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.039 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.040 DEBUG:boto:path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:44.040 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj111 2023-10-18T20:45:44.041 DEBUG:boto:Method: GET 2023-10-18T20:45:44.042 DEBUG:boto:Path: /bofrnf-59/?prefix=obj111 2023-10-18T20:45:44.042 DEBUG:boto:Data: 2023-10-18T20:45:44.043 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.043 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.044 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.044 DEBUG:boto:Params: {} 2023-10-18T20:45:44.045 DEBUG:boto:Token: None 2023-10-18T20:45:44.046 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.047 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041b968f7e050480d-00653043f8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.052 DEBUG:boto:b'bofrnf-59obj1111000falseobj1112023-10-18T20:43:09.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:45:44.053 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.054 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.055 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.055 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.056 DEBUG:boto:Method: GET 2023-10-18T20:45:44.056 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.057 DEBUG:boto:Data: 2023-10-18T20:45:44.057 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.058 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.058 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.059 DEBUG:boto:Params: {} 2023-10-18T20:45:44.059 DEBUG:boto:Token: None 2023-10-18T20:45:44.060 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.060 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mMrsPbBpEyKBw/8xkTPRgW5CgJc= 2023-10-18T20:45:44.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mMrsPbBpEyKBw/8xkTPRgW5CgJc=', 'Content-Length': '0'} 2023-10-18T20:45:44.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076d63f357c48681a-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.065 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.066 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.067 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.067 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.068 DEBUG:boto:Method: GET 2023-10-18T20:45:44.069 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.069 DEBUG:boto:Data: 2023-10-18T20:45:44.070 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.070 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.071 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.072 DEBUG:boto:Params: {} 2023-10-18T20:45:44.073 DEBUG:boto:Token: None 2023-10-18T20:45:44.074 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.074 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mMrsPbBpEyKBw/8xkTPRgW5CgJc= 2023-10-18T20:45:44.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mMrsPbBpEyKBw/8xkTPRgW5CgJc=', 'Content-Length': '0'} 2023-10-18T20:45:44.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd02a4819d327ea-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.118 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.119 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.120 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.120 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.120 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.121 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.121 DEBUG:boto:Data: 2023-10-18T20:45:44.122 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.122 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.123 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.123 DEBUG:boto:Params: {} 2023-10-18T20:45:44.124 DEBUG:boto:Token: None 2023-10-18T20:45:44.124 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.125 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PIHxnkqnc25njIkDX0dZha2Lvcw= 2023-10-18T20:45:44.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PIHxnkqnc25njIkDX0dZha2Lvcw=', 'Content-Length': '0'} 2023-10-18T20:45:44.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008450e6ac256f04fb-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.128 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.129 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.130 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.130 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.131 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.131 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.132 DEBUG:boto:Data: 2023-10-18T20:45:44.132 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.133 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.133 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.134 DEBUG:boto:Params: {} 2023-10-18T20:45:44.134 DEBUG:boto:Token: None 2023-10-18T20:45:44.135 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PIHxnkqnc25njIkDX0dZha2Lvcw= 2023-10-18T20:45:44.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PIHxnkqnc25njIkDX0dZha2Lvcw=', 'Content-Length': '0'} 2023-10-18T20:45:44.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bff675089b3d633-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:45:44.139 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.139 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.140 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.141 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.141 DEBUG:boto:Method: GET 2023-10-18T20:45:44.141 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.142 DEBUG:boto:Data: 2023-10-18T20:45:44.143 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.143 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.144 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.144 DEBUG:boto:Params: {} 2023-10-18T20:45:44.145 DEBUG:boto:Token: None 2023-10-18T20:45:44.145 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.146 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mMrsPbBpEyKBw/8xkTPRgW5CgJc= 2023-10-18T20:45:44.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mMrsPbBpEyKBw/8xkTPRgW5CgJc=', 'Content-Length': '0'} 2023-10-18T20:45:44.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f77c6e983693d193-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.190 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.192 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.192 DEBUG:boto:path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.193 DEBUG:boto:auth_path=/bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.194 DEBUG:boto:Method: GET 2023-10-18T20:45:44.195 DEBUG:boto:Path: /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.196 DEBUG:boto:Data: 2023-10-18T20:45:44.196 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.198 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.199 DEBUG:boto:Params: {} 2023-10-18T20:45:44.200 DEBUG:boto:Token: None 2023-10-18T20:45:44.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115?versionId=null 2023-10-18T20:45:44.202 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mMrsPbBpEyKBw/8xkTPRgW5CgJc= 2023-10-18T20:45:44.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mMrsPbBpEyKBw/8xkTPRgW5CgJc=', 'Content-Length': '0'} 2023-10-18T20:45:44.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b98b50d1ed6eb7-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.207 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.207 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.208 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.209 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.210 DEBUG:boto:Method: GET 2023-10-18T20:45:44.211 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.211 DEBUG:boto:Data: 2023-10-18T20:45:44.212 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.213 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.214 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.215 DEBUG:boto:Params: {} 2023-10-18T20:45:44.216 DEBUG:boto:Token: None 2023-10-18T20:45:44.216 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.217 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa2ab1fca96fbce0-00653043f8-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.222 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.223 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.223 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.223 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.224 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.225 DEBUG:boto:Method: GET 2023-10-18T20:45:44.226 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.227 DEBUG:boto:Data: 2023-10-18T20:45:44.227 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.228 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.229 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.229 DEBUG:boto:Params: {} 2023-10-18T20:45:44.230 DEBUG:boto:Token: None 2023-10-18T20:45:44.231 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.232 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec3fc1a8eabd3bc-00653043f8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.238 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.239 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.239 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.239 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.240 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:45:44.241 DEBUG:boto:Data: 2023-10-18T20:45:44.241 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.242 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.243 DEBUG:boto:Params: {} 2023-10-18T20:45:44.243 DEBUG:boto:Token: None 2023-10-18T20:45:44.244 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115 2023-10-18T20:45:44.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VDdTaTZOi4UGFrog/OdYE+R5xHk= 2023-10-18T20:45:44.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VDdTaTZOi4UGFrog/OdYE+R5xHk=', 'Content-Length': '0'} 2023-10-18T20:45:44.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e156672c59ce729-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.247 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.248 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.248 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.249 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:45:44.249 DEBUG:boto:Data: 2023-10-18T20:45:44.249 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.250 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.250 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.251 DEBUG:boto:Params: {} 2023-10-18T20:45:44.251 DEBUG:boto:Token: None 2023-10-18T20:45:44.251 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115 2023-10-18T20:45:44.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VDdTaTZOi4UGFrog/OdYE+R5xHk= 2023-10-18T20:45:44.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VDdTaTZOi4UGFrog/OdYE+R5xHk=', 'Content-Length': '0'} 2023-10-18T20:45:44.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2357dfce311f5d-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-18T20:45:44.255 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.256 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.256 DEBUG:boto:Method: GET 2023-10-18T20:45:44.257 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:45:44.257 DEBUG:boto:Data: 2023-10-18T20:45:44.258 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.258 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.258 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.259 DEBUG:boto:Params: {} 2023-10-18T20:45:44.259 DEBUG:boto:Token: None 2023-10-18T20:45:44.260 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115 2023-10-18T20:45:44.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B8wS7/AM6f0hE/x21NwGR+17wwE= 2023-10-18T20:45:44.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B8wS7/AM6f0hE/x21NwGR+17wwE=', 'Content-Length': '0'} 2023-10-18T20:45:44.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004266d6765179aad0-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.263 DEBUG:boto:path=/bofrnf-59/obj115 2023-10-18T20:45:44.264 DEBUG:boto:auth_path=/bofrnf-59/obj115 2023-10-18T20:45:44.264 DEBUG:boto:Method: GET 2023-10-18T20:45:44.265 DEBUG:boto:Path: /bofrnf-59/obj115 2023-10-18T20:45:44.265 DEBUG:boto:Data: 2023-10-18T20:45:44.266 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.266 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.269 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.270 DEBUG:boto:Params: {} 2023-10-18T20:45:44.270 DEBUG:boto:Token: None 2023-10-18T20:45:44.271 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj115 2023-10-18T20:45:44.271 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B8wS7/AM6f0hE/x21NwGR+17wwE= 2023-10-18T20:45:44.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B8wS7/AM6f0hE/x21NwGR+17wwE=', 'Content-Length': '0'} 2023-10-18T20:45:44.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b406a5ba70bd1d42-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.318 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.319 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.319 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.320 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.320 DEBUG:boto:Method: GET 2023-10-18T20:45:44.321 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.321 DEBUG:boto:Data: 2023-10-18T20:45:44.322 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.322 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.323 DEBUG:boto:Params: {} 2023-10-18T20:45:44.323 DEBUG:boto:Token: None 2023-10-18T20:45:44.324 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b81d2115205c5ac2-00653043f8-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.328 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.329 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.329 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.330 DEBUG:boto:path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.330 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.330 DEBUG:boto:Method: GET 2023-10-18T20:45:44.331 DEBUG:boto:Path: /bofrnf-59/?prefix=obj115 2023-10-18T20:45:44.331 DEBUG:boto:Data: 2023-10-18T20:45:44.332 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.332 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.332 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.333 DEBUG:boto:Params: {} 2023-10-18T20:45:44.333 DEBUG:boto:Token: None 2023-10-18T20:45:44.334 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.334 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f838f0c4e3b10ad-00653043f8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.339 DEBUG:boto:b'bofrnf-59obj1151000falseobj1152023-10-18T20:38:22.234Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:45:44.340 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.341 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.341 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.342 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.342 DEBUG:boto:Method: GET 2023-10-18T20:45:44.343 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.346 DEBUG:boto:Data: 2023-10-18T20:45:44.347 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.347 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.348 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.348 DEBUG:boto:Params: {} 2023-10-18T20:45:44.349 DEBUG:boto:Token: None 2023-10-18T20:45:44.350 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.350 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+X41EnSstaZ9CTUMm9zDm3ZlgUo= 2023-10-18T20:45:44.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+X41EnSstaZ9CTUMm9zDm3ZlgUo=', 'Content-Length': '0'} 2023-10-18T20:45:44.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b217cba7aacae2-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.394 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.395 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.396 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.396 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.396 DEBUG:boto:Method: GET 2023-10-18T20:45:44.397 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.397 DEBUG:boto:Data: 2023-10-18T20:45:44.398 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.399 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.399 DEBUG:boto:Params: {} 2023-10-18T20:45:44.400 DEBUG:boto:Token: None 2023-10-18T20:45:44.400 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+X41EnSstaZ9CTUMm9zDm3ZlgUo= 2023-10-18T20:45:44.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+X41EnSstaZ9CTUMm9zDm3ZlgUo=', 'Content-Length': '0'} 2023-10-18T20:45:44.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef806b65f2180549-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.405 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.405 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.406 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.406 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.407 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.407 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.407 DEBUG:boto:Data: 2023-10-18T20:45:44.408 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.408 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.409 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.409 DEBUG:boto:Params: {} 2023-10-18T20:45:44.410 DEBUG:boto:Token: None 2023-10-18T20:45:44.410 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XrPMJLtEdcD2E+p6vMmIBnmt1g0= 2023-10-18T20:45:44.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XrPMJLtEdcD2E+p6vMmIBnmt1g0=', 'Content-Length': '0'} 2023-10-18T20:45:44.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001af88266f690c416-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.415 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.416 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.416 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.417 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.418 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.418 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.419 DEBUG:boto:Data: 2023-10-18T20:45:44.420 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.420 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.420 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.421 DEBUG:boto:Params: {} 2023-10-18T20:45:44.422 DEBUG:boto:Token: None 2023-10-18T20:45:44.422 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.423 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XrPMJLtEdcD2E+p6vMmIBnmt1g0= 2023-10-18T20:45:44.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XrPMJLtEdcD2E+p6vMmIBnmt1g0=', 'Content-Length': '0'} 2023-10-18T20:45:44.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d840a38f96652449-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:45:44.426 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.427 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.427 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.428 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.429 DEBUG:boto:Method: GET 2023-10-18T20:45:44.429 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.430 DEBUG:boto:Data: 2023-10-18T20:45:44.430 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.431 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.431 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.432 DEBUG:boto:Params: {} 2023-10-18T20:45:44.432 DEBUG:boto:Token: None 2023-10-18T20:45:44.433 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.434 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+X41EnSstaZ9CTUMm9zDm3ZlgUo= 2023-10-18T20:45:44.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+X41EnSstaZ9CTUMm9zDm3ZlgUo=', 'Content-Length': '0'} 2023-10-18T20:45:44.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c5e51b3771a889-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.437 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.438 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.438 DEBUG:boto:path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.439 DEBUG:boto:auth_path=/bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.440 DEBUG:boto:Method: GET 2023-10-18T20:45:44.440 DEBUG:boto:Path: /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.441 DEBUG:boto:Data: 2023-10-18T20:45:44.442 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.443 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.443 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.444 DEBUG:boto:Params: {} 2023-10-18T20:45:44.445 DEBUG:boto:Token: None 2023-10-18T20:45:44.446 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116?versionId=null 2023-10-18T20:45:44.447 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+X41EnSstaZ9CTUMm9zDm3ZlgUo= 2023-10-18T20:45:44.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+X41EnSstaZ9CTUMm9zDm3ZlgUo=', 'Content-Length': '0'} 2023-10-18T20:45:44.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c052f8b60d4d146-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.451 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.451 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.452 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.452 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.453 DEBUG:boto:Method: GET 2023-10-18T20:45:44.453 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.453 DEBUG:boto:Data: 2023-10-18T20:45:44.454 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.455 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.455 DEBUG:boto:Params: {} 2023-10-18T20:45:44.456 DEBUG:boto:Token: None 2023-10-18T20:45:44.456 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.457 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6acb390add1e37b-00653043f8-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.460 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.461 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.461 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.462 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.463 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.463 DEBUG:boto:Method: GET 2023-10-18T20:45:44.463 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.464 DEBUG:boto:Data: 2023-10-18T20:45:44.464 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.465 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.465 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.466 DEBUG:boto:Params: {} 2023-10-18T20:45:44.466 DEBUG:boto:Token: None 2023-10-18T20:45:44.467 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.467 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a73aed56fe7ed8df-00653043f8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.474 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.475 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.475 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.475 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.476 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:45:44.476 DEBUG:boto:Data: 2023-10-18T20:45:44.476 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.477 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.477 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.478 DEBUG:boto:Params: {} 2023-10-18T20:45:44.478 DEBUG:boto:Token: None 2023-10-18T20:45:44.479 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116 2023-10-18T20:45:44.479 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kQc+McMIikeonwYtl1drZTmQgGc= 2023-10-18T20:45:44.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kQc+McMIikeonwYtl1drZTmQgGc=', 'Content-Length': '0'} 2023-10-18T20:45:44.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4528cfcbe05b2d2-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.483 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.483 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.484 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.484 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:45:44.485 DEBUG:boto:Data: 2023-10-18T20:45:44.485 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.486 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.486 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.486 DEBUG:boto:Params: {} 2023-10-18T20:45:44.487 DEBUG:boto:Token: None 2023-10-18T20:45:44.487 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116 2023-10-18T20:45:44.488 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kQc+McMIikeonwYtl1drZTmQgGc= 2023-10-18T20:45:44.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kQc+McMIikeonwYtl1drZTmQgGc=', 'Content-Length': '0'} 2023-10-18T20:45:44.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000726d24e6ed709d63-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-18T20:45:44.491 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.492 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.492 DEBUG:boto:Method: GET 2023-10-18T20:45:44.493 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:45:44.493 DEBUG:boto:Data: 2023-10-18T20:45:44.494 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.495 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.495 DEBUG:boto:Params: {} 2023-10-18T20:45:44.495 DEBUG:boto:Token: None 2023-10-18T20:45:44.496 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116 2023-10-18T20:45:44.497 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HD1D/EesPoYbUWYAEOjDvcviDkI= 2023-10-18T20:45:44.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HD1D/EesPoYbUWYAEOjDvcviDkI=', 'Content-Length': '0'} 2023-10-18T20:45:44.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d770ed1b8ffd09ec-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.546 DEBUG:boto:path=/bofrnf-59/obj116 2023-10-18T20:45:44.547 DEBUG:boto:auth_path=/bofrnf-59/obj116 2023-10-18T20:45:44.548 DEBUG:boto:Method: GET 2023-10-18T20:45:44.548 DEBUG:boto:Path: /bofrnf-59/obj116 2023-10-18T20:45:44.549 DEBUG:boto:Data: 2023-10-18T20:45:44.549 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.550 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.551 DEBUG:boto:Params: {} 2023-10-18T20:45:44.551 DEBUG:boto:Token: None 2023-10-18T20:45:44.552 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj116 2023-10-18T20:45:44.552 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HD1D/EesPoYbUWYAEOjDvcviDkI= 2023-10-18T20:45:44.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HD1D/EesPoYbUWYAEOjDvcviDkI=', 'Content-Length': '0'} 2023-10-18T20:45:44.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8ca46faeccec6e2-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.556 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.556 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.557 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.557 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.558 DEBUG:boto:Method: GET 2023-10-18T20:45:44.558 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.559 DEBUG:boto:Data: 2023-10-18T20:45:44.559 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.560 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.560 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.561 DEBUG:boto:Params: {} 2023-10-18T20:45:44.562 DEBUG:boto:Token: None 2023-10-18T20:45:44.562 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.563 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b335f6db6aee72f6-00653043f8-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.567 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.568 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.568 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.569 DEBUG:boto:path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.569 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.570 DEBUG:boto:Method: GET 2023-10-18T20:45:44.570 DEBUG:boto:Path: /bofrnf-59/?prefix=obj116 2023-10-18T20:45:44.571 DEBUG:boto:Data: 2023-10-18T20:45:44.572 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.572 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.573 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.573 DEBUG:boto:Params: {} 2023-10-18T20:45:44.574 DEBUG:boto:Token: None 2023-10-18T20:45:44.574 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.575 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bd3424685fb89a8-00653043f8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.622 DEBUG:boto:b'bofrnf-59obj1161000falseobj1162023-10-18T20:39:46.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-18T20:45:44.624 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.624 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.625 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.625 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.626 DEBUG:boto:Method: GET 2023-10-18T20:45:44.626 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.627 DEBUG:boto:Data: 2023-10-18T20:45:44.627 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.628 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.628 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.629 DEBUG:boto:Params: {} 2023-10-18T20:45:44.629 DEBUG:boto:Token: None 2023-10-18T20:45:44.630 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.630 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9H+C+in+uL/iXbsGgBPsIfHmft0= 2023-10-18T20:45:44.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9H+C+in+uL/iXbsGgBPsIfHmft0=', 'Content-Length': '0'} 2023-10-18T20:45:44.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008acb95004c1a6be8-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.635 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.635 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.636 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.636 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.637 DEBUG:boto:Method: GET 2023-10-18T20:45:44.637 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.638 DEBUG:boto:Data: 2023-10-18T20:45:44.638 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.638 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.639 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.639 DEBUG:boto:Params: {} 2023-10-18T20:45:44.640 DEBUG:boto:Token: None 2023-10-18T20:45:44.641 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.641 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9H+C+in+uL/iXbsGgBPsIfHmft0= 2023-10-18T20:45:44.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9H+C+in+uL/iXbsGgBPsIfHmft0=', 'Content-Length': '0'} 2023-10-18T20:45:44.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c75b0bf2b771e892-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.645 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.646 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.646 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.646 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.647 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.647 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.648 DEBUG:boto:Data: 2023-10-18T20:45:44.648 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.648 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.649 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.649 DEBUG:boto:Params: {} 2023-10-18T20:45:44.650 DEBUG:boto:Token: None 2023-10-18T20:45:44.650 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.651 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vAXnhTHI+/thhHxwJvh2ADAyx7U= 2023-10-18T20:45:44.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vAXnhTHI+/thhHxwJvh2ADAyx7U=', 'Content-Length': '0'} 2023-10-18T20:45:44.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebb05dd16106bb3-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.654 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.655 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.655 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.655 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.656 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.656 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.657 DEBUG:boto:Data: 2023-10-18T20:45:44.657 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.657 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.658 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.658 DEBUG:boto:Params: {} 2023-10-18T20:45:44.659 DEBUG:boto:Token: None 2023-10-18T20:45:44.659 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.660 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vAXnhTHI+/thhHxwJvh2ADAyx7U= 2023-10-18T20:45:44.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vAXnhTHI+/thhHxwJvh2ADAyx7U=', 'Content-Length': '0'} 2023-10-18T20:45:44.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007730f0feff79e936-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-18T20:45:44.664 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.664 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.665 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.665 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.666 DEBUG:boto:Method: GET 2023-10-18T20:45:44.666 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.667 DEBUG:boto:Data: 2023-10-18T20:45:44.667 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.668 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.668 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.669 DEBUG:boto:Params: {} 2023-10-18T20:45:44.669 DEBUG:boto:Token: None 2023-10-18T20:45:44.670 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.670 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9H+C+in+uL/iXbsGgBPsIfHmft0= 2023-10-18T20:45:44.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9H+C+in+uL/iXbsGgBPsIfHmft0=', 'Content-Length': '0'} 2023-10-18T20:45:44.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7c12c2f978fd4c-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.673 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.674 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.674 DEBUG:boto:path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.675 DEBUG:boto:auth_path=/bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.677 DEBUG:boto:Method: GET 2023-10-18T20:45:44.678 DEBUG:boto:Path: /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.679 DEBUG:boto:Data: 2023-10-18T20:45:44.679 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.681 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.681 DEBUG:boto:Params: {} 2023-10-18T20:45:44.682 DEBUG:boto:Token: None 2023-10-18T20:45:44.683 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124?versionId=null 2023-10-18T20:45:44.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9H+C+in+uL/iXbsGgBPsIfHmft0= 2023-10-18T20:45:44.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9H+C+in+uL/iXbsGgBPsIfHmft0=', 'Content-Length': '0'} 2023-10-18T20:45:44.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b59113c7bcf5d9-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.687 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.687 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.688 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.688 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.688 DEBUG:boto:Method: GET 2023-10-18T20:45:44.689 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.689 DEBUG:boto:Data: 2023-10-18T20:45:44.690 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.690 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.691 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.691 DEBUG:boto:Params: {} 2023-10-18T20:45:44.691 DEBUG:boto:Token: None 2023-10-18T20:45:44.692 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000726a003f50c64e75-00653043f8-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.696 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.697 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.697 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.698 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.698 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.699 DEBUG:boto:Method: GET 2023-10-18T20:45:44.699 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.699 DEBUG:boto:Data: 2023-10-18T20:45:44.700 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.700 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.700 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.701 DEBUG:boto:Params: {} 2023-10-18T20:45:44.701 DEBUG:boto:Token: None 2023-10-18T20:45:44.702 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.702 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d7b5fe947bea64c-00653043f8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.750 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.751 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.752 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.752 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.753 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:45:44.753 DEBUG:boto:Data: 2023-10-18T20:45:44.753 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.754 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.755 DEBUG:boto:Params: {} 2023-10-18T20:45:44.755 DEBUG:boto:Token: None 2023-10-18T20:45:44.756 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124 2023-10-18T20:45:44.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/Bmp2VCnjh9qbLJjfVVrW1ALKMs= 2023-10-18T20:45:44.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/Bmp2VCnjh9qbLJjfVVrW1ALKMs=', 'Content-Length': '0'} 2023-10-18T20:45:44.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001087b09d3699b6fc-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.760 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.760 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.761 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.761 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:45:44.762 DEBUG:boto:Data: 2023-10-18T20:45:44.762 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.763 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.763 DEBUG:boto:Params: {} 2023-10-18T20:45:44.764 DEBUG:boto:Token: None 2023-10-18T20:45:44.764 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124 2023-10-18T20:45:44.765 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/Bmp2VCnjh9qbLJjfVVrW1ALKMs= 2023-10-18T20:45:44.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/Bmp2VCnjh9qbLJjfVVrW1ALKMs=', 'Content-Length': '0'} 2023-10-18T20:45:44.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000989601985e2fe7a3-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-18T20:45:44.768 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.769 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.769 DEBUG:boto:Method: GET 2023-10-18T20:45:44.769 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:45:44.770 DEBUG:boto:Data: 2023-10-18T20:45:44.770 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.771 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.771 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.772 DEBUG:boto:Params: {} 2023-10-18T20:45:44.772 DEBUG:boto:Token: None 2023-10-18T20:45:44.772 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124 2023-10-18T20:45:44.773 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lKF3W3AGhjcaRwuXp6eU4Vu7wZY= 2023-10-18T20:45:44.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lKF3W3AGhjcaRwuXp6eU4Vu7wZY=', 'Content-Length': '0'} 2023-10-18T20:45:44.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cefcf037442fd9e-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.776 DEBUG:boto:path=/bofrnf-59/obj124 2023-10-18T20:45:44.776 DEBUG:boto:auth_path=/bofrnf-59/obj124 2023-10-18T20:45:44.777 DEBUG:boto:Method: GET 2023-10-18T20:45:44.777 DEBUG:boto:Path: /bofrnf-59/obj124 2023-10-18T20:45:44.778 DEBUG:boto:Data: 2023-10-18T20:45:44.778 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.779 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.779 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.779 DEBUG:boto:Params: {} 2023-10-18T20:45:44.780 DEBUG:boto:Token: None 2023-10-18T20:45:44.780 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj124 2023-10-18T20:45:44.781 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lKF3W3AGhjcaRwuXp6eU4Vu7wZY= 2023-10-18T20:45:44.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lKF3W3AGhjcaRwuXp6eU4Vu7wZY=', 'Content-Length': '0'} 2023-10-18T20:45:44.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e96bd6ee282bac5c-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.785 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.785 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.786 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.787 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.787 DEBUG:boto:Method: GET 2023-10-18T20:45:44.788 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.789 DEBUG:boto:Data: 2023-10-18T20:45:44.790 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.791 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.792 DEBUG:boto:Params: {} 2023-10-18T20:45:44.793 DEBUG:boto:Token: None 2023-10-18T20:45:44.794 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.795 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b6ce93be8fd5d7-00653043f8-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.838 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.839 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.840 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.840 DEBUG:boto:path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.841 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.841 DEBUG:boto:Method: GET 2023-10-18T20:45:44.842 DEBUG:boto:Path: /bofrnf-59/?prefix=obj124 2023-10-18T20:45:44.842 DEBUG:boto:Data: 2023-10-18T20:45:44.842 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.844 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.844 DEBUG:boto:Params: {} 2023-10-18T20:45:44.844 DEBUG:boto:Token: None 2023-10-18T20:45:44.845 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.845 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000791af8f8cf352771-00653043f8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.851 DEBUG:boto:b'bofrnf-59obj1241000falseobj1242023-10-18T20:41:07.828Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:45:44.852 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:44.852 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:44.853 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.853 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.854 DEBUG:boto:Method: GET 2023-10-18T20:45:44.855 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.855 DEBUG:boto:Data: 2023-10-18T20:45:44.856 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.856 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.857 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.857 DEBUG:boto:Params: {} 2023-10-18T20:45:44.858 DEBUG:boto:Token: None 2023-10-18T20:45:44.859 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uTBk8aFbwbSzniuRMaWM3JlbKm4= 2023-10-18T20:45:44.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uTBk8aFbwbSzniuRMaWM3JlbKm4=', 'Content-Length': '0'} 2023-10-18T20:45:44.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2d39262d51472c5-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.865 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:44.865 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:44.866 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.867 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.867 DEBUG:boto:Method: GET 2023-10-18T20:45:44.868 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.868 DEBUG:boto:Data: 2023-10-18T20:45:44.869 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.870 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.870 DEBUG:boto:Params: {} 2023-10-18T20:45:44.871 DEBUG:boto:Token: None 2023-10-18T20:45:44.871 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uTBk8aFbwbSzniuRMaWM3JlbKm4= 2023-10-18T20:45:44.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uTBk8aFbwbSzniuRMaWM3JlbKm4=', 'Content-Length': '0'} 2023-10-18T20:45:44.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e7ced101d91b97-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.919 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:44.919 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:44.920 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.920 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.921 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.921 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.922 DEBUG:boto:Data: 2023-10-18T20:45:44.922 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.923 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.924 DEBUG:boto:Params: {} 2023-10-18T20:45:44.924 DEBUG:boto:Token: None 2023-10-18T20:45:44.925 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.925 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q5s/cPYxjGZZOTW0PtEdHXVEI5c= 2023-10-18T20:45:44.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q5s/cPYxjGZZOTW0PtEdHXVEI5c=', 'Content-Length': '0'} 2023-10-18T20:45:44.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca6c148bd1db025-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.929 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:44.929 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:44.930 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.931 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.931 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.932 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.933 DEBUG:boto:Data: 2023-10-18T20:45:44.933 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.934 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.935 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.935 DEBUG:boto:Params: {} 2023-10-18T20:45:44.936 DEBUG:boto:Token: None 2023-10-18T20:45:44.937 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.937 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q5s/cPYxjGZZOTW0PtEdHXVEI5c= 2023-10-18T20:45:44.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q5s/cPYxjGZZOTW0PtEdHXVEI5c=', 'Content-Length': '0'} 2023-10-18T20:45:44.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de7605fbb3312385-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:45:44.942 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:44.943 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:44.944 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.944 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.945 DEBUG:boto:Method: GET 2023-10-18T20:45:44.945 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.946 DEBUG:boto:Data: 2023-10-18T20:45:44.947 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.947 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.948 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.949 DEBUG:boto:Params: {} 2023-10-18T20:45:44.950 DEBUG:boto:Token: None 2023-10-18T20:45:44.950 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uTBk8aFbwbSzniuRMaWM3JlbKm4= 2023-10-18T20:45:44.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uTBk8aFbwbSzniuRMaWM3JlbKm4=', 'Content-Length': '0'} 2023-10-18T20:45:44.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a2293c39fd6019-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.955 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:44.956 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:44.957 DEBUG:boto:path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.957 DEBUG:boto:auth_path=/bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.958 DEBUG:boto:Method: GET 2023-10-18T20:45:44.958 DEBUG:boto:Path: /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.959 DEBUG:boto:Data: 2023-10-18T20:45:44.959 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.960 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.961 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.961 DEBUG:boto:Params: {} 2023-10-18T20:45:44.962 DEBUG:boto:Token: None 2023-10-18T20:45:44.962 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj145?versionId=null 2023-10-18T20:45:44.963 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uTBk8aFbwbSzniuRMaWM3JlbKm4= 2023-10-18T20:45:44.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uTBk8aFbwbSzniuRMaWM3JlbKm4=', 'Content-Length': '0'} 2023-10-18T20:45:44.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d743df2bf774cf-00653043f8-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.966 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.967 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.967 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:44.968 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:44.968 DEBUG:boto:Method: GET 2023-10-18T20:45:44.968 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:45:44.969 DEBUG:boto:Data: 2023-10-18T20:45:44.969 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.970 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.970 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.971 DEBUG:boto:Params: {} 2023-10-18T20:45:44.971 DEBUG:boto:Token: None 2023-10-18T20:45:44.972 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.972 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004929f1441c96db4c-00653043f8-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.976 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.976 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:44.977 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:44.977 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:44.977 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:44.978 DEBUG:boto:Method: GET 2023-10-18T20:45:44.978 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:45:44.979 DEBUG:boto:Data: 2023-10-18T20:45:44.979 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.980 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.980 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.980 DEBUG:boto:Params: {} 2023-10-18T20:45:44.981 DEBUG:boto:Token: None 2023-10-18T20:45:44.981 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/ 2023-10-18T20:45:44.982 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A= 2023-10-18T20:45:44.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mlC7drIk3oA0twdeAr49OF/P32A=', 'Content-Length': '0'} 2023-10-18T20:45:44.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000171fd8f87b3c3ab3-00653043f8-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.987 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:44.987 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:44.988 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:44.988 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.989 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:45:44.989 DEBUG:boto:Data: 2023-10-18T20:45:44.989 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.990 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:44.990 DEBUG:boto:Port: 8001 2023-10-18T20:45:44.991 DEBUG:boto:Params: {} 2023-10-18T20:45:44.991 DEBUG:boto:Token: None 2023-10-18T20:45:44.992 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:44 GMT /bofrnf-59/obj145 2023-10-18T20:45:44.992 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rz5T2tCZPhi9cdmOydazznIPfdE= 2023-10-18T20:45:44.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:44 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rz5T2tCZPhi9cdmOydazznIPfdE=', 'Content-Length': '0'} 2023-10-18T20:45:44.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000171bbf44e0e5f24c-00653043f8-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:44.996 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:44.996 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:44.997 DEBUG:boto:Method: HEAD 2023-10-18T20:45:44.997 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:45:44.998 DEBUG:boto:Data: 2023-10-18T20:45:44.998 DEBUG:boto:Headers: {} 2023-10-18T20:45:44.998 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:44.999 DEBUG:boto:Port: 8000 2023-10-18T20:45:44.999 DEBUG:boto:Params: {} 2023-10-18T20:45:45.000 DEBUG:boto:Token: None 2023-10-18T20:45:45.000 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj145 2023-10-18T20:45:45.001 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fjMSf3vqzBFWFyj1zqTSa3rXDSc= 2023-10-18T20:45:45.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fjMSf3vqzBFWFyj1zqTSa3rXDSc=', 'Content-Length': '0'} 2023-10-18T20:45:45.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5545db296593ae0-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-18T20:45:45.004 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:45.004 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:45.005 DEBUG:boto:Method: GET 2023-10-18T20:45:45.005 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:45:45.006 DEBUG:boto:Data: 2023-10-18T20:45:45.006 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.007 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.008 DEBUG:boto:Params: {} 2023-10-18T20:45:45.008 DEBUG:boto:Token: None 2023-10-18T20:45:45.008 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj145 2023-10-18T20:45:45.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MdSup/3fKPJ8dWzd6wNuRaBkDz8= 2023-10-18T20:45:45.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MdSup/3fKPJ8dWzd6wNuRaBkDz8=', 'Content-Length': '0'} 2023-10-18T20:45:45.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e10f24b2ce2bac15-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.054 DEBUG:boto:path=/bofrnf-59/obj145 2023-10-18T20:45:45.055 DEBUG:boto:auth_path=/bofrnf-59/obj145 2023-10-18T20:45:45.055 DEBUG:boto:Method: GET 2023-10-18T20:45:45.056 DEBUG:boto:Path: /bofrnf-59/obj145 2023-10-18T20:45:45.056 DEBUG:boto:Data: 2023-10-18T20:45:45.057 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.057 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.057 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.058 DEBUG:boto:Params: {} 2023-10-18T20:45:45.058 DEBUG:boto:Token: None 2023-10-18T20:45:45.059 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj145 2023-10-18T20:45:45.059 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MdSup/3fKPJ8dWzd6wNuRaBkDz8= 2023-10-18T20:45:45.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MdSup/3fKPJ8dWzd6wNuRaBkDz8=', 'Content-Length': '0'} 2023-10-18T20:45:45.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dde45b26feaac25d-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.063 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.064 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.064 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:45.064 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:45.065 DEBUG:boto:Method: GET 2023-10-18T20:45:45.065 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:45:45.066 DEBUG:boto:Data: 2023-10-18T20:45:45.066 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.067 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.067 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.067 DEBUG:boto:Params: {} 2023-10-18T20:45:45.068 DEBUG:boto:Token: None 2023-10-18T20:45:45.068 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.069 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4614b1fcd9491ff-00653043f9-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.072 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.073 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.074 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.074 DEBUG:boto:path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:45.074 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj145 2023-10-18T20:45:45.075 DEBUG:boto:Method: GET 2023-10-18T20:45:45.075 DEBUG:boto:Path: /bofrnf-59/?prefix=obj145 2023-10-18T20:45:45.076 DEBUG:boto:Data: 2023-10-18T20:45:45.076 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.076 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.077 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.077 DEBUG:boto:Params: {} 2023-10-18T20:45:45.078 DEBUG:boto:Token: None 2023-10-18T20:45:45.078 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.078 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f09ab21b35df15bc-00653043f9-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.085 DEBUG:boto:b'bofrnf-59obj1451000falseobj1452023-10-18T20:39:47.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-18T20:45:45.087 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.087 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.088 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.088 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.088 DEBUG:boto:Method: GET 2023-10-18T20:45:45.089 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.089 DEBUG:boto:Data: 2023-10-18T20:45:45.090 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.090 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.091 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.091 DEBUG:boto:Params: {} 2023-10-18T20:45:45.091 DEBUG:boto:Token: None 2023-10-18T20:45:45.092 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.092 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwUE1mwHYDufovqHqnDX/SaQBfA= 2023-10-18T20:45:45.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwUE1mwHYDufovqHqnDX/SaQBfA=', 'Content-Length': '0'} 2023-10-18T20:45:45.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051613521444ada41-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.097 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.097 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.098 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.098 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.099 DEBUG:boto:Method: GET 2023-10-18T20:45:45.099 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.099 DEBUG:boto:Data: 2023-10-18T20:45:45.100 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.100 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.101 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.101 DEBUG:boto:Params: {} 2023-10-18T20:45:45.102 DEBUG:boto:Token: None 2023-10-18T20:45:45.102 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.102 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwUE1mwHYDufovqHqnDX/SaQBfA= 2023-10-18T20:45:45.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwUE1mwHYDufovqHqnDX/SaQBfA=', 'Content-Length': '0'} 2023-10-18T20:45:45.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b55cb222dc343d-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.146 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.147 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.148 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.148 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.148 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.149 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.149 DEBUG:boto:Data: 2023-10-18T20:45:45.150 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.150 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.151 DEBUG:boto:Params: {} 2023-10-18T20:45:45.151 DEBUG:boto:Token: None 2023-10-18T20:45:45.152 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YiaK/lANcwQbSUlWQCGoupYq+/s= 2023-10-18T20:45:45.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YiaK/lANcwQbSUlWQCGoupYq+/s=', 'Content-Length': '0'} 2023-10-18T20:45:45.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca0324b500b49b05-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.156 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.156 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.157 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.157 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.157 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.158 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.158 DEBUG:boto:Data: 2023-10-18T20:45:45.159 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.159 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.160 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.160 DEBUG:boto:Params: {} 2023-10-18T20:45:45.160 DEBUG:boto:Token: None 2023-10-18T20:45:45.161 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YiaK/lANcwQbSUlWQCGoupYq+/s= 2023-10-18T20:45:45.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YiaK/lANcwQbSUlWQCGoupYq+/s=', 'Content-Length': '0'} 2023-10-18T20:45:45.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c19bc760ae2a3a5-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-18T20:45:45.165 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.166 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.166 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.167 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.168 DEBUG:boto:Method: GET 2023-10-18T20:45:45.169 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.169 DEBUG:boto:Data: 2023-10-18T20:45:45.170 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.171 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.172 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.172 DEBUG:boto:Params: {} 2023-10-18T20:45:45.173 DEBUG:boto:Token: None 2023-10-18T20:45:45.174 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.175 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwUE1mwHYDufovqHqnDX/SaQBfA= 2023-10-18T20:45:45.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwUE1mwHYDufovqHqnDX/SaQBfA=', 'Content-Length': '0'} 2023-10-18T20:45:45.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005089602713eb197a-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.222 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.223 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.224 DEBUG:boto:path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.224 DEBUG:boto:auth_path=/bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.224 DEBUG:boto:Method: GET 2023-10-18T20:45:45.225 DEBUG:boto:Path: /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.225 DEBUG:boto:Data: 2023-10-18T20:45:45.226 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.227 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.227 DEBUG:boto:Params: {} 2023-10-18T20:45:45.228 DEBUG:boto:Token: None 2023-10-18T20:45:45.229 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148?versionId=null 2023-10-18T20:45:45.229 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JwUE1mwHYDufovqHqnDX/SaQBfA= 2023-10-18T20:45:45.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwUE1mwHYDufovqHqnDX/SaQBfA=', 'Content-Length': '0'} 2023-10-18T20:45:45.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000369483234609ce0b-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.233 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.234 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.234 DEBUG:boto:path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.235 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.235 DEBUG:boto:Method: GET 2023-10-18T20:45:45.236 DEBUG:boto:Path: /bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.236 DEBUG:boto:Data: 2023-10-18T20:45:45.237 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.237 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.237 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.238 DEBUG:boto:Params: {} 2023-10-18T20:45:45.238 DEBUG:boto:Token: None 2023-10-18T20:45:45.239 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.239 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b4143ae068160f7-00653043f9-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.243 DEBUG:boto:b'bofrnf-59obj1481000falseobj1482023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.244 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.245 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.245 DEBUG:boto:path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.246 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.246 DEBUG:boto:Method: GET 2023-10-18T20:45:45.247 DEBUG:boto:Path: /bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.247 DEBUG:boto:Data: 2023-10-18T20:45:45.247 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.248 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.249 DEBUG:boto:Params: {} 2023-10-18T20:45:45.250 DEBUG:boto:Token: None 2023-10-18T20:45:45.251 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.251 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd2af3f34943402d-00653043f9-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.258 DEBUG:boto:b'bofrnf-59obj1481000falseobj1482023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.259 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.259 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.259 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.260 DEBUG:boto:Path: /bofrnf-59/obj148 2023-10-18T20:45:45.260 DEBUG:boto:Data: 2023-10-18T20:45:45.261 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.261 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.262 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.262 DEBUG:boto:Params: {} 2023-10-18T20:45:45.262 DEBUG:boto:Token: None 2023-10-18T20:45:45.263 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148 2023-10-18T20:45:45.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gcZSNUCRzjw2JF+iBy04nYNwM1c= 2023-10-18T20:45:45.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gcZSNUCRzjw2JF+iBy04nYNwM1c=', 'Content-Length': '0'} 2023-10-18T20:45:45.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d2f04865ee8718d-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.267 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.268 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.268 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.268 DEBUG:boto:Path: /bofrnf-59/obj148 2023-10-18T20:45:45.269 DEBUG:boto:Data: 2023-10-18T20:45:45.269 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.270 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.270 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.270 DEBUG:boto:Params: {} 2023-10-18T20:45:45.271 DEBUG:boto:Token: None 2023-10-18T20:45:45.272 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148 2023-10-18T20:45:45.273 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gcZSNUCRzjw2JF+iBy04nYNwM1c= 2023-10-18T20:45:45.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gcZSNUCRzjw2JF+iBy04nYNwM1c=', 'Content-Length': '0'} 2023-10-18T20:45:45.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674b00f6bd2e1cf2-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-18T20:45:45.278 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.279 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.280 DEBUG:boto:Method: GET 2023-10-18T20:45:45.280 DEBUG:boto:Path: /bofrnf-59/obj148 2023-10-18T20:45:45.281 DEBUG:boto:Data: 2023-10-18T20:45:45.281 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.281 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.282 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.282 DEBUG:boto:Params: {} 2023-10-18T20:45:45.283 DEBUG:boto:Token: None 2023-10-18T20:45:45.283 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148 2023-10-18T20:45:45.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h7VdeC+KGJfmYE7ftosxWDIe6BA= 2023-10-18T20:45:45.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h7VdeC+KGJfmYE7ftosxWDIe6BA=', 'Content-Length': '0'} 2023-10-18T20:45:45.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4dbf3d5355d30dc-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.334 DEBUG:boto:path=/bofrnf-59/obj148 2023-10-18T20:45:45.335 DEBUG:boto:auth_path=/bofrnf-59/obj148 2023-10-18T20:45:45.335 DEBUG:boto:Method: GET 2023-10-18T20:45:45.336 DEBUG:boto:Path: /bofrnf-59/obj148 2023-10-18T20:45:45.336 DEBUG:boto:Data: 2023-10-18T20:45:45.337 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.337 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.338 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.338 DEBUG:boto:Params: {} 2023-10-18T20:45:45.339 DEBUG:boto:Token: None 2023-10-18T20:45:45.339 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj148 2023-10-18T20:45:45.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h7VdeC+KGJfmYE7ftosxWDIe6BA= 2023-10-18T20:45:45.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h7VdeC+KGJfmYE7ftosxWDIe6BA=', 'Content-Length': '0'} 2023-10-18T20:45:45.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e5c3aea78cf6813-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.343 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.344 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.344 DEBUG:boto:path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.344 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.345 DEBUG:boto:Method: GET 2023-10-18T20:45:45.345 DEBUG:boto:Path: /bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.346 DEBUG:boto:Data: 2023-10-18T20:45:45.346 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.347 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.347 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.347 DEBUG:boto:Params: {} 2023-10-18T20:45:45.348 DEBUG:boto:Token: None 2023-10-18T20:45:45.348 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.349 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb8ed6cf4f3a1aa6-00653043f9-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.353 DEBUG:boto:b'bofrnf-59obj1481000falseobj1482023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.353 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.354 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.354 DEBUG:boto:path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.355 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.355 DEBUG:boto:Method: GET 2023-10-18T20:45:45.356 DEBUG:boto:Path: /bofrnf-59/?prefix=obj148 2023-10-18T20:45:45.356 DEBUG:boto:Data: 2023-10-18T20:45:45.357 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.357 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.357 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.358 DEBUG:boto:Params: {} 2023-10-18T20:45:45.358 DEBUG:boto:Token: None 2023-10-18T20:45:45.359 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.359 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f6d7a52fb4033a7-00653043f9-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.410 DEBUG:boto:b'bofrnf-59obj1481000falseobj1482023-10-18T20:43:09.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-18T20:45:45.412 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.412 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.413 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.413 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.413 DEBUG:boto:Method: GET 2023-10-18T20:45:45.414 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.414 DEBUG:boto:Data: 2023-10-18T20:45:45.415 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.417 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.417 DEBUG:boto:Params: {} 2023-10-18T20:45:45.418 DEBUG:boto:Token: None 2023-10-18T20:45:45.418 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jaBL44J3+PfBbyKDQuU33vCp72o= 2023-10-18T20:45:45.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jaBL44J3+PfBbyKDQuU33vCp72o=', 'Content-Length': '0'} 2023-10-18T20:45:45.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5fe8861db605a32-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.423 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.423 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.424 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.424 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.425 DEBUG:boto:Method: GET 2023-10-18T20:45:45.425 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.426 DEBUG:boto:Data: 2023-10-18T20:45:45.426 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.427 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.428 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.428 DEBUG:boto:Params: {} 2023-10-18T20:45:45.429 DEBUG:boto:Token: None 2023-10-18T20:45:45.430 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.430 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jaBL44J3+PfBbyKDQuU33vCp72o= 2023-10-18T20:45:45.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jaBL44J3+PfBbyKDQuU33vCp72o=', 'Content-Length': '0'} 2023-10-18T20:45:45.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d628509208e56c5c-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.433 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.434 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.434 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.435 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.435 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.436 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.436 DEBUG:boto:Data: 2023-10-18T20:45:45.437 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.437 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.437 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.438 DEBUG:boto:Params: {} 2023-10-18T20:45:45.439 DEBUG:boto:Token: None 2023-10-18T20:45:45.439 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O1RrOoUuYJyD1i+V/RNZLvsPAMA= 2023-10-18T20:45:45.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O1RrOoUuYJyD1i+V/RNZLvsPAMA=', 'Content-Length': '0'} 2023-10-18T20:45:45.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dcfc55ef01117e0-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.443 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.443 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.444 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.444 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.444 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.445 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.445 DEBUG:boto:Data: 2023-10-18T20:45:45.446 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.446 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.446 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.447 DEBUG:boto:Params: {} 2023-10-18T20:45:45.447 DEBUG:boto:Token: None 2023-10-18T20:45:45.448 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O1RrOoUuYJyD1i+V/RNZLvsPAMA= 2023-10-18T20:45:45.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O1RrOoUuYJyD1i+V/RNZLvsPAMA=', 'Content-Length': '0'} 2023-10-18T20:45:45.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c52a61292dfb6b94-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-18T20:45:45.452 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.453 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.454 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.454 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.455 DEBUG:boto:Method: GET 2023-10-18T20:45:45.455 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.456 DEBUG:boto:Data: 2023-10-18T20:45:45.456 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.457 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.458 DEBUG:boto:Params: {} 2023-10-18T20:45:45.458 DEBUG:boto:Token: None 2023-10-18T20:45:45.459 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.459 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jaBL44J3+PfBbyKDQuU33vCp72o= 2023-10-18T20:45:45.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jaBL44J3+PfBbyKDQuU33vCp72o=', 'Content-Length': '0'} 2023-10-18T20:45:45.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dfc882ba4aea399-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.463 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.463 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.463 DEBUG:boto:path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.464 DEBUG:boto:auth_path=/bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.464 DEBUG:boto:Method: GET 2023-10-18T20:45:45.465 DEBUG:boto:Path: /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.465 DEBUG:boto:Data: 2023-10-18T20:45:45.466 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.466 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.467 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.467 DEBUG:boto:Params: {} 2023-10-18T20:45:45.468 DEBUG:boto:Token: None 2023-10-18T20:45:45.468 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155?versionId=null 2023-10-18T20:45:45.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jaBL44J3+PfBbyKDQuU33vCp72o= 2023-10-18T20:45:45.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jaBL44J3+PfBbyKDQuU33vCp72o=', 'Content-Length': '0'} 2023-10-18T20:45:45.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ba18da90830bb6e-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.472 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.472 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.473 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.473 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.474 DEBUG:boto:Method: GET 2023-10-18T20:45:45.474 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.475 DEBUG:boto:Data: 2023-10-18T20:45:45.475 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.476 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.476 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.477 DEBUG:boto:Params: {} 2023-10-18T20:45:45.477 DEBUG:boto:Token: None 2023-10-18T20:45:45.478 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.478 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073fa6ed22d1b3de8-00653043f9-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.482 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.483 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.483 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.484 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.484 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.484 DEBUG:boto:Method: GET 2023-10-18T20:45:45.485 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.485 DEBUG:boto:Data: 2023-10-18T20:45:45.487 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.487 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.487 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.488 DEBUG:boto:Params: {} 2023-10-18T20:45:45.488 DEBUG:boto:Token: None 2023-10-18T20:45:45.489 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.489 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079a27256c9795728-00653043f9-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.538 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.539 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.540 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.540 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.541 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:45:45.541 DEBUG:boto:Data: 2023-10-18T20:45:45.542 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.543 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.543 DEBUG:boto:Params: {} 2023-10-18T20:45:45.544 DEBUG:boto:Token: None 2023-10-18T20:45:45.544 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155 2023-10-18T20:45:45.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zE+Ex3E6ScSvIZmI3F0kg697rK0= 2023-10-18T20:45:45.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zE+Ex3E6ScSvIZmI3F0kg697rK0=', 'Content-Length': '0'} 2023-10-18T20:45:45.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b8dd5fe8b8174fa-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.548 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.548 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.549 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.549 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:45:45.549 DEBUG:boto:Data: 2023-10-18T20:45:45.550 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.551 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.551 DEBUG:boto:Params: {} 2023-10-18T20:45:45.552 DEBUG:boto:Token: None 2023-10-18T20:45:45.553 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155 2023-10-18T20:45:45.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:zE+Ex3E6ScSvIZmI3F0kg697rK0= 2023-10-18T20:45:45.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zE+Ex3E6ScSvIZmI3F0kg697rK0=', 'Content-Length': '0'} 2023-10-18T20:45:45.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82e90ec51a25819-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-18T20:45:45.557 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.557 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.558 DEBUG:boto:Method: GET 2023-10-18T20:45:45.558 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:45:45.559 DEBUG:boto:Data: 2023-10-18T20:45:45.559 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.560 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.560 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.561 DEBUG:boto:Params: {} 2023-10-18T20:45:45.561 DEBUG:boto:Token: None 2023-10-18T20:45:45.562 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155 2023-10-18T20:45:45.562 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hPUxB4Dbvdk9ZKKQTQFtuacZmJM= 2023-10-18T20:45:45.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hPUxB4Dbvdk9ZKKQTQFtuacZmJM=', 'Content-Length': '0'} 2023-10-18T20:45:45.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042924bb89b0a97b6-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.565 DEBUG:boto:path=/bofrnf-59/obj155 2023-10-18T20:45:45.566 DEBUG:boto:auth_path=/bofrnf-59/obj155 2023-10-18T20:45:45.566 DEBUG:boto:Method: GET 2023-10-18T20:45:45.567 DEBUG:boto:Path: /bofrnf-59/obj155 2023-10-18T20:45:45.567 DEBUG:boto:Data: 2023-10-18T20:45:45.568 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.568 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.568 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.569 DEBUG:boto:Params: {} 2023-10-18T20:45:45.569 DEBUG:boto:Token: None 2023-10-18T20:45:45.570 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj155 2023-10-18T20:45:45.571 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hPUxB4Dbvdk9ZKKQTQFtuacZmJM= 2023-10-18T20:45:45.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hPUxB4Dbvdk9ZKKQTQFtuacZmJM=', 'Content-Length': '0'} 2023-10-18T20:45:45.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d29fcd98a8714fe-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.574 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.574 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.575 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.575 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.576 DEBUG:boto:Method: GET 2023-10-18T20:45:45.576 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.577 DEBUG:boto:Data: 2023-10-18T20:45:45.577 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.578 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.578 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.579 DEBUG:boto:Params: {} 2023-10-18T20:45:45.579 DEBUG:boto:Token: None 2023-10-18T20:45:45.580 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.580 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000575cb23e3002e1c6-00653043f9-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.584 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.585 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.585 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.586 DEBUG:boto:path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.586 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.587 DEBUG:boto:Method: GET 2023-10-18T20:45:45.587 DEBUG:boto:Path: /bofrnf-59/?prefix=obj155 2023-10-18T20:45:45.588 DEBUG:boto:Data: 2023-10-18T20:45:45.588 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.589 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.589 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.589 DEBUG:boto:Params: {} 2023-10-18T20:45:45.590 DEBUG:boto:Token: None 2023-10-18T20:45:45.590 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.591 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087cfed6028951499-00653043f9-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.597 DEBUG:boto:b'bofrnf-59obj1551000falseobj1552023-10-18T20:41:07.890Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:45:45.598 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.598 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.599 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.599 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.599 DEBUG:boto:Method: GET 2023-10-18T20:45:45.600 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.600 DEBUG:boto:Data: 2023-10-18T20:45:45.601 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.601 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.602 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.602 DEBUG:boto:Params: {} 2023-10-18T20:45:45.603 DEBUG:boto:Token: None 2023-10-18T20:45:45.603 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.604 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7V7aUmjtyM64ML9fuObkAw4cIJk= 2023-10-18T20:45:45.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7V7aUmjtyM64ML9fuObkAw4cIJk=', 'Content-Length': '0'} 2023-10-18T20:45:45.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ffcba62f26edbaa-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.650 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.651 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.651 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.652 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.652 DEBUG:boto:Method: GET 2023-10-18T20:45:45.653 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.653 DEBUG:boto:Data: 2023-10-18T20:45:45.653 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.654 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.654 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.655 DEBUG:boto:Params: {} 2023-10-18T20:45:45.655 DEBUG:boto:Token: None 2023-10-18T20:45:45.655 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.656 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7V7aUmjtyM64ML9fuObkAw4cIJk= 2023-10-18T20:45:45.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7V7aUmjtyM64ML9fuObkAw4cIJk=', 'Content-Length': '0'} 2023-10-18T20:45:45.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8194835c7459db3-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.659 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.660 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.660 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.660 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.661 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.661 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.662 DEBUG:boto:Data: 2023-10-18T20:45:45.662 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.663 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.663 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.663 DEBUG:boto:Params: {} 2023-10-18T20:45:45.664 DEBUG:boto:Token: None 2023-10-18T20:45:45.664 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.665 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xCSinv2sUbq9MlXPc1QtVCh22IM= 2023-10-18T20:45:45.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xCSinv2sUbq9MlXPc1QtVCh22IM=', 'Content-Length': '0'} 2023-10-18T20:45:45.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005926d3de23a57bdc-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.668 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.668 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.668 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.669 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.669 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.670 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.670 DEBUG:boto:Data: 2023-10-18T20:45:45.670 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.671 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.672 DEBUG:boto:Params: {} 2023-10-18T20:45:45.672 DEBUG:boto:Token: None 2023-10-18T20:45:45.673 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xCSinv2sUbq9MlXPc1QtVCh22IM= 2023-10-18T20:45:45.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xCSinv2sUbq9MlXPc1QtVCh22IM=', 'Content-Length': '0'} 2023-10-18T20:45:45.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7efb803d6780704-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:45:45.677 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.677 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.677 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.678 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.678 DEBUG:boto:Method: GET 2023-10-18T20:45:45.679 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.679 DEBUG:boto:Data: 2023-10-18T20:45:45.680 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.680 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.681 DEBUG:boto:Params: {} 2023-10-18T20:45:45.681 DEBUG:boto:Token: None 2023-10-18T20:45:45.682 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.682 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7V7aUmjtyM64ML9fuObkAw4cIJk= 2023-10-18T20:45:45.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7V7aUmjtyM64ML9fuObkAw4cIJk=', 'Content-Length': '0'} 2023-10-18T20:45:45.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af04fe7a25acbdfa-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.685 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.686 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.686 DEBUG:boto:path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.687 DEBUG:boto:auth_path=/bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.687 DEBUG:boto:Method: GET 2023-10-18T20:45:45.687 DEBUG:boto:Path: /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.688 DEBUG:boto:Data: 2023-10-18T20:45:45.688 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.689 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.690 DEBUG:boto:Params: {} 2023-10-18T20:45:45.690 DEBUG:boto:Token: None 2023-10-18T20:45:45.691 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17?versionId=null 2023-10-18T20:45:45.691 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7V7aUmjtyM64ML9fuObkAw4cIJk= 2023-10-18T20:45:45.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7V7aUmjtyM64ML9fuObkAw4cIJk=', 'Content-Length': '0'} 2023-10-18T20:45:45.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5117047c38213bb-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.695 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.697 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.698 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.699 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.699 DEBUG:boto:Method: GET 2023-10-18T20:45:45.700 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.700 DEBUG:boto:Data: 2023-10-18T20:45:45.700 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.701 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.701 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.702 DEBUG:boto:Params: {} 2023-10-18T20:45:45.702 DEBUG:boto:Token: None 2023-10-18T20:45:45.703 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.703 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000608fc9fb7dedc07d-00653043f9-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.707 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.708 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.708 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.708 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.709 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.709 DEBUG:boto:Method: GET 2023-10-18T20:45:45.710 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.710 DEBUG:boto:Data: 2023-10-18T20:45:45.711 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.711 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.711 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.712 DEBUG:boto:Params: {} 2023-10-18T20:45:45.712 DEBUG:boto:Token: None 2023-10-18T20:45:45.713 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.713 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a4ec54c49d9390-00653043f9-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.719 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.719 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.720 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.720 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.721 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:45:45.721 DEBUG:boto:Data: 2023-10-18T20:45:45.721 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.722 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.722 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.723 DEBUG:boto:Params: {} 2023-10-18T20:45:45.723 DEBUG:boto:Token: None 2023-10-18T20:45:45.724 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17 2023-10-18T20:45:45.724 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qjLAVIowmY/olKs8Ht7mMgRhE9k= 2023-10-18T20:45:45.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qjLAVIowmY/olKs8Ht7mMgRhE9k=', 'Content-Length': '0'} 2023-10-18T20:45:45.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005516225febfe6d9f-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.727 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.727 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.728 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.728 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:45:45.729 DEBUG:boto:Data: 2023-10-18T20:45:45.729 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.729 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.730 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.730 DEBUG:boto:Params: {} 2023-10-18T20:45:45.731 DEBUG:boto:Token: None 2023-10-18T20:45:45.731 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17 2023-10-18T20:45:45.731 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qjLAVIowmY/olKs8Ht7mMgRhE9k= 2023-10-18T20:45:45.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qjLAVIowmY/olKs8Ht7mMgRhE9k=', 'Content-Length': '0'} 2023-10-18T20:45:45.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe9dd8f45facbf4-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-18T20:45:45.735 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.735 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.736 DEBUG:boto:Method: GET 2023-10-18T20:45:45.736 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:45:45.737 DEBUG:boto:Data: 2023-10-18T20:45:45.737 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.738 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.738 DEBUG:boto:Params: {} 2023-10-18T20:45:45.739 DEBUG:boto:Token: None 2023-10-18T20:45:45.739 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17 2023-10-18T20:45:45.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dp4BHOAhnZaXCCn/VmFIoZoEU64= 2023-10-18T20:45:45.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dp4BHOAhnZaXCCn/VmFIoZoEU64=', 'Content-Length': '0'} 2023-10-18T20:45:45.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d491fe91277b3371-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.786 DEBUG:boto:path=/bofrnf-59/obj17 2023-10-18T20:45:45.787 DEBUG:boto:auth_path=/bofrnf-59/obj17 2023-10-18T20:45:45.788 DEBUG:boto:Method: GET 2023-10-18T20:45:45.788 DEBUG:boto:Path: /bofrnf-59/obj17 2023-10-18T20:45:45.788 DEBUG:boto:Data: 2023-10-18T20:45:45.789 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.789 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.790 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.790 DEBUG:boto:Params: {} 2023-10-18T20:45:45.791 DEBUG:boto:Token: None 2023-10-18T20:45:45.791 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj17 2023-10-18T20:45:45.791 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dp4BHOAhnZaXCCn/VmFIoZoEU64= 2023-10-18T20:45:45.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dp4BHOAhnZaXCCn/VmFIoZoEU64=', 'Content-Length': '0'} 2023-10-18T20:45:45.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a046daa29680a40-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.795 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.795 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.796 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.796 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.797 DEBUG:boto:Method: GET 2023-10-18T20:45:45.797 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.797 DEBUG:boto:Data: 2023-10-18T20:45:45.798 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.798 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.799 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.799 DEBUG:boto:Params: {} 2023-10-18T20:45:45.800 DEBUG:boto:Token: None 2023-10-18T20:45:45.800 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029490b2a4fd465c8-00653043f9-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.804 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.805 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.805 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.806 DEBUG:boto:path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.806 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.807 DEBUG:boto:Method: GET 2023-10-18T20:45:45.807 DEBUG:boto:Path: /bofrnf-59/?prefix=obj17 2023-10-18T20:45:45.808 DEBUG:boto:Data: 2023-10-18T20:45:45.808 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.808 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.809 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.809 DEBUG:boto:Params: {} 2023-10-18T20:45:45.810 DEBUG:boto:Token: None 2023-10-18T20:45:45.810 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ebca4664aa9507d-00653043f9-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.816 DEBUG:boto:b'bofrnf-59obj171000falseobj172023-10-18T20:33:05.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-18T20:45:45.817 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:45.818 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:45.818 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.819 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.819 DEBUG:boto:Method: GET 2023-10-18T20:45:45.819 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.820 DEBUG:boto:Data: 2023-10-18T20:45:45.820 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.821 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.821 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.821 DEBUG:boto:Params: {} 2023-10-18T20:45:45.822 DEBUG:boto:Token: None 2023-10-18T20:45:45.822 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.823 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KRgcphbNqI+lqk+V1KA9iLDoH38= 2023-10-18T20:45:45.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KRgcphbNqI+lqk+V1KA9iLDoH38=', 'Content-Length': '0'} 2023-10-18T20:45:45.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea2430d833b1150-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.827 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:45.827 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:45.828 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.828 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.828 DEBUG:boto:Method: GET 2023-10-18T20:45:45.829 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.829 DEBUG:boto:Data: 2023-10-18T20:45:45.830 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.831 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.831 DEBUG:boto:Params: {} 2023-10-18T20:45:45.831 DEBUG:boto:Token: None 2023-10-18T20:45:45.832 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KRgcphbNqI+lqk+V1KA9iLDoH38= 2023-10-18T20:45:45.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KRgcphbNqI+lqk+V1KA9iLDoH38=', 'Content-Length': '0'} 2023-10-18T20:45:45.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000382620203d76a7c9-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.836 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:45.836 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:45.837 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.837 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.838 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.838 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.839 DEBUG:boto:Data: 2023-10-18T20:45:45.840 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.840 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.841 DEBUG:boto:Params: {} 2023-10-18T20:45:45.841 DEBUG:boto:Token: None 2023-10-18T20:45:45.842 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.842 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0l0xQdn+p0pO/Qw0Z3QJGzIe4Fc= 2023-10-18T20:45:45.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0l0xQdn+p0pO/Qw0Z3QJGzIe4Fc=', 'Content-Length': '0'} 2023-10-18T20:45:45.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c85f87014db5e74e-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.845 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:45.846 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:45.846 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.846 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.847 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.847 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.848 DEBUG:boto:Data: 2023-10-18T20:45:45.848 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.848 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.849 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.849 DEBUG:boto:Params: {} 2023-10-18T20:45:45.850 DEBUG:boto:Token: None 2023-10-18T20:45:45.850 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.851 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0l0xQdn+p0pO/Qw0Z3QJGzIe4Fc= 2023-10-18T20:45:45.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0l0xQdn+p0pO/Qw0Z3QJGzIe4Fc=', 'Content-Length': '0'} 2023-10-18T20:45:45.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7cf58157c19203-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-18T20:45:45.854 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:45.855 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:45.855 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.855 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.856 DEBUG:boto:Method: GET 2023-10-18T20:45:45.856 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.857 DEBUG:boto:Data: 2023-10-18T20:45:45.857 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.858 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.858 DEBUG:boto:Params: {} 2023-10-18T20:45:45.859 DEBUG:boto:Token: None 2023-10-18T20:45:45.859 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KRgcphbNqI+lqk+V1KA9iLDoH38= 2023-10-18T20:45:45.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KRgcphbNqI+lqk+V1KA9iLDoH38=', 'Content-Length': '0'} 2023-10-18T20:45:45.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebee9429c8cbd4a2-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.863 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:45.864 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:45.864 DEBUG:boto:path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.865 DEBUG:boto:auth_path=/bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.865 DEBUG:boto:Method: GET 2023-10-18T20:45:45.866 DEBUG:boto:Path: /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.866 DEBUG:boto:Data: 2023-10-18T20:45:45.866 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.867 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.867 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.868 DEBUG:boto:Params: {} 2023-10-18T20:45:45.868 DEBUG:boto:Token: None 2023-10-18T20:45:45.869 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj185?versionId=null 2023-10-18T20:45:45.869 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KRgcphbNqI+lqk+V1KA9iLDoH38= 2023-10-18T20:45:45.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KRgcphbNqI+lqk+V1KA9iLDoH38=', 'Content-Length': '0'} 2023-10-18T20:45:45.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000391699571100960-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.914 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.915 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.916 DEBUG:boto:path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:45.916 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:45.916 DEBUG:boto:Method: GET 2023-10-18T20:45:45.917 DEBUG:boto:Path: /bofrnf-59/?prefix=obj185 2023-10-18T20:45:45.917 DEBUG:boto:Data: 2023-10-18T20:45:45.918 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.919 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.919 DEBUG:boto:Params: {} 2023-10-18T20:45:45.920 DEBUG:boto:Token: None 2023-10-18T20:45:45.920 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.921 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005659c805e75042e7-00653043f9-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.925 DEBUG:boto:b'bofrnf-59obj1851000falseobj1852023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.925 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:45.926 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:45.926 DEBUG:boto:path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:45.927 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:45.927 DEBUG:boto:Method: GET 2023-10-18T20:45:45.927 DEBUG:boto:Path: /bofrnf-59/?prefix=obj185 2023-10-18T20:45:45.928 DEBUG:boto:Data: 2023-10-18T20:45:45.928 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.929 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.930 DEBUG:boto:Params: {} 2023-10-18T20:45:45.930 DEBUG:boto:Token: None 2023-10-18T20:45:45.931 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/ 2023-10-18T20:45:45.931 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg= 2023-10-18T20:45:45.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dZ3Se4Ln4pjAQ2pDQXlAmbt4LOg=', 'Content-Length': '0'} 2023-10-18T20:45:45.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000882e54e492d7cec7-00653043f9-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.937 DEBUG:boto:b'bofrnf-59obj1851000falseobj1852023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:45.938 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:45.938 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:45.938 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.939 DEBUG:boto:Path: /bofrnf-59/obj185 2023-10-18T20:45:45.939 DEBUG:boto:Data: 2023-10-18T20:45:45.940 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.940 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.941 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.941 DEBUG:boto:Params: {} 2023-10-18T20:45:45.941 DEBUG:boto:Token: None 2023-10-18T20:45:45.942 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj185 2023-10-18T20:45:45.942 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FzEjvjpSzi7+gMrrOZvYo0WV+OI= 2023-10-18T20:45:45.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FzEjvjpSzi7+gMrrOZvYo0WV+OI=', 'Content-Length': '0'} 2023-10-18T20:45:45.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda59c3070cfbd6b-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.946 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:45.946 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:45.947 DEBUG:boto:Method: HEAD 2023-10-18T20:45:45.947 DEBUG:boto:Path: /bofrnf-59/obj185 2023-10-18T20:45:45.947 DEBUG:boto:Data: 2023-10-18T20:45:45.948 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.948 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:45.949 DEBUG:boto:Port: 8000 2023-10-18T20:45:45.949 DEBUG:boto:Params: {} 2023-10-18T20:45:45.950 DEBUG:boto:Token: None 2023-10-18T20:45:45.950 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj185 2023-10-18T20:45:45.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FzEjvjpSzi7+gMrrOZvYo0WV+OI= 2023-10-18T20:45:45.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FzEjvjpSzi7+gMrrOZvYo0WV+OI=', 'Content-Length': '0'} 2023-10-18T20:45:45.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225f2e6b2444d27b-00653043f9-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:45.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-18T20:45:45.954 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:45.955 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:45.955 DEBUG:boto:Method: GET 2023-10-18T20:45:45.956 DEBUG:boto:Path: /bofrnf-59/obj185 2023-10-18T20:45:45.956 DEBUG:boto:Data: 2023-10-18T20:45:45.956 DEBUG:boto:Headers: {} 2023-10-18T20:45:45.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:45.957 DEBUG:boto:Port: 8001 2023-10-18T20:45:45.958 DEBUG:boto:Params: {} 2023-10-18T20:45:45.958 DEBUG:boto:Token: None 2023-10-18T20:45:45.959 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:45 GMT /bofrnf-59/obj185 2023-10-18T20:45:45.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mJtYIHUAVnRoOIsAhL1lpOE6dNQ= 2023-10-18T20:45:45.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:45 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mJtYIHUAVnRoOIsAhL1lpOE6dNQ=', 'Content-Length': '0'} 2023-10-18T20:45:45.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c7b4c23b39c8866-00653043f9-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.002 DEBUG:boto:path=/bofrnf-59/obj185 2023-10-18T20:45:46.003 DEBUG:boto:auth_path=/bofrnf-59/obj185 2023-10-18T20:45:46.003 DEBUG:boto:Method: GET 2023-10-18T20:45:46.004 DEBUG:boto:Path: /bofrnf-59/obj185 2023-10-18T20:45:46.004 DEBUG:boto:Data: 2023-10-18T20:45:46.005 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.005 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.006 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.006 DEBUG:boto:Params: {} 2023-10-18T20:45:46.007 DEBUG:boto:Token: None 2023-10-18T20:45:46.008 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj185 2023-10-18T20:45:46.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bxf+DfvJzlVyOseyCIPtkYbv7sM= 2023-10-18T20:45:46.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxf+DfvJzlVyOseyCIPtkYbv7sM=', 'Content-Length': '0'} 2023-10-18T20:45:46.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000633b6a7b96aba83a-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.014 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.014 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.015 DEBUG:boto:path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:46.016 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:46.016 DEBUG:boto:Method: GET 2023-10-18T20:45:46.017 DEBUG:boto:Path: /bofrnf-59/?prefix=obj185 2023-10-18T20:45:46.017 DEBUG:boto:Data: 2023-10-18T20:45:46.017 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.018 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.018 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.019 DEBUG:boto:Params: {} 2023-10-18T20:45:46.019 DEBUG:boto:Token: None 2023-10-18T20:45:46.020 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.020 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e4e6932c2809703-00653043fa-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.024 DEBUG:boto:b'bofrnf-59obj1851000falseobj1852023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.024 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.025 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.025 DEBUG:boto:path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:46.026 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj185 2023-10-18T20:45:46.026 DEBUG:boto:Method: GET 2023-10-18T20:45:46.027 DEBUG:boto:Path: /bofrnf-59/?prefix=obj185 2023-10-18T20:45:46.027 DEBUG:boto:Data: 2023-10-18T20:45:46.028 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.028 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.028 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.029 DEBUG:boto:Params: {} 2023-10-18T20:45:46.029 DEBUG:boto:Token: None 2023-10-18T20:45:46.030 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2ea53f6a0b62d76-00653043fa-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.036 DEBUG:boto:b'bofrnf-59obj1851000falseobj1852023-10-18T20:43:09.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:45:46.037 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.037 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.038 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.038 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.039 DEBUG:boto:Method: GET 2023-10-18T20:45:46.039 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.040 DEBUG:boto:Data: 2023-10-18T20:45:46.040 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.040 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.041 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.041 DEBUG:boto:Params: {} 2023-10-18T20:45:46.042 DEBUG:boto:Token: None 2023-10-18T20:45:46.042 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.043 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/A5GCxznRaPSPunMwcVFfereA0s= 2023-10-18T20:45:46.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/A5GCxznRaPSPunMwcVFfereA0s=', 'Content-Length': '0'} 2023-10-18T20:45:46.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160bf295fc97399f-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.046 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.047 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.047 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.047 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.048 DEBUG:boto:Method: GET 2023-10-18T20:45:46.048 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.049 DEBUG:boto:Data: 2023-10-18T20:45:46.049 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.049 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.050 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.050 DEBUG:boto:Params: {} 2023-10-18T20:45:46.051 DEBUG:boto:Token: None 2023-10-18T20:45:46.051 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/A5GCxznRaPSPunMwcVFfereA0s= 2023-10-18T20:45:46.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/A5GCxznRaPSPunMwcVFfereA0s=', 'Content-Length': '0'} 2023-10-18T20:45:46.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe5d2e8735c2a0b7-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.098 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.099 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.099 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.100 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.100 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.101 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.101 DEBUG:boto:Data: 2023-10-18T20:45:46.101 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.102 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.102 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.103 DEBUG:boto:Params: {} 2023-10-18T20:45:46.103 DEBUG:boto:Token: None 2023-10-18T20:45:46.105 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.105 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6eGojgZODSyRIGpxtSfpRZ7CmyU= 2023-10-18T20:45:46.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6eGojgZODSyRIGpxtSfpRZ7CmyU=', 'Content-Length': '0'} 2023-10-18T20:45:46.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000894b28c72599f6e4-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.109 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.109 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.110 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.110 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.111 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.111 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.111 DEBUG:boto:Data: 2023-10-18T20:45:46.112 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.112 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.113 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.113 DEBUG:boto:Params: {} 2023-10-18T20:45:46.114 DEBUG:boto:Token: None 2023-10-18T20:45:46.114 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.115 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6eGojgZODSyRIGpxtSfpRZ7CmyU= 2023-10-18T20:45:46.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6eGojgZODSyRIGpxtSfpRZ7CmyU=', 'Content-Length': '0'} 2023-10-18T20:45:46.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007581fb387b22489b-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:45:46.119 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.120 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.120 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.121 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.122 DEBUG:boto:Method: GET 2023-10-18T20:45:46.122 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.123 DEBUG:boto:Data: 2023-10-18T20:45:46.123 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.124 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.124 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.125 DEBUG:boto:Params: {} 2023-10-18T20:45:46.125 DEBUG:boto:Token: None 2023-10-18T20:45:46.126 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.126 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/A5GCxznRaPSPunMwcVFfereA0s= 2023-10-18T20:45:46.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/A5GCxznRaPSPunMwcVFfereA0s=', 'Content-Length': '0'} 2023-10-18T20:45:46.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ec05989f16c3324-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.129 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.130 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.130 DEBUG:boto:path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.131 DEBUG:boto:auth_path=/bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.131 DEBUG:boto:Method: GET 2023-10-18T20:45:46.131 DEBUG:boto:Path: /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.132 DEBUG:boto:Data: 2023-10-18T20:45:46.132 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.133 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.133 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.133 DEBUG:boto:Params: {} 2023-10-18T20:45:46.134 DEBUG:boto:Token: None 2023-10-18T20:45:46.134 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19?versionId=null 2023-10-18T20:45:46.135 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/A5GCxznRaPSPunMwcVFfereA0s= 2023-10-18T20:45:46.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/A5GCxznRaPSPunMwcVFfereA0s=', 'Content-Length': '0'} 2023-10-18T20:45:46.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c5d77d4c75763c9-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.138 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.139 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.139 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.140 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.140 DEBUG:boto:Method: GET 2023-10-18T20:45:46.141 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.141 DEBUG:boto:Data: 2023-10-18T20:45:46.141 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.142 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.143 DEBUG:boto:Params: {} 2023-10-18T20:45:46.143 DEBUG:boto:Token: None 2023-10-18T20:45:46.144 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a742c66f87347d-00653043fa-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.148 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.149 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.149 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.149 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.150 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.150 DEBUG:boto:Method: GET 2023-10-18T20:45:46.151 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.151 DEBUG:boto:Data: 2023-10-18T20:45:46.152 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.152 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.153 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.153 DEBUG:boto:Params: {} 2023-10-18T20:45:46.153 DEBUG:boto:Token: None 2023-10-18T20:45:46.154 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.154 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013eca9a4dd38cec8-00653043fa-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.160 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.161 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.161 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.162 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.162 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:45:46.162 DEBUG:boto:Data: 2023-10-18T20:45:46.163 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.163 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.164 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.164 DEBUG:boto:Params: {} 2023-10-18T20:45:46.165 DEBUG:boto:Token: None 2023-10-18T20:45:46.165 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19 2023-10-18T20:45:46.165 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:erzWd4jiGhMpZ8ne9Nf+k7SedB0= 2023-10-18T20:45:46.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:erzWd4jiGhMpZ8ne9Nf+k7SedB0=', 'Content-Length': '0'} 2023-10-18T20:45:46.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000242e953d4bd6030a-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.168 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.169 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.170 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.170 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:45:46.171 DEBUG:boto:Data: 2023-10-18T20:45:46.172 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.172 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.173 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.174 DEBUG:boto:Params: {} 2023-10-18T20:45:46.175 DEBUG:boto:Token: None 2023-10-18T20:45:46.175 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19 2023-10-18T20:45:46.176 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:erzWd4jiGhMpZ8ne9Nf+k7SedB0= 2023-10-18T20:45:46.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:erzWd4jiGhMpZ8ne9Nf+k7SedB0=', 'Content-Length': '0'} 2023-10-18T20:45:46.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000942056df5dd26f1-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-18T20:45:46.179 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.180 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.180 DEBUG:boto:Method: GET 2023-10-18T20:45:46.181 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:45:46.181 DEBUG:boto:Data: 2023-10-18T20:45:46.182 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.182 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.183 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.183 DEBUG:boto:Params: {} 2023-10-18T20:45:46.184 DEBUG:boto:Token: None 2023-10-18T20:45:46.184 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19 2023-10-18T20:45:46.185 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L892N7RpykRLyZZ5YMbklS8bguQ= 2023-10-18T20:45:46.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L892N7RpykRLyZZ5YMbklS8bguQ=', 'Content-Length': '0'} 2023-10-18T20:45:46.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005995f219a03eb93-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.230 DEBUG:boto:path=/bofrnf-59/obj19 2023-10-18T20:45:46.231 DEBUG:boto:auth_path=/bofrnf-59/obj19 2023-10-18T20:45:46.232 DEBUG:boto:Method: GET 2023-10-18T20:45:46.232 DEBUG:boto:Path: /bofrnf-59/obj19 2023-10-18T20:45:46.233 DEBUG:boto:Data: 2023-10-18T20:45:46.233 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.234 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.234 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.235 DEBUG:boto:Params: {} 2023-10-18T20:45:46.235 DEBUG:boto:Token: None 2023-10-18T20:45:46.236 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj19 2023-10-18T20:45:46.236 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:L892N7RpykRLyZZ5YMbklS8bguQ= 2023-10-18T20:45:46.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:L892N7RpykRLyZZ5YMbklS8bguQ=', 'Content-Length': '0'} 2023-10-18T20:45:46.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dbf625de8a8d8ec-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.240 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.241 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.241 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.242 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.242 DEBUG:boto:Method: GET 2023-10-18T20:45:46.243 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.244 DEBUG:boto:Data: 2023-10-18T20:45:46.245 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.246 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.246 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.247 DEBUG:boto:Params: {} 2023-10-18T20:45:46.248 DEBUG:boto:Token: None 2023-10-18T20:45:46.248 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.249 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008286a31da0659074-00653043fa-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.254 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.255 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.256 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.257 DEBUG:boto:path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.257 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.258 DEBUG:boto:Method: GET 2023-10-18T20:45:46.259 DEBUG:boto:Path: /bofrnf-59/?prefix=obj19 2023-10-18T20:45:46.260 DEBUG:boto:Data: 2023-10-18T20:45:46.260 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.261 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.261 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.261 DEBUG:boto:Params: {} 2023-10-18T20:45:46.262 DEBUG:boto:Token: None 2023-10-18T20:45:46.263 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3fa8f209dadb648-00653043fa-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.314 DEBUG:boto:b'bofrnf-59obj191000falseobj192023-10-18T20:34:23.373Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:45:46.316 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.316 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.317 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.317 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.317 DEBUG:boto:Method: GET 2023-10-18T20:45:46.318 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.318 DEBUG:boto:Data: 2023-10-18T20:45:46.319 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.320 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.321 DEBUG:boto:Params: {} 2023-10-18T20:45:46.322 DEBUG:boto:Token: None 2023-10-18T20:45:46.322 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.323 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7aXlr0ffSCpefvQZdAkzQ5/RfPQ= 2023-10-18T20:45:46.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7aXlr0ffSCpefvQZdAkzQ5/RfPQ=', 'Content-Length': '0'} 2023-10-18T20:45:46.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba36cbe15e2e1fe1-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.327 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.328 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.328 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.329 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.329 DEBUG:boto:Method: GET 2023-10-18T20:45:46.330 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.330 DEBUG:boto:Data: 2023-10-18T20:45:46.331 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.331 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.332 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.332 DEBUG:boto:Params: {} 2023-10-18T20:45:46.333 DEBUG:boto:Token: None 2023-10-18T20:45:46.333 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.334 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7aXlr0ffSCpefvQZdAkzQ5/RfPQ= 2023-10-18T20:45:46.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7aXlr0ffSCpefvQZdAkzQ5/RfPQ=', 'Content-Length': '0'} 2023-10-18T20:45:46.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006392478b22e67d8e-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.337 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.338 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.338 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.339 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.339 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.340 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.340 DEBUG:boto:Data: 2023-10-18T20:45:46.341 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.341 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.342 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.343 DEBUG:boto:Params: {} 2023-10-18T20:45:46.344 DEBUG:boto:Token: None 2023-10-18T20:45:46.344 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.345 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z78N60XVSo0/T2QUROuIPH9GJ3o= 2023-10-18T20:45:46.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z78N60XVSo0/T2QUROuIPH9GJ3o=', 'Content-Length': '0'} 2023-10-18T20:45:46.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae7541fb6eab8556-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.348 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.349 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.349 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.350 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.350 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.350 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.351 DEBUG:boto:Data: 2023-10-18T20:45:46.351 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.352 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.352 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.353 DEBUG:boto:Params: {} 2023-10-18T20:45:46.353 DEBUG:boto:Token: None 2023-10-18T20:45:46.354 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.355 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z78N60XVSo0/T2QUROuIPH9GJ3o= 2023-10-18T20:45:46.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z78N60XVSo0/T2QUROuIPH9GJ3o=', 'Content-Length': '0'} 2023-10-18T20:45:46.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb3e48879f21aea1-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:45:46.358 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.359 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.359 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.360 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.361 DEBUG:boto:Method: GET 2023-10-18T20:45:46.362 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.363 DEBUG:boto:Data: 2023-10-18T20:45:46.363 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.364 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.365 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.365 DEBUG:boto:Params: {} 2023-10-18T20:45:46.366 DEBUG:boto:Token: None 2023-10-18T20:45:46.366 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.367 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7aXlr0ffSCpefvQZdAkzQ5/RfPQ= 2023-10-18T20:45:46.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7aXlr0ffSCpefvQZdAkzQ5/RfPQ=', 'Content-Length': '0'} 2023-10-18T20:45:46.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ede495d2bcd756f-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.370 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.371 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.371 DEBUG:boto:path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.372 DEBUG:boto:auth_path=/bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.372 DEBUG:boto:Method: GET 2023-10-18T20:45:46.373 DEBUG:boto:Path: /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.373 DEBUG:boto:Data: 2023-10-18T20:45:46.373 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.375 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.376 DEBUG:boto:Params: {} 2023-10-18T20:45:46.376 DEBUG:boto:Token: None 2023-10-18T20:45:46.377 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22?versionId=null 2023-10-18T20:45:46.378 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7aXlr0ffSCpefvQZdAkzQ5/RfPQ= 2023-10-18T20:45:46.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7aXlr0ffSCpefvQZdAkzQ5/RfPQ=', 'Content-Length': '0'} 2023-10-18T20:45:46.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1ed88907bccc12-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.382 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.383 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.383 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.383 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.384 DEBUG:boto:Method: GET 2023-10-18T20:45:46.384 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.385 DEBUG:boto:Data: 2023-10-18T20:45:46.385 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.386 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.386 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.387 DEBUG:boto:Params: {} 2023-10-18T20:45:46.387 DEBUG:boto:Token: None 2023-10-18T20:45:46.388 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b21752956a16e28-00653043fa-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.392 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.393 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.394 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.395 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.395 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.396 DEBUG:boto:Method: GET 2023-10-18T20:45:46.396 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.397 DEBUG:boto:Data: 2023-10-18T20:45:46.398 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.399 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.400 DEBUG:boto:Params: {} 2023-10-18T20:45:46.400 DEBUG:boto:Token: None 2023-10-18T20:45:46.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.402 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000868cdc99799ca66d-00653043fa-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.408 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.409 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.409 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.410 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.410 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:45:46.411 DEBUG:boto:Data: 2023-10-18T20:45:46.411 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.412 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.412 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.412 DEBUG:boto:Params: {} 2023-10-18T20:45:46.413 DEBUG:boto:Token: None 2023-10-18T20:45:46.413 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22 2023-10-18T20:45:46.414 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G9KvTvNDYC0QbUG1Ymtwg5tpDbU= 2023-10-18T20:45:46.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G9KvTvNDYC0QbUG1Ymtwg5tpDbU=', 'Content-Length': '0'} 2023-10-18T20:45:46.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae9067d05b1482a-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.417 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.417 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.418 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.418 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:45:46.419 DEBUG:boto:Data: 2023-10-18T20:45:46.419 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.420 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.420 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.420 DEBUG:boto:Params: {} 2023-10-18T20:45:46.421 DEBUG:boto:Token: None 2023-10-18T20:45:46.421 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22 2023-10-18T20:45:46.422 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G9KvTvNDYC0QbUG1Ymtwg5tpDbU= 2023-10-18T20:45:46.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G9KvTvNDYC0QbUG1Ymtwg5tpDbU=', 'Content-Length': '0'} 2023-10-18T20:45:46.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a6d8d2752e777ae-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:45:46.425 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.426 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.426 DEBUG:boto:Method: GET 2023-10-18T20:45:46.427 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:45:46.427 DEBUG:boto:Data: 2023-10-18T20:45:46.427 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.428 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.428 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.429 DEBUG:boto:Params: {} 2023-10-18T20:45:46.429 DEBUG:boto:Token: None 2023-10-18T20:45:46.430 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22 2023-10-18T20:45:46.430 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gFwXfz5jLo+WWA04ksT0MAL26Ss= 2023-10-18T20:45:46.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gFwXfz5jLo+WWA04ksT0MAL26Ss=', 'Content-Length': '0'} 2023-10-18T20:45:46.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000576d16f01846d1a5-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.478 DEBUG:boto:path=/bofrnf-59/obj22 2023-10-18T20:45:46.479 DEBUG:boto:auth_path=/bofrnf-59/obj22 2023-10-18T20:45:46.480 DEBUG:boto:Method: GET 2023-10-18T20:45:46.480 DEBUG:boto:Path: /bofrnf-59/obj22 2023-10-18T20:45:46.480 DEBUG:boto:Data: 2023-10-18T20:45:46.481 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.482 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.482 DEBUG:boto:Params: {} 2023-10-18T20:45:46.483 DEBUG:boto:Token: None 2023-10-18T20:45:46.483 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj22 2023-10-18T20:45:46.483 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gFwXfz5jLo+WWA04ksT0MAL26Ss= 2023-10-18T20:45:46.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gFwXfz5jLo+WWA04ksT0MAL26Ss=', 'Content-Length': '0'} 2023-10-18T20:45:46.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090d2bc9c3370b92f-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.487 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.488 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.488 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.488 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.489 DEBUG:boto:Method: GET 2023-10-18T20:45:46.489 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.489 DEBUG:boto:Data: 2023-10-18T20:45:46.490 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.490 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.491 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.491 DEBUG:boto:Params: {} 2023-10-18T20:45:46.492 DEBUG:boto:Token: None 2023-10-18T20:45:46.492 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.493 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000778a4bf4d2da16ae-00653043fa-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.496 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.497 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.498 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.498 DEBUG:boto:path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.499 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.499 DEBUG:boto:Method: GET 2023-10-18T20:45:46.500 DEBUG:boto:Path: /bofrnf-59/?prefix=obj22 2023-10-18T20:45:46.500 DEBUG:boto:Data: 2023-10-18T20:45:46.500 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.501 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.501 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.502 DEBUG:boto:Params: {} 2023-10-18T20:45:46.502 DEBUG:boto:Token: None 2023-10-18T20:45:46.503 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.503 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055e330c815a3d081-00653043fa-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.550 DEBUG:boto:b'bofrnf-59obj221000falseobj222023-10-18T20:32:33.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:45:46.553 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.553 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.554 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.555 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.556 DEBUG:boto:Method: GET 2023-10-18T20:45:46.556 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.557 DEBUG:boto:Data: 2023-10-18T20:45:46.557 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.558 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.559 DEBUG:boto:Params: {} 2023-10-18T20:45:46.559 DEBUG:boto:Token: None 2023-10-18T20:45:46.560 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dTHXXhGN4OmixJZM7PlzEZZtrWw= 2023-10-18T20:45:46.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dTHXXhGN4OmixJZM7PlzEZZtrWw=', 'Content-Length': '0'} 2023-10-18T20:45:46.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b3fb356993b95f-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.564 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.565 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.565 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.566 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.566 DEBUG:boto:Method: GET 2023-10-18T20:45:46.567 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.568 DEBUG:boto:Data: 2023-10-18T20:45:46.568 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.569 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.569 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.570 DEBUG:boto:Params: {} 2023-10-18T20:45:46.570 DEBUG:boto:Token: None 2023-10-18T20:45:46.571 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.571 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dTHXXhGN4OmixJZM7PlzEZZtrWw= 2023-10-18T20:45:46.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dTHXXhGN4OmixJZM7PlzEZZtrWw=', 'Content-Length': '0'} 2023-10-18T20:45:46.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002002446769a91ee6-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.575 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.576 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.576 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.577 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.578 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.578 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.579 DEBUG:boto:Data: 2023-10-18T20:45:46.580 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.580 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.581 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.582 DEBUG:boto:Params: {} 2023-10-18T20:45:46.583 DEBUG:boto:Token: None 2023-10-18T20:45:46.584 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.584 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0s2TooNnUMJc8cJzqZWfKHV9Pe8= 2023-10-18T20:45:46.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0s2TooNnUMJc8cJzqZWfKHV9Pe8=', 'Content-Length': '0'} 2023-10-18T20:45:46.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0a93581074d2c46-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.587 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.588 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.588 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.589 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.589 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.590 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.590 DEBUG:boto:Data: 2023-10-18T20:45:46.590 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.591 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.591 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.592 DEBUG:boto:Params: {} 2023-10-18T20:45:46.592 DEBUG:boto:Token: None 2023-10-18T20:45:46.593 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.593 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0s2TooNnUMJc8cJzqZWfKHV9Pe8= 2023-10-18T20:45:46.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0s2TooNnUMJc8cJzqZWfKHV9Pe8=', 'Content-Length': '0'} 2023-10-18T20:45:46.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d48eff531e00b182-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:45:46.597 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.597 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.598 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.598 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.599 DEBUG:boto:Method: GET 2023-10-18T20:45:46.600 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.601 DEBUG:boto:Data: 2023-10-18T20:45:46.602 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.603 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.604 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.605 DEBUG:boto:Params: {} 2023-10-18T20:45:46.606 DEBUG:boto:Token: None 2023-10-18T20:45:46.607 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.607 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dTHXXhGN4OmixJZM7PlzEZZtrWw= 2023-10-18T20:45:46.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dTHXXhGN4OmixJZM7PlzEZZtrWw=', 'Content-Length': '0'} 2023-10-18T20:45:46.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7a17292b24f210a-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.654 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.655 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.656 DEBUG:boto:path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.656 DEBUG:boto:auth_path=/bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.657 DEBUG:boto:Method: GET 2023-10-18T20:45:46.657 DEBUG:boto:Path: /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.658 DEBUG:boto:Data: 2023-10-18T20:45:46.659 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.660 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.660 DEBUG:boto:Params: {} 2023-10-18T20:45:46.661 DEBUG:boto:Token: None 2023-10-18T20:45:46.661 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23?versionId=null 2023-10-18T20:45:46.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dTHXXhGN4OmixJZM7PlzEZZtrWw= 2023-10-18T20:45:46.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dTHXXhGN4OmixJZM7PlzEZZtrWw=', 'Content-Length': '0'} 2023-10-18T20:45:46.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018a2492e2a5c4ddc-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.665 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.666 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.666 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.667 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.667 DEBUG:boto:Method: GET 2023-10-18T20:45:46.668 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.668 DEBUG:boto:Data: 2023-10-18T20:45:46.669 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.669 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.670 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.670 DEBUG:boto:Params: {} 2023-10-18T20:45:46.671 DEBUG:boto:Token: None 2023-10-18T20:45:46.672 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.672 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039e4dd68d9757a08-00653043fa-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.676 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.677 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.677 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.678 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.678 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.679 DEBUG:boto:Method: GET 2023-10-18T20:45:46.679 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.680 DEBUG:boto:Data: 2023-10-18T20:45:46.680 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.681 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.682 DEBUG:boto:Params: {} 2023-10-18T20:45:46.683 DEBUG:boto:Token: None 2023-10-18T20:45:46.683 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.684 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e82d54124f23852-00653043fa-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.690 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.691 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.691 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.692 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.692 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:45:46.693 DEBUG:boto:Data: 2023-10-18T20:45:46.694 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.694 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.696 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.696 DEBUG:boto:Params: {} 2023-10-18T20:45:46.697 DEBUG:boto:Token: None 2023-10-18T20:45:46.698 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23 2023-10-18T20:45:46.699 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FhnVSrp2IOtRJbpI4rUXNCB8ycE= 2023-10-18T20:45:46.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FhnVSrp2IOtRJbpI4rUXNCB8ycE=', 'Content-Length': '0'} 2023-10-18T20:45:46.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009b8f2167357c9a0-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.703 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.704 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.704 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.705 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:45:46.706 DEBUG:boto:Data: 2023-10-18T20:45:46.707 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.708 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.710 DEBUG:boto:Params: {} 2023-10-18T20:45:46.711 DEBUG:boto:Token: None 2023-10-18T20:45:46.711 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23 2023-10-18T20:45:46.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FhnVSrp2IOtRJbpI4rUXNCB8ycE= 2023-10-18T20:45:46.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FhnVSrp2IOtRJbpI4rUXNCB8ycE=', 'Content-Length': '0'} 2023-10-18T20:45:46.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ebb4f629bc6b7eb-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-18T20:45:46.717 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.718 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.718 DEBUG:boto:Method: GET 2023-10-18T20:45:46.719 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:45:46.720 DEBUG:boto:Data: 2023-10-18T20:45:46.723 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.724 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.724 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.725 DEBUG:boto:Params: {} 2023-10-18T20:45:46.725 DEBUG:boto:Token: None 2023-10-18T20:45:46.726 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23 2023-10-18T20:45:46.726 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O6iDxqOvN4j4Z+Vt5ZfnxzMYZrU= 2023-10-18T20:45:46.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O6iDxqOvN4j4Z+Vt5ZfnxzMYZrU=', 'Content-Length': '0'} 2023-10-18T20:45:46.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb2d30b01afde63d-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.730 DEBUG:boto:path=/bofrnf-59/obj23 2023-10-18T20:45:46.730 DEBUG:boto:auth_path=/bofrnf-59/obj23 2023-10-18T20:45:46.731 DEBUG:boto:Method: GET 2023-10-18T20:45:46.731 DEBUG:boto:Path: /bofrnf-59/obj23 2023-10-18T20:45:46.732 DEBUG:boto:Data: 2023-10-18T20:45:46.732 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.733 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.734 DEBUG:boto:Params: {} 2023-10-18T20:45:46.734 DEBUG:boto:Token: None 2023-10-18T20:45:46.735 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj23 2023-10-18T20:45:46.736 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O6iDxqOvN4j4Z+Vt5ZfnxzMYZrU= 2023-10-18T20:45:46.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O6iDxqOvN4j4Z+Vt5ZfnxzMYZrU=', 'Content-Length': '0'} 2023-10-18T20:45:46.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033e379f3e729d50d-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.782 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.783 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.784 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.784 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.784 DEBUG:boto:Method: GET 2023-10-18T20:45:46.785 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.785 DEBUG:boto:Data: 2023-10-18T20:45:46.786 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.786 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.787 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.787 DEBUG:boto:Params: {} 2023-10-18T20:45:46.787 DEBUG:boto:Token: None 2023-10-18T20:45:46.788 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096b297e1735ab1cd-00653043fa-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.792 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.793 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.794 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.794 DEBUG:boto:path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.794 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.795 DEBUG:boto:Method: GET 2023-10-18T20:45:46.795 DEBUG:boto:Path: /bofrnf-59/?prefix=obj23 2023-10-18T20:45:46.796 DEBUG:boto:Data: 2023-10-18T20:45:46.796 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.797 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.797 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.797 DEBUG:boto:Params: {} 2023-10-18T20:45:46.798 DEBUG:boto:Token: None 2023-10-18T20:45:46.799 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b7a40cfe081798c-00653043fa-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.804 DEBUG:boto:b'bofrnf-59obj231000falseobj232023-10-18T20:38:22.071Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:45:46.805 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:46.806 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:46.806 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.807 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.807 DEBUG:boto:Method: GET 2023-10-18T20:45:46.807 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.808 DEBUG:boto:Data: 2023-10-18T20:45:46.808 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.809 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.810 DEBUG:boto:Params: {} 2023-10-18T20:45:46.810 DEBUG:boto:Token: None 2023-10-18T20:45:46.811 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.811 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2lEFrKh2AMCBc3vRhpMdBWeKCn8= 2023-10-18T20:45:46.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2lEFrKh2AMCBc3vRhpMdBWeKCn8=', 'Content-Length': '0'} 2023-10-18T20:45:46.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aa2d904c822107b-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.854 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:46.855 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:46.855 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.856 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.856 DEBUG:boto:Method: GET 2023-10-18T20:45:46.857 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.857 DEBUG:boto:Data: 2023-10-18T20:45:46.857 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.858 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.859 DEBUG:boto:Params: {} 2023-10-18T20:45:46.859 DEBUG:boto:Token: None 2023-10-18T20:45:46.860 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2lEFrKh2AMCBc3vRhpMdBWeKCn8= 2023-10-18T20:45:46.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2lEFrKh2AMCBc3vRhpMdBWeKCn8=', 'Content-Length': '0'} 2023-10-18T20:45:46.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f81a59073353c6c-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.864 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:46.864 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:46.864 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.865 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.865 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.866 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.866 DEBUG:boto:Data: 2023-10-18T20:45:46.866 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.867 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.867 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.868 DEBUG:boto:Params: {} 2023-10-18T20:45:46.868 DEBUG:boto:Token: None 2023-10-18T20:45:46.869 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.869 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0EDqbhpVFhJgQBihq3JAKWBOosc= 2023-10-18T20:45:46.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0EDqbhpVFhJgQBihq3JAKWBOosc=', 'Content-Length': '0'} 2023-10-18T20:45:46.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f806e9273bf1ed-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.872 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:46.873 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:46.873 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.873 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.874 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.874 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.875 DEBUG:boto:Data: 2023-10-18T20:45:46.875 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.876 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.876 DEBUG:boto:Params: {} 2023-10-18T20:45:46.877 DEBUG:boto:Token: None 2023-10-18T20:45:46.877 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.878 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0EDqbhpVFhJgQBihq3JAKWBOosc= 2023-10-18T20:45:46.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0EDqbhpVFhJgQBihq3JAKWBOosc=', 'Content-Length': '0'} 2023-10-18T20:45:46.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af733c5faf371342-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:45:46.881 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:46.882 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:46.882 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.883 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.883 DEBUG:boto:Method: GET 2023-10-18T20:45:46.883 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.884 DEBUG:boto:Data: 2023-10-18T20:45:46.884 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.885 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.885 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.885 DEBUG:boto:Params: {} 2023-10-18T20:45:46.886 DEBUG:boto:Token: None 2023-10-18T20:45:46.886 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.887 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2lEFrKh2AMCBc3vRhpMdBWeKCn8= 2023-10-18T20:45:46.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2lEFrKh2AMCBc3vRhpMdBWeKCn8=', 'Content-Length': '0'} 2023-10-18T20:45:46.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f474890e4d8ee9-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.890 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:46.891 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:46.891 DEBUG:boto:path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.892 DEBUG:boto:auth_path=/bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.892 DEBUG:boto:Method: GET 2023-10-18T20:45:46.892 DEBUG:boto:Path: /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.893 DEBUG:boto:Data: 2023-10-18T20:45:46.893 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.894 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.894 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.895 DEBUG:boto:Params: {} 2023-10-18T20:45:46.895 DEBUG:boto:Token: None 2023-10-18T20:45:46.896 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj29?versionId=null 2023-10-18T20:45:46.896 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2lEFrKh2AMCBc3vRhpMdBWeKCn8= 2023-10-18T20:45:46.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2lEFrKh2AMCBc3vRhpMdBWeKCn8=', 'Content-Length': '0'} 2023-10-18T20:45:46.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d2cd00cfda2e07-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.899 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.900 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.900 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:46.900 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:46.901 DEBUG:boto:Method: GET 2023-10-18T20:45:46.901 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:45:46.902 DEBUG:boto:Data: 2023-10-18T20:45:46.902 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.902 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.903 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.903 DEBUG:boto:Params: {} 2023-10-18T20:45:46.904 DEBUG:boto:Token: None 2023-10-18T20:45:46.904 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.905 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef5679c56f530b39-00653043fa-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.908 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.909 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:46.909 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:46.910 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:46.910 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:46.911 DEBUG:boto:Method: GET 2023-10-18T20:45:46.911 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:45:46.911 DEBUG:boto:Data: 2023-10-18T20:45:46.912 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.912 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.913 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.913 DEBUG:boto:Params: {} 2023-10-18T20:45:46.914 DEBUG:boto:Token: None 2023-10-18T20:45:46.914 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/ 2023-10-18T20:45:46.915 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI= 2023-10-18T20:45:46.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vWXXZcf32W2jpB6vcJOhN2zaeMI=', 'Content-Length': '0'} 2023-10-18T20:45:46.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069183e27c8d1e1f5-00653043fa-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.962 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:46.963 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:46.964 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:46.964 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.965 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:45:46.965 DEBUG:boto:Data: 2023-10-18T20:45:46.966 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.966 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.966 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.967 DEBUG:boto:Params: {} 2023-10-18T20:45:46.967 DEBUG:boto:Token: None 2023-10-18T20:45:46.968 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj29 2023-10-18T20:45:46.968 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AGob5JgXJe39DNU0eQqu97X8V7o= 2023-10-18T20:45:46.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AGob5JgXJe39DNU0eQqu97X8V7o=', 'Content-Length': '0'} 2023-10-18T20:45:46.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a7ab5aff273fedf-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.971 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:46.972 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:46.972 DEBUG:boto:Method: HEAD 2023-10-18T20:45:46.972 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:45:46.973 DEBUG:boto:Data: 2023-10-18T20:45:46.973 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:46.974 DEBUG:boto:Port: 8000 2023-10-18T20:45:46.975 DEBUG:boto:Params: {} 2023-10-18T20:45:46.975 DEBUG:boto:Token: None 2023-10-18T20:45:46.976 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj29 2023-10-18T20:45:46.976 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AGob5JgXJe39DNU0eQqu97X8V7o= 2023-10-18T20:45:46.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AGob5JgXJe39DNU0eQqu97X8V7o=', 'Content-Length': '0'} 2023-10-18T20:45:46.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e54a5c4ad4bc8a5c-00653043fa-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:46.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-18T20:45:46.980 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:46.980 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:46.981 DEBUG:boto:Method: GET 2023-10-18T20:45:46.981 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:45:46.982 DEBUG:boto:Data: 2023-10-18T20:45:46.982 DEBUG:boto:Headers: {} 2023-10-18T20:45:46.982 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:46.983 DEBUG:boto:Port: 8001 2023-10-18T20:45:46.983 DEBUG:boto:Params: {} 2023-10-18T20:45:46.984 DEBUG:boto:Token: None 2023-10-18T20:45:46.984 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:46 GMT /bofrnf-59/obj29 2023-10-18T20:45:46.985 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:upSq3RHBScr3RXkuWyyDp5O4Iyk= 2023-10-18T20:45:46.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:46 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:upSq3RHBScr3RXkuWyyDp5O4Iyk=', 'Content-Length': '0'} 2023-10-18T20:45:46.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70d7d68b11678d7-00653043fa-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.030 DEBUG:boto:path=/bofrnf-59/obj29 2023-10-18T20:45:47.031 DEBUG:boto:auth_path=/bofrnf-59/obj29 2023-10-18T20:45:47.031 DEBUG:boto:Method: GET 2023-10-18T20:45:47.032 DEBUG:boto:Path: /bofrnf-59/obj29 2023-10-18T20:45:47.033 DEBUG:boto:Data: 2023-10-18T20:45:47.033 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.033 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.034 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.035 DEBUG:boto:Params: {} 2023-10-18T20:45:47.035 DEBUG:boto:Token: None 2023-10-18T20:45:47.036 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj29 2023-10-18T20:45:47.036 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0DAQGdCXRybpmgQVEapjbHE4nFo= 2023-10-18T20:45:47.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0DAQGdCXRybpmgQVEapjbHE4nFo=', 'Content-Length': '0'} 2023-10-18T20:45:47.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c75be805a8627ab-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.039 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.040 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.040 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:47.041 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:47.041 DEBUG:boto:Method: GET 2023-10-18T20:45:47.042 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:45:47.042 DEBUG:boto:Data: 2023-10-18T20:45:47.043 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.043 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.043 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.044 DEBUG:boto:Params: {} 2023-10-18T20:45:47.044 DEBUG:boto:Token: None 2023-10-18T20:45:47.045 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.045 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7750d3eed9fda94-00653043fb-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.049 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.050 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.050 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.051 DEBUG:boto:path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:47.051 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj29 2023-10-18T20:45:47.051 DEBUG:boto:Method: GET 2023-10-18T20:45:47.052 DEBUG:boto:Path: /bofrnf-59/?prefix=obj29 2023-10-18T20:45:47.052 DEBUG:boto:Data: 2023-10-18T20:45:47.053 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.053 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.053 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.054 DEBUG:boto:Params: {} 2023-10-18T20:45:47.054 DEBUG:boto:Token: None 2023-10-18T20:45:47.055 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.055 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027209ea4009c72d6-00653043fb-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.060 DEBUG:boto:b'bofrnf-59obj291000falseobj292023-10-18T20:39:46.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-18T20:45:47.061 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.061 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.062 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.062 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.063 DEBUG:boto:Method: GET 2023-10-18T20:45:47.063 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.063 DEBUG:boto:Data: 2023-10-18T20:45:47.064 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.064 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.065 DEBUG:boto:Params: {} 2023-10-18T20:45:47.065 DEBUG:boto:Token: None 2023-10-18T20:45:47.066 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.066 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5AqXiVF4IhLfBbjh/0yRoqQlrfE= 2023-10-18T20:45:47.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5AqXiVF4IhLfBbjh/0yRoqQlrfE=', 'Content-Length': '0'} 2023-10-18T20:45:47.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c396eb2a43f9d140-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.070 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.070 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.071 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.071 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.071 DEBUG:boto:Method: GET 2023-10-18T20:45:47.072 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.072 DEBUG:boto:Data: 2023-10-18T20:45:47.073 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.074 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.074 DEBUG:boto:Params: {} 2023-10-18T20:45:47.075 DEBUG:boto:Token: None 2023-10-18T20:45:47.075 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.076 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5AqXiVF4IhLfBbjh/0yRoqQlrfE= 2023-10-18T20:45:47.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5AqXiVF4IhLfBbjh/0yRoqQlrfE=', 'Content-Length': '0'} 2023-10-18T20:45:47.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b9b8f8e156dc8a9-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.118 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.119 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.119 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.119 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.120 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.120 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.121 DEBUG:boto:Data: 2023-10-18T20:45:47.121 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.121 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.122 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.122 DEBUG:boto:Params: {} 2023-10-18T20:45:47.123 DEBUG:boto:Token: None 2023-10-18T20:45:47.123 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.123 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DT/5DA4uZACYY5xr5V5KYDJoN1M= 2023-10-18T20:45:47.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DT/5DA4uZACYY5xr5V5KYDJoN1M=', 'Content-Length': '0'} 2023-10-18T20:45:47.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a047a744ff76cb3-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.126 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.127 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.127 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.129 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.129 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.130 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.130 DEBUG:boto:Data: 2023-10-18T20:45:47.130 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.131 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.131 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.132 DEBUG:boto:Params: {} 2023-10-18T20:45:47.132 DEBUG:boto:Token: None 2023-10-18T20:45:47.132 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DT/5DA4uZACYY5xr5V5KYDJoN1M= 2023-10-18T20:45:47.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DT/5DA4uZACYY5xr5V5KYDJoN1M=', 'Content-Length': '0'} 2023-10-18T20:45:47.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d260657023d0bc1-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-18T20:45:47.136 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.137 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.137 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.137 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.138 DEBUG:boto:Method: GET 2023-10-18T20:45:47.138 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.139 DEBUG:boto:Data: 2023-10-18T20:45:47.139 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.139 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.140 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.140 DEBUG:boto:Params: {} 2023-10-18T20:45:47.141 DEBUG:boto:Token: None 2023-10-18T20:45:47.141 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.141 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5AqXiVF4IhLfBbjh/0yRoqQlrfE= 2023-10-18T20:45:47.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5AqXiVF4IhLfBbjh/0yRoqQlrfE=', 'Content-Length': '0'} 2023-10-18T20:45:47.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068b20e4660c7e783-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.145 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.145 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.146 DEBUG:boto:path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.146 DEBUG:boto:auth_path=/bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.146 DEBUG:boto:Method: GET 2023-10-18T20:45:47.147 DEBUG:boto:Path: /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.147 DEBUG:boto:Data: 2023-10-18T20:45:47.148 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.148 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.148 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.149 DEBUG:boto:Params: {} 2023-10-18T20:45:47.149 DEBUG:boto:Token: None 2023-10-18T20:45:47.150 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31?versionId=null 2023-10-18T20:45:47.150 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5AqXiVF4IhLfBbjh/0yRoqQlrfE= 2023-10-18T20:45:47.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5AqXiVF4IhLfBbjh/0yRoqQlrfE=', 'Content-Length': '0'} 2023-10-18T20:45:47.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007882bbd1bbe8848f-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.153 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.154 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.154 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.155 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.155 DEBUG:boto:Method: GET 2023-10-18T20:45:47.155 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.156 DEBUG:boto:Data: 2023-10-18T20:45:47.156 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.156 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.157 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.157 DEBUG:boto:Params: {} 2023-10-18T20:45:47.158 DEBUG:boto:Token: None 2023-10-18T20:45:47.158 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.159 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055fdd1103d5b3f12-00653043fb-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.162 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.163 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.163 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.164 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.164 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.165 DEBUG:boto:Method: GET 2023-10-18T20:45:47.165 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.165 DEBUG:boto:Data: 2023-10-18T20:45:47.166 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.166 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.167 DEBUG:boto:Params: {} 2023-10-18T20:45:47.167 DEBUG:boto:Token: None 2023-10-18T20:45:47.168 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c169088ca80ef2-00653043fb-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.173 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.174 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.174 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.175 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.175 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:45:47.176 DEBUG:boto:Data: 2023-10-18T20:45:47.176 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.176 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.177 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.177 DEBUG:boto:Params: {} 2023-10-18T20:45:47.177 DEBUG:boto:Token: None 2023-10-18T20:45:47.178 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31 2023-10-18T20:45:47.178 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HekROeaNHzt0aRV54/sRgQWfXFQ= 2023-10-18T20:45:47.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HekROeaNHzt0aRV54/sRgQWfXFQ=', 'Content-Length': '0'} 2023-10-18T20:45:47.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2a05f3253157293-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.181 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.182 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.182 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.183 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:45:47.183 DEBUG:boto:Data: 2023-10-18T20:45:47.183 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.184 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.184 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.185 DEBUG:boto:Params: {} 2023-10-18T20:45:47.185 DEBUG:boto:Token: None 2023-10-18T20:45:47.185 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31 2023-10-18T20:45:47.186 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HekROeaNHzt0aRV54/sRgQWfXFQ= 2023-10-18T20:45:47.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HekROeaNHzt0aRV54/sRgQWfXFQ=', 'Content-Length': '0'} 2023-10-18T20:45:47.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f325782275522d0-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-18T20:45:47.189 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.190 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.190 DEBUG:boto:Method: GET 2023-10-18T20:45:47.191 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:45:47.191 DEBUG:boto:Data: 2023-10-18T20:45:47.191 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.192 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.192 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.192 DEBUG:boto:Params: {} 2023-10-18T20:45:47.193 DEBUG:boto:Token: None 2023-10-18T20:45:47.193 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31 2023-10-18T20:45:47.194 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Krzjam1tKR84/8C2C3iKoK2dUWA= 2023-10-18T20:45:47.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Krzjam1tKR84/8C2C3iKoK2dUWA=', 'Content-Length': '0'} 2023-10-18T20:45:47.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf9c8d3c1f89ecc9-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.197 DEBUG:boto:path=/bofrnf-59/obj31 2023-10-18T20:45:47.197 DEBUG:boto:auth_path=/bofrnf-59/obj31 2023-10-18T20:45:47.198 DEBUG:boto:Method: GET 2023-10-18T20:45:47.198 DEBUG:boto:Path: /bofrnf-59/obj31 2023-10-18T20:45:47.199 DEBUG:boto:Data: 2023-10-18T20:45:47.199 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.200 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.200 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.200 DEBUG:boto:Params: {} 2023-10-18T20:45:47.201 DEBUG:boto:Token: None 2023-10-18T20:45:47.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj31 2023-10-18T20:45:47.202 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Krzjam1tKR84/8C2C3iKoK2dUWA= 2023-10-18T20:45:47.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Krzjam1tKR84/8C2C3iKoK2dUWA=', 'Content-Length': '0'} 2023-10-18T20:45:47.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f5cde4a16cdd81-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.250 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.251 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.251 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.252 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.252 DEBUG:boto:Method: GET 2023-10-18T20:45:47.253 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.253 DEBUG:boto:Data: 2023-10-18T20:45:47.253 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.254 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.254 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.255 DEBUG:boto:Params: {} 2023-10-18T20:45:47.255 DEBUG:boto:Token: None 2023-10-18T20:45:47.255 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.256 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4dbe6924e90648d-00653043fb-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.259 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.260 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.260 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.261 DEBUG:boto:path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.261 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.261 DEBUG:boto:Method: GET 2023-10-18T20:45:47.262 DEBUG:boto:Path: /bofrnf-59/?prefix=obj31 2023-10-18T20:45:47.262 DEBUG:boto:Data: 2023-10-18T20:45:47.263 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.263 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.264 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.264 DEBUG:boto:Params: {} 2023-10-18T20:45:47.265 DEBUG:boto:Token: None 2023-10-18T20:45:47.265 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.265 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000991ceb1e0eb20eb-00653043fb-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.271 DEBUG:boto:b'bofrnf-59obj311000falseobj312023-10-18T20:41:07.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:45:47.272 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.273 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.273 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.273 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.274 DEBUG:boto:Method: GET 2023-10-18T20:45:47.274 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.275 DEBUG:boto:Data: 2023-10-18T20:45:47.275 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.275 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.276 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.276 DEBUG:boto:Params: {} 2023-10-18T20:45:47.277 DEBUG:boto:Token: None 2023-10-18T20:45:47.277 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.278 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E1uVceadRoj98Sr4WYMqKCUpCAs= 2023-10-18T20:45:47.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E1uVceadRoj98Sr4WYMqKCUpCAs=', 'Content-Length': '0'} 2023-10-18T20:45:47.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729708bde9237eb4-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.281 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.282 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.282 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.283 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.283 DEBUG:boto:Method: GET 2023-10-18T20:45:47.283 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.284 DEBUG:boto:Data: 2023-10-18T20:45:47.284 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.285 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.285 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.285 DEBUG:boto:Params: {} 2023-10-18T20:45:47.286 DEBUG:boto:Token: None 2023-10-18T20:45:47.286 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.287 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E1uVceadRoj98Sr4WYMqKCUpCAs= 2023-10-18T20:45:47.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E1uVceadRoj98Sr4WYMqKCUpCAs=', 'Content-Length': '0'} 2023-10-18T20:45:47.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e858fa2631bd8172-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.290 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.291 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.291 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.291 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.292 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.292 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.293 DEBUG:boto:Data: 2023-10-18T20:45:47.293 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.293 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.294 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.294 DEBUG:boto:Params: {} 2023-10-18T20:45:47.295 DEBUG:boto:Token: None 2023-10-18T20:45:47.295 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.296 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EFWK8Tn6N9L4kyDfx9dGQqurRmA= 2023-10-18T20:45:47.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EFWK8Tn6N9L4kyDfx9dGQqurRmA=', 'Content-Length': '0'} 2023-10-18T20:45:47.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d2a18e878ec9975-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.299 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.299 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.299 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.300 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.300 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.301 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.301 DEBUG:boto:Data: 2023-10-18T20:45:47.302 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.302 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.303 DEBUG:boto:Params: {} 2023-10-18T20:45:47.303 DEBUG:boto:Token: None 2023-10-18T20:45:47.304 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EFWK8Tn6N9L4kyDfx9dGQqurRmA= 2023-10-18T20:45:47.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EFWK8Tn6N9L4kyDfx9dGQqurRmA=', 'Content-Length': '0'} 2023-10-18T20:45:47.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bcba4ab7e23234c-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:45:47.307 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.308 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.308 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.308 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.309 DEBUG:boto:Method: GET 2023-10-18T20:45:47.309 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.310 DEBUG:boto:Data: 2023-10-18T20:45:47.310 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.311 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.311 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.311 DEBUG:boto:Params: {} 2023-10-18T20:45:47.312 DEBUG:boto:Token: None 2023-10-18T20:45:47.312 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E1uVceadRoj98Sr4WYMqKCUpCAs= 2023-10-18T20:45:47.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E1uVceadRoj98Sr4WYMqKCUpCAs=', 'Content-Length': '0'} 2023-10-18T20:45:47.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a53e9510ca84e41-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.316 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.316 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.317 DEBUG:boto:path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.317 DEBUG:boto:auth_path=/bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.317 DEBUG:boto:Method: GET 2023-10-18T20:45:47.318 DEBUG:boto:Path: /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.318 DEBUG:boto:Data: 2023-10-18T20:45:47.319 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.319 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.320 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.320 DEBUG:boto:Params: {} 2023-10-18T20:45:47.320 DEBUG:boto:Token: None 2023-10-18T20:45:47.321 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33?versionId=null 2023-10-18T20:45:47.321 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E1uVceadRoj98Sr4WYMqKCUpCAs= 2023-10-18T20:45:47.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E1uVceadRoj98Sr4WYMqKCUpCAs=', 'Content-Length': '0'} 2023-10-18T20:45:47.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3dd34710fcab787-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.370 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.371 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.371 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.372 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.372 DEBUG:boto:Method: GET 2023-10-18T20:45:47.373 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.373 DEBUG:boto:Data: 2023-10-18T20:45:47.374 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.375 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.375 DEBUG:boto:Params: {} 2023-10-18T20:45:47.376 DEBUG:boto:Token: None 2023-10-18T20:45:47.376 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.377 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e8d0b50a1913068-00653043fb-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.380 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.381 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.382 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.382 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.383 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.383 DEBUG:boto:Method: GET 2023-10-18T20:45:47.384 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.384 DEBUG:boto:Data: 2023-10-18T20:45:47.384 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.385 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.385 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.386 DEBUG:boto:Params: {} 2023-10-18T20:45:47.386 DEBUG:boto:Token: None 2023-10-18T20:45:47.387 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.387 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003881d3acc4d97444-00653043fb-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.393 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.393 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.394 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.394 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.395 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:45:47.395 DEBUG:boto:Data: 2023-10-18T20:45:47.395 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.396 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.396 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.397 DEBUG:boto:Params: {} 2023-10-18T20:45:47.397 DEBUG:boto:Token: None 2023-10-18T20:45:47.398 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33 2023-10-18T20:45:47.398 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QoJNEaUyA7Mgo9ERupPtBEsevmY= 2023-10-18T20:45:47.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QoJNEaUyA7Mgo9ERupPtBEsevmY=', 'Content-Length': '0'} 2023-10-18T20:45:47.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9c9a35ac2e44999-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.401 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.402 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.402 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.403 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:45:47.403 DEBUG:boto:Data: 2023-10-18T20:45:47.404 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.404 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.404 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.405 DEBUG:boto:Params: {} 2023-10-18T20:45:47.405 DEBUG:boto:Token: None 2023-10-18T20:45:47.406 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33 2023-10-18T20:45:47.406 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QoJNEaUyA7Mgo9ERupPtBEsevmY= 2023-10-18T20:45:47.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QoJNEaUyA7Mgo9ERupPtBEsevmY=', 'Content-Length': '0'} 2023-10-18T20:45:47.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002574476bd4295149-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:45:47.410 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.410 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.410 DEBUG:boto:Method: GET 2023-10-18T20:45:47.411 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:45:47.411 DEBUG:boto:Data: 2023-10-18T20:45:47.412 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.412 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.413 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.413 DEBUG:boto:Params: {} 2023-10-18T20:45:47.414 DEBUG:boto:Token: None 2023-10-18T20:45:47.414 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33 2023-10-18T20:45:47.415 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mQ4/5JQ5YKTwsxmC2sb8WmuZ+Dw= 2023-10-18T20:45:47.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mQ4/5JQ5YKTwsxmC2sb8WmuZ+Dw=', 'Content-Length': '0'} 2023-10-18T20:45:47.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000976454d81a6d7544-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.462 DEBUG:boto:path=/bofrnf-59/obj33 2023-10-18T20:45:47.463 DEBUG:boto:auth_path=/bofrnf-59/obj33 2023-10-18T20:45:47.463 DEBUG:boto:Method: GET 2023-10-18T20:45:47.464 DEBUG:boto:Path: /bofrnf-59/obj33 2023-10-18T20:45:47.464 DEBUG:boto:Data: 2023-10-18T20:45:47.465 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.465 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.466 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.466 DEBUG:boto:Params: {} 2023-10-18T20:45:47.467 DEBUG:boto:Token: None 2023-10-18T20:45:47.467 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj33 2023-10-18T20:45:47.467 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mQ4/5JQ5YKTwsxmC2sb8WmuZ+Dw= 2023-10-18T20:45:47.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mQ4/5JQ5YKTwsxmC2sb8WmuZ+Dw=', 'Content-Length': '0'} 2023-10-18T20:45:47.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d83ae856a60299b-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.471 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.471 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.472 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.472 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.473 DEBUG:boto:Method: GET 2023-10-18T20:45:47.473 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.473 DEBUG:boto:Data: 2023-10-18T20:45:47.474 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.474 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.475 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.475 DEBUG:boto:Params: {} 2023-10-18T20:45:47.475 DEBUG:boto:Token: None 2023-10-18T20:45:47.476 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009924e5ae26351892-00653043fb-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.481 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.482 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.483 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.484 DEBUG:boto:path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.484 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.485 DEBUG:boto:Method: GET 2023-10-18T20:45:47.486 DEBUG:boto:Path: /bofrnf-59/?prefix=obj33 2023-10-18T20:45:47.487 DEBUG:boto:Data: 2023-10-18T20:45:47.488 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.488 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.489 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.489 DEBUG:boto:Params: {} 2023-10-18T20:45:47.490 DEBUG:boto:Token: None 2023-10-18T20:45:47.491 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.492 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554b97625652880f-00653043fb-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.538 DEBUG:boto:b'bofrnf-59obj331000falseobj332023-10-18T20:32:33.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:45:47.540 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.540 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.541 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.541 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.542 DEBUG:boto:Method: GET 2023-10-18T20:45:47.543 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.543 DEBUG:boto:Data: 2023-10-18T20:45:47.544 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.545 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.545 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.546 DEBUG:boto:Params: {} 2023-10-18T20:45:47.547 DEBUG:boto:Token: None 2023-10-18T20:45:47.548 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.548 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TFNon5YNMsymEHIggUe0ugTsxqk= 2023-10-18T20:45:47.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TFNon5YNMsymEHIggUe0ugTsxqk=', 'Content-Length': '0'} 2023-10-18T20:45:47.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000071694989a914680-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.553 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.554 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.555 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.555 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.556 DEBUG:boto:Method: GET 2023-10-18T20:45:47.557 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.557 DEBUG:boto:Data: 2023-10-18T20:45:47.558 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.558 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.559 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.560 DEBUG:boto:Params: {} 2023-10-18T20:45:47.560 DEBUG:boto:Token: None 2023-10-18T20:45:47.561 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.561 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TFNon5YNMsymEHIggUe0ugTsxqk= 2023-10-18T20:45:47.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TFNon5YNMsymEHIggUe0ugTsxqk=', 'Content-Length': '0'} 2023-10-18T20:45:47.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df41151e56a22bce-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.565 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.565 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.566 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.566 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.566 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.567 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.567 DEBUG:boto:Data: 2023-10-18T20:45:47.568 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.568 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.569 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.569 DEBUG:boto:Params: {} 2023-10-18T20:45:47.569 DEBUG:boto:Token: None 2023-10-18T20:45:47.570 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.570 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MxweVpOmBnkJhj9S99UnyCAbmhI= 2023-10-18T20:45:47.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MxweVpOmBnkJhj9S99UnyCAbmhI=', 'Content-Length': '0'} 2023-10-18T20:45:47.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022edf6b55e089671-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.574 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.574 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.574 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.575 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.575 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.576 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.576 DEBUG:boto:Data: 2023-10-18T20:45:47.576 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.577 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.577 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.578 DEBUG:boto:Params: {} 2023-10-18T20:45:47.578 DEBUG:boto:Token: None 2023-10-18T20:45:47.579 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.579 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MxweVpOmBnkJhj9S99UnyCAbmhI= 2023-10-18T20:45:47.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MxweVpOmBnkJhj9S99UnyCAbmhI=', 'Content-Length': '0'} 2023-10-18T20:45:47.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045ec6d74b25b8f7e-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:45:47.583 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.584 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.585 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.585 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.586 DEBUG:boto:Method: GET 2023-10-18T20:45:47.587 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.587 DEBUG:boto:Data: 2023-10-18T20:45:47.588 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.588 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.589 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.589 DEBUG:boto:Params: {} 2023-10-18T20:45:47.590 DEBUG:boto:Token: None 2023-10-18T20:45:47.590 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.591 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TFNon5YNMsymEHIggUe0ugTsxqk= 2023-10-18T20:45:47.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TFNon5YNMsymEHIggUe0ugTsxqk=', 'Content-Length': '0'} 2023-10-18T20:45:47.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b4d0226cc888a2-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.594 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.594 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.595 DEBUG:boto:path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.595 DEBUG:boto:auth_path=/bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.595 DEBUG:boto:Method: GET 2023-10-18T20:45:47.596 DEBUG:boto:Path: /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.596 DEBUG:boto:Data: 2023-10-18T20:45:47.597 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.597 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.598 DEBUG:boto:Params: {} 2023-10-18T20:45:47.598 DEBUG:boto:Token: None 2023-10-18T20:45:47.599 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34?versionId=null 2023-10-18T20:45:47.599 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TFNon5YNMsymEHIggUe0ugTsxqk= 2023-10-18T20:45:47.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TFNon5YNMsymEHIggUe0ugTsxqk=', 'Content-Length': '0'} 2023-10-18T20:45:47.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dcc8379a4668404-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.603 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.603 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.604 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.604 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.604 DEBUG:boto:Method: GET 2023-10-18T20:45:47.605 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.605 DEBUG:boto:Data: 2023-10-18T20:45:47.606 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.606 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.607 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.607 DEBUG:boto:Params: {} 2023-10-18T20:45:47.607 DEBUG:boto:Token: None 2023-10-18T20:45:47.608 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009881a1176ae9ff44-00653043fb-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.613 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.615 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.615 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.616 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.616 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.617 DEBUG:boto:Method: GET 2023-10-18T20:45:47.617 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.617 DEBUG:boto:Data: 2023-10-18T20:45:47.618 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.618 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.619 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.619 DEBUG:boto:Params: {} 2023-10-18T20:45:47.620 DEBUG:boto:Token: None 2023-10-18T20:45:47.620 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.621 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d28023c46b6af14-00653043fb-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.627 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.628 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.629 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.629 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.630 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:45:47.631 DEBUG:boto:Data: 2023-10-18T20:45:47.631 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.632 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.633 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.634 DEBUG:boto:Params: {} 2023-10-18T20:45:47.634 DEBUG:boto:Token: None 2023-10-18T20:45:47.635 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34 2023-10-18T20:45:47.636 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wUC/MhkIQG6gTWUXDoYyjekKIgY= 2023-10-18T20:45:47.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wUC/MhkIQG6gTWUXDoYyjekKIgY=', 'Content-Length': '0'} 2023-10-18T20:45:47.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa96fd8f13683481-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.640 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.640 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.641 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.641 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:45:47.642 DEBUG:boto:Data: 2023-10-18T20:45:47.642 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.643 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.643 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.643 DEBUG:boto:Params: {} 2023-10-18T20:45:47.644 DEBUG:boto:Token: None 2023-10-18T20:45:47.644 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34 2023-10-18T20:45:47.645 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wUC/MhkIQG6gTWUXDoYyjekKIgY= 2023-10-18T20:45:47.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wUC/MhkIQG6gTWUXDoYyjekKIgY=', 'Content-Length': '0'} 2023-10-18T20:45:47.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079dac4f5ac089658-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-18T20:45:47.648 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.649 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.649 DEBUG:boto:Method: GET 2023-10-18T20:45:47.650 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:45:47.650 DEBUG:boto:Data: 2023-10-18T20:45:47.651 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.651 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.651 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.652 DEBUG:boto:Params: {} 2023-10-18T20:45:47.652 DEBUG:boto:Token: None 2023-10-18T20:45:47.653 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34 2023-10-18T20:45:47.653 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6D1xnyVdzwI7yxZphppjdvvO9fw= 2023-10-18T20:45:47.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6D1xnyVdzwI7yxZphppjdvvO9fw=', 'Content-Length': '0'} 2023-10-18T20:45:47.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039aa6acadfe97e20-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.656 DEBUG:boto:path=/bofrnf-59/obj34 2023-10-18T20:45:47.657 DEBUG:boto:auth_path=/bofrnf-59/obj34 2023-10-18T20:45:47.657 DEBUG:boto:Method: GET 2023-10-18T20:45:47.657 DEBUG:boto:Path: /bofrnf-59/obj34 2023-10-18T20:45:47.658 DEBUG:boto:Data: 2023-10-18T20:45:47.658 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.659 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.660 DEBUG:boto:Params: {} 2023-10-18T20:45:47.660 DEBUG:boto:Token: None 2023-10-18T20:45:47.660 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj34 2023-10-18T20:45:47.661 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6D1xnyVdzwI7yxZphppjdvvO9fw= 2023-10-18T20:45:47.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6D1xnyVdzwI7yxZphppjdvvO9fw=', 'Content-Length': '0'} 2023-10-18T20:45:47.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4e4e2baca5868a-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.710 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.711 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.711 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.712 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.712 DEBUG:boto:Method: GET 2023-10-18T20:45:47.713 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.714 DEBUG:boto:Data: 2023-10-18T20:45:47.714 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.716 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.717 DEBUG:boto:Params: {} 2023-10-18T20:45:47.718 DEBUG:boto:Token: None 2023-10-18T20:45:47.718 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.719 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ef992dcaa4bec4e-00653043fb-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.724 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.724 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.725 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.725 DEBUG:boto:path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.726 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.726 DEBUG:boto:Method: GET 2023-10-18T20:45:47.727 DEBUG:boto:Path: /bofrnf-59/?prefix=obj34 2023-10-18T20:45:47.727 DEBUG:boto:Data: 2023-10-18T20:45:47.727 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.728 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.728 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.729 DEBUG:boto:Params: {} 2023-10-18T20:45:47.729 DEBUG:boto:Token: None 2023-10-18T20:45:47.730 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.730 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000304b2f1c6ed7e892-00653043fb-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.735 DEBUG:boto:b'bofrnf-59obj341000falseobj342023-10-18T20:33:05.349Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-18T20:45:47.736 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.737 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.737 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.738 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.738 DEBUG:boto:Method: GET 2023-10-18T20:45:47.738 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.739 DEBUG:boto:Data: 2023-10-18T20:45:47.739 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.741 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.741 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.742 DEBUG:boto:Params: {} 2023-10-18T20:45:47.742 DEBUG:boto:Token: None 2023-10-18T20:45:47.743 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.743 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vN0hdLyU4NfAiIGP0gpVOyXI7tA= 2023-10-18T20:45:47.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vN0hdLyU4NfAiIGP0gpVOyXI7tA=', 'Content-Length': '0'} 2023-10-18T20:45:47.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba64f9458cc16af8-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.794 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.795 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.795 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.796 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.796 DEBUG:boto:Method: GET 2023-10-18T20:45:47.797 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.797 DEBUG:boto:Data: 2023-10-18T20:45:47.798 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.798 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.799 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.799 DEBUG:boto:Params: {} 2023-10-18T20:45:47.800 DEBUG:boto:Token: None 2023-10-18T20:45:47.800 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.800 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vN0hdLyU4NfAiIGP0gpVOyXI7tA= 2023-10-18T20:45:47.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vN0hdLyU4NfAiIGP0gpVOyXI7tA=', 'Content-Length': '0'} 2023-10-18T20:45:47.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d480f3bb6a23628-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.804 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.804 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.805 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.805 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.806 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.806 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.807 DEBUG:boto:Data: 2023-10-18T20:45:47.807 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.808 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.808 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.808 DEBUG:boto:Params: {} 2023-10-18T20:45:47.809 DEBUG:boto:Token: None 2023-10-18T20:45:47.809 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.810 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AN4wG+5N3XLNj9gcMDkx/+gSUIw= 2023-10-18T20:45:47.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AN4wG+5N3XLNj9gcMDkx/+gSUIw=', 'Content-Length': '0'} 2023-10-18T20:45:47.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001beed274c8bb131f-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.813 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.813 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.814 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.814 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.815 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.815 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.816 DEBUG:boto:Data: 2023-10-18T20:45:47.816 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.816 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.817 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.818 DEBUG:boto:Params: {} 2023-10-18T20:45:47.818 DEBUG:boto:Token: None 2023-10-18T20:45:47.819 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.819 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AN4wG+5N3XLNj9gcMDkx/+gSUIw= 2023-10-18T20:45:47.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AN4wG+5N3XLNj9gcMDkx/+gSUIw=', 'Content-Length': '0'} 2023-10-18T20:45:47.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c200737a4f3c26bf-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-18T20:45:47.823 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.823 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.824 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.824 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.824 DEBUG:boto:Method: GET 2023-10-18T20:45:47.825 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.825 DEBUG:boto:Data: 2023-10-18T20:45:47.826 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.826 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.827 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.827 DEBUG:boto:Params: {} 2023-10-18T20:45:47.827 DEBUG:boto:Token: None 2023-10-18T20:45:47.828 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.828 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vN0hdLyU4NfAiIGP0gpVOyXI7tA= 2023-10-18T20:45:47.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vN0hdLyU4NfAiIGP0gpVOyXI7tA=', 'Content-Length': '0'} 2023-10-18T20:45:47.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000976578cc3027f773-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.832 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.832 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.833 DEBUG:boto:path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.833 DEBUG:boto:auth_path=/bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.834 DEBUG:boto:Method: GET 2023-10-18T20:45:47.835 DEBUG:boto:Path: /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.836 DEBUG:boto:Data: 2023-10-18T20:45:47.836 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.837 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.838 DEBUG:boto:Params: {} 2023-10-18T20:45:47.839 DEBUG:boto:Token: None 2023-10-18T20:45:47.840 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37?versionId=null 2023-10-18T20:45:47.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vN0hdLyU4NfAiIGP0gpVOyXI7tA= 2023-10-18T20:45:47.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vN0hdLyU4NfAiIGP0gpVOyXI7tA=', 'Content-Length': '0'} 2023-10-18T20:45:47.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3d3368e702011a-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.843 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.844 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.844 DEBUG:boto:path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.845 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.845 DEBUG:boto:Method: GET 2023-10-18T20:45:47.845 DEBUG:boto:Path: /bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.846 DEBUG:boto:Data: 2023-10-18T20:45:47.846 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.847 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.847 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.848 DEBUG:boto:Params: {} 2023-10-18T20:45:47.848 DEBUG:boto:Token: None 2023-10-18T20:45:47.849 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.849 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000168e509758612b14-00653043fb-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.854 DEBUG:boto:b'bofrnf-59obj371000falseobj372023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.855 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.856 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.856 DEBUG:boto:path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.857 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.858 DEBUG:boto:Method: GET 2023-10-18T20:45:47.859 DEBUG:boto:Path: /bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.859 DEBUG:boto:Data: 2023-10-18T20:45:47.860 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.860 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.860 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.861 DEBUG:boto:Params: {} 2023-10-18T20:45:47.861 DEBUG:boto:Token: None 2023-10-18T20:45:47.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000503d45080c0a6536-00653043fb-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.910 DEBUG:boto:b'bofrnf-59obj371000falseobj372023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.912 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.912 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.913 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.913 DEBUG:boto:Path: /bofrnf-59/obj37 2023-10-18T20:45:47.914 DEBUG:boto:Data: 2023-10-18T20:45:47.914 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.914 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.915 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.915 DEBUG:boto:Params: {} 2023-10-18T20:45:47.916 DEBUG:boto:Token: None 2023-10-18T20:45:47.916 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37 2023-10-18T20:45:47.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lQgwKGKC/T/mBeUGlOn/XL2Fhx4= 2023-10-18T20:45:47.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lQgwKGKC/T/mBeUGlOn/XL2Fhx4=', 'Content-Length': '0'} 2023-10-18T20:45:47.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e96cd3ee2a0ef03-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.920 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.920 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.921 DEBUG:boto:Method: HEAD 2023-10-18T20:45:47.921 DEBUG:boto:Path: /bofrnf-59/obj37 2023-10-18T20:45:47.922 DEBUG:boto:Data: 2023-10-18T20:45:47.922 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.923 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.923 DEBUG:boto:Params: {} 2023-10-18T20:45:47.924 DEBUG:boto:Token: None 2023-10-18T20:45:47.924 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37 2023-10-18T20:45:47.925 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lQgwKGKC/T/mBeUGlOn/XL2Fhx4= 2023-10-18T20:45:47.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lQgwKGKC/T/mBeUGlOn/XL2Fhx4=', 'Content-Length': '0'} 2023-10-18T20:45:47.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f2de6f36086667-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-18T20:45:47.929 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.929 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.930 DEBUG:boto:Method: GET 2023-10-18T20:45:47.930 DEBUG:boto:Path: /bofrnf-59/obj37 2023-10-18T20:45:47.931 DEBUG:boto:Data: 2023-10-18T20:45:47.931 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.932 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.932 DEBUG:boto:Params: {} 2023-10-18T20:45:47.933 DEBUG:boto:Token: None 2023-10-18T20:45:47.934 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37 2023-10-18T20:45:47.935 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TF1uKw/1qCMzCQ98NvGp94yOS+M= 2023-10-18T20:45:47.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TF1uKw/1qCMzCQ98NvGp94yOS+M=', 'Content-Length': '0'} 2023-10-18T20:45:47.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b33644cc92ed338-00653043fb-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.978 DEBUG:boto:path=/bofrnf-59/obj37 2023-10-18T20:45:47.979 DEBUG:boto:auth_path=/bofrnf-59/obj37 2023-10-18T20:45:47.979 DEBUG:boto:Method: GET 2023-10-18T20:45:47.980 DEBUG:boto:Path: /bofrnf-59/obj37 2023-10-18T20:45:47.980 DEBUG:boto:Data: 2023-10-18T20:45:47.981 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.981 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:47.982 DEBUG:boto:Port: 8000 2023-10-18T20:45:47.982 DEBUG:boto:Params: {} 2023-10-18T20:45:47.983 DEBUG:boto:Token: None 2023-10-18T20:45:47.983 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/obj37 2023-10-18T20:45:47.984 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TF1uKw/1qCMzCQ98NvGp94yOS+M= 2023-10-18T20:45:47.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TF1uKw/1qCMzCQ98NvGp94yOS+M=', 'Content-Length': '0'} 2023-10-18T20:45:47.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004356ed5d72987f8c-00653043fb-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.987 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.987 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.988 DEBUG:boto:path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.988 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.989 DEBUG:boto:Method: GET 2023-10-18T20:45:47.989 DEBUG:boto:Path: /bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.989 DEBUG:boto:Data: 2023-10-18T20:45:47.990 DEBUG:boto:Headers: {} 2023-10-18T20:45:47.990 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:47.991 DEBUG:boto:Port: 8001 2023-10-18T20:45:47.991 DEBUG:boto:Params: {} 2023-10-18T20:45:47.992 DEBUG:boto:Token: None 2023-10-18T20:45:47.992 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:47 GMT /bofrnf-59/ 2023-10-18T20:45:47.993 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw= 2023-10-18T20:45:47.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9igA991s8sWtwxOZEFI53G9Pmgw=', 'Content-Length': '0'} 2023-10-18T20:45:47.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f1499a426950f96-00653043fb-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:47.996 DEBUG:boto:b'bofrnf-59obj371000falseobj372023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:47.997 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:47.998 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:47.998 DEBUG:boto:path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.999 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj37 2023-10-18T20:45:47.999 DEBUG:boto:Method: GET 2023-10-18T20:45:48.000 DEBUG:boto:Path: /bofrnf-59/?prefix=obj37 2023-10-18T20:45:48.000 DEBUG:boto:Data: 2023-10-18T20:45:48.000 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.001 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.002 DEBUG:boto:Params: {} 2023-10-18T20:45:48.002 DEBUG:boto:Token: None 2023-10-18T20:45:48.003 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f83915ccf9d723-00653043fc-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.009 DEBUG:boto:b'bofrnf-59obj371000falseobj372023-10-18T20:43:09.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:45:48.010 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.011 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.011 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.011 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.012 DEBUG:boto:Method: GET 2023-10-18T20:45:48.012 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.013 DEBUG:boto:Data: 2023-10-18T20:45:48.013 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.014 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.015 DEBUG:boto:Params: {} 2023-10-18T20:45:48.015 DEBUG:boto:Token: None 2023-10-18T20:45:48.015 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.016 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rYqDg8WrTeQt37ww82NoKFv6mPw= 2023-10-18T20:45:48.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rYqDg8WrTeQt37ww82NoKFv6mPw=', 'Content-Length': '0'} 2023-10-18T20:45:48.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697b049ff0eeeab0-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.019 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.020 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.020 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.021 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.021 DEBUG:boto:Method: GET 2023-10-18T20:45:48.021 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.022 DEBUG:boto:Data: 2023-10-18T20:45:48.022 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.023 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.024 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.024 DEBUG:boto:Params: {} 2023-10-18T20:45:48.025 DEBUG:boto:Token: None 2023-10-18T20:45:48.026 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.027 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rYqDg8WrTeQt37ww82NoKFv6mPw= 2023-10-18T20:45:48.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rYqDg8WrTeQt37ww82NoKFv6mPw=', 'Content-Length': '0'} 2023-10-18T20:45:48.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083e2218856889c21-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.078 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.079 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.079 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.080 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.080 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.081 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.081 DEBUG:boto:Data: 2023-10-18T20:45:48.082 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.082 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.083 DEBUG:boto:Params: {} 2023-10-18T20:45:48.083 DEBUG:boto:Token: None 2023-10-18T20:45:48.084 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.084 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iusoTxSMP2Y0rsu4yRmsSfNGtF4= 2023-10-18T20:45:48.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iusoTxSMP2Y0rsu4yRmsSfNGtF4=', 'Content-Length': '0'} 2023-10-18T20:45:48.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000676731cfb2565a9-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.087 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.088 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.088 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.089 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.089 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.090 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.090 DEBUG:boto:Data: 2023-10-18T20:45:48.090 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.091 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.091 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.092 DEBUG:boto:Params: {} 2023-10-18T20:45:48.092 DEBUG:boto:Token: None 2023-10-18T20:45:48.092 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.093 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iusoTxSMP2Y0rsu4yRmsSfNGtF4= 2023-10-18T20:45:48.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iusoTxSMP2Y0rsu4yRmsSfNGtF4=', 'Content-Length': '0'} 2023-10-18T20:45:48.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e713fdb0869c65ae-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:45:48.096 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.097 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.097 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.098 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.098 DEBUG:boto:Method: GET 2023-10-18T20:45:48.099 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.099 DEBUG:boto:Data: 2023-10-18T20:45:48.100 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.100 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.100 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.101 DEBUG:boto:Params: {} 2023-10-18T20:45:48.101 DEBUG:boto:Token: None 2023-10-18T20:45:48.102 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.102 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rYqDg8WrTeQt37ww82NoKFv6mPw= 2023-10-18T20:45:48.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rYqDg8WrTeQt37ww82NoKFv6mPw=', 'Content-Length': '0'} 2023-10-18T20:45:48.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c286c47f31cae2-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.105 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.106 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.106 DEBUG:boto:path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.107 DEBUG:boto:auth_path=/bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.107 DEBUG:boto:Method: GET 2023-10-18T20:45:48.108 DEBUG:boto:Path: /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.108 DEBUG:boto:Data: 2023-10-18T20:45:48.108 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.109 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.110 DEBUG:boto:Params: {} 2023-10-18T20:45:48.110 DEBUG:boto:Token: None 2023-10-18T20:45:48.111 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38?versionId=null 2023-10-18T20:45:48.111 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:rYqDg8WrTeQt37ww82NoKFv6mPw= 2023-10-18T20:45:48.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rYqDg8WrTeQt37ww82NoKFv6mPw=', 'Content-Length': '0'} 2023-10-18T20:45:48.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad0255223472039-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.115 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.115 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.116 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.116 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.117 DEBUG:boto:Method: GET 2023-10-18T20:45:48.117 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.118 DEBUG:boto:Data: 2023-10-18T20:45:48.118 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.119 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.119 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.120 DEBUG:boto:Params: {} 2023-10-18T20:45:48.120 DEBUG:boto:Token: None 2023-10-18T20:45:48.121 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.121 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ff05059149723e9-00653043fc-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.125 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.125 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.126 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.126 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.127 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.127 DEBUG:boto:Method: GET 2023-10-18T20:45:48.128 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.128 DEBUG:boto:Data: 2023-10-18T20:45:48.129 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.129 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.130 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.131 DEBUG:boto:Params: {} 2023-10-18T20:45:48.132 DEBUG:boto:Token: None 2023-10-18T20:45:48.133 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.133 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004227bffc76bcbd4b-00653043fc-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.139 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.140 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.140 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.141 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.141 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:45:48.142 DEBUG:boto:Data: 2023-10-18T20:45:48.142 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.143 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.144 DEBUG:boto:Params: {} 2023-10-18T20:45:48.144 DEBUG:boto:Token: None 2023-10-18T20:45:48.145 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38 2023-10-18T20:45:48.145 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pogkfsy9+4VL7toPvwGxAvtXJy4= 2023-10-18T20:45:48.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pogkfsy9+4VL7toPvwGxAvtXJy4=', 'Content-Length': '0'} 2023-10-18T20:45:48.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b16cf1d409402676-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.148 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.149 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.149 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.150 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:45:48.150 DEBUG:boto:Data: 2023-10-18T20:45:48.151 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.151 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.151 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.152 DEBUG:boto:Params: {} 2023-10-18T20:45:48.152 DEBUG:boto:Token: None 2023-10-18T20:45:48.153 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38 2023-10-18T20:45:48.153 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pogkfsy9+4VL7toPvwGxAvtXJy4= 2023-10-18T20:45:48.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pogkfsy9+4VL7toPvwGxAvtXJy4=', 'Content-Length': '0'} 2023-10-18T20:45:48.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f9b8cd11c06ffaa-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-18T20:45:48.157 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.157 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.158 DEBUG:boto:Method: GET 2023-10-18T20:45:48.158 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:45:48.158 DEBUG:boto:Data: 2023-10-18T20:45:48.159 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.159 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.160 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.160 DEBUG:boto:Params: {} 2023-10-18T20:45:48.160 DEBUG:boto:Token: None 2023-10-18T20:45:48.161 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38 2023-10-18T20:45:48.161 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vVAmkHdhWnYfap1MB+qNqTw3448= 2023-10-18T20:45:48.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vVAmkHdhWnYfap1MB+qNqTw3448=', 'Content-Length': '0'} 2023-10-18T20:45:48.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e10d94f48129e528-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.165 DEBUG:boto:path=/bofrnf-59/obj38 2023-10-18T20:45:48.165 DEBUG:boto:auth_path=/bofrnf-59/obj38 2023-10-18T20:45:48.165 DEBUG:boto:Method: GET 2023-10-18T20:45:48.166 DEBUG:boto:Path: /bofrnf-59/obj38 2023-10-18T20:45:48.166 DEBUG:boto:Data: 2023-10-18T20:45:48.167 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.167 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.167 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.168 DEBUG:boto:Params: {} 2023-10-18T20:45:48.168 DEBUG:boto:Token: None 2023-10-18T20:45:48.169 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj38 2023-10-18T20:45:48.169 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vVAmkHdhWnYfap1MB+qNqTw3448= 2023-10-18T20:45:48.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vVAmkHdhWnYfap1MB+qNqTw3448=', 'Content-Length': '0'} 2023-10-18T20:45:48.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0b1dd6a77baf740-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.214 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.215 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.215 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.216 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.216 DEBUG:boto:Method: GET 2023-10-18T20:45:48.217 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.217 DEBUG:boto:Data: 2023-10-18T20:45:48.218 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.219 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.219 DEBUG:boto:Params: {} 2023-10-18T20:45:48.220 DEBUG:boto:Token: None 2023-10-18T20:45:48.220 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fccd934745cdfe9d-00653043fc-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.225 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.226 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.227 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.227 DEBUG:boto:path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.228 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.229 DEBUG:boto:Method: GET 2023-10-18T20:45:48.230 DEBUG:boto:Path: /bofrnf-59/?prefix=obj38 2023-10-18T20:45:48.230 DEBUG:boto:Data: 2023-10-18T20:45:48.231 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.231 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.232 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.232 DEBUG:boto:Params: {} 2023-10-18T20:45:48.233 DEBUG:boto:Token: None 2023-10-18T20:45:48.233 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.233 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025a38c9092353d2c-00653043fc-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.239 DEBUG:boto:b'bofrnf-59obj381000falseobj382023-10-18T20:34:23.394Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:45:48.240 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.240 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.241 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.241 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.241 DEBUG:boto:Method: GET 2023-10-18T20:45:48.242 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.242 DEBUG:boto:Data: 2023-10-18T20:45:48.243 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.243 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.244 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.244 DEBUG:boto:Params: {} 2023-10-18T20:45:48.245 DEBUG:boto:Token: None 2023-10-18T20:45:48.245 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.246 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ANOKE7OllbviJibz9VGW80+/I58= 2023-10-18T20:45:48.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ANOKE7OllbviJibz9VGW80+/I58=', 'Content-Length': '0'} 2023-10-18T20:45:48.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035892cb9ab8ad5fa-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.290 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.291 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.291 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.292 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.292 DEBUG:boto:Method: GET 2023-10-18T20:45:48.293 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.293 DEBUG:boto:Data: 2023-10-18T20:45:48.294 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.295 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.295 DEBUG:boto:Params: {} 2023-10-18T20:45:48.296 DEBUG:boto:Token: None 2023-10-18T20:45:48.296 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.297 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ANOKE7OllbviJibz9VGW80+/I58= 2023-10-18T20:45:48.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ANOKE7OllbviJibz9VGW80+/I58=', 'Content-Length': '0'} 2023-10-18T20:45:48.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000144cb5c79f31f8a3-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.300 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.300 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.301 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.301 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.302 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.302 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.303 DEBUG:boto:Data: 2023-10-18T20:45:48.303 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.303 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.304 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.304 DEBUG:boto:Params: {} 2023-10-18T20:45:48.305 DEBUG:boto:Token: None 2023-10-18T20:45:48.305 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.306 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eqRgYitegkwqv1CRyVwUbeJsDoI= 2023-10-18T20:45:48.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eqRgYitegkwqv1CRyVwUbeJsDoI=', 'Content-Length': '0'} 2023-10-18T20:45:48.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb2548dbad1270bf-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.309 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.309 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.310 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.310 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.311 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.311 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.312 DEBUG:boto:Data: 2023-10-18T20:45:48.312 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.313 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.313 DEBUG:boto:Params: {} 2023-10-18T20:45:48.314 DEBUG:boto:Token: None 2023-10-18T20:45:48.314 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eqRgYitegkwqv1CRyVwUbeJsDoI= 2023-10-18T20:45:48.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eqRgYitegkwqv1CRyVwUbeJsDoI=', 'Content-Length': '0'} 2023-10-18T20:45:48.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8bb9d24a7c375e-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:45:48.318 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.319 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.319 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.320 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.320 DEBUG:boto:Method: GET 2023-10-18T20:45:48.321 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.321 DEBUG:boto:Data: 2023-10-18T20:45:48.322 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.323 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.323 DEBUG:boto:Params: {} 2023-10-18T20:45:48.323 DEBUG:boto:Token: None 2023-10-18T20:45:48.324 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ANOKE7OllbviJibz9VGW80+/I58= 2023-10-18T20:45:48.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ANOKE7OllbviJibz9VGW80+/I58=', 'Content-Length': '0'} 2023-10-18T20:45:48.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e0600eda03723c7-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.328 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.328 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.329 DEBUG:boto:path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.329 DEBUG:boto:auth_path=/bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.329 DEBUG:boto:Method: GET 2023-10-18T20:45:48.330 DEBUG:boto:Path: /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.330 DEBUG:boto:Data: 2023-10-18T20:45:48.331 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.331 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.332 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.332 DEBUG:boto:Params: {} 2023-10-18T20:45:48.332 DEBUG:boto:Token: None 2023-10-18T20:45:48.333 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44?versionId=null 2023-10-18T20:45:48.333 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ANOKE7OllbviJibz9VGW80+/I58= 2023-10-18T20:45:48.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ANOKE7OllbviJibz9VGW80+/I58=', 'Content-Length': '0'} 2023-10-18T20:45:48.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d129e271e4ff31fd-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.337 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.337 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.338 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.338 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.339 DEBUG:boto:Method: GET 2023-10-18T20:45:48.339 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.339 DEBUG:boto:Data: 2023-10-18T20:45:48.340 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.340 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.341 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.341 DEBUG:boto:Params: {} 2023-10-18T20:45:48.342 DEBUG:boto:Token: None 2023-10-18T20:45:48.342 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.343 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000461489de447dc2a6-00653043fc-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.347 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.347 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.348 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.348 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.349 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.349 DEBUG:boto:Method: GET 2023-10-18T20:45:48.350 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.350 DEBUG:boto:Data: 2023-10-18T20:45:48.351 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.351 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.352 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.352 DEBUG:boto:Params: {} 2023-10-18T20:45:48.352 DEBUG:boto:Token: None 2023-10-18T20:45:48.353 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.354 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008449c9f3ffae8fc4-00653043fc-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.363 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.364 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.364 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.365 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.365 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:45:48.366 DEBUG:boto:Data: 2023-10-18T20:45:48.366 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.366 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.367 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.367 DEBUG:boto:Params: {} 2023-10-18T20:45:48.368 DEBUG:boto:Token: None 2023-10-18T20:45:48.368 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44 2023-10-18T20:45:48.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dk1umyz4Pc3GQU7gX+Xg7SNLdew= 2023-10-18T20:45:48.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dk1umyz4Pc3GQU7gX+Xg7SNLdew=', 'Content-Length': '0'} 2023-10-18T20:45:48.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac22029fa92f1e5b-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.372 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.372 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.373 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.373 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:45:48.373 DEBUG:boto:Data: 2023-10-18T20:45:48.374 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.375 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.375 DEBUG:boto:Params: {} 2023-10-18T20:45:48.375 DEBUG:boto:Token: None 2023-10-18T20:45:48.376 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44 2023-10-18T20:45:48.376 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Dk1umyz4Pc3GQU7gX+Xg7SNLdew= 2023-10-18T20:45:48.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Dk1umyz4Pc3GQU7gX+Xg7SNLdew=', 'Content-Length': '0'} 2023-10-18T20:45:48.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94df0b372b7403e-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:45:48.380 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.380 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.381 DEBUG:boto:Method: GET 2023-10-18T20:45:48.381 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:45:48.382 DEBUG:boto:Data: 2023-10-18T20:45:48.382 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.382 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.383 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.383 DEBUG:boto:Params: {} 2023-10-18T20:45:48.384 DEBUG:boto:Token: None 2023-10-18T20:45:48.384 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44 2023-10-18T20:45:48.385 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eRN/HHiNUqxUdZFKD/lkADU1PO0= 2023-10-18T20:45:48.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eRN/HHiNUqxUdZFKD/lkADU1PO0=', 'Content-Length': '0'} 2023-10-18T20:45:48.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ee19e8b815af0cd-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.434 DEBUG:boto:path=/bofrnf-59/obj44 2023-10-18T20:45:48.435 DEBUG:boto:auth_path=/bofrnf-59/obj44 2023-10-18T20:45:48.435 DEBUG:boto:Method: GET 2023-10-18T20:45:48.436 DEBUG:boto:Path: /bofrnf-59/obj44 2023-10-18T20:45:48.436 DEBUG:boto:Data: 2023-10-18T20:45:48.437 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.437 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.438 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.438 DEBUG:boto:Params: {} 2023-10-18T20:45:48.439 DEBUG:boto:Token: None 2023-10-18T20:45:48.439 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj44 2023-10-18T20:45:48.440 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eRN/HHiNUqxUdZFKD/lkADU1PO0= 2023-10-18T20:45:48.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eRN/HHiNUqxUdZFKD/lkADU1PO0=', 'Content-Length': '0'} 2023-10-18T20:45:48.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf40b9fae7e404eb-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.443 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.444 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.444 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.445 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.445 DEBUG:boto:Method: GET 2023-10-18T20:45:48.446 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.446 DEBUG:boto:Data: 2023-10-18T20:45:48.446 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.447 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.447 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.448 DEBUG:boto:Params: {} 2023-10-18T20:45:48.448 DEBUG:boto:Token: None 2023-10-18T20:45:48.449 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.449 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dce15e3f2c64fb7-00653043fc-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.453 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.454 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.454 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.454 DEBUG:boto:path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.455 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.455 DEBUG:boto:Method: GET 2023-10-18T20:45:48.456 DEBUG:boto:Path: /bofrnf-59/?prefix=obj44 2023-10-18T20:45:48.456 DEBUG:boto:Data: 2023-10-18T20:45:48.456 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.457 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.458 DEBUG:boto:Params: {} 2023-10-18T20:45:48.458 DEBUG:boto:Token: None 2023-10-18T20:45:48.459 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.459 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000653b1979011ff80e-00653043fc-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.464 DEBUG:boto:b'bofrnf-59obj441000falseobj442023-10-18T20:32:33.692Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:45:48.465 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.466 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.466 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.466 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.467 DEBUG:boto:Method: GET 2023-10-18T20:45:48.467 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.467 DEBUG:boto:Data: 2023-10-18T20:45:48.468 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.468 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.469 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.469 DEBUG:boto:Params: {} 2023-10-18T20:45:48.470 DEBUG:boto:Token: None 2023-10-18T20:45:48.470 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.470 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:14g8yGFngxC9qlYlKe+iYquZKFA= 2023-10-18T20:45:48.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:14g8yGFngxC9qlYlKe+iYquZKFA=', 'Content-Length': '0'} 2023-10-18T20:45:48.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056d9d7db146ed5fc-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.474 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.474 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.475 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.475 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.476 DEBUG:boto:Method: GET 2023-10-18T20:45:48.476 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.476 DEBUG:boto:Data: 2023-10-18T20:45:48.477 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.477 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.478 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.478 DEBUG:boto:Params: {} 2023-10-18T20:45:48.478 DEBUG:boto:Token: None 2023-10-18T20:45:48.479 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.479 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:14g8yGFngxC9qlYlKe+iYquZKFA= 2023-10-18T20:45:48.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:14g8yGFngxC9qlYlKe+iYquZKFA=', 'Content-Length': '0'} 2023-10-18T20:45:48.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e03c26899945afa-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.482 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.483 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.483 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.484 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.484 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.485 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.485 DEBUG:boto:Data: 2023-10-18T20:45:48.485 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.486 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.486 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.486 DEBUG:boto:Params: {} 2023-10-18T20:45:48.487 DEBUG:boto:Token: None 2023-10-18T20:45:48.487 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.488 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OOQJhm/zQT5z4fG0+rHbzXNRWuc= 2023-10-18T20:45:48.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OOQJhm/zQT5z4fG0+rHbzXNRWuc=', 'Content-Length': '0'} 2023-10-18T20:45:48.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33e1696958556a4-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.491 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.491 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.492 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.492 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.492 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.493 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.493 DEBUG:boto:Data: 2023-10-18T20:45:48.494 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.494 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.495 DEBUG:boto:Params: {} 2023-10-18T20:45:48.495 DEBUG:boto:Token: None 2023-10-18T20:45:48.496 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.496 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OOQJhm/zQT5z4fG0+rHbzXNRWuc= 2023-10-18T20:45:48.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OOQJhm/zQT5z4fG0+rHbzXNRWuc=', 'Content-Length': '0'} 2023-10-18T20:45:48.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f65a075f47cfceef-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:45:48.500 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.500 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.501 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.501 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.501 DEBUG:boto:Method: GET 2023-10-18T20:45:48.502 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.502 DEBUG:boto:Data: 2023-10-18T20:45:48.503 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.503 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.503 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.504 DEBUG:boto:Params: {} 2023-10-18T20:45:48.504 DEBUG:boto:Token: None 2023-10-18T20:45:48.505 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:14g8yGFngxC9qlYlKe+iYquZKFA= 2023-10-18T20:45:48.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:14g8yGFngxC9qlYlKe+iYquZKFA=', 'Content-Length': '0'} 2023-10-18T20:45:48.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc22d372d4485a1f-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.508 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.509 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.509 DEBUG:boto:path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.510 DEBUG:boto:auth_path=/bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.510 DEBUG:boto:Method: GET 2023-10-18T20:45:48.511 DEBUG:boto:Path: /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.511 DEBUG:boto:Data: 2023-10-18T20:45:48.512 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.512 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.513 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.513 DEBUG:boto:Params: {} 2023-10-18T20:45:48.514 DEBUG:boto:Token: None 2023-10-18T20:45:48.514 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46?versionId=null 2023-10-18T20:45:48.515 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:14g8yGFngxC9qlYlKe+iYquZKFA= 2023-10-18T20:45:48.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:14g8yGFngxC9qlYlKe+iYquZKFA=', 'Content-Length': '0'} 2023-10-18T20:45:48.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2d7501fc074cf6-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.558 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.559 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.561 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.561 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.562 DEBUG:boto:Method: GET 2023-10-18T20:45:48.562 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.563 DEBUG:boto:Data: 2023-10-18T20:45:48.563 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.564 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.564 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.565 DEBUG:boto:Params: {} 2023-10-18T20:45:48.565 DEBUG:boto:Token: None 2023-10-18T20:45:48.566 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.566 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ece59691f9b96f-00653043fc-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.570 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.571 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.571 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.572 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.572 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.573 DEBUG:boto:Method: GET 2023-10-18T20:45:48.573 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.573 DEBUG:boto:Data: 2023-10-18T20:45:48.574 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.574 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.575 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.575 DEBUG:boto:Params: {} 2023-10-18T20:45:48.576 DEBUG:boto:Token: None 2023-10-18T20:45:48.576 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000404b4be29d9a1f84-00653043fc-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.582 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.583 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.583 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.584 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.584 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:45:48.585 DEBUG:boto:Data: 2023-10-18T20:45:48.585 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.586 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.586 DEBUG:boto:Params: {} 2023-10-18T20:45:48.587 DEBUG:boto:Token: None 2023-10-18T20:45:48.587 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46 2023-10-18T20:45:48.587 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7zrLwU4DEdLtnKQQtIt11T/O6tU= 2023-10-18T20:45:48.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7zrLwU4DEdLtnKQQtIt11T/O6tU=', 'Content-Length': '0'} 2023-10-18T20:45:48.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003296dae5519f6e07-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.591 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.591 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.592 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.592 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:45:48.592 DEBUG:boto:Data: 2023-10-18T20:45:48.593 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.593 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.593 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.594 DEBUG:boto:Params: {} 2023-10-18T20:45:48.594 DEBUG:boto:Token: None 2023-10-18T20:45:48.595 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46 2023-10-18T20:45:48.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7zrLwU4DEdLtnKQQtIt11T/O6tU= 2023-10-18T20:45:48.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7zrLwU4DEdLtnKQQtIt11T/O6tU=', 'Content-Length': '0'} 2023-10-18T20:45:48.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e74a4db0613f4440-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-18T20:45:48.599 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.599 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.600 DEBUG:boto:Method: GET 2023-10-18T20:45:48.600 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:45:48.600 DEBUG:boto:Data: 2023-10-18T20:45:48.601 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.601 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.602 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.603 DEBUG:boto:Params: {} 2023-10-18T20:45:48.604 DEBUG:boto:Token: None 2023-10-18T20:45:48.605 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46 2023-10-18T20:45:48.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BNpAcB3i/MId1EZ2oVsLb4VtQd8= 2023-10-18T20:45:48.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BNpAcB3i/MId1EZ2oVsLb4VtQd8=', 'Content-Length': '0'} 2023-10-18T20:45:48.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad89180dd0f2dd3d-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.610 DEBUG:boto:path=/bofrnf-59/obj46 2023-10-18T20:45:48.611 DEBUG:boto:auth_path=/bofrnf-59/obj46 2023-10-18T20:45:48.612 DEBUG:boto:Method: GET 2023-10-18T20:45:48.612 DEBUG:boto:Path: /bofrnf-59/obj46 2023-10-18T20:45:48.613 DEBUG:boto:Data: 2023-10-18T20:45:48.614 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.615 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.616 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.616 DEBUG:boto:Params: {} 2023-10-18T20:45:48.617 DEBUG:boto:Token: None 2023-10-18T20:45:48.618 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj46 2023-10-18T20:45:48.619 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BNpAcB3i/MId1EZ2oVsLb4VtQd8= 2023-10-18T20:45:48.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BNpAcB3i/MId1EZ2oVsLb4VtQd8=', 'Content-Length': '0'} 2023-10-18T20:45:48.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b69d4ae7473be2-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.624 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.624 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.625 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.626 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.627 DEBUG:boto:Method: GET 2023-10-18T20:45:48.627 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.628 DEBUG:boto:Data: 2023-10-18T20:45:48.629 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.630 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.631 DEBUG:boto:Params: {} 2023-10-18T20:45:48.632 DEBUG:boto:Token: None 2023-10-18T20:45:48.633 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.634 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eca11d35588896d2-00653043fc-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.639 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.639 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.640 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.641 DEBUG:boto:path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.642 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.643 DEBUG:boto:Method: GET 2023-10-18T20:45:48.643 DEBUG:boto:Path: /bofrnf-59/?prefix=obj46 2023-10-18T20:45:48.644 DEBUG:boto:Data: 2023-10-18T20:45:48.645 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.646 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.646 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.647 DEBUG:boto:Params: {} 2023-10-18T20:45:48.648 DEBUG:boto:Token: None 2023-10-18T20:45:48.649 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.649 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c06fee6fe761509b-00653043fc-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.702 DEBUG:boto:b'bofrnf-59obj461000falseobj462023-10-18T20:38:22.132Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:45:48.704 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.704 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.705 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.705 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.706 DEBUG:boto:Method: GET 2023-10-18T20:45:48.706 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.706 DEBUG:boto:Data: 2023-10-18T20:45:48.707 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.708 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.708 DEBUG:boto:Params: {} 2023-10-18T20:45:48.709 DEBUG:boto:Token: None 2023-10-18T20:45:48.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sXD4cgu/jcXQonJB4O09sQubgyc= 2023-10-18T20:45:48.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sXD4cgu/jcXQonJB4O09sQubgyc=', 'Content-Length': '0'} 2023-10-18T20:45:48.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d823eeceddb407d-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.714 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.714 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.714 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.715 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.715 DEBUG:boto:Method: GET 2023-10-18T20:45:48.716 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.716 DEBUG:boto:Data: 2023-10-18T20:45:48.717 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.717 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.717 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.718 DEBUG:boto:Params: {} 2023-10-18T20:45:48.719 DEBUG:boto:Token: None 2023-10-18T20:45:48.719 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.720 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sXD4cgu/jcXQonJB4O09sQubgyc= 2023-10-18T20:45:48.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sXD4cgu/jcXQonJB4O09sQubgyc=', 'Content-Length': '0'} 2023-10-18T20:45:48.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000182fdf517d1d9c84-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.723 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.723 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.724 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.724 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.725 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.725 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.725 DEBUG:boto:Data: 2023-10-18T20:45:48.726 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.726 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.727 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.727 DEBUG:boto:Params: {} 2023-10-18T20:45:48.728 DEBUG:boto:Token: None 2023-10-18T20:45:48.728 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.729 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:27DJsnDeNCgnMS8S4Oj54K7MyOU= 2023-10-18T20:45:48.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:27DJsnDeNCgnMS8S4Oj54K7MyOU=', 'Content-Length': '0'} 2023-10-18T20:45:48.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3fa25846784198a-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.732 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.732 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.733 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.733 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.733 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.734 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.734 DEBUG:boto:Data: 2023-10-18T20:45:48.735 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.735 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.735 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.736 DEBUG:boto:Params: {} 2023-10-18T20:45:48.736 DEBUG:boto:Token: None 2023-10-18T20:45:48.737 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:27DJsnDeNCgnMS8S4Oj54K7MyOU= 2023-10-18T20:45:48.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:27DJsnDeNCgnMS8S4Oj54K7MyOU=', 'Content-Length': '0'} 2023-10-18T20:45:48.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b24db4234dbcccf-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:45:48.740 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.741 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.741 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.741 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.742 DEBUG:boto:Method: GET 2023-10-18T20:45:48.742 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.743 DEBUG:boto:Data: 2023-10-18T20:45:48.743 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.744 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.744 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.744 DEBUG:boto:Params: {} 2023-10-18T20:45:48.745 DEBUG:boto:Token: None 2023-10-18T20:45:48.746 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.746 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sXD4cgu/jcXQonJB4O09sQubgyc= 2023-10-18T20:45:48.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sXD4cgu/jcXQonJB4O09sQubgyc=', 'Content-Length': '0'} 2023-10-18T20:45:48.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f74640e270fdc0-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.749 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.750 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.750 DEBUG:boto:path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.751 DEBUG:boto:auth_path=/bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.751 DEBUG:boto:Method: GET 2023-10-18T20:45:48.751 DEBUG:boto:Path: /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.752 DEBUG:boto:Data: 2023-10-18T20:45:48.752 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.753 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.754 DEBUG:boto:Params: {} 2023-10-18T20:45:48.754 DEBUG:boto:Token: None 2023-10-18T20:45:48.755 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51?versionId=null 2023-10-18T20:45:48.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sXD4cgu/jcXQonJB4O09sQubgyc= 2023-10-18T20:45:48.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sXD4cgu/jcXQonJB4O09sQubgyc=', 'Content-Length': '0'} 2023-10-18T20:45:48.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a13b98e91a7c20ec-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.798 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.799 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.800 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.800 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.800 DEBUG:boto:Method: GET 2023-10-18T20:45:48.801 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.801 DEBUG:boto:Data: 2023-10-18T20:45:48.802 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.803 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.803 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.803 DEBUG:boto:Params: {} 2023-10-18T20:45:48.804 DEBUG:boto:Token: None 2023-10-18T20:45:48.804 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.805 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000031c1417c4661a21-00653043fc-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.809 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.810 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.810 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.811 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.811 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.811 DEBUG:boto:Method: GET 2023-10-18T20:45:48.812 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.812 DEBUG:boto:Data: 2023-10-18T20:45:48.813 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.813 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.814 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.814 DEBUG:boto:Params: {} 2023-10-18T20:45:48.815 DEBUG:boto:Token: None 2023-10-18T20:45:48.815 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000124b257dbde8a4d6-00653043fc-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.821 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.822 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.822 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.823 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.823 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:45:48.824 DEBUG:boto:Data: 2023-10-18T20:45:48.824 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.824 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.825 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.825 DEBUG:boto:Params: {} 2023-10-18T20:45:48.826 DEBUG:boto:Token: None 2023-10-18T20:45:48.826 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51 2023-10-18T20:45:48.827 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n38hl+QalGYYqai1T5XDkqnalJM= 2023-10-18T20:45:48.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n38hl+QalGYYqai1T5XDkqnalJM=', 'Content-Length': '0'} 2023-10-18T20:45:48.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000232147c3af587057-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.830 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.830 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.831 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.831 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:45:48.832 DEBUG:boto:Data: 2023-10-18T20:45:48.832 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.832 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.833 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.833 DEBUG:boto:Params: {} 2023-10-18T20:45:48.834 DEBUG:boto:Token: None 2023-10-18T20:45:48.834 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51 2023-10-18T20:45:48.835 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n38hl+QalGYYqai1T5XDkqnalJM= 2023-10-18T20:45:48.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n38hl+QalGYYqai1T5XDkqnalJM=', 'Content-Length': '0'} 2023-10-18T20:45:48.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d621e981404ef50-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-18T20:45:48.839 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.840 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.841 DEBUG:boto:Method: GET 2023-10-18T20:45:48.841 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:45:48.842 DEBUG:boto:Data: 2023-10-18T20:45:48.843 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.844 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.844 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.845 DEBUG:boto:Params: {} 2023-10-18T20:45:48.846 DEBUG:boto:Token: None 2023-10-18T20:45:48.847 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51 2023-10-18T20:45:48.847 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gM5rxxEvpqtrNOeFU3L+pEOVOo4= 2023-10-18T20:45:48.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gM5rxxEvpqtrNOeFU3L+pEOVOo4=', 'Content-Length': '0'} 2023-10-18T20:45:48.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c7f8d7ee2738f5-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.851 DEBUG:boto:path=/bofrnf-59/obj51 2023-10-18T20:45:48.851 DEBUG:boto:auth_path=/bofrnf-59/obj51 2023-10-18T20:45:48.852 DEBUG:boto:Method: GET 2023-10-18T20:45:48.852 DEBUG:boto:Path: /bofrnf-59/obj51 2023-10-18T20:45:48.852 DEBUG:boto:Data: 2023-10-18T20:45:48.853 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.853 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.854 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.854 DEBUG:boto:Params: {} 2023-10-18T20:45:48.855 DEBUG:boto:Token: None 2023-10-18T20:45:48.855 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj51 2023-10-18T20:45:48.856 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gM5rxxEvpqtrNOeFU3L+pEOVOo4= 2023-10-18T20:45:48.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gM5rxxEvpqtrNOeFU3L+pEOVOo4=', 'Content-Length': '0'} 2023-10-18T20:45:48.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007319d1df7bddc769-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.859 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.859 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.860 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.860 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.861 DEBUG:boto:Method: GET 2023-10-18T20:45:48.861 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.862 DEBUG:boto:Data: 2023-10-18T20:45:48.862 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.862 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.863 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.863 DEBUG:boto:Params: {} 2023-10-18T20:45:48.864 DEBUG:boto:Token: None 2023-10-18T20:45:48.864 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.865 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0be60f982aa771a-00653043fc-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.869 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.870 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:48.870 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:48.871 DEBUG:boto:path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.871 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.872 DEBUG:boto:Method: GET 2023-10-18T20:45:48.872 DEBUG:boto:Path: /bofrnf-59/?prefix=obj51 2023-10-18T20:45:48.873 DEBUG:boto:Data: 2023-10-18T20:45:48.874 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.876 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.876 DEBUG:boto:Params: {} 2023-10-18T20:45:48.877 DEBUG:boto:Token: None 2023-10-18T20:45:48.878 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/ 2023-10-18T20:45:48.879 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw= 2023-10-18T20:45:48.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eLP43C9GBQ8DNqx2vNWEbT1OHCw=', 'Content-Length': '0'} 2023-10-18T20:45:48.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe8c06393bad105b-00653043fc-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.930 DEBUG:boto:b'bofrnf-59obj511000falseobj512023-10-18T20:33:05.411Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:48.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:45:48.932 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:48.932 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:48.933 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.933 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.933 DEBUG:boto:Method: GET 2023-10-18T20:45:48.934 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.934 DEBUG:boto:Data: 2023-10-18T20:45:48.935 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.936 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.936 DEBUG:boto:Params: {} 2023-10-18T20:45:48.937 DEBUG:boto:Token: None 2023-10-18T20:45:48.938 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.939 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mgajwIj7p5igoibp/C9d/o7kwq4= 2023-10-18T20:45:48.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mgajwIj7p5igoibp/C9d/o7kwq4=', 'Content-Length': '0'} 2023-10-18T20:45:48.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001380bfa48548e8b4-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.944 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:48.945 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:48.946 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.946 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.947 DEBUG:boto:Method: GET 2023-10-18T20:45:48.948 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.949 DEBUG:boto:Data: 2023-10-18T20:45:48.949 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.950 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.951 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.952 DEBUG:boto:Params: {} 2023-10-18T20:45:48.953 DEBUG:boto:Token: None 2023-10-18T20:45:48.953 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.954 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mgajwIj7p5igoibp/C9d/o7kwq4= 2023-10-18T20:45:48.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mgajwIj7p5igoibp/C9d/o7kwq4=', 'Content-Length': '0'} 2023-10-18T20:45:48.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c0bc934835f5b46-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.958 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:48.959 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:48.960 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.960 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.961 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.961 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.961 DEBUG:boto:Data: 2023-10-18T20:45:48.962 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.962 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.963 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.964 DEBUG:boto:Params: {} 2023-10-18T20:45:48.964 DEBUG:boto:Token: None 2023-10-18T20:45:48.965 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.966 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7lOf218rhm1SrW2xSNS4yZG4XQw= 2023-10-18T20:45:48.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7lOf218rhm1SrW2xSNS4yZG4XQw=', 'Content-Length': '0'} 2023-10-18T20:45:48.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a9b127cab49bc7c-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.971 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:48.971 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:48.972 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.972 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.972 DEBUG:boto:Method: HEAD 2023-10-18T20:45:48.973 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.973 DEBUG:boto:Data: 2023-10-18T20:45:48.974 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.974 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:48.974 DEBUG:boto:Port: 8000 2023-10-18T20:45:48.975 DEBUG:boto:Params: {} 2023-10-18T20:45:48.975 DEBUG:boto:Token: None 2023-10-18T20:45:48.976 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.976 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7lOf218rhm1SrW2xSNS4yZG4XQw= 2023-10-18T20:45:48.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7lOf218rhm1SrW2xSNS4yZG4XQw=', 'Content-Length': '0'} 2023-10-18T20:45:48.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dc8bcbff49d81c2-00653043fc-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:48.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:45:48.980 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:48.980 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:48.981 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.981 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.982 DEBUG:boto:Method: GET 2023-10-18T20:45:48.983 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.983 DEBUG:boto:Data: 2023-10-18T20:45:48.983 DEBUG:boto:Headers: {} 2023-10-18T20:45:48.984 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:48.984 DEBUG:boto:Port: 8001 2023-10-18T20:45:48.985 DEBUG:boto:Params: {} 2023-10-18T20:45:48.985 DEBUG:boto:Token: None 2023-10-18T20:45:48.986 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:48 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:48.987 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mgajwIj7p5igoibp/C9d/o7kwq4= 2023-10-18T20:45:48.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:48 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mgajwIj7p5igoibp/C9d/o7kwq4=', 'Content-Length': '0'} 2023-10-18T20:45:48.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a462a87d80bfa295-00653043fc-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.030 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:49.031 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:49.031 DEBUG:boto:path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:49.032 DEBUG:boto:auth_path=/bofrnf-59/obj55?versionId=null 2023-10-18T20:45:49.032 DEBUG:boto:Method: GET 2023-10-18T20:45:49.033 DEBUG:boto:Path: /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:49.033 DEBUG:boto:Data: 2023-10-18T20:45:49.034 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.034 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.035 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.035 DEBUG:boto:Params: {} 2023-10-18T20:45:49.036 DEBUG:boto:Token: None 2023-10-18T20:45:49.036 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj55?versionId=null 2023-10-18T20:45:49.037 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mPzoyGCyvpNbH2bAmdQIPgTLoRo= 2023-10-18T20:45:49.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mPzoyGCyvpNbH2bAmdQIPgTLoRo=', 'Content-Length': '0'} 2023-10-18T20:45:49.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd79b33f4c0e67a-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.040 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.041 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.041 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.042 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.042 DEBUG:boto:Method: GET 2023-10-18T20:45:49.043 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.043 DEBUG:boto:Data: 2023-10-18T20:45:49.044 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.044 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.045 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.045 DEBUG:boto:Params: {} 2023-10-18T20:45:49.046 DEBUG:boto:Token: None 2023-10-18T20:45:49.047 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.047 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba89427428b19fb4-00653043fd-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.051 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.052 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.053 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.053 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.054 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.054 DEBUG:boto:Method: GET 2023-10-18T20:45:49.055 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.055 DEBUG:boto:Data: 2023-10-18T20:45:49.056 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.057 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.057 DEBUG:boto:Params: {} 2023-10-18T20:45:49.058 DEBUG:boto:Token: None 2023-10-18T20:45:49.058 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.059 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2cc9b3ba297ba69-00653043fd-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.065 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.066 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:49.066 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:49.067 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.067 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:45:49.068 DEBUG:boto:Data: 2023-10-18T20:45:49.068 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.069 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.069 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.070 DEBUG:boto:Params: {} 2023-10-18T20:45:49.070 DEBUG:boto:Token: None 2023-10-18T20:45:49.071 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj55 2023-10-18T20:45:49.071 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hfGtGA7sMEzkNmHXTWgGL/wNTRo= 2023-10-18T20:45:49.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hfGtGA7sMEzkNmHXTWgGL/wNTRo=', 'Content-Length': '0'} 2023-10-18T20:45:49.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9ba276dd223938-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.074 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:49.075 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:49.075 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.076 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:45:49.077 DEBUG:boto:Data: 2023-10-18T20:45:49.077 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.078 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.078 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.079 DEBUG:boto:Params: {} 2023-10-18T20:45:49.079 DEBUG:boto:Token: None 2023-10-18T20:45:49.079 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj55 2023-10-18T20:45:49.080 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hfGtGA7sMEzkNmHXTWgGL/wNTRo= 2023-10-18T20:45:49.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hfGtGA7sMEzkNmHXTWgGL/wNTRo=', 'Content-Length': '0'} 2023-10-18T20:45:49.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa648d64b56e0884-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:45:49.084 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:49.084 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:49.085 DEBUG:boto:Method: GET 2023-10-18T20:45:49.086 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:45:49.086 DEBUG:boto:Data: 2023-10-18T20:45:49.087 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.088 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.088 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.088 DEBUG:boto:Params: {} 2023-10-18T20:45:49.089 DEBUG:boto:Token: None 2023-10-18T20:45:49.089 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj55 2023-10-18T20:45:49.090 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c2p1m7TPUP5+yr2+8rSlR21HO1c= 2023-10-18T20:45:49.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c2p1m7TPUP5+yr2+8rSlR21HO1c=', 'Content-Length': '0'} 2023-10-18T20:45:49.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006214dbe98691bad8-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.094 DEBUG:boto:path=/bofrnf-59/obj55 2023-10-18T20:45:49.094 DEBUG:boto:auth_path=/bofrnf-59/obj55 2023-10-18T20:45:49.095 DEBUG:boto:Method: GET 2023-10-18T20:45:49.095 DEBUG:boto:Path: /bofrnf-59/obj55 2023-10-18T20:45:49.095 DEBUG:boto:Data: 2023-10-18T20:45:49.096 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.096 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.097 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.097 DEBUG:boto:Params: {} 2023-10-18T20:45:49.098 DEBUG:boto:Token: None 2023-10-18T20:45:49.098 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj55 2023-10-18T20:45:49.099 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:c2p1m7TPUP5+yr2+8rSlR21HO1c= 2023-10-18T20:45:49.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:c2p1m7TPUP5+yr2+8rSlR21HO1c=', 'Content-Length': '0'} 2023-10-18T20:45:49.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:32:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632f980a0690bedf-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.142 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.143 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.144 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.144 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.145 DEBUG:boto:Method: GET 2023-10-18T20:45:49.146 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.146 DEBUG:boto:Data: 2023-10-18T20:45:49.147 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.148 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.149 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.149 DEBUG:boto:Params: {} 2023-10-18T20:45:49.150 DEBUG:boto:Token: None 2023-10-18T20:45:49.151 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068c9fdb9a73af7d9-00653043fd-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.156 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.157 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.158 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.159 DEBUG:boto:path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.159 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.160 DEBUG:boto:Method: GET 2023-10-18T20:45:49.161 DEBUG:boto:Path: /bofrnf-59/?prefix=obj55 2023-10-18T20:45:49.161 DEBUG:boto:Data: 2023-10-18T20:45:49.162 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.163 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.164 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.164 DEBUG:boto:Params: {} 2023-10-18T20:45:49.165 DEBUG:boto:Token: None 2023-10-18T20:45:49.166 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.166 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc54467f507311d8-00653043fd-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.172 DEBUG:boto:b'bofrnf-59obj551000falseobj552023-10-18T20:32:33.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:45:49.174 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.174 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.175 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.175 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.176 DEBUG:boto:Method: GET 2023-10-18T20:45:49.176 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.177 DEBUG:boto:Data: 2023-10-18T20:45:49.177 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.178 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.179 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.180 DEBUG:boto:Params: {} 2023-10-18T20:45:49.181 DEBUG:boto:Token: None 2023-10-18T20:45:49.182 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.182 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JoEWmG2aW+doKO8vf+6jqXmPg5s= 2023-10-18T20:45:49.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JoEWmG2aW+doKO8vf+6jqXmPg5s=', 'Content-Length': '0'} 2023-10-18T20:45:49.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15bd856a44485cc-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.226 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.228 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.228 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.229 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.229 DEBUG:boto:Method: GET 2023-10-18T20:45:49.230 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.230 DEBUG:boto:Data: 2023-10-18T20:45:49.231 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.231 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.232 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.232 DEBUG:boto:Params: {} 2023-10-18T20:45:49.233 DEBUG:boto:Token: None 2023-10-18T20:45:49.234 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.234 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JoEWmG2aW+doKO8vf+6jqXmPg5s= 2023-10-18T20:45:49.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JoEWmG2aW+doKO8vf+6jqXmPg5s=', 'Content-Length': '0'} 2023-10-18T20:45:49.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065974a58c8c87787-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.238 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.239 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.240 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.241 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.241 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.242 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.243 DEBUG:boto:Data: 2023-10-18T20:45:49.244 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.244 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.245 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.246 DEBUG:boto:Params: {} 2023-10-18T20:45:49.247 DEBUG:boto:Token: None 2023-10-18T20:45:49.247 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.248 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C9z852BChWfFQDgSDcfAq/7TaLY= 2023-10-18T20:45:49.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C9z852BChWfFQDgSDcfAq/7TaLY=', 'Content-Length': '0'} 2023-10-18T20:45:49.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000440aba03284a9a41-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.253 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.253 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.254 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.255 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.255 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.256 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.257 DEBUG:boto:Data: 2023-10-18T20:45:49.257 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.258 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.259 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.260 DEBUG:boto:Params: {} 2023-10-18T20:45:49.260 DEBUG:boto:Token: None 2023-10-18T20:45:49.261 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.262 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:C9z852BChWfFQDgSDcfAq/7TaLY= 2023-10-18T20:45:49.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:C9z852BChWfFQDgSDcfAq/7TaLY=', 'Content-Length': '0'} 2023-10-18T20:45:49.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8b19238612a8c2e-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:45:49.267 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.268 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.269 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.269 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.270 DEBUG:boto:Method: GET 2023-10-18T20:45:49.271 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.272 DEBUG:boto:Data: 2023-10-18T20:45:49.272 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.273 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.273 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.274 DEBUG:boto:Params: {} 2023-10-18T20:45:49.275 DEBUG:boto:Token: None 2023-10-18T20:45:49.275 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JoEWmG2aW+doKO8vf+6jqXmPg5s= 2023-10-18T20:45:49.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JoEWmG2aW+doKO8vf+6jqXmPg5s=', 'Content-Length': '0'} 2023-10-18T20:45:49.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098b02fdc4b6e1366-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.279 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.280 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.280 DEBUG:boto:path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.280 DEBUG:boto:auth_path=/bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.281 DEBUG:boto:Method: GET 2023-10-18T20:45:49.281 DEBUG:boto:Path: /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.282 DEBUG:boto:Data: 2023-10-18T20:45:49.282 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.283 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.283 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.283 DEBUG:boto:Params: {} 2023-10-18T20:45:49.284 DEBUG:boto:Token: None 2023-10-18T20:45:49.284 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57?versionId=null 2023-10-18T20:45:49.285 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JoEWmG2aW+doKO8vf+6jqXmPg5s= 2023-10-18T20:45:49.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JoEWmG2aW+doKO8vf+6jqXmPg5s=', 'Content-Length': '0'} 2023-10-18T20:45:49.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd3d6d5ba2e1bc1-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.288 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.289 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.289 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.290 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.290 DEBUG:boto:Method: GET 2023-10-18T20:45:49.290 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.291 DEBUG:boto:Data: 2023-10-18T20:45:49.291 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.292 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.292 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.293 DEBUG:boto:Params: {} 2023-10-18T20:45:49.293 DEBUG:boto:Token: None 2023-10-18T20:45:49.294 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.294 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2e5a416195c6b3b-00653043fd-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.298 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.299 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.299 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.300 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.300 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.300 DEBUG:boto:Method: GET 2023-10-18T20:45:49.301 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.301 DEBUG:boto:Data: 2023-10-18T20:45:49.302 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.303 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.303 DEBUG:boto:Params: {} 2023-10-18T20:45:49.303 DEBUG:boto:Token: None 2023-10-18T20:45:49.304 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f46ffd4af700658b-00653043fd-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.310 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.310 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.311 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.311 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.312 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:45:49.312 DEBUG:boto:Data: 2023-10-18T20:45:49.312 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.313 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.313 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.314 DEBUG:boto:Params: {} 2023-10-18T20:45:49.314 DEBUG:boto:Token: None 2023-10-18T20:45:49.315 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57 2023-10-18T20:45:49.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2uPL/q3MCecG7ibMBNz0rzhGJmI= 2023-10-18T20:45:49.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2uPL/q3MCecG7ibMBNz0rzhGJmI=', 'Content-Length': '0'} 2023-10-18T20:45:49.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a846947cc285eca-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.318 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.319 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.319 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.319 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:45:49.320 DEBUG:boto:Data: 2023-10-18T20:45:49.320 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.321 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.321 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.321 DEBUG:boto:Params: {} 2023-10-18T20:45:49.322 DEBUG:boto:Token: None 2023-10-18T20:45:49.322 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57 2023-10-18T20:45:49.323 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2uPL/q3MCecG7ibMBNz0rzhGJmI= 2023-10-18T20:45:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2uPL/q3MCecG7ibMBNz0rzhGJmI=', 'Content-Length': '0'} 2023-10-18T20:45:49.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eacf2037dd5fbb2d-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-18T20:45:49.326 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.327 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.327 DEBUG:boto:Method: GET 2023-10-18T20:45:49.327 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:45:49.328 DEBUG:boto:Data: 2023-10-18T20:45:49.328 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.329 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.329 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.330 DEBUG:boto:Params: {} 2023-10-18T20:45:49.330 DEBUG:boto:Token: None 2023-10-18T20:45:49.331 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57 2023-10-18T20:45:49.331 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qKA4ZbVwKZpso8CIsWWuMNyqeBc= 2023-10-18T20:45:49.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qKA4ZbVwKZpso8CIsWWuMNyqeBc=', 'Content-Length': '0'} 2023-10-18T20:45:49.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3dfbc2e7bfaf6a2-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.334 DEBUG:boto:path=/bofrnf-59/obj57 2023-10-18T20:45:49.335 DEBUG:boto:auth_path=/bofrnf-59/obj57 2023-10-18T20:45:49.335 DEBUG:boto:Method: GET 2023-10-18T20:45:49.335 DEBUG:boto:Path: /bofrnf-59/obj57 2023-10-18T20:45:49.336 DEBUG:boto:Data: 2023-10-18T20:45:49.336 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.337 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.337 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.337 DEBUG:boto:Params: {} 2023-10-18T20:45:49.338 DEBUG:boto:Token: None 2023-10-18T20:45:49.338 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj57 2023-10-18T20:45:49.339 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qKA4ZbVwKZpso8CIsWWuMNyqeBc= 2023-10-18T20:45:49.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qKA4ZbVwKZpso8CIsWWuMNyqeBc=', 'Content-Length': '0'} 2023-10-18T20:45:49.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014b57c0cdaf0d9a6-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.342 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.342 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.342 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.343 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.343 DEBUG:boto:Method: GET 2023-10-18T20:45:49.344 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.344 DEBUG:boto:Data: 2023-10-18T20:45:49.345 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.345 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.346 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.346 DEBUG:boto:Params: {} 2023-10-18T20:45:49.346 DEBUG:boto:Token: None 2023-10-18T20:45:49.347 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.347 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb06caf389d51882-00653043fd-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.351 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.352 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.352 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.353 DEBUG:boto:path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.353 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.354 DEBUG:boto:Method: GET 2023-10-18T20:45:49.354 DEBUG:boto:Path: /bofrnf-59/?prefix=obj57 2023-10-18T20:45:49.355 DEBUG:boto:Data: 2023-10-18T20:45:49.355 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.355 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.356 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.356 DEBUG:boto:Params: {} 2023-10-18T20:45:49.357 DEBUG:boto:Token: None 2023-10-18T20:45:49.357 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.358 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ec69ccc87a69b7a-00653043fd-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.363 DEBUG:boto:b'bofrnf-59obj571000falseobj572023-10-18T20:34:23.453Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:45:49.364 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.364 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.365 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.365 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.366 DEBUG:boto:Method: GET 2023-10-18T20:45:49.366 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.367 DEBUG:boto:Data: 2023-10-18T20:45:49.367 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.368 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.368 DEBUG:boto:Params: {} 2023-10-18T20:45:49.369 DEBUG:boto:Token: None 2023-10-18T20:45:49.369 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.370 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aLIyO15xW34pSPUoZ0FspB7YaPA= 2023-10-18T20:45:49.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aLIyO15xW34pSPUoZ0FspB7YaPA=', 'Content-Length': '0'} 2023-10-18T20:45:49.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8f1ba0c2816f802-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.414 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.415 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.415 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.416 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.416 DEBUG:boto:Method: GET 2023-10-18T20:45:49.417 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.417 DEBUG:boto:Data: 2023-10-18T20:45:49.417 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.418 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.418 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.419 DEBUG:boto:Params: {} 2023-10-18T20:45:49.419 DEBUG:boto:Token: None 2023-10-18T20:45:49.420 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.420 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aLIyO15xW34pSPUoZ0FspB7YaPA= 2023-10-18T20:45:49.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aLIyO15xW34pSPUoZ0FspB7YaPA=', 'Content-Length': '0'} 2023-10-18T20:45:49.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b95e09c2b446747-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.424 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.425 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.425 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.425 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.426 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.426 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.427 DEBUG:boto:Data: 2023-10-18T20:45:49.427 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.427 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.428 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.428 DEBUG:boto:Params: {} 2023-10-18T20:45:49.429 DEBUG:boto:Token: None 2023-10-18T20:45:49.429 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.430 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DzBOGkT9IUFtP3AslUd28RpVNGc= 2023-10-18T20:45:49.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DzBOGkT9IUFtP3AslUd28RpVNGc=', 'Content-Length': '0'} 2023-10-18T20:45:49.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108d03ee71c68111-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.433 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.433 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.434 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.434 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.435 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.435 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.435 DEBUG:boto:Data: 2023-10-18T20:45:49.436 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.436 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.437 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.437 DEBUG:boto:Params: {} 2023-10-18T20:45:49.437 DEBUG:boto:Token: None 2023-10-18T20:45:49.438 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.438 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DzBOGkT9IUFtP3AslUd28RpVNGc= 2023-10-18T20:45:49.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DzBOGkT9IUFtP3AslUd28RpVNGc=', 'Content-Length': '0'} 2023-10-18T20:45:49.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437a6e75a7eb5a46-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:45:49.442 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.442 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.443 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.443 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.444 DEBUG:boto:Method: GET 2023-10-18T20:45:49.444 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.445 DEBUG:boto:Data: 2023-10-18T20:45:49.445 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.446 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.446 DEBUG:boto:Params: {} 2023-10-18T20:45:49.447 DEBUG:boto:Token: None 2023-10-18T20:45:49.447 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aLIyO15xW34pSPUoZ0FspB7YaPA= 2023-10-18T20:45:49.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aLIyO15xW34pSPUoZ0FspB7YaPA=', 'Content-Length': '0'} 2023-10-18T20:45:49.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000287ece8f185b8f2b-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.451 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.451 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.452 DEBUG:boto:path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.452 DEBUG:boto:auth_path=/bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.452 DEBUG:boto:Method: GET 2023-10-18T20:45:49.453 DEBUG:boto:Path: /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.453 DEBUG:boto:Data: 2023-10-18T20:45:49.454 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.455 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.455 DEBUG:boto:Params: {} 2023-10-18T20:45:49.455 DEBUG:boto:Token: None 2023-10-18T20:45:49.456 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58?versionId=null 2023-10-18T20:45:49.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aLIyO15xW34pSPUoZ0FspB7YaPA= 2023-10-18T20:45:49.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aLIyO15xW34pSPUoZ0FspB7YaPA=', 'Content-Length': '0'} 2023-10-18T20:45:49.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2efe8487d3a09c8-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.460 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.460 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.461 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.461 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.461 DEBUG:boto:Method: GET 2023-10-18T20:45:49.462 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.462 DEBUG:boto:Data: 2023-10-18T20:45:49.463 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.463 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.464 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.464 DEBUG:boto:Params: {} 2023-10-18T20:45:49.464 DEBUG:boto:Token: None 2023-10-18T20:45:49.465 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.465 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071b5bf5e5f7869b2-00653043fd-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.469 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.470 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.470 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.471 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.471 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.472 DEBUG:boto:Method: GET 2023-10-18T20:45:49.472 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.472 DEBUG:boto:Data: 2023-10-18T20:45:49.473 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.473 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.474 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.474 DEBUG:boto:Params: {} 2023-10-18T20:45:49.475 DEBUG:boto:Token: None 2023-10-18T20:45:49.475 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7caf769dd4a40e3-00653043fd-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.522 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.523 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.524 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.524 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.525 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:45:49.525 DEBUG:boto:Data: 2023-10-18T20:45:49.525 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.526 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.526 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.527 DEBUG:boto:Params: {} 2023-10-18T20:45:49.527 DEBUG:boto:Token: None 2023-10-18T20:45:49.527 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58 2023-10-18T20:45:49.528 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8Xmq3o9dNBxG0vTvS22E0qBr8o= 2023-10-18T20:45:49.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8Xmq3o9dNBxG0vTvS22E0qBr8o=', 'Content-Length': '0'} 2023-10-18T20:45:49.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b93df1d03eaa2e8-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.531 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.532 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.532 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.532 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:45:49.533 DEBUG:boto:Data: 2023-10-18T20:45:49.533 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.534 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.534 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.534 DEBUG:boto:Params: {} 2023-10-18T20:45:49.535 DEBUG:boto:Token: None 2023-10-18T20:45:49.535 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58 2023-10-18T20:45:49.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p8Xmq3o9dNBxG0vTvS22E0qBr8o= 2023-10-18T20:45:49.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p8Xmq3o9dNBxG0vTvS22E0qBr8o=', 'Content-Length': '0'} 2023-10-18T20:45:49.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbf145a44a65239d-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-18T20:45:49.539 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.540 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.540 DEBUG:boto:Method: GET 2023-10-18T20:45:49.540 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:45:49.541 DEBUG:boto:Data: 2023-10-18T20:45:49.541 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.542 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.543 DEBUG:boto:Params: {} 2023-10-18T20:45:49.543 DEBUG:boto:Token: None 2023-10-18T20:45:49.544 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58 2023-10-18T20:45:49.544 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FW4H0r5Lfw6Rh7BlLZjxRc1JFGA= 2023-10-18T20:45:49.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FW4H0r5Lfw6Rh7BlLZjxRc1JFGA=', 'Content-Length': '0'} 2023-10-18T20:45:49.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6877b37d73ab2ba-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.594 DEBUG:boto:path=/bofrnf-59/obj58 2023-10-18T20:45:49.595 DEBUG:boto:auth_path=/bofrnf-59/obj58 2023-10-18T20:45:49.595 DEBUG:boto:Method: GET 2023-10-18T20:45:49.596 DEBUG:boto:Path: /bofrnf-59/obj58 2023-10-18T20:45:49.596 DEBUG:boto:Data: 2023-10-18T20:45:49.597 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.598 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.598 DEBUG:boto:Params: {} 2023-10-18T20:45:49.599 DEBUG:boto:Token: None 2023-10-18T20:45:49.599 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj58 2023-10-18T20:45:49.600 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FW4H0r5Lfw6Rh7BlLZjxRc1JFGA= 2023-10-18T20:45:49.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FW4H0r5Lfw6Rh7BlLZjxRc1JFGA=', 'Content-Length': '0'} 2023-10-18T20:45:49.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009107f1797cdf59ed-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.603 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.604 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.605 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.605 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.606 DEBUG:boto:Method: GET 2023-10-18T20:45:49.606 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.607 DEBUG:boto:Data: 2023-10-18T20:45:49.607 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.608 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.608 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.609 DEBUG:boto:Params: {} 2023-10-18T20:45:49.609 DEBUG:boto:Token: None 2023-10-18T20:45:49.610 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.610 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e95f31472bbf5509-00653043fd-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.614 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.615 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.615 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.616 DEBUG:boto:path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.616 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.617 DEBUG:boto:Method: GET 2023-10-18T20:45:49.617 DEBUG:boto:Path: /bofrnf-59/?prefix=obj58 2023-10-18T20:45:49.617 DEBUG:boto:Data: 2023-10-18T20:45:49.618 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.618 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.619 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.619 DEBUG:boto:Params: {} 2023-10-18T20:45:49.620 DEBUG:boto:Token: None 2023-10-18T20:45:49.620 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.621 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f38fb4baf4f7bd-00653043fd-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.626 DEBUG:boto:b'bofrnf-59obj581000falseobj582023-10-18T20:39:46.913Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-18T20:45:49.627 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.628 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.628 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.628 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.629 DEBUG:boto:Method: GET 2023-10-18T20:45:49.629 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.630 DEBUG:boto:Data: 2023-10-18T20:45:49.630 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.630 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.631 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.631 DEBUG:boto:Params: {} 2023-10-18T20:45:49.632 DEBUG:boto:Token: None 2023-10-18T20:45:49.632 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.633 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wjdWX+6gmx0CNmAtXmPoesGD9+w= 2023-10-18T20:45:49.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wjdWX+6gmx0CNmAtXmPoesGD9+w=', 'Content-Length': '0'} 2023-10-18T20:45:49.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f78470e35210dc49-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.636 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.636 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.637 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.637 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.638 DEBUG:boto:Method: GET 2023-10-18T20:45:49.638 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.639 DEBUG:boto:Data: 2023-10-18T20:45:49.639 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.639 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.640 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.640 DEBUG:boto:Params: {} 2023-10-18T20:45:49.641 DEBUG:boto:Token: None 2023-10-18T20:45:49.641 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.642 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wjdWX+6gmx0CNmAtXmPoesGD9+w= 2023-10-18T20:45:49.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wjdWX+6gmx0CNmAtXmPoesGD9+w=', 'Content-Length': '0'} 2023-10-18T20:45:49.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304c70209379bfbb-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.686 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.687 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.687 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.687 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.688 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.688 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.689 DEBUG:boto:Data: 2023-10-18T20:45:49.689 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.690 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.690 DEBUG:boto:Params: {} 2023-10-18T20:45:49.691 DEBUG:boto:Token: None 2023-10-18T20:45:49.691 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.692 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1zIsVkGaHFyNVdO6KR4xX38yHJM= 2023-10-18T20:45:49.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1zIsVkGaHFyNVdO6KR4xX38yHJM=', 'Content-Length': '0'} 2023-10-18T20:45:49.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca26adfdba3a639e-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.695 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.695 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.696 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.696 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.696 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.697 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.697 DEBUG:boto:Data: 2023-10-18T20:45:49.698 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.698 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.699 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.699 DEBUG:boto:Params: {} 2023-10-18T20:45:49.699 DEBUG:boto:Token: None 2023-10-18T20:45:49.700 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.700 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1zIsVkGaHFyNVdO6KR4xX38yHJM= 2023-10-18T20:45:49.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1zIsVkGaHFyNVdO6KR4xX38yHJM=', 'Content-Length': '0'} 2023-10-18T20:45:49.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e49f6291a58c537-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-18T20:45:49.704 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.704 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.705 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.705 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.706 DEBUG:boto:Method: GET 2023-10-18T20:45:49.706 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.706 DEBUG:boto:Data: 2023-10-18T20:45:49.707 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.707 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.708 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.708 DEBUG:boto:Params: {} 2023-10-18T20:45:49.709 DEBUG:boto:Token: None 2023-10-18T20:45:49.709 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.710 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wjdWX+6gmx0CNmAtXmPoesGD9+w= 2023-10-18T20:45:49.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wjdWX+6gmx0CNmAtXmPoesGD9+w=', 'Content-Length': '0'} 2023-10-18T20:45:49.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000763999b8a6519a57-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.713 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.713 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.714 DEBUG:boto:path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.714 DEBUG:boto:auth_path=/bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.714 DEBUG:boto:Method: GET 2023-10-18T20:45:49.715 DEBUG:boto:Path: /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.715 DEBUG:boto:Data: 2023-10-18T20:45:49.715 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.716 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.716 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.717 DEBUG:boto:Params: {} 2023-10-18T20:45:49.717 DEBUG:boto:Token: None 2023-10-18T20:45:49.717 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62?versionId=null 2023-10-18T20:45:49.718 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wjdWX+6gmx0CNmAtXmPoesGD9+w= 2023-10-18T20:45:49.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wjdWX+6gmx0CNmAtXmPoesGD9+w=', 'Content-Length': '0'} 2023-10-18T20:45:49.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fac0f24a37c5fc61-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.721 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.721 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.722 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.722 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.723 DEBUG:boto:Method: GET 2023-10-18T20:45:49.723 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.723 DEBUG:boto:Data: 2023-10-18T20:45:49.724 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.724 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.725 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.725 DEBUG:boto:Params: {} 2023-10-18T20:45:49.725 DEBUG:boto:Token: None 2023-10-18T20:45:49.726 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.726 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a42910347732282-00653043fd-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.730 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.731 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.731 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.731 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.732 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.732 DEBUG:boto:Method: GET 2023-10-18T20:45:49.733 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.733 DEBUG:boto:Data: 2023-10-18T20:45:49.733 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.734 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.734 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.735 DEBUG:boto:Params: {} 2023-10-18T20:45:49.735 DEBUG:boto:Token: None 2023-10-18T20:45:49.736 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.736 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b718b4b091a027a-00653043fd-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.742 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.742 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.743 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.743 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.744 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:45:49.744 DEBUG:boto:Data: 2023-10-18T20:45:49.745 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.745 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.745 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.746 DEBUG:boto:Params: {} 2023-10-18T20:45:49.746 DEBUG:boto:Token: None 2023-10-18T20:45:49.747 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62 2023-10-18T20:45:49.747 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BukVRbCZzelOdSqY0t9KtGAZm2c= 2023-10-18T20:45:49.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BukVRbCZzelOdSqY0t9KtGAZm2c=', 'Content-Length': '0'} 2023-10-18T20:45:49.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d137fb871fe8311d-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.750 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.751 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.751 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.751 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:45:49.752 DEBUG:boto:Data: 2023-10-18T20:45:49.752 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.753 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.754 DEBUG:boto:Params: {} 2023-10-18T20:45:49.754 DEBUG:boto:Token: None 2023-10-18T20:45:49.755 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62 2023-10-18T20:45:49.755 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BukVRbCZzelOdSqY0t9KtGAZm2c= 2023-10-18T20:45:49.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BukVRbCZzelOdSqY0t9KtGAZm2c=', 'Content-Length': '0'} 2023-10-18T20:45:49.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000812a8f0c9d830f25-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-18T20:45:49.759 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.759 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.760 DEBUG:boto:Method: GET 2023-10-18T20:45:49.760 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:45:49.761 DEBUG:boto:Data: 2023-10-18T20:45:49.761 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.762 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.762 DEBUG:boto:Params: {} 2023-10-18T20:45:49.763 DEBUG:boto:Token: None 2023-10-18T20:45:49.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62 2023-10-18T20:45:49.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GwXkZrjDvYQ33Alzu1opxb7ymb4= 2023-10-18T20:45:49.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GwXkZrjDvYQ33Alzu1opxb7ymb4=', 'Content-Length': '0'} 2023-10-18T20:45:49.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009519c889cb14fa5f-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.767 DEBUG:boto:path=/bofrnf-59/obj62 2023-10-18T20:45:49.768 DEBUG:boto:auth_path=/bofrnf-59/obj62 2023-10-18T20:45:49.768 DEBUG:boto:Method: GET 2023-10-18T20:45:49.768 DEBUG:boto:Path: /bofrnf-59/obj62 2023-10-18T20:45:49.769 DEBUG:boto:Data: 2023-10-18T20:45:49.769 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.770 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.770 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.771 DEBUG:boto:Params: {} 2023-10-18T20:45:49.771 DEBUG:boto:Token: None 2023-10-18T20:45:49.771 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj62 2023-10-18T20:45:49.772 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GwXkZrjDvYQ33Alzu1opxb7ymb4= 2023-10-18T20:45:49.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GwXkZrjDvYQ33Alzu1opxb7ymb4=', 'Content-Length': '0'} 2023-10-18T20:45:49.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd73f29738aac7e0-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.818 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.819 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.819 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.820 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.820 DEBUG:boto:Method: GET 2023-10-18T20:45:49.821 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.821 DEBUG:boto:Data: 2023-10-18T20:45:49.821 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.822 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.822 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.823 DEBUG:boto:Params: {} 2023-10-18T20:45:49.823 DEBUG:boto:Token: None 2023-10-18T20:45:49.824 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.824 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee673f6dbf54bbe-00653043fd-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.828 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.828 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.829 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.829 DEBUG:boto:path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.830 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.830 DEBUG:boto:Method: GET 2023-10-18T20:45:49.831 DEBUG:boto:Path: /bofrnf-59/?prefix=obj62 2023-10-18T20:45:49.831 DEBUG:boto:Data: 2023-10-18T20:45:49.831 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.832 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.832 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.833 DEBUG:boto:Params: {} 2023-10-18T20:45:49.833 DEBUG:boto:Token: None 2023-10-18T20:45:49.834 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.834 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8cde602a846cda-00653043fd-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.840 DEBUG:boto:b'bofrnf-59obj621000falseobj622023-10-18T20:41:07.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:45:49.841 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:49.841 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:49.842 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.842 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.843 DEBUG:boto:Method: GET 2023-10-18T20:45:49.843 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.844 DEBUG:boto:Data: 2023-10-18T20:45:49.844 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.844 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.845 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.845 DEBUG:boto:Params: {} 2023-10-18T20:45:49.846 DEBUG:boto:Token: None 2023-10-18T20:45:49.846 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.847 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FhiZhe57KjOy8UJSoxGYFoSc8xc= 2023-10-18T20:45:49.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FhiZhe57KjOy8UJSoxGYFoSc8xc=', 'Content-Length': '0'} 2023-10-18T20:45:49.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c627181f48cb11e-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.894 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:49.895 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:49.895 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.896 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.896 DEBUG:boto:Method: GET 2023-10-18T20:45:49.896 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.897 DEBUG:boto:Data: 2023-10-18T20:45:49.897 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.898 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.898 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.898 DEBUG:boto:Params: {} 2023-10-18T20:45:49.899 DEBUG:boto:Token: None 2023-10-18T20:45:49.899 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.900 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FhiZhe57KjOy8UJSoxGYFoSc8xc= 2023-10-18T20:45:49.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FhiZhe57KjOy8UJSoxGYFoSc8xc=', 'Content-Length': '0'} 2023-10-18T20:45:49.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcdaa13f77b8e9c2-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.903 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:49.904 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:49.904 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.904 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.905 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.905 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.906 DEBUG:boto:Data: 2023-10-18T20:45:49.906 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.907 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.907 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.907 DEBUG:boto:Params: {} 2023-10-18T20:45:49.908 DEBUG:boto:Token: None 2023-10-18T20:45:49.908 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.909 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o9TuL1oVOyhrKTIfaCLKsNh04Ng= 2023-10-18T20:45:49.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o9TuL1oVOyhrKTIfaCLKsNh04Ng=', 'Content-Length': '0'} 2023-10-18T20:45:49.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e0aba442c93ca3-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.912 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:49.912 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:49.912 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.913 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.913 DEBUG:boto:Method: HEAD 2023-10-18T20:45:49.914 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.914 DEBUG:boto:Data: 2023-10-18T20:45:49.915 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.915 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.915 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.916 DEBUG:boto:Params: {} 2023-10-18T20:45:49.916 DEBUG:boto:Token: None 2023-10-18T20:45:49.917 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.917 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o9TuL1oVOyhrKTIfaCLKsNh04Ng= 2023-10-18T20:45:49.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o9TuL1oVOyhrKTIfaCLKsNh04Ng=', 'Content-Length': '0'} 2023-10-18T20:45:49.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077bd16115ef5189f-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:45:49.921 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:49.921 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:49.921 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.922 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.922 DEBUG:boto:Method: GET 2023-10-18T20:45:49.923 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.923 DEBUG:boto:Data: 2023-10-18T20:45:49.924 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.924 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.924 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.925 DEBUG:boto:Params: {} 2023-10-18T20:45:49.925 DEBUG:boto:Token: None 2023-10-18T20:45:49.926 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FhiZhe57KjOy8UJSoxGYFoSc8xc= 2023-10-18T20:45:49.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FhiZhe57KjOy8UJSoxGYFoSc8xc=', 'Content-Length': '0'} 2023-10-18T20:45:49.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000897403de1989662-00653043fd-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.929 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:49.930 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:49.930 DEBUG:boto:path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.931 DEBUG:boto:auth_path=/bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.932 DEBUG:boto:Method: GET 2023-10-18T20:45:49.932 DEBUG:boto:Path: /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.933 DEBUG:boto:Data: 2023-10-18T20:45:49.933 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.933 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.934 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.934 DEBUG:boto:Params: {} 2023-10-18T20:45:49.935 DEBUG:boto:Token: None 2023-10-18T20:45:49.935 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/obj68?versionId=null 2023-10-18T20:45:49.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FhiZhe57KjOy8UJSoxGYFoSc8xc= 2023-10-18T20:45:49.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FhiZhe57KjOy8UJSoxGYFoSc8xc=', 'Content-Length': '0'} 2023-10-18T20:45:49.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a6671ad0dc9c654-00653043fd-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.939 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.939 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.940 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:49.940 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:49.941 DEBUG:boto:Method: GET 2023-10-18T20:45:49.941 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:45:49.941 DEBUG:boto:Data: 2023-10-18T20:45:49.942 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.942 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:49.943 DEBUG:boto:Port: 8001 2023-10-18T20:45:49.943 DEBUG:boto:Params: {} 2023-10-18T20:45:49.944 DEBUG:boto:Token: None 2023-10-18T20:45:49.944 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.945 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b2bf46338c39ab9-00653043fd-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:49.948 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:49.949 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:49.949 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:49.950 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:49.950 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:49.951 DEBUG:boto:Method: GET 2023-10-18T20:45:49.951 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:45:49.951 DEBUG:boto:Data: 2023-10-18T20:45:49.952 DEBUG:boto:Headers: {} 2023-10-18T20:45:49.952 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:49.953 DEBUG:boto:Port: 8000 2023-10-18T20:45:49.953 DEBUG:boto:Params: {} 2023-10-18T20:45:49.954 DEBUG:boto:Token: None 2023-10-18T20:45:49.954 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:49 GMT /bofrnf-59/ 2023-10-18T20:45:49.955 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc= 2023-10-18T20:45:49.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CM6NWm8d189wh2a0vOe8ufQThsc=', 'Content-Length': '0'} 2023-10-18T20:45:49.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe548acdc77b608f-00653043fd-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.002 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.003 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:50.004 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:50.004 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.005 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:45:50.005 DEBUG:boto:Data: 2023-10-18T20:45:50.005 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.006 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.006 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.007 DEBUG:boto:Params: {} 2023-10-18T20:45:50.007 DEBUG:boto:Token: None 2023-10-18T20:45:50.008 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj68 2023-10-18T20:45:50.008 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/MVzUOhOI3zTLttPTOvSUD0+W1s= 2023-10-18T20:45:50.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/MVzUOhOI3zTLttPTOvSUD0+W1s=', 'Content-Length': '0'} 2023-10-18T20:45:50.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002514dc9c64edcc5-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.012 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:50.012 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:50.013 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.013 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:45:50.013 DEBUG:boto:Data: 2023-10-18T20:45:50.014 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.015 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.015 DEBUG:boto:Params: {} 2023-10-18T20:45:50.015 DEBUG:boto:Token: None 2023-10-18T20:45:50.016 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj68 2023-10-18T20:45:50.016 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/MVzUOhOI3zTLttPTOvSUD0+W1s= 2023-10-18T20:45:50.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/MVzUOhOI3zTLttPTOvSUD0+W1s=', 'Content-Length': '0'} 2023-10-18T20:45:50.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f6d50d605f78364-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-18T20:45:50.020 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:50.021 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:50.021 DEBUG:boto:Method: GET 2023-10-18T20:45:50.021 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:45:50.022 DEBUG:boto:Data: 2023-10-18T20:45:50.022 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.023 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.023 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.024 DEBUG:boto:Params: {} 2023-10-18T20:45:50.024 DEBUG:boto:Token: None 2023-10-18T20:45:50.024 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj68 2023-10-18T20:45:50.025 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBda+sC1WLJPt+y7RPtiZSXypzs= 2023-10-18T20:45:50.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBda+sC1WLJPt+y7RPtiZSXypzs=', 'Content-Length': '0'} 2023-10-18T20:45:50.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae2103b2ba7228e5-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.070 DEBUG:boto:path=/bofrnf-59/obj68 2023-10-18T20:45:50.071 DEBUG:boto:auth_path=/bofrnf-59/obj68 2023-10-18T20:45:50.072 DEBUG:boto:Method: GET 2023-10-18T20:45:50.072 DEBUG:boto:Path: /bofrnf-59/obj68 2023-10-18T20:45:50.072 DEBUG:boto:Data: 2023-10-18T20:45:50.073 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.074 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.074 DEBUG:boto:Params: {} 2023-10-18T20:45:50.075 DEBUG:boto:Token: None 2023-10-18T20:45:50.075 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj68 2023-10-18T20:45:50.076 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BBda+sC1WLJPt+y7RPtiZSXypzs= 2023-10-18T20:45:50.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BBda+sC1WLJPt+y7RPtiZSXypzs=', 'Content-Length': '0'} 2023-10-18T20:45:50.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bfd263721bdde17-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.079 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.079 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.080 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:50.080 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:50.081 DEBUG:boto:Method: GET 2023-10-18T20:45:50.081 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:45:50.082 DEBUG:boto:Data: 2023-10-18T20:45:50.082 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.082 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.083 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.083 DEBUG:boto:Params: {} 2023-10-18T20:45:50.084 DEBUG:boto:Token: None 2023-10-18T20:45:50.084 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.085 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e1d60c43d04f407-00653043fe-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.088 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.089 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.090 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.090 DEBUG:boto:path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:50.091 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj68 2023-10-18T20:45:50.091 DEBUG:boto:Method: GET 2023-10-18T20:45:50.091 DEBUG:boto:Path: /bofrnf-59/?prefix=obj68 2023-10-18T20:45:50.092 DEBUG:boto:Data: 2023-10-18T20:45:50.092 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.093 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.093 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.094 DEBUG:boto:Params: {} 2023-10-18T20:45:50.094 DEBUG:boto:Token: None 2023-10-18T20:45:50.094 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.095 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a227fe4f5a38bd-00653043fe-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.101 DEBUG:boto:b'bofrnf-59obj681000falseobj682023-10-18T20:33:05.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:45:50.102 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.102 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.103 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.103 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.104 DEBUG:boto:Method: GET 2023-10-18T20:45:50.104 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.104 DEBUG:boto:Data: 2023-10-18T20:45:50.105 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.105 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.106 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.106 DEBUG:boto:Params: {} 2023-10-18T20:45:50.107 DEBUG:boto:Token: None 2023-10-18T20:45:50.107 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.108 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VnQdvPGDXpC4FoL/iTzgd0neqlE= 2023-10-18T20:45:50.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VnQdvPGDXpC4FoL/iTzgd0neqlE=', 'Content-Length': '0'} 2023-10-18T20:45:50.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d34877e1a84e74e4-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.111 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.112 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.112 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.112 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.113 DEBUG:boto:Method: GET 2023-10-18T20:45:50.113 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.114 DEBUG:boto:Data: 2023-10-18T20:45:50.114 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.115 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.115 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.116 DEBUG:boto:Params: {} 2023-10-18T20:45:50.116 DEBUG:boto:Token: None 2023-10-18T20:45:50.116 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.117 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VnQdvPGDXpC4FoL/iTzgd0neqlE= 2023-10-18T20:45:50.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VnQdvPGDXpC4FoL/iTzgd0neqlE=', 'Content-Length': '0'} 2023-10-18T20:45:50.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000799a75ad30df91a1-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.162 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.163 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.164 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.164 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.164 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.165 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.165 DEBUG:boto:Data: 2023-10-18T20:45:50.166 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.166 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.166 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.167 DEBUG:boto:Params: {} 2023-10-18T20:45:50.167 DEBUG:boto:Token: None 2023-10-18T20:45:50.168 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.168 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t8si3grAN31C6lPpu9uErD+ogzg= 2023-10-18T20:45:50.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t8si3grAN31C6lPpu9uErD+ogzg=', 'Content-Length': '0'} 2023-10-18T20:45:50.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915d8938e2443620-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.171 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.172 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.172 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.173 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.173 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.174 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.174 DEBUG:boto:Data: 2023-10-18T20:45:50.174 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.175 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.175 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.176 DEBUG:boto:Params: {} 2023-10-18T20:45:50.176 DEBUG:boto:Token: None 2023-10-18T20:45:50.176 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.177 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:t8si3grAN31C6lPpu9uErD+ogzg= 2023-10-18T20:45:50.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:t8si3grAN31C6lPpu9uErD+ogzg=', 'Content-Length': '0'} 2023-10-18T20:45:50.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074e313ed7cd80b4a-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:45:50.180 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.181 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.181 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.182 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.182 DEBUG:boto:Method: GET 2023-10-18T20:45:50.183 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.183 DEBUG:boto:Data: 2023-10-18T20:45:50.184 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.184 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.184 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.185 DEBUG:boto:Params: {} 2023-10-18T20:45:50.185 DEBUG:boto:Token: None 2023-10-18T20:45:50.186 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.186 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VnQdvPGDXpC4FoL/iTzgd0neqlE= 2023-10-18T20:45:50.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VnQdvPGDXpC4FoL/iTzgd0neqlE=', 'Content-Length': '0'} 2023-10-18T20:45:50.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb43eeb72507888-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.234 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.235 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.235 DEBUG:boto:path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.236 DEBUG:boto:auth_path=/bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.236 DEBUG:boto:Method: GET 2023-10-18T20:45:50.237 DEBUG:boto:Path: /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.237 DEBUG:boto:Data: 2023-10-18T20:45:50.237 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.238 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.238 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.239 DEBUG:boto:Params: {} 2023-10-18T20:45:50.239 DEBUG:boto:Token: None 2023-10-18T20:45:50.240 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69?versionId=null 2023-10-18T20:45:50.240 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VnQdvPGDXpC4FoL/iTzgd0neqlE= 2023-10-18T20:45:50.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VnQdvPGDXpC4FoL/iTzgd0neqlE=', 'Content-Length': '0'} 2023-10-18T20:45:50.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e62ee08f867a1639-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.244 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.244 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.245 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.245 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.246 DEBUG:boto:Method: GET 2023-10-18T20:45:50.246 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.246 DEBUG:boto:Data: 2023-10-18T20:45:50.247 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.247 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.248 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.248 DEBUG:boto:Params: {} 2023-10-18T20:45:50.249 DEBUG:boto:Token: None 2023-10-18T20:45:50.249 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cabfeecfd2b296ef-00653043fe-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.254 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.254 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.255 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.255 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.256 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.256 DEBUG:boto:Method: GET 2023-10-18T20:45:50.256 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.257 DEBUG:boto:Data: 2023-10-18T20:45:50.257 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.258 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.258 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.258 DEBUG:boto:Params: {} 2023-10-18T20:45:50.259 DEBUG:boto:Token: None 2023-10-18T20:45:50.259 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.260 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2bbec1454237ff3-00653043fe-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.265 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.266 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.266 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.266 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.267 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:45:50.267 DEBUG:boto:Data: 2023-10-18T20:45:50.267 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.268 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.268 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.269 DEBUG:boto:Params: {} 2023-10-18T20:45:50.269 DEBUG:boto:Token: None 2023-10-18T20:45:50.269 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69 2023-10-18T20:45:50.270 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Wi9XFV9xIUYxjGgU7F0p6lILlUk= 2023-10-18T20:45:50.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Wi9XFV9xIUYxjGgU7F0p6lILlUk=', 'Content-Length': '0'} 2023-10-18T20:45:50.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ed647e9d0d1456-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.273 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.273 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.274 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.274 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:45:50.275 DEBUG:boto:Data: 2023-10-18T20:45:50.275 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.275 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.276 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.276 DEBUG:boto:Params: {} 2023-10-18T20:45:50.277 DEBUG:boto:Token: None 2023-10-18T20:45:50.277 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69 2023-10-18T20:45:50.278 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Wi9XFV9xIUYxjGgU7F0p6lILlUk= 2023-10-18T20:45:50.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Wi9XFV9xIUYxjGgU7F0p6lILlUk=', 'Content-Length': '0'} 2023-10-18T20:45:50.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad18ea3649101459-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-18T20:45:50.281 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.281 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.282 DEBUG:boto:Method: GET 2023-10-18T20:45:50.282 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:45:50.283 DEBUG:boto:Data: 2023-10-18T20:45:50.283 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.284 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.284 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.284 DEBUG:boto:Params: {} 2023-10-18T20:45:50.285 DEBUG:boto:Token: None 2023-10-18T20:45:50.285 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69 2023-10-18T20:45:50.286 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4lAui4tQcwOy1ZDKt5d3bN3hIi4= 2023-10-18T20:45:50.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4lAui4tQcwOy1ZDKt5d3bN3hIi4=', 'Content-Length': '0'} 2023-10-18T20:45:50.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8b1f717b4ff453-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.289 DEBUG:boto:path=/bofrnf-59/obj69 2023-10-18T20:45:50.290 DEBUG:boto:auth_path=/bofrnf-59/obj69 2023-10-18T20:45:50.290 DEBUG:boto:Method: GET 2023-10-18T20:45:50.290 DEBUG:boto:Path: /bofrnf-59/obj69 2023-10-18T20:45:50.291 DEBUG:boto:Data: 2023-10-18T20:45:50.291 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.292 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.292 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.293 DEBUG:boto:Params: {} 2023-10-18T20:45:50.293 DEBUG:boto:Token: None 2023-10-18T20:45:50.294 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj69 2023-10-18T20:45:50.294 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4lAui4tQcwOy1ZDKt5d3bN3hIi4= 2023-10-18T20:45:50.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4lAui4tQcwOy1ZDKt5d3bN3hIi4=', 'Content-Length': '0'} 2023-10-18T20:45:50.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c6404ad4380f5c-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.338 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.339 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.339 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.340 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.340 DEBUG:boto:Method: GET 2023-10-18T20:45:50.341 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.341 DEBUG:boto:Data: 2023-10-18T20:45:50.342 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.342 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.343 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.343 DEBUG:boto:Params: {} 2023-10-18T20:45:50.343 DEBUG:boto:Token: None 2023-10-18T20:45:50.344 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.344 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000043de0f76401b265-00653043fe-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.348 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.349 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.349 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.350 DEBUG:boto:path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.350 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.351 DEBUG:boto:Method: GET 2023-10-18T20:45:50.351 DEBUG:boto:Path: /bofrnf-59/?prefix=obj69 2023-10-18T20:45:50.351 DEBUG:boto:Data: 2023-10-18T20:45:50.352 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.352 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.353 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.353 DEBUG:boto:Params: {} 2023-10-18T20:45:50.354 DEBUG:boto:Token: None 2023-10-18T20:45:50.354 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.355 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc333dbc9f6b6476-00653043fe-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.360 DEBUG:boto:b'bofrnf-59obj691000falseobj692023-10-18T20:38:22.150Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-18T20:45:50.361 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.361 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.362 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.362 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.363 DEBUG:boto:Method: GET 2023-10-18T20:45:50.363 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.364 DEBUG:boto:Data: 2023-10-18T20:45:50.364 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.364 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.365 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.365 DEBUG:boto:Params: {} 2023-10-18T20:45:50.366 DEBUG:boto:Token: None 2023-10-18T20:45:50.366 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.367 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V5iu6SEK/En+PXoKHjpEVit9+Ok= 2023-10-18T20:45:50.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V5iu6SEK/En+PXoKHjpEVit9+Ok=', 'Content-Length': '0'} 2023-10-18T20:45:50.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa6727a42b1e7609-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.371 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.371 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.371 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.372 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.372 DEBUG:boto:Method: GET 2023-10-18T20:45:50.373 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.373 DEBUG:boto:Data: 2023-10-18T20:45:50.373 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.374 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.375 DEBUG:boto:Params: {} 2023-10-18T20:45:50.375 DEBUG:boto:Token: None 2023-10-18T20:45:50.376 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.376 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V5iu6SEK/En+PXoKHjpEVit9+Ok= 2023-10-18T20:45:50.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V5iu6SEK/En+PXoKHjpEVit9+Ok=', 'Content-Length': '0'} 2023-10-18T20:45:50.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dac9ec5e1255de14-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.379 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.380 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.380 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.381 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.381 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.382 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.382 DEBUG:boto:Data: 2023-10-18T20:45:50.382 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.383 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.383 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.384 DEBUG:boto:Params: {} 2023-10-18T20:45:50.384 DEBUG:boto:Token: None 2023-10-18T20:45:50.385 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.385 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5b3srdnh9eqN7x8455PTefAS7J0= 2023-10-18T20:45:50.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5b3srdnh9eqN7x8455PTefAS7J0=', 'Content-Length': '0'} 2023-10-18T20:45:50.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a1d5fd6548a3d01-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.388 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.389 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.389 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.389 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.390 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.390 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.391 DEBUG:boto:Data: 2023-10-18T20:45:50.391 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.391 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.392 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.392 DEBUG:boto:Params: {} 2023-10-18T20:45:50.393 DEBUG:boto:Token: None 2023-10-18T20:45:50.393 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.393 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5b3srdnh9eqN7x8455PTefAS7J0= 2023-10-18T20:45:50.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5b3srdnh9eqN7x8455PTefAS7J0=', 'Content-Length': '0'} 2023-10-18T20:45:50.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa930225089a49d9-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-18T20:45:50.397 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.398 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.398 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.399 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.399 DEBUG:boto:Method: GET 2023-10-18T20:45:50.401 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.402 DEBUG:boto:Data: 2023-10-18T20:45:50.402 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.403 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.403 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.403 DEBUG:boto:Params: {} 2023-10-18T20:45:50.404 DEBUG:boto:Token: None 2023-10-18T20:45:50.404 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.405 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V5iu6SEK/En+PXoKHjpEVit9+Ok= 2023-10-18T20:45:50.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V5iu6SEK/En+PXoKHjpEVit9+Ok=', 'Content-Length': '0'} 2023-10-18T20:45:50.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ab9d9fbd4465a80-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.408 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.409 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.409 DEBUG:boto:path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.410 DEBUG:boto:auth_path=/bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.410 DEBUG:boto:Method: GET 2023-10-18T20:45:50.410 DEBUG:boto:Path: /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.411 DEBUG:boto:Data: 2023-10-18T20:45:50.411 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.412 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.412 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.413 DEBUG:boto:Params: {} 2023-10-18T20:45:50.413 DEBUG:boto:Token: None 2023-10-18T20:45:50.413 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74?versionId=null 2023-10-18T20:45:50.414 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:V5iu6SEK/En+PXoKHjpEVit9+Ok= 2023-10-18T20:45:50.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V5iu6SEK/En+PXoKHjpEVit9+Ok=', 'Content-Length': '0'} 2023-10-18T20:45:50.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9a52cdc6fb73cf9-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.462 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.463 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.464 DEBUG:boto:path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.464 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.464 DEBUG:boto:Method: GET 2023-10-18T20:45:50.465 DEBUG:boto:Path: /bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.465 DEBUG:boto:Data: 2023-10-18T20:45:50.466 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.466 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.467 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.467 DEBUG:boto:Params: {} 2023-10-18T20:45:50.467 DEBUG:boto:Token: None 2023-10-18T20:45:50.468 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.468 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098d35a80d46a9497-00653043fe-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.472 DEBUG:boto:b'bofrnf-59obj741000falseobj742023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.473 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.473 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.474 DEBUG:boto:path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.474 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.475 DEBUG:boto:Method: GET 2023-10-18T20:45:50.475 DEBUG:boto:Path: /bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.475 DEBUG:boto:Data: 2023-10-18T20:45:50.476 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.476 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.477 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.477 DEBUG:boto:Params: {} 2023-10-18T20:45:50.477 DEBUG:boto:Token: None 2023-10-18T20:45:50.478 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.478 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba63142b93f8e372-00653043fe-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.484 DEBUG:boto:b'bofrnf-59obj741000falseobj742023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.484 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.485 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.485 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.485 DEBUG:boto:Path: /bofrnf-59/obj74 2023-10-18T20:45:50.486 DEBUG:boto:Data: 2023-10-18T20:45:50.486 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.487 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.487 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.487 DEBUG:boto:Params: {} 2023-10-18T20:45:50.488 DEBUG:boto:Token: None 2023-10-18T20:45:50.488 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74 2023-10-18T20:45:50.489 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5QujxhvDMptqxTOVNt3pO4b9nNo= 2023-10-18T20:45:50.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5QujxhvDMptqxTOVNt3pO4b9nNo=', 'Content-Length': '0'} 2023-10-18T20:45:50.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fbf05228cd8d7be-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.491 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.492 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.492 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.493 DEBUG:boto:Path: /bofrnf-59/obj74 2023-10-18T20:45:50.493 DEBUG:boto:Data: 2023-10-18T20:45:50.493 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.494 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.495 DEBUG:boto:Params: {} 2023-10-18T20:45:50.495 DEBUG:boto:Token: None 2023-10-18T20:45:50.496 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74 2023-10-18T20:45:50.496 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5QujxhvDMptqxTOVNt3pO4b9nNo= 2023-10-18T20:45:50.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5QujxhvDMptqxTOVNt3pO4b9nNo=', 'Content-Length': '0'} 2023-10-18T20:45:50.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6a20ca357526c59-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-18T20:45:50.500 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.500 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.500 DEBUG:boto:Method: GET 2023-10-18T20:45:50.501 DEBUG:boto:Path: /bofrnf-59/obj74 2023-10-18T20:45:50.501 DEBUG:boto:Data: 2023-10-18T20:45:50.502 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.502 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.503 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.503 DEBUG:boto:Params: {} 2023-10-18T20:45:50.504 DEBUG:boto:Token: None 2023-10-18T20:45:50.504 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74 2023-10-18T20:45:50.505 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+zNIWbuyUzGyYAZwXz1j1KMLbrA= 2023-10-18T20:45:50.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+zNIWbuyUzGyYAZwXz1j1KMLbrA=', 'Content-Length': '0'} 2023-10-18T20:45:50.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048a9aca9d467f69f-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.554 DEBUG:boto:path=/bofrnf-59/obj74 2023-10-18T20:45:50.555 DEBUG:boto:auth_path=/bofrnf-59/obj74 2023-10-18T20:45:50.556 DEBUG:boto:Method: GET 2023-10-18T20:45:50.556 DEBUG:boto:Path: /bofrnf-59/obj74 2023-10-18T20:45:50.556 DEBUG:boto:Data: 2023-10-18T20:45:50.557 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.557 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.558 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.558 DEBUG:boto:Params: {} 2023-10-18T20:45:50.559 DEBUG:boto:Token: None 2023-10-18T20:45:50.559 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj74 2023-10-18T20:45:50.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+zNIWbuyUzGyYAZwXz1j1KMLbrA= 2023-10-18T20:45:50.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+zNIWbuyUzGyYAZwXz1j1KMLbrA=', 'Content-Length': '0'} 2023-10-18T20:45:50.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:43:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e61f2ade7819e4c-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.563 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.563 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.564 DEBUG:boto:path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.564 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.564 DEBUG:boto:Method: GET 2023-10-18T20:45:50.565 DEBUG:boto:Path: /bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.565 DEBUG:boto:Data: 2023-10-18T20:45:50.566 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.566 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.567 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.567 DEBUG:boto:Params: {} 2023-10-18T20:45:50.568 DEBUG:boto:Token: None 2023-10-18T20:45:50.568 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.569 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e5bf29613b8ecdc-00653043fe-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.572 DEBUG:boto:b'bofrnf-59obj741000falseobj742023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.573 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.574 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.574 DEBUG:boto:path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.574 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.575 DEBUG:boto:Method: GET 2023-10-18T20:45:50.575 DEBUG:boto:Path: /bofrnf-59/?prefix=obj74 2023-10-18T20:45:50.576 DEBUG:boto:Data: 2023-10-18T20:45:50.576 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.576 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.577 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.577 DEBUG:boto:Params: {} 2023-10-18T20:45:50.578 DEBUG:boto:Token: None 2023-10-18T20:45:50.578 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.579 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af461c968ae807fa-00653043fe-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.626 DEBUG:boto:b'bofrnf-59obj741000falseobj742023-10-18T20:43:09.667Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:45:50.628 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.628 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.629 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.629 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.629 DEBUG:boto:Method: GET 2023-10-18T20:45:50.630 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.630 DEBUG:boto:Data: 2023-10-18T20:45:50.631 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.631 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.632 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.632 DEBUG:boto:Params: {} 2023-10-18T20:45:50.633 DEBUG:boto:Token: None 2023-10-18T20:45:50.633 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.634 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mqQznOCnr7JqbKHcsluLTwRWJCU= 2023-10-18T20:45:50.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mqQznOCnr7JqbKHcsluLTwRWJCU=', 'Content-Length': '0'} 2023-10-18T20:45:50.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003aaf68b8b94ec3c-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.637 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.638 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.638 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.639 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.639 DEBUG:boto:Method: GET 2023-10-18T20:45:50.640 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.640 DEBUG:boto:Data: 2023-10-18T20:45:50.640 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.641 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.642 DEBUG:boto:Params: {} 2023-10-18T20:45:50.642 DEBUG:boto:Token: None 2023-10-18T20:45:50.643 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mqQznOCnr7JqbKHcsluLTwRWJCU= 2023-10-18T20:45:50.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mqQznOCnr7JqbKHcsluLTwRWJCU=', 'Content-Length': '0'} 2023-10-18T20:45:50.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000791c9788a3d0ad13-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.647 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.647 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.647 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.648 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.648 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.649 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.649 DEBUG:boto:Data: 2023-10-18T20:45:50.649 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.650 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.650 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.651 DEBUG:boto:Params: {} 2023-10-18T20:45:50.651 DEBUG:boto:Token: None 2023-10-18T20:45:50.652 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0HbDZ1BZBRWiF0cSBOjw6ZtEHfM= 2023-10-18T20:45:50.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0HbDZ1BZBRWiF0cSBOjw6ZtEHfM=', 'Content-Length': '0'} 2023-10-18T20:45:50.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebcfb3acf2ff3b06-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.655 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.655 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.656 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.656 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.657 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.657 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.657 DEBUG:boto:Data: 2023-10-18T20:45:50.658 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.658 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.659 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.659 DEBUG:boto:Params: {} 2023-10-18T20:45:50.659 DEBUG:boto:Token: None 2023-10-18T20:45:50.660 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.660 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0HbDZ1BZBRWiF0cSBOjw6ZtEHfM= 2023-10-18T20:45:50.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0HbDZ1BZBRWiF0cSBOjw6ZtEHfM=', 'Content-Length': '0'} 2023-10-18T20:45:50.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac86c7bc572df025-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:45:50.664 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.664 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.665 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.665 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.666 DEBUG:boto:Method: GET 2023-10-18T20:45:50.666 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.667 DEBUG:boto:Data: 2023-10-18T20:45:50.667 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.668 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.668 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.668 DEBUG:boto:Params: {} 2023-10-18T20:45:50.669 DEBUG:boto:Token: None 2023-10-18T20:45:50.669 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.670 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mqQznOCnr7JqbKHcsluLTwRWJCU= 2023-10-18T20:45:50.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mqQznOCnr7JqbKHcsluLTwRWJCU=', 'Content-Length': '0'} 2023-10-18T20:45:50.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf064bcf70a2387-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.673 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.673 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.674 DEBUG:boto:path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.674 DEBUG:boto:auth_path=/bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.675 DEBUG:boto:Method: GET 2023-10-18T20:45:50.675 DEBUG:boto:Path: /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.675 DEBUG:boto:Data: 2023-10-18T20:45:50.676 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.676 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.677 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.677 DEBUG:boto:Params: {} 2023-10-18T20:45:50.677 DEBUG:boto:Token: None 2023-10-18T20:45:50.678 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76?versionId=null 2023-10-18T20:45:50.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mqQznOCnr7JqbKHcsluLTwRWJCU= 2023-10-18T20:45:50.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mqQznOCnr7JqbKHcsluLTwRWJCU=', 'Content-Length': '0'} 2023-10-18T20:45:50.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000751e489ad9ea0e2e-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.682 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.682 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.683 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.683 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.684 DEBUG:boto:Method: GET 2023-10-18T20:45:50.684 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.685 DEBUG:boto:Data: 2023-10-18T20:45:50.685 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.685 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.686 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.686 DEBUG:boto:Params: {} 2023-10-18T20:45:50.687 DEBUG:boto:Token: None 2023-10-18T20:45:50.687 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.688 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b22e45b12f1e6f8-00653043fe-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.691 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.692 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.693 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.693 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.693 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.694 DEBUG:boto:Method: GET 2023-10-18T20:45:50.694 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.695 DEBUG:boto:Data: 2023-10-18T20:45:50.695 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.696 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.696 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.696 DEBUG:boto:Params: {} 2023-10-18T20:45:50.697 DEBUG:boto:Token: None 2023-10-18T20:45:50.697 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.698 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001161bf0a0b067c8-00653043fe-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.703 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.704 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.704 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.705 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.705 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:45:50.705 DEBUG:boto:Data: 2023-10-18T20:45:50.706 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.707 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.707 DEBUG:boto:Params: {} 2023-10-18T20:45:50.708 DEBUG:boto:Token: None 2023-10-18T20:45:50.708 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76 2023-10-18T20:45:50.708 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TWUdnnZ9buK1DH8QoKXmUVJqS9g= 2023-10-18T20:45:50.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TWUdnnZ9buK1DH8QoKXmUVJqS9g=', 'Content-Length': '0'} 2023-10-18T20:45:50.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a09c83140453d1b7-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.712 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.712 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.712 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.713 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:45:50.713 DEBUG:boto:Data: 2023-10-18T20:45:50.714 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.714 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.715 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.715 DEBUG:boto:Params: {} 2023-10-18T20:45:50.715 DEBUG:boto:Token: None 2023-10-18T20:45:50.716 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76 2023-10-18T20:45:50.716 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TWUdnnZ9buK1DH8QoKXmUVJqS9g= 2023-10-18T20:45:50.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TWUdnnZ9buK1DH8QoKXmUVJqS9g=', 'Content-Length': '0'} 2023-10-18T20:45:50.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abb0a01f830124e7-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-18T20:45:50.720 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.720 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.721 DEBUG:boto:Method: GET 2023-10-18T20:45:50.721 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:45:50.722 DEBUG:boto:Data: 2023-10-18T20:45:50.722 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.722 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.723 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.723 DEBUG:boto:Params: {} 2023-10-18T20:45:50.724 DEBUG:boto:Token: None 2023-10-18T20:45:50.724 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76 2023-10-18T20:45:50.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+IIWy1n0tYhSeHEZsyXPwRGi9jc= 2023-10-18T20:45:50.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+IIWy1n0tYhSeHEZsyXPwRGi9jc=', 'Content-Length': '0'} 2023-10-18T20:45:50.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd67036d368b79be-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.774 DEBUG:boto:path=/bofrnf-59/obj76 2023-10-18T20:45:50.775 DEBUG:boto:auth_path=/bofrnf-59/obj76 2023-10-18T20:45:50.775 DEBUG:boto:Method: GET 2023-10-18T20:45:50.776 DEBUG:boto:Path: /bofrnf-59/obj76 2023-10-18T20:45:50.776 DEBUG:boto:Data: 2023-10-18T20:45:50.777 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.777 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.778 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.778 DEBUG:boto:Params: {} 2023-10-18T20:45:50.779 DEBUG:boto:Token: None 2023-10-18T20:45:50.779 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj76 2023-10-18T20:45:50.780 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+IIWy1n0tYhSeHEZsyXPwRGi9jc= 2023-10-18T20:45:50.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+IIWy1n0tYhSeHEZsyXPwRGi9jc=', 'Content-Length': '0'} 2023-10-18T20:45:50.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042ef78b5e3ff1037-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.783 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.784 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.784 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.784 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.785 DEBUG:boto:Method: GET 2023-10-18T20:45:50.785 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.786 DEBUG:boto:Data: 2023-10-18T20:45:50.786 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.787 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.787 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.787 DEBUG:boto:Params: {} 2023-10-18T20:45:50.788 DEBUG:boto:Token: None 2023-10-18T20:45:50.788 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.789 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8d639ba6d912fc-00653043fe-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.792 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.793 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.794 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.794 DEBUG:boto:path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.794 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.795 DEBUG:boto:Method: GET 2023-10-18T20:45:50.795 DEBUG:boto:Path: /bofrnf-59/?prefix=obj76 2023-10-18T20:45:50.796 DEBUG:boto:Data: 2023-10-18T20:45:50.796 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.797 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.797 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.798 DEBUG:boto:Params: {} 2023-10-18T20:45:50.798 DEBUG:boto:Token: None 2023-10-18T20:45:50.799 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb3b099b74feba66-00653043fe-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.805 DEBUG:boto:b'bofrnf-59obj761000falseobj762023-10-18T20:34:23.470Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:45:50.806 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.806 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.807 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.807 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.807 DEBUG:boto:Method: GET 2023-10-18T20:45:50.808 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.808 DEBUG:boto:Data: 2023-10-18T20:45:50.809 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.809 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.810 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.810 DEBUG:boto:Params: {} 2023-10-18T20:45:50.811 DEBUG:boto:Token: None 2023-10-18T20:45:50.811 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.812 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kFwHQdr/9LTs5lmJv64dO8PEYMQ= 2023-10-18T20:45:50.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kFwHQdr/9LTs5lmJv64dO8PEYMQ=', 'Content-Length': '0'} 2023-10-18T20:45:50.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026b04f176774670-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.815 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.816 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.816 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.817 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.817 DEBUG:boto:Method: GET 2023-10-18T20:45:50.818 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.818 DEBUG:boto:Data: 2023-10-18T20:45:50.819 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.819 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.819 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.820 DEBUG:boto:Params: {} 2023-10-18T20:45:50.820 DEBUG:boto:Token: None 2023-10-18T20:45:50.821 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.821 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kFwHQdr/9LTs5lmJv64dO8PEYMQ= 2023-10-18T20:45:50.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kFwHQdr/9LTs5lmJv64dO8PEYMQ=', 'Content-Length': '0'} 2023-10-18T20:45:50.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3ae839cddd2d5f-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.825 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.825 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.826 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.826 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.827 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.827 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.828 DEBUG:boto:Data: 2023-10-18T20:45:50.828 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.829 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.829 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.829 DEBUG:boto:Params: {} 2023-10-18T20:45:50.830 DEBUG:boto:Token: None 2023-10-18T20:45:50.830 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.831 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KXOWdaUuXvIB2NUS1dDT0avrYT0= 2023-10-18T20:45:50.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KXOWdaUuXvIB2NUS1dDT0avrYT0=', 'Content-Length': '0'} 2023-10-18T20:45:50.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a998e6e3c45be369-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.834 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.834 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.835 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.835 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.835 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.836 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.836 DEBUG:boto:Data: 2023-10-18T20:45:50.837 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.837 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.837 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.838 DEBUG:boto:Params: {} 2023-10-18T20:45:50.838 DEBUG:boto:Token: None 2023-10-18T20:45:50.839 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.839 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KXOWdaUuXvIB2NUS1dDT0avrYT0= 2023-10-18T20:45:50.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KXOWdaUuXvIB2NUS1dDT0avrYT0=', 'Content-Length': '0'} 2023-10-18T20:45:50.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5be75529f1a7cab-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:45:50.843 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.843 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.844 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.844 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.845 DEBUG:boto:Method: GET 2023-10-18T20:45:50.845 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.845 DEBUG:boto:Data: 2023-10-18T20:45:50.846 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.846 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.847 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.847 DEBUG:boto:Params: {} 2023-10-18T20:45:50.848 DEBUG:boto:Token: None 2023-10-18T20:45:50.848 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.849 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kFwHQdr/9LTs5lmJv64dO8PEYMQ= 2023-10-18T20:45:50.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kFwHQdr/9LTs5lmJv64dO8PEYMQ=', 'Content-Length': '0'} 2023-10-18T20:45:50.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ad2e270aad6ab6c-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.852 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.852 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.853 DEBUG:boto:path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.853 DEBUG:boto:auth_path=/bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.854 DEBUG:boto:Method: GET 2023-10-18T20:45:50.854 DEBUG:boto:Path: /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.854 DEBUG:boto:Data: 2023-10-18T20:45:50.855 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.856 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.856 DEBUG:boto:Params: {} 2023-10-18T20:45:50.857 DEBUG:boto:Token: None 2023-10-18T20:45:50.857 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85?versionId=null 2023-10-18T20:45:50.858 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kFwHQdr/9LTs5lmJv64dO8PEYMQ= 2023-10-18T20:45:50.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kFwHQdr/9LTs5lmJv64dO8PEYMQ=', 'Content-Length': '0'} 2023-10-18T20:45:50.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000059f70123672b4c-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.906 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.907 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.907 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:50.908 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:50.908 DEBUG:boto:Method: GET 2023-10-18T20:45:50.909 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:45:50.909 DEBUG:boto:Data: 2023-10-18T20:45:50.909 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.910 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.910 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.911 DEBUG:boto:Params: {} 2023-10-18T20:45:50.911 DEBUG:boto:Token: None 2023-10-18T20:45:50.912 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.912 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047266d5a3b11f997-00653043fe-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.916 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.917 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:50.917 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:50.917 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:50.918 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:50.918 DEBUG:boto:Method: GET 2023-10-18T20:45:50.919 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:45:50.919 DEBUG:boto:Data: 2023-10-18T20:45:50.919 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.920 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.920 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.920 DEBUG:boto:Params: {} 2023-10-18T20:45:50.921 DEBUG:boto:Token: None 2023-10-18T20:45:50.921 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/ 2023-10-18T20:45:50.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc= 2023-10-18T20:45:50.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nHalO661uTmk7tC81FKMMTdguJc=', 'Content-Length': '0'} 2023-10-18T20:45:50.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073b1053b06da72f7-00653043fe-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.927 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:50.928 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.928 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.929 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.929 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:45:50.929 DEBUG:boto:Data: 2023-10-18T20:45:50.930 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.930 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.931 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.931 DEBUG:boto:Params: {} 2023-10-18T20:45:50.931 DEBUG:boto:Token: None 2023-10-18T20:45:50.932 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85 2023-10-18T20:45:50.932 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GIpT+7uuliDa+IsXic+SXkWPAGQ= 2023-10-18T20:45:50.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GIpT+7uuliDa+IsXic+SXkWPAGQ=', 'Content-Length': '0'} 2023-10-18T20:45:50.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b39495f3f1b1f9ec-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.935 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.936 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.936 DEBUG:boto:Method: HEAD 2023-10-18T20:45:50.937 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:45:50.937 DEBUG:boto:Data: 2023-10-18T20:45:50.937 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.938 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.938 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.938 DEBUG:boto:Params: {} 2023-10-18T20:45:50.939 DEBUG:boto:Token: None 2023-10-18T20:45:50.939 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85 2023-10-18T20:45:50.940 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GIpT+7uuliDa+IsXic+SXkWPAGQ= 2023-10-18T20:45:50.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GIpT+7uuliDa+IsXic+SXkWPAGQ=', 'Content-Length': '0'} 2023-10-18T20:45:50.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006899e5fcfd2ea5ab-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-18T20:45:50.943 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.943 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.944 DEBUG:boto:Method: GET 2023-10-18T20:45:50.944 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:45:50.944 DEBUG:boto:Data: 2023-10-18T20:45:50.945 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:50.946 DEBUG:boto:Port: 8001 2023-10-18T20:45:50.946 DEBUG:boto:Params: {} 2023-10-18T20:45:50.947 DEBUG:boto:Token: None 2023-10-18T20:45:50.947 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85 2023-10-18T20:45:50.947 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TqZ8HvhHrdZ+nm5UT5mF2p80FF4= 2023-10-18T20:45:50.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TqZ8HvhHrdZ+nm5UT5mF2p80FF4=', 'Content-Length': '0'} 2023-10-18T20:45:50.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330cba61450e7c58-00653043fe-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:50.994 DEBUG:boto:path=/bofrnf-59/obj85 2023-10-18T20:45:50.995 DEBUG:boto:auth_path=/bofrnf-59/obj85 2023-10-18T20:45:50.995 DEBUG:boto:Method: GET 2023-10-18T20:45:50.996 DEBUG:boto:Path: /bofrnf-59/obj85 2023-10-18T20:45:50.996 DEBUG:boto:Data: 2023-10-18T20:45:50.997 DEBUG:boto:Headers: {} 2023-10-18T20:45:50.997 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:50.997 DEBUG:boto:Port: 8000 2023-10-18T20:45:50.998 DEBUG:boto:Params: {} 2023-10-18T20:45:50.998 DEBUG:boto:Token: None 2023-10-18T20:45:50.999 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:50 GMT /bofrnf-59/obj85 2023-10-18T20:45:50.999 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TqZ8HvhHrdZ+nm5UT5mF2p80FF4= 2023-10-18T20:45:51.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TqZ8HvhHrdZ+nm5UT5mF2p80FF4=', 'Content-Length': '0'} 2023-10-18T20:45:51.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:33:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000607d030d5cb433f8-00653043fe-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.003 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.003 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.003 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:51.004 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:51.004 DEBUG:boto:Method: GET 2023-10-18T20:45:51.005 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:45:51.005 DEBUG:boto:Data: 2023-10-18T20:45:51.005 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.006 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.006 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.007 DEBUG:boto:Params: {} 2023-10-18T20:45:51.007 DEBUG:boto:Token: None 2023-10-18T20:45:51.008 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.008 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cedd2542f601730b-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.012 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.012 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.013 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.013 DEBUG:boto:path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:51.014 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj85 2023-10-18T20:45:51.014 DEBUG:boto:Method: GET 2023-10-18T20:45:51.015 DEBUG:boto:Path: /bofrnf-59/?prefix=obj85 2023-10-18T20:45:51.015 DEBUG:boto:Data: 2023-10-18T20:45:51.015 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.016 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.016 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.017 DEBUG:boto:Params: {} 2023-10-18T20:45:51.017 DEBUG:boto:Token: None 2023-10-18T20:45:51.017 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.018 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000216c8b71b4bc08e3-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.023 DEBUG:boto:b'bofrnf-59obj851000falseobj852023-10-18T20:33:05.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:45:51.024 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.025 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.025 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.025 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.026 DEBUG:boto:Method: GET 2023-10-18T20:45:51.026 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.027 DEBUG:boto:Data: 2023-10-18T20:45:51.027 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.027 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.028 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.028 DEBUG:boto:Params: {} 2023-10-18T20:45:51.029 DEBUG:boto:Token: None 2023-10-18T20:45:51.029 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.030 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+KaNjFCKAerGnlAPljKEt/90MTI= 2023-10-18T20:45:51.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+KaNjFCKAerGnlAPljKEt/90MTI=', 'Content-Length': '0'} 2023-10-18T20:45:51.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acd88c4bc665eece-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.033 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.034 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.034 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.035 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.035 DEBUG:boto:Method: GET 2023-10-18T20:45:51.036 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.036 DEBUG:boto:Data: 2023-10-18T20:45:51.036 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.037 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.037 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.038 DEBUG:boto:Params: {} 2023-10-18T20:45:51.038 DEBUG:boto:Token: None 2023-10-18T20:45:51.039 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.039 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+KaNjFCKAerGnlAPljKEt/90MTI= 2023-10-18T20:45:51.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+KaNjFCKAerGnlAPljKEt/90MTI=', 'Content-Length': '0'} 2023-10-18T20:45:51.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccfa779e6ba0120f-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.086 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.087 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.087 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.088 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.088 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.089 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.089 DEBUG:boto:Data: 2023-10-18T20:45:51.089 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.090 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.090 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.091 DEBUG:boto:Params: {} 2023-10-18T20:45:51.091 DEBUG:boto:Token: None 2023-10-18T20:45:51.092 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.092 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+H9EdrCGt880PWZBC0RJsd9T8k= 2023-10-18T20:45:51.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+H9EdrCGt880PWZBC0RJsd9T8k=', 'Content-Length': '0'} 2023-10-18T20:45:51.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c3cc1d4e0c1bdd6-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.095 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.096 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.096 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.096 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.097 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.097 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.098 DEBUG:boto:Data: 2023-10-18T20:45:51.098 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.098 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.099 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.099 DEBUG:boto:Params: {} 2023-10-18T20:45:51.100 DEBUG:boto:Token: None 2023-10-18T20:45:51.100 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.101 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b+H9EdrCGt880PWZBC0RJsd9T8k= 2023-10-18T20:45:51.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b+H9EdrCGt880PWZBC0RJsd9T8k=', 'Content-Length': '0'} 2023-10-18T20:45:51.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ca27dba8d4f787d-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:45:51.104 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.105 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.105 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.105 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.106 DEBUG:boto:Method: GET 2023-10-18T20:45:51.106 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.107 DEBUG:boto:Data: 2023-10-18T20:45:51.107 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.108 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.108 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.108 DEBUG:boto:Params: {} 2023-10-18T20:45:51.109 DEBUG:boto:Token: None 2023-10-18T20:45:51.109 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.110 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+KaNjFCKAerGnlAPljKEt/90MTI= 2023-10-18T20:45:51.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+KaNjFCKAerGnlAPljKEt/90MTI=', 'Content-Length': '0'} 2023-10-18T20:45:51.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c822c57a251de10a-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.154 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.155 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.155 DEBUG:boto:path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.156 DEBUG:boto:auth_path=/bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.156 DEBUG:boto:Method: GET 2023-10-18T20:45:51.157 DEBUG:boto:Path: /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.157 DEBUG:boto:Data: 2023-10-18T20:45:51.158 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.158 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.159 DEBUG:boto:Params: {} 2023-10-18T20:45:51.159 DEBUG:boto:Token: None 2023-10-18T20:45:51.160 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87?versionId=null 2023-10-18T20:45:51.160 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+KaNjFCKAerGnlAPljKEt/90MTI= 2023-10-18T20:45:51.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+KaNjFCKAerGnlAPljKEt/90MTI=', 'Content-Length': '0'} 2023-10-18T20:45:51.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e563c4e15c99c977-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.164 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.164 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.165 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.165 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.165 DEBUG:boto:Method: GET 2023-10-18T20:45:51.166 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.166 DEBUG:boto:Data: 2023-10-18T20:45:51.167 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.167 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.168 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.168 DEBUG:boto:Params: {} 2023-10-18T20:45:51.168 DEBUG:boto:Token: None 2023-10-18T20:45:51.169 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.169 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000868f5cfec20972a1-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.173 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.174 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.174 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.174 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.175 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.175 DEBUG:boto:Method: GET 2023-10-18T20:45:51.176 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.176 DEBUG:boto:Data: 2023-10-18T20:45:51.177 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.177 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.177 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.178 DEBUG:boto:Params: {} 2023-10-18T20:45:51.178 DEBUG:boto:Token: None 2023-10-18T20:45:51.179 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.179 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073f6e19bdad1ca49-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.185 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.185 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.186 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.186 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.187 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:45:51.187 DEBUG:boto:Data: 2023-10-18T20:45:51.187 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.188 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.188 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.189 DEBUG:boto:Params: {} 2023-10-18T20:45:51.189 DEBUG:boto:Token: None 2023-10-18T20:45:51.190 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87 2023-10-18T20:45:51.190 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/l41GhEmdl5c4l3ltKPLcnylvHk= 2023-10-18T20:45:51.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/l41GhEmdl5c4l3ltKPLcnylvHk=', 'Content-Length': '0'} 2023-10-18T20:45:51.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012a3ad0424d963ec-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.193 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.194 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.194 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.194 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:45:51.195 DEBUG:boto:Data: 2023-10-18T20:45:51.195 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.196 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.196 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.196 DEBUG:boto:Params: {} 2023-10-18T20:45:51.197 DEBUG:boto:Token: None 2023-10-18T20:45:51.197 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87 2023-10-18T20:45:51.198 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/l41GhEmdl5c4l3ltKPLcnylvHk= 2023-10-18T20:45:51.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/l41GhEmdl5c4l3ltKPLcnylvHk=', 'Content-Length': '0'} 2023-10-18T20:45:51.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e294bf989d0e2621-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-18T20:45:51.201 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.202 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.202 DEBUG:boto:Method: GET 2023-10-18T20:45:51.202 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:45:51.203 DEBUG:boto:Data: 2023-10-18T20:45:51.203 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.204 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.204 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.205 DEBUG:boto:Params: {} 2023-10-18T20:45:51.205 DEBUG:boto:Token: None 2023-10-18T20:45:51.206 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87 2023-10-18T20:45:51.206 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SPHIv6KqI5GzEPl8XWFMFFxNRFY= 2023-10-18T20:45:51.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SPHIv6KqI5GzEPl8XWFMFFxNRFY=', 'Content-Length': '0'} 2023-10-18T20:45:51.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000378b9a20fe3a2ccf-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.209 DEBUG:boto:path=/bofrnf-59/obj87 2023-10-18T20:45:51.210 DEBUG:boto:auth_path=/bofrnf-59/obj87 2023-10-18T20:45:51.210 DEBUG:boto:Method: GET 2023-10-18T20:45:51.211 DEBUG:boto:Path: /bofrnf-59/obj87 2023-10-18T20:45:51.211 DEBUG:boto:Data: 2023-10-18T20:45:51.211 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.212 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.212 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.213 DEBUG:boto:Params: {} 2023-10-18T20:45:51.213 DEBUG:boto:Token: None 2023-10-18T20:45:51.214 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj87 2023-10-18T20:45:51.214 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SPHIv6KqI5GzEPl8XWFMFFxNRFY= 2023-10-18T20:45:51.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SPHIv6KqI5GzEPl8XWFMFFxNRFY=', 'Content-Length': '0'} 2023-10-18T20:45:51.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:39:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f2ca09f392e6b9-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.258 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.259 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.260 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.260 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.260 DEBUG:boto:Method: GET 2023-10-18T20:45:51.261 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.261 DEBUG:boto:Data: 2023-10-18T20:45:51.262 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.262 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.263 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.263 DEBUG:boto:Params: {} 2023-10-18T20:45:51.264 DEBUG:boto:Token: None 2023-10-18T20:45:51.264 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.265 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ef87f94fd1e7113-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.269 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.270 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.270 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.271 DEBUG:boto:path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.271 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.271 DEBUG:boto:Method: GET 2023-10-18T20:45:51.272 DEBUG:boto:Path: /bofrnf-59/?prefix=obj87 2023-10-18T20:45:51.272 DEBUG:boto:Data: 2023-10-18T20:45:51.273 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.273 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.274 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.274 DEBUG:boto:Params: {} 2023-10-18T20:45:51.275 DEBUG:boto:Token: None 2023-10-18T20:45:51.275 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005511b7dfa837e2f5-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.281 DEBUG:boto:b'bofrnf-59obj871000falseobj872023-10-18T20:39:46.932Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:45:51.283 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.283 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.284 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.284 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.285 DEBUG:boto:Method: GET 2023-10-18T20:45:51.285 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.286 DEBUG:boto:Data: 2023-10-18T20:45:51.286 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.286 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.287 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.287 DEBUG:boto:Params: {} 2023-10-18T20:45:51.288 DEBUG:boto:Token: None 2023-10-18T20:45:51.288 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.289 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PWsINbbjXACp1YxJYMdCLEdf9yU= 2023-10-18T20:45:51.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PWsINbbjXACp1YxJYMdCLEdf9yU=', 'Content-Length': '0'} 2023-10-18T20:45:51.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fb5083a0d8c1ba6-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.292 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.293 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.293 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.294 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.294 DEBUG:boto:Method: GET 2023-10-18T20:45:51.295 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.295 DEBUG:boto:Data: 2023-10-18T20:45:51.295 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.296 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.296 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.297 DEBUG:boto:Params: {} 2023-10-18T20:45:51.297 DEBUG:boto:Token: None 2023-10-18T20:45:51.298 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.298 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PWsINbbjXACp1YxJYMdCLEdf9yU= 2023-10-18T20:45:51.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PWsINbbjXACp1YxJYMdCLEdf9yU=', 'Content-Length': '0'} 2023-10-18T20:45:51.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437f8ffa498dd818-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.302 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.302 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.303 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.303 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.303 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.304 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.304 DEBUG:boto:Data: 2023-10-18T20:45:51.305 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.305 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.305 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.306 DEBUG:boto:Params: {} 2023-10-18T20:45:51.307 DEBUG:boto:Token: None 2023-10-18T20:45:51.307 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.308 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E8y921tXPZL7xW+kSjlz/7pS4PQ= 2023-10-18T20:45:51.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E8y921tXPZL7xW+kSjlz/7pS4PQ=', 'Content-Length': '0'} 2023-10-18T20:45:51.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008435b58eb370b9d8-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.311 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.311 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.312 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.312 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.313 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.313 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.313 DEBUG:boto:Data: 2023-10-18T20:45:51.314 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.315 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.315 DEBUG:boto:Params: {} 2023-10-18T20:45:51.315 DEBUG:boto:Token: None 2023-10-18T20:45:51.316 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.316 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:E8y921tXPZL7xW+kSjlz/7pS4PQ= 2023-10-18T20:45:51.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:E8y921tXPZL7xW+kSjlz/7pS4PQ=', 'Content-Length': '0'} 2023-10-18T20:45:51.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31e2e07d84a7d4b-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:45:51.320 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.320 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.321 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.321 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.322 DEBUG:boto:Method: GET 2023-10-18T20:45:51.322 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.323 DEBUG:boto:Data: 2023-10-18T20:45:51.323 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.323 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.324 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.324 DEBUG:boto:Params: {} 2023-10-18T20:45:51.325 DEBUG:boto:Token: None 2023-10-18T20:45:51.325 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.326 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PWsINbbjXACp1YxJYMdCLEdf9yU= 2023-10-18T20:45:51.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PWsINbbjXACp1YxJYMdCLEdf9yU=', 'Content-Length': '0'} 2023-10-18T20:45:51.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4f981e95d9d855a-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.374 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.375 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.376 DEBUG:boto:path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.377 DEBUG:boto:auth_path=/bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.378 DEBUG:boto:Method: GET 2023-10-18T20:45:51.378 DEBUG:boto:Path: /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.379 DEBUG:boto:Data: 2023-10-18T20:45:51.380 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.381 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.382 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.383 DEBUG:boto:Params: {} 2023-10-18T20:45:51.384 DEBUG:boto:Token: None 2023-10-18T20:45:51.384 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92?versionId=null 2023-10-18T20:45:51.385 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PWsINbbjXACp1YxJYMdCLEdf9yU= 2023-10-18T20:45:51.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PWsINbbjXACp1YxJYMdCLEdf9yU=', 'Content-Length': '0'} 2023-10-18T20:45:51.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e099a39f0b27617a-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.389 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.390 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.390 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.390 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.391 DEBUG:boto:Method: GET 2023-10-18T20:45:51.391 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.392 DEBUG:boto:Data: 2023-10-18T20:45:51.392 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.393 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.393 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.393 DEBUG:boto:Params: {} 2023-10-18T20:45:51.394 DEBUG:boto:Token: None 2023-10-18T20:45:51.395 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.395 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007368f989940b3cde-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.399 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.400 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.401 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.402 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.402 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.402 DEBUG:boto:Method: GET 2023-10-18T20:45:51.403 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.403 DEBUG:boto:Data: 2023-10-18T20:45:51.404 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.404 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.405 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.405 DEBUG:boto:Params: {} 2023-10-18T20:45:51.405 DEBUG:boto:Token: None 2023-10-18T20:45:51.406 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.406 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a9232987e280d14-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.412 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.413 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.413 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.413 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.414 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:45:51.414 DEBUG:boto:Data: 2023-10-18T20:45:51.415 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.416 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.416 DEBUG:boto:Params: {} 2023-10-18T20:45:51.417 DEBUG:boto:Token: None 2023-10-18T20:45:51.418 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92 2023-10-18T20:45:51.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W3vXnzr53jawU4JZeK1NYPzG7+4= 2023-10-18T20:45:51.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W3vXnzr53jawU4JZeK1NYPzG7+4=', 'Content-Length': '0'} 2023-10-18T20:45:51.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a5cb8d931652879-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.424 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.424 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.425 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.426 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:45:51.426 DEBUG:boto:Data: 2023-10-18T20:45:51.427 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.427 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.427 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.428 DEBUG:boto:Params: {} 2023-10-18T20:45:51.428 DEBUG:boto:Token: None 2023-10-18T20:45:51.429 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92 2023-10-18T20:45:51.430 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W3vXnzr53jawU4JZeK1NYPzG7+4= 2023-10-18T20:45:51.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W3vXnzr53jawU4JZeK1NYPzG7+4=', 'Content-Length': '0'} 2023-10-18T20:45:51.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9ee7eb0a9c8b29b-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-18T20:45:51.435 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.436 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.437 DEBUG:boto:Method: GET 2023-10-18T20:45:51.437 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:45:51.438 DEBUG:boto:Data: 2023-10-18T20:45:51.439 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.439 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.440 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.441 DEBUG:boto:Params: {} 2023-10-18T20:45:51.441 DEBUG:boto:Token: None 2023-10-18T20:45:51.442 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92 2023-10-18T20:45:51.443 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G/FgbHo8YbR/vweCAbBfpAVQM5o= 2023-10-18T20:45:51.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G/FgbHo8YbR/vweCAbBfpAVQM5o=', 'Content-Length': '0'} 2023-10-18T20:45:51.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001edd552c2e90c5ac-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.446 DEBUG:boto:path=/bofrnf-59/obj92 2023-10-18T20:45:51.448 DEBUG:boto:auth_path=/bofrnf-59/obj92 2023-10-18T20:45:51.449 DEBUG:boto:Method: GET 2023-10-18T20:45:51.450 DEBUG:boto:Path: /bofrnf-59/obj92 2023-10-18T20:45:51.451 DEBUG:boto:Data: 2023-10-18T20:45:51.452 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.453 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.454 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.455 DEBUG:boto:Params: {} 2023-10-18T20:45:51.456 DEBUG:boto:Token: None 2023-10-18T20:45:51.456 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj92 2023-10-18T20:45:51.457 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:G/FgbHo8YbR/vweCAbBfpAVQM5o= 2023-10-18T20:45:51.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:G/FgbHo8YbR/vweCAbBfpAVQM5o=', 'Content-Length': '0'} 2023-10-18T20:45:51.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ffb24948278d33-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.506 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.507 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.507 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.508 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.508 DEBUG:boto:Method: GET 2023-10-18T20:45:51.509 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.509 DEBUG:boto:Data: 2023-10-18T20:45:51.510 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.510 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.510 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.511 DEBUG:boto:Params: {} 2023-10-18T20:45:51.511 DEBUG:boto:Token: None 2023-10-18T20:45:51.512 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.513 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066457d929f59f4fe-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.517 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.518 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.519 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.519 DEBUG:boto:path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.520 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.520 DEBUG:boto:Method: GET 2023-10-18T20:45:51.520 DEBUG:boto:Path: /bofrnf-59/?prefix=obj92 2023-10-18T20:45:51.521 DEBUG:boto:Data: 2023-10-18T20:45:51.521 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.522 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.522 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.523 DEBUG:boto:Params: {} 2023-10-18T20:45:51.524 DEBUG:boto:Token: None 2023-10-18T20:45:51.525 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.526 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8cd5435dd8e58c-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.533 DEBUG:boto:b'bofrnf-59obj921000falseobj922023-10-18T20:38:22.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-18T20:45:51.534 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.535 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.536 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.536 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.537 DEBUG:boto:Method: GET 2023-10-18T20:45:51.537 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.538 DEBUG:boto:Data: 2023-10-18T20:45:51.539 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.540 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.542 DEBUG:boto:Params: {} 2023-10-18T20:45:51.543 DEBUG:boto:Token: None 2023-10-18T20:45:51.544 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y7e9yBOT23CZdeGCRbf/JByFvgk= 2023-10-18T20:45:51.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y7e9yBOT23CZdeGCRbf/JByFvgk=', 'Content-Length': '0'} 2023-10-18T20:45:51.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a0ecbf0fadf945-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.549 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.549 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.550 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.550 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.551 DEBUG:boto:Method: GET 2023-10-18T20:45:51.551 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.552 DEBUG:boto:Data: 2023-10-18T20:45:51.552 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.553 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.553 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.554 DEBUG:boto:Params: {} 2023-10-18T20:45:51.555 DEBUG:boto:Token: None 2023-10-18T20:45:51.555 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.556 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y7e9yBOT23CZdeGCRbf/JByFvgk= 2023-10-18T20:45:51.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y7e9yBOT23CZdeGCRbf/JByFvgk=', 'Content-Length': '0'} 2023-10-18T20:45:51.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b626129c228f1e1-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.560 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.561 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.561 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.562 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.563 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.563 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.564 DEBUG:boto:Data: 2023-10-18T20:45:51.564 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.565 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.565 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.566 DEBUG:boto:Params: {} 2023-10-18T20:45:51.566 DEBUG:boto:Token: None 2023-10-18T20:45:51.567 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uMvfBc/jhJqnYkLDQ0SDUr3oMq4= 2023-10-18T20:45:51.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uMvfBc/jhJqnYkLDQ0SDUr3oMq4=', 'Content-Length': '0'} 2023-10-18T20:45:51.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065388db932e0528b-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.571 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.571 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.572 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.572 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.573 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.573 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.573 DEBUG:boto:Data: 2023-10-18T20:45:51.574 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.574 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.575 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.575 DEBUG:boto:Params: {} 2023-10-18T20:45:51.576 DEBUG:boto:Token: None 2023-10-18T20:45:51.576 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.577 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uMvfBc/jhJqnYkLDQ0SDUr3oMq4= 2023-10-18T20:45:51.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uMvfBc/jhJqnYkLDQ0SDUr3oMq4=', 'Content-Length': '0'} 2023-10-18T20:45:51.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004079c8c1f4f05529-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-18T20:45:51.582 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.582 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.583 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.584 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.585 DEBUG:boto:Method: GET 2023-10-18T20:45:51.585 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.586 DEBUG:boto:Data: 2023-10-18T20:45:51.587 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.588 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.588 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.589 DEBUG:boto:Params: {} 2023-10-18T20:45:51.589 DEBUG:boto:Token: None 2023-10-18T20:45:51.590 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.590 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y7e9yBOT23CZdeGCRbf/JByFvgk= 2023-10-18T20:45:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y7e9yBOT23CZdeGCRbf/JByFvgk=', 'Content-Length': '0'} 2023-10-18T20:45:51.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf17f291be42095-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.634 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.637 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.637 DEBUG:boto:path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.638 DEBUG:boto:auth_path=/bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.638 DEBUG:boto:Method: GET 2023-10-18T20:45:51.639 DEBUG:boto:Path: /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.639 DEBUG:boto:Data: 2023-10-18T20:45:51.640 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.640 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.640 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.641 DEBUG:boto:Params: {} 2023-10-18T20:45:51.641 DEBUG:boto:Token: None 2023-10-18T20:45:51.642 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93?versionId=null 2023-10-18T20:45:51.642 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y7e9yBOT23CZdeGCRbf/JByFvgk= 2023-10-18T20:45:51.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y7e9yBOT23CZdeGCRbf/JByFvgk=', 'Content-Length': '0'} 2023-10-18T20:45:51.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ccf6b8317fad4b8-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.646 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.646 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.647 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.647 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.648 DEBUG:boto:Method: GET 2023-10-18T20:45:51.648 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.648 DEBUG:boto:Data: 2023-10-18T20:45:51.649 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.650 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.650 DEBUG:boto:Params: {} 2023-10-18T20:45:51.651 DEBUG:boto:Token: None 2023-10-18T20:45:51.651 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab1f3ed1743019c6-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.655 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.656 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.657 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.657 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.657 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.658 DEBUG:boto:Method: GET 2023-10-18T20:45:51.658 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.659 DEBUG:boto:Data: 2023-10-18T20:45:51.659 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.659 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.660 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.660 DEBUG:boto:Params: {} 2023-10-18T20:45:51.661 DEBUG:boto:Token: None 2023-10-18T20:45:51.661 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.662 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126e02c51f37f07f-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.667 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.668 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.668 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.669 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.669 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:45:51.670 DEBUG:boto:Data: 2023-10-18T20:45:51.670 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.671 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.672 DEBUG:boto:Params: {} 2023-10-18T20:45:51.672 DEBUG:boto:Token: None 2023-10-18T20:45:51.672 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93 2023-10-18T20:45:51.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UaSIfRCtH0GHqDkPTKoQQcoZRw0= 2023-10-18T20:45:51.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UaSIfRCtH0GHqDkPTKoQQcoZRw0=', 'Content-Length': '0'} 2023-10-18T20:45:51.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db76b41d2e91d64-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.676 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.676 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.677 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.677 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:45:51.678 DEBUG:boto:Data: 2023-10-18T20:45:51.678 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.679 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.679 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.679 DEBUG:boto:Params: {} 2023-10-18T20:45:51.680 DEBUG:boto:Token: None 2023-10-18T20:45:51.680 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93 2023-10-18T20:45:51.681 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UaSIfRCtH0GHqDkPTKoQQcoZRw0= 2023-10-18T20:45:51.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UaSIfRCtH0GHqDkPTKoQQcoZRw0=', 'Content-Length': '0'} 2023-10-18T20:45:51.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1223202b37d85b2-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-18T20:45:51.685 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.685 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.686 DEBUG:boto:Method: GET 2023-10-18T20:45:51.686 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:45:51.687 DEBUG:boto:Data: 2023-10-18T20:45:51.687 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.687 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.688 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.688 DEBUG:boto:Params: {} 2023-10-18T20:45:51.689 DEBUG:boto:Token: None 2023-10-18T20:45:51.689 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93 2023-10-18T20:45:51.690 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BA1LTUuj4AjBxp5fob7XSAmDZUU= 2023-10-18T20:45:51.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BA1LTUuj4AjBxp5fob7XSAmDZUU=', 'Content-Length': '0'} 2023-10-18T20:45:51.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007374d0d9a6a56a0a-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.693 DEBUG:boto:path=/bofrnf-59/obj93 2023-10-18T20:45:51.693 DEBUG:boto:auth_path=/bofrnf-59/obj93 2023-10-18T20:45:51.694 DEBUG:boto:Method: GET 2023-10-18T20:45:51.695 DEBUG:boto:Path: /bofrnf-59/obj93 2023-10-18T20:45:51.696 DEBUG:boto:Data: 2023-10-18T20:45:51.697 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.697 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.698 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.699 DEBUG:boto:Params: {} 2023-10-18T20:45:51.700 DEBUG:boto:Token: None 2023-10-18T20:45:51.701 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj93 2023-10-18T20:45:51.702 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BA1LTUuj4AjBxp5fob7XSAmDZUU= 2023-10-18T20:45:51.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BA1LTUuj4AjBxp5fob7XSAmDZUU=', 'Content-Length': '0'} 2023-10-18T20:45:51.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b30ece7057b530-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.707 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.708 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.708 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.709 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.710 DEBUG:boto:Method: GET 2023-10-18T20:45:51.710 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.711 DEBUG:boto:Data: 2023-10-18T20:45:51.712 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.713 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.714 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.714 DEBUG:boto:Params: {} 2023-10-18T20:45:51.715 DEBUG:boto:Token: None 2023-10-18T20:45:51.716 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009607fd8af50fd09d-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.722 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.723 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.724 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.725 DEBUG:boto:path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.726 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.727 DEBUG:boto:Method: GET 2023-10-18T20:45:51.727 DEBUG:boto:Path: /bofrnf-59/?prefix=obj93 2023-10-18T20:45:51.728 DEBUG:boto:Data: 2023-10-18T20:45:51.729 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.730 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.731 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.731 DEBUG:boto:Params: {} 2023-10-18T20:45:51.732 DEBUG:boto:Token: None 2023-10-18T20:45:51.733 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.734 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a564abb4531f68d-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.740 DEBUG:boto:b'bofrnf-59obj931000falseobj932023-10-18T20:41:07.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:45:51.741 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.741 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.742 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.742 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.743 DEBUG:boto:Method: GET 2023-10-18T20:45:51.743 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.743 DEBUG:boto:Data: 2023-10-18T20:45:51.744 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.744 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.745 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.745 DEBUG:boto:Params: {} 2023-10-18T20:45:51.746 DEBUG:boto:Token: None 2023-10-18T20:45:51.746 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.747 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4sWQaJUOxJasR8unhnVP+p5/Vi8= 2023-10-18T20:45:51.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4sWQaJUOxJasR8unhnVP+p5/Vi8=', 'Content-Length': '0'} 2023-10-18T20:45:51.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073a9d507d71c6b32-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.750 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.751 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.751 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.752 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.752 DEBUG:boto:Method: GET 2023-10-18T20:45:51.753 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.753 DEBUG:boto:Data: 2023-10-18T20:45:51.753 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.754 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.755 DEBUG:boto:Params: {} 2023-10-18T20:45:51.755 DEBUG:boto:Token: None 2023-10-18T20:45:51.756 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4sWQaJUOxJasR8unhnVP+p5/Vi8= 2023-10-18T20:45:51.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4sWQaJUOxJasR8unhnVP+p5/Vi8=', 'Content-Length': '0'} 2023-10-18T20:45:51.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df1acff7364795f-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.759 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.760 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.760 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.761 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.761 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.761 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.762 DEBUG:boto:Data: 2023-10-18T20:45:51.762 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.763 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.763 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.764 DEBUG:boto:Params: {} 2023-10-18T20:45:51.764 DEBUG:boto:Token: None 2023-10-18T20:45:51.765 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.765 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R2Y2wkYNxPrbPUojhRC8Fhbk04w= 2023-10-18T20:45:51.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R2Y2wkYNxPrbPUojhRC8Fhbk04w=', 'Content-Length': '0'} 2023-10-18T20:45:51.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc0cce99896fa6a9-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.768 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.768 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.769 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.769 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.770 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.770 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.771 DEBUG:boto:Data: 2023-10-18T20:45:51.771 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.771 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.772 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.772 DEBUG:boto:Params: {} 2023-10-18T20:45:51.773 DEBUG:boto:Token: None 2023-10-18T20:45:51.773 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R2Y2wkYNxPrbPUojhRC8Fhbk04w= 2023-10-18T20:45:51.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R2Y2wkYNxPrbPUojhRC8Fhbk04w=', 'Content-Length': '0'} 2023-10-18T20:45:51.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004940efcae77b836a-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:45:51.777 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.778 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.778 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.779 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.779 DEBUG:boto:Method: GET 2023-10-18T20:45:51.779 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.780 DEBUG:boto:Data: 2023-10-18T20:45:51.780 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.781 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.781 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.782 DEBUG:boto:Params: {} 2023-10-18T20:45:51.782 DEBUG:boto:Token: None 2023-10-18T20:45:51.783 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.783 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4sWQaJUOxJasR8unhnVP+p5/Vi8= 2023-10-18T20:45:51.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4sWQaJUOxJasR8unhnVP+p5/Vi8=', 'Content-Length': '0'} 2023-10-18T20:45:51.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bd98b1f1f151ce3-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.787 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.787 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.787 DEBUG:boto:path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.788 DEBUG:boto:auth_path=/bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.788 DEBUG:boto:Method: GET 2023-10-18T20:45:51.789 DEBUG:boto:Path: /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.789 DEBUG:boto:Data: 2023-10-18T20:45:51.790 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.791 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.791 DEBUG:boto:Params: {} 2023-10-18T20:45:51.792 DEBUG:boto:Token: None 2023-10-18T20:45:51.793 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95?versionId=null 2023-10-18T20:45:51.794 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4sWQaJUOxJasR8unhnVP+p5/Vi8= 2023-10-18T20:45:51.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4sWQaJUOxJasR8unhnVP+p5/Vi8=', 'Content-Length': '0'} 2023-10-18T20:45:51.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b44d98ddd3c53bb-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.838 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.840 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.840 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.840 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.841 DEBUG:boto:Method: GET 2023-10-18T20:45:51.841 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.842 DEBUG:boto:Data: 2023-10-18T20:45:51.842 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.843 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.844 DEBUG:boto:Params: {} 2023-10-18T20:45:51.844 DEBUG:boto:Token: None 2023-10-18T20:45:51.845 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.845 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078e86ca6e2caa189-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.849 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.850 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.850 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.851 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.851 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.852 DEBUG:boto:Method: GET 2023-10-18T20:45:51.852 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.852 DEBUG:boto:Data: 2023-10-18T20:45:51.853 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.853 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.854 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.854 DEBUG:boto:Params: {} 2023-10-18T20:45:51.855 DEBUG:boto:Token: None 2023-10-18T20:45:51.855 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.856 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095977330bdbbc426-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.861 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.862 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.863 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.863 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.863 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:45:51.864 DEBUG:boto:Data: 2023-10-18T20:45:51.864 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.865 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.865 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.865 DEBUG:boto:Params: {} 2023-10-18T20:45:51.866 DEBUG:boto:Token: None 2023-10-18T20:45:51.866 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95 2023-10-18T20:45:51.867 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fOJJc6Pe5v728bZopdL02XtipO8= 2023-10-18T20:45:51.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fOJJc6Pe5v728bZopdL02XtipO8=', 'Content-Length': '0'} 2023-10-18T20:45:51.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7b09dad79c02642-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.870 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.870 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.871 DEBUG:boto:Method: HEAD 2023-10-18T20:45:51.871 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:45:51.872 DEBUG:boto:Data: 2023-10-18T20:45:51.872 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.872 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.873 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.873 DEBUG:boto:Params: {} 2023-10-18T20:45:51.874 DEBUG:boto:Token: None 2023-10-18T20:45:51.874 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95 2023-10-18T20:45:51.874 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fOJJc6Pe5v728bZopdL02XtipO8= 2023-10-18T20:45:51.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fOJJc6Pe5v728bZopdL02XtipO8=', 'Content-Length': '0'} 2023-10-18T20:45:51.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae5e0f6609146587-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-18T20:45:51.878 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.878 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.879 DEBUG:boto:Method: GET 2023-10-18T20:45:51.879 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:45:51.879 DEBUG:boto:Data: 2023-10-18T20:45:51.880 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.880 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.881 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.881 DEBUG:boto:Params: {} 2023-10-18T20:45:51.882 DEBUG:boto:Token: None 2023-10-18T20:45:51.882 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95 2023-10-18T20:45:51.883 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oyiZvxFop0pdq2t+SHfHrfRcqpc= 2023-10-18T20:45:51.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oyiZvxFop0pdq2t+SHfHrfRcqpc=', 'Content-Length': '0'} 2023-10-18T20:45:51.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-59:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.28'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e28799e95ef872d8-00653043ff-1d9c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.886 DEBUG:boto:path=/bofrnf-59/obj95 2023-10-18T20:45:51.886 DEBUG:boto:auth_path=/bofrnf-59/obj95 2023-10-18T20:45:51.887 DEBUG:boto:Method: GET 2023-10-18T20:45:51.887 DEBUG:boto:Path: /bofrnf-59/obj95 2023-10-18T20:45:51.887 DEBUG:boto:Data: 2023-10-18T20:45:51.888 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.888 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.889 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.889 DEBUG:boto:Params: {} 2023-10-18T20:45:51.890 DEBUG:boto:Token: None 2023-10-18T20:45:51.890 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/obj95 2023-10-18T20:45:51.891 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oyiZvxFop0pdq2t+SHfHrfRcqpc= 2023-10-18T20:45:51.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oyiZvxFop0pdq2t+SHfHrfRcqpc=', 'Content-Length': '0'} 2023-10-18T20:45:51.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:34:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f874d4a19aa3b7f-00653043ff-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.894 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.894 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.895 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.895 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.895 DEBUG:boto:Method: GET 2023-10-18T20:45:51.896 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.896 DEBUG:boto:Data: 2023-10-18T20:45:51.897 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.897 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:45:51.898 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.898 DEBUG:boto:Params: {} 2023-10-18T20:45:51.898 DEBUG:boto:Token: None 2023-10-18T20:45:51.899 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.899 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000254d7079e8f1592d-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.903 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.904 DEBUG:boto:path=/bofrnf-59/ 2023-10-18T20:45:51.904 DEBUG:boto:auth_path=/bofrnf-59/ 2023-10-18T20:45:51.905 DEBUG:boto:path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.905 DEBUG:boto:auth_path=/bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.906 DEBUG:boto:Method: GET 2023-10-18T20:45:51.906 DEBUG:boto:Path: /bofrnf-59/?prefix=obj95 2023-10-18T20:45:51.906 DEBUG:boto:Data: 2023-10-18T20:45:51.907 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.907 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:45:51.908 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.908 DEBUG:boto:Params: {} 2023-10-18T20:45:51.909 DEBUG:boto:Token: None 2023-10-18T20:45:51.909 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:45:51 GMT /bofrnf-59/ 2023-10-18T20:45:51.910 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM= 2023-10-18T20:45:51.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:45:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W2giri3QDd7QrReUVq47xtnHlCM=', 'Content-Length': '0'} 2023-10-18T20:45:51.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081f39577b5e7abe1-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.915 DEBUG:boto:b'bofrnf-59obj951000falseobj952023-10-18T20:34:23.530Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:45:51.915 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-59 zones={a2, a1} 2023-10-18T20:45:51.916 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2023-10-18T20:45:51.920 INFO:rgw_multi.tests:create role zone=a1 name=roles-bofrnf-1 2023-10-18T20:45:51.921 DEBUG:boto:Method: POST 2023-10-18T20:45:51.921 DEBUG:boto:Path: / 2023-10-18T20:45:51.922 DEBUG:boto:Data: 2023-10-18T20:45:51.922 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.923 DEBUG:boto:Host: smithi082.front.sepia.ceph.com 2023-10-18T20:45:51.923 DEBUG:boto:Port: 8000 2023-10-18T20:45:51.924 DEBUG:boto:Params: {'RoleName': 'roles-bofrnf-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-10-18T20:45:51.924 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:45:51.924 DEBUG:boto:Token: None 2023-10-18T20:45:51.925 DEBUG:boto:CanonicalRequest: POST / host:smithi082.front.sepia.ceph.com:8000 x-amz-date:20231018T204551Z host;x-amz-date 467eb7e82bb66a13905342456bf9abf7b916db46b71e7fa6febabcec85105e2c 2023-10-18T20:45:51.926 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231018T204551Z 20231018/front/smithi082/aws4_request 4ecdc5e96a8c37acde922ff936bf56cd9386153c26f05d4197d252103021128a 2023-10-18T20:45:51.926 DEBUG:boto:Signature: a583c6135682559519604bbdb44013aa14afcaa7c98665d664810454ba38ba0c 2023-10-18T20:45:51.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231018T204551Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=6TC2VRGERC8I036KRP19/20231018/front/smithi082/aws4_request,SignedHeaders=host;x-amz-date,Signature=a583c6135682559519604bbdb44013aa14afcaa7c98665d664810454ba38ba0c', 'Host': 'smithi082.front.sepia.ceph.com:8000'} 2023-10-18T20:45:51.939 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004604d507b669b915-00653043ff-1664-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.939 DEBUG:boto:b'3225a92f-d285-485b-abe0-da83a7e7ffc0roles-bofrnf-1/arn:aws:iam:::role/roles-bofrnf-12023-10-18T20:45:51.930Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000004604d507b669b915-00653043ff-1664-a1' 2023-10-18T20:45:51.940 INFO:rgw_multi.tests:create role zone=a2 name=roles-bofrnf-2 2023-10-18T20:45:51.941 DEBUG:boto:Method: POST 2023-10-18T20:45:51.941 DEBUG:boto:Path: / 2023-10-18T20:45:51.941 DEBUG:boto:Data: 2023-10-18T20:45:51.942 DEBUG:boto:Headers: {} 2023-10-18T20:45:51.942 DEBUG:boto:Host: smithi082.front.sepia.ceph.com 2023-10-18T20:45:51.943 DEBUG:boto:Port: 8001 2023-10-18T20:45:51.943 DEBUG:boto:Params: {'RoleName': 'roles-bofrnf-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-10-18T20:45:51.944 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:45:51.944 DEBUG:boto:Token: None 2023-10-18T20:45:51.945 DEBUG:boto:CanonicalRequest: POST / host:smithi082.front.sepia.ceph.com:8001 x-amz-date:20231018T204551Z host;x-amz-date 845c4de935a9fe6a4637c99b73990d16660e67881226c028ed1e86fd4742133b 2023-10-18T20:45:51.945 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231018T204551Z 20231018/front/smithi082/aws4_request cd5707523a7136ad439f834cbf1d457e10c4039698c6aaf72ce7927bc679de8a 2023-10-18T20:45:51.945 DEBUG:boto:Signature: 17305cf78f32098784035a4aa4f7938950cf0a763f1232a01914f713f5fb79ec 2023-10-18T20:45:51.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231018T204551Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=6TC2VRGERC8I036KRP19/20231018/front/smithi082/aws4_request,SignedHeaders=host;x-amz-date,Signature=17305cf78f32098784035a4aa4f7938950cf0a763f1232a01914f713f5fb79ec', 'Host': 'smithi082.front.sepia.ceph.com:8001'} 2023-10-18T20:45:51.964 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000030d37c83d5da3f89-00653043ff-1d9c-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Wed, 18 Oct 2023 20:45:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:45:51.965 DEBUG:boto:b'a3082250-309c-4008-9016-250a7c2d081broles-bofrnf-2/arn:aws:iam:::role/roles-bofrnf-22023-10-18T20:45:51.959Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx0000030d37c83d5da3f89-00653043ff-1d9c-a2' 2023-10-18T20:45:51.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:51.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:51.967 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:45:52.021 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:52.021 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:52.053 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:45:52.066 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:45:52.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:52.066 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:45:52.067 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:43:09.583751Z" 2023-10-18T20:45:52.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:52.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:52.067 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:45:52.067 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T20:45:52.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:52.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:52.068 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:45:52.068 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:45:52.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:52.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:52.069 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:45:52.069 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T20:45:52.069 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:52.070 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:45:52.071 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:45:52.071 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:45:52.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:52.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:52.073 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:45:52.123 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:52.123 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:52.151 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:52.149+0000 7f0c1b146780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:45:52.152 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:52.150+0000 7f0c1b146780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:45:52.165 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:45:52.166 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:45:52.166 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:45:52.166 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:45:52.166 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:45:52.167 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:45:52.167 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:52.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:52.167 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:45:52.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:52.167 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:45:52.168 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:45:52.168 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:45:52.168 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:45:52.168 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:45:52.168 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:45:52.169 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:45:52.169 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:45:52.169 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:52.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:52.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:52.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:52.170 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:45:52.170 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:45:52.170 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:45:52.170 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:45:52.171 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:45:52.171 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:45:52.171 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:45:52.171 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:45:52.171 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:52.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:52.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:52.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:52.172 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:45:52.172 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:45:52.172 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:45:52.173 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:45:52.173 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:45:52.173 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:45:52.173 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:45:52.173 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:45:52.174 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:52.174 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:52.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:52.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:52.174 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:45:52.175 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:45:52.175 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:45:52.175 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:45:52.175 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:45:52.175 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:45:52.176 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:45:52.176 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:45:52.176 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:52.176 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:52.176 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:52.177 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:45:52.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:52.177 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:45:52.177 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:45:52.177 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:45:52.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:52.178 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:45:52Z" 2023-10-18T20:45:52.178 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:45:52.179 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:45:52Z" } 2023-10-18T20:45:52.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:45:52.180 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:45:52.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:45:52.181 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697661951.933926_367.1 target=1_1697658841.435004_114.1, shard=3 master=1_1697661951.953788_570.1 target= 2023-10-18T20:45:57.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:57.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:45:57.183 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:45:57.240 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:45:57.240 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:45:57.273 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:57.270+0000 7f98ee89b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:45:57.273 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:45:57.271+0000 7f98ee89b780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:45:57.287 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:45:57.288 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:45:57.288 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:45:57.288 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:45:57.288 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:45:57.289 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:45:57.289 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:57.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:57.289 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:45:57.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:57.290 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:45:57.290 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:45:57.290 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:45:57.290 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:45:57.290 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:45:57.291 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:45:57.291 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:45:57.291 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:45:57.291 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:57.291 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:57.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:57.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:57.292 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:45:57.292 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:45:57.292 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:45:57.293 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:45:57.293 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:45:57.293 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:45:57.293 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:45:57.293 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:45:57.294 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:57.294 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:57.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:57.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:57.294 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:45:57.295 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:45:57.295 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:45:57.295 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:45:57.295 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:45:57.295 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:45:57.296 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:45:57.296 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:45:57.296 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:57.296 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:57.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:57.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:45:57.297 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:45:57.297 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:45:57.297 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:45:57.297 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:45:57.297 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:45:57.298 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:45:57.298 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:45:57.298 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:45:57.298 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:45:57.299 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:57.299 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:45:57.299 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:45:57.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:57.299 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:45:57.300 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:45:57.300 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:45:57.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:45:57.300 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:45:57Z" 2023-10-18T20:45:57.300 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:45:57.301 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:45:57Z" } 2023-10-18T20:45:57.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:45:57.303 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:45:57.303 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:45:57.303 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697661951.933926_367.1 target=1_1697658841.435004_114.1, shard=3 master=1_1697661951.953788_570.1 target= 2023-10-18T20:46:02.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:02.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:02.306 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:02.364 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:02.364 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:02.396 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:02.394+0000 7f1b1a16f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:02.397 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:02.395+0000 7f1b1a16f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:02.411 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:02.412 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:02.412 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:02.412 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:02.412 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:02.413 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:02.413 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:02.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:02.413 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:02.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:02.414 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:02.414 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:02.414 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:02.414 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:02.415 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:02.415 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:02.415 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:02.415 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:02.415 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:02.416 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:02.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:02.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:02.416 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:02.416 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:02.417 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:02.417 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:46:02.417 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:02.417 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:02.417 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:02.417 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:46:02.418 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:02.418 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:02.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:02.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:02.418 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:02.419 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:02.419 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:02.419 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:02.419 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:02.419 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:02.420 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:02.420 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:02.420 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:02.420 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:02.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:02.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:02.421 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:02.421 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:02.421 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:02.421 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:02.421 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:02.422 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:02.422 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:02.422 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:02.422 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:02.422 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:02.423 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:02.423 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:02.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:02.423 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:02.423 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:02.424 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:02.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:02.424 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:02Z" 2023-10-18T20:46:02.424 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:02.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:02Z" } 2023-10-18T20:46:02.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:46:02.426 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:02.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:46:02.427 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697661951.933926_367.1 target=1_1697658841.435004_114.1, shard=3 master=1_1697661951.953788_570.1 target= 2023-10-18T20:46:07.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:07.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:07.429 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:07.490 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:07.490 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:07.522 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:07.521+0000 7f5ecaecb780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:07.523 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:07.522+0000 7f5ecaecb780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:07.537 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:07.537 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:07.537 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:07.538 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:07.538 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:07.538 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:07.538 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:07.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:07.539 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:07.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:07.539 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:07.539 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:07.540 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:07.540 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:07.540 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:07.540 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:07.540 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:07.541 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:07.541 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:07.541 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:07.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:07.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:07.542 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:07.542 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:07.542 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:07.542 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:46:07.543 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:07.543 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:07.543 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:07.543 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:46:07.543 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:07.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:07.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:07.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:07.544 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:07.544 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:07.545 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:07.545 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:07.545 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:07.545 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:07.546 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:07.546 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:07.546 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:07.546 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:07.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:07.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:07.547 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:07.547 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:07.547 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:07.547 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:07.548 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:07.548 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:07.548 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:07.548 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:07.548 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:07.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:07.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:07.549 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:07.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:07.549 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:07.550 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:07.550 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:07.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:07.550 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:07Z" 2023-10-18T20:46:07.551 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:07.552 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:07Z" } 2023-10-18T20:46:07.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:46:07.553 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:07.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:46:07.553 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697661951.933926_367.1 target=1_1697658841.435004_114.1, shard=3 master=1_1697661951.953788_570.1 target= 2023-10-18T20:46:12.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:12.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:12.556 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:12.614 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:12.614 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:12.647 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:12.645+0000 7fede58f2780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:12.648 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:12.646+0000 7fede58f2780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:12.661 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:12.662 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:12.662 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:12.662 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:12.662 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:12.663 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:12.663 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:12.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:12.663 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:12.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:12.664 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:12.664 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:12.664 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:12.664 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:12.665 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:12.665 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:12.665 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:12.665 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:12.665 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:12.666 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:12.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:12.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:12.666 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:12.667 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:12.667 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:12.667 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:46:12.667 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:12.667 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:12.668 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:12.668 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:46:12.668 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:12.668 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:12.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:12.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:12.669 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:12.669 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:12.669 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:12.669 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:12.670 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:12.670 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:12.670 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:12.670 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:12.671 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:12.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:12.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:12.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:12.671 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:12.672 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:12.672 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:12.672 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:12.672 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:12.672 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:12.673 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:12.673 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:12.673 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:12.673 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:12.673 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:12.674 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:12.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:12.674 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:12.674 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:12.674 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:12.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:12.675 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:12Z" 2023-10-18T20:46:12.675 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:12.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:12Z" } 2023-10-18T20:46:12.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:46:12.677 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:12.677 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:46:12.678 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697661951.933926_367.1 target=1_1697658841.435004_114.1, shard=3 master=1_1697661951.953788_570.1 target= 2023-10-18T20:46:17.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:17.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:17.680 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:17.737 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:17.737 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:17.769 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:17.767+0000 7f9001990780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:17.770 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:17.769+0000 7f9001990780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:17.783 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:46:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:46:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:17Z" 2023-10-18T20:46:17.798 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:17.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:17Z" } 2023-10-18T20:46:17.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:46:17.800 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:17.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:46:17.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697661951.933926_367.1 target=1_1697658841.435004_114.1, shard=3 master=1_1697661951.953788_570.1 target= 2023-10-18T20:46:22.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:22.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:22.804 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:22.862 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:22.863 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:22.895 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:22.894+0000 7f172774f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:22.896 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:22.895+0000 7f172774f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:22.911 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:46:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:46:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:22Z" 2023-10-18T20:46:22.925 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:22.926 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:22Z" } 2023-10-18T20:46:22.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:46:22.927 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:22.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:46:22.928 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697661951.933926_367.1 target=1_1697658841.435004_114.1, shard=3 master=1_1697661951.953788_570.1 target= 2023-10-18T20:46:27.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:27.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:27.931 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:27.987 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:27.988 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:28.020 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:28.019+0000 7fec45a33780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:28.021 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:28.020+0000 7fec45a33780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:28.037 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:28.037 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:28.037 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:28.037 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:28.038 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:28.038 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:28.038 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:28.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:28.039 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:28.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:28.039 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:28.039 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:28.039 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:28.039 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:28.040 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:28.040 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:28.040 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:28.040 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:28.040 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:28.041 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:28.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:28.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:28.041 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:28.041 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:28.042 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:28.042 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:46:28.042 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:28.042 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:28.042 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:28.043 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:46:28.043 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:28.043 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:28.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:28.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:28.043 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:28.044 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:28.044 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:28.044 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:28.044 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:28.045 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:28.045 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:28.045 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:28.045 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:28.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:28.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:28.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:28.046 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:28.046 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:28.046 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:28.047 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:28.047 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:28.047 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:28.047 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:28.047 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:28.048 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:28.048 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:28.048 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:28.048 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:28.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:28.049 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:28.049 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:28.049 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:28.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:28.049 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:28Z" 2023-10-18T20:46:28.049 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:28.050 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:28Z" } 2023-10-18T20:46:28.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:46:28.051 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:28.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:46:28.052 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697661951.933926_367.1 target=1_1697658841.435004_114.1, shard=3 master=1_1697661951.953788_570.1 target= 2023-10-18T20:46:33.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:33.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:33.055 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:33.111 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:33.112 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:33.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:33.142+0000 7f6bd5013780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:33.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:33.143+0000 7f6bd5013780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:33.158 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:33.158 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:33.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:33.160 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:33.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:33.160 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:33.160 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:33.160 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:33.161 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:33.161 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:33.161 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:33.161 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:33.161 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:33.162 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:33.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:33.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:33.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:33.162 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:33.163 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:33.163 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:33.163 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697658841.435004_114.1", 2023-10-18T20:46:33.163 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:33.163 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:33.164 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:33.164 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T20:46:33.164 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:33.164 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:33.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:33.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:33.165 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:33.165 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:33.165 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:33.165 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:33.166 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:33.166 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:33.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:33.166 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:33.166 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:33.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:33.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:33.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:33.167 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:33.167 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:33.168 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:33.168 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:33.168 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:33.168 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:33.168 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:33.169 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:33.169 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:33.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:33.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:33.169 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:33.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:33.170 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:33.170 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:33.170 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:33.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:33.170 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:33Z" 2023-10-18T20:46:33.171 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:33.172 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697658841.435004_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T19:54:01.435004Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:33Z" } 2023-10-18T20:46:33.172 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T20:46:33.173 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:33.173 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T20:46:33.173 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1697661951.933926_367.1 target=1_1697658841.435004_114.1, shard=3 master=1_1697661951.953788_570.1 target= 2023-10-18T20:46:38.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:38.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:38.176 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:38.234 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:38.234 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:38.267 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:38.266+0000 7f4a33117780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:38.268 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:38.267+0000 7f4a33117780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:38.281 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:38.282 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:38.282 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:38.282 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:38.282 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:38.282 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:38.283 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:38.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:38.283 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:38.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:38.283 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:38.284 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:38.284 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:38.284 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:38.284 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:38.284 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:38.285 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:38.285 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:38.285 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:38.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:38.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:38.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:38.286 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:38.286 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:38.286 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:38.286 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:46:38.286 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:38.287 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:38.287 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:38.287 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:46:38.287 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:38.287 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:38.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:38.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:38.288 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:38.288 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:38.288 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:38.289 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:38.289 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:38.289 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:38.289 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:38.289 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:38.289 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:38.290 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:38.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:38.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:38.290 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:38.290 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:38.291 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:38.291 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.950627_569.1", 2023-10-18T20:46:38.291 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:38.291 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:38.291 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:38.292 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.950627Z", 2023-10-18T20:46:38.292 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:38.292 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:38.292 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:38.292 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:38.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:38.293 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:38.293 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:38.293 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:38.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:38.293 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:38Z" 2023-10-18T20:46:38.294 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:38.295 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.950627_569.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.950627Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:38Z" } 2023-10-18T20:46:38.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.950627_569.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.950627Z', 'realm_epoch': 2}}] 2023-10-18T20:46:38.295 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:38.296 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.950627_569.1'} 2023-10-18T20:46:38.296 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697661951.953788_570.1 target=1_1697661951.950627_569.1 2023-10-18T20:46:43.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:43.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:43.299 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:43.356 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:43.356 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:43.388 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:43.386+0000 7fb3d12f3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:43.390 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:43.387+0000 7fb3d12f3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:43.404 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:43.404 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:43.404 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:43.404 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:43.405 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:43.405 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:43.405 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.405 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:43.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.406 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:43.406 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:43.406 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:43.407 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:43.407 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:43.407 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:43.407 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:43.407 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:43.408 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.408 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.409 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:43.409 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:43.409 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:43.409 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:46:43.409 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:43.410 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:43.410 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:43.410 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:46:43.410 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:46:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:46:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:43Z" 2023-10-18T20:46:43.417 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:43.418 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:43Z" } 2023-10-18T20:46:43.419 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:46:43.419 DEBUG:rgw_multi.tests:log_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:43.420 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:43.420 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:46:43.421 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-bofrnf-1 zones={a2, a1} 2023-10-18T20:46:43.421 DEBUG:boto:Method: POST 2023-10-18T20:46:43.421 DEBUG:boto:Path: / 2023-10-18T20:46:43.422 DEBUG:boto:Data: 2023-10-18T20:46:43.422 DEBUG:boto:Headers: {} 2023-10-18T20:46:43.423 DEBUG:boto:Host: smithi082.front.sepia.ceph.com 2023-10-18T20:46:43.423 DEBUG:boto:Port: 8001 2023-10-18T20:46:43.423 DEBUG:boto:Params: {'RoleName': 'roles-bofrnf-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-18T20:46:43.424 DEBUG:boto:Token: None 2023-10-18T20:46:43.425 DEBUG:boto:CanonicalRequest: POST / host:smithi082.front.sepia.ceph.com:8001 x-amz-date:20231018T204643Z host;x-amz-date c7397c7de11d3444f2f2d9ba061edd071c8dec4cae2759e154806e073c3a82ba 2023-10-18T20:46:43.425 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231018T204643Z 20231018/front/smithi082/aws4_request 559fad2cca039d580863c3362e1a0a9952ced44ebd76532111d41ee72bfa99f6 2023-10-18T20:46:43.425 DEBUG:boto:Signature: d9562e7f78e0c28fe205c16b86192dd0e68bc65cc34ba619a711004212935776 2023-10-18T20:46:43.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231018T204643Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=6TC2VRGERC8I036KRP19/20231018/front/smithi082/aws4_request,SignedHeaders=host;x-amz-date,Signature=d9562e7f78e0c28fe205c16b86192dd0e68bc65cc34ba619a711004212935776', 'Host': 'smithi082.front.sepia.ceph.com:8001'} 2023-10-18T20:46:43.428 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000097f12dbed6225d9b-0065304433-1d9c-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '710'), ('Date', 'Wed, 18 Oct 2023 20:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:46:43.429 DEBUG:boto:b'tx0000097f12dbed6225d9b-0065304433-1d9c-a23225a92f-d285-485b-abe0-da83a7e7ffc0roles-bofrnf-1/arn:aws:iam:::role/roles-bofrnf-12023-10-18T20:46:38.71Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-18T20:46:43.430 DEBUG:boto:Method: POST 2023-10-18T20:46:43.430 DEBUG:boto:Path: / 2023-10-18T20:46:43.431 DEBUG:boto:Data: 2023-10-18T20:46:43.431 DEBUG:boto:Headers: {} 2023-10-18T20:46:43.432 DEBUG:boto:Host: smithi082.front.sepia.ceph.com 2023-10-18T20:46:43.432 DEBUG:boto:Port: 8000 2023-10-18T20:46:43.432 DEBUG:boto:Params: {'RoleName': 'roles-bofrnf-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-18T20:46:43.433 DEBUG:boto:Token: None 2023-10-18T20:46:43.433 DEBUG:boto:CanonicalRequest: POST / host:smithi082.front.sepia.ceph.com:8000 x-amz-date:20231018T204643Z host;x-amz-date c7397c7de11d3444f2f2d9ba061edd071c8dec4cae2759e154806e073c3a82ba 2023-10-18T20:46:43.434 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231018T204643Z 20231018/front/smithi082/aws4_request 409ca315218197c7603e8755f1775c86e10091aced0d57b35d415cbf548b7403 2023-10-18T20:46:43.434 DEBUG:boto:Signature: ea4ed4a5913fe0670628f9dc64b3fa1aa39611973f49b733661618a97741c6f2 2023-10-18T20:46:43.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231018T204643Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=6TC2VRGERC8I036KRP19/20231018/front/smithi082/aws4_request,SignedHeaders=host;x-amz-date,Signature=ea4ed4a5913fe0670628f9dc64b3fa1aa39611973f49b733661618a97741c6f2', 'Host': 'smithi082.front.sepia.ceph.com:8000'} 2023-10-18T20:46:43.436 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8c2d5059b573563-0065304433-1664-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 18 Oct 2023 20:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:46:43.437 DEBUG:boto:b'tx00000f8c2d5059b573563-0065304433-1664-a13225a92f-d285-485b-abe0-da83a7e7ffc0roles-bofrnf-1/arn:aws:iam:::role/roles-bofrnf-12023-10-18T20:45:51.930Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-18T20:46:43.438 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-bofrnf-1 2023-10-18T20:46:43.438 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-bofrnf-1 zones={a2, a1} 2023-10-18T20:46:43.439 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-bofrnf-2 zones={a1, a2} 2023-10-18T20:46:43.439 DEBUG:boto:Method: POST 2023-10-18T20:46:43.440 DEBUG:boto:Path: / 2023-10-18T20:46:43.440 DEBUG:boto:Data: 2023-10-18T20:46:43.440 DEBUG:boto:Headers: {} 2023-10-18T20:46:43.441 DEBUG:boto:Host: smithi082.front.sepia.ceph.com 2023-10-18T20:46:43.441 DEBUG:boto:Port: 8000 2023-10-18T20:46:43.442 DEBUG:boto:Params: {'RoleName': 'roles-bofrnf-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-18T20:46:43.442 DEBUG:boto:Token: None 2023-10-18T20:46:43.443 DEBUG:boto:CanonicalRequest: POST / host:smithi082.front.sepia.ceph.com:8000 x-amz-date:20231018T204643Z host;x-amz-date 9309a630c65e80ef470b8c4616d437ada6087fbbca5d959f90a77acb85b1c0ce 2023-10-18T20:46:43.443 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231018T204643Z 20231018/front/smithi082/aws4_request 430003f7b2108e35fa11057cff9cb8b43e29aae7c1d591a43582587e31a3c431 2023-10-18T20:46:43.444 DEBUG:boto:Signature: d55aab9f89b90b0a4e469f3899a6319b4f9863eac599ace3b21f82f95a30ec77 2023-10-18T20:46:43.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231018T204643Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=6TC2VRGERC8I036KRP19/20231018/front/smithi082/aws4_request,SignedHeaders=host;x-amz-date,Signature=d55aab9f89b90b0a4e469f3899a6319b4f9863eac599ace3b21f82f95a30ec77', 'Host': 'smithi082.front.sepia.ceph.com:8000'} 2023-10-18T20:46:43.446 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000194b3186ca14c19e-0065304433-1664-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 18 Oct 2023 20:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:46:43.446 DEBUG:boto:b'tx00000194b3186ca14c19e-0065304433-1664-a1a3082250-309c-4008-9016-250a7c2d081broles-bofrnf-2/arn:aws:iam:::role/roles-bofrnf-22023-10-18T20:45:51.950Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-18T20:46:43.447 DEBUG:boto:Method: POST 2023-10-18T20:46:43.447 DEBUG:boto:Path: / 2023-10-18T20:46:43.448 DEBUG:boto:Data: 2023-10-18T20:46:43.448 DEBUG:boto:Headers: {} 2023-10-18T20:46:43.449 DEBUG:boto:Host: smithi082.front.sepia.ceph.com 2023-10-18T20:46:43.449 DEBUG:boto:Port: 8001 2023-10-18T20:46:43.449 DEBUG:boto:Params: {'RoleName': 'roles-bofrnf-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-18T20:46:43.450 DEBUG:boto:Token: None 2023-10-18T20:46:43.450 DEBUG:boto:CanonicalRequest: POST / host:smithi082.front.sepia.ceph.com:8001 x-amz-date:20231018T204643Z host;x-amz-date 9309a630c65e80ef470b8c4616d437ada6087fbbca5d959f90a77acb85b1c0ce 2023-10-18T20:46:43.451 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231018T204643Z 20231018/front/smithi082/aws4_request 5fb020effbdf41945ebd12c727dd383913586c86f8206e1412dbfaa6fffd0385 2023-10-18T20:46:43.451 DEBUG:boto:Signature: 67e61f04b0b227925be30a075fba5bebafe5c9fccb308d37e301352236efa1a6 2023-10-18T20:46:43.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231018T204643Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=6TC2VRGERC8I036KRP19/20231018/front/smithi082/aws4_request,SignedHeaders=host;x-amz-date,Signature=67e61f04b0b227925be30a075fba5bebafe5c9fccb308d37e301352236efa1a6', 'Host': 'smithi082.front.sepia.ceph.com:8001'} 2023-10-18T20:46:43.454 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000609057c3c6a4144d-0065304433-1d9c-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 18 Oct 2023 20:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:46:43.454 DEBUG:boto:b'tx00000609057c3c6a4144d-0065304433-1d9c-a2a3082250-309c-4008-9016-250a7c2d081broles-bofrnf-2/arn:aws:iam:::role/roles-bofrnf-22023-10-18T20:45:51.950Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-18T20:46:43.455 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-bofrnf-2 2023-10-18T20:46:43.455 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-bofrnf-2 zones={a1, a2} 2023-10-18T20:46:43.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-10-18T20:46:43.457 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:46:43.458 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:46:43.458 DEBUG:boto:Method: PUT 2023-10-18T20:46:43.458 DEBUG:boto:Path: /bofrnf-60/ 2023-10-18T20:46:43.459 DEBUG:boto:Data: 2023-10-18T20:46:43.459 DEBUG:boto:Headers: {} 2023-10-18T20:46:43.460 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:46:43.460 DEBUG:boto:Port: 8000 2023-10-18T20:46:43.460 DEBUG:boto:Params: {} 2023-10-18T20:46:43.461 DEBUG:boto:Token: None 2023-10-18T20:46:43.461 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:46:43 GMT /bofrnf-60/ 2023-10-18T20:46:43.462 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:biyJhrg9olejkClrhSpfTOPK/i0= 2023-10-18T20:46:43.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:46:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:biyJhrg9olejkClrhSpfTOPK/i0=', 'Content-Length': '0'} 2023-10-18T20:46:43.477 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ab4cd9b457a10893-0065304433-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:46:43.478 DEBUG:rgw_multi.tests:created bucket=bofrnf-60 2023-10-18T20:46:43.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:43.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:43.479 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:43.528 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:43.528 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:43.557 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:46:43.570 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:46:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.571 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662003.475465_1213.1", 2023-10-18T20:46:43.571 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:46:43.475465Z" 2023-10-18T20:46:43.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.572 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:46:43.572 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T20:46:43.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:46:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:46:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T20:46:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.574 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:46:43.575 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:43.576 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:46:43.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:43.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:43.577 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:43.626 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:43.626 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:43.655 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:43.653+0000 7f9446b20780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:43.656 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:43.654+0000 7f9446b20780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:43.668 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:43.669 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:43.669 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:43.669 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:43.669 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:43.670 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:43.670 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.670 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:43.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.671 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:43.671 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:43.671 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:43.671 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:43.671 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:43.672 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:43.672 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:43.672 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:43.672 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.672 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.673 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:43.673 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:43.673 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:43.674 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:46:43.674 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:43.674 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:43.674 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:43.675 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:46:43.675 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.675 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.675 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:43.676 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:43.676 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:43.676 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:43.676 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:43.676 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:43.677 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:43.677 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:43.677 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.677 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:43.678 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:43.678 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:43.678 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:43.678 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:46:43.679 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:43.679 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:43.679 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:43.679 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:46:43.679 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:43.680 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.680 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:43.680 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:43.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.680 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:43.680 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:43.681 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:43.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:43.681 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:43Z" 2023-10-18T20:46:43.681 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:43.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:43Z" } 2023-10-18T20:46:43.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:46:43.683 DEBUG:rgw_multi.tests:log_status={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:43.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:43.684 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697662003.475465_1213.1 target=1_1697661789.583751_1186.1 2023-10-18T20:46:48.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:48.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:48.686 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:48.743 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:48.743 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:48.776 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:48.774+0000 7f49f55b3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:48.777 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:48.775+0000 7f49f55b3780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:48.791 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:48.791 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:48.792 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:48.792 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:48.792 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:48.792 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:48.792 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:48.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:48.793 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:48.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:48.793 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:48.794 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:48.794 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:48.794 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:48.794 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:48.794 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:48.795 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:48.795 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:48.795 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:48.795 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:48.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:48.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:48.796 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:48.796 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:48.796 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:48.797 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:46:48.797 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:48.797 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:48.797 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:48.797 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:46:48.798 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:48.798 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:48.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:48.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:48.799 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:48.799 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:48.799 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:48.799 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:48.799 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:48.800 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:48.800 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:48.800 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:48.800 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:48.800 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:48.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:48.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:48.801 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:48.801 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:48.801 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:48.801 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:46:48.802 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:48.802 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:48.802 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:48.802 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:46:48.802 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:48.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:48.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:48.803 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:48.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:48.803 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:48.804 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:48.804 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:48.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:48.804 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:48Z" 2023-10-18T20:46:48.804 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:48.805 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:48Z" } 2023-10-18T20:46:48.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:46:48.806 DEBUG:rgw_multi.tests:log_status={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:48.807 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:48.807 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697662003.475465_1213.1 target=1_1697661789.583751_1186.1 2023-10-18T20:46:53.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:53.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:53.809 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:53.865 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:53.865 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:53.899 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:53.896+0000 7fbb1af5a780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:53.900 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:53.897+0000 7fbb1af5a780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:53.913 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:53.914 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:53.914 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:53.914 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:53.914 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:53.915 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:53.915 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:53.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:53.915 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:53.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:53.916 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:53.916 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:53.916 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:53.916 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:53.917 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:53.917 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:53.917 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:53.917 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:53.917 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:53.918 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:53.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:53.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:53.918 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:53.919 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:53.919 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:53.919 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:46:53.919 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:53.920 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:53.920 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:53.920 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:46:53.920 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:53.920 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:53.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:53.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:53.921 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:53.921 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:53.922 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:53.922 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:53.922 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:53.922 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:53.922 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:53.923 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:53.923 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:53.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:53.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:53.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:53.924 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:53.924 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:53.924 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:53.924 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:46:53.925 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:53.925 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:53.925 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:53.925 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:46:53.926 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:53.926 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:53.926 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:53.926 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:53.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:53.927 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:53.927 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:53.927 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:53.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:53.928 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:53Z" 2023-10-18T20:46:53.928 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:53.929 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:53Z" } 2023-10-18T20:46:53.930 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:46:53.930 DEBUG:rgw_multi.tests:log_status={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:53.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:53.931 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697662003.475465_1213.1 target=1_1697661789.583751_1186.1 2023-10-18T20:46:58.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:58.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:46:58.933 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:46:58.990 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:46:58.991 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:46:59.022 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:59.020+0000 7fae666d8780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:59.022 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:46:59.021+0000 7fae666d8780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:46:59.036 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:46:59.037 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:46:59.037 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:46:59.037 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:46:59.037 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:46:59.038 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:46:59.038 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:59.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:59.038 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:46:59.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:59.039 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:46:59.039 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:59.039 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:59.039 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:46:59.040 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:59.040 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:46:59.040 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:59.040 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:46:59.040 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:59.041 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:59.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:59.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:59.041 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:46:59.041 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:59.042 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:59.042 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:46:59.042 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:59.042 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:59.042 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:59.043 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:46:59.043 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:59.043 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:59.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:59.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:59.044 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:46:59.044 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:59.044 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:59.044 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:46:59.044 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:59.045 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:59.045 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:59.045 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:46:59.045 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:59.046 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:59.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:59.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:46:59.047 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:46:59.047 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:46:59.047 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:46:59.047 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:46:59.047 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:46:59.048 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:46:59.048 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:46:59.048 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:46:59.048 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:46:59.048 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:59.049 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:46:59.049 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:46:59.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:59.049 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:46:59.049 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:46:59.050 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:46:59.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:46:59.050 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:46:59Z" 2023-10-18T20:46:59.050 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:46:59.051 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:46:59Z" } 2023-10-18T20:46:59.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:46:59.052 DEBUG:rgw_multi.tests:log_status={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:59.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:46:59.053 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697662003.475465_1213.1 target=1_1697661789.583751_1186.1 2023-10-18T20:47:04.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:47:04.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:47:04.056 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:47:04.114 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:47:04.115 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:47:04.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:04.145+0000 7ff98c69f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:47:04.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:04.146+0000 7ff98c69f780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:47:04.161 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:47:04.162 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:47:04.162 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:47:04.162 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:47:04.162 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:47:04.163 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:47:04.163 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:04.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:04.163 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:47:04.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:04.164 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:47:04.164 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:04.164 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:04.164 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:47:04.165 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:04.165 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:47:04.165 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:04.165 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:47:04.165 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:04.166 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:04.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:04.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:04.166 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:47:04.166 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:04.167 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:04.167 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:47:04.167 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:04.167 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:04.168 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:04.168 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:47:04.168 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:04.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:04.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:04.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:04.169 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:47:04.169 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:04.170 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:04.170 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:47:04.170 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:04.170 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:04.171 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:04.171 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:47:04.171 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:04.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:04.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:04.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:04.172 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:47:04.172 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:04.173 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:04.173 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:47:04.173 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:04.173 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:04.173 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:04.174 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:47:04.174 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:04.174 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:04.174 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:04.174 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:47:04.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:04.175 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:47:04.175 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:47:04.175 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:47:04.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:04.176 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:47:04Z" 2023-10-18T20:47:04.176 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:47:04.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:47:04Z" } 2023-10-18T20:47:04.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:47:04.178 DEBUG:rgw_multi.tests:log_status={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:47:04.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:47:04.179 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697662003.475465_1213.1 target=1_1697661789.583751_1186.1 2023-10-18T20:47:09.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:47:09.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:47:09.182 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:47:09.239 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:47:09.240 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:47:09.272 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:09.271+0000 7f9d36802780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:47:09.273 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:09.272+0000 7f9d36802780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:47:09.287 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:47:09.288 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:47:09.288 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:47:09.288 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:47:09.288 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:47:09.289 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:47:09.289 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:09.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:09.289 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:47:09.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:09.290 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:47:09.290 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:09.290 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:09.290 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661789.583751_1186.1", 2023-10-18T20:47:09.291 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:09.291 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:47:09.291 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:09.291 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.583751Z", 2023-10-18T20:47:09.291 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:09.292 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:09.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:09.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:09.292 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:47:09.292 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:09.293 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:09.293 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:47:09.293 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:09.293 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:09.294 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:09.294 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:47:09.294 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:09.294 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:09.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:09.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:09.295 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:47:09.295 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:09.295 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:09.295 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:47:09.296 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:09.296 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:09.296 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:09.296 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:47:09.296 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:09.297 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:09.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:09.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:09.297 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:47:09.297 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:09.298 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:09.298 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:47:09.298 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:09.298 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:09.299 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:09.299 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:47:09.299 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:09.299 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:09.299 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:09.300 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:47:09.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:09.300 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:47:09.300 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:47:09.300 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:47:09.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:09.301 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:47:09Z" 2023-10-18T20:47:09.301 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:47:09.302 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697661789.583751_1186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:43:09.583751Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:47:09Z" } 2023-10-18T20:47:09.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697661789.583751_1186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.583751Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:47:09.303 DEBUG:rgw_multi.tests:log_status={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:47:09.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1697661789.583751_1186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:47:09.304 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697662003.475465_1213.1 target=1_1697661789.583751_1186.1 2023-10-18T20:47:14.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:47:14.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:47:14.307 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:47:14.361 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:47:14.362 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:47:14.393 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:14.391+0000 7f060a3b2780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:47:14.393 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:14.392+0000 7f060a3b2780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:47:14.406 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:47:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:47:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:47:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:47:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:47:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:47:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:47:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:47:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662003.469436_1210.1", 2023-10-18T20:47:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:47:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:46:43.469436Z", 2023-10-18T20:47:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:47:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:47:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:47:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:47:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:47:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:47:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:47:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:47:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:47:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:47:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:47:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:47:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:47:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:47:14Z" 2023-10-18T20:47:14.421 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:47:14.422 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697662003.469436_1210.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:46:43.469436Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:47:14Z" } 2023-10-18T20:47:14.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697662003.469436_1210.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:46:43.469436Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:47:14.423 DEBUG:rgw_multi.tests:log_status={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:47:14.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1697662003.469436_1210.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:47:14.424 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697662003.475465_1213.1 target=1_1697662003.469436_1210.1 2023-10-18T20:47:19.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:47:19.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:47:19.427 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:47:19.483 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:47:19.484 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:47:19.514 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:19.513+0000 7f3627130780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:47:19.515 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:19.514+0000 7f3627130780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:47:19.529 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:47:19.529 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:47:19.530 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:47:19.530 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:47:19.530 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:47:19.530 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:47:19.531 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:19.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:19.531 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:47:19.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:19.532 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:47:19.532 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:19.532 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:19.532 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662003.475465_1213.1", 2023-10-18T20:47:19.532 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:19.533 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:47:19.533 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:19.533 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:46:43.475465Z", 2023-10-18T20:47:19.533 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:19.533 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:19.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:19.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:19.534 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:47:19.534 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:19.535 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:19.535 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:47:19.535 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:19.535 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:19.535 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:19.536 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:47:19.536 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:19.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:19.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:19.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:19.537 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:47:19.537 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:19.537 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:19.537 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:47:19.538 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:19.538 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:19.538 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:19.538 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:47:19.538 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:19.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:19.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:19.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:47:19.539 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:47:19.539 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:47:19.540 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:47:19.540 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:47:19.540 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:47:19.540 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:47:19.541 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:47:19.541 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:47:19.541 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:47:19.541 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:19.541 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:47:19.542 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:47:19.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:19.542 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:47:19.542 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:47:19.542 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:47:19.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:47:19.543 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:47:19Z" 2023-10-18T20:47:19.543 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:47:19.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697662003.475465_1213.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:46:43.475465Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:47:19Z" } 2023-10-18T20:47:19.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697662003.475465_1213.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:46:43.475465Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:47:19.545 DEBUG:rgw_multi.tests:log_status={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:47:19.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1697662003.475465_1213.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:47:19.546 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:47:19.546 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-18T20:47:19.547 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T20:47:19.548 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:47:19.546+0000 7fae12b1b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 169636) UID: 0 2023-10-18T20:47:19.548 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:47:19.546+0000 7fae16d847c0 -1 shutting down 2023-10-18T20:47:20.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T20:47:25.648 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-18T20:47:25.649 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:47:25.650 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:47:25.650 DEBUG:boto:Method: HEAD 2023-10-18T20:47:25.651 DEBUG:boto:Path: /bofrnf-60/ 2023-10-18T20:47:25.651 DEBUG:boto:Data: 2023-10-18T20:47:25.652 DEBUG:boto:Headers: {} 2023-10-18T20:47:25.652 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.652 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.653 DEBUG:boto:Params: {} 2023-10-18T20:47:25.653 DEBUG:boto:Token: None 2023-10-18T20:47:25.654 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/ 2023-10-18T20:47:25.655 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI= 2023-10-18T20:47:25.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI=', 'Content-Length': '0'} 2023-10-18T20:47:25.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000303d32974294db12-006530445d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.659 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:47:25.659 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:47:25.659 DEBUG:boto:Method: PUT 2023-10-18T20:47:25.660 DEBUG:boto:Path: /bofrnf-60/obj11 2023-10-18T20:47:25.660 DEBUG:boto:Data: 2023-10-18T20:47:25.661 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:47:25.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.661 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.662 DEBUG:boto:Params: {} 2023-10-18T20:47:25.662 DEBUG:boto:Token: None 2023-10-18T20:47:25.663 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/obj11 2023-10-18T20:47:25.663 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A1QWUMyQfgQ5t1Fz1q7GAPejWK4= 2023-10-18T20:47:25.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A1QWUMyQfgQ5t1Fz1q7GAPejWK4='} 2023-10-18T20:47:25.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089457d343d2a2560-006530445d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.672 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:47:25.673 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:47:25.673 DEBUG:boto:Method: HEAD 2023-10-18T20:47:25.673 DEBUG:boto:Path: /bofrnf-60/ 2023-10-18T20:47:25.674 DEBUG:boto:Data: 2023-10-18T20:47:25.674 DEBUG:boto:Headers: {} 2023-10-18T20:47:25.674 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.675 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.675 DEBUG:boto:Params: {} 2023-10-18T20:47:25.676 DEBUG:boto:Token: None 2023-10-18T20:47:25.676 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/ 2023-10-18T20:47:25.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI= 2023-10-18T20:47:25.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI=', 'Content-Length': '0'} 2023-10-18T20:47:25.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fabe186fc189e57-006530445d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.679 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:47:25.680 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:47:25.680 DEBUG:boto:Method: PUT 2023-10-18T20:47:25.681 DEBUG:boto:Path: /bofrnf-60/obj22 2023-10-18T20:47:25.681 DEBUG:boto:Data: 2023-10-18T20:47:25.681 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:47:25.682 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.682 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.683 DEBUG:boto:Params: {} 2023-10-18T20:47:25.683 DEBUG:boto:Token: None 2023-10-18T20:47:25.684 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/obj22 2023-10-18T20:47:25.684 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NWHG7Avx1037zI2mjbf7nc8Skr4= 2023-10-18T20:47:25.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NWHG7Avx1037zI2mjbf7nc8Skr4='} 2023-10-18T20:47:25.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e25fbb976cae38fb-006530445d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.727 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:47:25.728 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:47:25.728 DEBUG:boto:Method: HEAD 2023-10-18T20:47:25.728 DEBUG:boto:Path: /bofrnf-60/ 2023-10-18T20:47:25.729 DEBUG:boto:Data: 2023-10-18T20:47:25.729 DEBUG:boto:Headers: {} 2023-10-18T20:47:25.730 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.730 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.731 DEBUG:boto:Params: {} 2023-10-18T20:47:25.731 DEBUG:boto:Token: None 2023-10-18T20:47:25.731 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/ 2023-10-18T20:47:25.732 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI= 2023-10-18T20:47:25.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI=', 'Content-Length': '0'} 2023-10-18T20:47:25.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b4994e8ea789394-006530445d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.735 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:47:25.736 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:47:25.736 DEBUG:boto:Method: PUT 2023-10-18T20:47:25.737 DEBUG:boto:Path: /bofrnf-60/obj33 2023-10-18T20:47:25.737 DEBUG:boto:Data: 2023-10-18T20:47:25.738 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:47:25.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.738 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.739 DEBUG:boto:Params: {} 2023-10-18T20:47:25.739 DEBUG:boto:Token: None 2023-10-18T20:47:25.740 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/obj33 2023-10-18T20:47:25.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a/i+6zfXtPLteGy0VPGLr8/NTFY= 2023-10-18T20:47:25.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a/i+6zfXtPLteGy0VPGLr8/NTFY='} 2023-10-18T20:47:25.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000582a9161c4b48d5c-006530445d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.747 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:47:25.748 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:47:25.748 DEBUG:boto:Method: HEAD 2023-10-18T20:47:25.749 DEBUG:boto:Path: /bofrnf-60/ 2023-10-18T20:47:25.749 DEBUG:boto:Data: 2023-10-18T20:47:25.749 DEBUG:boto:Headers: {} 2023-10-18T20:47:25.750 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.750 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.751 DEBUG:boto:Params: {} 2023-10-18T20:47:25.751 DEBUG:boto:Token: None 2023-10-18T20:47:25.752 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/ 2023-10-18T20:47:25.752 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI= 2023-10-18T20:47:25.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI=', 'Content-Length': '0'} 2023-10-18T20:47:25.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d282c3dd440a5ef2-006530445d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.755 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:47:25.756 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:47:25.756 DEBUG:boto:Method: PUT 2023-10-18T20:47:25.757 DEBUG:boto:Path: /bofrnf-60/obj44 2023-10-18T20:47:25.757 DEBUG:boto:Data: 2023-10-18T20:47:25.757 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:47:25.758 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.758 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.759 DEBUG:boto:Params: {} 2023-10-18T20:47:25.759 DEBUG:boto:Token: None 2023-10-18T20:47:25.760 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/obj44 2023-10-18T20:47:25.760 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:a/04F95s0MEv90EE64+bKkqnOrA= 2023-10-18T20:47:25.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:a/04F95s0MEv90EE64+bKkqnOrA='} 2023-10-18T20:47:25.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043ae356b35b27df2-006530445d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.803 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:47:25.804 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:47:25.804 DEBUG:boto:Method: HEAD 2023-10-18T20:47:25.804 DEBUG:boto:Path: /bofrnf-60/ 2023-10-18T20:47:25.805 DEBUG:boto:Data: 2023-10-18T20:47:25.805 DEBUG:boto:Headers: {} 2023-10-18T20:47:25.805 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.806 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.806 DEBUG:boto:Params: {} 2023-10-18T20:47:25.807 DEBUG:boto:Token: None 2023-10-18T20:47:25.807 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/ 2023-10-18T20:47:25.808 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI= 2023-10-18T20:47:25.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DoXuC2gPUqVXx6i+qsQR45szOlI=', 'Content-Length': '0'} 2023-10-18T20:47:25.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a1cec2a59386d68-006530445d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.811 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:47:25.812 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:47:25.812 DEBUG:boto:Method: PUT 2023-10-18T20:47:25.813 DEBUG:boto:Path: /bofrnf-60/obj55 2023-10-18T20:47:25.813 DEBUG:boto:Data: 2023-10-18T20:47:25.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:47:25.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:47:25.814 DEBUG:boto:Port: 8000 2023-10-18T20:47:25.815 DEBUG:boto:Params: {} 2023-10-18T20:47:25.815 DEBUG:boto:Token: None 2023-10-18T20:47:25.816 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:47:25 GMT /bofrnf-60/obj55 2023-10-18T20:47:25.816 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1bw7brhP12jd+H0fWfiIqrYS1IQ= 2023-10-18T20:47:25.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:47:25 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1bw7brhP12jd+H0fWfiIqrYS1IQ='} 2023-10-18T20:47:25.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fed7aa30068e93a-006530445d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:47:25.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:47:25.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:47:25.823 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:47:25.879 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:47:25.879 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:47:25.914 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:25.913+0000 7f9107acb780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-18T20:47:25.915 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:25.914+0000 7f9107acb780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-18T20:47:25.937 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-18T20:47:25.937 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T20:47:25.940 INFO:tasks.rgw.c1.client.1:Started 2023-10-18T20:47:25.940 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:47:25.961 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:47:25.961 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:47:25.962 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T20:47:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T20:47:26.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-18T20:47:26.964 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:47:26.987 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:47:26.988 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:47:26.988 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T20:47:26.989 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T20:47:26.990 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-60 2023-10-18T20:47:26.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:47:26.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:47:26.991 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:47:27.089 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:47:27.090 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:47:27.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:27.123+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:47:32.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:32.123+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:47:37.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:37.123+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:47:42.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:42.124+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:47:47.126 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:47.125+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:47:52.127 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:52.126+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:47:57.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:47:57.126+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:02.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:02.127+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:07.129 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:07.127+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:12.130 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:12.128+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:17.131 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:17.129+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:22.131 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:22.130+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:27.132 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:27.129+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:32.132 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:32.130+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:37.133 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:37.131+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:42.134 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:42.133+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:47.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:47.133+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:52.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:52.134+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:48:57.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:48:57.135+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:02.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:02.135+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:07.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:07.136+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:12.138 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:12.137+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:17.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:17.136+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:22.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:22.137+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:27.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:27.138+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:32.141 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:32.139+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:37.141 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:37.140+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:42.142 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:42.141+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:47.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:47.141+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:52.143 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:52.142+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:49:57.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:49:57.142+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:50:02.144 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:02.143+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:50:07.145 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:07.143+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:50:12.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:12.143+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:50:17.146 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:17.144+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:50:22.147 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:22.145+0000 7f53ca312780 1 waiting to reach incremental sync.. 2023-10-18T20:50:27.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:27.147+0000 7f53ca312780 1 bucket sync caught up with source: 2023-10-18T20:50:27.149 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.95.6, 00000000002.97.6, 00000000002.93.6] 2023-10-18T20:50:27.149 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.95.6, 00000000002.97.6, 00000000002.93.6] 2023-10-18T20:50:27.150 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:27.147+0000 7f53ca312780 0 bucket checkpoint complete 2023-10-18T20:50:27.166 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-60 2023-10-18T20:50:27.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:50:27.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:50:27.169 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:50:27.232 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:27.233 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:27.270 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:27.269+0000 7f20a4e6c780 1 bucket sync caught up with source: 2023-10-18T20:50:27.271 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.949.6, 00000000002.718.6, 00000000002.958.6] 2023-10-18T20:50:27.271 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.949.6, 00000000002.718.6, 00000000002.958.6] 2023-10-18T20:50:27.271 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:27.269+0000 7f20a4e6c780 0 bucket checkpoint complete 2023-10-18T20:50:27.284 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-60 zones={a2, a1} 2023-10-18T20:50:27.285 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.285 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.286 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.286 DEBUG:boto:Path: /bofrnf-60/ 2023-10-18T20:50:27.287 DEBUG:boto:Data: 2023-10-18T20:50:27.287 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.288 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.289 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.289 DEBUG:boto:Params: {} 2023-10-18T20:50:27.290 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:50:27.291 DEBUG:boto:Token: None 2023-10-18T20:50:27.291 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.292 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:43wemVbAOvYknXNwtsf78/k3yrM= 2023-10-18T20:50:27.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:43wemVbAOvYknXNwtsf78/k3yrM=', 'Content-Length': '0'} 2023-10-18T20:50:27.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d9fedfec70456b0-0065304513-1e64-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.298 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.298 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.298 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.299 DEBUG:boto:Path: /bofrnf-60/ 2023-10-18T20:50:27.299 DEBUG:boto:Data: 2023-10-18T20:50:27.300 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.300 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.300 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.301 DEBUG:boto:Params: {} 2023-10-18T20:50:27.301 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:50:27.302 DEBUG:boto:Token: None 2023-10-18T20:50:27.302 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.303 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:43wemVbAOvYknXNwtsf78/k3yrM= 2023-10-18T20:50:27.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:43wemVbAOvYknXNwtsf78/k3yrM=', 'Content-Length': '0'} 2023-10-18T20:50:27.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091ae802d680ab6a0-0065304513-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.307 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:50:27.308 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.308 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.309 DEBUG:boto:path=/bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.309 DEBUG:boto:auth_path=/bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.309 DEBUG:boto:Method: GET 2023-10-18T20:50:27.310 DEBUG:boto:Path: /bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.310 DEBUG:boto:Data: 2023-10-18T20:50:27.311 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.311 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.311 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.312 DEBUG:boto:Params: {} 2023-10-18T20:50:27.312 DEBUG:boto:Token: None 2023-10-18T20:50:27.313 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/?versions 2023-10-18T20:50:27.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ehrKeHMRR5gD4sOSd69PQm/tBvI= 2023-10-18T20:50:27.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ehrKeHMRR5gD4sOSd69PQm/tBvI=', 'Content-Length': '0'} 2023-10-18T20:50:27.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004429627ee15e8093-0065304513-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.317 DEBUG:boto:b'bofrnf-60999falseobj11nulltrue2023-10-18T20:47:25.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:47:25.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:47:25.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:47:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:47:25.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:50:27.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:50:27.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:50:27.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:50:27.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:50:27.320 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:50:27.320 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.321 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.321 DEBUG:boto:path=/bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.321 DEBUG:boto:auth_path=/bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.322 DEBUG:boto:Method: GET 2023-10-18T20:50:27.322 DEBUG:boto:Path: /bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.323 DEBUG:boto:Data: 2023-10-18T20:50:27.323 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.324 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.324 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.324 DEBUG:boto:Params: {} 2023-10-18T20:50:27.325 DEBUG:boto:Token: None 2023-10-18T20:50:27.325 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/?versions 2023-10-18T20:50:27.326 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ehrKeHMRR5gD4sOSd69PQm/tBvI= 2023-10-18T20:50:27.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ehrKeHMRR5gD4sOSd69PQm/tBvI=', 'Content-Length': '0'} 2023-10-18T20:50:27.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2b8f2f0ee034f8-0065304513-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.329 DEBUG:boto:b'bofrnf-60999falseobj11nulltrue2023-10-18T20:47:25.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:47:25.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:47:25.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:47:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:47:25.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:50:27.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:50:27.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:50:27.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:50:27.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:50:27.331 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.332 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.332 DEBUG:boto:path=/bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.333 DEBUG:boto:auth_path=/bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.333 DEBUG:boto:Method: GET 2023-10-18T20:50:27.334 DEBUG:boto:Path: /bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.334 DEBUG:boto:Data: 2023-10-18T20:50:27.334 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.335 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.335 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.336 DEBUG:boto:Params: {} 2023-10-18T20:50:27.336 DEBUG:boto:Token: None 2023-10-18T20:50:27.337 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/?versions 2023-10-18T20:50:27.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ehrKeHMRR5gD4sOSd69PQm/tBvI= 2023-10-18T20:50:27.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ehrKeHMRR5gD4sOSd69PQm/tBvI=', 'Content-Length': '0'} 2023-10-18T20:50:27.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc77acf5381d7491-0065304513-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.340 DEBUG:boto:b'bofrnf-60999falseobj11nulltrue2023-10-18T20:47:25.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:47:25.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:47:25.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:47:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:47:25.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.341 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.341 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.342 DEBUG:boto:path=/bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.342 DEBUG:boto:auth_path=/bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.343 DEBUG:boto:Method: GET 2023-10-18T20:50:27.343 DEBUG:boto:Path: /bofrnf-60/?versions&max-keys=999 2023-10-18T20:50:27.344 DEBUG:boto:Data: 2023-10-18T20:50:27.344 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.344 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.345 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.345 DEBUG:boto:Params: {} 2023-10-18T20:50:27.346 DEBUG:boto:Token: None 2023-10-18T20:50:27.346 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/?versions 2023-10-18T20:50:27.347 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ehrKeHMRR5gD4sOSd69PQm/tBvI= 2023-10-18T20:50:27.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ehrKeHMRR5gD4sOSd69PQm/tBvI=', 'Content-Length': '0'} 2023-10-18T20:50:27.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb793d585b736648-0065304513-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.350 DEBUG:boto:b'bofrnf-60999falseobj11nulltrue2023-10-18T20:47:25.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:47:25.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:47:25.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:47:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:47:25.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:50:27.351 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.352 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.352 DEBUG:boto:path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.352 DEBUG:boto:auth_path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.353 DEBUG:boto:Method: GET 2023-10-18T20:50:27.353 DEBUG:boto:Path: /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.354 DEBUG:boto:Data: 2023-10-18T20:50:27.354 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.355 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.355 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.355 DEBUG:boto:Params: {} 2023-10-18T20:50:27.356 DEBUG:boto:Token: None 2023-10-18T20:50:27.356 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.357 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9CZDoU0TkycwI7nFl2/hpwJjTVE= 2023-10-18T20:50:27.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9CZDoU0TkycwI7nFl2/hpwJjTVE=', 'Content-Length': '0'} 2023-10-18T20:50:27.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087bb0f7374fce8a7-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.360 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.361 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.361 DEBUG:boto:path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.361 DEBUG:boto:auth_path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.362 DEBUG:boto:Method: GET 2023-10-18T20:50:27.362 DEBUG:boto:Path: /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.363 DEBUG:boto:Data: 2023-10-18T20:50:27.363 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.364 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.364 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.364 DEBUG:boto:Params: {} 2023-10-18T20:50:27.365 DEBUG:boto:Token: None 2023-10-18T20:50:27.365 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.366 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9CZDoU0TkycwI7nFl2/hpwJjTVE= 2023-10-18T20:50:27.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9CZDoU0TkycwI7nFl2/hpwJjTVE=', 'Content-Length': '0'} 2023-10-18T20:50:27.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c8cf7b1a38070dc-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.369 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.369 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.369 DEBUG:boto:path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.370 DEBUG:boto:auth_path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.370 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.371 DEBUG:boto:Path: /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.371 DEBUG:boto:Data: 2023-10-18T20:50:27.372 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.372 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.372 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.373 DEBUG:boto:Params: {} 2023-10-18T20:50:27.373 DEBUG:boto:Token: None 2023-10-18T20:50:27.374 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.374 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ex8bDp01WO9s4kvKMohUFfIljr8= 2023-10-18T20:50:27.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ex8bDp01WO9s4kvKMohUFfIljr8=', 'Content-Length': '0'} 2023-10-18T20:50:27.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4ae0bdb2a6a20e-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.377 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.377 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.378 DEBUG:boto:path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.378 DEBUG:boto:auth_path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.378 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.379 DEBUG:boto:Path: /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.379 DEBUG:boto:Data: 2023-10-18T20:50:27.380 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.380 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.381 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.381 DEBUG:boto:Params: {} 2023-10-18T20:50:27.381 DEBUG:boto:Token: None 2023-10-18T20:50:27.382 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.382 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ex8bDp01WO9s4kvKMohUFfIljr8= 2023-10-18T20:50:27.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ex8bDp01WO9s4kvKMohUFfIljr8=', 'Content-Length': '0'} 2023-10-18T20:50:27.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7f93cd1982e8923-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:50:27.385 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.386 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.386 DEBUG:boto:path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.387 DEBUG:boto:auth_path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.387 DEBUG:boto:Method: GET 2023-10-18T20:50:27.388 DEBUG:boto:Path: /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.388 DEBUG:boto:Data: 2023-10-18T20:50:27.389 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.390 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.390 DEBUG:boto:Params: {} 2023-10-18T20:50:27.391 DEBUG:boto:Token: None 2023-10-18T20:50:27.391 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9CZDoU0TkycwI7nFl2/hpwJjTVE= 2023-10-18T20:50:27.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9CZDoU0TkycwI7nFl2/hpwJjTVE=', 'Content-Length': '0'} 2023-10-18T20:50:27.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085d8bfca9ace64de-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.395 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.395 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.396 DEBUG:boto:path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.396 DEBUG:boto:auth_path=/bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.396 DEBUG:boto:Method: GET 2023-10-18T20:50:27.397 DEBUG:boto:Path: /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.397 DEBUG:boto:Data: 2023-10-18T20:50:27.398 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.398 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.399 DEBUG:boto:Params: {} 2023-10-18T20:50:27.399 DEBUG:boto:Token: None 2023-10-18T20:50:27.400 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11?versionId=null 2023-10-18T20:50:27.400 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9CZDoU0TkycwI7nFl2/hpwJjTVE= 2023-10-18T20:50:27.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9CZDoU0TkycwI7nFl2/hpwJjTVE=', 'Content-Length': '0'} 2023-10-18T20:50:27.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000948be7ec08213221-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.403 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.404 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.404 DEBUG:boto:path=/bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.405 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.405 DEBUG:boto:Method: GET 2023-10-18T20:50:27.405 DEBUG:boto:Path: /bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.406 DEBUG:boto:Data: 2023-10-18T20:50:27.406 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.407 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.407 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.408 DEBUG:boto:Params: {} 2023-10-18T20:50:27.408 DEBUG:boto:Token: None 2023-10-18T20:50:27.409 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.409 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051b0669a6180128c-0065304513-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.412 DEBUG:boto:b'bofrnf-60obj111000falseobj112023-10-18T20:47:25.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.413 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.414 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.414 DEBUG:boto:path=/bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.414 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.415 DEBUG:boto:Method: GET 2023-10-18T20:50:27.415 DEBUG:boto:Path: /bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.416 DEBUG:boto:Data: 2023-10-18T20:50:27.416 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.417 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.417 DEBUG:boto:Params: {} 2023-10-18T20:50:27.418 DEBUG:boto:Token: None 2023-10-18T20:50:27.419 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c43c034b6f0bae7-0065304513-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.422 DEBUG:boto:b'bofrnf-60obj111000falseobj112023-10-18T20:47:25.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.423 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.423 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.424 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.424 DEBUG:boto:Path: /bofrnf-60/obj11 2023-10-18T20:50:27.425 DEBUG:boto:Data: 2023-10-18T20:50:27.425 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.426 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.426 DEBUG:boto:Params: {} 2023-10-18T20:50:27.427 DEBUG:boto:Token: None 2023-10-18T20:50:27.427 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11 2023-10-18T20:50:27.428 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Hu8YSj+q8xTo63DRLV79gJm5NIo= 2023-10-18T20:50:27.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Hu8YSj+q8xTo63DRLV79gJm5NIo=', 'Content-Length': '0'} 2023-10-18T20:50:27.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7430c9f88dab1e5-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.430 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.431 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.431 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.432 DEBUG:boto:Path: /bofrnf-60/obj11 2023-10-18T20:50:27.432 DEBUG:boto:Data: 2023-10-18T20:50:27.432 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.433 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.434 DEBUG:boto:Params: {} 2023-10-18T20:50:27.434 DEBUG:boto:Token: None 2023-10-18T20:50:27.434 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11 2023-10-18T20:50:27.435 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Hu8YSj+q8xTo63DRLV79gJm5NIo= 2023-10-18T20:50:27.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Hu8YSj+q8xTo63DRLV79gJm5NIo=', 'Content-Length': '0'} 2023-10-18T20:50:27.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5588f52dcf40042-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:50:27.438 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.439 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.439 DEBUG:boto:Method: GET 2023-10-18T20:50:27.439 DEBUG:boto:Path: /bofrnf-60/obj11 2023-10-18T20:50:27.440 DEBUG:boto:Data: 2023-10-18T20:50:27.440 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.441 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.441 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.442 DEBUG:boto:Params: {} 2023-10-18T20:50:27.442 DEBUG:boto:Token: None 2023-10-18T20:50:27.443 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11 2023-10-18T20:50:27.443 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PbIUwP0OkeE3D4+mTStHANI7PTE= 2023-10-18T20:50:27.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PbIUwP0OkeE3D4+mTStHANI7PTE=', 'Content-Length': '0'} 2023-10-18T20:50:27.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ece9daf0270585-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.446 DEBUG:boto:path=/bofrnf-60/obj11 2023-10-18T20:50:27.446 DEBUG:boto:auth_path=/bofrnf-60/obj11 2023-10-18T20:50:27.447 DEBUG:boto:Method: GET 2023-10-18T20:50:27.447 DEBUG:boto:Path: /bofrnf-60/obj11 2023-10-18T20:50:27.448 DEBUG:boto:Data: 2023-10-18T20:50:27.448 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.448 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.449 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.449 DEBUG:boto:Params: {} 2023-10-18T20:50:27.450 DEBUG:boto:Token: None 2023-10-18T20:50:27.450 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj11 2023-10-18T20:50:27.451 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PbIUwP0OkeE3D4+mTStHANI7PTE= 2023-10-18T20:50:27.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PbIUwP0OkeE3D4+mTStHANI7PTE=', 'Content-Length': '0'} 2023-10-18T20:50:27.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f408e38b1705b3a1-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.454 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.454 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.455 DEBUG:boto:path=/bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.455 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.456 DEBUG:boto:Method: GET 2023-10-18T20:50:27.456 DEBUG:boto:Path: /bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.456 DEBUG:boto:Data: 2023-10-18T20:50:27.457 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.457 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.458 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.458 DEBUG:boto:Params: {} 2023-10-18T20:50:27.459 DEBUG:boto:Token: None 2023-10-18T20:50:27.459 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202d03d3d5f8f9ef-0065304513-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.463 DEBUG:boto:b'bofrnf-60obj111000falseobj112023-10-18T20:47:25.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.463 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.464 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.464 DEBUG:boto:path=/bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.465 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.465 DEBUG:boto:Method: GET 2023-10-18T20:50:27.466 DEBUG:boto:Path: /bofrnf-60/?prefix=obj11 2023-10-18T20:50:27.466 DEBUG:boto:Data: 2023-10-18T20:50:27.466 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.467 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.467 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.468 DEBUG:boto:Params: {} 2023-10-18T20:50:27.468 DEBUG:boto:Token: None 2023-10-18T20:50:27.469 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000750e711eba115046-0065304513-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.473 DEBUG:boto:b'bofrnf-60obj111000falseobj112023-10-18T20:47:25.666Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:50:27.474 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.474 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.475 DEBUG:boto:path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.475 DEBUG:boto:auth_path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.476 DEBUG:boto:Method: GET 2023-10-18T20:50:27.476 DEBUG:boto:Path: /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.476 DEBUG:boto:Data: 2023-10-18T20:50:27.477 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.477 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.478 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.478 DEBUG:boto:Params: {} 2023-10-18T20:50:27.479 DEBUG:boto:Token: None 2023-10-18T20:50:27.479 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.480 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MT4VtTtYkGKzhUhJYx0uZklQFOo= 2023-10-18T20:50:27.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MT4VtTtYkGKzhUhJYx0uZklQFOo=', 'Content-Length': '0'} 2023-10-18T20:50:27.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f609e5552f3ad9b9-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.526 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.527 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.528 DEBUG:boto:path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.528 DEBUG:boto:auth_path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.528 DEBUG:boto:Method: GET 2023-10-18T20:50:27.529 DEBUG:boto:Path: /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.529 DEBUG:boto:Data: 2023-10-18T20:50:27.530 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.531 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.531 DEBUG:boto:Params: {} 2023-10-18T20:50:27.532 DEBUG:boto:Token: None 2023-10-18T20:50:27.532 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.533 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MT4VtTtYkGKzhUhJYx0uZklQFOo= 2023-10-18T20:50:27.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MT4VtTtYkGKzhUhJYx0uZklQFOo=', 'Content-Length': '0'} 2023-10-18T20:50:27.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd932205b6f6db9-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.536 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.536 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.536 DEBUG:boto:path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.537 DEBUG:boto:auth_path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.537 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.537 DEBUG:boto:Path: /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.538 DEBUG:boto:Data: 2023-10-18T20:50:27.538 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.539 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.539 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.539 DEBUG:boto:Params: {} 2023-10-18T20:50:27.540 DEBUG:boto:Token: None 2023-10-18T20:50:27.540 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.541 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YvlFutrS/rGEYXcQF9N60gyBZ88= 2023-10-18T20:50:27.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YvlFutrS/rGEYXcQF9N60gyBZ88=', 'Content-Length': '0'} 2023-10-18T20:50:27.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000698d5cdc5963201a-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.544 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.544 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.544 DEBUG:boto:path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.545 DEBUG:boto:auth_path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.545 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.546 DEBUG:boto:Path: /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.546 DEBUG:boto:Data: 2023-10-18T20:50:27.546 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.547 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.547 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.548 DEBUG:boto:Params: {} 2023-10-18T20:50:27.548 DEBUG:boto:Token: None 2023-10-18T20:50:27.549 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.549 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YvlFutrS/rGEYXcQF9N60gyBZ88= 2023-10-18T20:50:27.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YvlFutrS/rGEYXcQF9N60gyBZ88=', 'Content-Length': '0'} 2023-10-18T20:50:27.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d209ccc23aa01fa8-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:50:27.552 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.553 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.553 DEBUG:boto:path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.553 DEBUG:boto:auth_path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.554 DEBUG:boto:Method: GET 2023-10-18T20:50:27.554 DEBUG:boto:Path: /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.555 DEBUG:boto:Data: 2023-10-18T20:50:27.555 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.556 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.556 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.556 DEBUG:boto:Params: {} 2023-10-18T20:50:27.557 DEBUG:boto:Token: None 2023-10-18T20:50:27.557 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.558 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MT4VtTtYkGKzhUhJYx0uZklQFOo= 2023-10-18T20:50:27.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MT4VtTtYkGKzhUhJYx0uZklQFOo=', 'Content-Length': '0'} 2023-10-18T20:50:27.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053c99dbe6401711b-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.561 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.561 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.562 DEBUG:boto:path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.562 DEBUG:boto:auth_path=/bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.563 DEBUG:boto:Method: GET 2023-10-18T20:50:27.563 DEBUG:boto:Path: /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.563 DEBUG:boto:Data: 2023-10-18T20:50:27.564 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.564 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.565 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.565 DEBUG:boto:Params: {} 2023-10-18T20:50:27.565 DEBUG:boto:Token: None 2023-10-18T20:50:27.566 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22?versionId=null 2023-10-18T20:50:27.566 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MT4VtTtYkGKzhUhJYx0uZklQFOo= 2023-10-18T20:50:27.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MT4VtTtYkGKzhUhJYx0uZklQFOo=', 'Content-Length': '0'} 2023-10-18T20:50:27.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33c85d067fb76df-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.610 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.611 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.611 DEBUG:boto:path=/bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.612 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.612 DEBUG:boto:Method: GET 2023-10-18T20:50:27.613 DEBUG:boto:Path: /bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.613 DEBUG:boto:Data: 2023-10-18T20:50:27.614 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.614 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.615 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.615 DEBUG:boto:Params: {} 2023-10-18T20:50:27.615 DEBUG:boto:Token: None 2023-10-18T20:50:27.616 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.617 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e0a94707904ca12-0065304513-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.620 DEBUG:boto:b'bofrnf-60obj221000falseobj222023-10-18T20:47:25.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.621 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.621 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.622 DEBUG:boto:path=/bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.622 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.622 DEBUG:boto:Method: GET 2023-10-18T20:50:27.623 DEBUG:boto:Path: /bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.623 DEBUG:boto:Data: 2023-10-18T20:50:27.624 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.624 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.625 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.625 DEBUG:boto:Params: {} 2023-10-18T20:50:27.625 DEBUG:boto:Token: None 2023-10-18T20:50:27.626 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.626 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e353549d5a6687b1-0065304513-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.630 DEBUG:boto:b'bofrnf-60obj221000falseobj222023-10-18T20:47:25.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.631 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.631 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.631 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.632 DEBUG:boto:Path: /bofrnf-60/obj22 2023-10-18T20:50:27.632 DEBUG:boto:Data: 2023-10-18T20:50:27.633 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.633 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.634 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.634 DEBUG:boto:Params: {} 2023-10-18T20:50:27.634 DEBUG:boto:Token: None 2023-10-18T20:50:27.635 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22 2023-10-18T20:50:27.636 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pWHcOTjtEdhkYqtUtD9asj4XWIQ= 2023-10-18T20:50:27.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pWHcOTjtEdhkYqtUtD9asj4XWIQ=', 'Content-Length': '0'} 2023-10-18T20:50:27.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd70e26ac6a79b8-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.638 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.639 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.639 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.639 DEBUG:boto:Path: /bofrnf-60/obj22 2023-10-18T20:50:27.640 DEBUG:boto:Data: 2023-10-18T20:50:27.640 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.641 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.641 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.641 DEBUG:boto:Params: {} 2023-10-18T20:50:27.642 DEBUG:boto:Token: None 2023-10-18T20:50:27.642 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22 2023-10-18T20:50:27.643 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pWHcOTjtEdhkYqtUtD9asj4XWIQ= 2023-10-18T20:50:27.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pWHcOTjtEdhkYqtUtD9asj4XWIQ=', 'Content-Length': '0'} 2023-10-18T20:50:27.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46fa637ca577a75-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:50:27.646 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.647 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.647 DEBUG:boto:Method: GET 2023-10-18T20:50:27.648 DEBUG:boto:Path: /bofrnf-60/obj22 2023-10-18T20:50:27.648 DEBUG:boto:Data: 2023-10-18T20:50:27.648 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.649 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.650 DEBUG:boto:Params: {} 2023-10-18T20:50:27.650 DEBUG:boto:Token: None 2023-10-18T20:50:27.651 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22 2023-10-18T20:50:27.651 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9R2n9w6px48JJ+q8p9NwQNvg3GQ= 2023-10-18T20:50:27.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9R2n9w6px48JJ+q8p9NwQNvg3GQ=', 'Content-Length': '0'} 2023-10-18T20:50:27.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000757f853638092605-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.694 DEBUG:boto:path=/bofrnf-60/obj22 2023-10-18T20:50:27.695 DEBUG:boto:auth_path=/bofrnf-60/obj22 2023-10-18T20:50:27.695 DEBUG:boto:Method: GET 2023-10-18T20:50:27.696 DEBUG:boto:Path: /bofrnf-60/obj22 2023-10-18T20:50:27.696 DEBUG:boto:Data: 2023-10-18T20:50:27.697 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.697 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.697 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.698 DEBUG:boto:Params: {} 2023-10-18T20:50:27.698 DEBUG:boto:Token: None 2023-10-18T20:50:27.699 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj22 2023-10-18T20:50:27.699 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:9R2n9w6px48JJ+q8p9NwQNvg3GQ= 2023-10-18T20:50:27.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:9R2n9w6px48JJ+q8p9NwQNvg3GQ=', 'Content-Length': '0'} 2023-10-18T20:50:27.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035085c861bc40a6d-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.702 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.703 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.703 DEBUG:boto:path=/bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.704 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.704 DEBUG:boto:Method: GET 2023-10-18T20:50:27.705 DEBUG:boto:Path: /bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.705 DEBUG:boto:Data: 2023-10-18T20:50:27.706 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.707 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.707 DEBUG:boto:Params: {} 2023-10-18T20:50:27.707 DEBUG:boto:Token: None 2023-10-18T20:50:27.708 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.708 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b40282085146b5ca-0065304513-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.711 DEBUG:boto:b'bofrnf-60obj221000falseobj222023-10-18T20:47:25.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.712 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.713 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.713 DEBUG:boto:path=/bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.714 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.714 DEBUG:boto:Method: GET 2023-10-18T20:50:27.714 DEBUG:boto:Path: /bofrnf-60/?prefix=obj22 2023-10-18T20:50:27.715 DEBUG:boto:Data: 2023-10-18T20:50:27.715 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.716 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.716 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.717 DEBUG:boto:Params: {} 2023-10-18T20:50:27.717 DEBUG:boto:Token: None 2023-10-18T20:50:27.718 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.718 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c67107b9b3a38428-0065304513-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.721 DEBUG:boto:b'bofrnf-60obj221000falseobj222023-10-18T20:47:25.686Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:50:27.723 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.723 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.723 DEBUG:boto:path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.724 DEBUG:boto:auth_path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.724 DEBUG:boto:Method: GET 2023-10-18T20:50:27.725 DEBUG:boto:Path: /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.725 DEBUG:boto:Data: 2023-10-18T20:50:27.726 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.726 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.726 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.727 DEBUG:boto:Params: {} 2023-10-18T20:50:27.727 DEBUG:boto:Token: None 2023-10-18T20:50:27.728 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0/nKLZLdAfpX/SzoSc1ylRmNJgI= 2023-10-18T20:50:27.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0/nKLZLdAfpX/SzoSc1ylRmNJgI=', 'Content-Length': '0'} 2023-10-18T20:50:27.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003133c135b4d4efba-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.731 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.731 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.732 DEBUG:boto:path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.732 DEBUG:boto:auth_path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.733 DEBUG:boto:Method: GET 2023-10-18T20:50:27.733 DEBUG:boto:Path: /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.734 DEBUG:boto:Data: 2023-10-18T20:50:27.734 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.735 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.735 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.736 DEBUG:boto:Params: {} 2023-10-18T20:50:27.736 DEBUG:boto:Token: None 2023-10-18T20:50:27.737 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.737 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0/nKLZLdAfpX/SzoSc1ylRmNJgI= 2023-10-18T20:50:27.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0/nKLZLdAfpX/SzoSc1ylRmNJgI=', 'Content-Length': '0'} 2023-10-18T20:50:27.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ee8af22a10b444-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.740 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.741 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.741 DEBUG:boto:path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.741 DEBUG:boto:auth_path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.742 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.742 DEBUG:boto:Path: /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.743 DEBUG:boto:Data: 2023-10-18T20:50:27.743 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.743 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.744 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.744 DEBUG:boto:Params: {} 2023-10-18T20:50:27.745 DEBUG:boto:Token: None 2023-10-18T20:50:27.745 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.746 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SdMQaW5nCbFIVhY4meZRPlQk63o= 2023-10-18T20:50:27.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SdMQaW5nCbFIVhY4meZRPlQk63o=', 'Content-Length': '0'} 2023-10-18T20:50:27.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc78052585570f8f-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.748 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.749 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.749 DEBUG:boto:path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.750 DEBUG:boto:auth_path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.750 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.750 DEBUG:boto:Path: /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.751 DEBUG:boto:Data: 2023-10-18T20:50:27.751 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.752 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.752 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.753 DEBUG:boto:Params: {} 2023-10-18T20:50:27.753 DEBUG:boto:Token: None 2023-10-18T20:50:27.754 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.754 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SdMQaW5nCbFIVhY4meZRPlQk63o= 2023-10-18T20:50:27.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SdMQaW5nCbFIVhY4meZRPlQk63o=', 'Content-Length': '0'} 2023-10-18T20:50:27.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc0ce75f764d193-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:50:27.757 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.758 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.758 DEBUG:boto:path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.759 DEBUG:boto:auth_path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.759 DEBUG:boto:Method: GET 2023-10-18T20:50:27.759 DEBUG:boto:Path: /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.760 DEBUG:boto:Data: 2023-10-18T20:50:27.760 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.761 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.761 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.762 DEBUG:boto:Params: {} 2023-10-18T20:50:27.762 DEBUG:boto:Token: None 2023-10-18T20:50:27.763 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.763 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0/nKLZLdAfpX/SzoSc1ylRmNJgI= 2023-10-18T20:50:27.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0/nKLZLdAfpX/SzoSc1ylRmNJgI=', 'Content-Length': '0'} 2023-10-18T20:50:27.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086d494b1418be401-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.810 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.811 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.811 DEBUG:boto:path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.812 DEBUG:boto:auth_path=/bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.812 DEBUG:boto:Method: GET 2023-10-18T20:50:27.813 DEBUG:boto:Path: /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.813 DEBUG:boto:Data: 2023-10-18T20:50:27.814 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.814 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.815 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.815 DEBUG:boto:Params: {} 2023-10-18T20:50:27.816 DEBUG:boto:Token: None 2023-10-18T20:50:27.816 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33?versionId=null 2023-10-18T20:50:27.817 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0/nKLZLdAfpX/SzoSc1ylRmNJgI= 2023-10-18T20:50:27.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0/nKLZLdAfpX/SzoSc1ylRmNJgI=', 'Content-Length': '0'} 2023-10-18T20:50:27.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8702e191807dec8-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.820 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.820 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.821 DEBUG:boto:path=/bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.821 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.821 DEBUG:boto:Method: GET 2023-10-18T20:50:27.822 DEBUG:boto:Path: /bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.822 DEBUG:boto:Data: 2023-10-18T20:50:27.823 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.823 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.824 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.824 DEBUG:boto:Params: {} 2023-10-18T20:50:27.825 DEBUG:boto:Token: None 2023-10-18T20:50:27.825 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.826 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b97045860e0a7622-0065304513-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.829 DEBUG:boto:b'bofrnf-60obj331000falseobj332023-10-18T20:47:25.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.830 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.831 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.831 DEBUG:boto:path=/bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.832 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.833 DEBUG:boto:Method: GET 2023-10-18T20:50:27.833 DEBUG:boto:Path: /bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.834 DEBUG:boto:Data: 2023-10-18T20:50:27.834 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.835 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.835 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.836 DEBUG:boto:Params: {} 2023-10-18T20:50:27.836 DEBUG:boto:Token: None 2023-10-18T20:50:27.837 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.837 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dc304f225e8483f-0065304513-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.841 DEBUG:boto:b'bofrnf-60obj331000falseobj332023-10-18T20:47:25.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.841 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.842 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.842 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.843 DEBUG:boto:Path: /bofrnf-60/obj33 2023-10-18T20:50:27.843 DEBUG:boto:Data: 2023-10-18T20:50:27.843 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.844 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.844 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.845 DEBUG:boto:Params: {} 2023-10-18T20:50:27.845 DEBUG:boto:Token: None 2023-10-18T20:50:27.846 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33 2023-10-18T20:50:27.846 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HQMRQpPMIq2VYxGE3Gj8YYTpz28= 2023-10-18T20:50:27.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HQMRQpPMIq2VYxGE3Gj8YYTpz28=', 'Content-Length': '0'} 2023-10-18T20:50:27.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd80a5651bd69601-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.849 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.849 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.850 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.850 DEBUG:boto:Path: /bofrnf-60/obj33 2023-10-18T20:50:27.851 DEBUG:boto:Data: 2023-10-18T20:50:27.851 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.852 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.852 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.852 DEBUG:boto:Params: {} 2023-10-18T20:50:27.853 DEBUG:boto:Token: None 2023-10-18T20:50:27.853 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33 2023-10-18T20:50:27.854 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HQMRQpPMIq2VYxGE3Gj8YYTpz28= 2023-10-18T20:50:27.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HQMRQpPMIq2VYxGE3Gj8YYTpz28=', 'Content-Length': '0'} 2023-10-18T20:50:27.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003040ceaf9c804205-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:50:27.857 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.857 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.858 DEBUG:boto:Method: GET 2023-10-18T20:50:27.858 DEBUG:boto:Path: /bofrnf-60/obj33 2023-10-18T20:50:27.859 DEBUG:boto:Data: 2023-10-18T20:50:27.859 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.860 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.860 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.861 DEBUG:boto:Params: {} 2023-10-18T20:50:27.861 DEBUG:boto:Token: None 2023-10-18T20:50:27.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33 2023-10-18T20:50:27.862 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JUmPNnLKQycLwyk7O8ZaRfGTJ/s= 2023-10-18T20:50:27.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JUmPNnLKQycLwyk7O8ZaRfGTJ/s=', 'Content-Length': '0'} 2023-10-18T20:50:27.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcecfd27e20f748a-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.865 DEBUG:boto:path=/bofrnf-60/obj33 2023-10-18T20:50:27.865 DEBUG:boto:auth_path=/bofrnf-60/obj33 2023-10-18T20:50:27.866 DEBUG:boto:Method: GET 2023-10-18T20:50:27.866 DEBUG:boto:Path: /bofrnf-60/obj33 2023-10-18T20:50:27.867 DEBUG:boto:Data: 2023-10-18T20:50:27.870 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.870 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.870 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.871 DEBUG:boto:Params: {} 2023-10-18T20:50:27.871 DEBUG:boto:Token: None 2023-10-18T20:50:27.872 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj33 2023-10-18T20:50:27.872 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JUmPNnLKQycLwyk7O8ZaRfGTJ/s= 2023-10-18T20:50:27.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JUmPNnLKQycLwyk7O8ZaRfGTJ/s=', 'Content-Length': '0'} 2023-10-18T20:50:27.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072e4583c8f2a3514-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.876 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.876 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.877 DEBUG:boto:path=/bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.877 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.878 DEBUG:boto:Method: GET 2023-10-18T20:50:27.878 DEBUG:boto:Path: /bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.878 DEBUG:boto:Data: 2023-10-18T20:50:27.879 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.879 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.880 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.880 DEBUG:boto:Params: {} 2023-10-18T20:50:27.881 DEBUG:boto:Token: None 2023-10-18T20:50:27.881 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.882 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c43de47265e9ee0-0065304513-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.885 DEBUG:boto:b'bofrnf-60obj331000falseobj332023-10-18T20:47:25.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.885 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.886 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.886 DEBUG:boto:path=/bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.887 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.887 DEBUG:boto:Method: GET 2023-10-18T20:50:27.888 DEBUG:boto:Path: /bofrnf-60/?prefix=obj33 2023-10-18T20:50:27.888 DEBUG:boto:Data: 2023-10-18T20:50:27.888 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.889 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.889 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.890 DEBUG:boto:Params: {} 2023-10-18T20:50:27.890 DEBUG:boto:Token: None 2023-10-18T20:50:27.891 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.891 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:27.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f29a416f101e752-0065304513-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.938 DEBUG:boto:b'bofrnf-60obj331000falseobj332023-10-18T20:47:25.742Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:27.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:50:27.940 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:27.940 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:27.941 DEBUG:boto:path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.941 DEBUG:boto:auth_path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.941 DEBUG:boto:Method: GET 2023-10-18T20:50:27.942 DEBUG:boto:Path: /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.942 DEBUG:boto:Data: 2023-10-18T20:50:27.943 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.943 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.944 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.944 DEBUG:boto:Params: {} 2023-10-18T20:50:27.945 DEBUG:boto:Token: None 2023-10-18T20:50:27.945 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.946 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DIiZ8JATRbfcnBfmGTddz8vlvhc= 2023-10-18T20:50:27.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DIiZ8JATRbfcnBfmGTddz8vlvhc=', 'Content-Length': '0'} 2023-10-18T20:50:27.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b41d58a714de6bb-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.949 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:27.949 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:27.950 DEBUG:boto:path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.950 DEBUG:boto:auth_path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.951 DEBUG:boto:Method: GET 2023-10-18T20:50:27.951 DEBUG:boto:Path: /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.951 DEBUG:boto:Data: 2023-10-18T20:50:27.952 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.952 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.953 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.953 DEBUG:boto:Params: {} 2023-10-18T20:50:27.954 DEBUG:boto:Token: None 2023-10-18T20:50:27.954 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.955 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DIiZ8JATRbfcnBfmGTddz8vlvhc= 2023-10-18T20:50:27.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DIiZ8JATRbfcnBfmGTddz8vlvhc=', 'Content-Length': '0'} 2023-10-18T20:50:27.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c91ab9524e91e678-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.958 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:27.958 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:27.959 DEBUG:boto:path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.959 DEBUG:boto:auth_path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.960 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.960 DEBUG:boto:Path: /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.960 DEBUG:boto:Data: 2023-10-18T20:50:27.961 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.961 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.961 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.962 DEBUG:boto:Params: {} 2023-10-18T20:50:27.962 DEBUG:boto:Token: None 2023-10-18T20:50:27.963 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.963 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gvHzi9iO6pPAh6G92ligKwq3rBk= 2023-10-18T20:50:27.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gvHzi9iO6pPAh6G92ligKwq3rBk=', 'Content-Length': '0'} 2023-10-18T20:50:27.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c4610f68706bf6-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.966 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:27.966 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:27.967 DEBUG:boto:path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.967 DEBUG:boto:auth_path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.968 DEBUG:boto:Method: HEAD 2023-10-18T20:50:27.968 DEBUG:boto:Path: /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.969 DEBUG:boto:Data: 2023-10-18T20:50:27.969 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.970 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.970 DEBUG:boto:Params: {} 2023-10-18T20:50:27.971 DEBUG:boto:Token: None 2023-10-18T20:50:27.971 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.972 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gvHzi9iO6pPAh6G92ligKwq3rBk= 2023-10-18T20:50:27.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gvHzi9iO6pPAh6G92ligKwq3rBk=', 'Content-Length': '0'} 2023-10-18T20:50:27.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004283670df7b93ede-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:50:27.975 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:27.975 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:27.976 DEBUG:boto:path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.976 DEBUG:boto:auth_path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.976 DEBUG:boto:Method: GET 2023-10-18T20:50:27.977 DEBUG:boto:Path: /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.977 DEBUG:boto:Data: 2023-10-18T20:50:27.978 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.978 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.979 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.979 DEBUG:boto:Params: {} 2023-10-18T20:50:27.979 DEBUG:boto:Token: None 2023-10-18T20:50:27.980 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.980 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DIiZ8JATRbfcnBfmGTddz8vlvhc= 2023-10-18T20:50:27.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DIiZ8JATRbfcnBfmGTddz8vlvhc=', 'Content-Length': '0'} 2023-10-18T20:50:27.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f17dfdfcc767abfa-0065304513-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.983 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:27.984 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:27.984 DEBUG:boto:path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.985 DEBUG:boto:auth_path=/bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.985 DEBUG:boto:Method: GET 2023-10-18T20:50:27.986 DEBUG:boto:Path: /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.986 DEBUG:boto:Data: 2023-10-18T20:50:27.987 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.987 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:27.987 DEBUG:boto:Port: 8000 2023-10-18T20:50:27.988 DEBUG:boto:Params: {} 2023-10-18T20:50:27.988 DEBUG:boto:Token: None 2023-10-18T20:50:27.989 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/obj44?versionId=null 2023-10-18T20:50:27.989 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DIiZ8JATRbfcnBfmGTddz8vlvhc= 2023-10-18T20:50:27.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DIiZ8JATRbfcnBfmGTddz8vlvhc=', 'Content-Length': '0'} 2023-10-18T20:50:27.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d053a3a504ee22df-0065304513-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:27.992 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:27.992 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:27.993 DEBUG:boto:path=/bofrnf-60/?prefix=obj44 2023-10-18T20:50:27.993 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj44 2023-10-18T20:50:27.994 DEBUG:boto:Method: GET 2023-10-18T20:50:27.994 DEBUG:boto:Path: /bofrnf-60/?prefix=obj44 2023-10-18T20:50:27.995 DEBUG:boto:Data: 2023-10-18T20:50:27.995 DEBUG:boto:Headers: {} 2023-10-18T20:50:27.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:27.996 DEBUG:boto:Port: 8001 2023-10-18T20:50:27.996 DEBUG:boto:Params: {} 2023-10-18T20:50:27.997 DEBUG:boto:Token: None 2023-10-18T20:50:27.997 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:27 GMT /bofrnf-60/ 2023-10-18T20:50:27.998 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg= 2023-10-18T20:50:27.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:27 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sJ9NB+7olZSSCItUoNJQsHMc0Hg=', 'Content-Length': '0'} 2023-10-18T20:50:28.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c755fcec9f70add-0065304513-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.001 DEBUG:boto:b'bofrnf-60obj441000falseobj442023-10-18T20:47:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:28.002 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:28.002 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:28.003 DEBUG:boto:path=/bofrnf-60/?prefix=obj44 2023-10-18T20:50:28.003 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj44 2023-10-18T20:50:28.003 DEBUG:boto:Method: GET 2023-10-18T20:50:28.004 DEBUG:boto:Path: /bofrnf-60/?prefix=obj44 2023-10-18T20:50:28.004 DEBUG:boto:Data: 2023-10-18T20:50:28.005 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.005 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.006 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.006 DEBUG:boto:Params: {} 2023-10-18T20:50:28.007 DEBUG:boto:Token: None 2023-10-18T20:50:28.007 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/ 2023-10-18T20:50:28.008 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8= 2023-10-18T20:50:28.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8=', 'Content-Length': '0'} 2023-10-18T20:50:28.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000478a827c0ed07bd2-0065304514-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.011 DEBUG:boto:b'bofrnf-60obj441000falseobj442023-10-18T20:47:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:28.011 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:28.012 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:28.012 DEBUG:boto:Method: HEAD 2023-10-18T20:50:28.013 DEBUG:boto:Path: /bofrnf-60/obj44 2023-10-18T20:50:28.013 DEBUG:boto:Data: 2023-10-18T20:50:28.013 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.014 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.015 DEBUG:boto:Params: {} 2023-10-18T20:50:28.015 DEBUG:boto:Token: None 2023-10-18T20:50:28.016 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj44 2023-10-18T20:50:28.016 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fTwyV91jEqmmWszv7byVfhYf08s= 2023-10-18T20:50:28.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fTwyV91jEqmmWszv7byVfhYf08s=', 'Content-Length': '0'} 2023-10-18T20:50:28.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000861d883fe146c67-0065304514-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.019 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:28.019 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:28.020 DEBUG:boto:Method: HEAD 2023-10-18T20:50:28.020 DEBUG:boto:Path: /bofrnf-60/obj44 2023-10-18T20:50:28.020 DEBUG:boto:Data: 2023-10-18T20:50:28.021 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.021 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.022 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.022 DEBUG:boto:Params: {} 2023-10-18T20:50:28.023 DEBUG:boto:Token: None 2023-10-18T20:50:28.023 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj44 2023-10-18T20:50:28.024 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fTwyV91jEqmmWszv7byVfhYf08s= 2023-10-18T20:50:28.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fTwyV91jEqmmWszv7byVfhYf08s=', 'Content-Length': '0'} 2023-10-18T20:50:28.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e61a41887323891e-0065304514-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:50:28.027 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:28.027 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:28.028 DEBUG:boto:Method: GET 2023-10-18T20:50:28.028 DEBUG:boto:Path: /bofrnf-60/obj44 2023-10-18T20:50:28.029 DEBUG:boto:Data: 2023-10-18T20:50:28.029 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.029 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.030 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.030 DEBUG:boto:Params: {} 2023-10-18T20:50:28.031 DEBUG:boto:Token: None 2023-10-18T20:50:28.031 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj44 2023-10-18T20:50:28.032 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g0n8ACCO8DXac5qsh9r4N6EFBpI= 2023-10-18T20:50:28.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g0n8ACCO8DXac5qsh9r4N6EFBpI=', 'Content-Length': '0'} 2023-10-18T20:50:28.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aa2de664ae4832a-0065304514-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.082 DEBUG:boto:path=/bofrnf-60/obj44 2023-10-18T20:50:28.083 DEBUG:boto:auth_path=/bofrnf-60/obj44 2023-10-18T20:50:28.084 DEBUG:boto:Method: GET 2023-10-18T20:50:28.084 DEBUG:boto:Path: /bofrnf-60/obj44 2023-10-18T20:50:28.084 DEBUG:boto:Data: 2023-10-18T20:50:28.085 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.085 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.086 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.086 DEBUG:boto:Params: {} 2023-10-18T20:50:28.087 DEBUG:boto:Token: None 2023-10-18T20:50:28.087 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj44 2023-10-18T20:50:28.088 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g0n8ACCO8DXac5qsh9r4N6EFBpI= 2023-10-18T20:50:28.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g0n8ACCO8DXac5qsh9r4N6EFBpI=', 'Content-Length': '0'} 2023-10-18T20:50:28.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a65a5d685410d01-0065304514-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.091 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:28.091 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:28.092 DEBUG:boto:path=/bofrnf-60/?prefix=obj44 2023-10-18T20:50:28.092 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj44 2023-10-18T20:50:28.093 DEBUG:boto:Method: GET 2023-10-18T20:50:28.093 DEBUG:boto:Path: /bofrnf-60/?prefix=obj44 2023-10-18T20:50:28.094 DEBUG:boto:Data: 2023-10-18T20:50:28.094 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.094 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.095 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.095 DEBUG:boto:Params: {} 2023-10-18T20:50:28.096 DEBUG:boto:Token: None 2023-10-18T20:50:28.096 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/ 2023-10-18T20:50:28.097 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8= 2023-10-18T20:50:28.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8=', 'Content-Length': '0'} 2023-10-18T20:50:28.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac50b9091e0a660-0065304514-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.100 DEBUG:boto:b'bofrnf-60obj441000falseobj442023-10-18T20:47:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:28.101 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:28.101 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:28.102 DEBUG:boto:path=/bofrnf-60/?prefix=obj44 2023-10-18T20:50:28.102 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj44 2023-10-18T20:50:28.103 DEBUG:boto:Method: GET 2023-10-18T20:50:28.103 DEBUG:boto:Path: /bofrnf-60/?prefix=obj44 2023-10-18T20:50:28.104 DEBUG:boto:Data: 2023-10-18T20:50:28.104 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.104 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.105 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.105 DEBUG:boto:Params: {} 2023-10-18T20:50:28.106 DEBUG:boto:Token: None 2023-10-18T20:50:28.106 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/ 2023-10-18T20:50:28.107 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8= 2023-10-18T20:50:28.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8=', 'Content-Length': '0'} 2023-10-18T20:50:28.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050a7b633a2b85eb6-0065304514-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.150 DEBUG:boto:b'bofrnf-60obj441000falseobj442023-10-18T20:47:25.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:28.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:50:28.152 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.152 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.152 DEBUG:boto:path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.153 DEBUG:boto:auth_path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.153 DEBUG:boto:Method: GET 2023-10-18T20:50:28.154 DEBUG:boto:Path: /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.154 DEBUG:boto:Data: 2023-10-18T20:50:28.155 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.155 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.156 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.156 DEBUG:boto:Params: {} 2023-10-18T20:50:28.156 DEBUG:boto:Token: None 2023-10-18T20:50:28.157 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.157 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8DtRgOqNg8eFhvDvKPSOTeMUnyI= 2023-10-18T20:50:28.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8DtRgOqNg8eFhvDvKPSOTeMUnyI=', 'Content-Length': '0'} 2023-10-18T20:50:28.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004cbccdbacbdd084-0065304514-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.161 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.161 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.162 DEBUG:boto:path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.162 DEBUG:boto:auth_path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.163 DEBUG:boto:Method: GET 2023-10-18T20:50:28.163 DEBUG:boto:Path: /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.163 DEBUG:boto:Data: 2023-10-18T20:50:28.164 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.164 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.165 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.165 DEBUG:boto:Params: {} 2023-10-18T20:50:28.166 DEBUG:boto:Token: None 2023-10-18T20:50:28.166 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.167 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8DtRgOqNg8eFhvDvKPSOTeMUnyI= 2023-10-18T20:50:28.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8DtRgOqNg8eFhvDvKPSOTeMUnyI=', 'Content-Length': '0'} 2023-10-18T20:50:28.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b30bb13c3d15f5d5-0065304514-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.170 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.170 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.171 DEBUG:boto:path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.171 DEBUG:boto:auth_path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.172 DEBUG:boto:Method: HEAD 2023-10-18T20:50:28.172 DEBUG:boto:Path: /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.172 DEBUG:boto:Data: 2023-10-18T20:50:28.173 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.173 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.174 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.174 DEBUG:boto:Params: {} 2023-10-18T20:50:28.174 DEBUG:boto:Token: None 2023-10-18T20:50:28.175 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.175 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VflwvgHHADBDDdi1+FVHzhaNqi4= 2023-10-18T20:50:28.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VflwvgHHADBDDdi1+FVHzhaNqi4=', 'Content-Length': '0'} 2023-10-18T20:50:28.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046750df0c3f967e2-0065304514-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.178 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.179 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.179 DEBUG:boto:path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.180 DEBUG:boto:auth_path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.180 DEBUG:boto:Method: HEAD 2023-10-18T20:50:28.180 DEBUG:boto:Path: /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.181 DEBUG:boto:Data: 2023-10-18T20:50:28.181 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.182 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.182 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.182 DEBUG:boto:Params: {} 2023-10-18T20:50:28.183 DEBUG:boto:Token: None 2023-10-18T20:50:28.183 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.184 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VflwvgHHADBDDdi1+FVHzhaNqi4= 2023-10-18T20:50:28.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VflwvgHHADBDDdi1+FVHzhaNqi4=', 'Content-Length': '0'} 2023-10-18T20:50:28.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aa5f62da0f416e0-0065304514-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:50:28.187 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.187 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.188 DEBUG:boto:path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.188 DEBUG:boto:auth_path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.188 DEBUG:boto:Method: GET 2023-10-18T20:50:28.189 DEBUG:boto:Path: /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.189 DEBUG:boto:Data: 2023-10-18T20:50:28.190 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.191 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.191 DEBUG:boto:Params: {} 2023-10-18T20:50:28.192 DEBUG:boto:Token: None 2023-10-18T20:50:28.192 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.193 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8DtRgOqNg8eFhvDvKPSOTeMUnyI= 2023-10-18T20:50:28.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8DtRgOqNg8eFhvDvKPSOTeMUnyI=', 'Content-Length': '0'} 2023-10-18T20:50:28.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea1eed24004dbab0-0065304514-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.195 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.196 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.196 DEBUG:boto:path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.197 DEBUG:boto:auth_path=/bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.197 DEBUG:boto:Method: GET 2023-10-18T20:50:28.198 DEBUG:boto:Path: /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.198 DEBUG:boto:Data: 2023-10-18T20:50:28.199 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.199 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.200 DEBUG:boto:Params: {} 2023-10-18T20:50:28.200 DEBUG:boto:Token: None 2023-10-18T20:50:28.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55?versionId=null 2023-10-18T20:50:28.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8DtRgOqNg8eFhvDvKPSOTeMUnyI= 2023-10-18T20:50:28.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8DtRgOqNg8eFhvDvKPSOTeMUnyI=', 'Content-Length': '0'} 2023-10-18T20:50:28.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4237eb67c1d6e57-0065304514-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.204 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:28.205 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:28.205 DEBUG:boto:path=/bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.206 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.206 DEBUG:boto:Method: GET 2023-10-18T20:50:28.207 DEBUG:boto:Path: /bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.207 DEBUG:boto:Data: 2023-10-18T20:50:28.207 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.208 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.208 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.209 DEBUG:boto:Params: {} 2023-10-18T20:50:28.209 DEBUG:boto:Token: None 2023-10-18T20:50:28.210 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/ 2023-10-18T20:50:28.210 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8= 2023-10-18T20:50:28.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8=', 'Content-Length': '0'} 2023-10-18T20:50:28.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3fdf748fcf7a282-0065304514-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.214 DEBUG:boto:b'bofrnf-60obj551000falseobj552023-10-18T20:47:25.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:28.215 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:28.215 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:28.215 DEBUG:boto:path=/bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.216 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.216 DEBUG:boto:Method: GET 2023-10-18T20:50:28.217 DEBUG:boto:Path: /bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.217 DEBUG:boto:Data: 2023-10-18T20:50:28.217 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.218 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.218 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.219 DEBUG:boto:Params: {} 2023-10-18T20:50:28.219 DEBUG:boto:Token: None 2023-10-18T20:50:28.220 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/ 2023-10-18T20:50:28.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8= 2023-10-18T20:50:28.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8=', 'Content-Length': '0'} 2023-10-18T20:50:28.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075ffe6470ed0a6cd-0065304514-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.223 DEBUG:boto:b'bofrnf-60obj551000falseobj552023-10-18T20:47:25.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:28.224 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.225 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.225 DEBUG:boto:Method: HEAD 2023-10-18T20:50:28.225 DEBUG:boto:Path: /bofrnf-60/obj55 2023-10-18T20:50:28.226 DEBUG:boto:Data: 2023-10-18T20:50:28.226 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.227 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.227 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.227 DEBUG:boto:Params: {} 2023-10-18T20:50:28.228 DEBUG:boto:Token: None 2023-10-18T20:50:28.228 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55 2023-10-18T20:50:28.229 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jlDXN6N3I7AhrC9tWGSS9uX8i88= 2023-10-18T20:50:28.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jlDXN6N3I7AhrC9tWGSS9uX8i88=', 'Content-Length': '0'} 2023-10-18T20:50:28.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029d8968deda6dd44-0065304514-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.232 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.232 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.233 DEBUG:boto:Method: HEAD 2023-10-18T20:50:28.233 DEBUG:boto:Path: /bofrnf-60/obj55 2023-10-18T20:50:28.234 DEBUG:boto:Data: 2023-10-18T20:50:28.234 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.234 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.235 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.235 DEBUG:boto:Params: {} 2023-10-18T20:50:28.236 DEBUG:boto:Token: None 2023-10-18T20:50:28.236 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55 2023-10-18T20:50:28.237 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jlDXN6N3I7AhrC9tWGSS9uX8i88= 2023-10-18T20:50:28.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jlDXN6N3I7AhrC9tWGSS9uX8i88=', 'Content-Length': '0'} 2023-10-18T20:50:28.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000081fabb60e060e97-0065304514-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:50:28.240 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.240 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.241 DEBUG:boto:Method: GET 2023-10-18T20:50:28.241 DEBUG:boto:Path: /bofrnf-60/obj55 2023-10-18T20:50:28.241 DEBUG:boto:Data: 2023-10-18T20:50:28.242 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.242 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.243 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.243 DEBUG:boto:Params: {} 2023-10-18T20:50:28.244 DEBUG:boto:Token: None 2023-10-18T20:50:28.244 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55 2023-10-18T20:50:28.244 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NPZJgP7xxsVSetjLV3JVfhlahfY= 2023-10-18T20:50:28.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NPZJgP7xxsVSetjLV3JVfhlahfY=', 'Content-Length': '0'} 2023-10-18T20:50:28.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-60:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d4cc033feb371a-0065304514-1e64-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.290 DEBUG:boto:path=/bofrnf-60/obj55 2023-10-18T20:50:28.291 DEBUG:boto:auth_path=/bofrnf-60/obj55 2023-10-18T20:50:28.291 DEBUG:boto:Method: GET 2023-10-18T20:50:28.292 DEBUG:boto:Path: /bofrnf-60/obj55 2023-10-18T20:50:28.292 DEBUG:boto:Data: 2023-10-18T20:50:28.293 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.293 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.294 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.294 DEBUG:boto:Params: {} 2023-10-18T20:50:28.295 DEBUG:boto:Token: None 2023-10-18T20:50:28.295 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/obj55 2023-10-18T20:50:28.296 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NPZJgP7xxsVSetjLV3JVfhlahfY= 2023-10-18T20:50:28.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NPZJgP7xxsVSetjLV3JVfhlahfY=', 'Content-Length': '0'} 2023-10-18T20:50:28.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:47:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1aaadbbd24081d6-0065304514-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.299 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:28.299 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:28.300 DEBUG:boto:path=/bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.300 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.301 DEBUG:boto:Method: GET 2023-10-18T20:50:28.301 DEBUG:boto:Path: /bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.301 DEBUG:boto:Data: 2023-10-18T20:50:28.302 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:28.303 DEBUG:boto:Port: 8001 2023-10-18T20:50:28.303 DEBUG:boto:Params: {} 2023-10-18T20:50:28.304 DEBUG:boto:Token: None 2023-10-18T20:50:28.304 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/ 2023-10-18T20:50:28.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8= 2023-10-18T20:50:28.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8=', 'Content-Length': '0'} 2023-10-18T20:50:28.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ee134d4cb23fe8c-0065304514-1e64-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.308 DEBUG:boto:b'bofrnf-60obj551000falseobj552023-10-18T20:47:25.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:28.309 DEBUG:boto:path=/bofrnf-60/ 2023-10-18T20:50:28.309 DEBUG:boto:auth_path=/bofrnf-60/ 2023-10-18T20:50:28.310 DEBUG:boto:path=/bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.310 DEBUG:boto:auth_path=/bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.311 DEBUG:boto:Method: GET 2023-10-18T20:50:28.311 DEBUG:boto:Path: /bofrnf-60/?prefix=obj55 2023-10-18T20:50:28.312 DEBUG:boto:Data: 2023-10-18T20:50:28.312 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.313 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.313 DEBUG:boto:Params: {} 2023-10-18T20:50:28.314 DEBUG:boto:Token: None 2023-10-18T20:50:28.314 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-60/ 2023-10-18T20:50:28.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8= 2023-10-18T20:50:28.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SW2/qKc5Ns3s0hS3vGQq3yhasr8=', 'Content-Length': '0'} 2023-10-18T20:50:28.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000634958ccbba5feb2-0065304514-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.358 DEBUG:boto:b'bofrnf-60obj551000falseobj552023-10-18T20:47:25.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:50:28.359 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-60 zones={a2, a1} 2023-10-18T20:50:28.360 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-18T20:50:28.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.361 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:50:28.410 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:28.411 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:28.458 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:50:28.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.458 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000026788", 2023-10-18T20:50:28.459 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:47:45.429112Z" 2023-10-18T20:50:28.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.459 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000025837", 2023-10-18T20:50:28.460 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:47:45.431157Z" 2023-10-18T20:50:28.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.461 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000019177", 2023-10-18T20:50:28.461 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:43:09.614764Z" 2023-10-18T20:50:28.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.461 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000030741", 2023-10-18T20:50:28.462 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:47:25.765336Z" 2023-10-18T20:50:28.462 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.462 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:50:28.463 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000026788', 1: '00000000000000000000:00000000000000025837', 2: '00000000000000000000:00000000000000019177', 3: '00000000000000000000:00000000000000030741'} 2023-10-18T20:50:28.464 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:50:28.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.465 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:50:28.517 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:28.517 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:28.562 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:50:28.563 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:50:28.563 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:50:28.563 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:50:28.563 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:50:28.564 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 10677745729068821240 2023-10-18T20:50:28.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.564 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:50:28.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.565 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:50:28.565 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:28.565 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:50:28.565 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000026788", 2023-10-18T20:50:28.566 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:28.566 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:28.566 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:28.566 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:47:45.429112Z" 2023-10-18T20:50:28.566 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.567 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:50:28.567 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:28.567 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:50:28.568 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000025837", 2023-10-18T20:50:28.568 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:28.568 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:28.568 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:28.568 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:47:45.431157Z" 2023-10-18T20:50:28.569 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.569 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:50:28.570 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:28.570 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:50:28.570 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000019177", 2023-10-18T20:50:28.570 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:28.570 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T20:50:28.571 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:28.571 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:43:09.614764Z" 2023-10-18T20:50:28.571 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.572 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:50:28.572 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:28.572 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:50:28.572 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000030741", 2023-10-18T20:50:28.572 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:28.572 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 51, 2023-10-18T20:50:28.573 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:28.573 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:47:25.765336Z" 2023-10-18T20:50:28.573 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.573 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.573 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:50:28.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.574 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:50:28.574 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 60, 2023-10-18T20:50:28.574 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 60 2023-10-18T20:50:28.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.575 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:50:28Z" 2023-10-18T20:50:28.575 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:50:28.576 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10677745729068821240 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026788", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:47:45.429112Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025837", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:47:45.431157Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019177", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T20:43:09.614764Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000030741", "next_step_marker": "", "total_entries": 51, "pos": 0, "timestamp": "2023-10-18T20:47:25.765336Z" } } ] }, "full_sync": { "total": 60, "complete": 60 }, "current_time": "2023-10-18T20:50:28Z" } 2023-10-18T20:50:28.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026788', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:47:45.429112Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025837', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:47:45.431157Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019177', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T20:43:09.614764Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000030741', 'next_step_marker': '', 'total_entries': 51, 'pos': 0, 'timestamp': '2023-10-18T20:47:25.765336Z'}}] 2023-10-18T20:50:28.577 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000026788', 1: '00000000000000000000:00000000000000025837', 2: '00000000000000000000:00000000000000019177', 3: '00000000000000000000:00000000000000030741'} 2023-10-18T20:50:28.577 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000026788', 1: '00000000000000000000:00000000000000025837', 2: '00000000000000000000:00000000000000019177', 3: '00000000000000000000:00000000000000030741'} 2023-10-18T20:50:28.578 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:50:28.578 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-18T20:50:28.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.579 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:50:28.628 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:28.628 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:28.675 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:50:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000012919", 2023-10-18T20:50:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:47:48.047258Z" 2023-10-18T20:50:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000010549", 2023-10-18T20:50:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:48:59.802755Z" 2023-10-18T20:50:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005302", 2023-10-18T20:50:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:40.611119Z" 2023-10-18T20:50:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000014463", 2023-10-18T20:50:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:47:29.302110Z" 2023-10-18T20:50:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.679 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:50:28.680 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000012919', 1: '00000000000000000000:00000000000000010549', 2: '00000000000000000000:00000000000000005302', 3: '00000000000000000000:00000000000000014463'} 2023-10-18T20:50:28.681 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:50:28.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.682 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:50:28.733 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:28.733 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:28.776 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:50:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:50:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:50:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:50:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:50:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:50:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:50:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:50:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:50:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000012919", 2023-10-18T20:50:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:28.780 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:28.780 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:28.780 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:47:48.047258Z" 2023-10-18T20:50:28.780 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.781 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:50:28.781 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:28.782 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:50:28.782 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000010549", 2023-10-18T20:50:28.782 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:28.782 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:28.783 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:28.783 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:48:59.802755Z" 2023-10-18T20:50:28.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.784 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:50:28.784 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:28.784 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:50:28.784 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005302", 2023-10-18T20:50:28.784 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:28.785 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:28.785 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:28.785 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:40.611119Z" 2023-10-18T20:50:28.785 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.786 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:50:28.786 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:28.786 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:50:28.787 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000014463", 2023-10-18T20:50:28.787 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:28.787 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:28.787 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:28.787 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:47:29.302110Z" 2023-10-18T20:50:28.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.788 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:50:28.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.788 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:50:28.789 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:50:28.789 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:50:28.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.789 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:50:28Z" 2023-10-18T20:50:28.789 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:50:28.791 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012919", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:47:48.047258Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010549", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:48:59.802755Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005302", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:40.611119Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014463", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:47:29.302110Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:50:28Z" } 2023-10-18T20:50:28.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012919', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:47:48.047258Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010549', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:48:59.802755Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005302', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:40.611119Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014463', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:47:29.302110Z'}}] 2023-10-18T20:50:28.792 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012919', 1: '00000000000000000000:00000000000000010549', 2: '00000000000000000000:00000000000000005302', 3: '00000000000000000000:00000000000000014463'} 2023-10-18T20:50:28.792 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012919', 1: '00000000000000000000:00000000000000010549', 2: '00000000000000000000:00000000000000005302', 3: '00000000000000000000:00000000000000014463'} 2023-10-18T20:50:28.793 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:50:28.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2023-10-18T20:50:28.795 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:28.795 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:28.796 DEBUG:boto:Method: PUT 2023-10-18T20:50:28.796 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:28.796 DEBUG:boto:Data: 2023-10-18T20:50:28.797 DEBUG:boto:Headers: {} 2023-10-18T20:50:28.797 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:28.798 DEBUG:boto:Port: 8000 2023-10-18T20:50:28.798 DEBUG:boto:Params: {} 2023-10-18T20:50:28.799 DEBUG:boto:Token: None 2023-10-18T20:50:28.799 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:50:28 GMT /bofrnf-61/ 2023-10-18T20:50:28.799 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:47AA5ASZ2oaCBXgRSh165N8FEFA= 2023-10-18T20:50:28.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:47AA5ASZ2oaCBXgRSh165N8FEFA=', 'Content-Length': '0'} 2023-10-18T20:50:28.819 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000079454eb1857653-0065304514-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:28.820 DEBUG:rgw_multi.tests:created bucket=bofrnf-61 2023-10-18T20:50:28.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.822 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:50:28.870 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:28.870 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:28.899 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:50:28.912 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:50:28.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.912 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662228.817590_1239.1", 2023-10-18T20:50:28.913 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:50:28.817590Z" 2023-10-18T20:50:28.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.913 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:50:28.913 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T20:50:28.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.914 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:50:28.914 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:50:28.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:28.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:28.915 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:50:28.915 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T20:50:28.915 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:28.916 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:50:28.916 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697662228.817590_1239.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:50:28.917 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:50:28.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:28.918 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:50:28.969 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:28.970 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:28.998 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:28.996+0000 7fbbd3950780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:50:28.999 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:28.997+0000 7fbbd3950780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:50:29.015 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:50:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:50:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:50:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:50:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:50:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:50:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:50:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:50:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:50:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662228.811524_1230.1", 2023-10-18T20:50:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:50:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:50:28.811524Z", 2023-10-18T20:50:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:50:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:50:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:50:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:50:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:50:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:50:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:50:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:50:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:50:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:50:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:50:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:50:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:50:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:50:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:50:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:50:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:50:28Z" 2023-10-18T20:50:29.029 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:50:29.030 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697662228.811524_1230.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:50:28.811524Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:50:28Z" } 2023-10-18T20:50:29.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697662228.811524_1230.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:50:28.811524Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:50:29.031 DEBUG:rgw_multi.tests:log_status={0: '1_1697662228.817590_1239.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:50:29.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1697662228.811524_1230.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:50:29.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697662228.817590_1239.1 target=1_1697662228.811524_1230.1 2023-10-18T20:50:34.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:34.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:50:34.034 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:50:34.091 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:34.092 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:34.123 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:34.122+0000 7f16dd5d9780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:50:34.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:34.123+0000 7f16dd5d9780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:50:34.139 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:50:34.139 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:50:34.140 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:50:34.140 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:50:34.140 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:50:34.140 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:50:34.140 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:34.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:34.141 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:50:34.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:34.141 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:50:34.142 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:34.142 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:50:34.142 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662228.817590_1239.1", 2023-10-18T20:50:34.142 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:34.142 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:50:34.143 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:34.143 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:50:28.817590Z", 2023-10-18T20:50:34.143 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:34.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:34.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:34.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:34.144 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:50:34.144 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:34.144 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:50:34.145 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:50:34.145 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:34.145 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:34.145 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:34.145 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:50:34.146 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:34.146 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:34.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:34.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:34.147 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:50:34.147 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:34.147 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:50:34.147 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:50:34.148 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:34.148 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:34.148 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:34.148 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:50:34.148 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:34.149 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:34.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:34.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:50:34.149 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:50:34.149 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:50:34.150 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:50:34.150 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:50:34.150 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:50:34.150 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:50:34.151 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:50:34.151 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:50:34.151 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:50:34.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:34.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:50:34.152 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:50:34.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:34.152 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:50:34.152 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:50:34.152 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:50:34.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:50:34.153 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:50:34Z" 2023-10-18T20:50:34.153 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:50:34.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697662228.817590_1239.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:50:28.817590Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:50:34Z" } 2023-10-18T20:50:34.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697662228.817590_1239.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:50:28.817590Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:50:34.156 DEBUG:rgw_multi.tests:log_status={0: '1_1697662228.817590_1239.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:50:34.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1697662228.817590_1239.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:50:34.157 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:50:34.158 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-18T20:50:34.158 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T20:50:34.159 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:50:34.157+0000 7f77112ab700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 172938) UID: 0 2023-10-18T20:50:34.160 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:50:34.157+0000 7f77155147c0 -1 shutting down 2023-10-18T20:50:34.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T20:50:40.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2023-10-18T20:50:40.260 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-18T20:50:40.261 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.261 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.262 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.262 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.263 DEBUG:boto:Data: 2023-10-18T20:50:40.263 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.263 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.264 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.264 DEBUG:boto:Params: {} 2023-10-18T20:50:40.265 DEBUG:boto:Token: None 2023-10-18T20:50:40.265 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.266 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c90ab847996939df-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.270 DEBUG:boto:path=/bofrnf-61/obj11 2023-10-18T20:50:40.271 DEBUG:boto:auth_path=/bofrnf-61/obj11 2023-10-18T20:50:40.271 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.271 DEBUG:boto:Path: /bofrnf-61/obj11 2023-10-18T20:50:40.272 DEBUG:boto:Data: 2023-10-18T20:50:40.272 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.273 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.273 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.273 DEBUG:boto:Params: {} 2023-10-18T20:50:40.274 DEBUG:boto:Token: None 2023-10-18T20:50:40.274 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj11 2023-10-18T20:50:40.275 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+zCs3horFSsfJ9Q8cRTA1gigEUg= 2023-10-18T20:50:40.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+zCs3horFSsfJ9Q8cRTA1gigEUg='} 2023-10-18T20:50:40.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5004741568d9239-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.283 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.283 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.284 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.284 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.285 DEBUG:boto:Data: 2023-10-18T20:50:40.285 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.285 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.286 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.286 DEBUG:boto:Params: {} 2023-10-18T20:50:40.287 DEBUG:boto:Token: None 2023-10-18T20:50:40.287 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.288 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093049e21dc437e2c-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.291 DEBUG:boto:path=/bofrnf-61/obj22 2023-10-18T20:50:40.292 DEBUG:boto:auth_path=/bofrnf-61/obj22 2023-10-18T20:50:40.292 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.293 DEBUG:boto:Path: /bofrnf-61/obj22 2023-10-18T20:50:40.293 DEBUG:boto:Data: 2023-10-18T20:50:40.294 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.295 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.295 DEBUG:boto:Params: {} 2023-10-18T20:50:40.296 DEBUG:boto:Token: None 2023-10-18T20:50:40.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj22 2023-10-18T20:50:40.297 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Z/0aXtkPLKIzLd4eWFlBO3EcSQ= 2023-10-18T20:50:40.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Z/0aXtkPLKIzLd4eWFlBO3EcSQ='} 2023-10-18T20:50:40.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f338146c80ad387a-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.302 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.303 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.303 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.304 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.304 DEBUG:boto:Data: 2023-10-18T20:50:40.304 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.305 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.305 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.305 DEBUG:boto:Params: {} 2023-10-18T20:50:40.306 DEBUG:boto:Token: None 2023-10-18T20:50:40.306 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.307 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fb63639a86c1f6a-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.311 DEBUG:boto:path=/bofrnf-61/obj33 2023-10-18T20:50:40.311 DEBUG:boto:auth_path=/bofrnf-61/obj33 2023-10-18T20:50:40.312 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.312 DEBUG:boto:Path: /bofrnf-61/obj33 2023-10-18T20:50:40.312 DEBUG:boto:Data: 2023-10-18T20:50:40.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.313 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.314 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.314 DEBUG:boto:Params: {} 2023-10-18T20:50:40.314 DEBUG:boto:Token: None 2023-10-18T20:50:40.315 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj33 2023-10-18T20:50:40.315 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mwqsHeGCbUI7F62nEipJJbzQa5I= 2023-10-18T20:50:40.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mwqsHeGCbUI7F62nEipJJbzQa5I='} 2023-10-18T20:50:40.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ccb49bf4980694a3-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.360 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.361 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.361 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.362 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.363 DEBUG:boto:Data: 2023-10-18T20:50:40.364 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.364 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.365 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.366 DEBUG:boto:Params: {} 2023-10-18T20:50:40.367 DEBUG:boto:Token: None 2023-10-18T20:50:40.370 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.371 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015e1978443bef1e5-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.375 DEBUG:boto:path=/bofrnf-61/obj44 2023-10-18T20:50:40.375 DEBUG:boto:auth_path=/bofrnf-61/obj44 2023-10-18T20:50:40.376 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.376 DEBUG:boto:Path: /bofrnf-61/obj44 2023-10-18T20:50:40.376 DEBUG:boto:Data: 2023-10-18T20:50:40.377 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.377 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.377 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.378 DEBUG:boto:Params: {} 2023-10-18T20:50:40.378 DEBUG:boto:Token: None 2023-10-18T20:50:40.379 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj44 2023-10-18T20:50:40.379 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QqmSNimUVEQL6mbSG6kYV2S8ZvI= 2023-10-18T20:50:40.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QqmSNimUVEQL6mbSG6kYV2S8ZvI='} 2023-10-18T20:50:40.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087ad1c70cd9a20ba-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.387 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.387 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.388 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.388 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.388 DEBUG:boto:Data: 2023-10-18T20:50:40.389 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.389 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.390 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.390 DEBUG:boto:Params: {} 2023-10-18T20:50:40.391 DEBUG:boto:Token: None 2023-10-18T20:50:40.391 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.392 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004732890c70be52bd-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.396 DEBUG:boto:path=/bofrnf-61/obj55 2023-10-18T20:50:40.396 DEBUG:boto:auth_path=/bofrnf-61/obj55 2023-10-18T20:50:40.397 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.397 DEBUG:boto:Path: /bofrnf-61/obj55 2023-10-18T20:50:40.397 DEBUG:boto:Data: 2023-10-18T20:50:40.398 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.398 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.399 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.399 DEBUG:boto:Params: {} 2023-10-18T20:50:40.399 DEBUG:boto:Token: None 2023-10-18T20:50:40.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj55 2023-10-18T20:50:40.400 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MaKOW0fBYJAoGaOUMTv57xg7TWw= 2023-10-18T20:50:40.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MaKOW0fBYJAoGaOUMTv57xg7TWw='} 2023-10-18T20:50:40.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddfd97f8f015cd2b-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:40.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:40.444 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:50:40.506 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:40.507 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:40.554 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:50:40.554 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-61 2023-10-18T20:50:40.558 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 5 2023-10-18T20:50:40.595 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.596 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.596 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.596 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.597 DEBUG:boto:Data: 2023-10-18T20:50:40.597 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.598 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.598 DEBUG:boto:Params: {} 2023-10-18T20:50:40.599 DEBUG:boto:Token: None 2023-10-18T20:50:40.599 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.600 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f18ecc27e5cbbbd-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.603 DEBUG:boto:path=/bofrnf-61/obj17 2023-10-18T20:50:40.604 DEBUG:boto:auth_path=/bofrnf-61/obj17 2023-10-18T20:50:40.604 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.605 DEBUG:boto:Path: /bofrnf-61/obj17 2023-10-18T20:50:40.605 DEBUG:boto:Data: 2023-10-18T20:50:40.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.606 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.606 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.607 DEBUG:boto:Params: {} 2023-10-18T20:50:40.607 DEBUG:boto:Token: None 2023-10-18T20:50:40.608 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj17 2023-10-18T20:50:40.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5X5vRXOXbBl4waee9PSCNuzebh4= 2023-10-18T20:50:40.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5X5vRXOXbBl4waee9PSCNuzebh4='} 2023-10-18T20:50:40.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fd014ba2cc1cfb1-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.615 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.615 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.615 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.616 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.616 DEBUG:boto:Data: 2023-10-18T20:50:40.617 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.617 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.617 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.618 DEBUG:boto:Params: {} 2023-10-18T20:50:40.618 DEBUG:boto:Token: None 2023-10-18T20:50:40.619 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.619 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4643c9fd002e102-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.623 DEBUG:boto:path=/bofrnf-61/obj34 2023-10-18T20:50:40.624 DEBUG:boto:auth_path=/bofrnf-61/obj34 2023-10-18T20:50:40.624 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.624 DEBUG:boto:Path: /bofrnf-61/obj34 2023-10-18T20:50:40.625 DEBUG:boto:Data: 2023-10-18T20:50:40.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.626 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.626 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.626 DEBUG:boto:Params: {} 2023-10-18T20:50:40.627 DEBUG:boto:Token: None 2023-10-18T20:50:40.627 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj34 2023-10-18T20:50:40.628 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5M45YM0XfKwfRwpOqsIRgLvndrM= 2023-10-18T20:50:40.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5M45YM0XfKwfRwpOqsIRgLvndrM='} 2023-10-18T20:50:40.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b958ecc194de3fbd-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.671 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.672 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.672 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.673 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.673 DEBUG:boto:Data: 2023-10-18T20:50:40.673 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.674 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.674 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.675 DEBUG:boto:Params: {} 2023-10-18T20:50:40.675 DEBUG:boto:Token: None 2023-10-18T20:50:40.676 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd9e751965bcb58f-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.680 DEBUG:boto:path=/bofrnf-61/obj51 2023-10-18T20:50:40.680 DEBUG:boto:auth_path=/bofrnf-61/obj51 2023-10-18T20:50:40.680 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.681 DEBUG:boto:Path: /bofrnf-61/obj51 2023-10-18T20:50:40.681 DEBUG:boto:Data: 2023-10-18T20:50:40.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.682 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.683 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.683 DEBUG:boto:Params: {} 2023-10-18T20:50:40.683 DEBUG:boto:Token: None 2023-10-18T20:50:40.684 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj51 2023-10-18T20:50:40.684 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ZwjUOfM9Rb1CI3BkB0cSRfwRzpI= 2023-10-18T20:50:40.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZwjUOfM9Rb1CI3BkB0cSRfwRzpI='} 2023-10-18T20:50:40.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4bb2c7d85e2e1f1-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.690 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.691 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.691 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.691 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.692 DEBUG:boto:Data: 2023-10-18T20:50:40.692 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.693 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.693 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.694 DEBUG:boto:Params: {} 2023-10-18T20:50:40.694 DEBUG:boto:Token: None 2023-10-18T20:50:40.695 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.695 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002441051dc06a3b82-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.698 DEBUG:boto:path=/bofrnf-61/obj68 2023-10-18T20:50:40.701 DEBUG:boto:auth_path=/bofrnf-61/obj68 2023-10-18T20:50:40.702 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.702 DEBUG:boto:Path: /bofrnf-61/obj68 2023-10-18T20:50:40.703 DEBUG:boto:Data: 2023-10-18T20:50:40.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.704 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.704 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.705 DEBUG:boto:Params: {} 2023-10-18T20:50:40.705 DEBUG:boto:Token: None 2023-10-18T20:50:40.706 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj68 2023-10-18T20:50:40.706 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P7r0zKFLMCP68ag0Y98P+3HdF10= 2023-10-18T20:50:40.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P7r0zKFLMCP68ag0Y98P+3HdF10='} 2023-10-18T20:50:40.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6a2dd6762636311-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.751 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.752 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.752 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.752 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.753 DEBUG:boto:Data: 2023-10-18T20:50:40.753 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.754 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.754 DEBUG:boto:Params: {} 2023-10-18T20:50:40.755 DEBUG:boto:Token: None 2023-10-18T20:50:40.755 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a0dcb6c9716b5d4-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.760 DEBUG:boto:path=/bofrnf-61/obj85 2023-10-18T20:50:40.760 DEBUG:boto:auth_path=/bofrnf-61/obj85 2023-10-18T20:50:40.760 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.761 DEBUG:boto:Path: /bofrnf-61/obj85 2023-10-18T20:50:40.761 DEBUG:boto:Data: 2023-10-18T20:50:40.762 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.762 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.762 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.763 DEBUG:boto:Params: {} 2023-10-18T20:50:40.763 DEBUG:boto:Token: None 2023-10-18T20:50:40.764 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj85 2023-10-18T20:50:40.764 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ERDYnguzDBJ5bvHz2psiWcpzwUc= 2023-10-18T20:50:40.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ERDYnguzDBJ5bvHz2psiWcpzwUc='} 2023-10-18T20:50:40.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000673e0bfd1813a046-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:40.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:40.771 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:50:40.824 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:40.824 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:40.866 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:50:40.866 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-61 2023-10-18T20:50:40.871 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 10 2023-10-18T20:50:40.913 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.914 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.914 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.915 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.915 DEBUG:boto:Data: 2023-10-18T20:50:40.916 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.916 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.916 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.917 DEBUG:boto:Params: {} 2023-10-18T20:50:40.917 DEBUG:boto:Token: None 2023-10-18T20:50:40.917 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.918 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027ab0f3d003f4a9e-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.922 DEBUG:boto:path=/bofrnf-61/obj19 2023-10-18T20:50:40.923 DEBUG:boto:auth_path=/bofrnf-61/obj19 2023-10-18T20:50:40.923 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.923 DEBUG:boto:Path: /bofrnf-61/obj19 2023-10-18T20:50:40.924 DEBUG:boto:Data: 2023-10-18T20:50:40.924 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.924 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.925 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.925 DEBUG:boto:Params: {} 2023-10-18T20:50:40.926 DEBUG:boto:Token: None 2023-10-18T20:50:40.926 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj19 2023-10-18T20:50:40.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hO/RFzcTEKPT1MG/KH57NAWRE94= 2023-10-18T20:50:40.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hO/RFzcTEKPT1MG/KH57NAWRE94='} 2023-10-18T20:50:40.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a579d3a8ac891f21-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.933 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.933 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.933 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.934 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.934 DEBUG:boto:Data: 2023-10-18T20:50:40.935 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.935 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.935 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.936 DEBUG:boto:Params: {} 2023-10-18T20:50:40.936 DEBUG:boto:Token: None 2023-10-18T20:50:40.937 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/ 2023-10-18T20:50:40.937 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T20:50:40.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T20:50:40.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b0855f954684678-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.940 DEBUG:boto:path=/bofrnf-61/obj38 2023-10-18T20:50:40.941 DEBUG:boto:auth_path=/bofrnf-61/obj38 2023-10-18T20:50:40.941 DEBUG:boto:Method: PUT 2023-10-18T20:50:40.941 DEBUG:boto:Path: /bofrnf-61/obj38 2023-10-18T20:50:40.942 DEBUG:boto:Data: 2023-10-18T20:50:40.942 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:40.942 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.943 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.943 DEBUG:boto:Params: {} 2023-10-18T20:50:40.944 DEBUG:boto:Token: None 2023-10-18T20:50:40.944 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:40 GMT /bofrnf-61/obj38 2023-10-18T20:50:40.944 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Hc+7qJO3uA0saBzoOHaLzVm2e5A= 2023-10-18T20:50:40.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Hc+7qJO3uA0saBzoOHaLzVm2e5A='} 2023-10-18T20:50:40.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4fab1709946a2d8-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:40.996 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:40.996 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:40.997 DEBUG:boto:Method: HEAD 2023-10-18T20:50:40.997 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:40.997 DEBUG:boto:Data: 2023-10-18T20:50:40.998 DEBUG:boto:Headers: {} 2023-10-18T20:50:40.998 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:40.999 DEBUG:boto:Port: 8000 2023-10-18T20:50:40.999 DEBUG:boto:Params: {} 2023-10-18T20:50:41.000 DEBUG:boto:Token: None 2023-10-18T20:50:41.000 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.000 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093cf7d6ae72a6516-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.004 DEBUG:boto:path=/bofrnf-61/obj57 2023-10-18T20:50:41.004 DEBUG:boto:auth_path=/bofrnf-61/obj57 2023-10-18T20:50:41.005 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.005 DEBUG:boto:Path: /bofrnf-61/obj57 2023-10-18T20:50:41.006 DEBUG:boto:Data: 2023-10-18T20:50:41.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.007 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.007 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.007 DEBUG:boto:Params: {} 2023-10-18T20:50:41.008 DEBUG:boto:Token: None 2023-10-18T20:50:41.008 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj57 2023-10-18T20:50:41.009 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:As+YGEoMJP7Vd1swA0373E3uNHw= 2023-10-18T20:50:41.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:As+YGEoMJP7Vd1swA0373E3uNHw='} 2023-10-18T20:50:41.013 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e03945061ef6763-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.013 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.014 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.014 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.015 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.015 DEBUG:boto:Data: 2023-10-18T20:50:41.015 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.016 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.016 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.017 DEBUG:boto:Params: {} 2023-10-18T20:50:41.017 DEBUG:boto:Token: None 2023-10-18T20:50:41.017 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.018 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000241dd375b3d4e8ce-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.021 DEBUG:boto:path=/bofrnf-61/obj76 2023-10-18T20:50:41.022 DEBUG:boto:auth_path=/bofrnf-61/obj76 2023-10-18T20:50:41.022 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.023 DEBUG:boto:Path: /bofrnf-61/obj76 2023-10-18T20:50:41.023 DEBUG:boto:Data: 2023-10-18T20:50:41.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.024 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.024 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.024 DEBUG:boto:Params: {} 2023-10-18T20:50:41.025 DEBUG:boto:Token: None 2023-10-18T20:50:41.025 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj76 2023-10-18T20:50:41.026 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AcOHfRQZ0tP7XyxHBetqFwIjobs= 2023-10-18T20:50:41.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AcOHfRQZ0tP7XyxHBetqFwIjobs='} 2023-10-18T20:50:41.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000788b530c23c72b01-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.072 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.072 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.073 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.074 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.074 DEBUG:boto:Data: 2023-10-18T20:50:41.075 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.075 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.075 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.076 DEBUG:boto:Params: {} 2023-10-18T20:50:41.076 DEBUG:boto:Token: None 2023-10-18T20:50:41.077 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.077 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edac04ba3a2851b0-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.081 DEBUG:boto:path=/bofrnf-61/obj95 2023-10-18T20:50:41.081 DEBUG:boto:auth_path=/bofrnf-61/obj95 2023-10-18T20:50:41.082 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.082 DEBUG:boto:Path: /bofrnf-61/obj95 2023-10-18T20:50:41.082 DEBUG:boto:Data: 2023-10-18T20:50:41.083 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.083 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.084 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.084 DEBUG:boto:Params: {} 2023-10-18T20:50:41.085 DEBUG:boto:Token: None 2023-10-18T20:50:41.085 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj95 2023-10-18T20:50:41.086 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:LatHDjB7IOO1QMhS4GQR4BONr9Q= 2023-10-18T20:50:41.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LatHDjB7IOO1QMhS4GQR4BONr9Q='} 2023-10-18T20:50:41.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d883cb0486ff6b60-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:41.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:41.091 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:50:41.144 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:41.144 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:41.188 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:50:41.188 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-61 2023-10-18T20:50:41.194 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 15 2023-10-18T20:50:41.240 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.241 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.241 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.242 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.242 DEBUG:boto:Data: 2023-10-18T20:50:41.243 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.243 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.243 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.244 DEBUG:boto:Params: {} 2023-10-18T20:50:41.244 DEBUG:boto:Token: None 2023-10-18T20:50:41.244 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.245 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e6547115ca64973-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.249 DEBUG:boto:path=/bofrnf-61/obj23 2023-10-18T20:50:41.249 DEBUG:boto:auth_path=/bofrnf-61/obj23 2023-10-18T20:50:41.250 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.250 DEBUG:boto:Path: /bofrnf-61/obj23 2023-10-18T20:50:41.251 DEBUG:boto:Data: 2023-10-18T20:50:41.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.252 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.252 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.252 DEBUG:boto:Params: {} 2023-10-18T20:50:41.253 DEBUG:boto:Token: None 2023-10-18T20:50:41.253 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj23 2023-10-18T20:50:41.254 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JXfhaUcaKlwQHGZZxMzzRQkNxaw= 2023-10-18T20:50:41.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JXfhaUcaKlwQHGZZxMzzRQkNxaw='} 2023-10-18T20:50:41.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006577e658af898f67-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.260 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.260 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.261 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.261 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.261 DEBUG:boto:Data: 2023-10-18T20:50:41.262 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.262 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.263 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.263 DEBUG:boto:Params: {} 2023-10-18T20:50:41.264 DEBUG:boto:Token: None 2023-10-18T20:50:41.264 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.264 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc716804c106130d-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.268 DEBUG:boto:path=/bofrnf-61/obj46 2023-10-18T20:50:41.268 DEBUG:boto:auth_path=/bofrnf-61/obj46 2023-10-18T20:50:41.269 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.269 DEBUG:boto:Path: /bofrnf-61/obj46 2023-10-18T20:50:41.269 DEBUG:boto:Data: 2023-10-18T20:50:41.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.270 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.270 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.271 DEBUG:boto:Params: {} 2023-10-18T20:50:41.271 DEBUG:boto:Token: None 2023-10-18T20:50:41.272 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj46 2023-10-18T20:50:41.272 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EcXWhEheRv891gG5uk1m00F/kg8= 2023-10-18T20:50:41.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EcXWhEheRv891gG5uk1m00F/kg8='} 2023-10-18T20:50:41.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028bd3f971e3f53bb-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.315 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.315 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.316 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.316 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.317 DEBUG:boto:Data: 2023-10-18T20:50:41.317 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.317 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.318 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.318 DEBUG:boto:Params: {} 2023-10-18T20:50:41.319 DEBUG:boto:Token: None 2023-10-18T20:50:41.319 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.320 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aaaca00babe5b4c-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.323 DEBUG:boto:path=/bofrnf-61/obj69 2023-10-18T20:50:41.324 DEBUG:boto:auth_path=/bofrnf-61/obj69 2023-10-18T20:50:41.324 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.325 DEBUG:boto:Path: /bofrnf-61/obj69 2023-10-18T20:50:41.325 DEBUG:boto:Data: 2023-10-18T20:50:41.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.326 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.326 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.327 DEBUG:boto:Params: {} 2023-10-18T20:50:41.327 DEBUG:boto:Token: None 2023-10-18T20:50:41.327 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj69 2023-10-18T20:50:41.328 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yQxs/Cvmtyz3B+6pEtPQ62sk2K0= 2023-10-18T20:50:41.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yQxs/Cvmtyz3B+6pEtPQ62sk2K0='} 2023-10-18T20:50:41.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079a7200a2fb66790-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.334 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.334 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.335 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.335 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.336 DEBUG:boto:Data: 2023-10-18T20:50:41.336 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.336 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.337 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.337 DEBUG:boto:Params: {} 2023-10-18T20:50:41.338 DEBUG:boto:Token: None 2023-10-18T20:50:41.338 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.338 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f62295d458aedb88-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.342 DEBUG:boto:path=/bofrnf-61/obj92 2023-10-18T20:50:41.343 DEBUG:boto:auth_path=/bofrnf-61/obj92 2023-10-18T20:50:41.343 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.343 DEBUG:boto:Path: /bofrnf-61/obj92 2023-10-18T20:50:41.344 DEBUG:boto:Data: 2023-10-18T20:50:41.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.345 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.345 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.345 DEBUG:boto:Params: {} 2023-10-18T20:50:41.346 DEBUG:boto:Token: None 2023-10-18T20:50:41.346 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj92 2023-10-18T20:50:41.347 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:P+vWJizPVOLr1MjBBkG1sNVleEo= 2023-10-18T20:50:41.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P+vWJizPVOLr1MjBBkG1sNVleEo='} 2023-10-18T20:50:41.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f535b1715b56b04-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.391 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.391 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.392 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.392 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.393 DEBUG:boto:Data: 2023-10-18T20:50:41.393 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.393 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.394 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.394 DEBUG:boto:Params: {} 2023-10-18T20:50:41.395 DEBUG:boto:Token: None 2023-10-18T20:50:41.395 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.395 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e7ec169ad9e8e2a-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.399 DEBUG:boto:path=/bofrnf-61/obj115 2023-10-18T20:50:41.400 DEBUG:boto:auth_path=/bofrnf-61/obj115 2023-10-18T20:50:41.400 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.401 DEBUG:boto:Path: /bofrnf-61/obj115 2023-10-18T20:50:41.401 DEBUG:boto:Data: 2023-10-18T20:50:41.401 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.402 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.402 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.403 DEBUG:boto:Params: {} 2023-10-18T20:50:41.403 DEBUG:boto:Token: None 2023-10-18T20:50:41.403 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj115 2023-10-18T20:50:41.404 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yPI1G5UlPXkmrYyOnY7pogDwP+Y= 2023-10-18T20:50:41.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yPI1G5UlPXkmrYyOnY7pogDwP+Y='} 2023-10-18T20:50:41.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009664eedf1f6b8f44-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:41.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:41.411 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:50:41.465 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:41.465 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:41.513 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:50:41.514 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-61 2023-10-18T20:50:41.521 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 20 2023-10-18T20:50:41.570 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.571 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.571 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.571 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.572 DEBUG:boto:Data: 2023-10-18T20:50:41.572 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.572 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.573 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.573 DEBUG:boto:Params: {} 2023-10-18T20:50:41.574 DEBUG:boto:Token: None 2023-10-18T20:50:41.574 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.575 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.579 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e355fb93b69aee75-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.580 DEBUG:boto:path=/bofrnf-61/obj29 2023-10-18T20:50:41.580 DEBUG:boto:auth_path=/bofrnf-61/obj29 2023-10-18T20:50:41.581 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.581 DEBUG:boto:Path: /bofrnf-61/obj29 2023-10-18T20:50:41.581 DEBUG:boto:Data: 2023-10-18T20:50:41.582 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.583 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.583 DEBUG:boto:Params: {} 2023-10-18T20:50:41.584 DEBUG:boto:Token: None 2023-10-18T20:50:41.584 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj29 2023-10-18T20:50:41.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KU/tlibYxiEjkDISWDyU7RIvKjo= 2023-10-18T20:50:41.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KU/tlibYxiEjkDISWDyU7RIvKjo='} 2023-10-18T20:50:41.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b69c3557b8763bf0-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.631 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.632 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.632 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.632 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.633 DEBUG:boto:Data: 2023-10-18T20:50:41.633 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.634 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.634 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.634 DEBUG:boto:Params: {} 2023-10-18T20:50:41.635 DEBUG:boto:Token: None 2023-10-18T20:50:41.635 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.636 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf4abadc293bfcb5-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.641 DEBUG:boto:path=/bofrnf-61/obj58 2023-10-18T20:50:41.642 DEBUG:boto:auth_path=/bofrnf-61/obj58 2023-10-18T20:50:41.642 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.643 DEBUG:boto:Path: /bofrnf-61/obj58 2023-10-18T20:50:41.643 DEBUG:boto:Data: 2023-10-18T20:50:41.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.644 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.645 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.645 DEBUG:boto:Params: {} 2023-10-18T20:50:41.645 DEBUG:boto:Token: None 2023-10-18T20:50:41.646 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj58 2023-10-18T20:50:41.647 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ECT3PO3ZOEmHa3ZLotUr/KFIH80= 2023-10-18T20:50:41.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ECT3PO3ZOEmHa3ZLotUr/KFIH80='} 2023-10-18T20:50:41.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046ef1a2b5b82e0c8-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.653 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.654 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.654 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.654 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.655 DEBUG:boto:Data: 2023-10-18T20:50:41.655 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.655 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.656 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.656 DEBUG:boto:Params: {} 2023-10-18T20:50:41.657 DEBUG:boto:Token: None 2023-10-18T20:50:41.657 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.658 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff9e3d4ddd207ae1-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.662 DEBUG:boto:path=/bofrnf-61/obj87 2023-10-18T20:50:41.662 DEBUG:boto:auth_path=/bofrnf-61/obj87 2023-10-18T20:50:41.663 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.663 DEBUG:boto:Path: /bofrnf-61/obj87 2023-10-18T20:50:41.663 DEBUG:boto:Data: 2023-10-18T20:50:41.664 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.664 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.665 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.665 DEBUG:boto:Params: {} 2023-10-18T20:50:41.665 DEBUG:boto:Token: None 2023-10-18T20:50:41.666 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj87 2023-10-18T20:50:41.666 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BKoajGojlaOUkhSgSaZd68FkIq4= 2023-10-18T20:50:41.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BKoajGojlaOUkhSgSaZd68FkIq4='} 2023-10-18T20:50:41.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000481015a50fbb848-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.715 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.715 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.716 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.716 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.716 DEBUG:boto:Data: 2023-10-18T20:50:41.717 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.717 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.717 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.718 DEBUG:boto:Params: {} 2023-10-18T20:50:41.718 DEBUG:boto:Token: None 2023-10-18T20:50:41.719 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.719 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061532d843257603a-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.723 DEBUG:boto:path=/bofrnf-61/obj116 2023-10-18T20:50:41.724 DEBUG:boto:auth_path=/bofrnf-61/obj116 2023-10-18T20:50:41.724 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.724 DEBUG:boto:Path: /bofrnf-61/obj116 2023-10-18T20:50:41.725 DEBUG:boto:Data: 2023-10-18T20:50:41.725 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.725 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.726 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.726 DEBUG:boto:Params: {} 2023-10-18T20:50:41.727 DEBUG:boto:Token: None 2023-10-18T20:50:41.727 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj116 2023-10-18T20:50:41.728 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hXcdwUyi6brj7Kt5bYVtSqZ+BM0= 2023-10-18T20:50:41.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hXcdwUyi6brj7Kt5bYVtSqZ+BM0='} 2023-10-18T20:50:41.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023a5734ef7d569b2-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.734 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.734 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.735 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.735 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.736 DEBUG:boto:Data: 2023-10-18T20:50:41.736 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.736 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.737 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.737 DEBUG:boto:Params: {} 2023-10-18T20:50:41.738 DEBUG:boto:Token: None 2023-10-18T20:50:41.738 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.739 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d2ebb4179565dab-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.743 DEBUG:boto:path=/bofrnf-61/obj145 2023-10-18T20:50:41.743 DEBUG:boto:auth_path=/bofrnf-61/obj145 2023-10-18T20:50:41.744 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.744 DEBUG:boto:Path: /bofrnf-61/obj145 2023-10-18T20:50:41.745 DEBUG:boto:Data: 2023-10-18T20:50:41.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.746 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.747 DEBUG:boto:Params: {} 2023-10-18T20:50:41.747 DEBUG:boto:Token: None 2023-10-18T20:50:41.748 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj145 2023-10-18T20:50:41.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7PAcXBJ7iwzEGHK3brvBgF4ZMOU= 2023-10-18T20:50:41.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7PAcXBJ7iwzEGHK3brvBgF4ZMOU='} 2023-10-18T20:50:41.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b285c96010bb1a0e-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:41.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:41.792 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:50:41.845 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:41.845 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:41.894 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:50:41.894 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-61 2023-10-18T20:50:41.904 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 25 2023-10-18T20:50:41.961 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.961 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.962 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.962 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.962 DEBUG:boto:Data: 2023-10-18T20:50:41.963 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.963 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.964 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.964 DEBUG:boto:Params: {} 2023-10-18T20:50:41.965 DEBUG:boto:Token: None 2023-10-18T20:50:41.965 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.965 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9939f072f0f9a36-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.970 DEBUG:boto:path=/bofrnf-61/obj31 2023-10-18T20:50:41.970 DEBUG:boto:auth_path=/bofrnf-61/obj31 2023-10-18T20:50:41.971 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.971 DEBUG:boto:Path: /bofrnf-61/obj31 2023-10-18T20:50:41.971 DEBUG:boto:Data: 2023-10-18T20:50:41.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.973 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.973 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.973 DEBUG:boto:Params: {} 2023-10-18T20:50:41.974 DEBUG:boto:Token: None 2023-10-18T20:50:41.974 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj31 2023-10-18T20:50:41.975 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KWOYt6XP/F2DNu5XcWGZmOXwST4= 2023-10-18T20:50:41.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KWOYt6XP/F2DNu5XcWGZmOXwST4='} 2023-10-18T20:50:41.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ba98501fe123faa-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.981 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:41.982 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:41.982 DEBUG:boto:Method: HEAD 2023-10-18T20:50:41.983 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:41.983 DEBUG:boto:Data: 2023-10-18T20:50:41.984 DEBUG:boto:Headers: {} 2023-10-18T20:50:41.984 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.984 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.985 DEBUG:boto:Params: {} 2023-10-18T20:50:41.985 DEBUG:boto:Token: None 2023-10-18T20:50:41.986 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/ 2023-10-18T20:50:41.986 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T20:50:41.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T20:50:41.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbbcfc24b9ab4394-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:41.990 DEBUG:boto:path=/bofrnf-61/obj62 2023-10-18T20:50:41.991 DEBUG:boto:auth_path=/bofrnf-61/obj62 2023-10-18T20:50:41.991 DEBUG:boto:Method: PUT 2023-10-18T20:50:41.991 DEBUG:boto:Path: /bofrnf-61/obj62 2023-10-18T20:50:41.992 DEBUG:boto:Data: 2023-10-18T20:50:41.992 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:41.993 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:41.993 DEBUG:boto:Port: 8000 2023-10-18T20:50:41.994 DEBUG:boto:Params: {} 2023-10-18T20:50:41.994 DEBUG:boto:Token: None 2023-10-18T20:50:41.995 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:41 GMT /bofrnf-61/obj62 2023-10-18T20:50:41.995 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:p9yfzgZ14/6RN1dTEnE78uvpbJE= 2023-10-18T20:50:41.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p9yfzgZ14/6RN1dTEnE78uvpbJE='} 2023-10-18T20:50:42.001 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9d5e22e0a94ad49-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.001 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:42.002 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:42.003 DEBUG:boto:Method: HEAD 2023-10-18T20:50:42.003 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:42.004 DEBUG:boto:Data: 2023-10-18T20:50:42.004 DEBUG:boto:Headers: {} 2023-10-18T20:50:42.005 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.005 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.005 DEBUG:boto:Params: {} 2023-10-18T20:50:42.006 DEBUG:boto:Token: None 2023-10-18T20:50:42.006 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/ 2023-10-18T20:50:42.007 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T20:50:42.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T20:50:42.011 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f08eb02e8d6a81d3-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.011 DEBUG:boto:path=/bofrnf-61/obj93 2023-10-18T20:50:42.012 DEBUG:boto:auth_path=/bofrnf-61/obj93 2023-10-18T20:50:42.012 DEBUG:boto:Method: PUT 2023-10-18T20:50:42.012 DEBUG:boto:Path: /bofrnf-61/obj93 2023-10-18T20:50:42.013 DEBUG:boto:Data: 2023-10-18T20:50:42.013 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:42.014 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.014 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.015 DEBUG:boto:Params: {} 2023-10-18T20:50:42.015 DEBUG:boto:Token: None 2023-10-18T20:50:42.015 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/obj93 2023-10-18T20:50:42.016 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iINOw8rFtbRHBHbdM18CkO8cgyQ= 2023-10-18T20:50:42.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iINOw8rFtbRHBHbdM18CkO8cgyQ='} 2023-10-18T20:50:42.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcb24477849463e8-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.063 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:42.064 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:42.064 DEBUG:boto:Method: HEAD 2023-10-18T20:50:42.065 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:42.065 DEBUG:boto:Data: 2023-10-18T20:50:42.065 DEBUG:boto:Headers: {} 2023-10-18T20:50:42.066 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.066 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.067 DEBUG:boto:Params: {} 2023-10-18T20:50:42.067 DEBUG:boto:Token: None 2023-10-18T20:50:42.068 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/ 2023-10-18T20:50:42.068 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T20:50:42.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T20:50:42.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018e47b021cfe3168-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.073 DEBUG:boto:path=/bofrnf-61/obj124 2023-10-18T20:50:42.073 DEBUG:boto:auth_path=/bofrnf-61/obj124 2023-10-18T20:50:42.073 DEBUG:boto:Method: PUT 2023-10-18T20:50:42.074 DEBUG:boto:Path: /bofrnf-61/obj124 2023-10-18T20:50:42.074 DEBUG:boto:Data: 2023-10-18T20:50:42.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:42.075 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.075 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.076 DEBUG:boto:Params: {} 2023-10-18T20:50:42.076 DEBUG:boto:Token: None 2023-10-18T20:50:42.077 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/obj124 2023-10-18T20:50:42.077 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i+ur+qcqTEwrj3EfKRY5xIQr1gE= 2023-10-18T20:50:42.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i+ur+qcqTEwrj3EfKRY5xIQr1gE='} 2023-10-18T20:50:42.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a53fbe1ce2b8406d-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.083 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:42.084 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:42.084 DEBUG:boto:Method: HEAD 2023-10-18T20:50:42.085 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:42.085 DEBUG:boto:Data: 2023-10-18T20:50:42.086 DEBUG:boto:Headers: {} 2023-10-18T20:50:42.086 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.087 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.087 DEBUG:boto:Params: {} 2023-10-18T20:50:42.088 DEBUG:boto:Token: None 2023-10-18T20:50:42.088 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/ 2023-10-18T20:50:42.088 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T20:50:42.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T20:50:42.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000300c4006a84e89f5-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.092 DEBUG:boto:path=/bofrnf-61/obj155 2023-10-18T20:50:42.093 DEBUG:boto:auth_path=/bofrnf-61/obj155 2023-10-18T20:50:42.093 DEBUG:boto:Method: PUT 2023-10-18T20:50:42.094 DEBUG:boto:Path: /bofrnf-61/obj155 2023-10-18T20:50:42.094 DEBUG:boto:Data: 2023-10-18T20:50:42.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:42.095 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.095 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.096 DEBUG:boto:Params: {} 2023-10-18T20:50:42.096 DEBUG:boto:Token: None 2023-10-18T20:50:42.097 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/obj155 2023-10-18T20:50:42.097 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y5aKCZnIdGhKfjUy/xkqOQqJBWI= 2023-10-18T20:50:42.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y5aKCZnIdGhKfjUy/xkqOQqJBWI='} 2023-10-18T20:50:42.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e106619f4fd2197-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:42.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:42.144 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:50:42.198 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:42.198 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:42.247 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:50:42.247 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-61 2023-10-18T20:50:42.258 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 30 2023-10-18T20:50:42.318 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:42.318 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:42.319 DEBUG:boto:Method: HEAD 2023-10-18T20:50:42.319 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:42.319 DEBUG:boto:Data: 2023-10-18T20:50:42.320 DEBUG:boto:Headers: {} 2023-10-18T20:50:42.320 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.320 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.321 DEBUG:boto:Params: {} 2023-10-18T20:50:42.321 DEBUG:boto:Token: None 2023-10-18T20:50:42.322 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/ 2023-10-18T20:50:42.322 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T20:50:42.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T20:50:42.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6d025b11b77e4fe-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.327 DEBUG:boto:path=/bofrnf-61/obj37 2023-10-18T20:50:42.328 DEBUG:boto:auth_path=/bofrnf-61/obj37 2023-10-18T20:50:42.328 DEBUG:boto:Method: PUT 2023-10-18T20:50:42.329 DEBUG:boto:Path: /bofrnf-61/obj37 2023-10-18T20:50:42.329 DEBUG:boto:Data: 2023-10-18T20:50:42.330 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:42.330 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.331 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.331 DEBUG:boto:Params: {} 2023-10-18T20:50:42.332 DEBUG:boto:Token: None 2023-10-18T20:50:42.333 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/obj37 2023-10-18T20:50:42.333 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dAuCCm8qDeTOx1mwhewHofiB0KA= 2023-10-18T20:50:42.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dAuCCm8qDeTOx1mwhewHofiB0KA='} 2023-10-18T20:50:42.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075b5bdd786afe7c6-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.340 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:42.340 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:42.341 DEBUG:boto:Method: HEAD 2023-10-18T20:50:42.341 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:42.342 DEBUG:boto:Data: 2023-10-18T20:50:42.342 DEBUG:boto:Headers: {} 2023-10-18T20:50:42.343 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.343 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.344 DEBUG:boto:Params: {} 2023-10-18T20:50:42.344 DEBUG:boto:Token: None 2023-10-18T20:50:42.345 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/ 2023-10-18T20:50:42.345 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T20:50:42.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T20:50:42.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000677ec04ac70b676f-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.350 DEBUG:boto:path=/bofrnf-61/obj74 2023-10-18T20:50:42.350 DEBUG:boto:auth_path=/bofrnf-61/obj74 2023-10-18T20:50:42.351 DEBUG:boto:Method: PUT 2023-10-18T20:50:42.351 DEBUG:boto:Path: /bofrnf-61/obj74 2023-10-18T20:50:42.351 DEBUG:boto:Data: 2023-10-18T20:50:42.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:42.352 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.353 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.353 DEBUG:boto:Params: {} 2023-10-18T20:50:42.354 DEBUG:boto:Token: None 2023-10-18T20:50:42.354 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/obj74 2023-10-18T20:50:42.355 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RbtSlfZpzFjwDpS1dznruxBa2yw= 2023-10-18T20:50:42.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RbtSlfZpzFjwDpS1dznruxBa2yw='} 2023-10-18T20:50:42.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000769eccab0ca46c58-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.361 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:42.362 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:42.362 DEBUG:boto:Method: HEAD 2023-10-18T20:50:42.362 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:42.363 DEBUG:boto:Data: 2023-10-18T20:50:42.363 DEBUG:boto:Headers: {} 2023-10-18T20:50:42.364 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.364 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.364 DEBUG:boto:Params: {} 2023-10-18T20:50:42.365 DEBUG:boto:Token: None 2023-10-18T20:50:42.365 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/ 2023-10-18T20:50:42.366 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T20:50:42.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T20:50:42.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6d2707a8865fba0-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.371 DEBUG:boto:path=/bofrnf-61/obj111 2023-10-18T20:50:42.372 DEBUG:boto:auth_path=/bofrnf-61/obj111 2023-10-18T20:50:42.372 DEBUG:boto:Method: PUT 2023-10-18T20:50:42.373 DEBUG:boto:Path: /bofrnf-61/obj111 2023-10-18T20:50:42.373 DEBUG:boto:Data: 2023-10-18T20:50:42.373 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:42.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.374 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.375 DEBUG:boto:Params: {} 2023-10-18T20:50:42.375 DEBUG:boto:Token: None 2023-10-18T20:50:42.375 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/obj111 2023-10-18T20:50:42.376 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iLrH5XE3LLuQVMmMKYJla6T2Bkg= 2023-10-18T20:50:42.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iLrH5XE3LLuQVMmMKYJla6T2Bkg='} 2023-10-18T20:50:42.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d3175f69fa98bef-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.419 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:42.419 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:42.420 DEBUG:boto:Method: HEAD 2023-10-18T20:50:42.420 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:42.421 DEBUG:boto:Data: 2023-10-18T20:50:42.421 DEBUG:boto:Headers: {} 2023-10-18T20:50:42.421 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.422 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.422 DEBUG:boto:Params: {} 2023-10-18T20:50:42.422 DEBUG:boto:Token: None 2023-10-18T20:50:42.423 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/ 2023-10-18T20:50:42.423 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T20:50:42.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T20:50:42.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000651cb6076b73e2fa-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.428 DEBUG:boto:path=/bofrnf-61/obj148 2023-10-18T20:50:42.429 DEBUG:boto:auth_path=/bofrnf-61/obj148 2023-10-18T20:50:42.429 DEBUG:boto:Method: PUT 2023-10-18T20:50:42.429 DEBUG:boto:Path: /bofrnf-61/obj148 2023-10-18T20:50:42.430 DEBUG:boto:Data: 2023-10-18T20:50:42.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:42.430 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.431 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.431 DEBUG:boto:Params: {} 2023-10-18T20:50:42.432 DEBUG:boto:Token: None 2023-10-18T20:50:42.432 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/obj148 2023-10-18T20:50:42.432 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4GcIJKzhOlZAMq5YzkCDHhLPi1Y= 2023-10-18T20:50:42.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4GcIJKzhOlZAMq5YzkCDHhLPi1Y='} 2023-10-18T20:50:42.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015c5f6ac9873eb58-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.438 DEBUG:boto:path=/bofrnf-61/ 2023-10-18T20:50:42.439 DEBUG:boto:auth_path=/bofrnf-61/ 2023-10-18T20:50:42.439 DEBUG:boto:Method: HEAD 2023-10-18T20:50:42.439 DEBUG:boto:Path: /bofrnf-61/ 2023-10-18T20:50:42.440 DEBUG:boto:Data: 2023-10-18T20:50:42.440 DEBUG:boto:Headers: {} 2023-10-18T20:50:42.441 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.441 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.441 DEBUG:boto:Params: {} 2023-10-18T20:50:42.442 DEBUG:boto:Token: None 2023-10-18T20:50:42.442 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/ 2023-10-18T20:50:42.443 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T20:50:42.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T20:50:42.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4c4717726dc6e30-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.447 DEBUG:boto:path=/bofrnf-61/obj185 2023-10-18T20:50:42.447 DEBUG:boto:auth_path=/bofrnf-61/obj185 2023-10-18T20:50:42.448 DEBUG:boto:Method: PUT 2023-10-18T20:50:42.448 DEBUG:boto:Path: /bofrnf-61/obj185 2023-10-18T20:50:42.448 DEBUG:boto:Data: 2023-10-18T20:50:42.449 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:50:42.449 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:50:42.450 DEBUG:boto:Port: 8000 2023-10-18T20:50:42.450 DEBUG:boto:Params: {} 2023-10-18T20:50:42.451 DEBUG:boto:Token: None 2023-10-18T20:50:42.451 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:50:42 GMT /bofrnf-61/obj185 2023-10-18T20:50:42.451 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ByDtyjupSDpNvPEWRQiVgc4xjYo= 2023-10-18T20:50:42.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ByDtyjupSDpNvPEWRQiVgc4xjYo='} 2023-10-18T20:50:42.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2407c36c637750d-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:50:42.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:42.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:50:42.496 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:50:42.546 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:42.546 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:42.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:42.575+0000 7f573e733780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-18T20:50:42.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:42.577+0000 7f573e733780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-18T20:50:42.598 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-18T20:50:42.599 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T20:50:42.601 INFO:tasks.rgw.c1.client.1:Started 2023-10-18T20:50:42.602 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:42.623 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:50:42.623 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:50:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T20:50:42.625 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T20:50:43.626 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:50:43.648 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:50:43.648 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:50:43.650 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T20:50:43.650 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T20:50:43.650 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-61 2023-10-18T20:50:43.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:50:43.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:50:43.652 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:50:43.745 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:50:43.746 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:50:43.782 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:43.780+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:50:45.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-18T20:50:48.782 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:48.781+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:50:53.783 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:53.781+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:50:58.784 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:50:58.781+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:03.784 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:03.782+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:08.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:08.783+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:13.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:13.784+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:18.786 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:18.784+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:23.786 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:23.785+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:28.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:28.785+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:33.787 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:33.786+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:38.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:38.786+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:43.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:43.787+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:48.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:48.788+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:53.790 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:53.787+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:51:58.790 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:51:58.788+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:03.791 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:03.789+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:08.791 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:08.790+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:13.792 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:13.791+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:18.793 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:18.791+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:23.793 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:23.792+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:28.794 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:28.792+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:33.795 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:33.793+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:38.795 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:38.794+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:43.797 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:43.794+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:48.797 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:48.794+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:53.798 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:53.795+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:52:58.798 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:52:58.797+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:03.799 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:03.797+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:08.800 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:08.798+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:13.800 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:13.799+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:18.801 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:18.799+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:23.801 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:23.800+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:28.802 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:28.800+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:33.803 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:33.800+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:38.803 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:38.801+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:43.804 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:43.801+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:48.804 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:48.803+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:53.805 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:53.804+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:53:58.806 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:53:58.804+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:03.806 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:03.805+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:08.808 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:08.806+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:13.808 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:13.806+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:18.808 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:18.807+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:23.809 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:23.807+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:28.810 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:28.807+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:33.810 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:33.808+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:38.811 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:38.809+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:43.811 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:43.810+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:48.812 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:48.810+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:53.812 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:53.811+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:54:58.813 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:54:58.812+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:55:03.814 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:03.813+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:55:08.814 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:08.813+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:55:13.815 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:13.814+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:55:18.816 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:18.813+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:55:23.817 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:23.814+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:55:28.818 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:28.814+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:55:33.817 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:33.816+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T20:55:38.818 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:38.817+0000 7f7461192780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-10-18T20:55:38.819 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:38.817+0000 7f7461192780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T20:55:38.819 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:38.817+0000 7f7461192780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T20:55:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-18T20:55:38.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2023-10-18T20:55:38.840 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:38.840 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:38.841 DEBUG:boto:Method: PUT 2023-10-18T20:55:38.841 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:55:38.842 DEBUG:boto:Data: 2023-10-18T20:55:38.842 DEBUG:boto:Headers: {} 2023-10-18T20:55:38.843 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:38.843 DEBUG:boto:Port: 8000 2023-10-18T20:55:38.843 DEBUG:boto:Params: {} 2023-10-18T20:55:38.844 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:55:38.845 DEBUG:boto:Token: None 2023-10-18T20:55:38.845 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:55:38 GMT /bofrnf-62/ 2023-10-18T20:55:38.846 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VEgWisI6zkFUE/on5Ys5oG0hDIQ= 2023-10-18T20:55:38.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:38 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VEgWisI6zkFUE/on5Ys5oG0hDIQ=', 'Content-Length': '0'} 2023-10-18T20:55:38.866 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a2b6f70ba950ffea-006530464a-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:55:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:38.867 DEBUG:rgw_multi.tests:created bucket=bofrnf-62 2023-10-18T20:55:38.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:55:38.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:55:38.869 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:55:38.924 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:55:38.924 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:55:38.956 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:55:38.972 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:55:38.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:55:38.973 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662538.863797_1313.1", 2023-10-18T20:55:38.973 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:55:38.863797Z" 2023-10-18T20:55:38.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:55:38.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:55:38.974 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:55:38.974 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T20:55:38.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:55:38.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:55:38.975 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:55:38.975 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:55:38.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:55:38.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:55:38.976 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:55:38.976 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T20:55:38.976 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:55:38.976 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:55:38.977 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697662538.863797_1313.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:55:38.978 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:55:38.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:55:38.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:55:38.979 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:55:39.029 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:55:39.029 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:55:39.058 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:39.057+0000 7f21974a2780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:55:39.059 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:39.058+0000 7f21974a2780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:55:39.071 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:55:39.072 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:55:39.072 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:55:39.072 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:55:39.073 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:55:39.073 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:55:39.073 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:55:39.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:55:39.073 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:55:39.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:55:39.074 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:55:39.074 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:55:39.074 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:55:39.075 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662538.863797_1313.1", 2023-10-18T20:55:39.075 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:55:39.075 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:55:39.075 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:55:39.075 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:55:38.863797Z", 2023-10-18T20:55:39.076 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:55:39.076 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:55:39.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:55:39.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:55:39.077 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:55:39.077 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:55:39.077 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:55:39.077 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:55:39.078 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:55:39.078 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:55:39.078 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:55:39.078 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:55:39.079 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:55:39.079 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:55:39.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:55:39.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:55:39.079 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:55:39.080 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:55:39.080 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:55:39.080 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:55:39.080 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:55:39.081 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:55:39.081 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:55:39.081 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:55:39.081 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:55:39.081 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:55:39.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:55:39.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:55:39.082 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:55:39.082 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:55:39.083 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:55:39.083 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:55:39.083 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:55:39.083 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:55:39.083 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:55:39.084 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:55:39.084 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:55:39.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:55:39.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:55:39.084 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:55:39.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:55:39.085 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:55:39.085 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:55:39.085 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:55:39.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:55:39.086 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:55:39Z" 2023-10-18T20:55:39.086 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:55:39.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697662538.863797_1313.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:55:38.863797Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:55:39Z" } 2023-10-18T20:55:39.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697662538.863797_1313.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:55:38.863797Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:55:39.088 DEBUG:rgw_multi.tests:log_status={0: '1_1697662538.863797_1313.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:55:39.088 DEBUG:rgw_multi.tests:sync_status={0: '1_1697662538.863797_1313.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:55:39.089 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:55:39.089 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:39.090 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:39.090 DEBUG:boto:Method: HEAD 2023-10-18T20:55:39.091 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:55:39.091 DEBUG:boto:Data: 2023-10-18T20:55:39.091 DEBUG:boto:Headers: {} 2023-10-18T20:55:39.092 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:39.092 DEBUG:boto:Port: 8000 2023-10-18T20:55:39.093 DEBUG:boto:Params: {} 2023-10-18T20:55:39.093 DEBUG:boto:Token: None 2023-10-18T20:55:39.094 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:39 GMT /bofrnf-62/ 2023-10-18T20:55:39.094 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KbfbHTg0Cu20Ui3xtbMlye+qTLY= 2023-10-18T20:55:39.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KbfbHTg0Cu20Ui3xtbMlye+qTLY=', 'Content-Length': '0'} 2023-10-18T20:55:39.097 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006db6768d956a662e-006530464b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:39.098 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:39.098 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:39.099 DEBUG:boto:Method: PUT 2023-10-18T20:55:39.099 DEBUG:boto:Path: /bofrnf-62/dummy 2023-10-18T20:55:39.100 DEBUG:boto:Data: 2023-10-18T20:55:39.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:55:39.100 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:39.102 DEBUG:boto:Port: 8000 2023-10-18T20:55:39.102 DEBUG:boto:Params: {} 2023-10-18T20:55:39.103 DEBUG:boto:Token: None 2023-10-18T20:55:39.103 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:55:39 GMT /bofrnf-62/dummy 2023-10-18T20:55:39.104 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uGKHwshWiwmDjcb3rtSaqN4tTAI= 2023-10-18T20:55:39.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:55:39 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uGKHwshWiwmDjcb3rtSaqN4tTAI='} 2023-10-18T20:55:39.111 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a566b0755bfd97aa-006530464b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:39.111 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-62 2023-10-18T20:55:39.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:55:39.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:55:39.113 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:55:39.169 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:55:39.169 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:55:39.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:39.202+0000 7f0afd440780 1 waiting to reach incremental sync.. 2023-10-18T20:55:44.205 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:44.204+0000 7f0afd440780 1 bucket sync caught up with source: 2023-10-18T20:55:44.206 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:55:44.206 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:55:44.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:44.204+0000 7f0afd440780 0 bucket checkpoint complete 2023-10-18T20:55:44.223 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-62 2023-10-18T20:55:44.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:55:44.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:55:44.225 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:55:44.288 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:55:44.288 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:55:44.326 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:44.325+0000 7fd0c32e6780 1 waiting to reach incremental sync.. 2023-10-18T20:55:49.328 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:49.326+0000 7fd0c32e6780 1 bucket sync caught up with source: 2023-10-18T20:55:49.328 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.615.6, , , , , ] 2023-10-18T20:55:49.329 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.615.6, , , , , ] 2023-10-18T20:55:49.329 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:49.326+0000 7fd0c32e6780 0 bucket checkpoint complete 2023-10-18T20:55:49.345 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-62 zones={a2, a1} 2023-10-18T20:55:49.346 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.346 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.347 DEBUG:boto:Method: HEAD 2023-10-18T20:55:49.347 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:55:49.348 DEBUG:boto:Data: 2023-10-18T20:55:49.348 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.348 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.349 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.349 DEBUG:boto:Params: {} 2023-10-18T20:55:49.350 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:55:49.351 DEBUG:boto:Token: None 2023-10-18T20:55:49.351 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/ 2023-10-18T20:55:49.352 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VzeZVkEFnid+zBQM5KVm3T52bhs= 2023-10-18T20:55:49.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VzeZVkEFnid+zBQM5KVm3T52bhs=', 'Content-Length': '0'} 2023-10-18T20:55:49.357 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d45639fa009b1a8e-0065304655-1eec-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.358 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.358 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.359 DEBUG:boto:Method: HEAD 2023-10-18T20:55:49.359 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:55:49.360 DEBUG:boto:Data: 2023-10-18T20:55:49.360 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.360 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.361 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.361 DEBUG:boto:Params: {} 2023-10-18T20:55:49.362 DEBUG:boto:Token: None 2023-10-18T20:55:49.362 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/ 2023-10-18T20:55:49.363 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VzeZVkEFnid+zBQM5KVm3T52bhs= 2023-10-18T20:55:49.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VzeZVkEFnid+zBQM5KVm3T52bhs=', 'Content-Length': '0'} 2023-10-18T20:55:49.366 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e22c3946021d693-0065304655-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.367 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:55:49.368 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.368 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.369 DEBUG:boto:path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.370 DEBUG:boto:auth_path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.371 DEBUG:boto:Method: GET 2023-10-18T20:55:49.371 DEBUG:boto:Path: /bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.372 DEBUG:boto:Data: 2023-10-18T20:55:49.373 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.374 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.375 DEBUG:boto:Params: {} 2023-10-18T20:55:49.375 DEBUG:boto:Token: None 2023-10-18T20:55:49.376 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/?versions 2023-10-18T20:55:49.376 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6f/2QEnI+BA/ryNbNj8IIEsx7Hg= 2023-10-18T20:55:49.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6f/2QEnI+BA/ryNbNj8IIEsx7Hg=', 'Content-Length': '0'} 2023-10-18T20:55:49.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba34eb1de79f1cdc-0065304655-1eec-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.380 DEBUG:boto:b'bofrnf-62999falsedummynulltrue2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:55:49.381 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T20:55:49.381 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:55:49.382 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.382 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.382 DEBUG:boto:path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.383 DEBUG:boto:auth_path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.384 DEBUG:boto:Method: GET 2023-10-18T20:55:49.384 DEBUG:boto:Path: /bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.384 DEBUG:boto:Data: 2023-10-18T20:55:49.385 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.385 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.386 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.386 DEBUG:boto:Params: {} 2023-10-18T20:55:49.387 DEBUG:boto:Token: None 2023-10-18T20:55:49.387 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/?versions 2023-10-18T20:55:49.388 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6f/2QEnI+BA/ryNbNj8IIEsx7Hg= 2023-10-18T20:55:49.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6f/2QEnI+BA/ryNbNj8IIEsx7Hg=', 'Content-Length': '0'} 2023-10-18T20:55:49.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e95df99d8be98189-0065304655-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.392 DEBUG:boto:b'bofrnf-62999falsedummynulltrue2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:55:49.393 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T20:55:49.394 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.394 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.395 DEBUG:boto:path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.396 DEBUG:boto:auth_path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.396 DEBUG:boto:Method: GET 2023-10-18T20:55:49.397 DEBUG:boto:Path: /bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.398 DEBUG:boto:Data: 2023-10-18T20:55:49.398 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.400 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.400 DEBUG:boto:Params: {} 2023-10-18T20:55:49.401 DEBUG:boto:Token: None 2023-10-18T20:55:49.401 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/?versions 2023-10-18T20:55:49.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6f/2QEnI+BA/ryNbNj8IIEsx7Hg= 2023-10-18T20:55:49.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6f/2QEnI+BA/ryNbNj8IIEsx7Hg=', 'Content-Length': '0'} 2023-10-18T20:55:49.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ced0f7693705a3fa-0065304655-1eec-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.405 DEBUG:boto:b'bofrnf-62999falsedummynulltrue2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:55:49.406 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.406 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.407 DEBUG:boto:path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.407 DEBUG:boto:auth_path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.407 DEBUG:boto:Method: GET 2023-10-18T20:55:49.408 DEBUG:boto:Path: /bofrnf-62/?versions&max-keys=999 2023-10-18T20:55:49.408 DEBUG:boto:Data: 2023-10-18T20:55:49.408 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.409 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.410 DEBUG:boto:Params: {} 2023-10-18T20:55:49.410 DEBUG:boto:Token: None 2023-10-18T20:55:49.410 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/?versions 2023-10-18T20:55:49.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6f/2QEnI+BA/ryNbNj8IIEsx7Hg= 2023-10-18T20:55:49.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6f/2QEnI+BA/ryNbNj8IIEsx7Hg=', 'Content-Length': '0'} 2023-10-18T20:55:49.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d03b12e76bc3537f-0065304655-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.414 DEBUG:boto:b'bofrnf-62999falsedummynulltrue2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:55:49.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T20:55:49.415 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.416 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.416 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.417 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.417 DEBUG:boto:Method: GET 2023-10-18T20:55:49.417 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.418 DEBUG:boto:Data: 2023-10-18T20:55:49.418 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.419 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.419 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.419 DEBUG:boto:Params: {} 2023-10-18T20:55:49.420 DEBUG:boto:Token: None 2023-10-18T20:55:49.420 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.421 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PeyXOsIxD+uXZ1MeKKLt2gzYVA8= 2023-10-18T20:55:49.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PeyXOsIxD+uXZ1MeKKLt2gzYVA8=', 'Content-Length': '0'} 2023-10-18T20:55:49.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba433b8c3ef5ef3-0065304655-1eec-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.424 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.425 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.425 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.425 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.426 DEBUG:boto:Method: GET 2023-10-18T20:55:49.426 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.427 DEBUG:boto:Data: 2023-10-18T20:55:49.427 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.427 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.428 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.428 DEBUG:boto:Params: {} 2023-10-18T20:55:49.429 DEBUG:boto:Token: None 2023-10-18T20:55:49.429 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.430 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PeyXOsIxD+uXZ1MeKKLt2gzYVA8= 2023-10-18T20:55:49.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PeyXOsIxD+uXZ1MeKKLt2gzYVA8=', 'Content-Length': '0'} 2023-10-18T20:55:49.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e76993bdedb993-0065304655-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.433 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.433 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.434 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.434 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.435 DEBUG:boto:Method: HEAD 2023-10-18T20:55:49.435 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.435 DEBUG:boto:Data: 2023-10-18T20:55:49.436 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.436 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.436 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.437 DEBUG:boto:Params: {} 2023-10-18T20:55:49.437 DEBUG:boto:Token: None 2023-10-18T20:55:49.438 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.438 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:trex0KWEIlbbuv8HZT/pZDVY/0Y= 2023-10-18T20:55:49.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:trex0KWEIlbbuv8HZT/pZDVY/0Y=', 'Content-Length': '0'} 2023-10-18T20:55:49.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000189177518a1b8ce8-0065304655-1eec-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.442 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.443 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.443 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.444 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.444 DEBUG:boto:Method: HEAD 2023-10-18T20:55:49.444 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.445 DEBUG:boto:Data: 2023-10-18T20:55:49.445 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.445 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.446 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.446 DEBUG:boto:Params: {} 2023-10-18T20:55:49.447 DEBUG:boto:Token: None 2023-10-18T20:55:49.447 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.448 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:trex0KWEIlbbuv8HZT/pZDVY/0Y= 2023-10-18T20:55:49.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:trex0KWEIlbbuv8HZT/pZDVY/0Y=', 'Content-Length': '0'} 2023-10-18T20:55:49.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001ad2d749a52048b-0065304655-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T20:55:49.451 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.452 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.452 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.452 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.453 DEBUG:boto:Method: GET 2023-10-18T20:55:49.453 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.454 DEBUG:boto:Data: 2023-10-18T20:55:49.454 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.454 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.455 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.455 DEBUG:boto:Params: {} 2023-10-18T20:55:49.456 DEBUG:boto:Token: None 2023-10-18T20:55:49.456 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.456 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PeyXOsIxD+uXZ1MeKKLt2gzYVA8= 2023-10-18T20:55:49.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PeyXOsIxD+uXZ1MeKKLt2gzYVA8=', 'Content-Length': '0'} 2023-10-18T20:55:49.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048a973c237390d34-0065304655-1eec-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.459 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.460 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.460 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.461 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.461 DEBUG:boto:Method: GET 2023-10-18T20:55:49.462 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.462 DEBUG:boto:Data: 2023-10-18T20:55:49.462 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.463 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.463 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.463 DEBUG:boto:Params: {} 2023-10-18T20:55:49.464 DEBUG:boto:Token: None 2023-10-18T20:55:49.464 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:55:49.465 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PeyXOsIxD+uXZ1MeKKLt2gzYVA8= 2023-10-18T20:55:49.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PeyXOsIxD+uXZ1MeKKLt2gzYVA8=', 'Content-Length': '0'} 2023-10-18T20:55:49.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1b6ff24a378c53d-0065304655-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.468 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.469 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.469 DEBUG:boto:path=/bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.469 DEBUG:boto:auth_path=/bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.470 DEBUG:boto:Method: GET 2023-10-18T20:55:49.470 DEBUG:boto:Path: /bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.471 DEBUG:boto:Data: 2023-10-18T20:55:49.471 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.472 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.472 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.472 DEBUG:boto:Params: {} 2023-10-18T20:55:49.473 DEBUG:boto:Token: None 2023-10-18T20:55:49.473 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/ 2023-10-18T20:55:49.474 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JvuqQQxGzfi0ycHp1k2OvNipyVE= 2023-10-18T20:55:49.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JvuqQQxGzfi0ycHp1k2OvNipyVE=', 'Content-Length': '0'} 2023-10-18T20:55:49.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6d8d3f78f2858e8-0065304655-1eec-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.522 DEBUG:boto:b'bofrnf-62dummy1000falsedummy2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:55:49.526 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.527 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.527 DEBUG:boto:path=/bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.528 DEBUG:boto:auth_path=/bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.528 DEBUG:boto:Method: GET 2023-10-18T20:55:49.528 DEBUG:boto:Path: /bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.529 DEBUG:boto:Data: 2023-10-18T20:55:49.529 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.530 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.531 DEBUG:boto:Params: {} 2023-10-18T20:55:49.531 DEBUG:boto:Token: None 2023-10-18T20:55:49.532 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/ 2023-10-18T20:55:49.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JvuqQQxGzfi0ycHp1k2OvNipyVE= 2023-10-18T20:55:49.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JvuqQQxGzfi0ycHp1k2OvNipyVE=', 'Content-Length': '0'} 2023-10-18T20:55:49.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e5efe2dc39e266-0065304655-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.536 DEBUG:boto:b'bofrnf-62dummy1000falsedummy2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:55:49.537 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.538 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.538 DEBUG:boto:Method: HEAD 2023-10-18T20:55:49.539 DEBUG:boto:Path: /bofrnf-62/dummy 2023-10-18T20:55:49.539 DEBUG:boto:Data: 2023-10-18T20:55:49.540 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.541 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.541 DEBUG:boto:Params: {} 2023-10-18T20:55:49.542 DEBUG:boto:Token: None 2023-10-18T20:55:49.542 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy 2023-10-18T20:55:49.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i8HIzzOhb8/dNC6C/MMZzfL2fb4= 2023-10-18T20:55:49.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i8HIzzOhb8/dNC6C/MMZzfL2fb4=', 'Content-Length': '0'} 2023-10-18T20:55:49.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a03c2697196a5b-0065304655-1eec-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.546 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.547 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.547 DEBUG:boto:Method: HEAD 2023-10-18T20:55:49.547 DEBUG:boto:Path: /bofrnf-62/dummy 2023-10-18T20:55:49.548 DEBUG:boto:Data: 2023-10-18T20:55:49.548 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.549 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.549 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.549 DEBUG:boto:Params: {} 2023-10-18T20:55:49.550 DEBUG:boto:Token: None 2023-10-18T20:55:49.550 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy 2023-10-18T20:55:49.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i8HIzzOhb8/dNC6C/MMZzfL2fb4= 2023-10-18T20:55:49.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i8HIzzOhb8/dNC6C/MMZzfL2fb4=', 'Content-Length': '0'} 2023-10-18T20:55:49.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a7485ef463aafb4-0065304655-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T20:55:49.555 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.555 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.555 DEBUG:boto:Method: GET 2023-10-18T20:55:49.556 DEBUG:boto:Path: /bofrnf-62/dummy 2023-10-18T20:55:49.556 DEBUG:boto:Data: 2023-10-18T20:55:49.557 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.557 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.558 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.558 DEBUG:boto:Params: {} 2023-10-18T20:55:49.559 DEBUG:boto:Token: None 2023-10-18T20:55:49.559 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy 2023-10-18T20:55:49.560 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0y9Qm/DoAW74/G0dGeTTNrAH10U= 2023-10-18T20:55:49.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0y9Qm/DoAW74/G0dGeTTNrAH10U=', 'Content-Length': '0'} 2023-10-18T20:55:49.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17c14a22c709640-0065304655-1eec-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.563 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:55:49.563 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:55:49.564 DEBUG:boto:Method: GET 2023-10-18T20:55:49.564 DEBUG:boto:Path: /bofrnf-62/dummy 2023-10-18T20:55:49.565 DEBUG:boto:Data: 2023-10-18T20:55:49.566 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.567 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.568 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.569 DEBUG:boto:Params: {} 2023-10-18T20:55:49.569 DEBUG:boto:Token: None 2023-10-18T20:55:49.570 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/dummy 2023-10-18T20:55:49.571 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0y9Qm/DoAW74/G0dGeTTNrAH10U= 2023-10-18T20:55:49.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0y9Qm/DoAW74/G0dGeTTNrAH10U=', 'Content-Length': '0'} 2023-10-18T20:55:49.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022915c3c60380418-0065304655-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.576 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.577 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.577 DEBUG:boto:path=/bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.578 DEBUG:boto:auth_path=/bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.579 DEBUG:boto:Method: GET 2023-10-18T20:55:49.580 DEBUG:boto:Path: /bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.581 DEBUG:boto:Data: 2023-10-18T20:55:49.582 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.582 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:49.583 DEBUG:boto:Port: 8001 2023-10-18T20:55:49.584 DEBUG:boto:Params: {} 2023-10-18T20:55:49.585 DEBUG:boto:Token: None 2023-10-18T20:55:49.586 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/ 2023-10-18T20:55:49.587 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JvuqQQxGzfi0ycHp1k2OvNipyVE= 2023-10-18T20:55:49.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JvuqQQxGzfi0ycHp1k2OvNipyVE=', 'Content-Length': '0'} 2023-10-18T20:55:49.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff41db2b7c347509-0065304655-1eec-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.591 DEBUG:boto:b'bofrnf-62dummy1000falsedummy2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:55:49.593 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:49.593 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:49.594 DEBUG:boto:path=/bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.595 DEBUG:boto:auth_path=/bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.596 DEBUG:boto:Method: GET 2023-10-18T20:55:49.596 DEBUG:boto:Path: /bofrnf-62/?prefix=dummy 2023-10-18T20:55:49.597 DEBUG:boto:Data: 2023-10-18T20:55:49.598 DEBUG:boto:Headers: {} 2023-10-18T20:55:49.598 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:49.599 DEBUG:boto:Port: 8000 2023-10-18T20:55:49.600 DEBUG:boto:Params: {} 2023-10-18T20:55:49.601 DEBUG:boto:Token: None 2023-10-18T20:55:49.602 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:55:49 GMT /bofrnf-62/ 2023-10-18T20:55:49.602 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:JvuqQQxGzfi0ycHp1k2OvNipyVE= 2023-10-18T20:55:49.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JvuqQQxGzfi0ycHp1k2OvNipyVE=', 'Content-Length': '0'} 2023-10-18T20:55:49.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006422d101c305dde3-0065304655-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:49.646 DEBUG:boto:b'bofrnf-62dummy1000falsedummy2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:55:49.647 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-62 zones={a2, a1} 2023-10-18T20:55:49.648 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-18T20:55:49.648 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T20:55:49.650 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:55:49.648+0000 7fec29d1c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 175389) UID: 0 2023-10-18T20:55:49.651 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T20:55:49.648+0000 7fec2df857c0 -1 shutting down 2023-10-18T20:55:53.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T20:55:55.747 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-18T20:55:55.748 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:55.748 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:55.749 DEBUG:boto:Method: HEAD 2023-10-18T20:55:55.749 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:55:55.750 DEBUG:boto:Data: 2023-10-18T20:55:55.750 DEBUG:boto:Headers: {} 2023-10-18T20:55:55.750 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.751 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.751 DEBUG:boto:Params: {} 2023-10-18T20:55:55.752 DEBUG:boto:Token: None 2023-10-18T20:55:55.752 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/ 2023-10-18T20:55:55.752 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA= 2023-10-18T20:55:55.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA=', 'Content-Length': '0'} 2023-10-18T20:55:55.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049b6dc10e179a21f-006530465b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.757 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:55:55.757 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:55:55.757 DEBUG:boto:Method: PUT 2023-10-18T20:55:55.758 DEBUG:boto:Path: /bofrnf-62/obj11 2023-10-18T20:55:55.758 DEBUG:boto:Data: 2023-10-18T20:55:55.759 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:55:55.759 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.759 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.760 DEBUG:boto:Params: {} 2023-10-18T20:55:55.760 DEBUG:boto:Token: None 2023-10-18T20:55:55.761 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/obj11 2023-10-18T20:55:55.761 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ir2sT6qfu/VNo6qPxf7soTgHQ8A= 2023-10-18T20:55:55.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ir2sT6qfu/VNo6qPxf7soTgHQ8A='} 2023-10-18T20:55:55.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000132b21431c7d3bad-006530465b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.769 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:55.769 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:55.770 DEBUG:boto:Method: HEAD 2023-10-18T20:55:55.770 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:55:55.771 DEBUG:boto:Data: 2023-10-18T20:55:55.771 DEBUG:boto:Headers: {} 2023-10-18T20:55:55.771 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.772 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.772 DEBUG:boto:Params: {} 2023-10-18T20:55:55.773 DEBUG:boto:Token: None 2023-10-18T20:55:55.773 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/ 2023-10-18T20:55:55.774 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA= 2023-10-18T20:55:55.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA=', 'Content-Length': '0'} 2023-10-18T20:55:55.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2fc74d3acc5688a-006530465b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.777 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:55:55.778 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:55:55.778 DEBUG:boto:Method: PUT 2023-10-18T20:55:55.778 DEBUG:boto:Path: /bofrnf-62/obj22 2023-10-18T20:55:55.779 DEBUG:boto:Data: 2023-10-18T20:55:55.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:55:55.780 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.780 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.780 DEBUG:boto:Params: {} 2023-10-18T20:55:55.781 DEBUG:boto:Token: None 2023-10-18T20:55:55.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/obj22 2023-10-18T20:55:55.782 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vtMf0lJaSXRETQPDnnvXrl5aG/I= 2023-10-18T20:55:55.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vtMf0lJaSXRETQPDnnvXrl5aG/I='} 2023-10-18T20:55:55.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052e5d342118b32b4-006530465b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.788 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:55.788 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:55.788 DEBUG:boto:Method: HEAD 2023-10-18T20:55:55.789 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:55:55.789 DEBUG:boto:Data: 2023-10-18T20:55:55.790 DEBUG:boto:Headers: {} 2023-10-18T20:55:55.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.790 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.791 DEBUG:boto:Params: {} 2023-10-18T20:55:55.791 DEBUG:boto:Token: None 2023-10-18T20:55:55.792 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/ 2023-10-18T20:55:55.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA= 2023-10-18T20:55:55.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA=', 'Content-Length': '0'} 2023-10-18T20:55:55.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a95fab8cea72f56-006530465b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.796 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:55:55.796 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:55:55.797 DEBUG:boto:Method: PUT 2023-10-18T20:55:55.797 DEBUG:boto:Path: /bofrnf-62/obj33 2023-10-18T20:55:55.797 DEBUG:boto:Data: 2023-10-18T20:55:55.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:55:55.798 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.799 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.799 DEBUG:boto:Params: {} 2023-10-18T20:55:55.799 DEBUG:boto:Token: None 2023-10-18T20:55:55.800 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/obj33 2023-10-18T20:55:55.800 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KH52Rudr6f0SbMDLXeekWCwcXOc= 2023-10-18T20:55:55.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KH52Rudr6f0SbMDLXeekWCwcXOc='} 2023-10-18T20:55:55.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000137757eb327d4231-006530465b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.847 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:55.848 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:55.848 DEBUG:boto:Method: HEAD 2023-10-18T20:55:55.848 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:55:55.849 DEBUG:boto:Data: 2023-10-18T20:55:55.849 DEBUG:boto:Headers: {} 2023-10-18T20:55:55.849 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.850 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.850 DEBUG:boto:Params: {} 2023-10-18T20:55:55.851 DEBUG:boto:Token: None 2023-10-18T20:55:55.851 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/ 2023-10-18T20:55:55.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA= 2023-10-18T20:55:55.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA=', 'Content-Length': '0'} 2023-10-18T20:55:55.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000852d7549c22ad7e4-006530465b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.855 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:55:55.855 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:55:55.856 DEBUG:boto:Method: PUT 2023-10-18T20:55:55.856 DEBUG:boto:Path: /bofrnf-62/obj44 2023-10-18T20:55:55.856 DEBUG:boto:Data: 2023-10-18T20:55:55.857 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:55:55.857 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.858 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.858 DEBUG:boto:Params: {} 2023-10-18T20:55:55.858 DEBUG:boto:Token: None 2023-10-18T20:55:55.859 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/obj44 2023-10-18T20:55:55.859 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GEErOy5dPnZyBmmwJOvLddEWmz8= 2023-10-18T20:55:55.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GEErOy5dPnZyBmmwJOvLddEWmz8='} 2023-10-18T20:55:55.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f7ad35373a73fc3-006530465b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.866 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:55:55.867 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:55:55.867 DEBUG:boto:Method: HEAD 2023-10-18T20:55:55.868 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:55:55.868 DEBUG:boto:Data: 2023-10-18T20:55:55.868 DEBUG:boto:Headers: {} 2023-10-18T20:55:55.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.869 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.870 DEBUG:boto:Params: {} 2023-10-18T20:55:55.870 DEBUG:boto:Token: None 2023-10-18T20:55:55.871 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/ 2023-10-18T20:55:55.871 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA= 2023-10-18T20:55:55.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xpua6KcPInAS2LBokQEIqz844pA=', 'Content-Length': '0'} 2023-10-18T20:55:55.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef6f70dfbcfcc6d6-006530465b-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.875 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:55:55.875 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:55:55.875 DEBUG:boto:Method: PUT 2023-10-18T20:55:55.876 DEBUG:boto:Path: /bofrnf-62/obj55 2023-10-18T20:55:55.876 DEBUG:boto:Data: 2023-10-18T20:55:55.877 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:55:55.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:55:55.877 DEBUG:boto:Port: 8000 2023-10-18T20:55:55.878 DEBUG:boto:Params: {} 2023-10-18T20:55:55.878 DEBUG:boto:Token: None 2023-10-18T20:55:55.882 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:55:55 GMT /bofrnf-62/obj55 2023-10-18T20:55:55.882 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:84zZfzqElamPhydksxZH0ZTq4vE= 2023-10-18T20:55:55.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:55:55 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:84zZfzqElamPhydksxZH0ZTq4vE='} 2023-10-18T20:55:55.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a67bb3046c5fc4c2-006530465b-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:55:55.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:55:55.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:55:55.928 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:55:55.985 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:55:55.985 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:55:56.019 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:56.017+0000 7f1f74894780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-18T20:55:56.020 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:56.019+0000 7f1f74894780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-18T20:55:56.042 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-18T20:55:56.042 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T20:55:56.046 INFO:tasks.rgw.c1.client.1:Started 2023-10-18T20:55:56.047 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:56.068 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:55:56.069 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:55:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T20:55:56.071 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T20:55:57.072 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T20:55:57.094 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T20:55:57.095 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T20:55:57.096 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T20:55:57.096 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T20:55:57.097 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-62 2023-10-18T20:55:57.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:55:57.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:55:57.098 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:55:57.193 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:55:57.193 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:55:57.230 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:55:57.228+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:55:57.230 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:55:57.230 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:55:59.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-18T20:56:02.233 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:02.230+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:02.234 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:02.234 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:07.232 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:07.231+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:07.233 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:07.233 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:12.234 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:12.231+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:12.234 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:12.234 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:17.235 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:17.232+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:17.236 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:17.236 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:22.236 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:22.234+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:22.236 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:22.237 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:27.237 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:27.236+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:27.238 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:27.238 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:32.238 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:32.237+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:32.239 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:32.239 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:37.240 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:37.238+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:37.240 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:37.240 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:42.241 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:42.240+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:42.241 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:42.242 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:47.242 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:47.241+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:47.243 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:47.243 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:52.244 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:52.242+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:52.244 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:52.244 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:56:57.245 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:56:57.243+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:56:57.245 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:56:57.246 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:02.246 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:02.244+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:02.247 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:57:02.247 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:07.247 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:07.245+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:07.248 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:57:07.248 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:12.249 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:12.246+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:12.249 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:57:12.250 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:17.250 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:17.249+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:17.251 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:57:17.251 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:22.251 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:22.250+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:22.252 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:57:22.252 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:27.252 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:27.251+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:27.253 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , ] 2023-10-18T20:57:27.253 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:32.254 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:32.252+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:32.254 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , 00000000001.77.6] 2023-10-18T20:57:32.254 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:37.255 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:37.254+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:37.256 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , 00000000001.77.6] 2023-10-18T20:57:37.256 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:42.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:42.255+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:42.257 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , 00000000001.77.6] 2023-10-18T20:57:42.257 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:47.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:47.256+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:47.258 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , 00000000001.77.6] 2023-10-18T20:57:47.259 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:52.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:52.258+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:52.260 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , , 00000000001.77.6] 2023-10-18T20:57:52.260 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:57:57.260 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:57:57.258+0000 7f69f47c8780 1 waiting for incremental sync to catch up: 2023-10-18T20:57:57.261 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , , 00000000002.156.6, 00000000001.77.6] 2023-10-18T20:57:57.261 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:58:02.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:02.259+0000 7f69f47c8780 1 bucket sync caught up with source: 2023-10-18T20:58:02.262 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:58:02.262 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.152.6, , , 00000000001.100.6, 00000000002.156.6, 00000000002.79.6] 2023-10-18T20:58:02.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:02.259+0000 7f69f47c8780 0 bucket checkpoint complete 2023-10-18T20:58:02.279 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-62 2023-10-18T20:58:02.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:58:02.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:58:02.281 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:58:02.343 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:02.343 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:02.380 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:02.378+0000 7fd5b7664780 1 waiting for incremental sync to catch up: 2023-10-18T20:58:02.381 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.615.6, , , , , 00000000001.1424.6] 2023-10-18T20:58:02.381 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.615.6, , , 00000000001.968.6, 00000000002.619.6, 00000000002.1426.6] 2023-10-18T20:58:07.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:07.380+0000 7fd5b7664780 1 bucket sync caught up with source: 2023-10-18T20:58:07.382 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , 00000000001.615.6, , , 00000000001.968.6, 00000000002.619.6, 00000000002.1426.6] 2023-10-18T20:58:07.383 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , 00000000001.615.6, , , 00000000001.968.6, 00000000002.619.6, 00000000002.1426.6] 2023-10-18T20:58:07.383 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:07.380+0000 7fd5b7664780 0 bucket checkpoint complete 2023-10-18T20:58:07.399 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-62 zones={a2, a1} 2023-10-18T20:58:07.400 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.400 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.401 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.401 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:58:07.401 DEBUG:boto:Data: 2023-10-18T20:58:07.402 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.402 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.403 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.403 DEBUG:boto:Params: {} 2023-10-18T20:58:07.404 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T20:58:07.404 DEBUG:boto:Token: None 2023-10-18T20:58:07.405 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.405 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s3Ur3o+C/BNkKDC1FIwKlQxaHN0= 2023-10-18T20:58:07.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s3Ur3o+C/BNkKDC1FIwKlQxaHN0=', 'Content-Length': '0'} 2023-10-18T20:58:07.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016ed0ede6f2280fb-00653046df-1f24-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.413 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.413 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.414 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.414 DEBUG:boto:Path: /bofrnf-62/ 2023-10-18T20:58:07.414 DEBUG:boto:Data: 2023-10-18T20:58:07.415 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.415 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.416 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.416 DEBUG:boto:Params: {} 2023-10-18T20:58:07.416 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T20:58:07.417 DEBUG:boto:Token: None 2023-10-18T20:58:07.417 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.418 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:s3Ur3o+C/BNkKDC1FIwKlQxaHN0= 2023-10-18T20:58:07.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s3Ur3o+C/BNkKDC1FIwKlQxaHN0=', 'Content-Length': '0'} 2023-10-18T20:58:07.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054b5a1d3ee51a0c4-00653046df-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.423 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T20:58:07.423 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.424 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.424 DEBUG:boto:path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.424 DEBUG:boto:auth_path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.425 DEBUG:boto:Method: GET 2023-10-18T20:58:07.425 DEBUG:boto:Path: /bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.426 DEBUG:boto:Data: 2023-10-18T20:58:07.426 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.426 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.427 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.427 DEBUG:boto:Params: {} 2023-10-18T20:58:07.428 DEBUG:boto:Token: None 2023-10-18T20:58:07.428 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/?versions 2023-10-18T20:58:07.429 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6On19EvF+NXYXuICIYxw+/gr5jI= 2023-10-18T20:58:07.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6On19EvF+NXYXuICIYxw+/gr5jI=', 'Content-Length': '0'} 2023-10-18T20:58:07.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087465622186aa220-00653046df-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.432 DEBUG:boto:b'bofrnf-62999falsedummynulltrue2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-18T20:55:55.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:55:55.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:55:55.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:55:55.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:55:55.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.433 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T20:58:07.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:58:07.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:58:07.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:58:07.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:58:07.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:58:07.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T20:58:07.436 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.436 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.437 DEBUG:boto:path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.437 DEBUG:boto:auth_path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.437 DEBUG:boto:Method: GET 2023-10-18T20:58:07.438 DEBUG:boto:Path: /bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.438 DEBUG:boto:Data: 2023-10-18T20:58:07.439 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.439 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.439 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.440 DEBUG:boto:Params: {} 2023-10-18T20:58:07.440 DEBUG:boto:Token: None 2023-10-18T20:58:07.441 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/?versions 2023-10-18T20:58:07.441 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6On19EvF+NXYXuICIYxw+/gr5jI= 2023-10-18T20:58:07.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6On19EvF+NXYXuICIYxw+/gr5jI=', 'Content-Length': '0'} 2023-10-18T20:58:07.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f5c54905012771-00653046df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.445 DEBUG:boto:b'bofrnf-62999falsedummynulltrue2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-18T20:55:55.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:55:55.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:55:55.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:55:55.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:55:55.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.446 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-18T20:58:07.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T20:58:07.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T20:58:07.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T20:58:07.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T20:58:07.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T20:58:07.449 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.449 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.450 DEBUG:boto:path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.450 DEBUG:boto:auth_path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.451 DEBUG:boto:Method: GET 2023-10-18T20:58:07.451 DEBUG:boto:Path: /bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.452 DEBUG:boto:Data: 2023-10-18T20:58:07.452 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.452 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.453 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.453 DEBUG:boto:Params: {} 2023-10-18T20:58:07.454 DEBUG:boto:Token: None 2023-10-18T20:58:07.454 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/?versions 2023-10-18T20:58:07.455 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6On19EvF+NXYXuICIYxw+/gr5jI= 2023-10-18T20:58:07.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6On19EvF+NXYXuICIYxw+/gr5jI=', 'Content-Length': '0'} 2023-10-18T20:58:07.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002ef366441fb2990-00653046df-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.458 DEBUG:boto:b'bofrnf-62999falsedummynulltrue2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-18T20:55:55.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:55:55.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:55:55.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:55:55.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:55:55.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.459 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.459 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.460 DEBUG:boto:path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.461 DEBUG:boto:auth_path=/bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.461 DEBUG:boto:Method: GET 2023-10-18T20:58:07.461 DEBUG:boto:Path: /bofrnf-62/?versions&max-keys=999 2023-10-18T20:58:07.462 DEBUG:boto:Data: 2023-10-18T20:58:07.462 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.463 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.463 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.463 DEBUG:boto:Params: {} 2023-10-18T20:58:07.464 DEBUG:boto:Token: None 2023-10-18T20:58:07.464 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/?versions 2023-10-18T20:58:07.465 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6On19EvF+NXYXuICIYxw+/gr5jI= 2023-10-18T20:58:07.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6On19EvF+NXYXuICIYxw+/gr5jI=', 'Content-Length': '0'} 2023-10-18T20:58:07.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa152d9700762e4c-00653046df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.468 DEBUG:boto:b'bofrnf-62999falsedummynulltrue2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-18T20:55:55.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T20:55:55.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T20:55:55.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T20:55:55.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T20:55:55.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T20:58:07.470 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.470 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.471 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.471 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.472 DEBUG:boto:Method: GET 2023-10-18T20:58:07.472 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.473 DEBUG:boto:Data: 2023-10-18T20:58:07.473 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.473 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.474 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.474 DEBUG:boto:Params: {} 2023-10-18T20:58:07.475 DEBUG:boto:Token: None 2023-10-18T20:58:07.475 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.476 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BTQ2uNQc9fFhIwBA+UCtnAFvkdk= 2023-10-18T20:58:07.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BTQ2uNQc9fFhIwBA+UCtnAFvkdk=', 'Content-Length': '0'} 2023-10-18T20:58:07.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001adff8ff61291592-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.480 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.480 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.481 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.481 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.482 DEBUG:boto:Method: GET 2023-10-18T20:58:07.482 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.483 DEBUG:boto:Data: 2023-10-18T20:58:07.484 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.484 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.484 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.485 DEBUG:boto:Params: {} 2023-10-18T20:58:07.486 DEBUG:boto:Token: None 2023-10-18T20:58:07.486 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.487 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BTQ2uNQc9fFhIwBA+UCtnAFvkdk= 2023-10-18T20:58:07.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BTQ2uNQc9fFhIwBA+UCtnAFvkdk=', 'Content-Length': '0'} 2023-10-18T20:58:07.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040ad092f3db8fd85-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.490 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.491 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.491 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.492 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.492 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.493 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.493 DEBUG:boto:Data: 2023-10-18T20:58:07.493 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.494 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.494 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.495 DEBUG:boto:Params: {} 2023-10-18T20:58:07.495 DEBUG:boto:Token: None 2023-10-18T20:58:07.496 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.496 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M6N4s5KPxPZvoaDoq4IbEOmcsZA= 2023-10-18T20:58:07.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M6N4s5KPxPZvoaDoq4IbEOmcsZA=', 'Content-Length': '0'} 2023-10-18T20:58:07.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006596e4f414c8bafa-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.499 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.500 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.500 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.500 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.501 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.501 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.502 DEBUG:boto:Data: 2023-10-18T20:58:07.502 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.502 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.503 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.503 DEBUG:boto:Params: {} 2023-10-18T20:58:07.504 DEBUG:boto:Token: None 2023-10-18T20:58:07.504 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.504 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:M6N4s5KPxPZvoaDoq4IbEOmcsZA= 2023-10-18T20:58:07.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:M6N4s5KPxPZvoaDoq4IbEOmcsZA=', 'Content-Length': '0'} 2023-10-18T20:58:07.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017a0b7d41286f92a-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T20:58:07.508 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.508 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.509 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.509 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.510 DEBUG:boto:Method: GET 2023-10-18T20:58:07.510 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.511 DEBUG:boto:Data: 2023-10-18T20:58:07.511 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.512 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.512 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.512 DEBUG:boto:Params: {} 2023-10-18T20:58:07.513 DEBUG:boto:Token: None 2023-10-18T20:58:07.513 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.514 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BTQ2uNQc9fFhIwBA+UCtnAFvkdk= 2023-10-18T20:58:07.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BTQ2uNQc9fFhIwBA+UCtnAFvkdk=', 'Content-Length': '0'} 2023-10-18T20:58:07.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027a3beb23feae168-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.558 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.559 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.559 DEBUG:boto:path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.560 DEBUG:boto:auth_path=/bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.560 DEBUG:boto:Method: GET 2023-10-18T20:58:07.561 DEBUG:boto:Path: /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.561 DEBUG:boto:Data: 2023-10-18T20:58:07.561 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.562 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.562 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.563 DEBUG:boto:Params: {} 2023-10-18T20:58:07.563 DEBUG:boto:Token: None 2023-10-18T20:58:07.564 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy?versionId=null 2023-10-18T20:58:07.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BTQ2uNQc9fFhIwBA+UCtnAFvkdk= 2023-10-18T20:58:07.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BTQ2uNQc9fFhIwBA+UCtnAFvkdk=', 'Content-Length': '0'} 2023-10-18T20:58:07.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f5cada7fda78c61-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.567 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.568 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.568 DEBUG:boto:path=/bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.569 DEBUG:boto:auth_path=/bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.569 DEBUG:boto:Method: GET 2023-10-18T20:58:07.570 DEBUG:boto:Path: /bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.570 DEBUG:boto:Data: 2023-10-18T20:58:07.571 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.571 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.571 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.572 DEBUG:boto:Params: {} 2023-10-18T20:58:07.572 DEBUG:boto:Token: None 2023-10-18T20:58:07.573 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.573 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4= 2023-10-18T20:58:07.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4=', 'Content-Length': '0'} 2023-10-18T20:58:07.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c5566bd044420ff-00653046df-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.577 DEBUG:boto:b'bofrnf-62dummy1000falsedummy2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.578 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.578 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.579 DEBUG:boto:path=/bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.579 DEBUG:boto:auth_path=/bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.579 DEBUG:boto:Method: GET 2023-10-18T20:58:07.580 DEBUG:boto:Path: /bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.580 DEBUG:boto:Data: 2023-10-18T20:58:07.581 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.581 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.581 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.582 DEBUG:boto:Params: {} 2023-10-18T20:58:07.582 DEBUG:boto:Token: None 2023-10-18T20:58:07.583 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.583 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4= 2023-10-18T20:58:07.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4=', 'Content-Length': '0'} 2023-10-18T20:58:07.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c498f53b19d8413-00653046df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.587 DEBUG:boto:b'bofrnf-62dummy1000falsedummy2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.588 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.588 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.588 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.589 DEBUG:boto:Path: /bofrnf-62/dummy 2023-10-18T20:58:07.589 DEBUG:boto:Data: 2023-10-18T20:58:07.590 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.590 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.591 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.591 DEBUG:boto:Params: {} 2023-10-18T20:58:07.591 DEBUG:boto:Token: None 2023-10-18T20:58:07.592 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy 2023-10-18T20:58:07.592 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6lygN7FOlqmF8UHpzBzoYgGbHmU= 2023-10-18T20:58:07.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6lygN7FOlqmF8UHpzBzoYgGbHmU=', 'Content-Length': '0'} 2023-10-18T20:58:07.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1cc0fbd510e4f6f-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.596 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.596 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.596 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.597 DEBUG:boto:Path: /bofrnf-62/dummy 2023-10-18T20:58:07.597 DEBUG:boto:Data: 2023-10-18T20:58:07.598 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.598 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.599 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.599 DEBUG:boto:Params: {} 2023-10-18T20:58:07.599 DEBUG:boto:Token: None 2023-10-18T20:58:07.600 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy 2023-10-18T20:58:07.600 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6lygN7FOlqmF8UHpzBzoYgGbHmU= 2023-10-18T20:58:07.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6lygN7FOlqmF8UHpzBzoYgGbHmU=', 'Content-Length': '0'} 2023-10-18T20:58:07.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7b442d798653b9-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-18T20:58:07.603 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.604 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.604 DEBUG:boto:Method: GET 2023-10-18T20:58:07.605 DEBUG:boto:Path: /bofrnf-62/dummy 2023-10-18T20:58:07.605 DEBUG:boto:Data: 2023-10-18T20:58:07.606 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.606 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.606 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.607 DEBUG:boto:Params: {} 2023-10-18T20:58:07.607 DEBUG:boto:Token: None 2023-10-18T20:58:07.608 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy 2023-10-18T20:58:07.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WhXSVB96+z9KJgWGo5IJrFBvIwM= 2023-10-18T20:58:07.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WhXSVB96+z9KJgWGo5IJrFBvIwM=', 'Content-Length': '0'} 2023-10-18T20:58:07.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d27cf9e24c028b5e-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.612 DEBUG:boto:path=/bofrnf-62/dummy 2023-10-18T20:58:07.612 DEBUG:boto:auth_path=/bofrnf-62/dummy 2023-10-18T20:58:07.612 DEBUG:boto:Method: GET 2023-10-18T20:58:07.613 DEBUG:boto:Path: /bofrnf-62/dummy 2023-10-18T20:58:07.613 DEBUG:boto:Data: 2023-10-18T20:58:07.614 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.614 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.615 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.615 DEBUG:boto:Params: {} 2023-10-18T20:58:07.615 DEBUG:boto:Token: None 2023-10-18T20:58:07.616 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/dummy 2023-10-18T20:58:07.616 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WhXSVB96+z9KJgWGo5IJrFBvIwM= 2023-10-18T20:58:07.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WhXSVB96+z9KJgWGo5IJrFBvIwM=', 'Content-Length': '0'} 2023-10-18T20:58:07.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007562c295f820c202-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.662 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.663 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.663 DEBUG:boto:path=/bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.664 DEBUG:boto:auth_path=/bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.664 DEBUG:boto:Method: GET 2023-10-18T20:58:07.664 DEBUG:boto:Path: /bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.665 DEBUG:boto:Data: 2023-10-18T20:58:07.665 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.666 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.666 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.667 DEBUG:boto:Params: {} 2023-10-18T20:58:07.667 DEBUG:boto:Token: None 2023-10-18T20:58:07.668 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4= 2023-10-18T20:58:07.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4=', 'Content-Length': '0'} 2023-10-18T20:58:07.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b243f73c4f3c58e-00653046df-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.672 DEBUG:boto:b'bofrnf-62dummy1000falsedummy2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.673 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.673 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.674 DEBUG:boto:path=/bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.674 DEBUG:boto:auth_path=/bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.674 DEBUG:boto:Method: GET 2023-10-18T20:58:07.675 DEBUG:boto:Path: /bofrnf-62/?prefix=dummy 2023-10-18T20:58:07.675 DEBUG:boto:Data: 2023-10-18T20:58:07.676 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.676 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.677 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.677 DEBUG:boto:Params: {} 2023-10-18T20:58:07.677 DEBUG:boto:Token: None 2023-10-18T20:58:07.678 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.678 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4= 2023-10-18T20:58:07.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4=', 'Content-Length': '0'} 2023-10-18T20:58:07.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e41bbdc42243f0d-00653046df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.682 DEBUG:boto:b'bofrnf-62dummy1000falsedummy2023-10-18T20:55:39.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:58:07.683 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.684 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.684 DEBUG:boto:path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.684 DEBUG:boto:auth_path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.685 DEBUG:boto:Method: GET 2023-10-18T20:58:07.685 DEBUG:boto:Path: /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.686 DEBUG:boto:Data: 2023-10-18T20:58:07.686 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.686 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.687 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.687 DEBUG:boto:Params: {} 2023-10-18T20:58:07.688 DEBUG:boto:Token: None 2023-10-18T20:58:07.688 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.689 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WAOoVueefWOIDaW5ZyqTrpKnGE8= 2023-10-18T20:58:07.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WAOoVueefWOIDaW5ZyqTrpKnGE8=', 'Content-Length': '0'} 2023-10-18T20:58:07.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f838677e9d5af41-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.734 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.735 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.735 DEBUG:boto:path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.736 DEBUG:boto:auth_path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.736 DEBUG:boto:Method: GET 2023-10-18T20:58:07.737 DEBUG:boto:Path: /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.737 DEBUG:boto:Data: 2023-10-18T20:58:07.737 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.738 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.738 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.739 DEBUG:boto:Params: {} 2023-10-18T20:58:07.739 DEBUG:boto:Token: None 2023-10-18T20:58:07.740 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.740 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WAOoVueefWOIDaW5ZyqTrpKnGE8= 2023-10-18T20:58:07.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WAOoVueefWOIDaW5ZyqTrpKnGE8=', 'Content-Length': '0'} 2023-10-18T20:58:07.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000508ca2d0bff95ca2-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.743 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.744 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.744 DEBUG:boto:path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.745 DEBUG:boto:auth_path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.745 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.745 DEBUG:boto:Path: /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.746 DEBUG:boto:Data: 2023-10-18T20:58:07.746 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.747 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.747 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.748 DEBUG:boto:Params: {} 2023-10-18T20:58:07.748 DEBUG:boto:Token: None 2023-10-18T20:58:07.748 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lq4grjHtwE8V+Rb+RQ2YhUgR0bw= 2023-10-18T20:58:07.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lq4grjHtwE8V+Rb+RQ2YhUgR0bw=', 'Content-Length': '0'} 2023-10-18T20:58:07.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000971192f6294b0541-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.752 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.752 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.753 DEBUG:boto:path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.753 DEBUG:boto:auth_path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.753 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.754 DEBUG:boto:Path: /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.754 DEBUG:boto:Data: 2023-10-18T20:58:07.755 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.755 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.755 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.756 DEBUG:boto:Params: {} 2023-10-18T20:58:07.756 DEBUG:boto:Token: None 2023-10-18T20:58:07.757 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.757 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lq4grjHtwE8V+Rb+RQ2YhUgR0bw= 2023-10-18T20:58:07.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lq4grjHtwE8V+Rb+RQ2YhUgR0bw=', 'Content-Length': '0'} 2023-10-18T20:58:07.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e30decf9c24de7a7-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:58:07.760 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.761 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.761 DEBUG:boto:path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.762 DEBUG:boto:auth_path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.762 DEBUG:boto:Method: GET 2023-10-18T20:58:07.763 DEBUG:boto:Path: /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.763 DEBUG:boto:Data: 2023-10-18T20:58:07.764 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.764 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.764 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.765 DEBUG:boto:Params: {} 2023-10-18T20:58:07.765 DEBUG:boto:Token: None 2023-10-18T20:58:07.766 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.766 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WAOoVueefWOIDaW5ZyqTrpKnGE8= 2023-10-18T20:58:07.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WAOoVueefWOIDaW5ZyqTrpKnGE8=', 'Content-Length': '0'} 2023-10-18T20:58:07.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007556ae5ac648b7e8-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.769 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.770 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.770 DEBUG:boto:path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.771 DEBUG:boto:auth_path=/bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.771 DEBUG:boto:Method: GET 2023-10-18T20:58:07.772 DEBUG:boto:Path: /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.772 DEBUG:boto:Data: 2023-10-18T20:58:07.773 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.773 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.773 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.774 DEBUG:boto:Params: {} 2023-10-18T20:58:07.774 DEBUG:boto:Token: None 2023-10-18T20:58:07.775 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11?versionId=null 2023-10-18T20:58:07.776 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WAOoVueefWOIDaW5ZyqTrpKnGE8= 2023-10-18T20:58:07.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WAOoVueefWOIDaW5ZyqTrpKnGE8=', 'Content-Length': '0'} 2023-10-18T20:58:07.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cc797de146a37fe-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.779 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.779 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.780 DEBUG:boto:path=/bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.780 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.780 DEBUG:boto:Method: GET 2023-10-18T20:58:07.781 DEBUG:boto:Path: /bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.781 DEBUG:boto:Data: 2023-10-18T20:58:07.782 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.782 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.783 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.783 DEBUG:boto:Params: {} 2023-10-18T20:58:07.783 DEBUG:boto:Token: None 2023-10-18T20:58:07.784 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.784 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4= 2023-10-18T20:58:07.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4=', 'Content-Length': '0'} 2023-10-18T20:58:07.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dde006d74645b4de-00653046df-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.788 DEBUG:boto:b'bofrnf-62obj111000falseobj112023-10-18T20:55:55.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.789 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.789 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.790 DEBUG:boto:path=/bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.790 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.790 DEBUG:boto:Method: GET 2023-10-18T20:58:07.791 DEBUG:boto:Path: /bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.791 DEBUG:boto:Data: 2023-10-18T20:58:07.792 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.792 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.792 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.793 DEBUG:boto:Params: {} 2023-10-18T20:58:07.793 DEBUG:boto:Token: None 2023-10-18T20:58:07.794 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.794 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4= 2023-10-18T20:58:07.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4=', 'Content-Length': '0'} 2023-10-18T20:58:07.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8c930265dcf6ff4-00653046df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.838 DEBUG:boto:b'bofrnf-62obj111000falseobj112023-10-18T20:55:55.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.839 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.840 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.840 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.841 DEBUG:boto:Path: /bofrnf-62/obj11 2023-10-18T20:58:07.841 DEBUG:boto:Data: 2023-10-18T20:58:07.842 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.842 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.842 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.843 DEBUG:boto:Params: {} 2023-10-18T20:58:07.844 DEBUG:boto:Token: None 2023-10-18T20:58:07.844 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11 2023-10-18T20:58:07.844 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fEcUrszJ+cGTx/s/064me+lrQPQ= 2023-10-18T20:58:07.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fEcUrszJ+cGTx/s/064me+lrQPQ=', 'Content-Length': '0'} 2023-10-18T20:58:07.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed91181beb4b11e9-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.848 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.848 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.849 DEBUG:boto:Method: HEAD 2023-10-18T20:58:07.849 DEBUG:boto:Path: /bofrnf-62/obj11 2023-10-18T20:58:07.850 DEBUG:boto:Data: 2023-10-18T20:58:07.850 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.851 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.851 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.851 DEBUG:boto:Params: {} 2023-10-18T20:58:07.852 DEBUG:boto:Token: None 2023-10-18T20:58:07.852 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11 2023-10-18T20:58:07.853 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fEcUrszJ+cGTx/s/064me+lrQPQ= 2023-10-18T20:58:07.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fEcUrszJ+cGTx/s/064me+lrQPQ=', 'Content-Length': '0'} 2023-10-18T20:58:07.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080c33bc2a0061d57-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T20:58:07.858 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.858 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.859 DEBUG:boto:Method: GET 2023-10-18T20:58:07.859 DEBUG:boto:Path: /bofrnf-62/obj11 2023-10-18T20:58:07.860 DEBUG:boto:Data: 2023-10-18T20:58:07.860 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.861 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.861 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.861 DEBUG:boto:Params: {} 2023-10-18T20:58:07.862 DEBUG:boto:Token: None 2023-10-18T20:58:07.862 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11 2023-10-18T20:58:07.863 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KNbwKr6biyaQwLc41rqPKhq4HzE= 2023-10-18T20:58:07.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KNbwKr6biyaQwLc41rqPKhq4HzE=', 'Content-Length': '0'} 2023-10-18T20:58:07.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d17c0d9f0cd768-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.906 DEBUG:boto:path=/bofrnf-62/obj11 2023-10-18T20:58:07.907 DEBUG:boto:auth_path=/bofrnf-62/obj11 2023-10-18T20:58:07.907 DEBUG:boto:Method: GET 2023-10-18T20:58:07.908 DEBUG:boto:Path: /bofrnf-62/obj11 2023-10-18T20:58:07.908 DEBUG:boto:Data: 2023-10-18T20:58:07.909 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.909 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.909 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.910 DEBUG:boto:Params: {} 2023-10-18T20:58:07.910 DEBUG:boto:Token: None 2023-10-18T20:58:07.911 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj11 2023-10-18T20:58:07.911 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KNbwKr6biyaQwLc41rqPKhq4HzE= 2023-10-18T20:58:07.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KNbwKr6biyaQwLc41rqPKhq4HzE=', 'Content-Length': '0'} 2023-10-18T20:58:07.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5dd1704f4b3a24d-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.915 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.915 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.916 DEBUG:boto:path=/bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.916 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.917 DEBUG:boto:Method: GET 2023-10-18T20:58:07.917 DEBUG:boto:Path: /bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.917 DEBUG:boto:Data: 2023-10-18T20:58:07.918 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.919 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.919 DEBUG:boto:Params: {} 2023-10-18T20:58:07.920 DEBUG:boto:Token: None 2023-10-18T20:58:07.921 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.921 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4= 2023-10-18T20:58:07.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4=', 'Content-Length': '0'} 2023-10-18T20:58:07.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db2cfcffe07b934e-00653046df-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.925 DEBUG:boto:b'bofrnf-62obj111000falseobj112023-10-18T20:55:55.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.926 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:07.927 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:07.927 DEBUG:boto:path=/bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.927 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.928 DEBUG:boto:Method: GET 2023-10-18T20:58:07.928 DEBUG:boto:Path: /bofrnf-62/?prefix=obj11 2023-10-18T20:58:07.929 DEBUG:boto:Data: 2023-10-18T20:58:07.929 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.930 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.930 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.931 DEBUG:boto:Params: {} 2023-10-18T20:58:07.931 DEBUG:boto:Token: None 2023-10-18T20:58:07.932 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/ 2023-10-18T20:58:07.932 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4= 2023-10-18T20:58:07.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uk8RzyGb0wSby7nliRgCt6znSV4=', 'Content-Length': '0'} 2023-10-18T20:58:07.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f61c37f994d64d7-00653046df-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.936 DEBUG:boto:b'bofrnf-62obj111000falseobj112023-10-18T20:55:55.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:07.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:58:07.937 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:07.938 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:07.938 DEBUG:boto:path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:07.938 DEBUG:boto:auth_path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:07.939 DEBUG:boto:Method: GET 2023-10-18T20:58:07.939 DEBUG:boto:Path: /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:07.940 DEBUG:boto:Data: 2023-10-18T20:58:07.940 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.941 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:07.941 DEBUG:boto:Port: 8001 2023-10-18T20:58:07.941 DEBUG:boto:Params: {} 2023-10-18T20:58:07.942 DEBUG:boto:Token: None 2023-10-18T20:58:07.942 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:07.943 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FZOtU968GyTtQ6Bjb2VtxXMDAe0= 2023-10-18T20:58:07.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FZOtU968GyTtQ6Bjb2VtxXMDAe0=', 'Content-Length': '0'} 2023-10-18T20:58:07.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4bc358c8dd22d48-00653046df-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.946 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:07.947 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:07.947 DEBUG:boto:path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:07.948 DEBUG:boto:auth_path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:07.948 DEBUG:boto:Method: GET 2023-10-18T20:58:07.948 DEBUG:boto:Path: /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:07.949 DEBUG:boto:Data: 2023-10-18T20:58:07.949 DEBUG:boto:Headers: {} 2023-10-18T20:58:07.950 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:07.950 DEBUG:boto:Port: 8000 2023-10-18T20:58:07.951 DEBUG:boto:Params: {} 2023-10-18T20:58:07.951 DEBUG:boto:Token: None 2023-10-18T20:58:07.952 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:07 GMT /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:07.952 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FZOtU968GyTtQ6Bjb2VtxXMDAe0= 2023-10-18T20:58:07.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FZOtU968GyTtQ6Bjb2VtxXMDAe0=', 'Content-Length': '0'} 2023-10-18T20:58:07.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058fcef502794c0a5-00653046df-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:07.998 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:07.999 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:07.999 DEBUG:boto:path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.000 DEBUG:boto:auth_path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.000 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.001 DEBUG:boto:Path: /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.001 DEBUG:boto:Data: 2023-10-18T20:58:08.001 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.002 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.002 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.003 DEBUG:boto:Params: {} 2023-10-18T20:58:08.003 DEBUG:boto:Token: None 2023-10-18T20:58:08.004 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.004 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YETYqjR0ldVhdL82y6kEBX2KC3Q= 2023-10-18T20:58:08.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YETYqjR0ldVhdL82y6kEBX2KC3Q=', 'Content-Length': '0'} 2023-10-18T20:58:08.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000711c9262d12cb5e2-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.007 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:08.007 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:08.008 DEBUG:boto:path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.008 DEBUG:boto:auth_path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.009 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.009 DEBUG:boto:Path: /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.009 DEBUG:boto:Data: 2023-10-18T20:58:08.010 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.010 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.011 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.011 DEBUG:boto:Params: {} 2023-10-18T20:58:08.011 DEBUG:boto:Token: None 2023-10-18T20:58:08.012 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:YETYqjR0ldVhdL82y6kEBX2KC3Q= 2023-10-18T20:58:08.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:YETYqjR0ldVhdL82y6kEBX2KC3Q=', 'Content-Length': '0'} 2023-10-18T20:58:08.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000546e18aabe73ffa1-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:58:08.016 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:08.017 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:08.017 DEBUG:boto:path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.017 DEBUG:boto:auth_path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.018 DEBUG:boto:Method: GET 2023-10-18T20:58:08.018 DEBUG:boto:Path: /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.019 DEBUG:boto:Data: 2023-10-18T20:58:08.019 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.020 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.020 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.021 DEBUG:boto:Params: {} 2023-10-18T20:58:08.021 DEBUG:boto:Token: None 2023-10-18T20:58:08.022 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.022 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0MubAmnL+tjzC7iqZfktPa6pQuA= 2023-10-18T20:58:08.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0MubAmnL+tjzC7iqZfktPa6pQuA=', 'Content-Length': '0'} 2023-10-18T20:58:08.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b8e3cc3204f8d5-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.025 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:08.026 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:08.026 DEBUG:boto:path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.026 DEBUG:boto:auth_path=/bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.027 DEBUG:boto:Method: GET 2023-10-18T20:58:08.027 DEBUG:boto:Path: /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.028 DEBUG:boto:Data: 2023-10-18T20:58:08.028 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.029 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.029 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.029 DEBUG:boto:Params: {} 2023-10-18T20:58:08.030 DEBUG:boto:Token: None 2023-10-18T20:58:08.030 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj22?versionId=null 2023-10-18T20:58:08.031 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0MubAmnL+tjzC7iqZfktPa6pQuA= 2023-10-18T20:58:08.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0MubAmnL+tjzC7iqZfktPa6pQuA=', 'Content-Length': '0'} 2023-10-18T20:58:08.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf8420f2993d601-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.034 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.035 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.035 DEBUG:boto:path=/bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.035 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.036 DEBUG:boto:Method: GET 2023-10-18T20:58:08.036 DEBUG:boto:Path: /bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.037 DEBUG:boto:Data: 2023-10-18T20:58:08.037 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.038 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.038 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.038 DEBUG:boto:Params: {} 2023-10-18T20:58:08.039 DEBUG:boto:Token: None 2023-10-18T20:58:08.039 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.040 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b63a32f1f4e7580f-00653046e0-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.043 DEBUG:boto:b'bofrnf-62obj221000falseobj222023-10-18T20:55:55.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.044 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.045 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.045 DEBUG:boto:path=/bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.045 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.046 DEBUG:boto:Method: GET 2023-10-18T20:58:08.046 DEBUG:boto:Path: /bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.047 DEBUG:boto:Data: 2023-10-18T20:58:08.047 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.048 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.048 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.048 DEBUG:boto:Params: {} 2023-10-18T20:58:08.049 DEBUG:boto:Token: None 2023-10-18T20:58:08.049 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.050 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a0bfa10486e2da4-00653046e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.054 DEBUG:boto:b'bofrnf-62obj221000falseobj222023-10-18T20:55:55.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.054 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:08.055 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:08.055 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.056 DEBUG:boto:Path: /bofrnf-62/obj22 2023-10-18T20:58:08.056 DEBUG:boto:Data: 2023-10-18T20:58:08.057 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.057 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.057 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.058 DEBUG:boto:Params: {} 2023-10-18T20:58:08.058 DEBUG:boto:Token: None 2023-10-18T20:58:08.059 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj22 2023-10-18T20:58:08.059 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8jlpBIHmg2Obnl02rpKXzrK0nzE= 2023-10-18T20:58:08.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8jlpBIHmg2Obnl02rpKXzrK0nzE=', 'Content-Length': '0'} 2023-10-18T20:58:08.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037f497b256573ca-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.062 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:08.062 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:08.063 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.063 DEBUG:boto:Path: /bofrnf-62/obj22 2023-10-18T20:58:08.064 DEBUG:boto:Data: 2023-10-18T20:58:08.064 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.064 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.065 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.065 DEBUG:boto:Params: {} 2023-10-18T20:58:08.066 DEBUG:boto:Token: None 2023-10-18T20:58:08.066 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj22 2023-10-18T20:58:08.067 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8jlpBIHmg2Obnl02rpKXzrK0nzE= 2023-10-18T20:58:08.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8jlpBIHmg2Obnl02rpKXzrK0nzE=', 'Content-Length': '0'} 2023-10-18T20:58:08.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062a23b9b07673938-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T20:58:08.070 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:08.071 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:08.071 DEBUG:boto:Method: GET 2023-10-18T20:58:08.072 DEBUG:boto:Path: /bofrnf-62/obj22 2023-10-18T20:58:08.072 DEBUG:boto:Data: 2023-10-18T20:58:08.072 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.073 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.073 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.074 DEBUG:boto:Params: {} 2023-10-18T20:58:08.074 DEBUG:boto:Token: None 2023-10-18T20:58:08.075 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj22 2023-10-18T20:58:08.075 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RtDzx9r/tiHQQ4vTLvEU+ARfo1o= 2023-10-18T20:58:08.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RtDzx9r/tiHQQ4vTLvEU+ARfo1o=', 'Content-Length': '0'} 2023-10-18T20:58:08.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee3dee908f04d55-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.122 DEBUG:boto:path=/bofrnf-62/obj22 2023-10-18T20:58:08.123 DEBUG:boto:auth_path=/bofrnf-62/obj22 2023-10-18T20:58:08.124 DEBUG:boto:Method: GET 2023-10-18T20:58:08.124 DEBUG:boto:Path: /bofrnf-62/obj22 2023-10-18T20:58:08.125 DEBUG:boto:Data: 2023-10-18T20:58:08.125 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.125 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.126 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.126 DEBUG:boto:Params: {} 2023-10-18T20:58:08.127 DEBUG:boto:Token: None 2023-10-18T20:58:08.127 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj22 2023-10-18T20:58:08.128 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RtDzx9r/tiHQQ4vTLvEU+ARfo1o= 2023-10-18T20:58:08.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RtDzx9r/tiHQQ4vTLvEU+ARfo1o=', 'Content-Length': '0'} 2023-10-18T20:58:08.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b0bb85810cd11e-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.131 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.131 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.132 DEBUG:boto:path=/bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.132 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.133 DEBUG:boto:Method: GET 2023-10-18T20:58:08.133 DEBUG:boto:Path: /bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.134 DEBUG:boto:Data: 2023-10-18T20:58:08.134 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.134 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.135 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.135 DEBUG:boto:Params: {} 2023-10-18T20:58:08.136 DEBUG:boto:Token: None 2023-10-18T20:58:08.136 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.137 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001caeab07ac02e762-00653046e0-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.141 DEBUG:boto:b'bofrnf-62obj221000falseobj222023-10-18T20:55:55.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.142 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.142 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.142 DEBUG:boto:path=/bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.143 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.143 DEBUG:boto:Method: GET 2023-10-18T20:58:08.144 DEBUG:boto:Path: /bofrnf-62/?prefix=obj22 2023-10-18T20:58:08.144 DEBUG:boto:Data: 2023-10-18T20:58:08.145 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.145 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.146 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.146 DEBUG:boto:Params: {} 2023-10-18T20:58:08.146 DEBUG:boto:Token: None 2023-10-18T20:58:08.147 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.147 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d198d23b49c49f2e-00653046e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.194 DEBUG:boto:b'bofrnf-62obj221000falseobj222023-10-18T20:55:55.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:58:08.196 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.196 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.196 DEBUG:boto:path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.197 DEBUG:boto:auth_path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.197 DEBUG:boto:Method: GET 2023-10-18T20:58:08.198 DEBUG:boto:Path: /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.198 DEBUG:boto:Data: 2023-10-18T20:58:08.199 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.199 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.199 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.200 DEBUG:boto:Params: {} 2023-10-18T20:58:08.200 DEBUG:boto:Token: None 2023-10-18T20:58:08.201 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.201 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kitLPhclPBhYPRRQu+BMSuBW/A8= 2023-10-18T20:58:08.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kitLPhclPBhYPRRQu+BMSuBW/A8=', 'Content-Length': '0'} 2023-10-18T20:58:08.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d0878971a59edd-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.205 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.205 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.206 DEBUG:boto:path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.206 DEBUG:boto:auth_path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.207 DEBUG:boto:Method: GET 2023-10-18T20:58:08.207 DEBUG:boto:Path: /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.208 DEBUG:boto:Data: 2023-10-18T20:58:08.208 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.208 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.209 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.209 DEBUG:boto:Params: {} 2023-10-18T20:58:08.210 DEBUG:boto:Token: None 2023-10-18T20:58:08.210 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kitLPhclPBhYPRRQu+BMSuBW/A8= 2023-10-18T20:58:08.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kitLPhclPBhYPRRQu+BMSuBW/A8=', 'Content-Length': '0'} 2023-10-18T20:58:08.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40c62eb82367428-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.214 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.215 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.215 DEBUG:boto:path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.215 DEBUG:boto:auth_path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.216 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.216 DEBUG:boto:Path: /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.217 DEBUG:boto:Data: 2023-10-18T20:58:08.217 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.217 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.218 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.218 DEBUG:boto:Params: {} 2023-10-18T20:58:08.219 DEBUG:boto:Token: None 2023-10-18T20:58:08.219 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.220 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aJNUWXsCTJrFirz5nEucUm2qkUI= 2023-10-18T20:58:08.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aJNUWXsCTJrFirz5nEucUm2qkUI=', 'Content-Length': '0'} 2023-10-18T20:58:08.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c45b687d8c3cf3-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.223 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.223 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.224 DEBUG:boto:path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.224 DEBUG:boto:auth_path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.224 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.225 DEBUG:boto:Path: /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.225 DEBUG:boto:Data: 2023-10-18T20:58:08.226 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.226 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.227 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.227 DEBUG:boto:Params: {} 2023-10-18T20:58:08.227 DEBUG:boto:Token: None 2023-10-18T20:58:08.228 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.228 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aJNUWXsCTJrFirz5nEucUm2qkUI= 2023-10-18T20:58:08.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aJNUWXsCTJrFirz5nEucUm2qkUI=', 'Content-Length': '0'} 2023-10-18T20:58:08.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6f7b47384f502ef-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:58:08.232 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.232 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.233 DEBUG:boto:path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.233 DEBUG:boto:auth_path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.234 DEBUG:boto:Method: GET 2023-10-18T20:58:08.234 DEBUG:boto:Path: /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.235 DEBUG:boto:Data: 2023-10-18T20:58:08.235 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.236 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.236 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.236 DEBUG:boto:Params: {} 2023-10-18T20:58:08.237 DEBUG:boto:Token: None 2023-10-18T20:58:08.237 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.238 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kitLPhclPBhYPRRQu+BMSuBW/A8= 2023-10-18T20:58:08.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kitLPhclPBhYPRRQu+BMSuBW/A8=', 'Content-Length': '0'} 2023-10-18T20:58:08.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a01d8cf4e7ae8d99-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.241 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.242 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.242 DEBUG:boto:path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.243 DEBUG:boto:auth_path=/bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.243 DEBUG:boto:Method: GET 2023-10-18T20:58:08.244 DEBUG:boto:Path: /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.244 DEBUG:boto:Data: 2023-10-18T20:58:08.245 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.245 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.245 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.246 DEBUG:boto:Params: {} 2023-10-18T20:58:08.246 DEBUG:boto:Token: None 2023-10-18T20:58:08.247 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33?versionId=null 2023-10-18T20:58:08.247 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kitLPhclPBhYPRRQu+BMSuBW/A8= 2023-10-18T20:58:08.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kitLPhclPBhYPRRQu+BMSuBW/A8=', 'Content-Length': '0'} 2023-10-18T20:58:08.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d06c574bb020d5f-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.251 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.251 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.251 DEBUG:boto:path=/bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.252 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.252 DEBUG:boto:Method: GET 2023-10-18T20:58:08.253 DEBUG:boto:Path: /bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.253 DEBUG:boto:Data: 2023-10-18T20:58:08.254 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.254 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.254 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.255 DEBUG:boto:Params: {} 2023-10-18T20:58:08.255 DEBUG:boto:Token: None 2023-10-18T20:58:08.256 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.256 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c5227a9ba0cda6f-00653046e0-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.260 DEBUG:boto:b'bofrnf-62obj331000falseobj332023-10-18T20:55:55.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.261 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.261 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.261 DEBUG:boto:path=/bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.262 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.262 DEBUG:boto:Method: GET 2023-10-18T20:58:08.263 DEBUG:boto:Path: /bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.263 DEBUG:boto:Data: 2023-10-18T20:58:08.263 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.264 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.264 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.265 DEBUG:boto:Params: {} 2023-10-18T20:58:08.265 DEBUG:boto:Token: None 2023-10-18T20:58:08.266 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.266 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e1027c8750779f1-00653046e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.310 DEBUG:boto:b'bofrnf-62obj331000falseobj332023-10-18T20:55:55.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.311 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.311 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.312 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.312 DEBUG:boto:Path: /bofrnf-62/obj33 2023-10-18T20:58:08.313 DEBUG:boto:Data: 2023-10-18T20:58:08.313 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.314 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.314 DEBUG:boto:Params: {} 2023-10-18T20:58:08.315 DEBUG:boto:Token: None 2023-10-18T20:58:08.315 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33 2023-10-18T20:58:08.316 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f3AtyHcgGKmpRJYcUsYjoQLjrEw= 2023-10-18T20:58:08.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f3AtyHcgGKmpRJYcUsYjoQLjrEw=', 'Content-Length': '0'} 2023-10-18T20:58:08.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000273ec210f5737777-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.319 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.319 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.320 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.320 DEBUG:boto:Path: /bofrnf-62/obj33 2023-10-18T20:58:08.321 DEBUG:boto:Data: 2023-10-18T20:58:08.321 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.321 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.322 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.322 DEBUG:boto:Params: {} 2023-10-18T20:58:08.323 DEBUG:boto:Token: None 2023-10-18T20:58:08.323 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33 2023-10-18T20:58:08.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:f3AtyHcgGKmpRJYcUsYjoQLjrEw= 2023-10-18T20:58:08.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f3AtyHcgGKmpRJYcUsYjoQLjrEw=', 'Content-Length': '0'} 2023-10-18T20:58:08.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006921774f03bb9dea-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T20:58:08.327 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.328 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.328 DEBUG:boto:Method: GET 2023-10-18T20:58:08.329 DEBUG:boto:Path: /bofrnf-62/obj33 2023-10-18T20:58:08.329 DEBUG:boto:Data: 2023-10-18T20:58:08.329 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.330 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.330 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.331 DEBUG:boto:Params: {} 2023-10-18T20:58:08.331 DEBUG:boto:Token: None 2023-10-18T20:58:08.332 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33 2023-10-18T20:58:08.332 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iGMlaDcXsXi9CObe/QPcMLiAKrI= 2023-10-18T20:58:08.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iGMlaDcXsXi9CObe/QPcMLiAKrI=', 'Content-Length': '0'} 2023-10-18T20:58:08.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eaa8ed22464680c-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.335 DEBUG:boto:path=/bofrnf-62/obj33 2023-10-18T20:58:08.336 DEBUG:boto:auth_path=/bofrnf-62/obj33 2023-10-18T20:58:08.336 DEBUG:boto:Method: GET 2023-10-18T20:58:08.337 DEBUG:boto:Path: /bofrnf-62/obj33 2023-10-18T20:58:08.337 DEBUG:boto:Data: 2023-10-18T20:58:08.338 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.338 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.339 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.339 DEBUG:boto:Params: {} 2023-10-18T20:58:08.339 DEBUG:boto:Token: None 2023-10-18T20:58:08.340 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj33 2023-10-18T20:58:08.340 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iGMlaDcXsXi9CObe/QPcMLiAKrI= 2023-10-18T20:58:08.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iGMlaDcXsXi9CObe/QPcMLiAKrI=', 'Content-Length': '0'} 2023-10-18T20:58:08.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e784ae43d1aaf95-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.343 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.344 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.344 DEBUG:boto:path=/bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.345 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.345 DEBUG:boto:Method: GET 2023-10-18T20:58:08.346 DEBUG:boto:Path: /bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.346 DEBUG:boto:Data: 2023-10-18T20:58:08.347 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.347 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.347 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.348 DEBUG:boto:Params: {} 2023-10-18T20:58:08.348 DEBUG:boto:Token: None 2023-10-18T20:58:08.349 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.349 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff7957e0f1478b3-00653046e0-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.398 DEBUG:boto:b'bofrnf-62obj331000falseobj332023-10-18T20:55:55.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.399 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.400 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.400 DEBUG:boto:path=/bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.400 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.401 DEBUG:boto:Method: GET 2023-10-18T20:58:08.401 DEBUG:boto:Path: /bofrnf-62/?prefix=obj33 2023-10-18T20:58:08.402 DEBUG:boto:Data: 2023-10-18T20:58:08.402 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.403 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.403 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.404 DEBUG:boto:Params: {} 2023-10-18T20:58:08.404 DEBUG:boto:Token: None 2023-10-18T20:58:08.404 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.405 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000946f0dce3c2d0cfd-00653046e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.409 DEBUG:boto:b'bofrnf-62obj331000falseobj332023-10-18T20:55:55.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:58:08.410 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.410 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.411 DEBUG:boto:path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.411 DEBUG:boto:auth_path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.411 DEBUG:boto:Method: GET 2023-10-18T20:58:08.412 DEBUG:boto:Path: /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.412 DEBUG:boto:Data: 2023-10-18T20:58:08.413 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.413 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.414 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.414 DEBUG:boto:Params: {} 2023-10-18T20:58:08.415 DEBUG:boto:Token: None 2023-10-18T20:58:08.415 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.416 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y195yHf4bGMB/nWmfOlcWREOuSA= 2023-10-18T20:58:08.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y195yHf4bGMB/nWmfOlcWREOuSA=', 'Content-Length': '0'} 2023-10-18T20:58:08.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d62d95b73ccf654-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.420 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.420 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.421 DEBUG:boto:path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.421 DEBUG:boto:auth_path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.422 DEBUG:boto:Method: GET 2023-10-18T20:58:08.422 DEBUG:boto:Path: /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.423 DEBUG:boto:Data: 2023-10-18T20:58:08.423 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.423 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.424 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.424 DEBUG:boto:Params: {} 2023-10-18T20:58:08.425 DEBUG:boto:Token: None 2023-10-18T20:58:08.425 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.426 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y195yHf4bGMB/nWmfOlcWREOuSA= 2023-10-18T20:58:08.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y195yHf4bGMB/nWmfOlcWREOuSA=', 'Content-Length': '0'} 2023-10-18T20:58:08.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000688e821fa5c96f18-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.429 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.429 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.430 DEBUG:boto:path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.430 DEBUG:boto:auth_path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.431 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.431 DEBUG:boto:Path: /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.431 DEBUG:boto:Data: 2023-10-18T20:58:08.432 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.432 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.433 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.433 DEBUG:boto:Params: {} 2023-10-18T20:58:08.434 DEBUG:boto:Token: None 2023-10-18T20:58:08.434 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.434 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:48H62e+atJOaf94ZDAJDMlTWg+U= 2023-10-18T20:58:08.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:48H62e+atJOaf94ZDAJDMlTWg+U=', 'Content-Length': '0'} 2023-10-18T20:58:08.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682cf0aa9c2683e2-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.437 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.438 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.438 DEBUG:boto:path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.439 DEBUG:boto:auth_path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.439 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.439 DEBUG:boto:Path: /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.440 DEBUG:boto:Data: 2023-10-18T20:58:08.440 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.441 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.441 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.441 DEBUG:boto:Params: {} 2023-10-18T20:58:08.442 DEBUG:boto:Token: None 2023-10-18T20:58:08.442 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.443 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:48H62e+atJOaf94ZDAJDMlTWg+U= 2023-10-18T20:58:08.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:48H62e+atJOaf94ZDAJDMlTWg+U=', 'Content-Length': '0'} 2023-10-18T20:58:08.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15036690d0adc8a-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:58:08.446 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.447 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.447 DEBUG:boto:path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.448 DEBUG:boto:auth_path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.448 DEBUG:boto:Method: GET 2023-10-18T20:58:08.448 DEBUG:boto:Path: /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.449 DEBUG:boto:Data: 2023-10-18T20:58:08.449 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.450 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.450 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.451 DEBUG:boto:Params: {} 2023-10-18T20:58:08.451 DEBUG:boto:Token: None 2023-10-18T20:58:08.452 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.452 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y195yHf4bGMB/nWmfOlcWREOuSA= 2023-10-18T20:58:08.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y195yHf4bGMB/nWmfOlcWREOuSA=', 'Content-Length': '0'} 2023-10-18T20:58:08.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3d8499e4a347b0b-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.456 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.456 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.456 DEBUG:boto:path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.457 DEBUG:boto:auth_path=/bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.457 DEBUG:boto:Method: GET 2023-10-18T20:58:08.458 DEBUG:boto:Path: /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.458 DEBUG:boto:Data: 2023-10-18T20:58:08.459 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.459 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.459 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.460 DEBUG:boto:Params: {} 2023-10-18T20:58:08.460 DEBUG:boto:Token: None 2023-10-18T20:58:08.461 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44?versionId=null 2023-10-18T20:58:08.461 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y195yHf4bGMB/nWmfOlcWREOuSA= 2023-10-18T20:58:08.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y195yHf4bGMB/nWmfOlcWREOuSA=', 'Content-Length': '0'} 2023-10-18T20:58:08.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c5e53b6781bfb98-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.464 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.465 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.465 DEBUG:boto:path=/bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.466 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.466 DEBUG:boto:Method: GET 2023-10-18T20:58:08.466 DEBUG:boto:Path: /bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.467 DEBUG:boto:Data: 2023-10-18T20:58:08.467 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.468 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.468 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.469 DEBUG:boto:Params: {} 2023-10-18T20:58:08.469 DEBUG:boto:Token: None 2023-10-18T20:58:08.470 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.470 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008945752bb90b796b-00653046e0-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.518 DEBUG:boto:b'bofrnf-62obj441000falseobj442023-10-18T20:55:55.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.519 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.520 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.520 DEBUG:boto:path=/bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.520 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.521 DEBUG:boto:Method: GET 2023-10-18T20:58:08.521 DEBUG:boto:Path: /bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.522 DEBUG:boto:Data: 2023-10-18T20:58:08.522 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.523 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.523 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.523 DEBUG:boto:Params: {} 2023-10-18T20:58:08.524 DEBUG:boto:Token: None 2023-10-18T20:58:08.524 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.525 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fee0df76bf60208-00653046e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.529 DEBUG:boto:b'bofrnf-62obj441000falseobj442023-10-18T20:55:55.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.530 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.530 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.531 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.531 DEBUG:boto:Path: /bofrnf-62/obj44 2023-10-18T20:58:08.532 DEBUG:boto:Data: 2023-10-18T20:58:08.532 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.533 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.533 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.533 DEBUG:boto:Params: {} 2023-10-18T20:58:08.534 DEBUG:boto:Token: None 2023-10-18T20:58:08.534 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44 2023-10-18T20:58:08.535 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gqzCGYSVxbM2qYRS6GbvkYg07AU= 2023-10-18T20:58:08.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gqzCGYSVxbM2qYRS6GbvkYg07AU=', 'Content-Length': '0'} 2023-10-18T20:58:08.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032a81a68aa1bd2f6-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.538 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.538 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.539 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.539 DEBUG:boto:Path: /bofrnf-62/obj44 2023-10-18T20:58:08.540 DEBUG:boto:Data: 2023-10-18T20:58:08.540 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.541 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.541 DEBUG:boto:Params: {} 2023-10-18T20:58:08.542 DEBUG:boto:Token: None 2023-10-18T20:58:08.542 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44 2023-10-18T20:58:08.542 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gqzCGYSVxbM2qYRS6GbvkYg07AU= 2023-10-18T20:58:08.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gqzCGYSVxbM2qYRS6GbvkYg07AU=', 'Content-Length': '0'} 2023-10-18T20:58:08.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a7ae4c6eaae925-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T20:58:08.546 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.546 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.547 DEBUG:boto:Method: GET 2023-10-18T20:58:08.547 DEBUG:boto:Path: /bofrnf-62/obj44 2023-10-18T20:58:08.548 DEBUG:boto:Data: 2023-10-18T20:58:08.548 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.548 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.549 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.549 DEBUG:boto:Params: {} 2023-10-18T20:58:08.550 DEBUG:boto:Token: None 2023-10-18T20:58:08.550 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44 2023-10-18T20:58:08.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3NvqiyOj5YidytIDsUiHVl4orwI= 2023-10-18T20:58:08.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3NvqiyOj5YidytIDsUiHVl4orwI=', 'Content-Length': '0'} 2023-10-18T20:58:08.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb2c5ffdb00a3bb-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.554 DEBUG:boto:path=/bofrnf-62/obj44 2023-10-18T20:58:08.554 DEBUG:boto:auth_path=/bofrnf-62/obj44 2023-10-18T20:58:08.555 DEBUG:boto:Method: GET 2023-10-18T20:58:08.555 DEBUG:boto:Path: /bofrnf-62/obj44 2023-10-18T20:58:08.555 DEBUG:boto:Data: 2023-10-18T20:58:08.556 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.556 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.557 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.557 DEBUG:boto:Params: {} 2023-10-18T20:58:08.558 DEBUG:boto:Token: None 2023-10-18T20:58:08.558 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj44 2023-10-18T20:58:08.559 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3NvqiyOj5YidytIDsUiHVl4orwI= 2023-10-18T20:58:08.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3NvqiyOj5YidytIDsUiHVl4orwI=', 'Content-Length': '0'} 2023-10-18T20:58:08.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2dd5264d526f963-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.562 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.562 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.563 DEBUG:boto:path=/bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.563 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.564 DEBUG:boto:Method: GET 2023-10-18T20:58:08.564 DEBUG:boto:Path: /bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.564 DEBUG:boto:Data: 2023-10-18T20:58:08.565 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.565 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.566 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.566 DEBUG:boto:Params: {} 2023-10-18T20:58:08.567 DEBUG:boto:Token: None 2023-10-18T20:58:08.567 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.568 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000222e978a3904123f-00653046e0-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.571 DEBUG:boto:b'bofrnf-62obj441000falseobj442023-10-18T20:55:55.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.572 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.573 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.573 DEBUG:boto:path=/bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.574 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.574 DEBUG:boto:Method: GET 2023-10-18T20:58:08.575 DEBUG:boto:Path: /bofrnf-62/?prefix=obj44 2023-10-18T20:58:08.575 DEBUG:boto:Data: 2023-10-18T20:58:08.576 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.576 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.577 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.577 DEBUG:boto:Params: {} 2023-10-18T20:58:08.578 DEBUG:boto:Token: None 2023-10-18T20:58:08.578 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.579 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3a8aad09e2cc6d4-00653046e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.582 DEBUG:boto:b'bofrnf-62obj441000falseobj442023-10-18T20:55:55.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:58:08.583 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.584 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.584 DEBUG:boto:path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.585 DEBUG:boto:auth_path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.585 DEBUG:boto:Method: GET 2023-10-18T20:58:08.586 DEBUG:boto:Path: /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.586 DEBUG:boto:Data: 2023-10-18T20:58:08.586 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.587 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.587 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.588 DEBUG:boto:Params: {} 2023-10-18T20:58:08.588 DEBUG:boto:Token: None 2023-10-18T20:58:08.589 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.589 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pfnu1Q2AdN3C64nm6CWzmFTe0l0= 2023-10-18T20:58:08.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pfnu1Q2AdN3C64nm6CWzmFTe0l0=', 'Content-Length': '0'} 2023-10-18T20:58:08.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b13cb74c4d5be49a-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.593 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.593 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.594 DEBUG:boto:path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.594 DEBUG:boto:auth_path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.594 DEBUG:boto:Method: GET 2023-10-18T20:58:08.595 DEBUG:boto:Path: /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.595 DEBUG:boto:Data: 2023-10-18T20:58:08.596 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.596 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.597 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.597 DEBUG:boto:Params: {} 2023-10-18T20:58:08.598 DEBUG:boto:Token: None 2023-10-18T20:58:08.598 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.599 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pfnu1Q2AdN3C64nm6CWzmFTe0l0= 2023-10-18T20:58:08.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pfnu1Q2AdN3C64nm6CWzmFTe0l0=', 'Content-Length': '0'} 2023-10-18T20:58:08.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d00092fa6b6f9be-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.602 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.602 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.603 DEBUG:boto:path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.603 DEBUG:boto:auth_path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.604 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.604 DEBUG:boto:Path: /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.605 DEBUG:boto:Data: 2023-10-18T20:58:08.605 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.606 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.606 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.607 DEBUG:boto:Params: {} 2023-10-18T20:58:08.607 DEBUG:boto:Token: None 2023-10-18T20:58:08.608 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+W1pW3xlnKw2Al70UDgOTs22ASE= 2023-10-18T20:58:08.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+W1pW3xlnKw2Al70UDgOTs22ASE=', 'Content-Length': '0'} 2023-10-18T20:58:08.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e49158a07272ec9-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.611 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.612 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.612 DEBUG:boto:path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.612 DEBUG:boto:auth_path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.613 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.613 DEBUG:boto:Path: /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.614 DEBUG:boto:Data: 2023-10-18T20:58:08.614 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.615 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.615 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.616 DEBUG:boto:Params: {} 2023-10-18T20:58:08.616 DEBUG:boto:Token: None 2023-10-18T20:58:08.617 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.617 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+W1pW3xlnKw2Al70UDgOTs22ASE= 2023-10-18T20:58:08.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+W1pW3xlnKw2Al70UDgOTs22ASE=', 'Content-Length': '0'} 2023-10-18T20:58:08.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd565e218d25bd4b-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:58:08.621 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.621 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.622 DEBUG:boto:path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.622 DEBUG:boto:auth_path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.623 DEBUG:boto:Method: GET 2023-10-18T20:58:08.623 DEBUG:boto:Path: /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.624 DEBUG:boto:Data: 2023-10-18T20:58:08.624 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.625 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.626 DEBUG:boto:Params: {} 2023-10-18T20:58:08.626 DEBUG:boto:Token: None 2023-10-18T20:58:08.627 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.627 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pfnu1Q2AdN3C64nm6CWzmFTe0l0= 2023-10-18T20:58:08.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pfnu1Q2AdN3C64nm6CWzmFTe0l0=', 'Content-Length': '0'} 2023-10-18T20:58:08.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f640f34f6b4bd235-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.674 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.675 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.675 DEBUG:boto:path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.676 DEBUG:boto:auth_path=/bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.676 DEBUG:boto:Method: GET 2023-10-18T20:58:08.677 DEBUG:boto:Path: /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.677 DEBUG:boto:Data: 2023-10-18T20:58:08.678 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.678 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.678 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.679 DEBUG:boto:Params: {} 2023-10-18T20:58:08.679 DEBUG:boto:Token: None 2023-10-18T20:58:08.680 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55?versionId=null 2023-10-18T20:58:08.680 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Pfnu1Q2AdN3C64nm6CWzmFTe0l0= 2023-10-18T20:58:08.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Pfnu1Q2AdN3C64nm6CWzmFTe0l0=', 'Content-Length': '0'} 2023-10-18T20:58:08.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001730a9ba9aebd8d8-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.684 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.684 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.684 DEBUG:boto:path=/bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.685 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.685 DEBUG:boto:Method: GET 2023-10-18T20:58:08.686 DEBUG:boto:Path: /bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.686 DEBUG:boto:Data: 2023-10-18T20:58:08.687 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.687 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.687 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.688 DEBUG:boto:Params: {} 2023-10-18T20:58:08.688 DEBUG:boto:Token: None 2023-10-18T20:58:08.689 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.689 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a01414148c1ce2b-00653046e0-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.693 DEBUG:boto:b'bofrnf-62obj551000falseobj552023-10-18T20:55:55.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.694 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.694 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.695 DEBUG:boto:path=/bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.695 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.695 DEBUG:boto:Method: GET 2023-10-18T20:58:08.696 DEBUG:boto:Path: /bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.696 DEBUG:boto:Data: 2023-10-18T20:58:08.697 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.697 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.697 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.698 DEBUG:boto:Params: {} 2023-10-18T20:58:08.698 DEBUG:boto:Token: None 2023-10-18T20:58:08.699 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.699 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0c7a57e2369091f-00653046e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.703 DEBUG:boto:b'bofrnf-62obj551000falseobj552023-10-18T20:55:55.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.704 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.704 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.705 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.705 DEBUG:boto:Path: /bofrnf-62/obj55 2023-10-18T20:58:08.705 DEBUG:boto:Data: 2023-10-18T20:58:08.706 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.707 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.707 DEBUG:boto:Params: {} 2023-10-18T20:58:08.707 DEBUG:boto:Token: None 2023-10-18T20:58:08.708 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55 2023-10-18T20:58:08.708 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nagg7BGkXGRxPo89hiybStVn9lI= 2023-10-18T20:58:08.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nagg7BGkXGRxPo89hiybStVn9lI=', 'Content-Length': '0'} 2023-10-18T20:58:08.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d6014bda889ce5-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.711 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.712 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.712 DEBUG:boto:Method: HEAD 2023-10-18T20:58:08.713 DEBUG:boto:Path: /bofrnf-62/obj55 2023-10-18T20:58:08.713 DEBUG:boto:Data: 2023-10-18T20:58:08.714 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.714 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.714 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.715 DEBUG:boto:Params: {} 2023-10-18T20:58:08.715 DEBUG:boto:Token: None 2023-10-18T20:58:08.716 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55 2023-10-18T20:58:08.716 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nagg7BGkXGRxPo89hiybStVn9lI= 2023-10-18T20:58:08.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nagg7BGkXGRxPo89hiybStVn9lI=', 'Content-Length': '0'} 2023-10-18T20:58:08.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018abee729cb7df8d-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T20:58:08.720 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.720 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.721 DEBUG:boto:Method: GET 2023-10-18T20:58:08.721 DEBUG:boto:Path: /bofrnf-62/obj55 2023-10-18T20:58:08.721 DEBUG:boto:Data: 2023-10-18T20:58:08.722 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.722 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.723 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.723 DEBUG:boto:Params: {} 2023-10-18T20:58:08.724 DEBUG:boto:Token: None 2023-10-18T20:58:08.724 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55 2023-10-18T20:58:08.725 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pfPKsYBeJ208roYMDlgnUsoUgKg= 2023-10-18T20:58:08.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pfPKsYBeJ208roYMDlgnUsoUgKg=', 'Content-Length': '0'} 2023-10-18T20:58:08.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-62:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.31'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b798b0b434923b30-00653046e0-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.728 DEBUG:boto:path=/bofrnf-62/obj55 2023-10-18T20:58:08.728 DEBUG:boto:auth_path=/bofrnf-62/obj55 2023-10-18T20:58:08.728 DEBUG:boto:Method: GET 2023-10-18T20:58:08.729 DEBUG:boto:Path: /bofrnf-62/obj55 2023-10-18T20:58:08.729 DEBUG:boto:Data: 2023-10-18T20:58:08.730 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.730 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.731 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.731 DEBUG:boto:Params: {} 2023-10-18T20:58:08.731 DEBUG:boto:Token: None 2023-10-18T20:58:08.732 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/obj55 2023-10-18T20:58:08.732 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pfPKsYBeJ208roYMDlgnUsoUgKg= 2023-10-18T20:58:08.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pfPKsYBeJ208roYMDlgnUsoUgKg=', 'Content-Length': '0'} 2023-10-18T20:58:08.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:55:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dc6aa1fe47d4618-00653046e0-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.778 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.779 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.779 DEBUG:boto:path=/bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.780 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.780 DEBUG:boto:Method: GET 2023-10-18T20:58:08.781 DEBUG:boto:Path: /bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.781 DEBUG:boto:Data: 2023-10-18T20:58:08.781 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.782 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T20:58:08.782 DEBUG:boto:Port: 8001 2023-10-18T20:58:08.783 DEBUG:boto:Params: {} 2023-10-18T20:58:08.783 DEBUG:boto:Token: None 2023-10-18T20:58:08.784 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.784 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033402c36a72d631e-00653046e0-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.788 DEBUG:boto:b'bofrnf-62obj551000falseobj552023-10-18T20:55:55.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.789 DEBUG:boto:path=/bofrnf-62/ 2023-10-18T20:58:08.789 DEBUG:boto:auth_path=/bofrnf-62/ 2023-10-18T20:58:08.789 DEBUG:boto:path=/bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.790 DEBUG:boto:auth_path=/bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.790 DEBUG:boto:Method: GET 2023-10-18T20:58:08.791 DEBUG:boto:Path: /bofrnf-62/?prefix=obj55 2023-10-18T20:58:08.791 DEBUG:boto:Data: 2023-10-18T20:58:08.792 DEBUG:boto:Headers: {} 2023-10-18T20:58:08.792 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:08.792 DEBUG:boto:Port: 8000 2023-10-18T20:58:08.793 DEBUG:boto:Params: {} 2023-10-18T20:58:08.793 DEBUG:boto:Token: None 2023-10-18T20:58:08.794 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 20:58:08 GMT /bofrnf-62/ 2023-10-18T20:58:08.794 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4= 2023-10-18T20:58:08.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NlM5KI+jmWRYbP3Zgw323rO6tc4=', 'Content-Length': '0'} 2023-10-18T20:58:08.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baaccc5ae2886ff9-00653046e0-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:58:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:08.798 DEBUG:boto:b'bofrnf-62obj551000falseobj552023-10-18T20:55:55.884Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T20:58:08.798 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-62 zones={a2, a1} 2023-10-18T20:58:08.799 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-18T20:58:08.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:08.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:08.800 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:08.854 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:08.854 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:08.905 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:58:08.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:08.905 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032869", 2023-10-18T20:58:08.906 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:56:11.442407Z" 2023-10-18T20:58:08.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:08.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:08.906 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032493", 2023-10-18T20:58:08.907 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:56:11.444193Z" 2023-10-18T20:58:08.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:08.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:08.907 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000023565", 2023-10-18T20:58:08.907 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:50:42.299372Z" 2023-10-18T20:58:08.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:08.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:08.908 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000037532", 2023-10-18T20:58:08.908 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:55:55.864811Z" 2023-10-18T20:58:08.909 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:08.909 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:58:08.910 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032869', 1: '00000000000000000000:00000000000000032493', 2: '00000000000000000000:00000000000000023565', 3: '00000000000000000000:00000000000000037532'} 2023-10-18T20:58:08.910 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:58:08.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:08.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:08.912 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:08.963 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:08.963 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:09.007 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:58:09.007 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:58:09.007 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:58:09.007 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:58:09.008 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:58:09.008 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 10838132146158513877 2023-10-18T20:58:09.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.008 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:58:09.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.009 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:58:09.009 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:09.009 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:09.009 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032869", 2023-10-18T20:58:09.010 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:09.010 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:09.010 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:09.010 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:56:11.442407Z" 2023-10-18T20:58:09.010 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.011 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:58:09.011 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:09.012 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:09.012 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032493", 2023-10-18T20:58:09.012 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:09.012 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:09.012 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:09.013 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:56:11.444193Z" 2023-10-18T20:58:09.013 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.013 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:58:09.014 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:09.014 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:09.014 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000023565", 2023-10-18T20:58:09.014 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:09.014 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T20:58:09.015 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:09.015 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:50:42.299372Z" 2023-10-18T20:58:09.015 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.016 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:58:09.016 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:09.016 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:09.016 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000037532", 2023-10-18T20:58:09.017 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:09.017 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 53, 2023-10-18T20:58:09.017 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:09.017 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:55:55.864811Z" 2023-10-18T20:58:09.017 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.018 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.018 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:58:09.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.018 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:58:09.018 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 62, 2023-10-18T20:58:09.019 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 62 2023-10-18T20:58:09.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.019 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:58:08Z" 2023-10-18T20:58:09.019 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:58:09.020 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10838132146158513877 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032869", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:56:11.442407Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:56:11.444193Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023565", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T20:50:42.299372Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037532", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-18T20:55:55.864811Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-10-18T20:58:08Z" } 2023-10-18T20:58:09.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032869', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:56:11.442407Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:56:11.444193Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023565', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T20:50:42.299372Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037532', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-18T20:55:55.864811Z'}}] 2023-10-18T20:58:09.021 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032869', 1: '00000000000000000000:00000000000000032493', 2: '00000000000000000000:00000000000000023565', 3: '00000000000000000000:00000000000000037532'} 2023-10-18T20:58:09.022 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032869', 1: '00000000000000000000:00000000000000032493', 2: '00000000000000000000:00000000000000023565', 3: '00000000000000000000:00000000000000037532'} 2023-10-18T20:58:09.022 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T20:58:09.022 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-18T20:58:09.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:09.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:09.024 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:09.072 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:09.072 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:09.121 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:58:09.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.122 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000014188", 2023-10-18T20:58:09.122 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:58:08.153316Z" 2023-10-18T20:58:09.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.123 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000011541", 2023-10-18T20:58:09.123 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:58:00.461309Z" 2023-10-18T20:58:09.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.124 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005725", 2023-10-18T20:58:09.124 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:58:00.432732Z" 2023-10-18T20:58:09.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.125 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000015309", 2023-10-18T20:58:09.125 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:58:00.465500Z" 2023-10-18T20:58:09.125 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.125 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:58:09.126 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000014188', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:09.127 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:58:09.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:09.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:09.128 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:09.179 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:09.179 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:09.226 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:58:09.226 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:58:09.226 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:58:09.226 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:58:09.227 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:58:09.227 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:58:09.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.227 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:58:09.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.228 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:58:09.228 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:09.228 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:09.228 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000014047", 2023-10-18T20:58:09.229 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:09.229 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:09.229 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:09.229 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:57:56.270898Z" 2023-10-18T20:58:09.229 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.230 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:58:09.230 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:09.230 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:09.231 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000011541", 2023-10-18T20:58:09.231 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:09.231 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:09.231 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:09.231 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.461309Z" 2023-10-18T20:58:09.232 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.232 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:58:09.233 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:09.233 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:09.233 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005725", 2023-10-18T20:58:09.233 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:09.233 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:09.234 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:09.234 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.432732Z" 2023-10-18T20:58:09.234 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:09.235 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:58:09.235 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:09.235 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:09.235 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000015309", 2023-10-18T20:58:09.236 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:09.236 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:09.236 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:09.236 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.465500Z" 2023-10-18T20:58:09.236 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.236 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:09.237 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:58:09.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.237 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:58:09.237 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:58:09.237 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:58:09.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:09.238 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:58:09Z" 2023-10-18T20:58:09.238 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:58:09.239 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014047", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:57:56.270898Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011541", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.461309Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005725", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.432732Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015309", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.465500Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:58:09Z" } 2023-10-18T20:58:09.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014047', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:57:56.270898Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011541', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.461309Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005725', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.432732Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015309', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.465500Z'}}] 2023-10-18T20:58:09.240 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014188', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:09.240 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014047', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:09.241 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000014188 target=00000000000000000000:00000000000000014047 2023-10-18T20:58:14.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:14.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:14.243 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:14.303 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:14.303 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:14.350 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:58:14.351 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:58:14.351 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:58:14.351 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:58:14.351 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:58:14.352 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:58:14.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:14.352 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:58:14.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:14.352 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:58:14.353 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:14.353 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:14.353 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000014047", 2023-10-18T20:58:14.353 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:14.354 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:14.354 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:14.354 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:57:56.270898Z" 2023-10-18T20:58:14.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:14.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:14.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:14.355 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:58:14.355 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:14.355 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:14.356 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000011541", 2023-10-18T20:58:14.356 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:14.356 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:14.356 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:14.356 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.461309Z" 2023-10-18T20:58:14.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:14.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:14.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:14.357 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:58:14.357 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:14.358 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:14.358 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005725", 2023-10-18T20:58:14.358 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:14.358 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:14.358 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:14.359 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.432732Z" 2023-10-18T20:58:14.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:14.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:14.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:14.360 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:58:14.360 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:14.360 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:14.360 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000015309", 2023-10-18T20:58:14.361 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:14.361 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:14.361 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:14.361 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.465500Z" 2023-10-18T20:58:14.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:14.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:14.362 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:58:14.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:14.362 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:58:14.363 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:58:14.363 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:58:14.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:14.363 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:58:14Z" 2023-10-18T20:58:14.364 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:58:14.365 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014047", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:57:56.270898Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011541", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.461309Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005725", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.432732Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015309", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.465500Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:58:14Z" } 2023-10-18T20:58:14.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014047', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:57:56.270898Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011541', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.461309Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005725', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.432732Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015309', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.465500Z'}}] 2023-10-18T20:58:14.366 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014188', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:14.366 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014047', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:14.367 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000014188 target=00000000000000000000:00000000000000014047 2023-10-18T20:58:19.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:19.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:19.369 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:19.427 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:19.428 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:19.481 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:58:19.482 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:58:19.482 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:58:19.482 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:58:19.482 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:58:19.482 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:58:19.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:19.483 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:58:19.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:19.483 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:58:19.484 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:19.484 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:19.484 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000014047", 2023-10-18T20:58:19.484 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:19.484 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:19.485 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:19.485 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:57:56.270898Z" 2023-10-18T20:58:19.485 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:19.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:19.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:19.486 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:58:19.486 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:19.486 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:19.486 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000011541", 2023-10-18T20:58:19.487 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:19.487 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:19.487 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:19.487 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.461309Z" 2023-10-18T20:58:19.487 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:19.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:19.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:19.488 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:58:19.488 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:19.488 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:19.489 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005725", 2023-10-18T20:58:19.489 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:19.489 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:19.489 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:19.489 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.432732Z" 2023-10-18T20:58:19.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:19.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:19.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:19.490 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:58:19.490 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:19.491 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:19.491 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000015309", 2023-10-18T20:58:19.491 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:19.491 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:19.491 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:19.492 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.465500Z" 2023-10-18T20:58:19.492 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:19.492 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:19.492 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:58:19.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:19.493 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:58:19.493 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:58:19.493 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:58:19.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:19.494 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:58:19Z" 2023-10-18T20:58:19.494 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:58:19.495 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014047", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:57:56.270898Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011541", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.461309Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005725", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.432732Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015309", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.465500Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:58:19Z" } 2023-10-18T20:58:19.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014047', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:57:56.270898Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011541', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.461309Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005725', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.432732Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015309', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.465500Z'}}] 2023-10-18T20:58:19.496 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014188', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:19.496 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014047', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:19.497 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000014188 target=00000000000000000000:00000000000000014047 2023-10-18T20:58:24.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:24.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:24.499 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:24.559 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:24.560 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:24.608 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:58:24.608 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:58:24.608 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:58:24.608 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:58:24.609 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:58:24.609 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:58:24.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:24.609 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:58:24.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:24.610 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:58:24.610 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:24.610 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:24.611 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000014047", 2023-10-18T20:58:24.611 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:24.611 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:24.611 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:24.612 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:57:56.270898Z" 2023-10-18T20:58:24.612 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:24.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:24.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:24.612 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:58:24.613 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:24.613 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:24.613 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000011541", 2023-10-18T20:58:24.613 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:24.614 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:24.614 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:24.614 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.461309Z" 2023-10-18T20:58:24.614 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:24.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:24.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:24.615 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:58:24.615 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:24.615 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:24.615 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005725", 2023-10-18T20:58:24.616 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:24.616 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:24.616 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:24.616 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.432732Z" 2023-10-18T20:58:24.617 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:24.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:24.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:24.617 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:58:24.617 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:24.618 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:24.618 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000015309", 2023-10-18T20:58:24.618 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:24.618 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:24.619 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:24.619 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.465500Z" 2023-10-18T20:58:24.619 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:24.619 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:24.620 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:58:24.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:24.620 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:58:24.620 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:58:24.620 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:58:24.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:24.621 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:58:24Z" 2023-10-18T20:58:24.621 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:58:24.622 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014047", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:57:56.270898Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011541", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.461309Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005725", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.432732Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015309", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.465500Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:58:24Z" } 2023-10-18T20:58:24.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014047', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:57:56.270898Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011541', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.461309Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005725', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.432732Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015309', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.465500Z'}}] 2023-10-18T20:58:24.623 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014188', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:24.624 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014047', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:24.624 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000014188 target=00000000000000000000:00000000000000014047 2023-10-18T20:58:29.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:29.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:29.626 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:29.685 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:29.685 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:29.733 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:58:29.734 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:58:29.734 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:58:29.734 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:58:29.734 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:58:29.735 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T20:58:29.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.735 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:58:29.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.736 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:58:29.736 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:29.736 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:29.736 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000014188", 2023-10-18T20:58:29.736 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:29.737 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:29.737 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:29.737 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:08.153316Z" 2023-10-18T20:58:29.737 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.738 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:58:29.738 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:29.738 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:29.739 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000011541", 2023-10-18T20:58:29.739 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:29.739 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:29.739 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:29.739 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.461309Z" 2023-10-18T20:58:29.740 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.740 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:58:29.740 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:29.741 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:29.741 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000005725", 2023-10-18T20:58:29.741 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:29.741 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:29.741 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:29.742 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.432732Z" 2023-10-18T20:58:29.742 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.743 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:58:29.743 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:29.743 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T20:58:29.743 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000015309", 2023-10-18T20:58:29.743 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:29.744 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:29.744 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:29.744 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:00.465500Z" 2023-10-18T20:58:29.744 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.745 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:58:29.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.745 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:58:29.745 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T20:58:29.746 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T20:58:29.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.746 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:58:29Z" 2023-10-18T20:58:29.746 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:58:29.747 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014188", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:08.153316Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011541", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.461309Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005725", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.432732Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015309", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:58:00.465500Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T20:58:29Z" } 2023-10-18T20:58:29.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014188', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:08.153316Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011541', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.461309Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005725', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.432732Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015309', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:58:00.465500Z'}}] 2023-10-18T20:58:29.748 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014188', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:29.749 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014188', 1: '00000000000000000000:00000000000000011541', 2: '00000000000000000000:00000000000000005725', 3: '00000000000000000000:00000000000000015309'} 2023-10-18T20:58:29.749 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T20:58:29.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2023-10-18T20:58:29.751 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:29.751 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:29.752 DEBUG:boto:Method: PUT 2023-10-18T20:58:29.752 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:29.753 DEBUG:boto:Data: 2023-10-18T20:58:29.753 DEBUG:boto:Headers: {} 2023-10-18T20:58:29.753 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:29.754 DEBUG:boto:Port: 8000 2023-10-18T20:58:29.754 DEBUG:boto:Params: {} 2023-10-18T20:58:29.755 DEBUG:boto:Token: None 2023-10-18T20:58:29.755 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 20:58:29 GMT /bofrnf-63/ 2023-10-18T20:58:29.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:esiPySdIkgPqGBUYBPOOjzeMmCI= 2023-10-18T20:58:29.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:esiPySdIkgPqGBUYBPOOjzeMmCI=', 'Content-Length': '0'} 2023-10-18T20:58:29.772 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cf0a49e39ccc2b1e-00653046f5-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:29.773 DEBUG:rgw_multi.tests:created bucket=bofrnf-63 2023-10-18T20:58:29.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:29.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:29.774 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:29.824 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:29.824 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:29.857 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T20:58:29.870 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T20:58:29.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.871 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662709.769954_1341.1", 2023-10-18T20:58:29.871 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:58:29.769954Z" 2023-10-18T20:58:29.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.872 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:58:29.872 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T20:58:29.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.873 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:58:29.873 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T20:58:29.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.873 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:58:29.874 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T20:58:29.874 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.874 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T20:58:29.875 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697662709.769954_1341.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:58:29.876 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T20:58:29.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:29.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T20:58:29.877 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T20:58:29.925 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:29.926 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:29.956 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:29.955+0000 7fc902dc2780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:58:29.957 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:29.956+0000 7fc902dc2780 20 RGW-SYNC:meta: read sync status 2023-10-18T20:58:29.970 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T20:58:29.970 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T20:58:29.970 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T20:58:29.970 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T20:58:29.971 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T20:58:29.971 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T20:58:29.971 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:58:29.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.971 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T20:58:29.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.972 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T20:58:29.972 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:29.972 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:58:29.972 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697662709.769954_1341.1", 2023-10-18T20:58:29.973 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:29.973 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T20:58:29.973 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:29.973 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:58:29.769954Z", 2023-10-18T20:58:29.973 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:58:29.973 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.974 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T20:58:29.974 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:29.974 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:58:29.975 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T20:58:29.975 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:29.975 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:29.975 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:29.976 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T20:58:29.976 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:58:29.976 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.977 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T20:58:29.977 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:29.977 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:58:29.977 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T20:58:29.977 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:29.977 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:29.978 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:29.978 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T20:58:29.978 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:58:29.978 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T20:58:29.979 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T20:58:29.979 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T20:58:29.979 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T20:58:29.979 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T20:58:29.980 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T20:58:29.980 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T20:58:29.980 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T20:58:29.980 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T20:58:29.980 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T20:58:29.980 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.981 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T20:58:29.981 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T20:58:29.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.981 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T20:58:29.981 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T20:58:29.982 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T20:58:29.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T20:58:29.982 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T20:58:29Z" 2023-10-18T20:58:29.982 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T20:58:29.983 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697662709.769954_1341.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T20:58:29.769954Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T20:58:29Z" } 2023-10-18T20:58:29.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697662709.769954_1341.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:58:29.769954Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T20:58:29.984 DEBUG:rgw_multi.tests:log_status={0: '1_1697662709.769954_1341.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:58:29.985 DEBUG:rgw_multi.tests:sync_status={0: '1_1697662709.769954_1341.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T20:58:29.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T20:58:29.985 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:29.986 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:29.986 DEBUG:boto:Method: HEAD 2023-10-18T20:58:29.987 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:29.987 DEBUG:boto:Data: 2023-10-18T20:58:29.987 DEBUG:boto:Headers: {} 2023-10-18T20:58:29.988 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:29.988 DEBUG:boto:Port: 8000 2023-10-18T20:58:29.989 DEBUG:boto:Params: {} 2023-10-18T20:58:29.989 DEBUG:boto:Token: None 2023-10-18T20:58:29.989 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:29 GMT /bofrnf-63/ 2023-10-18T20:58:29.990 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+E8hJKqrJZOCO18BDa8B0RfdY5Q= 2023-10-18T20:58:29.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+E8hJKqrJZOCO18BDa8B0RfdY5Q=', 'Content-Length': '0'} 2023-10-18T20:58:29.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000626cc1889a7a7828-00653046f5-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:29.994 DEBUG:boto:path=/bofrnf-63/obj11 2023-10-18T20:58:29.994 DEBUG:boto:auth_path=/bofrnf-63/obj11 2023-10-18T20:58:29.994 DEBUG:boto:Method: PUT 2023-10-18T20:58:29.995 DEBUG:boto:Path: /bofrnf-63/obj11 2023-10-18T20:58:29.995 DEBUG:boto:Data: 2023-10-18T20:58:29.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:29.996 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:29.996 DEBUG:boto:Port: 8000 2023-10-18T20:58:29.997 DEBUG:boto:Params: {} 2023-10-18T20:58:29.997 DEBUG:boto:Token: None 2023-10-18T20:58:29.998 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:29 GMT /bofrnf-63/obj11 2023-10-18T20:58:29.998 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xaru9oMWL8Zy5DnVtl0RzMm4kMM= 2023-10-18T20:58:29.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xaru9oMWL8Zy5DnVtl0RzMm4kMM='} 2023-10-18T20:58:30.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d7417adebad4228-00653046f5-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.005 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.006 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.006 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.006 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.007 DEBUG:boto:Data: 2023-10-18T20:58:30.007 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.008 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.008 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.008 DEBUG:boto:Params: {} 2023-10-18T20:58:30.009 DEBUG:boto:Token: None 2023-10-18T20:58:30.009 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.010 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d87182a0a6cd6da-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.013 DEBUG:boto:path=/bofrnf-63/obj22 2023-10-18T20:58:30.013 DEBUG:boto:auth_path=/bofrnf-63/obj22 2023-10-18T20:58:30.014 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.014 DEBUG:boto:Path: /bofrnf-63/obj22 2023-10-18T20:58:30.014 DEBUG:boto:Data: 2023-10-18T20:58:30.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.015 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.016 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.016 DEBUG:boto:Params: {} 2023-10-18T20:58:30.016 DEBUG:boto:Token: None 2023-10-18T20:58:30.017 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj22 2023-10-18T20:58:30.017 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3FXZjzhTaKOKqosy11k1TnYNLb4= 2023-10-18T20:58:30.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3FXZjzhTaKOKqosy11k1TnYNLb4='} 2023-10-18T20:58:30.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051336ccf498cfc5e-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.022 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.023 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.023 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.023 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.024 DEBUG:boto:Data: 2023-10-18T20:58:30.024 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.025 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.025 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.025 DEBUG:boto:Params: {} 2023-10-18T20:58:30.026 DEBUG:boto:Token: None 2023-10-18T20:58:30.026 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.027 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009791abb35afb66b-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.030 DEBUG:boto:path=/bofrnf-63/obj33 2023-10-18T20:58:30.030 DEBUG:boto:auth_path=/bofrnf-63/obj33 2023-10-18T20:58:30.031 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.031 DEBUG:boto:Path: /bofrnf-63/obj33 2023-10-18T20:58:30.031 DEBUG:boto:Data: 2023-10-18T20:58:30.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.032 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.033 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.033 DEBUG:boto:Params: {} 2023-10-18T20:58:30.033 DEBUG:boto:Token: None 2023-10-18T20:58:30.034 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj33 2023-10-18T20:58:30.034 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FRMLYArKTWTcS3HRXOkQPqGQGG0= 2023-10-18T20:58:30.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FRMLYArKTWTcS3HRXOkQPqGQGG0='} 2023-10-18T20:58:30.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0583cec00b111e1-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.040 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.041 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.041 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.042 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.042 DEBUG:boto:Data: 2023-10-18T20:58:30.042 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.043 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.043 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.044 DEBUG:boto:Params: {} 2023-10-18T20:58:30.044 DEBUG:boto:Token: None 2023-10-18T20:58:30.045 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.045 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029decb1f1c584234-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.048 DEBUG:boto:path=/bofrnf-63/obj44 2023-10-18T20:58:30.049 DEBUG:boto:auth_path=/bofrnf-63/obj44 2023-10-18T20:58:30.049 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.050 DEBUG:boto:Path: /bofrnf-63/obj44 2023-10-18T20:58:30.050 DEBUG:boto:Data: 2023-10-18T20:58:30.050 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.051 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.051 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.052 DEBUG:boto:Params: {} 2023-10-18T20:58:30.052 DEBUG:boto:Token: None 2023-10-18T20:58:30.053 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj44 2023-10-18T20:58:30.053 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1g1pud1A0JKYLVBTdqxIk9TIE5Q= 2023-10-18T20:58:30.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1g1pud1A0JKYLVBTdqxIk9TIE5Q='} 2023-10-18T20:58:30.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029516b1c1b58f7c5-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.099 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.099 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.100 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.100 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.100 DEBUG:boto:Data: 2023-10-18T20:58:30.101 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.101 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.101 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.102 DEBUG:boto:Params: {} 2023-10-18T20:58:30.102 DEBUG:boto:Token: None 2023-10-18T20:58:30.103 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.103 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095f0917cf1ff6a79-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.106 DEBUG:boto:path=/bofrnf-63/obj55 2023-10-18T20:58:30.107 DEBUG:boto:auth_path=/bofrnf-63/obj55 2023-10-18T20:58:30.107 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.107 DEBUG:boto:Path: /bofrnf-63/obj55 2023-10-18T20:58:30.108 DEBUG:boto:Data: 2023-10-18T20:58:30.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.109 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.109 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.109 DEBUG:boto:Params: {} 2023-10-18T20:58:30.110 DEBUG:boto:Token: None 2023-10-18T20:58:30.110 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj55 2023-10-18T20:58:30.111 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gqw50Q50/TdhqLeuyJRVCOTrd0M= 2023-10-18T20:58:30.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gqw50Q50/TdhqLeuyJRVCOTrd0M='} 2023-10-18T20:58:30.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4eafa0d09e3b585-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:30.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:30.116 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:58:30.168 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:30.168 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:30.213 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:58:30.213 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-63 2023-10-18T20:58:30.216 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 5 2023-10-18T20:58:30.253 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.253 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.254 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.254 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.254 DEBUG:boto:Data: 2023-10-18T20:58:30.255 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.255 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.255 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.256 DEBUG:boto:Params: {} 2023-10-18T20:58:30.256 DEBUG:boto:Token: None 2023-10-18T20:58:30.257 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.257 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004de7999d888f5688-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.261 DEBUG:boto:path=/bofrnf-63/obj17 2023-10-18T20:58:30.262 DEBUG:boto:auth_path=/bofrnf-63/obj17 2023-10-18T20:58:30.262 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.262 DEBUG:boto:Path: /bofrnf-63/obj17 2023-10-18T20:58:30.263 DEBUG:boto:Data: 2023-10-18T20:58:30.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.264 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.264 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.264 DEBUG:boto:Params: {} 2023-10-18T20:58:30.265 DEBUG:boto:Token: None 2023-10-18T20:58:30.265 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj17 2023-10-18T20:58:30.266 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/LrFkzu55NzrKRVhKtzKVZEkP7U= 2023-10-18T20:58:30.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/LrFkzu55NzrKRVhKtzKVZEkP7U='} 2023-10-18T20:58:30.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000185c546249df269c-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.273 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.273 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.274 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.274 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.275 DEBUG:boto:Data: 2023-10-18T20:58:30.275 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.275 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.276 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.276 DEBUG:boto:Params: {} 2023-10-18T20:58:30.276 DEBUG:boto:Token: None 2023-10-18T20:58:30.277 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.277 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9fc55a1e2c67962-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.281 DEBUG:boto:path=/bofrnf-63/obj34 2023-10-18T20:58:30.281 DEBUG:boto:auth_path=/bofrnf-63/obj34 2023-10-18T20:58:30.282 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.282 DEBUG:boto:Path: /bofrnf-63/obj34 2023-10-18T20:58:30.283 DEBUG:boto:Data: 2023-10-18T20:58:30.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.283 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.284 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.284 DEBUG:boto:Params: {} 2023-10-18T20:58:30.285 DEBUG:boto:Token: None 2023-10-18T20:58:30.285 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj34 2023-10-18T20:58:30.285 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5e7OgFW/6X0qHaFaLpzc49irIq0= 2023-10-18T20:58:30.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5e7OgFW/6X0qHaFaLpzc49irIq0='} 2023-10-18T20:58:30.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fa1b60ac4e79875-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.292 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.292 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.292 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.293 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.293 DEBUG:boto:Data: 2023-10-18T20:58:30.294 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.294 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.295 DEBUG:boto:Params: {} 2023-10-18T20:58:30.295 DEBUG:boto:Token: None 2023-10-18T20:58:30.296 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.296 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.299 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e782163d7cffa5e1-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.300 DEBUG:boto:path=/bofrnf-63/obj51 2023-10-18T20:58:30.300 DEBUG:boto:auth_path=/bofrnf-63/obj51 2023-10-18T20:58:30.300 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.301 DEBUG:boto:Path: /bofrnf-63/obj51 2023-10-18T20:58:30.301 DEBUG:boto:Data: 2023-10-18T20:58:30.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.302 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.302 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.303 DEBUG:boto:Params: {} 2023-10-18T20:58:30.303 DEBUG:boto:Token: None 2023-10-18T20:58:30.304 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj51 2023-10-18T20:58:30.304 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ju6NJioPld1uceh4j2naIo7zYrw= 2023-10-18T20:58:30.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ju6NJioPld1uceh4j2naIo7zYrw='} 2023-10-18T20:58:30.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db20aade8ee4b112-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.347 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.347 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.348 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.348 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.349 DEBUG:boto:Data: 2023-10-18T20:58:30.349 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.349 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.350 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.350 DEBUG:boto:Params: {} 2023-10-18T20:58:30.350 DEBUG:boto:Token: None 2023-10-18T20:58:30.351 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.351 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d561a278245fca0c-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.355 DEBUG:boto:path=/bofrnf-63/obj68 2023-10-18T20:58:30.355 DEBUG:boto:auth_path=/bofrnf-63/obj68 2023-10-18T20:58:30.355 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.356 DEBUG:boto:Path: /bofrnf-63/obj68 2023-10-18T20:58:30.356 DEBUG:boto:Data: 2023-10-18T20:58:30.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.357 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.357 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.358 DEBUG:boto:Params: {} 2023-10-18T20:58:30.358 DEBUG:boto:Token: None 2023-10-18T20:58:30.359 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj68 2023-10-18T20:58:30.359 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OWyT3IGRStQ7sf7T0dVggwvQRpw= 2023-10-18T20:58:30.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OWyT3IGRStQ7sf7T0dVggwvQRpw='} 2023-10-18T20:58:30.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001753c9089e17cf88-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.365 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.365 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.366 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.366 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.367 DEBUG:boto:Data: 2023-10-18T20:58:30.367 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.368 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.368 DEBUG:boto:Params: {} 2023-10-18T20:58:30.368 DEBUG:boto:Token: None 2023-10-18T20:58:30.369 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.369 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1fc373fe33130ca-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.373 DEBUG:boto:path=/bofrnf-63/obj85 2023-10-18T20:58:30.373 DEBUG:boto:auth_path=/bofrnf-63/obj85 2023-10-18T20:58:30.374 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.374 DEBUG:boto:Path: /bofrnf-63/obj85 2023-10-18T20:58:30.374 DEBUG:boto:Data: 2023-10-18T20:58:30.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.375 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.376 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.376 DEBUG:boto:Params: {} 2023-10-18T20:58:30.376 DEBUG:boto:Token: None 2023-10-18T20:58:30.377 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj85 2023-10-18T20:58:30.377 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Viy5nZ/Ux9BaxM0wN1kIsF0EtlU= 2023-10-18T20:58:30.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Viy5nZ/Ux9BaxM0wN1kIsF0EtlU='} 2023-10-18T20:58:30.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001f45f489d65e81c-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:30.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:30.384 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:58:30.436 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:30.436 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:30.481 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:58:30.481 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-63 2023-10-18T20:58:30.487 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 10 2023-10-18T20:58:30.530 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.530 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.531 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.531 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.532 DEBUG:boto:Data: 2023-10-18T20:58:30.532 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.532 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.533 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.533 DEBUG:boto:Params: {} 2023-10-18T20:58:30.534 DEBUG:boto:Token: None 2023-10-18T20:58:30.534 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.535 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000116dd4ccbda4bdc9-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.539 DEBUG:boto:path=/bofrnf-63/obj19 2023-10-18T20:58:30.539 DEBUG:boto:auth_path=/bofrnf-63/obj19 2023-10-18T20:58:30.539 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.540 DEBUG:boto:Path: /bofrnf-63/obj19 2023-10-18T20:58:30.540 DEBUG:boto:Data: 2023-10-18T20:58:30.541 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.541 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.541 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.542 DEBUG:boto:Params: {} 2023-10-18T20:58:30.542 DEBUG:boto:Token: None 2023-10-18T20:58:30.543 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj19 2023-10-18T20:58:30.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:RBbDnHZMGztlmdj53XUImqMxxSA= 2023-10-18T20:58:30.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RBbDnHZMGztlmdj53XUImqMxxSA='} 2023-10-18T20:58:30.549 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4a5513f305f6f48-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.549 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.549 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.550 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.550 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.551 DEBUG:boto:Data: 2023-10-18T20:58:30.551 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.551 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.552 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.552 DEBUG:boto:Params: {} 2023-10-18T20:58:30.553 DEBUG:boto:Token: None 2023-10-18T20:58:30.553 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.554 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003af29c67302aec2e-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.558 DEBUG:boto:path=/bofrnf-63/obj38 2023-10-18T20:58:30.558 DEBUG:boto:auth_path=/bofrnf-63/obj38 2023-10-18T20:58:30.558 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.559 DEBUG:boto:Path: /bofrnf-63/obj38 2023-10-18T20:58:30.559 DEBUG:boto:Data: 2023-10-18T20:58:30.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.560 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.560 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.561 DEBUG:boto:Params: {} 2023-10-18T20:58:30.561 DEBUG:boto:Token: None 2023-10-18T20:58:30.562 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj38 2023-10-18T20:58:30.562 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PehESfneRAiSmw67BFXBTy+mnLM= 2023-10-18T20:58:30.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PehESfneRAiSmw67BFXBTy+mnLM='} 2023-10-18T20:58:30.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000234d3bcb39ecd12f-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.607 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.608 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.608 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.609 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.610 DEBUG:boto:Data: 2023-10-18T20:58:30.610 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.610 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.611 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.611 DEBUG:boto:Params: {} 2023-10-18T20:58:30.612 DEBUG:boto:Token: None 2023-10-18T20:58:30.612 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.613 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073860f8cfeb7647f-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.617 DEBUG:boto:path=/bofrnf-63/obj57 2023-10-18T20:58:30.617 DEBUG:boto:auth_path=/bofrnf-63/obj57 2023-10-18T20:58:30.618 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.618 DEBUG:boto:Path: /bofrnf-63/obj57 2023-10-18T20:58:30.619 DEBUG:boto:Data: 2023-10-18T20:58:30.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.619 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.620 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.620 DEBUG:boto:Params: {} 2023-10-18T20:58:30.621 DEBUG:boto:Token: None 2023-10-18T20:58:30.621 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj57 2023-10-18T20:58:30.621 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DVF64RZIRRdiDLgom7icLtbk6L4= 2023-10-18T20:58:30.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DVF64RZIRRdiDLgom7icLtbk6L4='} 2023-10-18T20:58:30.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007833e6b24e0fcaa3-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.626 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.627 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.627 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.627 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.628 DEBUG:boto:Data: 2023-10-18T20:58:30.628 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.628 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.629 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.629 DEBUG:boto:Params: {} 2023-10-18T20:58:30.630 DEBUG:boto:Token: None 2023-10-18T20:58:30.630 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.631 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000924eaf0e8e55674e-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.635 DEBUG:boto:path=/bofrnf-63/obj76 2023-10-18T20:58:30.635 DEBUG:boto:auth_path=/bofrnf-63/obj76 2023-10-18T20:58:30.635 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.636 DEBUG:boto:Path: /bofrnf-63/obj76 2023-10-18T20:58:30.636 DEBUG:boto:Data: 2023-10-18T20:58:30.637 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.637 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.637 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.638 DEBUG:boto:Params: {} 2023-10-18T20:58:30.638 DEBUG:boto:Token: None 2023-10-18T20:58:30.639 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj76 2023-10-18T20:58:30.639 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xZNcGuz5HJu72wQdWx3zU6npyvU= 2023-10-18T20:58:30.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xZNcGuz5HJu72wQdWx3zU6npyvU='} 2023-10-18T20:58:30.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000709f5a4399425dc6-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.645 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.645 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.646 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.646 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.646 DEBUG:boto:Data: 2023-10-18T20:58:30.647 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.647 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.648 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.648 DEBUG:boto:Params: {} 2023-10-18T20:58:30.648 DEBUG:boto:Token: None 2023-10-18T20:58:30.649 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.649 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3a5c848236b71dc-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.653 DEBUG:boto:path=/bofrnf-63/obj95 2023-10-18T20:58:30.654 DEBUG:boto:auth_path=/bofrnf-63/obj95 2023-10-18T20:58:30.654 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.654 DEBUG:boto:Path: /bofrnf-63/obj95 2023-10-18T20:58:30.655 DEBUG:boto:Data: 2023-10-18T20:58:30.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.656 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.656 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.656 DEBUG:boto:Params: {} 2023-10-18T20:58:30.657 DEBUG:boto:Token: None 2023-10-18T20:58:30.657 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj95 2023-10-18T20:58:30.658 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:g739m2NDOYln1G9PTRETiNKWK38= 2023-10-18T20:58:30.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g739m2NDOYln1G9PTRETiNKWK38='} 2023-10-18T20:58:30.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000031123912827bfb5-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:30.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:30.704 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:58:30.756 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:30.756 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:30.800 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:58:30.800 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-63 2023-10-18T20:58:30.807 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 15 2023-10-18T20:58:30.852 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.853 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.853 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.854 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.854 DEBUG:boto:Data: 2023-10-18T20:58:30.855 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.855 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.856 DEBUG:boto:Params: {} 2023-10-18T20:58:30.856 DEBUG:boto:Token: None 2023-10-18T20:58:30.857 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.857 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018320f291c3ab751-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.861 DEBUG:boto:path=/bofrnf-63/obj23 2023-10-18T20:58:30.861 DEBUG:boto:auth_path=/bofrnf-63/obj23 2023-10-18T20:58:30.862 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.862 DEBUG:boto:Path: /bofrnf-63/obj23 2023-10-18T20:58:30.863 DEBUG:boto:Data: 2023-10-18T20:58:30.863 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.864 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.864 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.864 DEBUG:boto:Params: {} 2023-10-18T20:58:30.865 DEBUG:boto:Token: None 2023-10-18T20:58:30.865 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj23 2023-10-18T20:58:30.866 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dK69cRXYuO0DR5azR8mHIX+hNdk= 2023-10-18T20:58:30.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dK69cRXYuO0DR5azR8mHIX+hNdk='} 2023-10-18T20:58:30.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fd02e945a0428c4-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.872 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.872 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.873 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.873 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.874 DEBUG:boto:Data: 2023-10-18T20:58:30.874 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.874 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.875 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.875 DEBUG:boto:Params: {} 2023-10-18T20:58:30.876 DEBUG:boto:Token: None 2023-10-18T20:58:30.876 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.876 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055943bc4977032ad-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.880 DEBUG:boto:path=/bofrnf-63/obj46 2023-10-18T20:58:30.881 DEBUG:boto:auth_path=/bofrnf-63/obj46 2023-10-18T20:58:30.881 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.881 DEBUG:boto:Path: /bofrnf-63/obj46 2023-10-18T20:58:30.882 DEBUG:boto:Data: 2023-10-18T20:58:30.882 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.883 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.883 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.883 DEBUG:boto:Params: {} 2023-10-18T20:58:30.884 DEBUG:boto:Token: None 2023-10-18T20:58:30.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj46 2023-10-18T20:58:30.885 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vBL45m0ayN+2ldl0KhlrjxIAUQk= 2023-10-18T20:58:30.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vBL45m0ayN+2ldl0KhlrjxIAUQk='} 2023-10-18T20:58:30.891 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022b0d1b9826f3300-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.891 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.892 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.892 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.892 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.893 DEBUG:boto:Data: 2023-10-18T20:58:30.893 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.894 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.894 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.894 DEBUG:boto:Params: {} 2023-10-18T20:58:30.895 DEBUG:boto:Token: None 2023-10-18T20:58:30.895 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.896 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1a3b0d77cd31e60-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.899 DEBUG:boto:path=/bofrnf-63/obj69 2023-10-18T20:58:30.900 DEBUG:boto:auth_path=/bofrnf-63/obj69 2023-10-18T20:58:30.900 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.900 DEBUG:boto:Path: /bofrnf-63/obj69 2023-10-18T20:58:30.901 DEBUG:boto:Data: 2023-10-18T20:58:30.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.902 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.902 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.902 DEBUG:boto:Params: {} 2023-10-18T20:58:30.903 DEBUG:boto:Token: None 2023-10-18T20:58:30.903 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj69 2023-10-18T20:58:30.904 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4Hjw6dnP7ew7GJX0K0e2FUCEv6o= 2023-10-18T20:58:30.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4Hjw6dnP7ew7GJX0K0e2FUCEv6o='} 2023-10-18T20:58:30.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6056d14e4660ef7-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.947 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.948 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.948 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.948 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.949 DEBUG:boto:Data: 2023-10-18T20:58:30.949 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.950 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.950 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.950 DEBUG:boto:Params: {} 2023-10-18T20:58:30.951 DEBUG:boto:Token: None 2023-10-18T20:58:30.951 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.952 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c66a84565c859ede-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.956 DEBUG:boto:path=/bofrnf-63/obj92 2023-10-18T20:58:30.956 DEBUG:boto:auth_path=/bofrnf-63/obj92 2023-10-18T20:58:30.956 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.957 DEBUG:boto:Path: /bofrnf-63/obj92 2023-10-18T20:58:30.957 DEBUG:boto:Data: 2023-10-18T20:58:30.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.958 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.959 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.959 DEBUG:boto:Params: {} 2023-10-18T20:58:30.959 DEBUG:boto:Token: None 2023-10-18T20:58:30.960 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj92 2023-10-18T20:58:30.960 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Fq/s6egPVugluMONwycQ1YSzVLM= 2023-10-18T20:58:30.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Fq/s6egPVugluMONwycQ1YSzVLM='} 2023-10-18T20:58:30.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1e6ba18ddef07d6-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.966 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:30.967 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:30.967 DEBUG:boto:Method: HEAD 2023-10-18T20:58:30.968 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:30.968 DEBUG:boto:Data: 2023-10-18T20:58:30.968 DEBUG:boto:Headers: {} 2023-10-18T20:58:30.969 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.969 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.970 DEBUG:boto:Params: {} 2023-10-18T20:58:30.970 DEBUG:boto:Token: None 2023-10-18T20:58:30.970 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/ 2023-10-18T20:58:30.971 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T20:58:30.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T20:58:30.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e5aaa590b621fb1-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:30.974 DEBUG:boto:path=/bofrnf-63/obj115 2023-10-18T20:58:30.975 DEBUG:boto:auth_path=/bofrnf-63/obj115 2023-10-18T20:58:30.975 DEBUG:boto:Method: PUT 2023-10-18T20:58:30.976 DEBUG:boto:Path: /bofrnf-63/obj115 2023-10-18T20:58:30.976 DEBUG:boto:Data: 2023-10-18T20:58:30.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:30.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:30.977 DEBUG:boto:Port: 8000 2023-10-18T20:58:30.978 DEBUG:boto:Params: {} 2023-10-18T20:58:30.978 DEBUG:boto:Token: None 2023-10-18T20:58:30.978 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:30 GMT /bofrnf-63/obj115 2023-10-18T20:58:30.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AlenhjI9HjheMPfoLWCMrzvNR5A= 2023-10-18T20:58:30.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AlenhjI9HjheMPfoLWCMrzvNR5A='} 2023-10-18T20:58:31.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf040cbb49b271d2-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:31.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:31.028 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:58:31.081 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:31.081 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:31.127 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:58:31.127 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-63 2023-10-18T20:58:31.135 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 20 2023-10-18T20:58:31.185 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.186 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.187 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.187 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.187 DEBUG:boto:Data: 2023-10-18T20:58:31.188 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.188 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.189 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.189 DEBUG:boto:Params: {} 2023-10-18T20:58:31.189 DEBUG:boto:Token: None 2023-10-18T20:58:31.190 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.190 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffddcee777021e01-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.195 DEBUG:boto:path=/bofrnf-63/obj29 2023-10-18T20:58:31.195 DEBUG:boto:auth_path=/bofrnf-63/obj29 2023-10-18T20:58:31.196 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.196 DEBUG:boto:Path: /bofrnf-63/obj29 2023-10-18T20:58:31.197 DEBUG:boto:Data: 2023-10-18T20:58:31.197 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.198 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.198 DEBUG:boto:Params: {} 2023-10-18T20:58:31.199 DEBUG:boto:Token: None 2023-10-18T20:58:31.200 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj29 2023-10-18T20:58:31.200 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5ESbkCNdQKcfMzDteJ+7X6TOobs= 2023-10-18T20:58:31.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5ESbkCNdQKcfMzDteJ+7X6TOobs='} 2023-10-18T20:58:31.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000335695faa5c741dd-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.206 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.207 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.207 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.208 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.208 DEBUG:boto:Data: 2023-10-18T20:58:31.208 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.209 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.209 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.210 DEBUG:boto:Params: {} 2023-10-18T20:58:31.210 DEBUG:boto:Token: None 2023-10-18T20:58:31.211 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bab2f24589b66548-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.215 DEBUG:boto:path=/bofrnf-63/obj58 2023-10-18T20:58:31.215 DEBUG:boto:auth_path=/bofrnf-63/obj58 2023-10-18T20:58:31.216 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.216 DEBUG:boto:Path: /bofrnf-63/obj58 2023-10-18T20:58:31.217 DEBUG:boto:Data: 2023-10-18T20:58:31.217 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.217 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.218 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.218 DEBUG:boto:Params: {} 2023-10-18T20:58:31.219 DEBUG:boto:Token: None 2023-10-18T20:58:31.219 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj58 2023-10-18T20:58:31.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:K8gYmn+9/8nx7RYJlcbpoKnh9RY= 2023-10-18T20:58:31.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:K8gYmn+9/8nx7RYJlcbpoKnh9RY='} 2023-10-18T20:58:31.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6a1b19fe8a5f6f4-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.227 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.228 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.228 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.229 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.229 DEBUG:boto:Data: 2023-10-18T20:58:31.230 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.231 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.231 DEBUG:boto:Params: {} 2023-10-18T20:58:31.232 DEBUG:boto:Token: None 2023-10-18T20:58:31.232 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.233 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da01c1c56aad83b2-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.237 DEBUG:boto:path=/bofrnf-63/obj87 2023-10-18T20:58:31.238 DEBUG:boto:auth_path=/bofrnf-63/obj87 2023-10-18T20:58:31.238 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.238 DEBUG:boto:Path: /bofrnf-63/obj87 2023-10-18T20:58:31.239 DEBUG:boto:Data: 2023-10-18T20:58:31.239 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.240 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.240 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.240 DEBUG:boto:Params: {} 2023-10-18T20:58:31.241 DEBUG:boto:Token: None 2023-10-18T20:58:31.241 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj87 2023-10-18T20:58:31.242 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8O3zSw8TgsuuqdRkaua31VJCSr4= 2023-10-18T20:58:31.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8O3zSw8TgsuuqdRkaua31VJCSr4='} 2023-10-18T20:58:31.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d203ec4a370f934c-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.291 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.291 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.292 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.292 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.293 DEBUG:boto:Data: 2023-10-18T20:58:31.293 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.294 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.294 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.294 DEBUG:boto:Params: {} 2023-10-18T20:58:31.295 DEBUG:boto:Token: None 2023-10-18T20:58:31.296 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.296 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c48a97173b63cbbc-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.300 DEBUG:boto:path=/bofrnf-63/obj116 2023-10-18T20:58:31.301 DEBUG:boto:auth_path=/bofrnf-63/obj116 2023-10-18T20:58:31.301 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.302 DEBUG:boto:Path: /bofrnf-63/obj116 2023-10-18T20:58:31.302 DEBUG:boto:Data: 2023-10-18T20:58:31.303 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.303 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.303 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.304 DEBUG:boto:Params: {} 2023-10-18T20:58:31.304 DEBUG:boto:Token: None 2023-10-18T20:58:31.305 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj116 2023-10-18T20:58:31.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:UyP7Xv/QXuCJPyN7OllDoB6QzZc= 2023-10-18T20:58:31.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UyP7Xv/QXuCJPyN7OllDoB6QzZc='} 2023-10-18T20:58:31.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff0ad44343c3bdc1-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.311 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.311 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.312 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.312 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.313 DEBUG:boto:Data: 2023-10-18T20:58:31.313 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.314 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.314 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.314 DEBUG:boto:Params: {} 2023-10-18T20:58:31.315 DEBUG:boto:Token: None 2023-10-18T20:58:31.315 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.316 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdeb85826b692f38-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.320 DEBUG:boto:path=/bofrnf-63/obj145 2023-10-18T20:58:31.320 DEBUG:boto:auth_path=/bofrnf-63/obj145 2023-10-18T20:58:31.321 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.321 DEBUG:boto:Path: /bofrnf-63/obj145 2023-10-18T20:58:31.321 DEBUG:boto:Data: 2023-10-18T20:58:31.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.323 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.323 DEBUG:boto:Params: {} 2023-10-18T20:58:31.324 DEBUG:boto:Token: None 2023-10-18T20:58:31.324 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj145 2023-10-18T20:58:31.325 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:VQ0CA4OgS+U8B76Xa25o7qKMrEE= 2023-10-18T20:58:31.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VQ0CA4OgS+U8B76Xa25o7qKMrEE='} 2023-10-18T20:58:31.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbd5961a57906524-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:31.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:31.372 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:58:31.425 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:31.426 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:31.474 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:58:31.474 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-63 2023-10-18T20:58:31.484 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 25 2023-10-18T20:58:31.540 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.540 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.541 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.541 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.542 DEBUG:boto:Data: 2023-10-18T20:58:31.542 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.543 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.543 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.543 DEBUG:boto:Params: {} 2023-10-18T20:58:31.544 DEBUG:boto:Token: None 2023-10-18T20:58:31.544 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a33b6c4497a214c-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.550 DEBUG:boto:path=/bofrnf-63/obj31 2023-10-18T20:58:31.550 DEBUG:boto:auth_path=/bofrnf-63/obj31 2023-10-18T20:58:31.551 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.551 DEBUG:boto:Path: /bofrnf-63/obj31 2023-10-18T20:58:31.551 DEBUG:boto:Data: 2023-10-18T20:58:31.552 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.552 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.553 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.553 DEBUG:boto:Params: {} 2023-10-18T20:58:31.554 DEBUG:boto:Token: None 2023-10-18T20:58:31.554 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj31 2023-10-18T20:58:31.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yhA18IgbRQKtt4OGQu3GGprw0zk= 2023-10-18T20:58:31.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yhA18IgbRQKtt4OGQu3GGprw0zk='} 2023-10-18T20:58:31.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006909c8a8de842fd6-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.561 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.561 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.562 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.562 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.563 DEBUG:boto:Data: 2023-10-18T20:58:31.563 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.563 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.564 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.564 DEBUG:boto:Params: {} 2023-10-18T20:58:31.565 DEBUG:boto:Token: None 2023-10-18T20:58:31.565 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.566 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000febdda13c97d1d37-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.571 DEBUG:boto:path=/bofrnf-63/obj62 2023-10-18T20:58:31.571 DEBUG:boto:auth_path=/bofrnf-63/obj62 2023-10-18T20:58:31.571 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.572 DEBUG:boto:Path: /bofrnf-63/obj62 2023-10-18T20:58:31.573 DEBUG:boto:Data: 2023-10-18T20:58:31.573 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.573 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.574 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.574 DEBUG:boto:Params: {} 2023-10-18T20:58:31.574 DEBUG:boto:Token: None 2023-10-18T20:58:31.575 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj62 2023-10-18T20:58:31.575 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kqYLlzD5bid1hbllk/ILJUqOqMY= 2023-10-18T20:58:31.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kqYLlzD5bid1hbllk/ILJUqOqMY='} 2023-10-18T20:58:31.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ed4c172487ff10f-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.581 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.581 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.582 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.583 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.583 DEBUG:boto:Data: 2023-10-18T20:58:31.583 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.584 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.584 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.585 DEBUG:boto:Params: {} 2023-10-18T20:58:31.585 DEBUG:boto:Token: None 2023-10-18T20:58:31.585 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.586 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003009fab0fe2bbebc-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.590 DEBUG:boto:path=/bofrnf-63/obj93 2023-10-18T20:58:31.590 DEBUG:boto:auth_path=/bofrnf-63/obj93 2023-10-18T20:58:31.591 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.591 DEBUG:boto:Path: /bofrnf-63/obj93 2023-10-18T20:58:31.592 DEBUG:boto:Data: 2023-10-18T20:58:31.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.592 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.593 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.593 DEBUG:boto:Params: {} 2023-10-18T20:58:31.594 DEBUG:boto:Token: None 2023-10-18T20:58:31.594 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj93 2023-10-18T20:58:31.595 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:549fEJ8JMkb+OmW/mWJCquzOKW0= 2023-10-18T20:58:31.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:549fEJ8JMkb+OmW/mWJCquzOKW0='} 2023-10-18T20:58:31.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3ede4cb7edbaf32-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.601 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.601 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.601 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.602 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.602 DEBUG:boto:Data: 2023-10-18T20:58:31.603 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.603 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.604 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.604 DEBUG:boto:Params: {} 2023-10-18T20:58:31.604 DEBUG:boto:Token: None 2023-10-18T20:58:31.605 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000193a4f1ba1568d9f-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.609 DEBUG:boto:path=/bofrnf-63/obj124 2023-10-18T20:58:31.610 DEBUG:boto:auth_path=/bofrnf-63/obj124 2023-10-18T20:58:31.610 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.611 DEBUG:boto:Path: /bofrnf-63/obj124 2023-10-18T20:58:31.611 DEBUG:boto:Data: 2023-10-18T20:58:31.612 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.612 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.613 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.613 DEBUG:boto:Params: {} 2023-10-18T20:58:31.614 DEBUG:boto:Token: None 2023-10-18T20:58:31.614 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj124 2023-10-18T20:58:31.615 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:69J+VS4By/LO7oCgneBR1SHi2hA= 2023-10-18T20:58:31.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:69J+VS4By/LO7oCgneBR1SHi2hA='} 2023-10-18T20:58:31.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087ccd83f93e157b1-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.659 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.660 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.660 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.660 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.661 DEBUG:boto:Data: 2023-10-18T20:58:31.661 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.662 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.662 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.662 DEBUG:boto:Params: {} 2023-10-18T20:58:31.663 DEBUG:boto:Token: None 2023-10-18T20:58:31.663 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.664 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052b2c9b864e91441-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.668 DEBUG:boto:path=/bofrnf-63/obj155 2023-10-18T20:58:31.668 DEBUG:boto:auth_path=/bofrnf-63/obj155 2023-10-18T20:58:31.669 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.669 DEBUG:boto:Path: /bofrnf-63/obj155 2023-10-18T20:58:31.670 DEBUG:boto:Data: 2023-10-18T20:58:31.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.670 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.671 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.671 DEBUG:boto:Params: {} 2023-10-18T20:58:31.672 DEBUG:boto:Token: None 2023-10-18T20:58:31.672 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj155 2023-10-18T20:58:31.673 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:21cV7j+LgtEy904i+NiTgPfmzZQ= 2023-10-18T20:58:31.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:21cV7j+LgtEy904i+NiTgPfmzZQ='} 2023-10-18T20:58:31.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b83e059bfecf3d44-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:31.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T20:58:31.680 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T20:58:31.732 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:31.732 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:31.783 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T20:58:31.783 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-63 2023-10-18T20:58:31.794 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 30 2023-10-18T20:58:31.853 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.853 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.854 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.854 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.855 DEBUG:boto:Data: 2023-10-18T20:58:31.855 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.855 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.856 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.856 DEBUG:boto:Params: {} 2023-10-18T20:58:31.857 DEBUG:boto:Token: None 2023-10-18T20:58:31.857 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.857 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4fc535d8f9c4c12-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.862 DEBUG:boto:path=/bofrnf-63/obj37 2023-10-18T20:58:31.862 DEBUG:boto:auth_path=/bofrnf-63/obj37 2023-10-18T20:58:31.863 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.863 DEBUG:boto:Path: /bofrnf-63/obj37 2023-10-18T20:58:31.864 DEBUG:boto:Data: 2023-10-18T20:58:31.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.864 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.865 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.865 DEBUG:boto:Params: {} 2023-10-18T20:58:31.866 DEBUG:boto:Token: None 2023-10-18T20:58:31.866 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj37 2023-10-18T20:58:31.866 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:49Pd/txzmMHEV6cZRNk0xyg0Eqo= 2023-10-18T20:58:31.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:49Pd/txzmMHEV6cZRNk0xyg0Eqo='} 2023-10-18T20:58:31.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2e7dd6e3d5906ad-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.873 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.873 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.873 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.874 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.874 DEBUG:boto:Data: 2023-10-18T20:58:31.875 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.875 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.875 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.876 DEBUG:boto:Params: {} 2023-10-18T20:58:31.876 DEBUG:boto:Token: None 2023-10-18T20:58:31.877 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.877 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000065fac21ccbd2e5e-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.881 DEBUG:boto:path=/bofrnf-63/obj74 2023-10-18T20:58:31.881 DEBUG:boto:auth_path=/bofrnf-63/obj74 2023-10-18T20:58:31.882 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.882 DEBUG:boto:Path: /bofrnf-63/obj74 2023-10-18T20:58:31.883 DEBUG:boto:Data: 2023-10-18T20:58:31.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.884 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.884 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.884 DEBUG:boto:Params: {} 2023-10-18T20:58:31.885 DEBUG:boto:Token: None 2023-10-18T20:58:31.885 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj74 2023-10-18T20:58:31.886 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Vw+HWikspvpRCIeDc5uBHh4Ju1s= 2023-10-18T20:58:31.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vw+HWikspvpRCIeDc5uBHh4Ju1s='} 2023-10-18T20:58:31.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e64a1c647e0be4c-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.931 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.931 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.932 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.932 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.932 DEBUG:boto:Data: 2023-10-18T20:58:31.933 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.933 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.934 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.934 DEBUG:boto:Params: {} 2023-10-18T20:58:31.935 DEBUG:boto:Token: None 2023-10-18T20:58:31.935 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.939 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001101a6b242c7da31-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.940 DEBUG:boto:path=/bofrnf-63/obj111 2023-10-18T20:58:31.940 DEBUG:boto:auth_path=/bofrnf-63/obj111 2023-10-18T20:58:31.941 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.941 DEBUG:boto:Path: /bofrnf-63/obj111 2023-10-18T20:58:31.942 DEBUG:boto:Data: 2023-10-18T20:58:31.942 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.943 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.943 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.943 DEBUG:boto:Params: {} 2023-10-18T20:58:31.944 DEBUG:boto:Token: None 2023-10-18T20:58:31.945 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj111 2023-10-18T20:58:31.945 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xfose7/t8dcmGGPtR3RSbC+8dyY= 2023-10-18T20:58:31.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xfose7/t8dcmGGPtR3RSbC+8dyY='} 2023-10-18T20:58:31.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2b21b626b826885-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.951 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:31.952 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:31.952 DEBUG:boto:Method: HEAD 2023-10-18T20:58:31.953 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:31.953 DEBUG:boto:Data: 2023-10-18T20:58:31.953 DEBUG:boto:Headers: {} 2023-10-18T20:58:31.954 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.954 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.955 DEBUG:boto:Params: {} 2023-10-18T20:58:31.955 DEBUG:boto:Token: None 2023-10-18T20:58:31.955 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/ 2023-10-18T20:58:31.956 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T20:58:31.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T20:58:31.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8cb5f567c959d3e-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:31.960 DEBUG:boto:path=/bofrnf-63/obj148 2023-10-18T20:58:31.961 DEBUG:boto:auth_path=/bofrnf-63/obj148 2023-10-18T20:58:31.961 DEBUG:boto:Method: PUT 2023-10-18T20:58:31.962 DEBUG:boto:Path: /bofrnf-63/obj148 2023-10-18T20:58:31.962 DEBUG:boto:Data: 2023-10-18T20:58:31.963 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:31.963 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:31.963 DEBUG:boto:Port: 8000 2023-10-18T20:58:31.964 DEBUG:boto:Params: {} 2023-10-18T20:58:31.964 DEBUG:boto:Token: None 2023-10-18T20:58:31.965 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:31 GMT /bofrnf-63/obj148 2023-10-18T20:58:31.965 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:b1SvQ0yI8nVfjmRe82+qYX7eqzQ= 2023-10-18T20:58:31.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b1SvQ0yI8nVfjmRe82+qYX7eqzQ='} 2023-10-18T20:58:32.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f410356dd80a8847-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:32.011 DEBUG:boto:path=/bofrnf-63/ 2023-10-18T20:58:32.011 DEBUG:boto:auth_path=/bofrnf-63/ 2023-10-18T20:58:32.012 DEBUG:boto:Method: HEAD 2023-10-18T20:58:32.012 DEBUG:boto:Path: /bofrnf-63/ 2023-10-18T20:58:32.012 DEBUG:boto:Data: 2023-10-18T20:58:32.013 DEBUG:boto:Headers: {} 2023-10-18T20:58:32.013 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:32.014 DEBUG:boto:Port: 8000 2023-10-18T20:58:32.014 DEBUG:boto:Params: {} 2023-10-18T20:58:32.015 DEBUG:boto:Token: None 2023-10-18T20:58:32.015 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 20:58:32 GMT /bofrnf-63/ 2023-10-18T20:58:32.015 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:aBBUFVdTZH43i7We+FzM8Ltyi9k= 2023-10-18T20:58:32.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aBBUFVdTZH43i7We+FzM8Ltyi9k=', 'Content-Length': '0'} 2023-10-18T20:58:32.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041144c3132dc6283-00653046f8-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:32.020 DEBUG:boto:path=/bofrnf-63/obj185 2023-10-18T20:58:32.020 DEBUG:boto:auth_path=/bofrnf-63/obj185 2023-10-18T20:58:32.021 DEBUG:boto:Method: PUT 2023-10-18T20:58:32.021 DEBUG:boto:Path: /bofrnf-63/obj185 2023-10-18T20:58:32.022 DEBUG:boto:Data: 2023-10-18T20:58:32.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T20:58:32.023 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T20:58:32.023 DEBUG:boto:Port: 8000 2023-10-18T20:58:32.023 DEBUG:boto:Params: {} 2023-10-18T20:58:32.024 DEBUG:boto:Token: None 2023-10-18T20:58:32.024 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 20:58:32 GMT /bofrnf-63/obj185 2023-10-18T20:58:32.025 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ph+pyzTtiVbh2DwlaBIhbK2JFGI= 2023-10-18T20:58:32.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ph+pyzTtiVbh2DwlaBIhbK2JFGI='} 2023-10-18T20:58:32.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a2f44a441cb6827-00653046f8-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T20:58:32.031 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-63 2023-10-18T20:58:32.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:58:32.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:58:32.032 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:58:32.087 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:32.088 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:32.123 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:32.121+0000 7f1d7a60d780 1 waiting to reach incremental sync.. 2023-10-18T20:58:37.123 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:37.122+0000 7f1d7a60d780 1 waiting to reach incremental sync.. 2023-10-18T20:58:42.124 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:42.123+0000 7f1d7a60d780 1 waiting to reach incremental sync.. 2023-10-18T20:58:47.125 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:47.122+0000 7f1d7a60d780 1 waiting to reach incremental sync.. 2023-10-18T20:58:52.128 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:52.126+0000 7f1d7a60d780 1 bucket sync caught up with source: 2023-10-18T20:58:52.129 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.115.6, , , , , 00000000002.129.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.115.6] 2023-10-18T20:58:52.129 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.115.6, , , , , 00000000002.129.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.115.6] 2023-10-18T20:58:52.129 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:52.126+0000 7f1d7a60d780 0 bucket checkpoint complete 2023-10-18T20:58:52.145 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-63 2023-10-18T20:58:52.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:58:52.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T20:58:52.147 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T20:58:52.210 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T20:58:52.210 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T20:58:52.246 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:52.244+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:58:57.248 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:58:57.245+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:02.248 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:02.247+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:07.249 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:07.247+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:12.249 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:12.247+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:17.249 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:17.248+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:22.250 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:22.249+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:27.251 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:27.250+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:32.252 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:32.250+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:37.252 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:37.251+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:42.253 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:42.251+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:47.253 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:47.251+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:52.254 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:52.252+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T20:59:57.255 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T20:59:57.254+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:02.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:02.254+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:07.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:07.255+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:12.257 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:12.255+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:17.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:17.256+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:22.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:22.256+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:27.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:27.257+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:32.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:32.257+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:37.260 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:37.258+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:42.261 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:42.259+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:47.261 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:47.260+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:52.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:52.260+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:00:57.263 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:00:57.261+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:02.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:02.262+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:07.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:07.262+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:12.266 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:12.263+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:17.265 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:17.264+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:22.266 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:22.263+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:27.266 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:27.264+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:32.267 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:32.265+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:37.267 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:37.266+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:42.268 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:42.266+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:47.268 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:47.267+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:52.269 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:52.268+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:01:57.270 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:01:57.269+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:02.271 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:02.269+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:07.271 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:07.270+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:12.272 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:12.271+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:17.273 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:17.271+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:22.274 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:22.271+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:27.274 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:27.272+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:32.274 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:32.273+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:37.275 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:37.274+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:42.276 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:42.274+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:47.276 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:47.275+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:52.277 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:52.275+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:02:57.278 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:02:57.276+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:02.279 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:02.277+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:07.279 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:07.277+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:12.280 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:12.278+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:17.281 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:17.279+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:22.281 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:22.280+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:27.282 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:27.281+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:32.283 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:32.281+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:37.282+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:42.284 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:42.282+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:03:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:47.283+0000 7fc6c5775780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-10-18T21:03:47.285 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:47.283+0000 7fc6c5775780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:03:47.285 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:47.283+0000 7fc6c5775780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:03:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-18T21:03:47.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-10-18T21:03:47.304 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:03:47.305 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:03:47.305 DEBUG:boto:Method: PUT 2023-10-18T21:03:47.305 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:03:47.306 DEBUG:boto:Data: 2023-10-18T21:03:47.306 DEBUG:boto:Headers: {} 2023-10-18T21:03:47.307 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.307 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.308 DEBUG:boto:Params: {} 2023-10-18T21:03:47.308 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:03:47.309 DEBUG:boto:Token: None 2023-10-18T21:03:47.309 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/ 2023-10-18T21:03:47.310 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gsRi8Y+QHfxrf7UcdER9jNlSw3I= 2023-10-18T21:03:47.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gsRi8Y+QHfxrf7UcdER9jNlSw3I=', 'Content-Length': '0'} 2023-10-18T21:03:47.331 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000037e0ae8e2045dba8-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.331 DEBUG:rgw_multi.tests:created bucket=bofrnf-64 2023-10-18T21:03:47.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:03:47.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:03:47.333 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:03:47.386 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:03:47.386 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:03:47.422 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:03:47.435 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:03:47.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:03:47.435 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663027.328641_1439.1", 2023-10-18T21:03:47.436 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:03:47.328641Z" 2023-10-18T21:03:47.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:03:47.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:03:47.436 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:03:47.436 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:03:47.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:03:47.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:03:47.437 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:03:47.437 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:03:47.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:03:47.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:03:47.438 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:03:47.438 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:03:47.438 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:03:47.439 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:03:47.440 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663027.328641_1439.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:03:47.440 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:03:47.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:03:47.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:03:47.441 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:03:47.492 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:03:47.492 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:03:47.520 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:47.519+0000 7fc0aed04780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:03:47.521 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:47.520+0000 7fc0aed04780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:03:47.534 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:03:47.534 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:03:47.535 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:03:47.535 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:03:47.535 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:03:47.536 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:03:47.536 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:03:47.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:03:47.536 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:03:47.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:03:47.537 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:03:47.537 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:03:47.538 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:03:47.538 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663027.328641_1439.1", 2023-10-18T21:03:47.538 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:03:47.539 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:03:47.539 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:03:47.539 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:03:47.328641Z", 2023-10-18T21:03:47.539 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:03:47.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:03:47.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:03:47.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:03:47.541 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:03:47.541 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:03:47.541 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:03:47.541 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:03:47.542 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:03:47.542 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:03:47.542 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:03:47.543 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:03:47.543 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:03:47.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:03:47.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:03:47.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:03:47.544 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:03:47.544 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:03:47.545 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:03:47.545 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:03:47.545 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:03:47.546 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:03:47.546 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:03:47.546 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:03:47.547 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:03:47.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:03:47.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:03:47.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:03:47.548 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:03:47.548 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:03:47.548 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:03:47.549 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:03:47.549 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:03:47.549 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:03:47.549 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:03:47.550 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:03:47.550 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:03:47.550 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:03:47.551 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:03:47.551 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:03:47.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:03:47.552 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:03:47.552 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:03:47.552 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:03:47.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:03:47.553 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:03:47Z" 2023-10-18T21:03:47.553 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:03:47.554 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663027.328641_1439.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:03:47.328641Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:03:47Z" } 2023-10-18T21:03:47.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663027.328641_1439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:03:47.328641Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:03:47.556 DEBUG:rgw_multi.tests:log_status={0: '1_1697663027.328641_1439.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:03:47.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663027.328641_1439.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:03:47.557 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:03:47.558 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:03:47.558 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:03:47.559 DEBUG:boto:Method: HEAD 2023-10-18T21:03:47.559 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:03:47.560 DEBUG:boto:Data: 2023-10-18T21:03:47.561 DEBUG:boto:Headers: {} 2023-10-18T21:03:47.561 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.562 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.562 DEBUG:boto:Params: {} 2023-10-18T21:03:47.563 DEBUG:boto:Token: None 2023-10-18T21:03:47.564 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/ 2023-10-18T21:03:47.564 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:03:47.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:03:47.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052425925bacd91f5-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.568 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:03:47.569 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:03:47.569 DEBUG:boto:Method: PUT 2023-10-18T21:03:47.570 DEBUG:boto:Path: /bofrnf-64/obj11 2023-10-18T21:03:47.570 DEBUG:boto:Data: 2023-10-18T21:03:47.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:03:47.571 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.571 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.572 DEBUG:boto:Params: {} 2023-10-18T21:03:47.572 DEBUG:boto:Token: None 2023-10-18T21:03:47.573 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/obj11 2023-10-18T21:03:47.573 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kJMMs1aTQBEnQLDuENU2svQJT5o= 2023-10-18T21:03:47.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kJMMs1aTQBEnQLDuENU2svQJT5o='} 2023-10-18T21:03:47.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d5ea8c2adba5a35-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.580 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:03:47.581 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:03:47.581 DEBUG:boto:Method: HEAD 2023-10-18T21:03:47.581 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:03:47.582 DEBUG:boto:Data: 2023-10-18T21:03:47.582 DEBUG:boto:Headers: {} 2023-10-18T21:03:47.583 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.583 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.584 DEBUG:boto:Params: {} 2023-10-18T21:03:47.584 DEBUG:boto:Token: None 2023-10-18T21:03:47.585 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/ 2023-10-18T21:03:47.585 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:03:47.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:03:47.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000918fd72caeb25936-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.588 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:03:47.589 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:03:47.589 DEBUG:boto:Method: PUT 2023-10-18T21:03:47.590 DEBUG:boto:Path: /bofrnf-64/obj22 2023-10-18T21:03:47.590 DEBUG:boto:Data: 2023-10-18T21:03:47.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:03:47.591 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.592 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.592 DEBUG:boto:Params: {} 2023-10-18T21:03:47.592 DEBUG:boto:Token: None 2023-10-18T21:03:47.593 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/obj22 2023-10-18T21:03:47.593 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MzfOEmfuUqZZzd3GSY/GhHAumQk= 2023-10-18T21:03:47.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MzfOEmfuUqZZzd3GSY/GhHAumQk='} 2023-10-18T21:03:47.597 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1d4c187c68c5381-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.598 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:03:47.598 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:03:47.599 DEBUG:boto:Method: HEAD 2023-10-18T21:03:47.599 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:03:47.600 DEBUG:boto:Data: 2023-10-18T21:03:47.600 DEBUG:boto:Headers: {} 2023-10-18T21:03:47.600 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.601 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.601 DEBUG:boto:Params: {} 2023-10-18T21:03:47.602 DEBUG:boto:Token: None 2023-10-18T21:03:47.602 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/ 2023-10-18T21:03:47.603 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:03:47.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:03:47.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000005b29a1c0a7ef32-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.605 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:03:47.606 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:03:47.606 DEBUG:boto:Method: PUT 2023-10-18T21:03:47.607 DEBUG:boto:Path: /bofrnf-64/obj33 2023-10-18T21:03:47.607 DEBUG:boto:Data: 2023-10-18T21:03:47.607 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:03:47.608 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.608 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.608 DEBUG:boto:Params: {} 2023-10-18T21:03:47.609 DEBUG:boto:Token: None 2023-10-18T21:03:47.609 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/obj33 2023-10-18T21:03:47.610 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ojAOBVnkTxoLgi0d+Oel1PODrR4= 2023-10-18T21:03:47.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ojAOBVnkTxoLgi0d+Oel1PODrR4='} 2023-10-18T21:03:47.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcf6867ec619b352-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.616 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:03:47.616 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:03:47.617 DEBUG:boto:Method: HEAD 2023-10-18T21:03:47.617 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:03:47.617 DEBUG:boto:Data: 2023-10-18T21:03:47.618 DEBUG:boto:Headers: {} 2023-10-18T21:03:47.618 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.619 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.619 DEBUG:boto:Params: {} 2023-10-18T21:03:47.619 DEBUG:boto:Token: None 2023-10-18T21:03:47.620 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/ 2023-10-18T21:03:47.620 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:03:47.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:03:47.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060bcabd7027a5c9e-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.623 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:03:47.624 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:03:47.624 DEBUG:boto:Method: PUT 2023-10-18T21:03:47.625 DEBUG:boto:Path: /bofrnf-64/obj44 2023-10-18T21:03:47.625 DEBUG:boto:Data: 2023-10-18T21:03:47.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:03:47.626 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.626 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.627 DEBUG:boto:Params: {} 2023-10-18T21:03:47.627 DEBUG:boto:Token: None 2023-10-18T21:03:47.627 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/obj44 2023-10-18T21:03:47.628 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7tuPfoz37Jich9jiSdpIQS/f4/M= 2023-10-18T21:03:47.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7tuPfoz37Jich9jiSdpIQS/f4/M='} 2023-10-18T21:03:47.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0005c618f635a67-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.671 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:03:47.671 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:03:47.672 DEBUG:boto:Method: HEAD 2023-10-18T21:03:47.672 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:03:47.673 DEBUG:boto:Data: 2023-10-18T21:03:47.673 DEBUG:boto:Headers: {} 2023-10-18T21:03:47.673 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.674 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.674 DEBUG:boto:Params: {} 2023-10-18T21:03:47.675 DEBUG:boto:Token: None 2023-10-18T21:03:47.675 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/ 2023-10-18T21:03:47.675 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:03:47.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:03:47.678 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7f0f333f81cb2e5-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.679 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:03:47.679 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:03:47.680 DEBUG:boto:Method: PUT 2023-10-18T21:03:47.680 DEBUG:boto:Path: /bofrnf-64/obj55 2023-10-18T21:03:47.680 DEBUG:boto:Data: 2023-10-18T21:03:47.681 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:03:47.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:03:47.681 DEBUG:boto:Port: 8000 2023-10-18T21:03:47.682 DEBUG:boto:Params: {} 2023-10-18T21:03:47.682 DEBUG:boto:Token: None 2023-10-18T21:03:47.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:03:47 GMT /bofrnf-64/obj55 2023-10-18T21:03:47.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mVWsxe1jfrRVn25Qb2iO8pJtgek= 2023-10-18T21:03:47.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mVWsxe1jfrRVn25Qb2iO8pJtgek='} 2023-10-18T21:03:47.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091f23c3124d4da68-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:03:47.688 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-64 2023-10-18T21:03:47.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:03:47.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:03:47.689 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:03:47.745 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:03:47.746 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:03:47.779 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:47.778+0000 7ff04123e780 1 waiting to reach incremental sync.. 2023-10-18T21:03:52.780 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:52.779+0000 7ff04123e780 1 waiting to reach incremental sync.. 2023-10-18T21:03:57.781 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:03:57.779+0000 7ff04123e780 1 waiting to reach incremental sync.. 2023-10-18T21:04:02.784 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:02.781+0000 7ff04123e780 1 bucket sync caught up with source: 2023-10-18T21:04:02.784 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.119.6, 00000000002.97.6, 00000000002.121.6] 2023-10-18T21:04:02.785 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.119.6, 00000000002.97.6, 00000000002.121.6] 2023-10-18T21:04:02.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:02.781+0000 7ff04123e780 0 bucket checkpoint complete 2023-10-18T21:04:02.801 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=bofrnf-64 2023-10-18T21:04:02.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:04:02.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:04:02.802 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:04:02.863 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:04:02.863 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:04:02.900 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:02.897+0000 7f466f907780 1 waiting to reach incremental sync.. 2023-10-18T21:04:07.901 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:07.899+0000 7f466f907780 1 bucket sync caught up with source: 2023-10-18T21:04:07.902 INFO:teuthology.orchestra.run.smithi082.stderr: local status: [, , , , , , , , 00000000001.332.6, 00000000002.855.6, 00000000002.725.6] 2023-10-18T21:04:07.902 INFO:teuthology.orchestra.run.smithi082.stderr: remote markers: [, , , , , , , , 00000000001.332.6, 00000000002.855.6, 00000000002.725.6] 2023-10-18T21:04:07.902 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:07.899+0000 7f466f907780 0 bucket checkpoint complete 2023-10-18T21:04:07.916 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bofrnf-64 zones={a2, a1} 2023-10-18T21:04:07.916 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:07.917 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:07.917 DEBUG:boto:Method: HEAD 2023-10-18T21:04:07.918 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:07.918 DEBUG:boto:Data: 2023-10-18T21:04:07.919 DEBUG:boto:Headers: {} 2023-10-18T21:04:07.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:07.919 DEBUG:boto:Port: 8001 2023-10-18T21:04:07.920 DEBUG:boto:Params: {} 2023-10-18T21:04:07.920 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:04:07.921 DEBUG:boto:Token: None 2023-10-18T21:04:07.921 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:07 GMT /bofrnf-64/ 2023-10-18T21:04:07.922 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hG7lL7f4EFehigKdiQG8WSihlzE= 2023-10-18T21:04:07.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hG7lL7f4EFehigKdiQG8WSihlzE=', 'Content-Length': '0'} 2023-10-18T21:04:07.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007112ade23e62e733-0065304847-1f24-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:07.927 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:07.927 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:07.928 DEBUG:boto:Method: HEAD 2023-10-18T21:04:07.928 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:07.929 DEBUG:boto:Data: 2023-10-18T21:04:07.929 DEBUG:boto:Headers: {} 2023-10-18T21:04:07.929 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:07.930 DEBUG:boto:Port: 8000 2023-10-18T21:04:07.930 DEBUG:boto:Params: {} 2023-10-18T21:04:07.930 DEBUG:boto:Token: None 2023-10-18T21:04:07.931 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:07 GMT /bofrnf-64/ 2023-10-18T21:04:07.931 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hG7lL7f4EFehigKdiQG8WSihlzE= 2023-10-18T21:04:07.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hG7lL7f4EFehigKdiQG8WSihlzE=', 'Content-Length': '0'} 2023-10-18T21:04:07.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007303c5a32d3b4855-0065304847-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:07.935 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-18T21:04:07.935 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:07.935 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:07.936 DEBUG:boto:path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.936 DEBUG:boto:auth_path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.937 DEBUG:boto:Method: GET 2023-10-18T21:04:07.937 DEBUG:boto:Path: /bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.937 DEBUG:boto:Data: 2023-10-18T21:04:07.938 DEBUG:boto:Headers: {} 2023-10-18T21:04:07.938 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:07.939 DEBUG:boto:Port: 8001 2023-10-18T21:04:07.939 DEBUG:boto:Params: {} 2023-10-18T21:04:07.939 DEBUG:boto:Token: None 2023-10-18T21:04:07.940 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:07 GMT /bofrnf-64/?versions 2023-10-18T21:04:07.940 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY= 2023-10-18T21:04:07.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY=', 'Content-Length': '0'} 2023-10-18T21:04:07.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e1d385468474f87-0065304847-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:07.943 DEBUG:boto:b'bofrnf-64999falseobj11nulltrue2023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:07.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T21:04:07.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T21:04:07.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T21:04:07.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T21:04:07.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T21:04:07.946 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-18T21:04:07.947 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:07.947 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:07.948 DEBUG:boto:path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.948 DEBUG:boto:auth_path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.948 DEBUG:boto:Method: GET 2023-10-18T21:04:07.949 DEBUG:boto:Path: /bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.949 DEBUG:boto:Data: 2023-10-18T21:04:07.950 DEBUG:boto:Headers: {} 2023-10-18T21:04:07.950 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:07.950 DEBUG:boto:Port: 8000 2023-10-18T21:04:07.951 DEBUG:boto:Params: {} 2023-10-18T21:04:07.951 DEBUG:boto:Token: None 2023-10-18T21:04:07.952 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:07 GMT /bofrnf-64/?versions 2023-10-18T21:04:07.952 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY= 2023-10-18T21:04:07.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY=', 'Content-Length': '0'} 2023-10-18T21:04:07.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002424d6c1cc8bc0cd-0065304847-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:07.958 DEBUG:boto:b'bofrnf-64999falseobj11nulltrue2023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:07.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-18T21:04:07.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-18T21:04:07.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-18T21:04:07.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-18T21:04:07.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-18T21:04:07.960 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:07.961 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:07.961 DEBUG:boto:path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.962 DEBUG:boto:auth_path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.962 DEBUG:boto:Method: GET 2023-10-18T21:04:07.962 DEBUG:boto:Path: /bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.963 DEBUG:boto:Data: 2023-10-18T21:04:07.963 DEBUG:boto:Headers: {} 2023-10-18T21:04:07.964 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:07.964 DEBUG:boto:Port: 8001 2023-10-18T21:04:07.964 DEBUG:boto:Params: {} 2023-10-18T21:04:07.965 DEBUG:boto:Token: None 2023-10-18T21:04:07.965 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:07 GMT /bofrnf-64/?versions 2023-10-18T21:04:07.966 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY= 2023-10-18T21:04:07.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY=', 'Content-Length': '0'} 2023-10-18T21:04:07.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000201f6f2f5efbd4d4-0065304847-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:07.969 DEBUG:boto:b'bofrnf-64999falseobj11nulltrue2023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:07.970 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:07.970 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:07.970 DEBUG:boto:path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.971 DEBUG:boto:auth_path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.971 DEBUG:boto:Method: GET 2023-10-18T21:04:07.972 DEBUG:boto:Path: /bofrnf-64/?versions&max-keys=999 2023-10-18T21:04:07.972 DEBUG:boto:Data: 2023-10-18T21:04:07.972 DEBUG:boto:Headers: {} 2023-10-18T21:04:07.973 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:07.973 DEBUG:boto:Port: 8000 2023-10-18T21:04:07.974 DEBUG:boto:Params: {} 2023-10-18T21:04:07.974 DEBUG:boto:Token: None 2023-10-18T21:04:07.975 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:07 GMT /bofrnf-64/?versions 2023-10-18T21:04:07.975 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY= 2023-10-18T21:04:07.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY=', 'Content-Length': '0'} 2023-10-18T21:04:07.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c168948e6c42f1eb-0065304847-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:07.978 DEBUG:boto:b'bofrnf-64999falseobj11nulltrue2023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:07.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T21:04:07.979 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:07.980 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:07.980 DEBUG:boto:path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:07.981 DEBUG:boto:auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:07.981 DEBUG:boto:Method: GET 2023-10-18T21:04:07.981 DEBUG:boto:Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:07.982 DEBUG:boto:Data: 2023-10-18T21:04:07.982 DEBUG:boto:Headers: {} 2023-10-18T21:04:07.983 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:07.983 DEBUG:boto:Port: 8001 2023-10-18T21:04:07.983 DEBUG:boto:Params: {} 2023-10-18T21:04:07.984 DEBUG:boto:Token: None 2023-10-18T21:04:07.984 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:07 GMT /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:07.985 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/AuSZUhqAF7a4UhpftlnXM/ocvU= 2023-10-18T21:04:07.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/AuSZUhqAF7a4UhpftlnXM/ocvU=', 'Content-Length': '0'} 2023-10-18T21:04:07.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9cd14cf86f0edd7-0065304847-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:07.988 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:07.988 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:07.989 DEBUG:boto:path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:07.989 DEBUG:boto:auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:07.990 DEBUG:boto:Method: GET 2023-10-18T21:04:07.990 DEBUG:boto:Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:07.991 DEBUG:boto:Data: 2023-10-18T21:04:07.991 DEBUG:boto:Headers: {} 2023-10-18T21:04:07.991 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:07.992 DEBUG:boto:Port: 8000 2023-10-18T21:04:07.992 DEBUG:boto:Params: {} 2023-10-18T21:04:07.993 DEBUG:boto:Token: None 2023-10-18T21:04:07.993 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:07 GMT /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:07.994 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/AuSZUhqAF7a4UhpftlnXM/ocvU= 2023-10-18T21:04:07.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/AuSZUhqAF7a4UhpftlnXM/ocvU=', 'Content-Length': '0'} 2023-10-18T21:04:07.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f2c4f944f89b8b4-0065304847-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.038 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:08.039 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:08.039 DEBUG:boto:path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.040 DEBUG:boto:auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.040 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.040 DEBUG:boto:Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.041 DEBUG:boto:Data: 2023-10-18T21:04:08.041 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.041 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.042 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.042 DEBUG:boto:Params: {} 2023-10-18T21:04:08.043 DEBUG:boto:Token: None 2023-10-18T21:04:08.043 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.044 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i7/YOoHOM7OQ1ZaF3Rc7NukMy2I= 2023-10-18T21:04:08.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i7/YOoHOM7OQ1ZaF3Rc7NukMy2I=', 'Content-Length': '0'} 2023-10-18T21:04:08.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b49759cda6b6e454-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.047 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:08.047 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:08.048 DEBUG:boto:path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.048 DEBUG:boto:auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.048 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.049 DEBUG:boto:Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.049 DEBUG:boto:Data: 2023-10-18T21:04:08.050 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.050 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.050 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.051 DEBUG:boto:Params: {} 2023-10-18T21:04:08.051 DEBUG:boto:Token: None 2023-10-18T21:04:08.052 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.052 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:i7/YOoHOM7OQ1ZaF3Rc7NukMy2I= 2023-10-18T21:04:08.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i7/YOoHOM7OQ1ZaF3Rc7NukMy2I=', 'Content-Length': '0'} 2023-10-18T21:04:08.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004299d20e76f56e9a-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T21:04:08.055 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:08.055 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:08.056 DEBUG:boto:path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.056 DEBUG:boto:auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.057 DEBUG:boto:Method: GET 2023-10-18T21:04:08.057 DEBUG:boto:Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.057 DEBUG:boto:Data: 2023-10-18T21:04:08.058 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.058 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.059 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.059 DEBUG:boto:Params: {} 2023-10-18T21:04:08.060 DEBUG:boto:Token: None 2023-10-18T21:04:08.060 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.061 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cJV3yEnOIq45DVMCWS2ZSOpScz0= 2023-10-18T21:04:08.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cJV3yEnOIq45DVMCWS2ZSOpScz0=', 'Content-Length': '0'} 2023-10-18T21:04:08.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f5e99f9fa24d8af-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.064 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:08.064 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:08.065 DEBUG:boto:path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.065 DEBUG:boto:auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.065 DEBUG:boto:Method: GET 2023-10-18T21:04:08.066 DEBUG:boto:Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.066 DEBUG:boto:Data: 2023-10-18T21:04:08.067 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.067 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.068 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.068 DEBUG:boto:Params: {} 2023-10-18T21:04:08.068 DEBUG:boto:Token: None 2023-10-18T21:04:08.069 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj11?versionId=null 2023-10-18T21:04:08.069 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cJV3yEnOIq45DVMCWS2ZSOpScz0= 2023-10-18T21:04:08.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cJV3yEnOIq45DVMCWS2ZSOpScz0=', 'Content-Length': '0'} 2023-10-18T21:04:08.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6ac73bb0d15f436-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.072 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.073 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.073 DEBUG:boto:path=/bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.074 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.074 DEBUG:boto:Method: GET 2023-10-18T21:04:08.075 DEBUG:boto:Path: /bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.075 DEBUG:boto:Data: 2023-10-18T21:04:08.075 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.076 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.076 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.077 DEBUG:boto:Params: {} 2023-10-18T21:04:08.077 DEBUG:boto:Token: None 2023-10-18T21:04:08.078 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.078 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fbafe1d1c1751b2-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.122 DEBUG:boto:b'bofrnf-64obj111000falseobj112023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.123 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.123 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.124 DEBUG:boto:path=/bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.124 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.125 DEBUG:boto:Method: GET 2023-10-18T21:04:08.125 DEBUG:boto:Path: /bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.126 DEBUG:boto:Data: 2023-10-18T21:04:08.126 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.126 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.127 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.127 DEBUG:boto:Params: {} 2023-10-18T21:04:08.128 DEBUG:boto:Token: None 2023-10-18T21:04:08.128 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.129 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b1e277ac4edc535-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.132 DEBUG:boto:b'bofrnf-64obj111000falseobj112023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.133 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:08.133 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:08.134 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.134 DEBUG:boto:Path: /bofrnf-64/obj11 2023-10-18T21:04:08.134 DEBUG:boto:Data: 2023-10-18T21:04:08.135 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.135 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.135 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.136 DEBUG:boto:Params: {} 2023-10-18T21:04:08.136 DEBUG:boto:Token: None 2023-10-18T21:04:08.137 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj11 2023-10-18T21:04:08.137 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QQiSYTTjqoxdtHrmRyjvdyqn7II= 2023-10-18T21:04:08.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QQiSYTTjqoxdtHrmRyjvdyqn7II=', 'Content-Length': '0'} 2023-10-18T21:04:08.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1d59c1256d68b4-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.140 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:08.140 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:08.141 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.141 DEBUG:boto:Path: /bofrnf-64/obj11 2023-10-18T21:04:08.141 DEBUG:boto:Data: 2023-10-18T21:04:08.142 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.142 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.143 DEBUG:boto:Params: {} 2023-10-18T21:04:08.143 DEBUG:boto:Token: None 2023-10-18T21:04:08.144 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj11 2023-10-18T21:04:08.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QQiSYTTjqoxdtHrmRyjvdyqn7II= 2023-10-18T21:04:08.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QQiSYTTjqoxdtHrmRyjvdyqn7II=', 'Content-Length': '0'} 2023-10-18T21:04:08.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd5b0d8c54439f88-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-18T21:04:08.148 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:08.148 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:08.148 DEBUG:boto:Method: GET 2023-10-18T21:04:08.149 DEBUG:boto:Path: /bofrnf-64/obj11 2023-10-18T21:04:08.149 DEBUG:boto:Data: 2023-10-18T21:04:08.150 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.150 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.151 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.151 DEBUG:boto:Params: {} 2023-10-18T21:04:08.151 DEBUG:boto:Token: None 2023-10-18T21:04:08.152 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj11 2023-10-18T21:04:08.152 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HL2xnvxnwazUweeaApxAhDfHl6I= 2023-10-18T21:04:08.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HL2xnvxnwazUweeaApxAhDfHl6I=', 'Content-Length': '0'} 2023-10-18T21:04:08.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017b7e6dcf64c2cdc-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.155 DEBUG:boto:path=/bofrnf-64/obj11 2023-10-18T21:04:08.156 DEBUG:boto:auth_path=/bofrnf-64/obj11 2023-10-18T21:04:08.156 DEBUG:boto:Method: GET 2023-10-18T21:04:08.157 DEBUG:boto:Path: /bofrnf-64/obj11 2023-10-18T21:04:08.157 DEBUG:boto:Data: 2023-10-18T21:04:08.157 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.158 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.158 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.161 DEBUG:boto:Params: {} 2023-10-18T21:04:08.162 DEBUG:boto:Token: None 2023-10-18T21:04:08.162 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj11 2023-10-18T21:04:08.163 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HL2xnvxnwazUweeaApxAhDfHl6I= 2023-10-18T21:04:08.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HL2xnvxnwazUweeaApxAhDfHl6I=', 'Content-Length': '0'} 2023-10-18T21:04:08.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025208d87e3d5e90b-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.206 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.207 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.207 DEBUG:boto:path=/bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.208 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.208 DEBUG:boto:Method: GET 2023-10-18T21:04:08.209 DEBUG:boto:Path: /bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.209 DEBUG:boto:Data: 2023-10-18T21:04:08.210 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.210 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.210 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.211 DEBUG:boto:Params: {} 2023-10-18T21:04:08.211 DEBUG:boto:Token: None 2023-10-18T21:04:08.212 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.212 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3d91fb7f81aeee-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.216 DEBUG:boto:b'bofrnf-64obj111000falseobj112023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.217 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.217 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.218 DEBUG:boto:path=/bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.218 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.219 DEBUG:boto:Method: GET 2023-10-18T21:04:08.219 DEBUG:boto:Path: /bofrnf-64/?prefix=obj11 2023-10-18T21:04:08.219 DEBUG:boto:Data: 2023-10-18T21:04:08.220 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.220 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.221 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.221 DEBUG:boto:Params: {} 2023-10-18T21:04:08.222 DEBUG:boto:Token: None 2023-10-18T21:04:08.222 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.223 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b95fa17385459138-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.226 DEBUG:boto:b'bofrnf-64obj111000falseobj112023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T21:04:08.227 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.227 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.228 DEBUG:boto:path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.228 DEBUG:boto:auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.229 DEBUG:boto:Method: GET 2023-10-18T21:04:08.229 DEBUG:boto:Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.230 DEBUG:boto:Data: 2023-10-18T21:04:08.230 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.230 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.231 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.231 DEBUG:boto:Params: {} 2023-10-18T21:04:08.232 DEBUG:boto:Token: None 2023-10-18T21:04:08.232 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.233 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U= 2023-10-18T21:04:08.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U=', 'Content-Length': '0'} 2023-10-18T21:04:08.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007637789e8ec3bad1-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.236 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.236 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.237 DEBUG:boto:path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.237 DEBUG:boto:auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.238 DEBUG:boto:Method: GET 2023-10-18T21:04:08.238 DEBUG:boto:Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.239 DEBUG:boto:Data: 2023-10-18T21:04:08.239 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.239 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.240 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.240 DEBUG:boto:Params: {} 2023-10-18T21:04:08.241 DEBUG:boto:Token: None 2023-10-18T21:04:08.241 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.242 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U= 2023-10-18T21:04:08.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U=', 'Content-Length': '0'} 2023-10-18T21:04:08.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003323f3592deb9a7a-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.245 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.245 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.246 DEBUG:boto:path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.246 DEBUG:boto:auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.246 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.247 DEBUG:boto:Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.247 DEBUG:boto:Data: 2023-10-18T21:04:08.248 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.249 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.249 DEBUG:boto:Params: {} 2023-10-18T21:04:08.249 DEBUG:boto:Token: None 2023-10-18T21:04:08.250 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.250 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EDmF9U34rcI0fuRJJYp1YqI7Wuw= 2023-10-18T21:04:08.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EDmF9U34rcI0fuRJJYp1YqI7Wuw=', 'Content-Length': '0'} 2023-10-18T21:04:08.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000600b5cfd13e002e4-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.253 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.254 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.254 DEBUG:boto:path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.254 DEBUG:boto:auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.255 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.255 DEBUG:boto:Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.256 DEBUG:boto:Data: 2023-10-18T21:04:08.256 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.256 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.257 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.257 DEBUG:boto:Params: {} 2023-10-18T21:04:08.258 DEBUG:boto:Token: None 2023-10-18T21:04:08.258 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.259 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EDmF9U34rcI0fuRJJYp1YqI7Wuw= 2023-10-18T21:04:08.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EDmF9U34rcI0fuRJJYp1YqI7Wuw=', 'Content-Length': '0'} 2023-10-18T21:04:08.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f0746f4f16bac5c-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T21:04:08.262 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.262 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.263 DEBUG:boto:path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.263 DEBUG:boto:auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.264 DEBUG:boto:Method: GET 2023-10-18T21:04:08.264 DEBUG:boto:Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.264 DEBUG:boto:Data: 2023-10-18T21:04:08.265 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.265 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.266 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.266 DEBUG:boto:Params: {} 2023-10-18T21:04:08.267 DEBUG:boto:Token: None 2023-10-18T21:04:08.267 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.268 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U= 2023-10-18T21:04:08.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U=', 'Content-Length': '0'} 2023-10-18T21:04:08.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd1ee416b9a926f3-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.270 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.271 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.271 DEBUG:boto:path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.272 DEBUG:boto:auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.272 DEBUG:boto:Method: GET 2023-10-18T21:04:08.273 DEBUG:boto:Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.273 DEBUG:boto:Data: 2023-10-18T21:04:08.274 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.274 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.274 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.275 DEBUG:boto:Params: {} 2023-10-18T21:04:08.275 DEBUG:boto:Token: None 2023-10-18T21:04:08.276 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22?versionId=null 2023-10-18T21:04:08.276 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U= 2023-10-18T21:04:08.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U=', 'Content-Length': '0'} 2023-10-18T21:04:08.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0e4ea0cc5f31165-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.279 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.279 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.280 DEBUG:boto:path=/bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.280 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.281 DEBUG:boto:Method: GET 2023-10-18T21:04:08.281 DEBUG:boto:Path: /bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.281 DEBUG:boto:Data: 2023-10-18T21:04:08.282 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.282 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.283 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.283 DEBUG:boto:Params: {} 2023-10-18T21:04:08.284 DEBUG:boto:Token: None 2023-10-18T21:04:08.284 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.284 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cc0326b295bf4b7-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.292 DEBUG:boto:b'bofrnf-64obj221000falseobj222023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.293 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.293 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.294 DEBUG:boto:path=/bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.294 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.295 DEBUG:boto:Method: GET 2023-10-18T21:04:08.295 DEBUG:boto:Path: /bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.295 DEBUG:boto:Data: 2023-10-18T21:04:08.296 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.296 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.297 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.297 DEBUG:boto:Params: {} 2023-10-18T21:04:08.297 DEBUG:boto:Token: None 2023-10-18T21:04:08.298 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.298 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009366328567674dd3-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.302 DEBUG:boto:b'bofrnf-64obj221000falseobj222023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.302 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.303 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.303 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.304 DEBUG:boto:Path: /bofrnf-64/obj22 2023-10-18T21:04:08.304 DEBUG:boto:Data: 2023-10-18T21:04:08.304 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.305 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.305 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.305 DEBUG:boto:Params: {} 2023-10-18T21:04:08.306 DEBUG:boto:Token: None 2023-10-18T21:04:08.306 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22 2023-10-18T21:04:08.307 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ytP5Bp79TzjZ5xFFqvp8rIRqe3A= 2023-10-18T21:04:08.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ytP5Bp79TzjZ5xFFqvp8rIRqe3A=', 'Content-Length': '0'} 2023-10-18T21:04:08.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7eda0c0a3f6b92d-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.310 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.310 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.311 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.311 DEBUG:boto:Path: /bofrnf-64/obj22 2023-10-18T21:04:08.311 DEBUG:boto:Data: 2023-10-18T21:04:08.312 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.312 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.313 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.313 DEBUG:boto:Params: {} 2023-10-18T21:04:08.313 DEBUG:boto:Token: None 2023-10-18T21:04:08.314 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22 2023-10-18T21:04:08.314 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ytP5Bp79TzjZ5xFFqvp8rIRqe3A= 2023-10-18T21:04:08.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ytP5Bp79TzjZ5xFFqvp8rIRqe3A=', 'Content-Length': '0'} 2023-10-18T21:04:08.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a52a2e3c57ca05-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-18T21:04:08.318 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.318 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.319 DEBUG:boto:Method: GET 2023-10-18T21:04:08.319 DEBUG:boto:Path: /bofrnf-64/obj22 2023-10-18T21:04:08.319 DEBUG:boto:Data: 2023-10-18T21:04:08.320 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.320 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.321 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.321 DEBUG:boto:Params: {} 2023-10-18T21:04:08.322 DEBUG:boto:Token: None 2023-10-18T21:04:08.322 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22 2023-10-18T21:04:08.323 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SbmjAegI+JRqs8QjN8BqODkQuJc= 2023-10-18T21:04:08.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SbmjAegI+JRqs8QjN8BqODkQuJc=', 'Content-Length': '0'} 2023-10-18T21:04:08.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ca79fcfcf51ac7-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.370 DEBUG:boto:path=/bofrnf-64/obj22 2023-10-18T21:04:08.371 DEBUG:boto:auth_path=/bofrnf-64/obj22 2023-10-18T21:04:08.371 DEBUG:boto:Method: GET 2023-10-18T21:04:08.372 DEBUG:boto:Path: /bofrnf-64/obj22 2023-10-18T21:04:08.372 DEBUG:boto:Data: 2023-10-18T21:04:08.373 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.373 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.374 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.374 DEBUG:boto:Params: {} 2023-10-18T21:04:08.374 DEBUG:boto:Token: None 2023-10-18T21:04:08.375 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj22 2023-10-18T21:04:08.375 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:SbmjAegI+JRqs8QjN8BqODkQuJc= 2023-10-18T21:04:08.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SbmjAegI+JRqs8QjN8BqODkQuJc=', 'Content-Length': '0'} 2023-10-18T21:04:08.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3de6eabf0f15b1e-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.378 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.379 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.379 DEBUG:boto:path=/bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.380 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.380 DEBUG:boto:Method: GET 2023-10-18T21:04:08.381 DEBUG:boto:Path: /bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.381 DEBUG:boto:Data: 2023-10-18T21:04:08.381 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.382 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.382 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.383 DEBUG:boto:Params: {} 2023-10-18T21:04:08.383 DEBUG:boto:Token: None 2023-10-18T21:04:08.384 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.384 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd65c08550f0c68d-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.387 DEBUG:boto:b'bofrnf-64obj221000falseobj222023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.388 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.388 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.389 DEBUG:boto:path=/bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.389 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.389 DEBUG:boto:Method: GET 2023-10-18T21:04:08.390 DEBUG:boto:Path: /bofrnf-64/?prefix=obj22 2023-10-18T21:04:08.390 DEBUG:boto:Data: 2023-10-18T21:04:08.391 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.391 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.391 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.392 DEBUG:boto:Params: {} 2023-10-18T21:04:08.392 DEBUG:boto:Token: None 2023-10-18T21:04:08.393 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.393 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094932f06439e4f52-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.397 DEBUG:boto:b'bofrnf-64obj221000falseobj222023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T21:04:08.398 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.398 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.398 DEBUG:boto:path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.399 DEBUG:boto:auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.399 DEBUG:boto:Method: GET 2023-10-18T21:04:08.400 DEBUG:boto:Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.400 DEBUG:boto:Data: 2023-10-18T21:04:08.400 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.401 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.401 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.402 DEBUG:boto:Params: {} 2023-10-18T21:04:08.402 DEBUG:boto:Token: None 2023-10-18T21:04:08.403 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.403 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM= 2023-10-18T21:04:08.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM=', 'Content-Length': '0'} 2023-10-18T21:04:08.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a385edb1cd5b3eba-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.406 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.406 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.407 DEBUG:boto:path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.407 DEBUG:boto:auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.407 DEBUG:boto:Method: GET 2023-10-18T21:04:08.408 DEBUG:boto:Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.408 DEBUG:boto:Data: 2023-10-18T21:04:08.408 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.409 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.410 DEBUG:boto:Params: {} 2023-10-18T21:04:08.410 DEBUG:boto:Token: None 2023-10-18T21:04:08.411 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM= 2023-10-18T21:04:08.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM=', 'Content-Length': '0'} 2023-10-18T21:04:08.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d67d46279b1fea6a-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.414 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.414 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.415 DEBUG:boto:path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.415 DEBUG:boto:auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.416 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.416 DEBUG:boto:Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.416 DEBUG:boto:Data: 2023-10-18T21:04:08.417 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.417 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.417 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.418 DEBUG:boto:Params: {} 2023-10-18T21:04:08.418 DEBUG:boto:Token: None 2023-10-18T21:04:08.419 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.419 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oBSj3j9hej3U7r8Ycbn+5uyUCAA= 2023-10-18T21:04:08.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oBSj3j9hej3U7r8Ycbn+5uyUCAA=', 'Content-Length': '0'} 2023-10-18T21:04:08.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4f4fa7e0b0198d0-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.421 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.422 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.422 DEBUG:boto:path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.423 DEBUG:boto:auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.423 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.423 DEBUG:boto:Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.424 DEBUG:boto:Data: 2023-10-18T21:04:08.424 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.425 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.425 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.425 DEBUG:boto:Params: {} 2023-10-18T21:04:08.426 DEBUG:boto:Token: None 2023-10-18T21:04:08.426 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.426 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oBSj3j9hej3U7r8Ycbn+5uyUCAA= 2023-10-18T21:04:08.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oBSj3j9hej3U7r8Ycbn+5uyUCAA=', 'Content-Length': '0'} 2023-10-18T21:04:08.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d7fd2e64ff5476-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T21:04:08.430 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.430 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.431 DEBUG:boto:path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.431 DEBUG:boto:auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.431 DEBUG:boto:Method: GET 2023-10-18T21:04:08.432 DEBUG:boto:Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.432 DEBUG:boto:Data: 2023-10-18T21:04:08.433 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.433 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.433 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.434 DEBUG:boto:Params: {} 2023-10-18T21:04:08.434 DEBUG:boto:Token: None 2023-10-18T21:04:08.435 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.435 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM= 2023-10-18T21:04:08.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM=', 'Content-Length': '0'} 2023-10-18T21:04:08.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006757be7645d5225a-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.438 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.438 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.439 DEBUG:boto:path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.439 DEBUG:boto:auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.440 DEBUG:boto:Method: GET 2023-10-18T21:04:08.440 DEBUG:boto:Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.440 DEBUG:boto:Data: 2023-10-18T21:04:08.441 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.441 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.441 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.442 DEBUG:boto:Params: {} 2023-10-18T21:04:08.442 DEBUG:boto:Token: None 2023-10-18T21:04:08.443 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33?versionId=null 2023-10-18T21:04:08.443 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM= 2023-10-18T21:04:08.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM=', 'Content-Length': '0'} 2023-10-18T21:04:08.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d6876897510989-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.446 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.446 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.447 DEBUG:boto:path=/bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.447 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.447 DEBUG:boto:Method: GET 2023-10-18T21:04:08.448 DEBUG:boto:Path: /bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.448 DEBUG:boto:Data: 2023-10-18T21:04:08.449 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.449 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.449 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.450 DEBUG:boto:Params: {} 2023-10-18T21:04:08.450 DEBUG:boto:Token: None 2023-10-18T21:04:08.451 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.451 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14b0a78060f2152-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.455 DEBUG:boto:b'bofrnf-64obj331000falseobj332023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.455 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.456 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.456 DEBUG:boto:path=/bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.456 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.457 DEBUG:boto:Method: GET 2023-10-18T21:04:08.457 DEBUG:boto:Path: /bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.457 DEBUG:boto:Data: 2023-10-18T21:04:08.458 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.458 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.459 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.459 DEBUG:boto:Params: {} 2023-10-18T21:04:08.460 DEBUG:boto:Token: None 2023-10-18T21:04:08.460 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e507b31224dfb643-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.464 DEBUG:boto:b'bofrnf-64obj331000falseobj332023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.464 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.465 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.465 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.465 DEBUG:boto:Path: /bofrnf-64/obj33 2023-10-18T21:04:08.466 DEBUG:boto:Data: 2023-10-18T21:04:08.466 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.466 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.467 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.467 DEBUG:boto:Params: {} 2023-10-18T21:04:08.467 DEBUG:boto:Token: None 2023-10-18T21:04:08.468 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33 2023-10-18T21:04:08.468 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X0idiJytnT3d0liI/gZqF44JGYM= 2023-10-18T21:04:08.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X0idiJytnT3d0liI/gZqF44JGYM=', 'Content-Length': '0'} 2023-10-18T21:04:08.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a290aa9b05386a1-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.471 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.471 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.472 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.472 DEBUG:boto:Path: /bofrnf-64/obj33 2023-10-18T21:04:08.473 DEBUG:boto:Data: 2023-10-18T21:04:08.473 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.473 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.474 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.474 DEBUG:boto:Params: {} 2023-10-18T21:04:08.474 DEBUG:boto:Token: None 2023-10-18T21:04:08.475 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33 2023-10-18T21:04:08.475 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X0idiJytnT3d0liI/gZqF44JGYM= 2023-10-18T21:04:08.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X0idiJytnT3d0liI/gZqF44JGYM=', 'Content-Length': '0'} 2023-10-18T21:04:08.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04829eb2252c50f-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-18T21:04:08.478 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.479 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.479 DEBUG:boto:Method: GET 2023-10-18T21:04:08.480 DEBUG:boto:Path: /bofrnf-64/obj33 2023-10-18T21:04:08.480 DEBUG:boto:Data: 2023-10-18T21:04:08.480 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.481 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.481 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.482 DEBUG:boto:Params: {} 2023-10-18T21:04:08.482 DEBUG:boto:Token: None 2023-10-18T21:04:08.483 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33 2023-10-18T21:04:08.483 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n4ne0CYB9cwBB0Ihg+5MjECrtyI= 2023-10-18T21:04:08.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n4ne0CYB9cwBB0Ihg+5MjECrtyI=', 'Content-Length': '0'} 2023-10-18T21:04:08.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e896ff27003a76fc-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.530 DEBUG:boto:path=/bofrnf-64/obj33 2023-10-18T21:04:08.531 DEBUG:boto:auth_path=/bofrnf-64/obj33 2023-10-18T21:04:08.532 DEBUG:boto:Method: GET 2023-10-18T21:04:08.532 DEBUG:boto:Path: /bofrnf-64/obj33 2023-10-18T21:04:08.532 DEBUG:boto:Data: 2023-10-18T21:04:08.533 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.533 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.534 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.534 DEBUG:boto:Params: {} 2023-10-18T21:04:08.535 DEBUG:boto:Token: None 2023-10-18T21:04:08.535 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj33 2023-10-18T21:04:08.536 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:n4ne0CYB9cwBB0Ihg+5MjECrtyI= 2023-10-18T21:04:08.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n4ne0CYB9cwBB0Ihg+5MjECrtyI=', 'Content-Length': '0'} 2023-10-18T21:04:08.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af1ceae86ab746a-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.539 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.539 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.540 DEBUG:boto:path=/bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.540 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.541 DEBUG:boto:Method: GET 2023-10-18T21:04:08.541 DEBUG:boto:Path: /bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.541 DEBUG:boto:Data: 2023-10-18T21:04:08.542 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.542 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.543 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.543 DEBUG:boto:Params: {} 2023-10-18T21:04:08.544 DEBUG:boto:Token: None 2023-10-18T21:04:08.545 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.545 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c3cbc4bbde9b095-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.548 DEBUG:boto:b'bofrnf-64obj331000falseobj332023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.549 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.549 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.550 DEBUG:boto:path=/bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.550 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.551 DEBUG:boto:Method: GET 2023-10-18T21:04:08.551 DEBUG:boto:Path: /bofrnf-64/?prefix=obj33 2023-10-18T21:04:08.552 DEBUG:boto:Data: 2023-10-18T21:04:08.552 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.553 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.553 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.553 DEBUG:boto:Params: {} 2023-10-18T21:04:08.554 DEBUG:boto:Token: None 2023-10-18T21:04:08.554 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.555 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044724e334710008b-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.558 DEBUG:boto:b'bofrnf-64obj331000falseobj332023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T21:04:08.559 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.560 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.560 DEBUG:boto:path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.560 DEBUG:boto:auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.561 DEBUG:boto:Method: GET 2023-10-18T21:04:08.561 DEBUG:boto:Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.562 DEBUG:boto:Data: 2023-10-18T21:04:08.562 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.563 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.563 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.563 DEBUG:boto:Params: {} 2023-10-18T21:04:08.564 DEBUG:boto:Token: None 2023-10-18T21:04:08.564 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.565 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc= 2023-10-18T21:04:08.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc=', 'Content-Length': '0'} 2023-10-18T21:04:08.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd82f5facbbe61ba-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.568 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.568 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.569 DEBUG:boto:path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.569 DEBUG:boto:auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.570 DEBUG:boto:Method: GET 2023-10-18T21:04:08.570 DEBUG:boto:Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.570 DEBUG:boto:Data: 2023-10-18T21:04:08.571 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.571 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.573 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.573 DEBUG:boto:Params: {} 2023-10-18T21:04:08.574 DEBUG:boto:Token: None 2023-10-18T21:04:08.574 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.575 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc= 2023-10-18T21:04:08.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc=', 'Content-Length': '0'} 2023-10-18T21:04:08.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b96e10ecedefc6-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.578 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.578 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.579 DEBUG:boto:path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.579 DEBUG:boto:auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.580 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.580 DEBUG:boto:Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.580 DEBUG:boto:Data: 2023-10-18T21:04:08.581 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.581 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.582 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.582 DEBUG:boto:Params: {} 2023-10-18T21:04:08.583 DEBUG:boto:Token: None 2023-10-18T21:04:08.583 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.583 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FVO9Q8v1tbeYdo86GBSXbwAQrF8= 2023-10-18T21:04:08.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVO9Q8v1tbeYdo86GBSXbwAQrF8=', 'Content-Length': '0'} 2023-10-18T21:04:08.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000223c444ebc60d24-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.586 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.587 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.587 DEBUG:boto:path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.588 DEBUG:boto:auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.588 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.588 DEBUG:boto:Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.589 DEBUG:boto:Data: 2023-10-18T21:04:08.589 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.590 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.590 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.590 DEBUG:boto:Params: {} 2023-10-18T21:04:08.591 DEBUG:boto:Token: None 2023-10-18T21:04:08.591 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.592 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FVO9Q8v1tbeYdo86GBSXbwAQrF8= 2023-10-18T21:04:08.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVO9Q8v1tbeYdo86GBSXbwAQrF8=', 'Content-Length': '0'} 2023-10-18T21:04:08.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cf572b0f8a626b4-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T21:04:08.595 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.595 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.596 DEBUG:boto:path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.596 DEBUG:boto:auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.597 DEBUG:boto:Method: GET 2023-10-18T21:04:08.597 DEBUG:boto:Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.597 DEBUG:boto:Data: 2023-10-18T21:04:08.598 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.598 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.599 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.599 DEBUG:boto:Params: {} 2023-10-18T21:04:08.600 DEBUG:boto:Token: None 2023-10-18T21:04:08.600 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.600 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc= 2023-10-18T21:04:08.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc=', 'Content-Length': '0'} 2023-10-18T21:04:08.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2860e3ed40d7195-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.604 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.604 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.604 DEBUG:boto:path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.605 DEBUG:boto:auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.605 DEBUG:boto:Method: GET 2023-10-18T21:04:08.606 DEBUG:boto:Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.606 DEBUG:boto:Data: 2023-10-18T21:04:08.606 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.607 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.607 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.608 DEBUG:boto:Params: {} 2023-10-18T21:04:08.608 DEBUG:boto:Token: None 2023-10-18T21:04:08.609 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44?versionId=null 2023-10-18T21:04:08.609 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc= 2023-10-18T21:04:08.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc=', 'Content-Length': '0'} 2023-10-18T21:04:08.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423bdf3bc88b443b-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.612 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.613 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.613 DEBUG:boto:path=/bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.613 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.614 DEBUG:boto:Method: GET 2023-10-18T21:04:08.614 DEBUG:boto:Path: /bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.614 DEBUG:boto:Data: 2023-10-18T21:04:08.615 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.615 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.616 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.616 DEBUG:boto:Params: {} 2023-10-18T21:04:08.616 DEBUG:boto:Token: None 2023-10-18T21:04:08.617 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.617 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e2d53ff4b16568c-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.662 DEBUG:boto:b'bofrnf-64obj441000falseobj442023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.663 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.663 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.664 DEBUG:boto:path=/bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.664 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.664 DEBUG:boto:Method: GET 2023-10-18T21:04:08.665 DEBUG:boto:Path: /bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.665 DEBUG:boto:Data: 2023-10-18T21:04:08.666 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.666 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.666 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.667 DEBUG:boto:Params: {} 2023-10-18T21:04:08.667 DEBUG:boto:Token: None 2023-10-18T21:04:08.667 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.668 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082e721551ce07362-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.671 DEBUG:boto:b'bofrnf-64obj441000falseobj442023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.672 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.672 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.673 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.673 DEBUG:boto:Path: /bofrnf-64/obj44 2023-10-18T21:04:08.673 DEBUG:boto:Data: 2023-10-18T21:04:08.674 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.674 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.674 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.675 DEBUG:boto:Params: {} 2023-10-18T21:04:08.675 DEBUG:boto:Token: None 2023-10-18T21:04:08.676 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44 2023-10-18T21:04:08.676 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hlRgC63GcHmaaGRJRKTVfn0gaK0= 2023-10-18T21:04:08.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hlRgC63GcHmaaGRJRKTVfn0gaK0=', 'Content-Length': '0'} 2023-10-18T21:04:08.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c173bb352443933d-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.679 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.679 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.680 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.680 DEBUG:boto:Path: /bofrnf-64/obj44 2023-10-18T21:04:08.680 DEBUG:boto:Data: 2023-10-18T21:04:08.681 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.682 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.682 DEBUG:boto:Params: {} 2023-10-18T21:04:08.682 DEBUG:boto:Token: None 2023-10-18T21:04:08.683 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44 2023-10-18T21:04:08.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hlRgC63GcHmaaGRJRKTVfn0gaK0= 2023-10-18T21:04:08.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hlRgC63GcHmaaGRJRKTVfn0gaK0=', 'Content-Length': '0'} 2023-10-18T21:04:08.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d6d1c095708447-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-18T21:04:08.686 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.687 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.687 DEBUG:boto:Method: GET 2023-10-18T21:04:08.688 DEBUG:boto:Path: /bofrnf-64/obj44 2023-10-18T21:04:08.688 DEBUG:boto:Data: 2023-10-18T21:04:08.688 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.689 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.689 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.690 DEBUG:boto:Params: {} 2023-10-18T21:04:08.690 DEBUG:boto:Token: None 2023-10-18T21:04:08.691 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44 2023-10-18T21:04:08.691 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xvDDh+1pBRgQdQGssUcVGO2WQwQ= 2023-10-18T21:04:08.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xvDDh+1pBRgQdQGssUcVGO2WQwQ=', 'Content-Length': '0'} 2023-10-18T21:04:08.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdcca31f34a3678b-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.694 DEBUG:boto:path=/bofrnf-64/obj44 2023-10-18T21:04:08.695 DEBUG:boto:auth_path=/bofrnf-64/obj44 2023-10-18T21:04:08.695 DEBUG:boto:Method: GET 2023-10-18T21:04:08.695 DEBUG:boto:Path: /bofrnf-64/obj44 2023-10-18T21:04:08.696 DEBUG:boto:Data: 2023-10-18T21:04:08.696 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.697 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.697 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.697 DEBUG:boto:Params: {} 2023-10-18T21:04:08.698 DEBUG:boto:Token: None 2023-10-18T21:04:08.698 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj44 2023-10-18T21:04:08.699 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xvDDh+1pBRgQdQGssUcVGO2WQwQ= 2023-10-18T21:04:08.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xvDDh+1pBRgQdQGssUcVGO2WQwQ=', 'Content-Length': '0'} 2023-10-18T21:04:08.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d6a40e3ffec1fc9-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.702 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.702 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.703 DEBUG:boto:path=/bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.703 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.704 DEBUG:boto:Method: GET 2023-10-18T21:04:08.704 DEBUG:boto:Path: /bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.704 DEBUG:boto:Data: 2023-10-18T21:04:08.705 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.705 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.706 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.706 DEBUG:boto:Params: {} 2023-10-18T21:04:08.707 DEBUG:boto:Token: None 2023-10-18T21:04:08.707 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.708 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009afab75c98500265-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.711 DEBUG:boto:b'bofrnf-64obj441000falseobj442023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.711 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.712 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.712 DEBUG:boto:path=/bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.713 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.713 DEBUG:boto:Method: GET 2023-10-18T21:04:08.713 DEBUG:boto:Path: /bofrnf-64/?prefix=obj44 2023-10-18T21:04:08.714 DEBUG:boto:Data: 2023-10-18T21:04:08.714 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.715 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.715 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.716 DEBUG:boto:Params: {} 2023-10-18T21:04:08.716 DEBUG:boto:Token: None 2023-10-18T21:04:08.716 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.717 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026362bcec413ffae-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.720 DEBUG:boto:b'bofrnf-64obj441000falseobj442023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T21:04:08.721 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.721 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.722 DEBUG:boto:path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.722 DEBUG:boto:auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.723 DEBUG:boto:Method: GET 2023-10-18T21:04:08.723 DEBUG:boto:Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.723 DEBUG:boto:Data: 2023-10-18T21:04:08.724 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.724 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.725 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.725 DEBUG:boto:Params: {} 2023-10-18T21:04:08.725 DEBUG:boto:Token: None 2023-10-18T21:04:08.726 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.726 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I= 2023-10-18T21:04:08.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I=', 'Content-Length': '0'} 2023-10-18T21:04:08.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040838bdaf698856d-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.729 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.730 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.730 DEBUG:boto:path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.730 DEBUG:boto:auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.731 DEBUG:boto:Method: GET 2023-10-18T21:04:08.731 DEBUG:boto:Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.732 DEBUG:boto:Data: 2023-10-18T21:04:08.732 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.733 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.733 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.733 DEBUG:boto:Params: {} 2023-10-18T21:04:08.734 DEBUG:boto:Token: None 2023-10-18T21:04:08.734 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.735 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I= 2023-10-18T21:04:08.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I=', 'Content-Length': '0'} 2023-10-18T21:04:08.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0a5ada9b7ed25e-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.778 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.779 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.779 DEBUG:boto:path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.780 DEBUG:boto:auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.780 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.780 DEBUG:boto:Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.781 DEBUG:boto:Data: 2023-10-18T21:04:08.781 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.782 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.782 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.782 DEBUG:boto:Params: {} 2023-10-18T21:04:08.783 DEBUG:boto:Token: None 2023-10-18T21:04:08.783 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.784 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1E+jsSsKOzITrXWjvt/DjXRBMLA= 2023-10-18T21:04:08.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1E+jsSsKOzITrXWjvt/DjXRBMLA=', 'Content-Length': '0'} 2023-10-18T21:04:08.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b21cd94c3f919f6-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.787 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.787 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.788 DEBUG:boto:path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.788 DEBUG:boto:auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.789 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.789 DEBUG:boto:Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.789 DEBUG:boto:Data: 2023-10-18T21:04:08.790 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.790 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.791 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.791 DEBUG:boto:Params: {} 2023-10-18T21:04:08.792 DEBUG:boto:Token: None 2023-10-18T21:04:08.792 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.792 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1E+jsSsKOzITrXWjvt/DjXRBMLA= 2023-10-18T21:04:08.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1E+jsSsKOzITrXWjvt/DjXRBMLA=', 'Content-Length': '0'} 2023-10-18T21:04:08.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000979812c52392e4a1-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T21:04:08.796 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.796 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.797 DEBUG:boto:path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.797 DEBUG:boto:auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.797 DEBUG:boto:Method: GET 2023-10-18T21:04:08.798 DEBUG:boto:Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.798 DEBUG:boto:Data: 2023-10-18T21:04:08.799 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.799 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.799 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.800 DEBUG:boto:Params: {} 2023-10-18T21:04:08.800 DEBUG:boto:Token: None 2023-10-18T21:04:08.801 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.801 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I= 2023-10-18T21:04:08.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I=', 'Content-Length': '0'} 2023-10-18T21:04:08.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0ef29856799dd8-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.846 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.847 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.847 DEBUG:boto:path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.847 DEBUG:boto:auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.848 DEBUG:boto:Method: GET 2023-10-18T21:04:08.848 DEBUG:boto:Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.849 DEBUG:boto:Data: 2023-10-18T21:04:08.849 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.849 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.850 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.850 DEBUG:boto:Params: {} 2023-10-18T21:04:08.851 DEBUG:boto:Token: None 2023-10-18T21:04:08.851 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55?versionId=null 2023-10-18T21:04:08.852 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I= 2023-10-18T21:04:08.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I=', 'Content-Length': '0'} 2023-10-18T21:04:08.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c617669dbc499db4-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.855 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.855 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.856 DEBUG:boto:path=/bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.856 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.856 DEBUG:boto:Method: GET 2023-10-18T21:04:08.857 DEBUG:boto:Path: /bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.857 DEBUG:boto:Data: 2023-10-18T21:04:08.858 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.858 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.858 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.859 DEBUG:boto:Params: {} 2023-10-18T21:04:08.859 DEBUG:boto:Token: None 2023-10-18T21:04:08.860 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.860 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8b745fa6dd862d-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.863 DEBUG:boto:b'bofrnf-64obj551000falseobj552023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.864 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.865 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.865 DEBUG:boto:path=/bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.866 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.866 DEBUG:boto:Method: GET 2023-10-18T21:04:08.867 DEBUG:boto:Path: /bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.867 DEBUG:boto:Data: 2023-10-18T21:04:08.867 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.868 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.869 DEBUG:boto:Params: {} 2023-10-18T21:04:08.869 DEBUG:boto:Token: None 2023-10-18T21:04:08.870 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.870 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7976a524d3060fa-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.874 DEBUG:boto:b'bofrnf-64obj551000falseobj552023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.875 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.875 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.875 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.876 DEBUG:boto:Path: /bofrnf-64/obj55 2023-10-18T21:04:08.876 DEBUG:boto:Data: 2023-10-18T21:04:08.877 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.877 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.877 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.878 DEBUG:boto:Params: {} 2023-10-18T21:04:08.878 DEBUG:boto:Token: None 2023-10-18T21:04:08.879 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55 2023-10-18T21:04:08.879 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BTV67fGnRvtsk/wfqM4PNjih7lw= 2023-10-18T21:04:08.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BTV67fGnRvtsk/wfqM4PNjih7lw=', 'Content-Length': '0'} 2023-10-18T21:04:08.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e83a48c8f5f85d-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.882 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.882 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.883 DEBUG:boto:Method: HEAD 2023-10-18T21:04:08.883 DEBUG:boto:Path: /bofrnf-64/obj55 2023-10-18T21:04:08.884 DEBUG:boto:Data: 2023-10-18T21:04:08.884 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.884 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.885 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.885 DEBUG:boto:Params: {} 2023-10-18T21:04:08.886 DEBUG:boto:Token: None 2023-10-18T21:04:08.886 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55 2023-10-18T21:04:08.887 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BTV67fGnRvtsk/wfqM4PNjih7lw= 2023-10-18T21:04:08.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BTV67fGnRvtsk/wfqM4PNjih7lw=', 'Content-Length': '0'} 2023-10-18T21:04:08.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c84247aacdd854af-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-18T21:04:08.890 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.890 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.891 DEBUG:boto:Method: GET 2023-10-18T21:04:08.891 DEBUG:boto:Path: /bofrnf-64/obj55 2023-10-18T21:04:08.892 DEBUG:boto:Data: 2023-10-18T21:04:08.892 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.893 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.893 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.893 DEBUG:boto:Params: {} 2023-10-18T21:04:08.894 DEBUG:boto:Token: None 2023-10-18T21:04:08.894 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55 2023-10-18T21:04:08.895 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GbHY1U4oQD8UHBm9ViYtJlGzeyg= 2023-10-18T21:04:08.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GbHY1U4oQD8UHBm9ViYtJlGzeyg=', 'Content-Length': '0'} 2023-10-18T21:04:08.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca128905657ccbda-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.897 DEBUG:boto:path=/bofrnf-64/obj55 2023-10-18T21:04:08.898 DEBUG:boto:auth_path=/bofrnf-64/obj55 2023-10-18T21:04:08.898 DEBUG:boto:Method: GET 2023-10-18T21:04:08.899 DEBUG:boto:Path: /bofrnf-64/obj55 2023-10-18T21:04:08.899 DEBUG:boto:Data: 2023-10-18T21:04:08.900 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.900 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.900 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.901 DEBUG:boto:Params: {} 2023-10-18T21:04:08.901 DEBUG:boto:Token: None 2023-10-18T21:04:08.902 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/obj55 2023-10-18T21:04:08.902 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GbHY1U4oQD8UHBm9ViYtJlGzeyg= 2023-10-18T21:04:08.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GbHY1U4oQD8UHBm9ViYtJlGzeyg=', 'Content-Length': '0'} 2023-10-18T21:04:08.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c11dc2278cf7ac7e-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.905 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.905 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.906 DEBUG:boto:path=/bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.906 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.906 DEBUG:boto:Method: GET 2023-10-18T21:04:08.907 DEBUG:boto:Path: /bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.907 DEBUG:boto:Data: 2023-10-18T21:04:08.908 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.908 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:08.909 DEBUG:boto:Port: 8001 2023-10-18T21:04:08.909 DEBUG:boto:Params: {} 2023-10-18T21:04:08.909 DEBUG:boto:Token: None 2023-10-18T21:04:08.910 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.910 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ebfd5bdcdd4c695-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.913 DEBUG:boto:b'bofrnf-64obj551000falseobj552023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.914 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:08.914 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:08.915 DEBUG:boto:path=/bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.915 DEBUG:boto:auth_path=/bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.916 DEBUG:boto:Method: GET 2023-10-18T21:04:08.916 DEBUG:boto:Path: /bofrnf-64/?prefix=obj55 2023-10-18T21:04:08.916 DEBUG:boto:Data: 2023-10-18T21:04:08.917 DEBUG:boto:Headers: {} 2023-10-18T21:04:08.917 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:08.917 DEBUG:boto:Port: 8000 2023-10-18T21:04:08.918 DEBUG:boto:Params: {} 2023-10-18T21:04:08.918 DEBUG:boto:Token: None 2023-10-18T21:04:08.919 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:04:08 GMT /bofrnf-64/ 2023-10-18T21:04:08.919 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:04:08.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:04:08.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c218ae6ebc70c56-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:08.922 DEBUG:boto:b'bofrnf-64obj551000falseobj552023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:04:08.923 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bofrnf-64 zones={a2, a1} 2023-10-18T21:04:08.924 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-18T21:04:08.924 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:04:08.925 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T21:04:08.922+0000 7ff122b99700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 176793) UID: 0 2023-10-18T21:04:08.925 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T21:04:08.922+0000 7ff126e027c0 -1 shutting down 2023-10-18T21:04:10.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T21:04:15.023 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-18T21:04:15.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:15.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:15.026 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:04:15.087 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:04:15.088 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:04:15.133 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T21:04:15.133 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-64 2023-10-18T21:04:15.137 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 5 2023-10-18T21:04:15.174 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.175 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.176 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.177 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.178 DEBUG:boto:Data: 2023-10-18T21:04:15.178 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.179 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.180 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.180 DEBUG:boto:Params: {} 2023-10-18T21:04:15.181 DEBUG:boto:Token: None 2023-10-18T21:04:15.182 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.183 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e34ebf078087c9f6-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.188 DEBUG:boto:path=/bofrnf-64/obj17 2023-10-18T21:04:15.189 DEBUG:boto:auth_path=/bofrnf-64/obj17 2023-10-18T21:04:15.190 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.191 DEBUG:boto:Path: /bofrnf-64/obj17 2023-10-18T21:04:15.191 DEBUG:boto:Data: 2023-10-18T21:04:15.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.193 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.193 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.194 DEBUG:boto:Params: {} 2023-10-18T21:04:15.195 DEBUG:boto:Token: None 2023-10-18T21:04:15.196 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj17 2023-10-18T21:04:15.197 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NrntbniOlHTRPoDcOJ1lmck1jcY= 2023-10-18T21:04:15.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NrntbniOlHTRPoDcOJ1lmck1jcY='} 2023-10-18T21:04:15.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000878e82c71c1ff91c-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.243 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.244 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.245 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.246 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.246 DEBUG:boto:Data: 2023-10-18T21:04:15.247 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.248 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.249 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.249 DEBUG:boto:Params: {} 2023-10-18T21:04:15.250 DEBUG:boto:Token: None 2023-10-18T21:04:15.251 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.252 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a463868ff2a34b6f-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.256 DEBUG:boto:path=/bofrnf-64/obj34 2023-10-18T21:04:15.257 DEBUG:boto:auth_path=/bofrnf-64/obj34 2023-10-18T21:04:15.257 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.258 DEBUG:boto:Path: /bofrnf-64/obj34 2023-10-18T21:04:15.259 DEBUG:boto:Data: 2023-10-18T21:04:15.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.260 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.261 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.262 DEBUG:boto:Params: {} 2023-10-18T21:04:15.262 DEBUG:boto:Token: None 2023-10-18T21:04:15.263 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj34 2023-10-18T21:04:15.264 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vcgPA7yxDmJfdTcr4IXL5mNUFH4= 2023-10-18T21:04:15.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vcgPA7yxDmJfdTcr4IXL5mNUFH4='} 2023-10-18T21:04:15.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003900c8c3e0f89c5-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.271 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.272 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.272 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.273 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.273 DEBUG:boto:Data: 2023-10-18T21:04:15.274 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.275 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.276 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.276 DEBUG:boto:Params: {} 2023-10-18T21:04:15.277 DEBUG:boto:Token: None 2023-10-18T21:04:15.278 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.278 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046a27bab10418f7a-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.283 DEBUG:boto:path=/bofrnf-64/obj51 2023-10-18T21:04:15.284 DEBUG:boto:auth_path=/bofrnf-64/obj51 2023-10-18T21:04:15.284 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.285 DEBUG:boto:Path: /bofrnf-64/obj51 2023-10-18T21:04:15.286 DEBUG:boto:Data: 2023-10-18T21:04:15.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.287 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.288 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.289 DEBUG:boto:Params: {} 2023-10-18T21:04:15.290 DEBUG:boto:Token: None 2023-10-18T21:04:15.290 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj51 2023-10-18T21:04:15.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3qVJiRpwwE/va4AFiu3lpipEVo8= 2023-10-18T21:04:15.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3qVJiRpwwE/va4AFiu3lpipEVo8='} 2023-10-18T21:04:15.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcd9347c3ca7b063-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.335 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.336 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.337 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.337 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.338 DEBUG:boto:Data: 2023-10-18T21:04:15.339 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.339 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.340 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.341 DEBUG:boto:Params: {} 2023-10-18T21:04:15.342 DEBUG:boto:Token: None 2023-10-18T21:04:15.342 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.343 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000616c3a480b1d5cfc-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.348 DEBUG:boto:path=/bofrnf-64/obj68 2023-10-18T21:04:15.348 DEBUG:boto:auth_path=/bofrnf-64/obj68 2023-10-18T21:04:15.349 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.350 DEBUG:boto:Path: /bofrnf-64/obj68 2023-10-18T21:04:15.351 DEBUG:boto:Data: 2023-10-18T21:04:15.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.352 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.353 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.353 DEBUG:boto:Params: {} 2023-10-18T21:04:15.354 DEBUG:boto:Token: None 2023-10-18T21:04:15.355 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj68 2023-10-18T21:04:15.356 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7UcniNPxTCAT+Ah9Alow1WuuqJs= 2023-10-18T21:04:15.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7UcniNPxTCAT+Ah9Alow1WuuqJs='} 2023-10-18T21:04:15.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b99573fd1f910b54-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.363 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.364 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.364 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.365 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.366 DEBUG:boto:Data: 2023-10-18T21:04:15.366 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.367 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.368 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.368 DEBUG:boto:Params: {} 2023-10-18T21:04:15.369 DEBUG:boto:Token: None 2023-10-18T21:04:15.370 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.371 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b31327c990e8619e-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.375 DEBUG:boto:path=/bofrnf-64/obj85 2023-10-18T21:04:15.376 DEBUG:boto:auth_path=/bofrnf-64/obj85 2023-10-18T21:04:15.377 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.377 DEBUG:boto:Path: /bofrnf-64/obj85 2023-10-18T21:04:15.378 DEBUG:boto:Data: 2023-10-18T21:04:15.379 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.379 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.380 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.381 DEBUG:boto:Params: {} 2023-10-18T21:04:15.382 DEBUG:boto:Token: None 2023-10-18T21:04:15.382 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj85 2023-10-18T21:04:15.383 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QvXzdjkfA9yew1UfL71X/+uUR3Q= 2023-10-18T21:04:15.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QvXzdjkfA9yew1UfL71X/+uUR3Q='} 2023-10-18T21:04:15.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d1f47a3acb569fe-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:15.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:15.428 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:04:15.481 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:04:15.482 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:04:15.525 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T21:04:15.526 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-64 2023-10-18T21:04:15.532 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 10 2023-10-18T21:04:15.575 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.575 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.575 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.576 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.576 DEBUG:boto:Data: 2023-10-18T21:04:15.577 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.577 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.577 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.578 DEBUG:boto:Params: {} 2023-10-18T21:04:15.578 DEBUG:boto:Token: None 2023-10-18T21:04:15.579 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.579 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5f5f0cc5352b663-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.583 DEBUG:boto:path=/bofrnf-64/obj19 2023-10-18T21:04:15.583 DEBUG:boto:auth_path=/bofrnf-64/obj19 2023-10-18T21:04:15.584 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.584 DEBUG:boto:Path: /bofrnf-64/obj19 2023-10-18T21:04:15.584 DEBUG:boto:Data: 2023-10-18T21:04:15.585 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.585 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.585 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.586 DEBUG:boto:Params: {} 2023-10-18T21:04:15.586 DEBUG:boto:Token: None 2023-10-18T21:04:15.587 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj19 2023-10-18T21:04:15.587 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qfAaWmCKK3Xtp3R9UBXL12y+Phk= 2023-10-18T21:04:15.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qfAaWmCKK3Xtp3R9UBXL12y+Phk='} 2023-10-18T21:04:15.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac5f5a801ee266ff-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.593 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.593 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.594 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.594 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.594 DEBUG:boto:Data: 2023-10-18T21:04:15.595 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.595 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.595 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.596 DEBUG:boto:Params: {} 2023-10-18T21:04:15.596 DEBUG:boto:Token: None 2023-10-18T21:04:15.597 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.597 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7af20a156552ed9-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.601 DEBUG:boto:path=/bofrnf-64/obj38 2023-10-18T21:04:15.601 DEBUG:boto:auth_path=/bofrnf-64/obj38 2023-10-18T21:04:15.601 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.602 DEBUG:boto:Path: /bofrnf-64/obj38 2023-10-18T21:04:15.602 DEBUG:boto:Data: 2023-10-18T21:04:15.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.603 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.603 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.604 DEBUG:boto:Params: {} 2023-10-18T21:04:15.604 DEBUG:boto:Token: None 2023-10-18T21:04:15.605 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj38 2023-10-18T21:04:15.605 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0I6V8oTnRABjnoipRUsNxe/NiOY= 2023-10-18T21:04:15.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0I6V8oTnRABjnoipRUsNxe/NiOY='} 2023-10-18T21:04:15.609 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053de607309a416dd-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.610 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.610 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.610 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.611 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.611 DEBUG:boto:Data: 2023-10-18T21:04:15.611 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.612 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.612 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.612 DEBUG:boto:Params: {} 2023-10-18T21:04:15.613 DEBUG:boto:Token: None 2023-10-18T21:04:15.613 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.614 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3d5884aa07b40de-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.617 DEBUG:boto:path=/bofrnf-64/obj57 2023-10-18T21:04:15.618 DEBUG:boto:auth_path=/bofrnf-64/obj57 2023-10-18T21:04:15.618 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.619 DEBUG:boto:Path: /bofrnf-64/obj57 2023-10-18T21:04:15.619 DEBUG:boto:Data: 2023-10-18T21:04:15.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.620 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.620 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.621 DEBUG:boto:Params: {} 2023-10-18T21:04:15.621 DEBUG:boto:Token: None 2023-10-18T21:04:15.621 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj57 2023-10-18T21:04:15.622 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:y2kC1j3Pg8elMPGROtZvUx2LfHk= 2023-10-18T21:04:15.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y2kC1j3Pg8elMPGROtZvUx2LfHk='} 2023-10-18T21:04:15.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f254fe238ab7793-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.627 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.627 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.627 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.628 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.628 DEBUG:boto:Data: 2023-10-18T21:04:15.629 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.629 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.629 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.630 DEBUG:boto:Params: {} 2023-10-18T21:04:15.630 DEBUG:boto:Token: None 2023-10-18T21:04:15.631 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.631 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002df6f52828d70801-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.635 DEBUG:boto:path=/bofrnf-64/obj76 2023-10-18T21:04:15.635 DEBUG:boto:auth_path=/bofrnf-64/obj76 2023-10-18T21:04:15.635 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.636 DEBUG:boto:Path: /bofrnf-64/obj76 2023-10-18T21:04:15.636 DEBUG:boto:Data: 2023-10-18T21:04:15.637 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.637 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.637 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.638 DEBUG:boto:Params: {} 2023-10-18T21:04:15.638 DEBUG:boto:Token: None 2023-10-18T21:04:15.639 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj76 2023-10-18T21:04:15.639 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:w+N2Y4sDo99mqeEz0mQzGZTkp9U= 2023-10-18T21:04:15.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w+N2Y4sDo99mqeEz0mQzGZTkp9U='} 2023-10-18T21:04:15.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d050df8199c34619-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.645 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.645 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.646 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.646 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.647 DEBUG:boto:Data: 2023-10-18T21:04:15.647 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.647 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.648 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.648 DEBUG:boto:Params: {} 2023-10-18T21:04:15.649 DEBUG:boto:Token: None 2023-10-18T21:04:15.649 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.649 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7012c5660098b59-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.653 DEBUG:boto:path=/bofrnf-64/obj95 2023-10-18T21:04:15.653 DEBUG:boto:auth_path=/bofrnf-64/obj95 2023-10-18T21:04:15.654 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.654 DEBUG:boto:Path: /bofrnf-64/obj95 2023-10-18T21:04:15.655 DEBUG:boto:Data: 2023-10-18T21:04:15.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.655 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.656 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.656 DEBUG:boto:Params: {} 2023-10-18T21:04:15.657 DEBUG:boto:Token: None 2023-10-18T21:04:15.657 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj95 2023-10-18T21:04:15.658 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2x8q8PGQT2A9vebMvcoWdgr+kqs= 2023-10-18T21:04:15.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2x8q8PGQT2A9vebMvcoWdgr+kqs='} 2023-10-18T21:04:15.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006532da93468e1789-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:15.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:15.704 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:04:15.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-18T21:04:15.756 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:04:15.756 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:04:15.801 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T21:04:15.801 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-64 2023-10-18T21:04:15.808 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 15 2023-10-18T21:04:15.853 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.854 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.854 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.855 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.855 DEBUG:boto:Data: 2023-10-18T21:04:15.855 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.856 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.856 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.856 DEBUG:boto:Params: {} 2023-10-18T21:04:15.857 DEBUG:boto:Token: None 2023-10-18T21:04:15.857 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.858 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cf4183dc6f92875-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.862 DEBUG:boto:path=/bofrnf-64/obj23 2023-10-18T21:04:15.862 DEBUG:boto:auth_path=/bofrnf-64/obj23 2023-10-18T21:04:15.862 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.863 DEBUG:boto:Path: /bofrnf-64/obj23 2023-10-18T21:04:15.863 DEBUG:boto:Data: 2023-10-18T21:04:15.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.864 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.864 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.865 DEBUG:boto:Params: {} 2023-10-18T21:04:15.865 DEBUG:boto:Token: None 2023-10-18T21:04:15.866 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj23 2023-10-18T21:04:15.866 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:miTEU3Qs0a6J0Fvnu2YZXWwvS3U= 2023-10-18T21:04:15.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:miTEU3Qs0a6J0Fvnu2YZXWwvS3U='} 2023-10-18T21:04:15.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072fde9172bade3d7-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.872 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.872 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.873 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.873 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.874 DEBUG:boto:Data: 2023-10-18T21:04:15.874 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.874 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.875 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.875 DEBUG:boto:Params: {} 2023-10-18T21:04:15.875 DEBUG:boto:Token: None 2023-10-18T21:04:15.876 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.876 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ae40b2f57a4d70d-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.880 DEBUG:boto:path=/bofrnf-64/obj46 2023-10-18T21:04:15.880 DEBUG:boto:auth_path=/bofrnf-64/obj46 2023-10-18T21:04:15.880 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.881 DEBUG:boto:Path: /bofrnf-64/obj46 2023-10-18T21:04:15.881 DEBUG:boto:Data: 2023-10-18T21:04:15.882 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.882 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.883 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.883 DEBUG:boto:Params: {} 2023-10-18T21:04:15.883 DEBUG:boto:Token: None 2023-10-18T21:04:15.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj46 2023-10-18T21:04:15.884 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EH6nySVuYDwb1kFRQH8KYyzmWAc= 2023-10-18T21:04:15.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EH6nySVuYDwb1kFRQH8KYyzmWAc='} 2023-10-18T21:04:15.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a51e0a2f99e7f6c9-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.890 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.891 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.891 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.891 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.892 DEBUG:boto:Data: 2023-10-18T21:04:15.892 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.893 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.893 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.893 DEBUG:boto:Params: {} 2023-10-18T21:04:15.894 DEBUG:boto:Token: None 2023-10-18T21:04:15.894 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.895 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb135227f6a304a4-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.899 DEBUG:boto:path=/bofrnf-64/obj69 2023-10-18T21:04:15.899 DEBUG:boto:auth_path=/bofrnf-64/obj69 2023-10-18T21:04:15.899 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.900 DEBUG:boto:Path: /bofrnf-64/obj69 2023-10-18T21:04:15.900 DEBUG:boto:Data: 2023-10-18T21:04:15.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.901 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.901 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.902 DEBUG:boto:Params: {} 2023-10-18T21:04:15.902 DEBUG:boto:Token: None 2023-10-18T21:04:15.903 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj69 2023-10-18T21:04:15.903 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cTMrgEmuoMwr5UPeDW3yr+ejkrc= 2023-10-18T21:04:15.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cTMrgEmuoMwr5UPeDW3yr+ejkrc='} 2023-10-18T21:04:15.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cfaa12c00ed7046-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.947 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.947 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.948 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.948 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.948 DEBUG:boto:Data: 2023-10-18T21:04:15.949 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.949 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.949 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.950 DEBUG:boto:Params: {} 2023-10-18T21:04:15.950 DEBUG:boto:Token: None 2023-10-18T21:04:15.951 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.951 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039d069f3dc773b21-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.955 DEBUG:boto:path=/bofrnf-64/obj92 2023-10-18T21:04:15.955 DEBUG:boto:auth_path=/bofrnf-64/obj92 2023-10-18T21:04:15.956 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.956 DEBUG:boto:Path: /bofrnf-64/obj92 2023-10-18T21:04:15.956 DEBUG:boto:Data: 2023-10-18T21:04:15.957 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.957 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.958 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.958 DEBUG:boto:Params: {} 2023-10-18T21:04:15.959 DEBUG:boto:Token: None 2023-10-18T21:04:15.959 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj92 2023-10-18T21:04:15.959 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7ZY+Op2OHhvH8eKLcNU0kDtn0ko= 2023-10-18T21:04:15.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7ZY+Op2OHhvH8eKLcNU0kDtn0ko='} 2023-10-18T21:04:15.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7f7795450a219d1-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.965 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:15.966 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:15.966 DEBUG:boto:Method: HEAD 2023-10-18T21:04:15.967 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:15.967 DEBUG:boto:Data: 2023-10-18T21:04:15.967 DEBUG:boto:Headers: {} 2023-10-18T21:04:15.968 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.968 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.968 DEBUG:boto:Params: {} 2023-10-18T21:04:15.969 DEBUG:boto:Token: None 2023-10-18T21:04:15.969 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/ 2023-10-18T21:04:15.970 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:04:15.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:04:15.973 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ae88fc788cd65e3-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:15.973 DEBUG:boto:path=/bofrnf-64/obj115 2023-10-18T21:04:15.974 DEBUG:boto:auth_path=/bofrnf-64/obj115 2023-10-18T21:04:15.974 DEBUG:boto:Method: PUT 2023-10-18T21:04:15.975 DEBUG:boto:Path: /bofrnf-64/obj115 2023-10-18T21:04:15.975 DEBUG:boto:Data: 2023-10-18T21:04:15.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:15.976 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:15.976 DEBUG:boto:Port: 8000 2023-10-18T21:04:15.976 DEBUG:boto:Params: {} 2023-10-18T21:04:15.977 DEBUG:boto:Token: None 2023-10-18T21:04:15.977 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:15 GMT /bofrnf-64/obj115 2023-10-18T21:04:15.978 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sARDa4dDHpXEEhHXw23FZEaMcJk= 2023-10-18T21:04:15.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sARDa4dDHpXEEhHXw23FZEaMcJk='} 2023-10-18T21:04:16.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbda3440f5d43242-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:16.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:16.024 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:04:16.075 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:04:16.076 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:04:16.122 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T21:04:16.122 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-64 2023-10-18T21:04:16.129 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 20 2023-10-18T21:04:16.179 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.180 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.180 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.181 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.181 DEBUG:boto:Data: 2023-10-18T21:04:16.181 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.182 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.182 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.183 DEBUG:boto:Params: {} 2023-10-18T21:04:16.183 DEBUG:boto:Token: None 2023-10-18T21:04:16.183 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.184 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a44dbf8bc780768b-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.188 DEBUG:boto:path=/bofrnf-64/obj29 2023-10-18T21:04:16.189 DEBUG:boto:auth_path=/bofrnf-64/obj29 2023-10-18T21:04:16.189 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.189 DEBUG:boto:Path: /bofrnf-64/obj29 2023-10-18T21:04:16.190 DEBUG:boto:Data: 2023-10-18T21:04:16.190 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.191 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.191 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.191 DEBUG:boto:Params: {} 2023-10-18T21:04:16.192 DEBUG:boto:Token: None 2023-10-18T21:04:16.192 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj29 2023-10-18T21:04:16.193 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0Us8prGuf5odkhB8y0x0V87Gn+8= 2023-10-18T21:04:16.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0Us8prGuf5odkhB8y0x0V87Gn+8='} 2023-10-18T21:04:16.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ba1d1d74b284cc9-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.199 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.199 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.200 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.200 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.200 DEBUG:boto:Data: 2023-10-18T21:04:16.201 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.201 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.201 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.202 DEBUG:boto:Params: {} 2023-10-18T21:04:16.202 DEBUG:boto:Token: None 2023-10-18T21:04:16.203 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.203 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.206 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000585a07531bc6f836-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.207 DEBUG:boto:path=/bofrnf-64/obj58 2023-10-18T21:04:16.207 DEBUG:boto:auth_path=/bofrnf-64/obj58 2023-10-18T21:04:16.207 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.208 DEBUG:boto:Path: /bofrnf-64/obj58 2023-10-18T21:04:16.208 DEBUG:boto:Data: 2023-10-18T21:04:16.209 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.209 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.209 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.210 DEBUG:boto:Params: {} 2023-10-18T21:04:16.210 DEBUG:boto:Token: None 2023-10-18T21:04:16.210 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj58 2023-10-18T21:04:16.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:h97oeUJNH7pdFjzRTyBs65lBn9Q= 2023-10-18T21:04:16.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h97oeUJNH7pdFjzRTyBs65lBn9Q='} 2023-10-18T21:04:16.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c6837b18aa15330-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.217 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.217 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.218 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.218 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.218 DEBUG:boto:Data: 2023-10-18T21:04:16.219 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.219 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.219 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.220 DEBUG:boto:Params: {} 2023-10-18T21:04:16.220 DEBUG:boto:Token: None 2023-10-18T21:04:16.220 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.221 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2f6bd7dde4cb31f-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.225 DEBUG:boto:path=/bofrnf-64/obj87 2023-10-18T21:04:16.225 DEBUG:boto:auth_path=/bofrnf-64/obj87 2023-10-18T21:04:16.225 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.226 DEBUG:boto:Path: /bofrnf-64/obj87 2023-10-18T21:04:16.226 DEBUG:boto:Data: 2023-10-18T21:04:16.227 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.227 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.227 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.228 DEBUG:boto:Params: {} 2023-10-18T21:04:16.228 DEBUG:boto:Token: None 2023-10-18T21:04:16.228 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj87 2023-10-18T21:04:16.229 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Ts+wGPw7TGvrHRKdnuK7JoQ/UVw= 2023-10-18T21:04:16.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ts+wGPw7TGvrHRKdnuK7JoQ/UVw='} 2023-10-18T21:04:16.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4926ed1810bf30e-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.275 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.275 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.276 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.276 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.276 DEBUG:boto:Data: 2023-10-18T21:04:16.277 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.277 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.277 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.278 DEBUG:boto:Params: {} 2023-10-18T21:04:16.278 DEBUG:boto:Token: None 2023-10-18T21:04:16.279 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.279 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae54454c83d4ed9e-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.283 DEBUG:boto:path=/bofrnf-64/obj116 2023-10-18T21:04:16.284 DEBUG:boto:auth_path=/bofrnf-64/obj116 2023-10-18T21:04:16.284 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.284 DEBUG:boto:Path: /bofrnf-64/obj116 2023-10-18T21:04:16.285 DEBUG:boto:Data: 2023-10-18T21:04:16.285 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.285 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.286 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.286 DEBUG:boto:Params: {} 2023-10-18T21:04:16.287 DEBUG:boto:Token: None 2023-10-18T21:04:16.287 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj116 2023-10-18T21:04:16.287 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:X7YHmXXN7wIgkn2eib7KcZBb7E4= 2023-10-18T21:04:16.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X7YHmXXN7wIgkn2eib7KcZBb7E4='} 2023-10-18T21:04:16.293 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052695e33bdab3229-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.293 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.294 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.294 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.295 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.295 DEBUG:boto:Data: 2023-10-18T21:04:16.295 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.296 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.296 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.296 DEBUG:boto:Params: {} 2023-10-18T21:04:16.297 DEBUG:boto:Token: None 2023-10-18T21:04:16.297 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.297 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b14524279f99536-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.301 DEBUG:boto:path=/bofrnf-64/obj145 2023-10-18T21:04:16.302 DEBUG:boto:auth_path=/bofrnf-64/obj145 2023-10-18T21:04:16.302 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.302 DEBUG:boto:Path: /bofrnf-64/obj145 2023-10-18T21:04:16.303 DEBUG:boto:Data: 2023-10-18T21:04:16.303 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.304 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.304 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.304 DEBUG:boto:Params: {} 2023-10-18T21:04:16.305 DEBUG:boto:Token: None 2023-10-18T21:04:16.305 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj145 2023-10-18T21:04:16.305 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3z1nEnAqaBZ8CCCt6JNnGUZ43a0= 2023-10-18T21:04:16.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3z1nEnAqaBZ8CCCt6JNnGUZ43a0='} 2023-10-18T21:04:16.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4edf970137237be-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:16.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:16.352 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:04:16.403 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:04:16.403 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:04:16.451 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T21:04:16.452 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-64 2023-10-18T21:04:16.461 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 25 2023-10-18T21:04:16.518 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.519 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.519 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.520 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.520 DEBUG:boto:Data: 2023-10-18T21:04:16.520 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.521 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.521 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.522 DEBUG:boto:Params: {} 2023-10-18T21:04:16.522 DEBUG:boto:Token: None 2023-10-18T21:04:16.523 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.523 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ff3c1121842b9af-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.527 DEBUG:boto:path=/bofrnf-64/obj31 2023-10-18T21:04:16.528 DEBUG:boto:auth_path=/bofrnf-64/obj31 2023-10-18T21:04:16.528 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.528 DEBUG:boto:Path: /bofrnf-64/obj31 2023-10-18T21:04:16.529 DEBUG:boto:Data: 2023-10-18T21:04:16.529 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.530 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.530 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.531 DEBUG:boto:Params: {} 2023-10-18T21:04:16.531 DEBUG:boto:Token: None 2023-10-18T21:04:16.531 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj31 2023-10-18T21:04:16.532 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KdTQMJjvA8AouOzapYe0slspVXE= 2023-10-18T21:04:16.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KdTQMJjvA8AouOzapYe0slspVXE='} 2023-10-18T21:04:16.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060b1f058816e6e82-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.538 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.538 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.539 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.539 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.540 DEBUG:boto:Data: 2023-10-18T21:04:16.540 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.540 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.541 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.541 DEBUG:boto:Params: {} 2023-10-18T21:04:16.542 DEBUG:boto:Token: None 2023-10-18T21:04:16.542 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.543 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb72a33d78af0973-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.547 DEBUG:boto:path=/bofrnf-64/obj62 2023-10-18T21:04:16.547 DEBUG:boto:auth_path=/bofrnf-64/obj62 2023-10-18T21:04:16.548 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.548 DEBUG:boto:Path: /bofrnf-64/obj62 2023-10-18T21:04:16.548 DEBUG:boto:Data: 2023-10-18T21:04:16.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.549 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.549 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.550 DEBUG:boto:Params: {} 2023-10-18T21:04:16.550 DEBUG:boto:Token: None 2023-10-18T21:04:16.551 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj62 2023-10-18T21:04:16.551 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KvTTm5bzOtibUCIPg67x0XCCgOg= 2023-10-18T21:04:16.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KvTTm5bzOtibUCIPg67x0XCCgOg='} 2023-10-18T21:04:16.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4d43e9e61690295-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.595 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.595 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.596 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.596 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.597 DEBUG:boto:Data: 2023-10-18T21:04:16.597 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.597 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.598 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.598 DEBUG:boto:Params: {} 2023-10-18T21:04:16.599 DEBUG:boto:Token: None 2023-10-18T21:04:16.599 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.599 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050a8e00abb777bdd-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.604 DEBUG:boto:path=/bofrnf-64/obj93 2023-10-18T21:04:16.604 DEBUG:boto:auth_path=/bofrnf-64/obj93 2023-10-18T21:04:16.605 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.605 DEBUG:boto:Path: /bofrnf-64/obj93 2023-10-18T21:04:16.605 DEBUG:boto:Data: 2023-10-18T21:04:16.606 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.606 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.607 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.607 DEBUG:boto:Params: {} 2023-10-18T21:04:16.607 DEBUG:boto:Token: None 2023-10-18T21:04:16.608 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj93 2023-10-18T21:04:16.608 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MxyOkgIVaie8N43mgQTZoRtK2zo= 2023-10-18T21:04:16.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MxyOkgIVaie8N43mgQTZoRtK2zo='} 2023-10-18T21:04:16.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003c9dc32640781fc-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.614 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.615 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.615 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.615 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.616 DEBUG:boto:Data: 2023-10-18T21:04:16.616 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.616 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.617 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.617 DEBUG:boto:Params: {} 2023-10-18T21:04:16.617 DEBUG:boto:Token: None 2023-10-18T21:04:16.618 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.618 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008418cd34e02f81b6-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.622 DEBUG:boto:path=/bofrnf-64/obj124 2023-10-18T21:04:16.623 DEBUG:boto:auth_path=/bofrnf-64/obj124 2023-10-18T21:04:16.623 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.624 DEBUG:boto:Path: /bofrnf-64/obj124 2023-10-18T21:04:16.624 DEBUG:boto:Data: 2023-10-18T21:04:16.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.625 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.625 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.626 DEBUG:boto:Params: {} 2023-10-18T21:04:16.626 DEBUG:boto:Token: None 2023-10-18T21:04:16.626 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj124 2023-10-18T21:04:16.627 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gxvNwd5khEDxaCdoE1jfJAlZGlk= 2023-10-18T21:04:16.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gxvNwd5khEDxaCdoE1jfJAlZGlk='} 2023-10-18T21:04:16.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b79e309d069d2dea-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.671 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.671 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.672 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.672 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.672 DEBUG:boto:Data: 2023-10-18T21:04:16.673 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.673 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.673 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.674 DEBUG:boto:Params: {} 2023-10-18T21:04:16.674 DEBUG:boto:Token: None 2023-10-18T21:04:16.675 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.675 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.678 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009319aff4aa795c9a-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.679 DEBUG:boto:path=/bofrnf-64/obj155 2023-10-18T21:04:16.679 DEBUG:boto:auth_path=/bofrnf-64/obj155 2023-10-18T21:04:16.680 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.680 DEBUG:boto:Path: /bofrnf-64/obj155 2023-10-18T21:04:16.681 DEBUG:boto:Data: 2023-10-18T21:04:16.681 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.681 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.682 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.682 DEBUG:boto:Params: {} 2023-10-18T21:04:16.682 DEBUG:boto:Token: None 2023-10-18T21:04:16.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj155 2023-10-18T21:04:16.683 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:iE9yV78dHB/DmxME41XExQnypKY= 2023-10-18T21:04:16.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iE9yV78dHB/DmxME41XExQnypKY='} 2023-10-18T21:04:16.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083843a762e259a8d-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:16.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:16.690 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:04:16.743 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:04:16.743 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:04:16.794 INFO:teuthology.orchestra.run.smithi082.stdout:tenant: 2023-10-18T21:04:16.794 INFO:teuthology.orchestra.run.smithi082.stdout:bucket name: bofrnf-64 2023-10-18T21:04:16.805 INFO:teuthology.orchestra.run.smithi082.stdout:total entries: 30 2023-10-18T21:04:16.866 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.867 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.867 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.868 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.868 DEBUG:boto:Data: 2023-10-18T21:04:16.869 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.869 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.869 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.870 DEBUG:boto:Params: {} 2023-10-18T21:04:16.870 DEBUG:boto:Token: None 2023-10-18T21:04:16.871 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.871 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa1b437e9f6df9bd-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.876 DEBUG:boto:path=/bofrnf-64/obj37 2023-10-18T21:04:16.876 DEBUG:boto:auth_path=/bofrnf-64/obj37 2023-10-18T21:04:16.877 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.877 DEBUG:boto:Path: /bofrnf-64/obj37 2023-10-18T21:04:16.877 DEBUG:boto:Data: 2023-10-18T21:04:16.878 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.878 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.879 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.879 DEBUG:boto:Params: {} 2023-10-18T21:04:16.879 DEBUG:boto:Token: None 2023-10-18T21:04:16.880 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj37 2023-10-18T21:04:16.880 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+Ca/zmIoD+dI2+oAdi98dG9Lj5w= 2023-10-18T21:04:16.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+Ca/zmIoD+dI2+oAdi98dG9Lj5w='} 2023-10-18T21:04:16.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e9c3ff44de601d1-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.923 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.923 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.924 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.924 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.924 DEBUG:boto:Data: 2023-10-18T21:04:16.925 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.925 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.925 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.926 DEBUG:boto:Params: {} 2023-10-18T21:04:16.926 DEBUG:boto:Token: None 2023-10-18T21:04:16.927 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.927 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037341a399e7bbd0a-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.932 DEBUG:boto:path=/bofrnf-64/obj74 2023-10-18T21:04:16.932 DEBUG:boto:auth_path=/bofrnf-64/obj74 2023-10-18T21:04:16.932 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.933 DEBUG:boto:Path: /bofrnf-64/obj74 2023-10-18T21:04:16.933 DEBUG:boto:Data: 2023-10-18T21:04:16.934 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.934 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.934 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.935 DEBUG:boto:Params: {} 2023-10-18T21:04:16.935 DEBUG:boto:Token: None 2023-10-18T21:04:16.936 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj74 2023-10-18T21:04:16.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:vLxMfXLH0Mmf8TNAN3geaG0E35c= 2023-10-18T21:04:16.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vLxMfXLH0Mmf8TNAN3geaG0E35c='} 2023-10-18T21:04:16.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aa850b1b5f27463-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.942 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.943 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:16.943 DEBUG:boto:Method: HEAD 2023-10-18T21:04:16.943 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:16.944 DEBUG:boto:Data: 2023-10-18T21:04:16.944 DEBUG:boto:Headers: {} 2023-10-18T21:04:16.945 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.945 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.945 DEBUG:boto:Params: {} 2023-10-18T21:04:16.946 DEBUG:boto:Token: None 2023-10-18T21:04:16.946 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/ 2023-10-18T21:04:16.947 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:04:16.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:04:16.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a29644f9e4bf98d-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.951 DEBUG:boto:path=/bofrnf-64/obj111 2023-10-18T21:04:16.951 DEBUG:boto:auth_path=/bofrnf-64/obj111 2023-10-18T21:04:16.952 DEBUG:boto:Method: PUT 2023-10-18T21:04:16.952 DEBUG:boto:Path: /bofrnf-64/obj111 2023-10-18T21:04:16.952 DEBUG:boto:Data: 2023-10-18T21:04:16.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:16.953 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:16.954 DEBUG:boto:Port: 8000 2023-10-18T21:04:16.954 DEBUG:boto:Params: {} 2023-10-18T21:04:16.954 DEBUG:boto:Token: None 2023-10-18T21:04:16.955 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:16 GMT /bofrnf-64/obj111 2023-10-18T21:04:16.955 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gF2QzDgB596Pwj69zv1+dq74/CU= 2023-10-18T21:04:16.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gF2QzDgB596Pwj69zv1+dq74/CU='} 2023-10-18T21:04:16.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5e195fc62f8062e-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:16.999 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:16.999 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:17.000 DEBUG:boto:Method: HEAD 2023-10-18T21:04:17.000 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:17.000 DEBUG:boto:Data: 2023-10-18T21:04:17.001 DEBUG:boto:Headers: {} 2023-10-18T21:04:17.001 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:17.001 DEBUG:boto:Port: 8000 2023-10-18T21:04:17.002 DEBUG:boto:Params: {} 2023-10-18T21:04:17.002 DEBUG:boto:Token: None 2023-10-18T21:04:17.003 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:17 GMT /bofrnf-64/ 2023-10-18T21:04:17.003 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q0hKGy4sF1ZCPfe1ADjrYonNX28= 2023-10-18T21:04:17.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q0hKGy4sF1ZCPfe1ADjrYonNX28=', 'Content-Length': '0'} 2023-10-18T21:04:17.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a404ecaf6e8cad03-0065304851-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:17.008 DEBUG:boto:path=/bofrnf-64/obj148 2023-10-18T21:04:17.008 DEBUG:boto:auth_path=/bofrnf-64/obj148 2023-10-18T21:04:17.009 DEBUG:boto:Method: PUT 2023-10-18T21:04:17.009 DEBUG:boto:Path: /bofrnf-64/obj148 2023-10-18T21:04:17.010 DEBUG:boto:Data: 2023-10-18T21:04:17.010 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:17.010 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:17.011 DEBUG:boto:Port: 8000 2023-10-18T21:04:17.011 DEBUG:boto:Params: {} 2023-10-18T21:04:17.011 DEBUG:boto:Token: None 2023-10-18T21:04:17.012 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:17 GMT /bofrnf-64/obj148 2023-10-18T21:04:17.012 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:T+fyg5esgpGNE1c/Li0P5q7Cfao= 2023-10-18T21:04:17.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T+fyg5esgpGNE1c/Li0P5q7Cfao='} 2023-10-18T21:04:17.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005357316cb5806c97-0065304851-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:17.018 DEBUG:boto:path=/bofrnf-64/ 2023-10-18T21:04:17.019 DEBUG:boto:auth_path=/bofrnf-64/ 2023-10-18T21:04:17.019 DEBUG:boto:Method: HEAD 2023-10-18T21:04:17.019 DEBUG:boto:Path: /bofrnf-64/ 2023-10-18T21:04:17.020 DEBUG:boto:Data: 2023-10-18T21:04:17.020 DEBUG:boto:Headers: {} 2023-10-18T21:04:17.021 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:17.021 DEBUG:boto:Port: 8000 2023-10-18T21:04:17.021 DEBUG:boto:Params: {} 2023-10-18T21:04:17.022 DEBUG:boto:Token: None 2023-10-18T21:04:17.022 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:04:17 GMT /bofrnf-64/ 2023-10-18T21:04:17.023 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Q0hKGy4sF1ZCPfe1ADjrYonNX28= 2023-10-18T21:04:17.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q0hKGy4sF1ZCPfe1ADjrYonNX28=', 'Content-Length': '0'} 2023-10-18T21:04:17.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e2e25f096c31456-0065304851-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:17.027 DEBUG:boto:path=/bofrnf-64/obj185 2023-10-18T21:04:17.027 DEBUG:boto:auth_path=/bofrnf-64/obj185 2023-10-18T21:04:17.028 DEBUG:boto:Method: PUT 2023-10-18T21:04:17.028 DEBUG:boto:Path: /bofrnf-64/obj185 2023-10-18T21:04:17.028 DEBUG:boto:Data: 2023-10-18T21:04:17.029 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:04:17.029 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:04:17.030 DEBUG:boto:Port: 8000 2023-10-18T21:04:17.030 DEBUG:boto:Params: {} 2023-10-18T21:04:17.030 DEBUG:boto:Token: None 2023-10-18T21:04:17.031 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 18 Oct 2023 21:04:17 GMT /bofrnf-64/obj185 2023-10-18T21:04:17.031 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:k9CLn46GT6lafE0QhpfXPu3rhMA= 2023-10-18T21:04:17.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:k9CLn46GT6lafE0QhpfXPu3rhMA='} 2023-10-18T21:04:17.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b925caa5dbc73e7-0065304851-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:04:17.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:17.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:04:17.076 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:04:17.127 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:04:17.127 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:04:17.158 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:17.156+0000 7efd078c3780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-18T21:04:17.159 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:17.158+0000 7efd078c3780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-18T21:04:17.180 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-18T21:04:17.181 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T21:04:17.186 INFO:tasks.rgw.c1.client.1:Started 2023-10-18T21:04:17.187 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-18T21:04:17.209 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T21:04:17.209 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T21:04:17.210 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T21:04:18.211 DEBUG:teuthology.orchestra.run.smithi082:> curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T21:04:18.233 INFO:teuthology.orchestra.run.smithi082.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T21:04:18.233 INFO:teuthology.orchestra.run.smithi082.stderr: Dload Upload Total Spent Left Speed 2023-10-18T21:04:18.234 INFO:teuthology.orchestra.run.smithi082.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T21:04:18.235 INFO:teuthology.orchestra.run.smithi082.stdout:anonymous 2023-10-18T21:04:18.235 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=bofrnf-64 2023-10-18T21:04:18.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:04:18.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:04:18.237 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:04:18.333 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:04:18.333 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:04:18.370 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:18.369+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:04:23.371 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:23.370+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:04:28.372 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:28.371+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:04:33.373 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:33.371+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:04:38.373 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:38.372+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:04:43.374 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:43.373+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:04:48.375 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:48.373+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:04:53.376 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:53.373+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:04:58.376 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:04:58.374+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:03.377 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:03.375+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:08.378 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:08.376+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:13.378 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:13.377+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:18.379 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:18.377+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:23.380 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:23.378+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:28.380 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:28.379+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:33.381 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:33.380+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:38.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:38.380+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:43.382 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:43.380+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:48.383 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:48.381+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:53.384 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:53.381+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:05:58.384 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:05:58.383+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:06:03.384 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:03.383+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:06:08.386 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:08.384+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:06:13.386 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:13.385+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:06:18.387 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:18.385+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:06:23.388 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:23.386+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:06:28.388 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:28.387+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:06:33.389 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:33.387+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:06:38.389 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:38.387+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:06:43.390 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:43.388+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:06:48.391 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:48.389+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:06:53.391 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:53.390+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:06:58.392 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:06:58.391+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:03.393 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:03.391+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:08.393 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:08.392+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:13.394 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:13.393+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:18.394 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:18.393+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:23.395 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:23.394+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:28.396 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:28.394+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:33.397 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:33.394+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:38.397 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:38.395+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:43.398 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:43.397+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:48.399 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:48.397+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:53.399 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:53.398+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:07:58.400 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:07:58.398+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:03.400 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:03.399+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:08.401 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:08.400+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:13.402 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:13.401+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:18.403 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:18.401+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:23.403 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:23.401+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:28.403 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:28.402+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:33.404 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:33.402+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:38.405 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:38.403+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:43.405 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:43.404+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:48.406 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:48.405+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:53.407 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:53.405+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:08:58.407 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:08:58.406+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:09:03.408 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:03.407+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:09:08.409 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:08.407+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:09:13.409 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:13.407+0000 7fdae3324780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-18T21:09:13.409 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:13.407+0000 7fdae3324780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:09:13.410 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:13.407+0000 7fdae3324780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:09:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-18T21:09:13.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2023-10-18T21:09:13.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:13.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:13.432 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:13.488 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:13.488 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:13.522 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:09:13.536 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:09:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:04:16.812634Z" 2023-10-18T21:09:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:09:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:09:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:09:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:13.540 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:09:13.541 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:13.541 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:09:13.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:13.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:13.543 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:13.592 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:13.593 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:13.621 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:13.619+0000 7f0016e6c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:13.622 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:13.620+0000 7f0016e6c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:13.634 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:13.635 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:13.635 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:13.635 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:13.636 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:13.636 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:13.636 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:13.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:13.637 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:13.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:13.637 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:13.637 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:13.637 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:13.638 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:13.638 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:13.638 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:13.638 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:13.638 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:13.639 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:13.639 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:13.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:13.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:13.639 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:13.640 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:13.640 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:13.640 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:13.640 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:13.641 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:13.641 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:13.641 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:13.641 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:13.641 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:13.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:13.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:13.642 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:13.642 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:13.643 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:13.643 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:13.643 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:13.643 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:13.643 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:13.644 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:13.644 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:13.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:13.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:13.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:13.645 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:13.645 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:13.645 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:13.645 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:13.645 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:13.646 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:13.646 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:13.646 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:13.646 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:13.647 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:13.647 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:13.647 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:13.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:13.647 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:13.648 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:13.648 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:13.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:13.648 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:13Z" 2023-10-18T21:09:13.648 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:13.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:13Z" } 2023-10-18T21:09:13.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:13.651 DEBUG:rgw_multi.tests:log_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:13.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:13.651 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:09:13.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:13.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:13.653 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:13.698 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:13.698 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:13.742 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[]} 2023-10-18T21:09:13.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:13.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:13.743 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:13.791 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:13.792 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:13.838 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:09:13.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:13.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:13.839 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:13.888 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:13.889 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:13.934 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-18T21:09:13.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:13.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:13.935 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:13.982 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:13.982 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:14.027 INFO:teuthology.orchestra.run.smithi082.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-10-18T21:09:14.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.028 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:14.075 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:14.076 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:14.120 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-18T21:09:14.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.121 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:14.174 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:14.174 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:14.221 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:09:14.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.223 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:14.277 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:14.277 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:14.322 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:09:14.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.323 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:14.375 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:14.375 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:14.421 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:09:14.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.422 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:14.469 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:14.470 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:14.514 INFO:teuthology.orchestra.run.smithi082.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-10-18T21:09:14.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.515 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:14.563 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:14.563 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:14.612 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:14.613 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:14.613 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 13, 2023-10-18T21:09:14.613 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:09:14.614 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:09:14.614 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:09:14.614 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:14.614 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:09:14.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.615 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:09:14.615 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:09:14.615 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:09:14.615 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:09:14.616 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:09:14.616 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:09:14.616 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.616 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:09:14.616 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:09:14.617 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:09:14.617 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:09:14.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.617 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:09:14.617 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:09:14.617 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:09:14.618 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:09:14.618 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.618 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:09:14.618 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:09:14.619 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:09:14.619 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:09:14.619 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:09:14.619 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:09:14.619 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:09:14.620 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:09:14.620 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:09:14.620 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:09:14.620 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:09:14.620 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.621 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:09:14.621 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:09:14.621 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:09:14.621 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:09:14.622 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.622 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:09:14.622 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:09:14.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:09:14.623 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:09:14.623 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:09:14.623 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:09:14.623 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:09:14.623 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:09:14.624 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:09:14.624 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:09:14.624 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:09:14.624 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.624 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.625 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.625 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:09:14.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.625 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:09:14.625 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:09:14.626 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:09:14.626 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:09:14.626 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.626 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.627 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.627 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:09:14.627 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:09:14.627 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:09:14.627 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [ 2023-10-18T21:09:14.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.628 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-group", 2023-10-18T21:09:14.628 INFO:teuthology.orchestra.run.smithi082.stdout: "data_flow": { 2023-10-18T21:09:14.628 INFO:teuthology.orchestra.run.smithi082.stdout: "symmetrical": [ 2023-10-18T21:09:14.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.629 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-flow1", 2023-10-18T21:09:14.629 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:09:14.629 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:09:14.629 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:09:14.630 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.630 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.630 INFO:teuthology.orchestra.run.smithi082.stdout: "directional": [ 2023-10-18T21:09:14.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.631 INFO:teuthology.orchestra.run.smithi082.stdout: "source_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:09:14.631 INFO:teuthology.orchestra.run.smithi082.stdout: "dest_zone": "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:09:14.631 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.631 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.632 INFO:teuthology.orchestra.run.smithi082.stdout: "pipes": [ 2023-10-18T21:09:14.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.632 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe", 2023-10-18T21:09:14.632 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:09:14.633 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:09:14.633 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:09:14.633 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:09:14.633 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:09:14.633 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.634 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:09:14.634 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:09:14.634 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:09:14.634 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:09:14.635 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:09:14.635 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.635 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:09:14.636 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:09:14.636 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:09:14.636 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:09:14.636 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.637 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:09:14.637 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:09:14.637 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:09:14.637 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:09:14.637 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.638 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.638 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "enabled" 2023-10-18T21:09:14.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.639 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.639 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:09:14.639 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:09:14.639 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.640 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:09:14.640 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:09:14.640 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:09:14.641 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:09:14.641 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:09:14.641 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:09:14.641 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.642 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:09:14.642 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:09:14.642 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:09:14.642 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:09:14.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.643 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:09:14.643 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:09:14.643 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:09:14.643 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:09:14.643 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.644 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:09:14.644 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:09:14.644 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:09:14.644 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:09:14.644 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:09:14.645 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:09:14.645 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:09:14.645 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:09:14.645 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:09:14.645 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:09:14.646 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:09:14.646 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.646 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:09:14.647 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:09:14.647 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:09:14.647 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:09:14.647 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.647 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:09:14.648 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:09:14.648 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:09:14.648 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:09:14.648 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:09:14.648 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:09:14.649 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:09:14.649 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:09:14.649 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:09:14.649 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:09:14.649 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:09:14.649 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.650 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.650 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.650 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:09:14.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.651 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:09:14.651 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:09:14.651 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:09:14.651 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:09:14.651 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.652 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.652 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.652 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:09:14.652 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:09:14.652 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:09:14.653 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:09:14.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.653 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:09:14.654 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:09:14.654 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.655 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.655 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:09:14.655 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:09:14.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.655 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:09:14.656 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:09:14.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.656 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:09:14.656 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:09:14.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.657 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:09:14.657 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:09:14.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:14.658 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:09:14.658 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:09:14.658 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.658 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:14.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.659 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:09:14.659 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:09:14.659 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:09:14.659 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:09:14.660 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:09:14.660 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:09:14.660 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:09:14.660 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:09:14.660 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:09:14.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.661 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:09:14.661 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:09:14.661 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:09:14.661 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:09:14.662 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:09:14.662 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:09:14.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.662 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:09:14.662 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:09:14.663 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:09:14.663 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:09:14.663 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:09:14.663 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:09:14.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.664 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:09:14.664 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:09:14.664 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:09:14.664 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:09:14.664 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:09:14.665 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:09:14.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.665 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:09:14.665 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:09:14.666 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:09:14.666 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:09:14.666 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:09:14.666 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:09:14.666 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:14.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:14.667 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:09:14.667 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:09:14.667 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:14.667 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:14.668 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-65 2023-10-18T21:09:14.668 DEBUG:boto:path=/bofrnf-65/ 2023-10-18T21:09:14.669 DEBUG:boto:auth_path=/bofrnf-65/ 2023-10-18T21:09:14.669 DEBUG:boto:Method: PUT 2023-10-18T21:09:14.670 DEBUG:boto:Path: /bofrnf-65/ 2023-10-18T21:09:14.670 DEBUG:boto:Data: 2023-10-18T21:09:14.671 DEBUG:boto:Headers: {} 2023-10-18T21:09:14.671 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:09:14.671 DEBUG:boto:Port: 8000 2023-10-18T21:09:14.672 DEBUG:boto:Params: {} 2023-10-18T21:09:14.672 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:09:14.673 DEBUG:boto:Token: None 2023-10-18T21:09:14.674 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:09:14 GMT /bofrnf-65/ 2023-10-18T21:09:14.674 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6nTOiV9/kb7BUd7FEBF1WOHPRrA= 2023-10-18T21:09:14.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:09:14 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6nTOiV9/kb7BUd7FEBF1WOHPRrA=', 'Content-Length': '0'} 2023-10-18T21:09:14.931 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e8419ad4b538531-006530497a-2095-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:09:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:09:14.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:14.932 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:15.000 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:15.001 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:15.111 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:09:15.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.113 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:15.188 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:15.189 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:15.259 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-18T21:09:15.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.260 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:15.318 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:15.318 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:15.374 INFO:teuthology.orchestra.run.smithi082.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-10-18T21:09:15.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.376 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:15.472 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-18T21:09:15.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.473 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:15.529 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:15.529 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:15.578 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:09:15.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.580 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:15.636 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:15.636 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:15.684 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:09:15.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.686 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:15.739 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:15.739 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:15.787 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-18T21:09:15.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:09:15.789 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:09:15.838 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:15.839 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:15.883 INFO:teuthology.orchestra.run.smithi082.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-10-18T21:09:15.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:15.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:15.885 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:15.934 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:15.935 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:15.963 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:09:15.976 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:09:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663355.773395_1608.1", 2023-10-18T21:09:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:09:15.773395Z" 2023-10-18T21:09:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:09:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:09:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:09:15.980 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:15.980 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:09:15.981 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:15.981 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:09:15.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:15.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:15.983 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:16.032 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:16.032 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:16.061 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:16.059+0000 7f32e98fd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:16.061 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:16.060+0000 7f32e98fd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:16.075 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:16.076 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:16.076 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:16.076 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:16.077 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:16.077 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:16.077 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:16.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:16.077 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:16.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:16.078 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:16.078 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:16.078 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:16.078 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:16.079 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:16.079 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:16.079 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:16.079 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:16.079 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:16.080 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:16.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:16.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:16.080 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:16.081 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:16.081 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:16.081 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:16.081 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:16.081 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:16.082 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:16.082 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:16.082 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:16.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:16.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:16.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:16.083 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:16.083 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:16.083 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:16.084 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:16.084 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:16.084 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:16.084 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:16.084 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:16.085 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:16.085 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:16.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:16.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:16.085 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:16.086 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:16.086 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:16.086 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:16.086 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:16.086 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:16.087 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:16.087 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:16.087 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:16.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:16.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:16.087 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:16.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:16.088 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:16.088 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:16.088 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:16.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:16.089 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:16Z" 2023-10-18T21:09:16.089 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:16.090 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:16Z" } 2023-10-18T21:09:16.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:16.091 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:16.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:16.091 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:09:21.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:21.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:21.094 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:21.151 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:21.151 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:21.183 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:21.182+0000 7f014257d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:21.184 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:21.182+0000 7f014257d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:21.198 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:21.198 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:21.199 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:21.199 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:21.199 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:21.199 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:21.200 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:21.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:21.200 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:21.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:21.200 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:21.201 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:21.201 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:21.201 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:21.201 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:21.201 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:21.202 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:21.202 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:21.202 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:21.202 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:21.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:21.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:21.203 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:21.203 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:21.203 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:21.203 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:21.204 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:21.204 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:21.204 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:21.204 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:21.204 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:21.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:21.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:21.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:21.205 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:21.205 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:21.206 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:21.206 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:21.206 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:21.206 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:21.206 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:21.207 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:21.207 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:21.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:21.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:21.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:21.208 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:21.208 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:21.208 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:21.208 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:21.208 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:21.209 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:21.209 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:21.209 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:21.209 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:21.209 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:21.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:21.210 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:21.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:21.210 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:21.210 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:21.211 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:21.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:21.211 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:21Z" 2023-10-18T21:09:21.211 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:21.212 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:21Z" } 2023-10-18T21:09:21.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:21.213 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:21.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:21.214 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:09:26.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:26.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:26.217 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:26.273 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:26.274 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:26.307 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:26.306+0000 7fee928a7780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:26.309 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:26.307+0000 7fee928a7780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:26.322 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:26.323 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:26.323 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:26.323 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:26.324 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:26.324 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:26.324 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:26.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:26.325 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:26.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:26.325 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:26.326 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:26.326 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:26.326 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:26.326 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:26.327 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:26.327 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:26.327 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:26.327 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:26.327 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:26.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:26.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:26.328 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:26.328 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:26.329 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:26.329 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:26.329 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:26.329 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:26.330 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:26.330 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:26.330 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:26.330 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:26.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:26.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:26.331 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:26.331 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:26.331 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:26.332 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:26.332 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:26.332 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:26.332 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:26.333 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:26.333 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:26.333 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:26.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:26.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:26.334 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:26.334 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:26.334 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:26.334 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:26.335 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:26.335 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:26.335 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:26.335 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:26.335 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:26.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:26.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:26.336 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:26.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:26.336 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:26.337 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:26.337 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:26.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:26.337 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:26Z" 2023-10-18T21:09:26.337 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:26.339 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:26Z" } 2023-10-18T21:09:26.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:26.340 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:26.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:26.340 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:09:31.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:31.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:31.343 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:31.400 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:31.400 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:31.433 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:31.431+0000 7efeeb119780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:31.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:31.432+0000 7efeeb119780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:31.447 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:31.447 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:31.448 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:31.448 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:31.448 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:31.448 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:31.448 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:31.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:31.449 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:31.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:31.449 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:31.449 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:31.450 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:31.450 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:31.450 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:31.450 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:31.451 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:31.451 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:31.451 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:31.451 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:31.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:31.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:31.452 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:31.452 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:31.452 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:31.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:31.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:31.455 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:31.455 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:31.456 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:31.456 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:31.456 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:31.457 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:31.457 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:31.457 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:31.457 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:31.458 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:31.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:31.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:31.458 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:31.459 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:31.459 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:31.459 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:31.459 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:31.460 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:31.460 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:31.460 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:31.460 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:31.461 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:31.461 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:31.461 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:31.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:31.461 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:31.462 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:31.462 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:31.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:31.462 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:31Z" 2023-10-18T21:09:31.463 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:31.464 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:31Z" } 2023-10-18T21:09:31.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:31.465 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:31.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:31.465 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:09:36.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:36.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:36.468 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:36.524 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:36.525 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:36.557 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:36.556+0000 7f044f1bd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:36.558 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:36.557+0000 7f044f1bd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:36.573 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:36.573 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:36.573 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:36.574 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:36.574 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:36.574 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:36.574 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:36.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:36.575 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:36.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:36.575 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:36.575 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:36.575 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:36.576 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:36.576 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:36.576 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:36.576 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:36.576 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:36.577 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:36.577 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:36.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:36.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:36.577 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:36.578 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:36.578 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:36.578 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:36.578 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:36.578 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:36.579 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:36.579 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:36.579 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:36.579 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:36.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:36.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:36.580 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:36.580 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:36.580 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:36.580 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:36.581 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:36.581 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:36.581 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:36.581 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:36.581 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:36.582 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:36.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:36.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:36.582 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:36.582 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:36.583 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:36.583 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:36.583 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:36.583 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:36.583 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:36.584 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:36.584 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:36.584 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:36.584 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:36.584 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:36.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:36.585 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:36.585 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:36.585 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:36.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:36.585 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:36Z" 2023-10-18T21:09:36.586 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:36.587 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:36Z" } 2023-10-18T21:09:36.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:36.588 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:36.588 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:36.588 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:09:41.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:41.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:41.591 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:41.648 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:41.649 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:41.681 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:41.680+0000 7f5ceecee780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:41.681 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:41.681+0000 7f5ceecee780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:41.695 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:41.696 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:41.696 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:41.696 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:41.696 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:41.696 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:41.697 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:41.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:41.697 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:41.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:41.697 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:41.698 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:41.698 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:41.698 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:41.698 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:41.698 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:41.699 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:41.699 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:41.699 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:41.699 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:41.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:41.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:41.700 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:41.700 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:41.700 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:41.700 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:41.701 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:41.701 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:41.701 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:41.701 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:41.701 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:41.702 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:41.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:41.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:41.702 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:41.702 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:41.703 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:41.703 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:41.703 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:41.703 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:41.704 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:41.704 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:41.704 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:41.704 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:41.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:41.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:41.705 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:41.705 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:41.705 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:41.705 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:41.705 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:41.706 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:41.706 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:41.706 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:41.706 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:41.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:41.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:41.707 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:41.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:41.707 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:41.707 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:41.708 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:41.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:41.708 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:41Z" 2023-10-18T21:09:41.708 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:41.709 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:41Z" } 2023-10-18T21:09:41.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:41.710 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:41.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:41.711 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:09:46.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:46.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:46.714 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:46.771 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:46.772 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:46.804 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:46.803+0000 7f5dad925780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:46.805 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:46.804+0000 7f5dad925780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:46.818 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:46.819 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:46.819 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:46.819 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:46.819 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:46.820 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:46.820 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:46.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:46.820 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:46.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:46.821 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:46.821 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:46.821 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:46.821 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:46.822 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:46.822 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:46.822 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:46.822 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:46.823 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:46.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:46.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:46.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:46.823 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:46.824 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:46.824 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:46.824 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:46.824 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:46.825 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:46.825 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:46.825 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:46.825 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:46.825 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:46.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:46.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:46.826 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:46.826 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:46.827 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:46.827 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:46.827 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:46.827 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:46.828 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:46.828 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:46.828 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:46.828 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:46.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:46.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:46.829 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:46.829 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:46.829 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:46.830 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:46.830 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:46.830 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:46.830 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:46.830 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:46.831 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:46.831 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:46.831 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:46.831 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:46.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:46.832 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:46.832 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:46.832 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:46.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:46.833 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:46Z" 2023-10-18T21:09:46.833 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:46.834 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:46Z" } 2023-10-18T21:09:46.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:46.835 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:46.835 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:46.836 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:09:51.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:51.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:51.838 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:51.898 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:51.898 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:51.930 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:51.929+0000 7fb78201d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:51.931 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:51.930+0000 7fb78201d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:51.945 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:51Z" 2023-10-18T21:09:51.959 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:51.960 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:51Z" } 2023-10-18T21:09:51.960 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:51.961 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:51.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:51.962 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:09:56.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:56.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:09:56.965 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:09:57.022 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:09:57.022 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:09:57.064 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:57.063+0000 7fe92dde0780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:57.065 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:09:57.064+0000 7fe92dde0780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:09:57.078 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:09:57.079 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:09:57.079 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:09:57.079 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:09:57.079 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:09:57.080 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:09:57.080 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:57.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:57.080 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:09:57.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:57.081 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:09:57.081 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:57.081 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:57.081 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:09:57.081 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:57.082 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:09:57.082 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:57.082 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:09:57.082 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:57.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:57.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:57.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:57.083 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:09:57.083 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:57.083 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:57.084 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:09:57.084 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:57.084 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:57.084 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:57.084 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:09:57.085 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:57.085 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:57.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:57.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:57.085 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:09:57.086 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:57.086 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:57.086 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:09:57.086 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:57.086 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:57.087 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:57.087 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:09:57.087 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:57.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:57.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:57.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:09:57.088 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:09:57.088 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:09:57.088 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:09:57.088 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:09:57.089 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:09:57.089 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:09:57.089 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:09:57.089 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:09:57.089 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:09:57.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:57.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:09:57.090 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:09:57.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:57.090 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:09:57.091 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:09:57.091 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:09:57.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:09:57.091 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:09:57Z" 2023-10-18T21:09:57.091 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:09:57.092 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:09:57Z" } 2023-10-18T21:09:57.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:09:57.094 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:57.094 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:09:57.095 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:10:02.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:02.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:02.098 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:02.153 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:02.154 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:02.188 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:02.187+0000 7f8271599780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:02.189 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:02.188+0000 7f8271599780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:02.202 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:10:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:10:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:02Z" 2023-10-18T21:10:02.216 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:02.217 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:02Z" } 2023-10-18T21:10:02.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:02.218 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:02.219 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:02.219 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:10:07.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:07.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:07.222 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:07.278 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:07.279 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:07.311 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:07.308+0000 7f617e677780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:07.311 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:07.310+0000 7f617e677780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:07.327 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:07.327 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:07.327 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:07.328 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:07.328 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:07.328 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:07.328 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:07.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:07.329 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:07.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:07.329 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:07.329 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:07.330 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:07.330 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:10:07.330 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:07.330 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:07.330 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:07.331 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:10:07.331 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:07.331 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:07.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:07.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:07.332 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:07.332 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:07.332 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:07.332 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:07.333 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:07.333 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:07.333 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:07.333 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:07.333 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:07.334 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:07.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:07.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:07.334 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:07.335 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:07.335 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:07.335 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:07.335 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:07.336 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:07.336 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:07.336 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:07.336 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:07.337 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:07.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:07.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:07.337 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:07.338 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:07.338 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:07.338 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:07.338 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:07.338 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:07.339 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:07.339 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:07.339 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:07.339 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:07.339 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:07.340 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:07.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:07.340 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:07.340 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:07.340 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:07.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:07.341 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:07Z" 2023-10-18T21:10:07.341 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:07.342 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:07Z" } 2023-10-18T21:10:07.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:07.343 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:07.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:07.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:10:12.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:12.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:12.347 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:12.401 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:12.402 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:12.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:12.432+0000 7fe944fd7780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:12.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:12.433+0000 7fe944fd7780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:12.449 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:12.449 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:12.450 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:12.450 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:12.450 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:12.450 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:12.450 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:12.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:12.451 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:12.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:12.451 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:12.452 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:12.452 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:12.452 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:10:12.452 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:12.452 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:12.453 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:12.453 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:10:12.453 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:12.453 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:12.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:12.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:12.454 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:12.454 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:12.454 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:12.455 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:12.455 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:12.455 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:12.455 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:12.455 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:12.456 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:12.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:12.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:12.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:12.456 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:12.457 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:12.457 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:12.457 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:12.457 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:12.457 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:12.458 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:12.458 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:12.458 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:12.458 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:12.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:12.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:12.459 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:12.459 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:12.459 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:12.460 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:12.460 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:12.460 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:12.460 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:12.460 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:12.461 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:12.461 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:12.461 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:12.461 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:12.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:12.462 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:12.462 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:12.462 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:12.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:12.462 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:12Z" 2023-10-18T21:10:12.463 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:12.464 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:12Z" } 2023-10-18T21:10:12.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:12.465 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:12.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:12.465 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:10:17.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:17.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:17.467 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:17.526 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:17.526 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:17.558 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:17.556+0000 7fa83600a780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:17.558 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:17.557+0000 7fa83600a780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:17.572 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:17.573 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:17.573 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:17.573 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:17.574 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:17.574 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:17.574 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:17.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:17.575 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:17.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:17.575 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:17.575 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:17.575 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:17.576 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:10:17.576 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:17.576 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:17.576 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:17.576 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:10:17.577 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:17.577 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:17.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:17.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:17.577 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:17.578 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:17.578 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:17.578 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:17.578 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:17.579 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:17.579 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:17.579 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:17.579 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:17.579 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:17.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:17.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:17.580 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:17.580 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:17.581 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:17.581 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:17.581 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:17.581 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:17.581 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:17.582 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:17.582 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:17.582 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:17.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:17.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:17.583 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:17.583 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:17.583 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:17.584 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:17.584 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:17.584 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:17.584 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:17.584 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:17.585 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:17.585 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:17.585 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:17.585 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:17.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:17.586 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:17.586 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:17.586 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:17.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:17.586 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:17Z" 2023-10-18T21:10:17.587 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:17.588 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:17Z" } 2023-10-18T21:10:17.588 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:17.589 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:17.589 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:17.589 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:10:22.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:22.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:22.592 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:22.650 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:22.650 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:22.683 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:22.682+0000 7fd7e542c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:22.684 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:22.683+0000 7fd7e542c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:22.697 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:22.697 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:22.698 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:22.698 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:22.698 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:22.698 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:22.699 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:22.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:22.699 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:22.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:22.699 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:22.700 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:22.700 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:22.700 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663056.812634_1500.1", 2023-10-18T21:10:22.700 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:22.701 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:22.701 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:22.701 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.812634Z", 2023-10-18T21:10:22.701 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:22.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:22.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:22.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:22.702 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:22.702 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:22.703 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:22.703 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:22.703 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:22.703 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:22.704 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:22.704 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:22.704 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:22.704 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:22.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:22.708 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:22.708 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:22.708 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:22.708 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:22.708 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:22.709 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:22.709 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:22.709 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:22.709 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:22.709 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:22.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:22.710 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:22.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:22.710 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:22.711 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:22.711 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:22.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:22.711 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:22Z" 2023-10-18T21:10:22.711 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:22.712 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663056.812634_1500.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:04:16.812634Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:22Z" } 2023-10-18T21:10:22.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663056.812634_1500.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.812634Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:22.713 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:22.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663056.812634_1500.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:22.714 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663056.812634_1500.1 2023-10-18T21:10:27.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:27.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:27.716 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:27.773 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:27.773 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:27.806 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:27.805+0000 7f9d6dfbf780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:27.807 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:27.806+0000 7f9d6dfbf780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:27.821 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:27.821 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:27.821 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:27.821 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:27.822 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:27.822 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:27.822 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:27.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:27.823 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:27.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:27.823 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:27.823 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:27.823 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:27.824 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663355.561440_1594.1", 2023-10-18T21:10:27.824 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:27.824 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:27.824 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:27.824 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:09:15.561440Z", 2023-10-18T21:10:27.825 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:27.825 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:27.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:27.825 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:27.825 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:27.826 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:27.826 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:27.826 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:27.826 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:27.827 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:27.827 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:27.827 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:27.827 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:27.827 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:27.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:27.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:27.828 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:27.828 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:27.828 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:27.829 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:27.829 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:27.829 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:27.829 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:27.829 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:27.829 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:27.830 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:27.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:27.830 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:27.830 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:27.830 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:27.831 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:27.831 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:27.831 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:27.831 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:27.831 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:27.832 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:27.832 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:27.832 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:27.832 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:27.833 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:27.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:27.833 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:27.833 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:27.833 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:27.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:27.834 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:27Z" 2023-10-18T21:10:27.834 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:27.835 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663355.561440_1594.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:09:15.561440Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:27Z" } 2023-10-18T21:10:27.836 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663355.561440_1594.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:09:15.561440Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:27.836 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:27.836 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663355.561440_1594.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:27.837 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663355.773395_1608.1 target=1_1697663355.561440_1594.1 2023-10-18T21:10:32.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:32.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:32.839 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:32.893 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:32.894 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:32.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:32.923+0000 7fdb3b11f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:32.925 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:32.924+0000 7fdb3b11f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:32.939 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:32.939 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:32.939 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:32.939 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:32.940 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:32.940 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:32.940 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:32.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:32.940 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:32.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:32.941 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:32.941 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:32.941 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:32.941 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663355.773395_1608.1", 2023-10-18T21:10:32.942 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:32.942 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:32.942 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:32.942 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:09:15.773395Z", 2023-10-18T21:10:32.943 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:32.943 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:32.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:32.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:32.943 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:32.944 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:32.944 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:32.944 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:32.944 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:32.944 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:32.945 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:32.945 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:32.945 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:32.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:32.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:32.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:32.946 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:32.946 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:32.946 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:32.946 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:32.947 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:32.947 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:32.947 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:32.947 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:32.947 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:32.948 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:32.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:32.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:32.948 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:32.948 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:32.949 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:32.949 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:32.949 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:32.949 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:32.949 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:32.950 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:32.950 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:32.950 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:32.950 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:32.950 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:32.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:32.951 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:32.951 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:32.951 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:32.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:32.952 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:32Z" 2023-10-18T21:10:32.952 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:32.952 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663355.773395_1608.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:09:15.773395Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:32Z" } 2023-10-18T21:10:32.953 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663355.773395_1608.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:09:15.773395Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:32.953 DEBUG:rgw_multi.tests:log_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:32.954 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:32.954 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:10:32.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:32.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:32.955 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:33.007 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:33.007 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:33.056 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:10:33.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:33.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:33.057 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:33.113 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:33.113 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:33.162 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:10:33.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:33.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:33.163 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:33.218 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:33.219 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:33.267 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-18T21:10:33.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:33.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:33.269 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:33.317 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:33.318 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:33.366 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:10:33.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:33.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'bofrnf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:33.367 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket bofrnf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:33.413 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:33.414 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:33.466 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[]} 2023-10-18T21:10:33.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:33.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:33.467 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:33.515 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:33.515 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:33.544 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:10:33.557 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:10:33.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:33.558 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:10:33.558 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:10:33.351663Z" 2023-10-18T21:10:33.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:33.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:33.559 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:33.559 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:10:33.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:33.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:33.559 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:33.560 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:10:33.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:33.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:33.560 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:33.560 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:10:33.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:33.561 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:10:33.562 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:33.562 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:10:33.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:33.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:33.563 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:33.613 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:33.614 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:33.642 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:33.641+0000 7f567a860780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:33.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:33.642+0000 7f567a860780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:33.655 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:33.656 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:33.656 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:33.656 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:33.656 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:33.656 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:33.657 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:33.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:33.657 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:33.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:33.657 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:33.658 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:33.658 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:33.658 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663355.773395_1608.1", 2023-10-18T21:10:33.658 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:33.658 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:33.659 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:33.659 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:09:15.773395Z", 2023-10-18T21:10:33.659 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:33.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:33.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:33.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:33.660 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:33.660 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:33.660 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:33.660 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:33.661 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:33.661 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:33.661 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:33.661 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:33.661 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:33.661 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:33.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:33.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:33.662 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:33.662 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:33.662 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:33.663 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:33.663 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:33.663 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:33.663 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:33.663 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:33.664 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:33.664 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:33.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:33.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:33.664 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:33.665 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:33.665 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:33.665 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:33.665 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:33.665 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:33.665 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:33.666 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:33.666 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:33.666 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:33.666 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:33.666 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:33.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:33.667 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:33.667 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:33.667 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:33.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:33.668 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:33Z" 2023-10-18T21:10:33.668 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:33.669 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663355.773395_1608.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:09:15.773395Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:33Z" } 2023-10-18T21:10:33.669 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663355.773395_1608.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:09:15.773395Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:33.669 DEBUG:rgw_multi.tests:log_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:33.670 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663355.773395_1608.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:33.670 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663433.351663_1634.1 target=1_1697663355.773395_1608.1 2023-10-18T21:10:38.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:38.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:38.672 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:38.729 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:38.729 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:38.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:38.760+0000 7f76225a4780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:38.762 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:38.761+0000 7f76225a4780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:38.775 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:38.776 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:38.776 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:38.776 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:38.776 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:38.777 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:38.777 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:38.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:38.777 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:38.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:38.777 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:38.778 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:38.778 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:38.778 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:10:38.778 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:38.779 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:38.779 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:38.779 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:10:38.779 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:38.779 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:38.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:38.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:38.780 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:38.780 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:38.780 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:38.780 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:38.781 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:38.781 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:38.781 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:38.781 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:38.781 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:38.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:38.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:38.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:38.782 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:38.782 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:38.782 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:38.783 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:38.783 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:38.783 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:38.783 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:38.784 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:38.784 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:38.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:38.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:38.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:38.784 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:38.785 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:38.785 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:38.785 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:38.785 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:38.786 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:38.786 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:38.786 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:38.786 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:38.786 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:38.787 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:38.787 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:38.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:38.787 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:38.787 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:38.788 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:38.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:38.788 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:38Z" 2023-10-18T21:10:38.788 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:38.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:38Z" } 2023-10-18T21:10:38.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:38.790 DEBUG:rgw_multi.tests:log_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:38.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:38.791 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:10:38.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:38.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:38.792 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:38.844 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:38.844 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:38.890 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:10:38.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:38.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:38.891 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:38.945 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:38.945 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:38.990 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:10:38.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:38.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:38.991 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:39.043 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:39.043 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:39.089 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-18T21:10:39.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.090 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:39.138 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:39.139 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:39.185 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:10:39.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.186 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:39.231 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:39.232 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:39.276 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[]} 2023-10-18T21:10:39.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.277 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:39.325 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:39.326 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:39.376 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:39.376 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:39.377 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 14, 2023-10-18T21:10:39.377 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:10:39.377 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:10:39.377 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:10:39.377 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:39.378 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:10:39.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.378 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:10:39.378 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:10:39.379 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:10:39.379 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:10:39.379 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:39.379 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:10:39.379 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.380 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:10:39.380 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:10:39.380 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:39.380 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:10:39.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.381 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:39.381 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:10:39.381 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:39.381 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:10:39.381 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.382 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:10:39.382 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:10:39.382 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:10:39.382 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:10:39.382 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:10:39.383 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:10:39.383 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:10:39.383 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:10:39.383 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:10:39.383 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:10:39.384 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:39.384 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.384 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:10:39.385 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:10:39.385 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:39.385 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:10:39.385 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.385 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:10:39.386 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:10:39.386 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:10:39.386 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:10:39.386 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:10:39.386 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:10:39.387 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:10:39.387 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:10:39.387 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:10:39.387 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:10:39.388 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:39.388 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.388 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.388 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:10:39.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.389 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:10:39.389 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:10:39.389 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:10:39.389 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:10:39.390 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.390 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.390 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.390 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:10:39.391 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:10:39.391 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:10:39.391 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:10:39.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.391 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:10:39.392 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:39.392 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.392 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:10:39.393 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:10:39.393 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:10:39.393 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:10:39.393 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:39.393 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:10:39.394 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.394 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:10:39.394 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:10:39.394 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:10:39.394 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:10:39.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.395 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:10:39.395 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:10:39.395 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:39.395 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:10:39.396 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.396 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:10:39.396 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:10:39.396 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:10:39.396 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:10:39.397 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:10:39.397 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:10:39.397 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:10:39.397 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:10:39.397 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:10:39.397 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:10:39.398 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:39.399 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.399 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:10:39.399 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:10:39.400 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:39.400 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:10:39.400 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.400 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:10:39.400 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:10:39.401 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:10:39.401 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:10:39.401 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:10:39.401 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:10:39.401 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:10:39.402 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:10:39.402 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:10:39.402 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:10:39.402 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:39.402 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.403 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.403 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.403 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:10:39.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.403 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:10:39.404 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:10:39.404 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:10:39.404 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:10:39.404 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.405 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.405 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.405 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:10:39.405 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:10:39.405 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:10:39.405 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:10:39.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.406 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:10:39.406 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:39.406 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.406 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.407 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:39.407 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:10:39.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.407 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:39.407 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:10:39.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.408 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:10:39.408 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:10:39.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.409 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:10:39.409 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:10:39.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.410 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:10:39.410 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:10:39.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.410 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.411 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:10:39.411 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:39.411 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:10:39.411 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:10:39.411 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:10:39.412 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:10:39.412 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:10:39.412 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:10:39.412 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:10:39.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.413 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:10:39.413 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:10:39.413 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:10:39.413 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:10:39.413 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:10:39.414 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:10:39.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.414 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:10:39.414 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:10:39.414 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:10:39.415 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:10:39.415 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:10:39.415 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:10:39.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.415 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:10:39.416 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:10:39.416 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:10:39.416 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:10:39.416 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:10:39.416 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:10:39.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.417 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:10:39.417 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:10:39.417 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:10:39.417 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:10:39.417 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:10:39.418 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:10:39.418 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.418 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:10:39.418 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:10:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:39.419 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:39.419 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2023-10-18T21:10:39.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:39.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:39.422 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:39.479 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:39.479 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:39.529 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:10:39.568 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:10:39.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.568 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:10:39.569 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:10:33.351663Z" 2023-10-18T21:10:39.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.569 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:10:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:39.570 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:10:39.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.571 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:39.571 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:10:39.571 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.572 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:10:39.573 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:39.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:10:39.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:39.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:39.575 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:39.646 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:39.646 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:39.734 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:39.733+0000 7f12a0e76780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:39.736 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:39.735+0000 7f12a0e76780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:39.777 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:39.777 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:39.778 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:39.778 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:39.778 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:39.778 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:39.779 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:39.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.779 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:39.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.780 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:39.780 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:39.780 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:39.780 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:10:39.780 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:39.781 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:39.781 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:39.781 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:10:39.781 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:39.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.782 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:39.783 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:39.783 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:39.783 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:39.783 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:39.784 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:39.784 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:39.784 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:39.784 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:39.785 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.785 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:39.785 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:39.785 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:39.786 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:39.786 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:39.786 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:39.786 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:39.787 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:39.787 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:39.787 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:39.788 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:39.788 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:39.788 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:39.788 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:39.789 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:39.789 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:39.789 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:39.789 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:39.789 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:39.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:39.790 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:39.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.790 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:39.791 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:39.791 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:39.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:39.791 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:39Z" 2023-10-18T21:10:39.791 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:39.792 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:39Z" } 2023-10-18T21:10:39.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:39.792 DEBUG:rgw_multi.tests:log_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:39.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:39.793 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:10:39.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.795 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:39.860 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:39.860 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:39.934 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[]} 2023-10-18T21:10:39.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:39.936 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:40.000 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:40.000 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:40.066 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:10:40.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.068 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:40.128 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:40.129 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:40.174 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-18T21:10:40.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.175 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:40.227 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:40.228 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:40.275 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:10:40.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.276 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:40.326 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:40.326 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:40.371 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:10:40.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.373 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:40.420 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:40.421 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:40.469 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:40.470 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:40.470 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 15, 2023-10-18T21:10:40.470 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:10:40.471 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:10:40.471 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:10:40.471 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:40.471 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:10:40.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.472 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:10:40.472 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:10:40.472 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:10:40.472 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:10:40.473 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:40.473 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:10:40.473 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.473 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:10:40.474 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:10:40.474 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:40.474 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:10:40.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.474 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:40.475 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:10:40.475 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:40.475 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:10:40.475 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.475 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:10:40.476 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:10:40.476 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:10:40.476 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:10:40.476 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:10:40.476 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:10:40.477 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:10:40.477 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:10:40.477 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:10:40.477 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:10:40.478 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:40.478 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.479 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:10:40.479 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:10:40.479 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:40.479 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:10:40.480 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.480 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:10:40.480 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:10:40.480 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:10:40.481 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:10:40.481 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:10:40.481 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:10:40.481 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:10:40.482 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:10:40.482 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:10:40.482 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:10:40.482 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:40.483 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.483 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.483 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:10:40.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.484 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:10:40.484 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:10:40.484 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:10:40.485 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:10:40.485 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.485 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.485 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.485 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:10:40.486 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:10:40.486 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:10:40.486 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [ 2023-10-18T21:10:40.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.487 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-group", 2023-10-18T21:10:40.487 INFO:teuthology.orchestra.run.smithi082.stdout: "data_flow": { 2023-10-18T21:10:40.487 INFO:teuthology.orchestra.run.smithi082.stdout: "symmetrical": [ 2023-10-18T21:10:40.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.488 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-flow1", 2023-10-18T21:10:40.488 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:10:40.488 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:40.488 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:10:40.489 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.489 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.489 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.490 INFO:teuthology.orchestra.run.smithi082.stdout: "pipes": [ 2023-10-18T21:10:40.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.490 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe", 2023-10-18T21:10:40.490 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:10:40.491 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:10:40.491 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:10:40.491 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:40.491 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:10:40.492 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.492 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:10:40.492 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:10:40.493 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:10:40.493 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:40.493 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:10:40.493 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.494 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:10:40.494 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:10:40.494 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:10:40.495 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:10:40.495 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.495 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:10:40.496 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:10:40.496 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:10:40.496 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:10:40.496 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.497 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.497 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.497 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "enabled" 2023-10-18T21:10:40.497 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.498 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.498 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:10:40.498 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:40.499 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.499 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:10:40.499 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:10:40.500 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:10:40.500 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:10:40.500 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:40.500 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:10:40.501 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.501 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:10:40.501 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:10:40.501 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:10:40.502 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:10:40.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.502 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:10:40.502 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:10:40.502 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:40.503 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:10:40.503 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.503 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:10:40.503 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:10:40.504 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:10:40.504 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:10:40.504 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:10:40.504 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:10:40.504 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:10:40.505 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:10:40.505 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:10:40.505 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:10:40.505 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:40.505 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.506 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:10:40.506 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:10:40.507 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:10:40.507 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:10:40.507 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.507 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:10:40.507 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:10:40.508 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:10:40.508 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:10:40.508 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:10:40.508 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:10:40.509 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:10:40.509 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:10:40.509 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:10:40.509 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:10:40.509 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:40.510 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.510 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.510 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.510 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:10:40.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.511 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:10:40.511 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:10:40.511 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:10:40.512 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:10:40.512 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.512 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.512 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.512 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:10:40.513 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:10:40.513 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:10:40.513 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:10:40.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.514 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:10:40.514 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:10:40.514 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.514 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.514 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:10:40.515 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:10:40.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.515 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:40.515 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:10:40.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.516 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:10:40.516 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:10:40.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.518 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:10:40.518 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:10:40.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.519 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:10:40.519 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:10:40.520 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.520 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:40.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.520 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:10:40.520 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:10:40.521 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:10:40.521 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:10:40.521 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:10:40.521 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:10:40.522 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:10:40.522 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:10:40.522 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:10:40.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.523 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:10:40.523 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:10:40.523 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:10:40.523 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:10:40.524 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:10:40.524 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:10:40.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.524 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:10:40.524 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:10:40.525 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:10:40.525 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:10:40.525 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:10:40.525 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:10:40.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.526 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:10:40.526 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:10:40.526 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:10:40.527 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:10:40.527 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:10:40.527 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:10:40.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.528 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:10:40.528 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:10:40.528 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:10:40.528 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:10:40.528 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:10:40.529 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:10:40.529 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.529 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:10:40.530 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:10:40.530 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:40.530 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:40.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:10:40.532 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:10:40.587 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:40.587 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:40.706 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:10:40.706 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-66 2023-10-18T21:10:40.707 DEBUG:boto:path=/bofrnf-66/ 2023-10-18T21:10:40.707 DEBUG:boto:auth_path=/bofrnf-66/ 2023-10-18T21:10:40.708 DEBUG:boto:Method: PUT 2023-10-18T21:10:40.708 DEBUG:boto:Path: /bofrnf-66/ 2023-10-18T21:10:40.708 DEBUG:boto:Data: 2023-10-18T21:10:40.709 DEBUG:boto:Headers: {} 2023-10-18T21:10:40.709 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:10:40.709 DEBUG:boto:Port: 8000 2023-10-18T21:10:40.710 DEBUG:boto:Params: {} 2023-10-18T21:10:40.710 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:10:40.711 DEBUG:boto:Token: None 2023-10-18T21:10:40.711 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:10:40 GMT /bofrnf-66/ 2023-10-18T21:10:40.712 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pgrX/8LHFcUa91+FIeES4UaeL0M= 2023-10-18T21:10:40.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:10:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pgrX/8LHFcUa91+FIeES4UaeL0M=', 'Content-Length': '0'} 2023-10-18T21:10:40.743 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005470fa06b89b383c-00653049d0-2216-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:10:40.744 DEBUG:boto:path=/bofrnf-66/ 2023-10-18T21:10:40.744 DEBUG:boto:auth_path=/bofrnf-66/ 2023-10-18T21:10:40.744 DEBUG:boto:Method: HEAD 2023-10-18T21:10:40.745 DEBUG:boto:Path: /bofrnf-66/ 2023-10-18T21:10:40.745 DEBUG:boto:Data: 2023-10-18T21:10:40.745 DEBUG:boto:Headers: {} 2023-10-18T21:10:40.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:10:40.746 DEBUG:boto:Port: 8000 2023-10-18T21:10:40.746 DEBUG:boto:Params: {} 2023-10-18T21:10:40.747 DEBUG:boto:Token: None 2023-10-18T21:10:40.747 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:10:40 GMT /bofrnf-66/ 2023-10-18T21:10:40.748 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ugdW3KxvO3aQ2CADLzgVCBe8QoA= 2023-10-18T21:10:40.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:10:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ugdW3KxvO3aQ2CADLzgVCBe8QoA=', 'Content-Length': '0'} 2023-10-18T21:10:40.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc49c2d0ac0611bd-00653049d0-2216-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:10:40.752 DEBUG:boto:path=/bofrnf-66/obj1 2023-10-18T21:10:40.752 DEBUG:boto:auth_path=/bofrnf-66/obj1 2023-10-18T21:10:40.753 DEBUG:boto:Method: PUT 2023-10-18T21:10:40.753 DEBUG:boto:Path: /bofrnf-66/obj1 2023-10-18T21:10:40.753 DEBUG:boto:Data: 2023-10-18T21:10:40.754 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:10:40.754 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:10:40.755 DEBUG:boto:Port: 8000 2023-10-18T21:10:40.755 DEBUG:boto:Params: {} 2023-10-18T21:10:40.755 DEBUG:boto:Token: None 2023-10-18T21:10:40.756 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:10:40 GMT /bofrnf-66/obj1 2023-10-18T21:10:40.756 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pLu3BSRCVUXUkN/48R5zzcShQe4= 2023-10-18T21:10:40.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:10:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pLu3BSRCVUXUkN/48R5zzcShQe4='} 2023-10-18T21:10:40.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d491b890039c259-00653049d0-2216-a1'), ('Date', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:10:40.791 INFO:rgw_multi.tests:create bucket zone=a2 name=bofrnf-67 2023-10-18T21:10:40.791 DEBUG:boto:path=/bofrnf-67/ 2023-10-18T21:10:40.792 DEBUG:boto:auth_path=/bofrnf-67/ 2023-10-18T21:10:40.792 DEBUG:boto:Method: PUT 2023-10-18T21:10:40.793 DEBUG:boto:Path: /bofrnf-67/ 2023-10-18T21:10:40.793 DEBUG:boto:Data: 2023-10-18T21:10:40.794 DEBUG:boto:Headers: {} 2023-10-18T21:10:40.794 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:10:40.794 DEBUG:boto:Port: 8001 2023-10-18T21:10:40.795 DEBUG:boto:Params: {} 2023-10-18T21:10:40.795 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:10:40.796 DEBUG:boto:Token: None 2023-10-18T21:10:40.796 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:10:40 GMT /bofrnf-67/ 2023-10-18T21:10:40.797 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tqccay/g5AbaKTiLqDv5rLHETFw= 2023-10-18T21:10:40.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:10:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tqccay/g5AbaKTiLqDv5rLHETFw=', 'Content-Length': '0'} 2023-10-18T21:10:40.837 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ab718fc2ab8349b-00653049d0-2218-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:10:40.837 DEBUG:boto:path=/bofrnf-67/ 2023-10-18T21:10:40.838 DEBUG:boto:auth_path=/bofrnf-67/ 2023-10-18T21:10:40.838 DEBUG:boto:Method: HEAD 2023-10-18T21:10:40.839 DEBUG:boto:Path: /bofrnf-67/ 2023-10-18T21:10:40.839 DEBUG:boto:Data: 2023-10-18T21:10:40.839 DEBUG:boto:Headers: {} 2023-10-18T21:10:40.840 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:10:40.840 DEBUG:boto:Port: 8001 2023-10-18T21:10:40.840 DEBUG:boto:Params: {} 2023-10-18T21:10:40.841 DEBUG:boto:Token: None 2023-10-18T21:10:40.841 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:10:40 GMT /bofrnf-67/ 2023-10-18T21:10:40.842 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z+bpZKd06uC51G8iXK6igymvLe0= 2023-10-18T21:10:40.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:10:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z+bpZKd06uC51G8iXK6igymvLe0=', 'Content-Length': '0'} 2023-10-18T21:10:40.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d57a7cec63588d53-00653049d0-2218-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:10:40.846 DEBUG:boto:path=/bofrnf-67/obj2 2023-10-18T21:10:40.847 DEBUG:boto:auth_path=/bofrnf-67/obj2 2023-10-18T21:10:40.847 DEBUG:boto:Method: PUT 2023-10-18T21:10:40.847 DEBUG:boto:Path: /bofrnf-67/obj2 2023-10-18T21:10:40.848 DEBUG:boto:Data: 2023-10-18T21:10:40.848 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:10:40.849 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:10:40.849 DEBUG:boto:Port: 8001 2023-10-18T21:10:40.849 DEBUG:boto:Params: {} 2023-10-18T21:10:40.850 DEBUG:boto:Token: None 2023-10-18T21:10:40.850 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:10:40 GMT /bofrnf-67/obj2 2023-10-18T21:10:40.851 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tcEn5ycj8hfb7y6Hc8atN0KIHdA= 2023-10-18T21:10:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:10:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tcEn5ycj8hfb7y6Hc8atN0KIHdA='} 2023-10-18T21:10:40.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078ef7aca0110a0ae-00653049d0-2218-a2'), ('Date', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:10:40.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:40.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:40.862 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:40.924 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:40.924 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:40.971 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:10:40.989 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:10:40.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.990 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663440.822914_1716.1", 2023-10-18T21:10:40.990 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:10:40.822914Z" 2023-10-18T21:10:40.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.991 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:40.991 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:10:40.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.991 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:40.992 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:10:40.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:40.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:40.992 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:40.992 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:10:40.992 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:40.993 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:10:40.994 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:40.994 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:10:40.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:40.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:40.995 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:41.053 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:41.053 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:41.082 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:41.081+0000 7f47f4820780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:41.083 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:41.082+0000 7f47f4820780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:41.095 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:10:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:10:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:41.102 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:41.102 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:41.102 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:41.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:41.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:41.103 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:41.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:41.105 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:41.106 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:41.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:41.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:41.107 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:41.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:41.108 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:41Z" 2023-10-18T21:10:41.108 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:41.109 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:41Z" } 2023-10-18T21:10:41.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:41.110 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:41.111 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:41.111 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:10:46.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:46.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:46.113 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:46.170 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:46.171 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:46.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:46.203+0000 7f40cf836780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:46.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:46.205+0000 7f40cf836780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:46.219 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:46.219 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:46.220 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:46.220 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:46.220 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:46.220 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:46.221 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:46.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:46.221 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:46.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:46.221 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:46.222 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:46.222 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:46.222 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:10:46.222 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:46.222 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:46.223 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:46.223 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:10:46.223 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:46.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:46.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:46.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:46.224 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:46.224 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:46.224 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:46.225 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:46.225 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:46.225 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:46.225 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:46.225 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:46.225 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:46.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:46.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:46.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:46.226 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:46.226 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:46.227 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:46.227 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:46.227 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:46.227 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:46.227 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:46.228 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:46.228 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:46.228 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:46.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:46.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:46.229 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:46.229 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:46.229 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:46.229 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:46.229 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:46.230 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:46.230 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:46.230 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:46.230 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:46.230 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:46.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:46.231 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:46.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:46.231 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:46.231 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:46.232 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:46.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:46.232 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:46Z" 2023-10-18T21:10:46.232 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:46.233 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:46Z" } 2023-10-18T21:10:46.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:46.234 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:46.235 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:46.235 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:10:51.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:51.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:51.237 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:51.295 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:51.295 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:51.327 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:51.326+0000 7f27a9eda780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:51.328 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:51.327+0000 7f27a9eda780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:51.342 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:51.342 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:51.342 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:51.342 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:51.343 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:51.343 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:51.343 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:51.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:51.344 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:51.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:51.344 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:51.344 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:51.344 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:51.345 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:10:51.345 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:51.345 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:51.345 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:51.346 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:10:51.346 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:51.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:51.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:51.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:51.347 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:51.347 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:51.347 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:51.347 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:51.347 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:51.348 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:51.348 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:51.348 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:51.348 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:51.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:51.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:51.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:51.349 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:51.349 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:51.350 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:51.350 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:51.350 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:51.350 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:51.351 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:51.351 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:51.351 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:51.351 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:51.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:51.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:51.352 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:51.352 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:51.352 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:51.353 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:51.353 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:51.353 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:51.353 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:51.354 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:51.354 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:51.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:51.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:51.354 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:51.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:51.355 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:51.355 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:51.355 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:51.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:51.356 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:51Z" 2023-10-18T21:10:51.356 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:51.357 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:51Z" } 2023-10-18T21:10:51.357 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:51.358 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:51.358 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:51.359 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:10:56.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:56.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:10:56.361 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:10:56.418 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:10:56.418 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:10:56.450 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:56.449+0000 7f45c180d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:56.450 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:10:56.450+0000 7f45c180d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:10:56.478 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:10:56.479 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:10:56.479 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:10:56.479 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:10:56.479 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:10:56.479 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:10:56.480 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:56.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:56.480 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:10:56.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:56.481 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:10:56.481 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:56.481 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:56.481 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:10:56.481 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:56.482 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:10:56.482 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:56.482 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:10:56.482 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:56.482 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:56.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:56.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:56.483 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:10:56.483 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:56.484 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:56.484 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:10:56.484 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:56.484 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:56.484 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:56.485 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:10:56.485 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:56.485 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:56.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:56.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:56.485 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:10:56.486 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:56.486 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:56.486 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:10:56.486 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:56.486 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:56.487 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:56.487 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:10:56.487 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:56.487 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:56.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:56.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:10:56.488 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:10:56.488 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:10:56.488 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:10:56.488 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:10:56.489 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:10:56.489 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:10:56.489 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:10:56.489 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:10:56.489 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:10:56.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:56.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:10:56.490 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:10:56.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:56.490 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:10:56.491 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:10:56.491 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:10:56.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:10:56.491 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:10:56Z" 2023-10-18T21:10:56.491 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:10:56.492 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:10:56Z" } 2023-10-18T21:10:56.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:10:56.493 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:56.494 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:10:56.494 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:01.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:01.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:01.497 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:01.554 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:01.555 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:01.587 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:01.585+0000 7fad3af2d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:01.587 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:01.586+0000 7fad3af2d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:01.601 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:01.601 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:01.601 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:01.602 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:01.602 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:01.602 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:01.602 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:01.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:01.603 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:01.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:01.603 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:01.603 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:01.603 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:01.604 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:01.604 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:01.604 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:01.604 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:01.604 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:01.605 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:01.605 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:01.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:01.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:01.605 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:01.605 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:01.606 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:01.606 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:01.606 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:01.606 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:01.606 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:01.607 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:01.607 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:01.607 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:01.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:01.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:01.608 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:01.608 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:01.608 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:01.608 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:01.608 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:01.608 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:01.609 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:01.609 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:01.609 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:01.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:01.611 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:01.611 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:01.611 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:01.611 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:01.611 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:01.612 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:01.612 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:01.612 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:01.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:01.612 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:01.613 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:01.613 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:01.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:01.613 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:01Z" 2023-10-18T21:11:01.613 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:01.614 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:01Z" } 2023-10-18T21:11:01.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:01.615 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:01.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:01.616 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:06.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:06.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:06.618 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:06.676 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:06.676 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:06.708 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:06.706+0000 7f4c028d2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:06.709 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:06.707+0000 7f4c028d2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:06.722 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:06.722 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:06.723 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:06.723 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:06.723 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:06.723 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:06.724 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:06.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:06.724 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:06.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:06.725 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:06.725 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:06.725 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:06.725 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:06.725 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:06.725 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:06.726 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:06.726 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:06.726 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:06.726 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:06.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:06.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:06.727 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:06.727 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:06.727 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:06.728 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:06.728 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:06.728 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:06.728 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:06.728 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:06.729 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:06.729 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:06.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:06.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:06.729 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:06.730 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:06.730 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:06.730 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:06.730 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:06.730 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:06.731 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:06.731 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:06.731 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:06.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:06.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:06.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:06.732 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:06.732 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:06.732 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:06.732 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:06.732 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:06.733 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:06.733 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:06.733 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:06.733 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:06.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:06.734 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:06.734 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:06.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:06.734 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:06.734 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:06.735 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:06.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:06.735 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:06Z" 2023-10-18T21:11:06.735 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:06.736 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:06Z" } 2023-10-18T21:11:06.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:06.737 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:06.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:06.738 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:11.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:11.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:11.740 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:11.796 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:11.796 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:11.826 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:11.825+0000 7f890b9ee780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:11.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:11.826+0000 7f890b9ee780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:11.841 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:11.841 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:11.842 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:11.842 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:11.842 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:11.842 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:11.842 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:11.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:11.843 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:11.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:11.843 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:11.844 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:11.844 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:11.844 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:11.844 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:11.844 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:11.845 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:11.845 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:11.845 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:11.845 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:11.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:11.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:11.846 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:11.846 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:11.846 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:11.846 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:11.847 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:11.847 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:11.847 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:11.847 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:11.847 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:11.848 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:11.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:11.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:11.848 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:11.849 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:11.849 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:11.849 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:11.849 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:11.849 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:11.850 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:11.850 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:11.850 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:11.850 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:11.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:11.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:11.851 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:11.851 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:11.851 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:11.851 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:11.852 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:11.852 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:11.852 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:11.852 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:11.852 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:11.853 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:11.853 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:11.853 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:11.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:11.853 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:11.854 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:11.854 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:11.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:11.854 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:11Z" 2023-10-18T21:11:11.854 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:11.855 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:11Z" } 2023-10-18T21:11:11.856 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:11.856 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:11.856 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:11.857 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:16.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:16.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:16.860 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:16.916 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:16.917 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:16.949 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:16.948+0000 7fc8bd7e2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:16.950 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:16.949+0000 7fc8bd7e2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:16.963 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:16.963 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:16.964 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:16.964 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:16.964 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:16.964 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:16.964 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:16.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:16.965 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:16.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:16.965 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:16.965 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:16.965 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:16.966 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:16.966 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:16.966 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:16.966 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:16.967 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:16.967 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:16.967 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:16.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:16.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:16.967 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:16.968 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:16.968 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:16.968 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:16.968 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:16.968 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:16.969 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:16.969 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:16.969 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:16.969 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:16.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:16.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:16.970 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:16.970 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:16.970 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:16.970 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:16.970 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:16.971 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:16.971 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:16.971 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:16.971 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:16.972 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:16.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:16.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:16.972 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:16.972 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:16.972 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:16.973 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:16.973 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:16.973 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:16.973 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:16.973 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:16.974 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:16.974 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:16.974 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:16.974 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:16.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:16.975 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:16.975 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:16.975 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:16.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:16.975 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:16Z" 2023-10-18T21:11:16.976 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:16.976 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:16Z" } 2023-10-18T21:11:16.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:16.977 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:16.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:16.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:21.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:21.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:21.981 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:22.038 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:22.038 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:22.073 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:22.072+0000 7f7df090a780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:22.074 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:22.073+0000 7f7df090a780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:22.088 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:22.088 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:22.088 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:22.089 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:22.089 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:22.089 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:22.089 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:22.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:22.090 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:22.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:22.090 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:22.090 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:22.091 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:22.091 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:22.091 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:22.091 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:22.092 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:22.092 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:22.092 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:22.092 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:22.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:22.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:22.094 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:22.094 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:22.095 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:22.095 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:22.095 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:22.095 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:22.096 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:22.096 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:22.096 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:22.097 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:22.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:22.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:22.097 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:22.098 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:22.098 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:22.098 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:22.099 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:22.099 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:22.099 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:22.099 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:22.100 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:22.100 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:22.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:22.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:22.100 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:22.101 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:22.101 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:22.101 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:22.101 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:22.101 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:22.102 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:22.102 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:22.102 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:22.102 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:22.103 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:22.103 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:22.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:22.103 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:22.103 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:22.104 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:22.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:22.104 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:22Z" 2023-10-18T21:11:22.104 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:22.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:22Z" } 2023-10-18T21:11:22.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:22.106 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:22.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:22.107 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:27.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:27.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:27.109 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:27.166 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:27.167 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:27.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:27.198+0000 7f301f253780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:27.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:27.199+0000 7f301f253780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:27.213 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:27.214 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:27.214 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:27.214 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:27.214 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:27.215 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:27.215 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:27.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:27.215 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:27.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:27.216 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:27.216 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:27.216 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:27.216 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:27.216 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:27.217 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:27.217 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:27.217 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:27.217 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:27.217 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:27.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:27.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:27.218 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:27.218 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:27.218 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:27.219 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:27.219 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:27.219 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:27.219 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:27.220 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:27.220 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:27.220 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:27.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:27.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:27.220 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:27.221 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:27.221 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:27.221 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:27.221 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:27.221 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:27.222 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:27.222 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:27.222 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:27.222 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:27.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:27.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:27.223 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:27.223 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:27.223 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:27.223 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:27.223 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:27.224 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:27.224 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:27.224 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:27.224 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:27.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:27.225 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:27.225 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:27.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:27.225 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:27.225 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:27.225 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:27.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:27.226 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:27Z" 2023-10-18T21:11:27.226 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:27.227 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:27Z" } 2023-10-18T21:11:27.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:27.228 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:27.228 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:27.229 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:32.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:32.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:32.231 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:32.288 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:32.288 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:32.320 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:32.319+0000 7f1254d88780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:32.320 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:32.320+0000 7f1254d88780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:32.334 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:32Z" 2023-10-18T21:11:32.348 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:32.349 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:32Z" } 2023-10-18T21:11:32.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:32.350 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:32.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:32.351 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:37.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:37.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:37.353 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:37.411 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:37.412 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:37.444 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:37.443+0000 7f7d39ce3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:37.444 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:37.444+0000 7f7d39ce3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:37.459 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:37.459 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:37.459 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:37.459 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:37.460 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:37.460 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:37.460 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:37.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:37.461 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:37.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:37.461 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:37.461 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:37.462 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:37.462 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:37.462 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:37.462 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:37.463 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:37.463 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:37.463 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:37.463 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:37.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:37.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:37.464 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:37.464 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:37.464 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:37.464 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:37.465 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:37.465 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:37.465 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:37.465 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:37.466 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:37.466 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:37.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:37.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:37.467 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:37.467 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:37.467 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:37.467 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:37.467 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:37.468 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:37.468 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:37.468 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:37.468 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:37.468 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:37.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:37.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:37.469 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:37.469 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:37.470 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:37.470 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:37.470 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:37.471 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:37.471 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:37.471 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:37.471 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:37.472 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:37.472 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:37.472 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:37.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:37.472 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:37.472 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:37.473 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:37.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:37.473 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:37Z" 2023-10-18T21:11:37.473 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:37.474 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:37Z" } 2023-10-18T21:11:37.475 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:37.476 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:37.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:37.477 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:42.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:42.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:42.480 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:42.536 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:42.569 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:42.568+0000 7fd016f7f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:42.570 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:42.569+0000 7fd016f7f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:42.585 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:42.585 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:42.586 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:42.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:42.587 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:42.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:42.587 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:42.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:42.590 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:42.590 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:42.590 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:42.591 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:42.592 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:42.593 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:42.593 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:42.593 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:42.594 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:42.594 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:42.594 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:42.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:42.596 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:42.597 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:42.598 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:42.599 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:42.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:42.599 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:42Z" 2023-10-18T21:11:42.599 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:42.600 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:42Z" } 2023-10-18T21:11:42.601 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:42.601 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:42.602 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:42.602 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:47.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:47.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:47.605 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:47.661 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:47.662 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:47.694 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:47.693+0000 7f8ea4e12780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:47.695 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:47.694+0000 7f8ea4e12780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:47.709 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:47Z" 2023-10-18T21:11:47.723 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:47.724 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:47Z" } 2023-10-18T21:11:47.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:47.725 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:47.726 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:47.726 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:52.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:52.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:52.728 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:52.786 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:52.786 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:52.818 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:52.816+0000 7f4392aac780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:52.819 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:52.817+0000 7f4392aac780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:52.833 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:52.833 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:52.833 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:52.833 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:52.834 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:52.834 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:52.834 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:52.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:52.835 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:52.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:52.835 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:52.835 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:52.835 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:52.836 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:52.836 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:52.836 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:52.836 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:52.837 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:52.837 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:52.837 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:52.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:52.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:52.838 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:52.838 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:52.838 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:52.838 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:52.839 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:52.839 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:52.839 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:52.839 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:52.840 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:52.840 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:52.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:52.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:52.840 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:52.841 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:52.841 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:52.842 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:52.842 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:52.842 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:52.842 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:52.843 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:52.843 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:52.843 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:52.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:52.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:52.844 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:52.844 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:52.845 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:52.845 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:52.846 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:52.846 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:52.846 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:52.847 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:52.847 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:52.847 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:52.847 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:52.848 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:52.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:52.848 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:52.849 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:52.849 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:52.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:52.850 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:52Z" 2023-10-18T21:11:52.850 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:52.851 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:52Z" } 2023-10-18T21:11:52.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:52.852 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:52.853 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:52.853 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:11:57.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:57.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:11:57.855 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:11:57.913 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:11:57.913 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:11:57.944 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:57.942+0000 7f6568772780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:57.945 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:11:57.944+0000 7f6568772780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:11:57.960 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:11:57.960 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:11:57.960 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:11:57.960 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:11:57.961 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:11:57.961 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:11:57.961 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:57.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:57.961 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:11:57.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:57.962 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:11:57.962 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:57.962 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:57.963 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:11:57.963 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:57.963 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:11:57.963 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:57.963 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:11:57.964 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:57.964 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:57.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:57.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:57.965 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:11:57.965 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:57.965 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:57.965 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:11:57.965 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:57.966 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:57.966 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:57.966 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:11:57.966 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:57.966 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:57.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:57.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:57.967 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:11:57.967 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:57.968 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:57.968 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:11:57.968 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:57.968 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:57.968 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:57.969 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:11:57.969 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:57.969 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:57.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:57.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:11:57.970 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:11:57.970 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:11:57.970 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:11:57.970 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:11:57.970 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:11:57.971 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:11:57.971 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:11:57.971 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:11:57.971 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:11:57.972 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:57.972 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:11:57.972 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:11:57.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:57.972 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:11:57.973 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:11:57.973 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:11:57.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:11:57.973 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:11:57Z" 2023-10-18T21:11:57.973 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:11:57.974 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:11:57Z" } 2023-10-18T21:11:57.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:11:57.975 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:57.976 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:11:57.976 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:02.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:02.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:02.979 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:03.035 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:03.036 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:03.068 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:03.066+0000 7f2d4637f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:03.069 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:03.067+0000 7f2d4637f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:03.083 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:03.084 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:03.084 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:03.084 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:03.084 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:03.084 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:03.085 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:03.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:03.085 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:03.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:03.085 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:03.086 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:03.086 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:03.086 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:12:03.086 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:03.087 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:03.087 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:03.087 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:12:03.087 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:03.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:03.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:03.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:03.088 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:03.088 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:03.088 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:03.089 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:03.089 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:03.089 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:03.089 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:03.089 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:03.090 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:03.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:03.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:03.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:03.091 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:03.091 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:03.091 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:03.091 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:03.091 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:03.092 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:03.092 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:03.092 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:03.092 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:03.092 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:03.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:03.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:03.093 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:03.093 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:03.093 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:03.094 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:03.094 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:03.094 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:03.094 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:03.095 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:03.095 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:03.095 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:03.095 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:03.095 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:03.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:03.096 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:03.096 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:03.096 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:03.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:03.097 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:03Z" 2023-10-18T21:12:03.097 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:03.098 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:03Z" } 2023-10-18T21:12:03.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:03.099 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:03.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:03.100 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:08.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:08.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:08.102 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:08.158 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:08.159 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:08.191 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:08.190+0000 7fdba45b2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:08.192 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:08.191+0000 7fdba45b2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:08.206 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:08.207 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:08.207 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:08.207 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:08.207 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:08.207 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:08.208 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:08.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:08.208 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:08.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:08.209 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:08.209 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:08.209 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:08.209 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:12:08.209 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:08.210 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:08.210 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:08.210 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:12:08.210 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:08.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:08.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:08.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:08.211 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:08.211 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:08.211 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:08.212 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:08.212 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:08.212 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:08.212 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:08.213 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:08.213 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:08.213 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:08.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:08.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:08.213 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:08.214 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:08.214 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:08.214 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:08.214 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:08.215 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:08.215 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:08.215 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:08.215 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:08.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:08.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:08.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:08.216 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:08.216 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:08.216 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:08.217 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:08.217 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:08.217 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:08.217 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:08.217 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:08.218 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:08.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:08.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:08.218 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:08.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:08.219 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:08.219 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:08.219 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:08.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:08.220 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:08Z" 2023-10-18T21:12:08.220 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:08.221 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:08Z" } 2023-10-18T21:12:08.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:08.222 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:08.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:08.223 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:13.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:13.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:13.225 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:13.283 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:13.283 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:13.316 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:13.315+0000 7f6cc637b780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:13.316 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:13.316+0000 7f6cc637b780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:13.330 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:13.331 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:13.331 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:13.331 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:13.331 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:13.331 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:13.332 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:13.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:13.332 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:13.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:13.332 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:13.333 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:13.333 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:13.333 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:12:13.333 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:13.334 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:13.334 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:13.334 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:12:13.334 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:13.334 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:13.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:13.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:13.335 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:13.335 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:13.335 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:13.336 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:13.336 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:13.336 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:13.336 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:13.336 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:13.337 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:13.337 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:13.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:13.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:13.337 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:13.338 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:13.338 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:13.338 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:13.338 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:13.339 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:13.339 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:13.339 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:13.339 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:13.339 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:13.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:13.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:13.340 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:13.340 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:13.340 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:13.341 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:13.341 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:13.341 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:13.341 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:13.341 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:13.342 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:13.342 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:13.342 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:13.342 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:13.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:13.343 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:13.343 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:13.343 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:13.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:13.343 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:13Z" 2023-10-18T21:12:13.344 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:13.345 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:13Z" } 2023-10-18T21:12:13.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:13.345 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:13.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:13.346 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:18.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:18.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:18.349 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:18.406 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:18.407 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:18.440 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:18.439+0000 7f6abf34c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:18.441 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:18.440+0000 7f6abf34c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:18.455 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:18.455 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:18.455 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:18.455 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:18.455 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:18.456 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:18.456 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:18.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:18.456 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:18.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:18.457 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:18.457 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:18.457 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:18.457 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:12:18.457 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:18.458 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:18.458 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:18.458 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:12:18.458 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:18.458 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:18.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:18.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:18.459 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:18.459 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:18.459 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:18.460 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:18.460 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:18.460 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:18.460 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:18.460 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:18.461 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:18.461 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:18.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:18.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:18.461 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:18.461 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:18.462 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:18.462 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:18.462 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:18.462 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:18.463 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:18.463 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:18.463 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:18.463 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:18.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:18.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:18.464 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:18.464 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:18.464 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:18.464 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:18.464 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:18.465 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:18.465 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:18.465 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:18.465 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:18.466 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:18.466 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:18.466 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:18.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:18.467 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:18.467 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:18.467 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:18.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:18.467 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:18Z" 2023-10-18T21:12:18.468 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:18.468 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:18Z" } 2023-10-18T21:12:18.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:18.469 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:18.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:18.470 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:23.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:23.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:23.473 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:23.532 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:23.532 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:23.565 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:23.564+0000 7f215fd61780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:23.566 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:23.565+0000 7f215fd61780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:23.579 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:23.580 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:23.580 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:23.580 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:23.580 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:23.581 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:23.581 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:23.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:23.581 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:23.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:23.582 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:23.582 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:23.582 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:23.583 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:12:23.583 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:23.583 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:23.583 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:23.583 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:12:23.584 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:23.584 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:23.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:23.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:23.585 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:23.585 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:23.586 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:23.586 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:23.586 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:23.587 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:23.587 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:23.587 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:23.587 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:23.588 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:23.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:23.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:23.588 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:23.588 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:23.589 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:23.589 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:23.589 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:23.589 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:23.589 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:23.589 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:23.590 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:23.590 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:23.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:23.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:23.591 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:23.591 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:23.591 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:23.591 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:23.591 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:23.592 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:23.592 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:23.592 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:23.592 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:23.592 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:23.592 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:23.593 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:23.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:23.593 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:23.594 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:23.594 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:23.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:23.594 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:23Z" 2023-10-18T21:12:23.594 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:23.595 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:23Z" } 2023-10-18T21:12:23.596 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:23.597 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:23.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:23.597 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:28.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:28.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:28.599 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:28.657 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:28.657 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:28.691 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:28.690+0000 7f699fc99780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:28.692 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:28.691+0000 7f699fc99780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:28.706 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:12:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:12:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:28Z" 2023-10-18T21:12:28.719 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:28.720 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:28Z" } 2023-10-18T21:12:28.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:28.721 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:28.721 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:28.722 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:33.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:33.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:33.724 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:33.780 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:33.781 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:33.813 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:33.812+0000 7f12e3772780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:33.814 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:33.813+0000 7f12e3772780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:33.827 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:33.828 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:33.828 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:33.828 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:33.828 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:33.830 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:33.830 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:33.830 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:12:33.830 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:12:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:33.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:33.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:33.832 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:33.832 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:33.836 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:33.836 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:33.836 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:33.836 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:33.837 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:33.837 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:33.837 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:33Z" 2023-10-18T21:12:33.842 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:33.843 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:33Z" } 2023-10-18T21:12:33.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:33.844 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:33.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:33.844 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:38.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:38.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:38.847 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:38.904 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:38.904 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:38.938 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:38.937+0000 7f3da2bab780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:38.938 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:38.938+0000 7f3da2bab780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:38.952 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:38.952 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:38.953 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:38.953 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:38.953 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:38.953 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:38.953 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:38.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:38.954 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:38.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:38.954 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:38.955 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:38.955 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:38.955 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:12:38.955 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:38.955 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:38.956 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:38.956 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:12:38.956 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:38.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:38.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:38.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:38.957 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:38.957 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:38.957 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:38.958 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:38.958 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:38.958 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:38.958 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:38.958 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:38.959 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:38.959 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:38.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:38.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:38.960 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:38.960 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:38.960 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:38.960 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:38.960 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:38.961 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:38.961 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:38.961 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:38.961 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:38.961 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:38.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:38.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:38.962 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:38.962 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:38.962 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:38.963 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:38.963 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:38.963 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:38.963 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:38.963 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:38.964 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:38.964 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:38.964 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:38.964 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:38.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:38.965 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:38.965 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:38.965 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:38.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:38.965 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:38Z" 2023-10-18T21:12:38.966 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:38.967 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:38Z" } 2023-10-18T21:12:38.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:38.968 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:38.968 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:38.969 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:41.664 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T21:12:41.663+0000 7fc3d5be8700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-18T21:12:43.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:43.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:43.971 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:44.028 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:44.029 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:44.063 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:44.061+0000 7f66a8a9d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:44.064 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:44.062+0000 7f66a8a9d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:44.077 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:44.077 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:44.078 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:44.078 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:44.078 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:44.078 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:44.078 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:44.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:44.079 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:44.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:44.079 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:44.080 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:44.080 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:44.080 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663433.351663_1634.1", 2023-10-18T21:12:44.080 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:44.080 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:44.081 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:44.081 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:33.351663Z", 2023-10-18T21:12:44.081 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:44.081 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:44.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:44.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:44.082 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:44.082 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:44.082 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:44.082 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:44.083 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:44.083 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:44.083 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:44.083 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:44.083 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:44.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:44.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:44.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:44.084 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:44.084 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:44.084 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:44.085 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:44.085 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:44.085 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:44.085 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:44.085 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:44.086 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:44.086 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:44.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:44.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:44.086 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:44.087 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:44.087 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:44.087 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:44.087 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:44.087 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:44.087 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:44.088 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:44.088 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:44.088 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:44.088 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:44.088 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:44.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:44.089 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:44.089 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:44.089 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:44.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:44.090 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:44Z" 2023-10-18T21:12:44.090 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:44.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663433.351663_1634.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:33.351663Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:44Z" } 2023-10-18T21:12:44.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663433.351663_1634.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:33.351663Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:44.092 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:44.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663433.351663_1634.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:44.092 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663440.822914_1716.1 target=1_1697663433.351663_1634.1 2023-10-18T21:12:49.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.095 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:49.153 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:49.153 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:49.185 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:49.183+0000 7f3a83f6b780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:49.186 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:49.184+0000 7f3a83f6b780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:49.200 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:49.200 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:49.200 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:49.200 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:49.201 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:49.201 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:49.201 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.202 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:49.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.202 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:49.202 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.202 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:49.203 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663440.822914_1716.1", 2023-10-18T21:12:49.203 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.203 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:49.203 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.204 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.822914Z", 2023-10-18T21:12:49.204 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.205 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:49.205 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.205 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:49.205 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:49.206 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.206 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:49.206 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.206 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:49.206 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.207 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:49.207 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.208 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:49.208 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:49.208 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.208 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:49.208 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.209 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:49.209 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.209 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.210 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:49.210 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.210 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:49.210 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:49.210 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.211 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:49.211 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.211 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:49.211 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.211 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.212 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:49.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.212 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:49.212 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:49.213 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:49.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.213 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:49Z" 2023-10-18T21:12:49.213 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:49.214 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663440.822914_1716.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:40.822914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:49Z" } 2023-10-18T21:12:49.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663440.822914_1716.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.822914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:49.215 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:49.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:49.216 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:12:49.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.217 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:49.266 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:49.266 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:49.319 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:12:49.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.319 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:12:49.319 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:12:49.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.320 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:12:49.320 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:12:49.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.321 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:12:49.321 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:12:49.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.322 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:12:49.322 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:12:49.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.322 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:12:49.323 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:12:49.324 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:12:49.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.325 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:49.375 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:49.376 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:49.419 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:49.419 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:49.419 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:49.419 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:49.420 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:49.420 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:12:49.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.420 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:49.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.421 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:49.421 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.421 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:49.421 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:12:49.421 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.422 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:49.422 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.422 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:12:49.422 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.423 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:49.423 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.423 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:49.423 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:12:49.424 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.424 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:49.424 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.424 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:12:49.424 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.425 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:49.425 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.425 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:12:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:12:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:12:49.427 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.427 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:49.427 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.428 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:49.428 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:12:49.428 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.428 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:12:49.429 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.429 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:12:49.429 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.429 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.429 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:49.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.430 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:49.430 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:12:49.430 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:12:49.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.431 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:49Z" 2023-10-18T21:12:49.431 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:49.432 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:12:49Z" } 2023-10-18T21:12:49.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:12:49.433 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:12:49.433 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:12:49.433 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:12:49.434 DEBUG:boto:path=/bofrnf-66/ 2023-10-18T21:12:49.434 DEBUG:boto:auth_path=/bofrnf-66/ 2023-10-18T21:12:49.435 DEBUG:boto:Method: HEAD 2023-10-18T21:12:49.435 DEBUG:boto:Path: /bofrnf-66/ 2023-10-18T21:12:49.435 DEBUG:boto:Data: 2023-10-18T21:12:49.436 DEBUG:boto:Headers: {} 2023-10-18T21:12:49.436 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:12:49.436 DEBUG:boto:Port: 8001 2023-10-18T21:12:49.437 DEBUG:boto:Params: {} 2023-10-18T21:12:49.437 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:12:49.438 DEBUG:boto:Token: None 2023-10-18T21:12:49.438 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:12:49 GMT /bofrnf-66/ 2023-10-18T21:12:49.439 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oqFuVYGa6Lt9CBgaSk0+9xAZAPg= 2023-10-18T21:12:49.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oqFuVYGa6Lt9CBgaSk0+9xAZAPg=', 'Content-Length': '0'} 2023-10-18T21:12:49.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000967a3ec30d765342-0065304a51-2218-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:12:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:49.446 DEBUG:boto:path=/bofrnf-66/obj1 2023-10-18T21:12:49.447 DEBUG:boto:auth_path=/bofrnf-66/obj1 2023-10-18T21:12:49.447 DEBUG:boto:Method: HEAD 2023-10-18T21:12:49.447 DEBUG:boto:Path: /bofrnf-66/obj1 2023-10-18T21:12:49.448 DEBUG:boto:Data: 2023-10-18T21:12:49.448 DEBUG:boto:Headers: {} 2023-10-18T21:12:49.449 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:12:49.449 DEBUG:boto:Port: 8001 2023-10-18T21:12:49.449 DEBUG:boto:Params: {} 2023-10-18T21:12:49.450 DEBUG:boto:Token: None 2023-10-18T21:12:49.450 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:12:49 GMT /bofrnf-66/obj1 2023-10-18T21:12:49.451 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:StCVES41uzoe/PbwzlZzpgHpYyY= 2023-10-18T21:12:49.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:StCVES41uzoe/PbwzlZzpgHpYyY=', 'Content-Length': '0'} 2023-10-18T21:12:49.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-66:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fff2d3ae3e87549d-0065304a51-2218-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:12:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:49.455 DEBUG:boto:path=/bofrnf-66/obj1 2023-10-18T21:12:49.455 DEBUG:boto:auth_path=/bofrnf-66/obj1 2023-10-18T21:12:49.456 DEBUG:boto:Method: GET 2023-10-18T21:12:49.456 DEBUG:boto:Path: /bofrnf-66/obj1 2023-10-18T21:12:49.457 DEBUG:boto:Data: 2023-10-18T21:12:49.457 DEBUG:boto:Headers: {} 2023-10-18T21:12:49.458 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:12:49.458 DEBUG:boto:Port: 8001 2023-10-18T21:12:49.459 DEBUG:boto:Params: {} 2023-10-18T21:12:49.459 DEBUG:boto:Token: None 2023-10-18T21:12:49.459 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:12:49 GMT /bofrnf-66/obj1 2023-10-18T21:12:49.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:HqofynrlbDN7QIUZcQg8Q6tiqck= 2023-10-18T21:12:49.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HqofynrlbDN7QIUZcQg8Q6tiqck=', 'Content-Length': '0'} 2023-10-18T21:12:49.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-66:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c28820c9e42ffc3-0065304a51-2218-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:12:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:49.463 DEBUG:boto:path=/bofrnf-67/ 2023-10-18T21:12:49.463 DEBUG:boto:auth_path=/bofrnf-67/ 2023-10-18T21:12:49.464 DEBUG:boto:Method: HEAD 2023-10-18T21:12:49.464 DEBUG:boto:Path: /bofrnf-67/ 2023-10-18T21:12:49.464 DEBUG:boto:Data: 2023-10-18T21:12:49.465 DEBUG:boto:Headers: {} 2023-10-18T21:12:49.465 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:49.466 DEBUG:boto:Port: 8000 2023-10-18T21:12:49.466 DEBUG:boto:Params: {} 2023-10-18T21:12:49.467 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:12:49.467 DEBUG:boto:Token: None 2023-10-18T21:12:49.468 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:12:49 GMT /bofrnf-67/ 2023-10-18T21:12:49.468 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cYq77gpHLt6F2LIBdwOl/KRyUHc= 2023-10-18T21:12:49.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cYq77gpHLt6F2LIBdwOl/KRyUHc=', 'Content-Length': '0'} 2023-10-18T21:12:49.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea3e28d904e29c22-0065304a51-2216-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:12:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:49.473 DEBUG:boto:path=/bofrnf-67/obj2 2023-10-18T21:12:49.473 DEBUG:boto:auth_path=/bofrnf-67/obj2 2023-10-18T21:12:49.474 DEBUG:boto:Method: HEAD 2023-10-18T21:12:49.474 DEBUG:boto:Path: /bofrnf-67/obj2 2023-10-18T21:12:49.475 DEBUG:boto:Data: 2023-10-18T21:12:49.475 DEBUG:boto:Headers: {} 2023-10-18T21:12:49.476 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:49.476 DEBUG:boto:Port: 8000 2023-10-18T21:12:49.476 DEBUG:boto:Params: {} 2023-10-18T21:12:49.477 DEBUG:boto:Token: None 2023-10-18T21:12:49.477 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:12:49 GMT /bofrnf-67/obj2 2023-10-18T21:12:49.478 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qyYb1MmVT2X7R6PoB5yjFGYiFwU= 2023-10-18T21:12:49.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qyYb1MmVT2X7R6PoB5yjFGYiFwU=', 'Content-Length': '0'} 2023-10-18T21:12:49.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-67:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f43f389cf44659c-0065304a51-2216-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:12:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:49.481 DEBUG:boto:path=/bofrnf-67/obj2 2023-10-18T21:12:49.481 DEBUG:boto:auth_path=/bofrnf-67/obj2 2023-10-18T21:12:49.482 DEBUG:boto:Method: GET 2023-10-18T21:12:49.482 DEBUG:boto:Path: /bofrnf-67/obj2 2023-10-18T21:12:49.482 DEBUG:boto:Data: 2023-10-18T21:12:49.483 DEBUG:boto:Headers: {} 2023-10-18T21:12:49.483 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:49.484 DEBUG:boto:Port: 8000 2023-10-18T21:12:49.484 DEBUG:boto:Params: {} 2023-10-18T21:12:49.484 DEBUG:boto:Token: None 2023-10-18T21:12:49.485 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:12:49 GMT /bofrnf-67/obj2 2023-10-18T21:12:49.485 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qCD89hSdXdPjpbOHt8v2APVyLdk= 2023-10-18T21:12:49.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qCD89hSdXdPjpbOHt8v2APVyLdk=', 'Content-Length': '0'} 2023-10-18T21:12:49.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:10:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-67:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c35f358f31f10139-0065304a51-2216-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:12:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:49.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:49.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:49.489 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:49.536 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:49.537 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:49.581 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:12:49.582 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ok 2023-10-18T21:12:49.583 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-10-18T21:12:49.584 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-10-18T21:12:49.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.585 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:49.633 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:49.634 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:49.664 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:12:49.677 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:12:49.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.677 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663440.822914_1716.1", 2023-10-18T21:12:49.678 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:10:40.822914Z" 2023-10-18T21:12:49.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.678 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:12:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:12:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:12:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.681 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:12:49.682 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:49.682 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:12:49.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:49.683 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:49.733 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:49.733 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:49.761 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:49.759+0000 7fe71661b780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:49.762 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:49.760+0000 7fe71661b780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:49.774 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:49.775 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:49.775 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:49.775 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:49.775 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:49.775 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:49.776 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.776 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:49.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.777 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:49.777 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.777 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:49.777 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663440.822914_1716.1", 2023-10-18T21:12:49.777 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.778 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:49.778 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.778 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.822914Z", 2023-10-18T21:12:49.778 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.778 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.779 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:49.779 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.780 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:49.780 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:49.780 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.780 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:49.780 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.781 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:49.781 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.781 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.782 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:49.782 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.782 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:49.782 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:49.783 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.783 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:49.783 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.783 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:49.783 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:49.784 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:49.784 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:49.785 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:49.785 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:49.785 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:49.785 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:49.785 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:49.786 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:49.786 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:49.786 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.786 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:49.787 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:49.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.787 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:49.787 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:49.787 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:49.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:49.788 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:49Z" 2023-10-18T21:12:49.788 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:49.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663440.822914_1716.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:10:40.822914Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:49Z" } 2023-10-18T21:12:49.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663440.822914_1716.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.822914Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:49.790 DEBUG:rgw_multi.tests:log_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:49.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663440.822914_1716.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:49.791 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:12:49.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:49.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:49.792 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:49.881 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[]} 2023-10-18T21:12:49.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:49.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:49.883 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:49.928 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:49.929 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:49.973 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[]} 2023-10-18T21:12:49.974 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-68 2023-10-18T21:12:49.974 DEBUG:boto:path=/bofrnf-68/ 2023-10-18T21:12:49.975 DEBUG:boto:auth_path=/bofrnf-68/ 2023-10-18T21:12:49.975 DEBUG:boto:Method: PUT 2023-10-18T21:12:49.976 DEBUG:boto:Path: /bofrnf-68/ 2023-10-18T21:12:49.976 DEBUG:boto:Data: 2023-10-18T21:12:49.977 DEBUG:boto:Headers: {} 2023-10-18T21:12:49.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:49.977 DEBUG:boto:Port: 8000 2023-10-18T21:12:49.978 DEBUG:boto:Params: {} 2023-10-18T21:12:49.978 DEBUG:boto:Token: None 2023-10-18T21:12:49.979 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:12:49 GMT /bofrnf-68/ 2023-10-18T21:12:49.979 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ucIVsKhCZNPExgXhQ3fmMPbYFlw= 2023-10-18T21:12:49.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:49 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ucIVsKhCZNPExgXhQ3fmMPbYFlw=', 'Content-Length': '0'} 2023-10-18T21:12:49.994 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b10087556f6aa1f6-0065304a51-2216-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:12:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:49.995 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-69 2023-10-18T21:12:49.995 DEBUG:boto:path=/bofrnf-69/ 2023-10-18T21:12:49.996 DEBUG:boto:auth_path=/bofrnf-69/ 2023-10-18T21:12:49.996 DEBUG:boto:Method: PUT 2023-10-18T21:12:49.997 DEBUG:boto:Path: /bofrnf-69/ 2023-10-18T21:12:49.997 DEBUG:boto:Data: 2023-10-18T21:12:49.997 DEBUG:boto:Headers: {} 2023-10-18T21:12:49.998 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:49.998 DEBUG:boto:Port: 8000 2023-10-18T21:12:49.999 DEBUG:boto:Params: {} 2023-10-18T21:12:49.999 DEBUG:boto:Token: None 2023-10-18T21:12:50.000 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:12:50 GMT /bofrnf-69/ 2023-10-18T21:12:50.000 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sidyljyMf8ON+hmKMR2nyZDnAnI= 2023-10-18T21:12:50.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:50 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sidyljyMf8ON+hmKMR2nyZDnAnI=', 'Content-Length': '0'} 2023-10-18T21:12:50.015 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000094aeca846c2abbda-0065304a52-2216-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:12:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:50.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:50.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:50.017 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:50.068 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:50.068 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:50.103 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:12:50.116 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:12:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:12:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:12:50.013765Z" 2023-10-18T21:12:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:12:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:12:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:12:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.120 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:12:50.121 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:50.122 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:12:50.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:50.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:50.123 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:50.172 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:50.173 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:50.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:50.201+0000 7f25de413780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:50.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:50.202+0000 7f25de413780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:50.216 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:50.217 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:50.217 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:50.217 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:50.217 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:50.218 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:50.218 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:50.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.218 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:50.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.219 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:50.219 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:50.219 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:50.219 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:12:50.219 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:50.220 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:50.220 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:50.220 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:12:50.220 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:50.220 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.221 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:50.221 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:50.221 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:50.222 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:50.222 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:50.222 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:50.222 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:50.222 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:50.223 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:50.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.223 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:50.224 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:50.224 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:50.224 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:50.224 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:50.224 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:50.224 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:50.225 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:50.225 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:50.225 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.226 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:50.226 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:50.226 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:50.226 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:50.227 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:50.227 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:50.227 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:50.227 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:50.227 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:50.227 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.228 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.228 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.228 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:50.229 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:50.229 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:50.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.229 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:50Z" 2023-10-18T21:12:50.230 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:50.231 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:50Z" } 2023-10-18T21:12:50.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:50.232 DEBUG:rgw_multi.tests:log_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:50.232 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:50.232 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:12:50.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.234 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:50.282 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:50.282 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:50.328 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:12:50.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.330 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:50.383 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:50.384 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:50.429 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-18T21:12:50.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=bofrnf-68', '--dest-bucket=bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=bofrnf-68', '--dest-bucket=bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.432 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=bofrnf-68 --dest-bucket=bofrnf-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:50.486 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:50.487 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:50.531 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"bofrnf-68","zones":["a1","a2"]},"dest":{"bucket":"bofrnf-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:12:50.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.533 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:50.583 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:50.583 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:50.627 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"bofrnf-68","zones":["a1","a2"]},"dest":{"bucket":"bofrnf-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:12:50.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.629 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:50.673 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:50.674 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:50.718 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"bofrnf-68","zones":["a1","a2"]},"dest":{"bucket":"bofrnf-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:12:50.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.720 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:50.767 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:50.768 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:50.818 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:50.818 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:50.818 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 16, 2023-10-18T21:12:50.819 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:12:50.819 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:12:50.819 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:12:50.819 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:50.819 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:12:50.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.820 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:12:50.820 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:12:50.820 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:12:50.820 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:12:50.821 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:12:50.821 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:12:50.821 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.821 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:12:50.821 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:12:50.822 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:12:50.822 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:12:50.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.822 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:12:50.823 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:12:50.823 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:12:50.823 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:12:50.823 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.823 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:12:50.824 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:12:50.824 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:12:50.824 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:12:50.824 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:12:50.825 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:12:50.825 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:12:50.825 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:12:50.825 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:12:50.825 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:12:50.826 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:12:50.826 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.826 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:12:50.827 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:12:50.827 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:12:50.827 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:12:50.827 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.828 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:12:50.828 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:12:50.828 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:12:50.828 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:12:50.828 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:12:50.829 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:12:50.829 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:12:50.829 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:12:50.829 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:12:50.829 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:12:50.829 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:12:50.830 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.830 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.830 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.830 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:12:50.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.831 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:12:50.831 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:12:50.831 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:12:50.831 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:12:50.831 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.832 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.832 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.832 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:12:50.832 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:12:50.832 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:12:50.833 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [ 2023-10-18T21:12:50.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.833 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-group", 2023-10-18T21:12:50.833 INFO:teuthology.orchestra.run.smithi082.stdout: "data_flow": { 2023-10-18T21:12:50.833 INFO:teuthology.orchestra.run.smithi082.stdout: "symmetrical": [ 2023-10-18T21:12:50.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.834 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-flow1", 2023-10-18T21:12:50.834 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:12:50.834 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:12:50.834 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:12:50.835 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.835 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.835 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.835 INFO:teuthology.orchestra.run.smithi082.stdout: "pipes": [ 2023-10-18T21:12:50.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.836 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe", 2023-10-18T21:12:50.836 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:12:50.836 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-68", 2023-10-18T21:12:50.837 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:12:50.837 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:12:50.837 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:12:50.837 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.838 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:12:50.838 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-68", 2023-10-18T21:12:50.838 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:12:50.838 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:12:50.838 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:12:50.839 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.839 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:12:50.839 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:12:50.839 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:12:50.840 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:12:50.840 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.840 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:12:50.840 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:12:50.841 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:12:50.841 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:12:50.841 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.841 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.841 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.842 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "enabled" 2023-10-18T21:12:50.842 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.842 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.842 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:12:50.843 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:12:50.843 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.844 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:12:50.844 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:12:50.844 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:12:50.844 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:12:50.844 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:12:50.845 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:12:50.845 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.845 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:12:50.845 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:12:50.845 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:12:50.846 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:12:50.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.846 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:12:50.846 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:12:50.846 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:12:50.847 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:12:50.847 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.847 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:12:50.847 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:12:50.847 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:12:50.848 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:12:50.848 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:12:50.848 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:12:50.848 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:12:50.849 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:12:50.849 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:12:50.849 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:12:50.849 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:12:50.849 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.850 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:12:50.850 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:12:50.851 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:12:50.851 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:12:50.851 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.851 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:12:50.852 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:12:50.852 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:12:50.852 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:12:50.852 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:12:50.852 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:12:50.853 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:12:50.853 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:12:50.853 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:12:50.853 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:12:50.853 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:12:50.854 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.854 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.854 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.854 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:12:50.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.855 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:12:50.855 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:12:50.855 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:12:50.855 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:12:50.855 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.856 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.856 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.856 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:12:50.856 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:12:50.856 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:12:50.857 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:12:50.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.857 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:12:50.857 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:12:50.857 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.858 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.858 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:12:50.858 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:12:50.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.858 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:12:50.859 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:12:50.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.859 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:12:50.860 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:12:50.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.861 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:12:50.861 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:12:50.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:50.862 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:12:50.862 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:12:50.862 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.862 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:50.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.863 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:12:50.863 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:12:50.863 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:12:50.864 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:12:50.864 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:12:50.864 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:12:50.864 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:12:50.864 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:12:50.865 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:12:50.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.865 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:12:50.865 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:12:50.865 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:12:50.866 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:12:50.866 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:12:50.866 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:12:50.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.867 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:12:50.867 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:12:50.867 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:12:50.867 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:12:50.868 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:12:50.868 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:12:50.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.869 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:12:50.869 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:12:50.869 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:12:50.869 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:12:50.869 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:12:50.870 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:12:50.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.870 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:12:50.870 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:12:50.870 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:12:50.871 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:12:50.871 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:12:50.871 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:12:50.871 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:50.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:50.872 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:12:50.872 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:12:50.872 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:50.872 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:50.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:12:50.874 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:12:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:51.053 INFO:teuthology.orchestra.run.smithi082.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"bofrnf-68"},"dest":{"zone":"a1","bucket":"bofrnf-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"bofrnf-68"},"dest":{"zone":"a2","bucket":"bofrnf-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-10-18T21:12:51.053 DEBUG:boto:path=/bofrnf-68/ 2023-10-18T21:12:51.054 DEBUG:boto:auth_path=/bofrnf-68/ 2023-10-18T21:12:51.054 DEBUG:boto:Method: HEAD 2023-10-18T21:12:51.055 DEBUG:boto:Path: /bofrnf-68/ 2023-10-18T21:12:51.055 DEBUG:boto:Data: 2023-10-18T21:12:51.055 DEBUG:boto:Headers: {} 2023-10-18T21:12:51.056 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:51.056 DEBUG:boto:Port: 8000 2023-10-18T21:12:51.057 DEBUG:boto:Params: {} 2023-10-18T21:12:51.057 DEBUG:boto:Token: None 2023-10-18T21:12:51.058 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:12:51 GMT /bofrnf-68/ 2023-10-18T21:12:51.058 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:45CG+wlNpLOoFZiJL4NbyCuEc3k= 2023-10-18T21:12:51.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:45CG+wlNpLOoFZiJL4NbyCuEc3k=', 'Content-Length': '0'} 2023-10-18T21:12:51.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbd8193f0165eaca-0065304a53-2376-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:51.072 DEBUG:boto:path=/bofrnf-68/obj1 2023-10-18T21:12:51.073 DEBUG:boto:auth_path=/bofrnf-68/obj1 2023-10-18T21:12:51.073 DEBUG:boto:Method: PUT 2023-10-18T21:12:51.074 DEBUG:boto:Path: /bofrnf-68/obj1 2023-10-18T21:12:51.074 DEBUG:boto:Data: 2023-10-18T21:12:51.074 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:12:51.075 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:51.075 DEBUG:boto:Port: 8000 2023-10-18T21:12:51.076 DEBUG:boto:Params: {} 2023-10-18T21:12:51.076 DEBUG:boto:Token: None 2023-10-18T21:12:51.077 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:12:51 GMT /bofrnf-68/obj1 2023-10-18T21:12:51.077 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3XyjP9VJ4Gg6Kz3sq/9yJ2L0Um0= 2023-10-18T21:12:51.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:12:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3XyjP9VJ4Gg6Kz3sq/9yJ2L0Um0='} 2023-10-18T21:12:51.100 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057dadcbde43a69a2-0065304a53-2376-a1'), ('Date', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:51.100 DEBUG:boto:path=/bofrnf-68/ 2023-10-18T21:12:51.101 DEBUG:boto:auth_path=/bofrnf-68/ 2023-10-18T21:12:51.101 DEBUG:boto:Method: HEAD 2023-10-18T21:12:51.102 DEBUG:boto:Path: /bofrnf-68/ 2023-10-18T21:12:51.102 DEBUG:boto:Data: 2023-10-18T21:12:51.102 DEBUG:boto:Headers: {} 2023-10-18T21:12:51.103 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:51.103 DEBUG:boto:Port: 8000 2023-10-18T21:12:51.104 DEBUG:boto:Params: {} 2023-10-18T21:12:51.104 DEBUG:boto:Token: None 2023-10-18T21:12:51.104 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:12:51 GMT /bofrnf-68/ 2023-10-18T21:12:51.105 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:45CG+wlNpLOoFZiJL4NbyCuEc3k= 2023-10-18T21:12:51.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:45CG+wlNpLOoFZiJL4NbyCuEc3k=', 'Content-Length': '0'} 2023-10-18T21:12:51.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073c1a9660fabe876-0065304a53-2376-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:51.110 DEBUG:boto:path=/bofrnf-68/obj2 2023-10-18T21:12:51.110 DEBUG:boto:auth_path=/bofrnf-68/obj2 2023-10-18T21:12:51.111 DEBUG:boto:Method: PUT 2023-10-18T21:12:51.111 DEBUG:boto:Path: /bofrnf-68/obj2 2023-10-18T21:12:51.111 DEBUG:boto:Data: 2023-10-18T21:12:51.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:12:51.112 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:51.113 DEBUG:boto:Port: 8000 2023-10-18T21:12:51.113 DEBUG:boto:Params: {} 2023-10-18T21:12:51.114 DEBUG:boto:Token: None 2023-10-18T21:12:51.114 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:12:51 GMT /bofrnf-68/obj2 2023-10-18T21:12:51.114 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:O0SaqxLkWbFKsLyRfhioKlw7Y68= 2023-10-18T21:12:51.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:12:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O0SaqxLkWbFKsLyRfhioKlw7Y68='} 2023-10-18T21:12:51.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1d12fb471e4f1da-0065304a53-2376-a1'), ('Date', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:51.163 DEBUG:boto:path=/bofrnf-68/ 2023-10-18T21:12:51.163 DEBUG:boto:auth_path=/bofrnf-68/ 2023-10-18T21:12:51.164 DEBUG:boto:Method: HEAD 2023-10-18T21:12:51.164 DEBUG:boto:Path: /bofrnf-68/ 2023-10-18T21:12:51.165 DEBUG:boto:Data: 2023-10-18T21:12:51.165 DEBUG:boto:Headers: {} 2023-10-18T21:12:51.165 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:51.167 DEBUG:boto:Port: 8000 2023-10-18T21:12:51.167 DEBUG:boto:Params: {} 2023-10-18T21:12:51.168 DEBUG:boto:Token: None 2023-10-18T21:12:51.168 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:12:51 GMT /bofrnf-68/ 2023-10-18T21:12:51.169 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:45CG+wlNpLOoFZiJL4NbyCuEc3k= 2023-10-18T21:12:51.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:45CG+wlNpLOoFZiJL4NbyCuEc3k=', 'Content-Length': '0'} 2023-10-18T21:12:51.172 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efec1642a9a63ef4-0065304a53-2376-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:51.173 DEBUG:boto:path=/bofrnf-68/obj3 2023-10-18T21:12:51.174 DEBUG:boto:auth_path=/bofrnf-68/obj3 2023-10-18T21:12:51.174 DEBUG:boto:Method: PUT 2023-10-18T21:12:51.174 DEBUG:boto:Path: /bofrnf-68/obj3 2023-10-18T21:12:51.175 DEBUG:boto:Data: 2023-10-18T21:12:51.175 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:12:51.176 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:51.176 DEBUG:boto:Port: 8000 2023-10-18T21:12:51.176 DEBUG:boto:Params: {} 2023-10-18T21:12:51.177 DEBUG:boto:Token: None 2023-10-18T21:12:51.177 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:12:51 GMT /bofrnf-68/obj3 2023-10-18T21:12:51.178 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OKGrSHCtnBjOrJkyuUzs/8k6CVw= 2023-10-18T21:12:51.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:12:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OKGrSHCtnBjOrJkyuUzs/8k6CVw='} 2023-10-18T21:12:51.189 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000809ac56cdfc5e651-0065304a53-2376-a1'), ('Date', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:51.189 DEBUG:boto:path=/bofrnf-69/ 2023-10-18T21:12:51.190 DEBUG:boto:auth_path=/bofrnf-69/ 2023-10-18T21:12:51.190 DEBUG:boto:Method: HEAD 2023-10-18T21:12:51.191 DEBUG:boto:Path: /bofrnf-69/ 2023-10-18T21:12:51.191 DEBUG:boto:Data: 2023-10-18T21:12:51.191 DEBUG:boto:Headers: {} 2023-10-18T21:12:51.192 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:51.192 DEBUG:boto:Port: 8000 2023-10-18T21:12:51.193 DEBUG:boto:Params: {} 2023-10-18T21:12:51.193 DEBUG:boto:Token: None 2023-10-18T21:12:51.194 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:12:51 GMT /bofrnf-69/ 2023-10-18T21:12:51.194 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Umz9cQVsUQGFtIwW+5+VE0Pt6+4= 2023-10-18T21:12:51.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:12:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Umz9cQVsUQGFtIwW+5+VE0Pt6+4=', 'Content-Length': '0'} 2023-10-18T21:12:51.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040af7f6100eabdfc-0065304a53-2376-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:51.200 DEBUG:boto:path=/bofrnf-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-18T21:12:51.200 DEBUG:boto:auth_path=/bofrnf-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-18T21:12:51.200 DEBUG:boto:Method: PUT 2023-10-18T21:12:51.201 DEBUG:boto:Path: /bofrnf-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-18T21:12:51.201 DEBUG:boto:Data: 2023-10-18T21:12:51.202 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:12:51.202 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:12:51.202 DEBUG:boto:Port: 8000 2023-10-18T21:12:51.203 DEBUG:boto:Params: {} 2023-10-18T21:12:51.203 DEBUG:boto:Token: None 2023-10-18T21:12:51.204 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:12:51 GMT /bofrnf-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-18T21:12:51.204 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:XnSU2IuravE7DytocoO3tUFIwYk= 2023-10-18T21:12:51.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:12:51 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XnSU2IuravE7DytocoO3tUFIwYk='} 2023-10-18T21:12:51.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a680db0111538944-0065304a53-2376-a1'), ('Date', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:12:51.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:51.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:51.248 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:51.320 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:51.320 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:51.367 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:12:51.390 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:12:51.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.390 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:12:51.390 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:12:50.013765Z" 2023-10-18T21:12:51.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.391 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:51.391 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:12:51.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.392 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:51.392 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:12:51.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.393 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:51.393 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:12:51.393 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.394 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:12:51.395 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:51.395 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:12:51.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:51.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:51.397 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:51.464 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:51.465 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:51.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:51.510+0000 7fb50781f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:51.513 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:12:51.511+0000 7fb50781f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:12:51.530 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:51.530 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:51.531 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:51.531 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:51.531 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:51.531 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:12:51.532 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:51.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.532 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:51.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.532 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:51.533 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:51.533 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:51.533 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:12:51.533 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:51.533 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:12:51.534 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:51.534 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:12:51.534 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:51.534 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.535 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:51.535 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:51.535 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:51.536 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:12:51.536 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:51.536 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:51.536 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:51.536 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:12:51.537 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:51.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.537 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:51.538 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:51.538 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:51.538 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:12:51.538 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:51.539 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:51.539 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:51.539 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:12:51.539 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:51.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.540 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:51.540 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:51.541 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:12:51.541 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:12:51.541 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:51.541 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:51.541 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:51.542 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:12:51.542 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:12:51.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.542 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:51.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.543 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:51.543 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:12:51.543 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:12:51.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.544 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:51Z" 2023-10-18T21:12:51.544 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:51.545 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:12:51Z" } 2023-10-18T21:12:51.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:12:51.546 DEBUG:rgw_multi.tests:log_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:51.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:12:51.547 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:12:51.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:51.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:51.548 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:51.600 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:51.601 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:12:51.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.651 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:12:51.651 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:12:51.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.652 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:12:51.652 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:12:51.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.652 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:12:51.653 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:12:51.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.653 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051253", 2023-10-18T21:12:51.654 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:12:51.187035Z" 2023-10-18T21:12:51.654 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.654 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:12:51.655 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:12:51.655 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:12:51.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:51.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:51.657 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:51.707 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:51.707 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:51.750 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:51.751 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:51.751 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:51.751 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:51.751 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:51.752 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:12:51.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.752 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:51.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.753 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:51.753 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:51.753 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:51.753 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:12:51.754 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:51.754 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:51.754 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:51.754 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:12:51.754 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.755 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:51.756 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:51.756 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:51.756 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:12:51.756 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:51.756 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:51.757 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:51.757 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:12:51.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.758 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:51.758 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:51.758 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:51.758 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:12:51.758 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:51.759 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:12:51.759 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:51.759 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:12:51.759 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:51.760 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:51.760 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:51.760 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:51.760 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:12:51.761 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:51.761 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:12:51.761 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:51.761 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:12:51.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:51.762 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:51.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.762 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:51.762 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:12:51.763 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:12:51.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:51.763 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:51Z" 2023-10-18T21:12:51.763 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:51.764 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:12:51Z" } 2023-10-18T21:12:51.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:12:51.765 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:12:51.766 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:12:51.766 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:12:56.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:56.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:12:56.768 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:12:56.827 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:12:56.827 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:12:56.875 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:12:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:12:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:12:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:12:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:12:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:12:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:12:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:12:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:12:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:12:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:12:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:12:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:12:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:12:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:12:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:12:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:12:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:12:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:12:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:12:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:12:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:12:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:12:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:12:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:12:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:12:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:12:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:12:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:12:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:12:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:12:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:12:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:12:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:12:56Z" 2023-10-18T21:12:56.887 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:12:56.888 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:12:56Z" } 2023-10-18T21:12:56.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:12:56.889 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:12:56.890 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:12:56.890 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:01.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:01.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:01.892 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:01.952 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:01.953 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:02.000 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:02.000 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:02.002 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:02.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:02.002 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:02.002 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:02.003 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:02.003 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:02.003 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:02.003 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:02.003 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:02.004 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:02.004 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:02.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:02.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:02.004 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:02.005 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:02.005 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:02.005 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:02.005 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:02.005 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:02.006 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:02.006 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:02.006 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:02.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:02.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:02.007 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:02.007 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:02.007 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:02.007 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:02.007 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:02.008 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:02.008 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:02.008 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:02.008 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:02.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:02.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:02.009 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:02.009 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:02.009 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:02.010 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:02.010 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:02.010 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:02.010 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:02.011 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:02.011 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:02.011 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:02.011 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:02.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:02.012 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:02.012 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:02.012 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:02.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:02.012 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:01Z" 2023-10-18T21:13:02.013 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:02.014 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:01Z" } 2023-10-18T21:13:02.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:02.015 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:02.015 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:02.015 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:07.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:07.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:07.017 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:07.076 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:07.076 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:07.124 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:07.124 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:07.125 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:07.125 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:07.125 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:07.125 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:07.128 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:07.128 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:07.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:07.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:07.128 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:07.129 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:07.129 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:07.129 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:07.129 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:07.129 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:07.130 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:07.130 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:07.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:07.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:07.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:07.131 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:07.131 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:07.131 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:07.131 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:07.132 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:07.132 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:07.132 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:07.132 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:07.132 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:07.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:07.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:07.133 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:07.133 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:07.134 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:07.134 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:07.134 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:07.134 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:07.134 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:07.135 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:07.135 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:07.135 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:07.135 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:07.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:07.136 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:07.136 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:07.136 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:07.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:07.136 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:07Z" 2023-10-18T21:13:07.137 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:07.138 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:07Z" } 2023-10-18T21:13:07.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:07.139 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:07.139 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:07.139 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:12.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:12.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:12.143 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:12.201 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:12.202 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:12.251 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:12.252 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:12.252 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:12.252 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:12.253 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:12.253 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:12.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:12.253 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:12.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:12.254 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:12.254 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:12.255 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:12.255 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:12.255 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:12.255 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:12.256 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:12.256 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:12.256 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:12.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:12.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:12.257 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:12.257 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:12.257 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:12.257 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:12.258 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:12.258 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:12.258 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:12.258 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:12.258 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:12.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:12.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:12.259 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:12.259 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:12.260 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:12.260 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:12.260 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:12.260 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:12.260 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:12.261 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:12.261 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:12.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:12.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:12.262 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:12.262 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:12.262 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:12.262 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:12.262 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:12.263 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:12.263 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:12.263 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:12.263 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:12.264 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:12.264 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:12.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:12.264 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:12.264 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:12.265 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:12.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:12.265 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:12Z" 2023-10-18T21:13:12.265 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:12.267 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:12Z" } 2023-10-18T21:13:12.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:12.268 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:12.268 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:12.268 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:17.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:17.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:17.271 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:17.332 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:17.332 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:17.380 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:17.380 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:17.381 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:17.381 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:17.381 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:17.381 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:17.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:17.382 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:17.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:17.382 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:17.383 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:17.383 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:17.383 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:17.383 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:17.383 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:17.384 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:17.384 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:17.384 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:17.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:17.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:17.385 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:17.385 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:17.385 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:17.385 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:17.385 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:17.386 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:17.386 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:17.386 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:17.386 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:17.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:17.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:17.387 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:17.387 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:17.387 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:17.388 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:17.388 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:17.388 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:17.388 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:17.388 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:17.389 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:17.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:17.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:17.389 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:17.390 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:17.390 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:17.390 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:17.390 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:17.390 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:17.391 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:17.391 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:17.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:17.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:17.391 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:17.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:17.392 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:17.392 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:17.392 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:17.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:17.393 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:17Z" 2023-10-18T21:13:17.393 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:17.394 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:17Z" } 2023-10-18T21:13:17.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:17.395 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:17.395 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:17.396 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:22.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:22.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:22.399 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:22.459 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:22.460 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:22.508 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:22.509 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:22.509 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:22.509 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:22.509 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:22.510 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:22.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:22.510 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:22.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:22.511 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:22.511 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:22.511 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:22.511 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:22.512 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:22.512 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:22.512 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:22.512 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:22.512 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:22.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:22.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:22.513 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:22.513 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:22.516 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:22.516 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:22.516 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:22.516 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:22.517 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:22.517 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:22.517 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:22.517 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:22.517 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:22.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:22.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:22.518 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:22.518 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:22.519 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:22.519 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:22.519 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:22.519 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:22.520 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:22.520 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:22.520 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:22.520 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:22.520 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:22.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:22.521 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:22.521 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:22.522 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:22.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:22.522 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:22Z" 2023-10-18T21:13:22.522 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:22.524 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:22Z" } 2023-10-18T21:13:22.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:22.527 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:22.527 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:22.527 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:27.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:27.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:27.530 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:27.589 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:27.589 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:27.638 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:27Z" 2023-10-18T21:13:27.651 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:27.652 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:27Z" } 2023-10-18T21:13:27.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:27.653 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:27.654 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:27.654 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:32.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:32.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:32.656 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:32.716 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:32.717 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:32.767 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:32.767 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:32.767 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:32.768 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:32.768 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:32.768 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:32.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:32.768 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:32.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:32.769 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:32.769 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:32.769 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:32.769 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:32.770 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:32.770 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:32.770 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:32.770 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:32.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:32.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:32.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:32.771 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:32.771 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:32.772 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:32.772 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:32.772 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:32.772 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:32.773 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:32.773 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:32.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:32.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:32.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:32.774 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:32.774 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:32.774 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:32.774 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:32.774 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:32.775 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:32.775 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:32.775 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:32.775 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:32.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:32.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:32.776 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:32.776 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:32.776 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:32.776 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:32.777 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:32.777 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:32.777 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:32.777 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:32.777 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:32.778 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:32.778 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:32.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:32.778 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:32.779 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:32.779 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:32.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:32.779 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:32Z" 2023-10-18T21:13:32.780 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:32.781 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:32Z" } 2023-10-18T21:13:32.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:32.782 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:32.782 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:32.783 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:37.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:37.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:37.785 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:37.844 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:37.844 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:37.893 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:37.893 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:37.893 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:37.893 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:37.894 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:37.894 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:37.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:37.894 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:37.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:37.895 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:37.895 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:37.895 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:37.895 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:37.896 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:37.896 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:37.896 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:37.896 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:37.896 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:37.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:37.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:37.897 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:37.897 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:37.897 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:37.898 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:37.898 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:37.899 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:37.899 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:37.899 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:37.900 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:37.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:37.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:37.900 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:37.900 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:37.901 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:37.901 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:37.901 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:37.901 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:37.901 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:37.902 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:37.902 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:37.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:37.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:37.903 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:37.903 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:37.903 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:37.903 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:37.904 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:37.904 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:37.904 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:37.904 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:37.904 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:37.905 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:37.905 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:37.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:37.905 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:37.906 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:37.906 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:37.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:37.906 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:37Z" 2023-10-18T21:13:37.907 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:37.908 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:37Z" } 2023-10-18T21:13:37.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:37.908 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:37.909 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:37.909 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:42.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:42.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:42.911 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:42.972 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:42.972 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:43.020 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:43.021 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:43.021 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:43.021 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:43.021 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:43.021 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:43.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:43.022 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:43.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:43.022 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:43.023 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:43.023 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:43.023 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:43.023 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:43.023 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:43.024 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:43.024 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:43.024 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:43.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:43.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:43.024 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:43.025 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:43.025 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:43.025 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:43.025 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:43.025 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:43.026 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:43.026 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:43.026 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:43.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:43.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:43.027 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:43.027 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:43.027 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:43.027 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:43.027 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:43.028 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:43.028 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:43.028 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:43.028 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:43.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:43.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:43.029 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:43.029 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:43.029 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:43.029 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:43.030 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:43.030 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:43.030 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:43.030 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:43.031 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:43.031 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:43.031 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:43.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:43.031 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:43.031 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:43.032 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:43.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:43.032 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:43Z" 2023-10-18T21:13:43.032 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:43.033 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:43Z" } 2023-10-18T21:13:43.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:43.034 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:43.035 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:43.035 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:48.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:48.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:48.038 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:48.099 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:48.100 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:48.149 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:48.149 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:48.149 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:48.149 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:48.150 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:48.150 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:48.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:48.150 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:48.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:48.151 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:48.151 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:48.151 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:48.151 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:48.151 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:48.152 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:48.152 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:48.152 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:48.152 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:48.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:48.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:48.153 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:48.153 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:48.153 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:48.153 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:48.154 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:48.154 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:48.154 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:48.154 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:48.155 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:48.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:48.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:48.155 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:48.155 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:48.155 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:48.156 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:48.156 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:48.156 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:48.156 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:48.156 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:48.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:48.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:48.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:48.157 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:48.157 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:48.158 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:48.158 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:48.158 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:48.158 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:48.158 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:48.159 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:48.159 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:48.159 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:48.159 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:48.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:48.160 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:48.160 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:48.160 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:48.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:48.160 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:48Z" 2023-10-18T21:13:48.161 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:48.161 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:48Z" } 2023-10-18T21:13:48.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:48.162 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:48.163 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:48.163 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:53.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:53.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:53.166 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:53.225 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:53.225 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:53.274 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:53.274 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:53.275 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:53.275 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:53.275 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:53.275 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:53.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:53.276 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:53.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:53.276 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:53.276 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:53.276 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:53.277 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:53.277 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:53.277 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:53.277 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:53.278 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:53.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:53.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:53.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:53.279 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:53.279 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:53.279 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:53.279 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:53.279 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:53.280 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:53.280 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:53.280 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:53.280 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:53.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:53.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:53.281 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:53.281 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:53.281 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:53.282 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:53.282 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:53.282 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:53.282 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:53.282 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:53.283 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:53.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:53.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:53.283 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:53.284 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:53.284 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:53.284 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:53.284 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:53.284 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:53.285 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:53.285 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:53.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:53.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:53.285 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:53.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:53.286 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:53.286 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:53.286 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:53.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:53.287 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:53Z" 2023-10-18T21:13:53.287 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:53.288 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:53Z" } 2023-10-18T21:13:53.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:53.289 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:53.289 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:53.289 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:13:58.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:58.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:13:58.291 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:13:58.351 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:13:58.351 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:13:58.401 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:13:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:13:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:13:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:13:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:13:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:13:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:13:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:13:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:13:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:13:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:13:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:13:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:13:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:13:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:13:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:13:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:13:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:13:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:13:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:13:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:13:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:13:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:13:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:13:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:13:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:13:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:13:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:13:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:13:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:13:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:13:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:13:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:13:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:13:58Z" 2023-10-18T21:13:58.415 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:13:58.416 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:13:58Z" } 2023-10-18T21:13:58.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:13:58.417 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:13:58.417 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:13:58.417 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:03.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:03.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:03.420 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:03.482 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:03.482 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:03.534 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:03.534 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:03.535 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:03.535 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:03.535 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:03.535 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:03.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:03.536 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:03.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:03.537 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:03.537 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:03.537 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:03.538 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:03.538 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:03.539 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:03.539 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:03.539 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:03.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:03.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:03.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:03.541 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:03.541 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:03.541 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:03.541 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:14:03.541 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:03.542 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:03.542 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:03.542 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:14:03.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:03.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:03.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:03.543 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:03.543 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:03.543 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:03.544 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:03.544 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:03.544 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:03.544 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:03.544 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:03.545 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:03.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:03.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:03.545 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:03.545 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:03.546 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:03.546 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:03.546 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:03.546 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:03.546 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:03.547 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:03.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:03.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:03.547 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:03.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:03.548 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:03.548 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:03.548 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:03.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:03.549 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:03Z" 2023-10-18T21:14:03.549 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:03.551 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:03Z" } 2023-10-18T21:14:03.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:03.553 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:03.553 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:03.554 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:08.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:08.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:08.557 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:08.616 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:08.617 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:08.665 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:08.665 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:08.665 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:08.665 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:08.666 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:08.666 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:08.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:08.666 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:08.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:08.667 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:08.667 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:08.667 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:08.668 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:08.668 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:08.668 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:08.668 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:08.668 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:08.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:08.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:08.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:08.669 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:08.669 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:08.670 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:08.670 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:14:08.670 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:08.670 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:08.671 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:08.671 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:14:08.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:08.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:08.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:08.672 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:08.672 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:08.672 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:08.672 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:08.672 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:08.673 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:08.673 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:08.673 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:08.673 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:08.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:08.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:08.674 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:08.674 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:08.674 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:08.675 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:08.675 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:08.675 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:08.675 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:08.675 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:08.676 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:08.676 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:08.676 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:08.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:08.676 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:08.677 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:08.677 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:08.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:08.677 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:08Z" 2023-10-18T21:14:08.678 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:08.679 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:08Z" } 2023-10-18T21:14:08.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:08.680 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:08.680 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:08.681 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:13.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:13.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:13.683 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:13.742 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:13.743 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:13.790 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:13.791 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:13.791 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:13.791 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:13.791 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:13.792 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:13.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:13.792 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:13.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:13.793 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:13.793 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:13.793 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:13.793 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:13.793 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:13.794 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:13.794 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:13.794 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:13.794 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:13.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:13.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:13.795 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:13.795 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:13.795 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:13.796 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:14:13.796 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:13.796 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:13.796 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:13.797 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:14:13.797 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:13.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:13.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:13.797 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:13.798 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:13.798 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:13.798 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:13.798 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:13.798 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:13.799 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:13.799 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:13.799 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:13.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:13.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:13.800 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:13.800 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:13.800 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:13.801 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:13.801 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:13.801 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:13.801 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:13.802 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:13.802 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:13.802 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:13.803 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:13.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:13.804 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:13.804 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:13.804 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:13.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:13.805 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:13Z" 2023-10-18T21:14:13.805 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:13.807 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:13Z" } 2023-10-18T21:14:13.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:13.809 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:13.810 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:13.810 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:18.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:18.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:18.813 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:18.871 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:18.871 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:18.920 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:18.920 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:18.920 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:18.921 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:18.921 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:18.921 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:18.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:18.922 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:18.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:18.922 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:18.922 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:18.923 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:18.923 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:18.923 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:18.923 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:18.923 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:18.924 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:18.924 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:18.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:18.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:18.925 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:18.925 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:18.925 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:18.925 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045379", 2023-10-18T21:14:18.926 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:18.926 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:18.926 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:18.926 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:11:11.762358Z" 2023-10-18T21:14:18.926 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:18.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:18.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:18.927 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:18.927 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:18.927 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:18.928 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:18.928 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:18.928 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:18.929 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:18.929 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:18.929 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:18.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:18.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:18.930 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:18.930 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:18.930 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:18.930 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:18.931 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:18.931 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:18.931 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:18.931 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:18.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:18.932 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:18.932 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:18.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:18.932 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:18.933 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:18.933 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:18.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:18.933 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:18Z" 2023-10-18T21:14:18.934 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:18.935 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045379", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:11:11.762358Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:18Z" } 2023-10-18T21:14:18.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045379', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:11:11.762358Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:18.936 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:18.936 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045379', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:18.937 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=1 master=00000000000000000000:00000000000000045519 target=00000000000000000000:00000000000000045379, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:23.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:23.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:23.939 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:23.999 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:23.999 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:24.047 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:24.047 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:24.048 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:24.048 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:24.048 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:24.048 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:24.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:24.049 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:24.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:24.049 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:24.049 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:24.050 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:24.050 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:24.050 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:24.050 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:24.050 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:24.051 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:24.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:24.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:24.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:24.051 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:24.052 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:24.052 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:24.052 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:14:24.052 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:24.052 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:24.053 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:24.053 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:14:24.053 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:24.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:24.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:24.054 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:24.054 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:24.054 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:24.054 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:24.055 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:24.055 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:24.055 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:24.055 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:24.055 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:24.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:24.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:24.056 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:24.056 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:24.057 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:24.057 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:24.057 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:24.057 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:24.057 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:24.057 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:24.058 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:24.058 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:24.058 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:24.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:24.059 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:24.059 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:24.059 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:24.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:24.059 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:24Z" 2023-10-18T21:14:24.060 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:24.061 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:24Z" } 2023-10-18T21:14:24.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:24.062 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:24.062 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:24.062 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:29.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:29.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:29.065 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:29.123 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:29.124 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:29.172 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:29.173 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:29.173 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:29.173 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:29.173 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:29.174 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:29.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:29.174 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:29.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:29.175 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:29.175 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:29.175 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:29.175 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:29.175 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:29.176 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:29.176 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:29.176 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:29.176 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:29.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:29.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:29.177 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:29.177 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:29.177 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:29.177 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:14:29.178 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:29.178 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:29.178 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:29.178 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:14:29.179 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:29.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:29.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:29.179 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:29.179 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:29.180 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:29.180 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:29.180 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:29.180 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:29.180 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:29.181 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:29.181 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:29.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:29.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:29.181 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:29.182 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:29.182 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:29.182 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:29.182 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:29.182 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:29.183 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:29.183 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:29.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:29.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:29.184 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:29.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:29.184 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:29.184 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:29.184 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:29.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:29.185 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:29Z" 2023-10-18T21:14:29.185 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:29.186 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:29Z" } 2023-10-18T21:14:29.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:29.187 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:29.187 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:29.188 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:34.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:34.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:34.190 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:34.249 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:34.249 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:34.297 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:34.297 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:34.298 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:34.298 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:34.298 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:34.298 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:34.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:34.299 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:34.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:34.299 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:34.299 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:34.300 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:34.300 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:34.300 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:34.300 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:34.300 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:34.301 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:34.301 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:34.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:34.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:34.301 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:34.301 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:34.302 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:34.302 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:14:34.302 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:34.302 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:34.303 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:34.303 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:14:34.303 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:34.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:34.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:34.303 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:34.304 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:34.304 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:34.304 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:34.304 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:34.304 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:34.305 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:34.305 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:34.305 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:34.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:34.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:34.306 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:34.306 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:34.306 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:34.306 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:34.306 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:34.307 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:34.307 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:34.307 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:34.307 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:34.307 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:34.307 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:34.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:34.308 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:34.308 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:34.308 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:34.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:34.309 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:34Z" 2023-10-18T21:14:34.309 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:34.310 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:34Z" } 2023-10-18T21:14:34.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:34.311 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:34.311 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:34.311 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:39.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:39.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:39.314 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:39.373 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:39.373 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:39.421 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:14:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:14:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:39Z" 2023-10-18T21:14:39.434 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:39.435 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:39Z" } 2023-10-18T21:14:39.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:39.436 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:39.436 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:39.437 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:44.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:44.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:44.439 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:44.498 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:44.498 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:44.550 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:44.550 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:44.550 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:44.550 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:44.551 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:44.551 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:44.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:44.551 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:44.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:44.552 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:44.552 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:44.552 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:44.552 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:44.552 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:44.553 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:44.553 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:44.553 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:44.553 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:44.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:44.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:44.554 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:44.554 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:44.554 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:44.554 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:14:44.555 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:44.555 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:44.555 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:44.555 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:14:44.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:44.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:44.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:44.556 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:44.556 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:44.556 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:44.557 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:44.557 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:44.557 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:44.557 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:44.557 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:44.558 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:44.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:44.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:44.558 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:44.558 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:44.559 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:44.559 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:44.559 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:44.559 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:44.559 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:44.560 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:44.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:44.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:44.560 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:44.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:44.561 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:44.561 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:44.561 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:44.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:44.561 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:44Z" 2023-10-18T21:14:44.562 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:44.562 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:44Z" } 2023-10-18T21:14:44.563 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:44.564 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:44.564 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:44.564 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:49.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:49.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:49.566 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:49.626 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:49.626 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:49.674 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:49.674 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:49.674 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:49.675 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:49.675 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:49.675 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:49.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:49.675 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:49.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:49.676 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:49.676 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:49.676 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:49.677 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044184", 2023-10-18T21:14:49.677 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:49.677 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:49.677 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:49.677 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:16.885046Z" 2023-10-18T21:14:49.678 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:49.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:49.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:49.678 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:49.678 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:14:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:49.679 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:14:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:49.680 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:49.681 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:49.681 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:49.681 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:49.681 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:49.681 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:49.682 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:49.682 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:49.682 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:49.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:49.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:49.683 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:49.683 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:49.683 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:49.683 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000050973", 2023-10-18T21:14:49.684 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:49.684 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:49.684 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:49.684 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:10:40.767903Z" 2023-10-18T21:14:49.684 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:49.685 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:49.685 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:49.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:49.685 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:49.685 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:49.686 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:49.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:49.686 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:49Z" 2023-10-18T21:14:49.686 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:49.687 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:04:16.885046Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050973", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:10:40.767903Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:49Z" } 2023-10-18T21:14:49.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:04:16.885046Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050973', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:10:40.767903Z'}}] 2023-10-18T21:14:49.688 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:49.688 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044184', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000050973'} 2023-10-18T21:14:49.689 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044326 target=00000000000000000000:00000000000000044184, shard=3 master=00000000000000000000:00000000000000051253 target=00000000000000000000:00000000000000050973 2023-10-18T21:14:54.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:54.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:54.691 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:54.750 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:54.750 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:54.799 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:54.800 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:54.800 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:54.800 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:54.800 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:14:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:54.801 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:54.802 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:54.802 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:54.802 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:14:54.802 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:54.802 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:14:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:54.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:54.804 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:14:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:14:54.805 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:54.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:54.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:54.806 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:54.806 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:54.806 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:14:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:14:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:54.807 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:14:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:54.808 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:54.809 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:14:54.809 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051253", 2023-10-18T21:14:54.809 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:54.809 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:14:54.809 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:54.810 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.187035Z" 2023-10-18T21:14:54.810 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:54.810 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:54.810 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:14:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:14:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:54.811 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:54Z" 2023-10-18T21:14:54.812 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:54.813 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051253", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:12:51.187035Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:14:54Z" } 2023-10-18T21:14:54.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051253', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.187035Z'}}] 2023-10-18T21:14:54.814 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:54.814 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051253'} 2023-10-18T21:14:54.814 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:14:54.815 DEBUG:boto:path=/bofrnf-68/ 2023-10-18T21:14:54.815 DEBUG:boto:auth_path=/bofrnf-68/ 2023-10-18T21:14:54.816 DEBUG:boto:Method: HEAD 2023-10-18T21:14:54.816 DEBUG:boto:Path: /bofrnf-68/ 2023-10-18T21:14:54.816 DEBUG:boto:Data: 2023-10-18T21:14:54.817 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.817 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.818 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.818 DEBUG:boto:Params: {} 2023-10-18T21:14:54.819 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:14:54.819 DEBUG:boto:Token: None 2023-10-18T21:14:54.820 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-68/ 2023-10-18T21:14:54.820 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3Qwp5J7i1lL5c4tBu6mWr8Ydb5A= 2023-10-18T21:14:54.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Qwp5J7i1lL5c4tBu6mWr8Ydb5A=', 'Content-Length': '0'} 2023-10-18T21:14:54.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fdf9ebbc4dfd157-0065304ace-237a-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.827 DEBUG:boto:path=/bofrnf-68/obj1 2023-10-18T21:14:54.828 DEBUG:boto:auth_path=/bofrnf-68/obj1 2023-10-18T21:14:54.828 DEBUG:boto:Method: HEAD 2023-10-18T21:14:54.828 DEBUG:boto:Path: /bofrnf-68/obj1 2023-10-18T21:14:54.829 DEBUG:boto:Data: 2023-10-18T21:14:54.829 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.830 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.830 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.830 DEBUG:boto:Params: {} 2023-10-18T21:14:54.831 DEBUG:boto:Token: None 2023-10-18T21:14:54.831 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-68/obj1 2023-10-18T21:14:54.832 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FP1EvcMyKl15S6gtP6f8Yc0fOUU= 2023-10-18T21:14:54.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FP1EvcMyKl15S6gtP6f8Yc0fOUU=', 'Content-Length': '0'} 2023-10-18T21:14:54.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-68:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2ffb0e47570c3ef-0065304ace-237a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.835 DEBUG:boto:path=/bofrnf-68/obj1 2023-10-18T21:14:54.835 DEBUG:boto:auth_path=/bofrnf-68/obj1 2023-10-18T21:14:54.836 DEBUG:boto:Method: GET 2023-10-18T21:14:54.836 DEBUG:boto:Path: /bofrnf-68/obj1 2023-10-18T21:14:54.837 DEBUG:boto:Data: 2023-10-18T21:14:54.837 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.838 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.838 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.838 DEBUG:boto:Params: {} 2023-10-18T21:14:54.839 DEBUG:boto:Token: None 2023-10-18T21:14:54.839 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-68/obj1 2023-10-18T21:14:54.840 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:K81oVGWsjQ3WzT96Uois1K2K+zI= 2023-10-18T21:14:54.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:K81oVGWsjQ3WzT96Uois1K2K+zI=', 'Content-Length': '0'} 2023-10-18T21:14:54.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-68:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e321aa2761d0df62-0065304ace-237a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.843 DEBUG:boto:path=/bofrnf-68/obj2 2023-10-18T21:14:54.843 DEBUG:boto:auth_path=/bofrnf-68/obj2 2023-10-18T21:14:54.844 DEBUG:boto:Method: HEAD 2023-10-18T21:14:54.844 DEBUG:boto:Path: /bofrnf-68/obj2 2023-10-18T21:14:54.845 DEBUG:boto:Data: 2023-10-18T21:14:54.845 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.845 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.846 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.846 DEBUG:boto:Params: {} 2023-10-18T21:14:54.847 DEBUG:boto:Token: None 2023-10-18T21:14:54.847 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-68/obj2 2023-10-18T21:14:54.847 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xog3Ah75rF+HVrnN5Mn/vHbEWeU= 2023-10-18T21:14:54.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xog3Ah75rF+HVrnN5Mn/vHbEWeU=', 'Content-Length': '0'} 2023-10-18T21:14:54.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-68:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e501ee5baf8b75fc-0065304ace-237a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.850 DEBUG:boto:path=/bofrnf-68/obj2 2023-10-18T21:14:54.851 DEBUG:boto:auth_path=/bofrnf-68/obj2 2023-10-18T21:14:54.851 DEBUG:boto:Method: GET 2023-10-18T21:14:54.852 DEBUG:boto:Path: /bofrnf-68/obj2 2023-10-18T21:14:54.852 DEBUG:boto:Data: 2023-10-18T21:14:54.852 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.853 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.853 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.854 DEBUG:boto:Params: {} 2023-10-18T21:14:54.854 DEBUG:boto:Token: None 2023-10-18T21:14:54.855 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-68/obj2 2023-10-18T21:14:54.855 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0Duk12mLJDpzN4aYrz6Vy9/rzbY= 2023-10-18T21:14:54.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0Duk12mLJDpzN4aYrz6Vy9/rzbY=', 'Content-Length': '0'} 2023-10-18T21:14:54.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-68:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000360f20c6de260945-0065304ace-237a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.898 DEBUG:boto:path=/bofrnf-68/obj3 2023-10-18T21:14:54.899 DEBUG:boto:auth_path=/bofrnf-68/obj3 2023-10-18T21:14:54.899 DEBUG:boto:Method: HEAD 2023-10-18T21:14:54.899 DEBUG:boto:Path: /bofrnf-68/obj3 2023-10-18T21:14:54.900 DEBUG:boto:Data: 2023-10-18T21:14:54.900 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.901 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.901 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.901 DEBUG:boto:Params: {} 2023-10-18T21:14:54.902 DEBUG:boto:Token: None 2023-10-18T21:14:54.902 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-68/obj3 2023-10-18T21:14:54.903 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:+H98HCP971Q7egx0U/a/5mn7RYE= 2023-10-18T21:14:54.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+H98HCP971Q7egx0U/a/5mn7RYE=', 'Content-Length': '0'} 2023-10-18T21:14:54.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-68:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082b6ddccdc5e143b-0065304ace-237a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.906 DEBUG:boto:path=/bofrnf-68/obj3 2023-10-18T21:14:54.906 DEBUG:boto:auth_path=/bofrnf-68/obj3 2023-10-18T21:14:54.907 DEBUG:boto:Method: GET 2023-10-18T21:14:54.907 DEBUG:boto:Path: /bofrnf-68/obj3 2023-10-18T21:14:54.907 DEBUG:boto:Data: 2023-10-18T21:14:54.908 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.908 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.909 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.909 DEBUG:boto:Params: {} 2023-10-18T21:14:54.909 DEBUG:boto:Token: None 2023-10-18T21:14:54.910 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-68/obj3 2023-10-18T21:14:54.910 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7ah1D3xI8l5AmwDPb3ZhS2yrd3o= 2023-10-18T21:14:54.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7ah1D3xI8l5AmwDPb3ZhS2yrd3o=', 'Content-Length': '0'} 2023-10-18T21:14:54.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:12:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-68:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aae338652b8cb8ab-0065304ace-237a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.914 DEBUG:boto:path=/bofrnf-69/ 2023-10-18T21:14:54.914 DEBUG:boto:auth_path=/bofrnf-69/ 2023-10-18T21:14:54.914 DEBUG:boto:Method: HEAD 2023-10-18T21:14:54.915 DEBUG:boto:Path: /bofrnf-69/ 2023-10-18T21:14:54.915 DEBUG:boto:Data: 2023-10-18T21:14:54.916 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.916 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.916 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.917 DEBUG:boto:Params: {} 2023-10-18T21:14:54.917 DEBUG:boto:Token: None 2023-10-18T21:14:54.918 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-69/ 2023-10-18T21:14:54.918 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dnJQaoC8IHjQxO2FWIiHDf6eJxo= 2023-10-18T21:14:54.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dnJQaoC8IHjQxO2FWIiHDf6eJxo=', 'Content-Length': '0'} 2023-10-18T21:14:54.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077f7a2ede5f6a618-0065304ace-237a-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.922 DEBUG:boto:path=/bofrnf-69/obj1 2023-10-18T21:14:54.922 DEBUG:boto:auth_path=/bofrnf-69/obj1 2023-10-18T21:14:54.923 DEBUG:boto:Method: HEAD 2023-10-18T21:14:54.923 DEBUG:boto:Path: /bofrnf-69/obj1 2023-10-18T21:14:54.923 DEBUG:boto:Data: 2023-10-18T21:14:54.924 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.924 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.925 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.925 DEBUG:boto:Params: {} 2023-10-18T21:14:54.925 DEBUG:boto:Token: None 2023-10-18T21:14:54.926 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-69/obj1 2023-10-18T21:14:54.926 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:QNlgLgAwakLYT16VqSmhXWLDG84= 2023-10-18T21:14:54.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QNlgLgAwakLYT16VqSmhXWLDG84=', 'Content-Length': '0'} 2023-10-18T21:14:54.928 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000cba1c8b8a67dd031-0065304ace-237a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.929 DEBUG:boto:path=/bofrnf-69/obj2 2023-10-18T21:14:54.929 DEBUG:boto:auth_path=/bofrnf-69/obj2 2023-10-18T21:14:54.930 DEBUG:boto:Method: HEAD 2023-10-18T21:14:54.930 DEBUG:boto:Path: /bofrnf-69/obj2 2023-10-18T21:14:54.931 DEBUG:boto:Data: 2023-10-18T21:14:54.931 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.931 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.932 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.932 DEBUG:boto:Params: {} 2023-10-18T21:14:54.933 DEBUG:boto:Token: None 2023-10-18T21:14:54.933 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-69/obj2 2023-10-18T21:14:54.933 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:2bNJTua5/x8ZKo1QYyigfSjaAGo= 2023-10-18T21:14:54.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2bNJTua5/x8ZKo1QYyigfSjaAGo=', 'Content-Length': '0'} 2023-10-18T21:14:54.935 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002775d843b5bbb9c8-0065304ace-237a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.936 DEBUG:boto:path=/bofrnf-69/obj3 2023-10-18T21:14:54.936 DEBUG:boto:auth_path=/bofrnf-69/obj3 2023-10-18T21:14:54.937 DEBUG:boto:Method: HEAD 2023-10-18T21:14:54.937 DEBUG:boto:Path: /bofrnf-69/obj3 2023-10-18T21:14:54.937 DEBUG:boto:Data: 2023-10-18T21:14:54.938 DEBUG:boto:Headers: {} 2023-10-18T21:14:54.938 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:14:54.939 DEBUG:boto:Port: 8001 2023-10-18T21:14:54.939 DEBUG:boto:Params: {} 2023-10-18T21:14:54.939 DEBUG:boto:Token: None 2023-10-18T21:14:54.940 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:54 GMT /bofrnf-69/obj3 2023-10-18T21:14:54.940 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qeghgi9wjDzk2YumMNauEPRmNDI= 2023-10-18T21:14:54.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qeghgi9wjDzk2YumMNauEPRmNDI=', 'Content-Length': '0'} 2023-10-18T21:14:54.942 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b5b65de5d8c69e10-0065304ace-237a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:14:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:54.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:54.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:54.944 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:54.996 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:54.996 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:55.041 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:14:55.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.043 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:55.096 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:55.096 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:55.141 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:14:55.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.142 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:55.191 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:55.191 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:55.236 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:14:55.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.237 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:55.285 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:55.286 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:55.337 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:14:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 17, 2023-10-18T21:14:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:14:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:14:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:14:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:14:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:14:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:14:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:14:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:14:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:14:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:14:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:14:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:14:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:14:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:14:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:14:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:14:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:14:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:14:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:14:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:14:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:14:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:14:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:14:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:14:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:14:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:14:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:14:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:14:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:14:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:14:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:14:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:14:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:14:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:14:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:14:55.346 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:14:55.346 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:14:55.346 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:14:55.346 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:14:55.346 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:14:55.347 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:14:55.347 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:14:55.347 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:14:55.347 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:14:55.347 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:14:55.348 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.348 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.348 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.348 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:14:55.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.348 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:14:55.349 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:14:55.349 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:14:55.349 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:14:55.349 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.350 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.350 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:14:55.350 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:14:55.350 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:14:55.351 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [ 2023-10-18T21:14:55.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.351 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-group", 2023-10-18T21:14:55.351 INFO:teuthology.orchestra.run.smithi082.stdout: "data_flow": { 2023-10-18T21:14:55.352 INFO:teuthology.orchestra.run.smithi082.stdout: "symmetrical": [ 2023-10-18T21:14:55.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.352 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-flow1", 2023-10-18T21:14:55.352 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:14:55.352 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:14:55.353 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:14:55.353 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.353 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.354 INFO:teuthology.orchestra.run.smithi082.stdout: "pipes": [ 2023-10-18T21:14:55.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.354 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe", 2023-10-18T21:14:55.354 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:14:55.354 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:14:55.354 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:14:55.355 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:14:55.355 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:14:55.355 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.355 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:14:55.356 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:14:55.356 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:14:55.356 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:14:55.356 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:14:55.356 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.357 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:14:55.357 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:14:55.357 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:14:55.357 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:14:55.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.358 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:14:55.358 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:14:55.358 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:14:55.359 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:14:55.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.359 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.359 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "allowed" 2023-10-18T21:14:55.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.360 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.360 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:14:55.360 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:14:55.360 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.361 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:14:55.361 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:14:55.361 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:14:55.361 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:14:55.362 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:14:55.362 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:14:55.362 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.362 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:14:55.362 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:14:55.363 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:14:55.363 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:14:55.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.363 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:14:55.363 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:14:55.364 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:14:55.364 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:14:55.364 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.364 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:14:55.365 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:14:55.365 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:14:55.365 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:14:55.365 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:14:55.365 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:14:55.365 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:14:55.366 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:14:55.366 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:14:55.366 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:14:55.366 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:14:55.366 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.367 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:14:55.367 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:14:55.367 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:14:55.368 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:14:55.368 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.368 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:14:55.368 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:14:55.368 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:14:55.369 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:14:55.369 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:14:55.369 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:14:55.369 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:14:55.369 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:14:55.369 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:14:55.370 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:14:55.370 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:14:55.370 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.370 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.370 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.371 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:14:55.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.371 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:14:55.371 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:14:55.371 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:14:55.371 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:14:55.372 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.372 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.372 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:14:55.372 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:14:55.372 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:14:55.373 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:14:55.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.373 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:14:55.373 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:14:55.373 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.374 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.374 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:14:55.374 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:14:55.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.374 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:14:55.374 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:14:55.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.375 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:14:55.375 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:14:55.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.376 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:14:55.377 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:14:55.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:55.377 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:14:55.377 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:14:55.378 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.378 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:55.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.378 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:14:55.378 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:14:55.379 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:14:55.379 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:14:55.379 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:14:55.379 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:14:55.379 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:14:55.379 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:14:55.380 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:14:55.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.380 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:14:55.380 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:14:55.380 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:14:55.381 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:14:55.381 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:14:55.381 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:14:55.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.382 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:14:55.383 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:14:55.383 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:14:55.383 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:14:55.384 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:14:55.384 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:14:55.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.385 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:14:55.385 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:14:55.385 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:14:55.386 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:14:55.386 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:14:55.386 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:14:55.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.387 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:14:55.387 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:14:55.387 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:14:55.388 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:14:55.388 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:14:55.388 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:14:55.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:55.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:55.389 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:14:55.389 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:14:55.389 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:14:55.389 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:55.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.391 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:55.443 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:55.443 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:55.510 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:14:55.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.511 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket bofrnf-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:55.580 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:55.580 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:55.668 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:14:55.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.670 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket bofrnf-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:55.742 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:55.742 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:55.831 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-18T21:14:55.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.833 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=bofrnf-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:55.906 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:55.906 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:55.978 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-18T21:14:55.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:55.979 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket bofrnf-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:56.036 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:56.036 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:56.094 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-18T21:14:56.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:56.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:14:56.095 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket bofrnf-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:14:56.142 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:56.142 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:56.187 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-18T21:14:56.187 DEBUG:boto:path=/bofrnf-68/ 2023-10-18T21:14:56.188 DEBUG:boto:auth_path=/bofrnf-68/ 2023-10-18T21:14:56.188 DEBUG:boto:Method: HEAD 2023-10-18T21:14:56.189 DEBUG:boto:Path: /bofrnf-68/ 2023-10-18T21:14:56.189 DEBUG:boto:Data: 2023-10-18T21:14:56.189 DEBUG:boto:Headers: {} 2023-10-18T21:14:56.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:14:56.190 DEBUG:boto:Port: 8000 2023-10-18T21:14:56.190 DEBUG:boto:Params: {} 2023-10-18T21:14:56.191 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:14:56.191 DEBUG:boto:Token: None 2023-10-18T21:14:56.192 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:56 GMT /bofrnf-68/ 2023-10-18T21:14:56.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:NDclmjd9/oYztk1dlGVDOBDT8Kk= 2023-10-18T21:14:56.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NDclmjd9/oYztk1dlGVDOBDT8Kk=', 'Content-Length': '0'} 2023-10-18T21:14:56.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab581fb59fb5c7e8-0065304ad0-2486-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:14:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:56.199 DEBUG:boto:path=/bofrnf-68/obj3 2023-10-18T21:14:56.199 DEBUG:boto:auth_path=/bofrnf-68/obj3 2023-10-18T21:14:56.199 DEBUG:boto:Method: PUT 2023-10-18T21:14:56.200 DEBUG:boto:Path: /bofrnf-68/obj3 2023-10-18T21:14:56.200 DEBUG:boto:Data: 2023-10-18T21:14:56.201 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:14:56.201 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:14:56.201 DEBUG:boto:Port: 8000 2023-10-18T21:14:56.202 DEBUG:boto:Params: {} 2023-10-18T21:14:56.202 DEBUG:boto:Token: None 2023-10-18T21:14:56.202 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:14:56 GMT /bofrnf-68/obj3 2023-10-18T21:14:56.203 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:fMip9hZ70WVwY+YbfmnZo5IxED0= 2023-10-18T21:14:56.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:14:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:fMip9hZ70WVwY+YbfmnZo5IxED0='} 2023-10-18T21:14:56.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae6ee876ad0695dd-0065304ad0-2486-a1'), ('Date', 'Wed, 18 Oct 2023 21:14:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:56.213 DEBUG:boto:path=/bofrnf-69/ 2023-10-18T21:14:56.213 DEBUG:boto:auth_path=/bofrnf-69/ 2023-10-18T21:14:56.213 DEBUG:boto:Method: HEAD 2023-10-18T21:14:56.214 DEBUG:boto:Path: /bofrnf-69/ 2023-10-18T21:14:56.214 DEBUG:boto:Data: 2023-10-18T21:14:56.215 DEBUG:boto:Headers: {} 2023-10-18T21:14:56.215 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:14:56.215 DEBUG:boto:Port: 8000 2023-10-18T21:14:56.216 DEBUG:boto:Params: {} 2023-10-18T21:14:56.216 DEBUG:boto:Token: None 2023-10-18T21:14:56.217 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:14:56 GMT /bofrnf-69/ 2023-10-18T21:14:56.217 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:d2DLlY903UdjUM1yNUr7HIQJ2So= 2023-10-18T21:14:56.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:14:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d2DLlY903UdjUM1yNUr7HIQJ2So=', 'Content-Length': '0'} 2023-10-18T21:14:56.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002ec45f240175299-0065304ad0-2486-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:14:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:56.221 DEBUG:boto:path=/bofrnf-69/obj3 2023-10-18T21:14:56.222 DEBUG:boto:auth_path=/bofrnf-69/obj3 2023-10-18T21:14:56.222 DEBUG:boto:Method: PUT 2023-10-18T21:14:56.223 DEBUG:boto:Path: /bofrnf-69/obj3 2023-10-18T21:14:56.223 DEBUG:boto:Data: 2023-10-18T21:14:56.223 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:14:56.224 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:14:56.224 DEBUG:boto:Port: 8000 2023-10-18T21:14:56.225 DEBUG:boto:Params: {} 2023-10-18T21:14:56.225 DEBUG:boto:Token: None 2023-10-18T21:14:56.225 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:14:56 GMT /bofrnf-69/obj3 2023-10-18T21:14:56.226 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dhjohHF9MwnDmkG7WkUL81n23Ik= 2023-10-18T21:14:56.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:14:56 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dhjohHF9MwnDmkG7WkUL81n23Ik='} 2023-10-18T21:14:56.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ae25fae1c66512b-0065304ad0-2486-a1'), ('Date', 'Wed, 18 Oct 2023 21:14:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:14:56.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:56.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:56.235 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:56.282 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:56.282 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:56.312 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:14:56.325 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:14:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663696.079492_1849.1", 2023-10-18T21:14:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:14:56.079492Z" 2023-10-18T21:14:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:14:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:14:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:14:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:14:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:14:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:14:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:56.328 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:14:56.329 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:14:56.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:14:56.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:56.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:14:56.331 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:14:56.379 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:14:56.380 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:14:56.408 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:14:56.407+0000 7fe9c3e47780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:14:56.409 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:14:56.408+0000 7fe9c3e47780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:14:56.422 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:14:56.422 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:14:56.423 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:14:56.423 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:14:56.423 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:14:56.423 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:14:56.424 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:14:56.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:56.424 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:14:56.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:56.424 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:14:56.425 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:56.425 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:14:56.425 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:14:56.425 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:56.425 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:14:56.426 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:56.426 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:14:56.426 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:14:56.426 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:56.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:56.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:56.427 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:14:56.427 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:56.427 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:14:56.427 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:14:56.428 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:56.428 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:56.428 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:56.428 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:14:56.428 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:14:56.429 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:56.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:56.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:56.429 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:14:56.429 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:56.430 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:14:56.430 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:14:56.430 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:56.430 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:56.430 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:56.431 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:14:56.431 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:14:56.431 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:56.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:56.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:14:56.431 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:14:56.432 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:14:56.432 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:14:56.432 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:14:56.432 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:14:56.432 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:14:56.433 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:14:56.433 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:14:56.433 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:14:56.433 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:56.433 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:14:56.434 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:14:56.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:56.434 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:14:56.434 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:14:56.434 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:14:56.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:14:56.435 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:14:56Z" 2023-10-18T21:14:56.435 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:14:56.438 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:14:56Z" } 2023-10-18T21:14:56.438 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:14:56.439 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:14:56.439 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:14:56.439 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:01.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:01.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:01.442 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:01.501 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:01.501 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:01.535 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:01.534+0000 7fc0381f6780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:01.536 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:01.535+0000 7fc0381f6780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:01.549 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:01.549 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:01.550 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:01.550 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:01.550 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:01.550 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:01.551 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:01.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:01.551 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:01.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:01.552 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:01.553 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:01.553 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:01.553 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:01.553 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:01.553 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:01.554 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:01.555 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:01.555 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:01.555 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:01.555 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:01.555 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:01.556 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:01.556 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:01.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:01.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:01.557 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:01.557 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:01.557 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:01.557 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:01.558 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:01.558 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:01.558 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:01.558 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:01.558 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:01.559 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:01.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:01.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:01.559 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:01.559 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:01.560 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:01.560 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:01.560 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:01.560 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:01.560 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:01.561 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:01.561 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:01.561 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:01.561 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:01.562 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:01.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:01.562 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:01.562 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:01.562 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:01.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:01.563 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:01Z" 2023-10-18T21:15:01.563 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:01.564 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:01Z" } 2023-10-18T21:15:01.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:01.565 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:01.565 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:01.565 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:06.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:06.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:06.568 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:06.625 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:06.625 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:06.658 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:06.657+0000 7f96ab68d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:06.659 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:06.658+0000 7f96ab68d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:06.672 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:06.673 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:06.673 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:06.673 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:06.673 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:06.674 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:06.674 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:06.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:06.674 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:06.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:06.674 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:06.675 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:06.675 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:06.675 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:06.675 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:06.675 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:06.676 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:06.676 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:06.676 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:06.676 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:06.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:06.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:06.677 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:06.677 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:06.677 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:06.677 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:06.677 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:06.678 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:06.678 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:06.678 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:06.678 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:06.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:06.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:06.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:06.679 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:06.679 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:06.680 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:06.680 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:06.680 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:06.680 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:06.680 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:06.680 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:06.681 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:06.681 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:06.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:06.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:06.681 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:06.682 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:06.682 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:06.682 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:06.682 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:06.682 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:06.683 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:06.683 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:06.683 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:06.683 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:06.684 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:06.684 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:06.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:06.684 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:06.684 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:06.684 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:06.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:06.685 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:06Z" 2023-10-18T21:15:06.685 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:06.686 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:06Z" } 2023-10-18T21:15:06.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:06.687 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:06.687 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:06.687 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:11.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:11.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:11.690 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:11.748 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:11.748 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:11.780 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:11.779+0000 7f0ff6bda780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:11.781 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:11.780+0000 7f0ff6bda780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:11.796 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:11.796 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:11.796 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:11.796 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:11.797 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:11.797 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:11.797 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:11.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:11.798 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:11.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:11.798 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:11.798 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:11.799 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:11.799 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:11.799 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:11.799 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:11.799 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:11.800 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:11.800 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:11.800 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:11.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:11.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:11.801 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:11.801 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:11.801 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:11.801 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:11.802 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:11.802 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:11.802 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:11.802 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:11.802 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:11.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:11.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:11.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:11.803 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:11.803 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:11.804 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:11.804 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:11.804 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:11.804 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:11.805 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:11.805 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:11.805 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:11.805 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:11.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:11.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:11.806 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:11.806 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:11.807 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:11.807 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:11.807 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:11.807 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:11.807 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:11.808 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:11.808 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:11.808 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:11.808 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:11.809 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:11.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:11.809 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:11.809 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:11.810 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:11.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:11.810 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:11Z" 2023-10-18T21:15:11.810 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:11.811 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:11Z" } 2023-10-18T21:15:11.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:11.812 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:11.813 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:11.813 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:16.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:16.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:16.815 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:16.872 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:16.872 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:16.905 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:16.904+0000 7f469e78c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:16.905 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:16.905+0000 7f469e78c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:16.919 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:16.919 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:16.920 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:16.920 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:16.920 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:16.920 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:16.920 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:16.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:16.921 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:16.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:16.921 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:16.921 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:16.922 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:16.922 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:16.922 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:16.922 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:16.922 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:16.923 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:16.923 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:16.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:16.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:16.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:16.924 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:16.924 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:16.924 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:16.924 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:16.924 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:16.925 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:16.925 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:16.925 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:16.925 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:16.925 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:16.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:16.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:16.926 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:16.926 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:16.927 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:16.927 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:16.927 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:16.927 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:16.927 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:16.928 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:16.928 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:16.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:16.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:16.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:16.929 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:16.929 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:16.929 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:16.929 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:16.930 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:16.930 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:16.930 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:16.930 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:16.930 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:16.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:16.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:16.931 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:16.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:16.931 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:16.932 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:16.932 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:16.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:16.932 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:16Z" 2023-10-18T21:15:16.932 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:16.933 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:16Z" } 2023-10-18T21:15:16.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:16.934 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:16.935 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:16.935 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:21.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:21.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:21.938 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:21.994 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:21.995 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:22.026 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:22.024+0000 7fbae9a55780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:22.027 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:22.025+0000 7fbae9a55780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:22.040 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:22.041 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:22.041 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:22.041 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:22.041 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:22.042 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:22.042 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:22.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:22.042 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:22.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:22.043 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:22.043 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:22.043 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:22.043 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:22.044 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:22.044 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:22.044 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:22.044 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:22.044 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:22.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:22.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:22.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:22.045 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:22.046 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:22.046 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:22.046 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:22.046 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:22.046 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:22.047 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:22.047 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:22.047 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:22.047 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:22.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:22.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:22.048 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:22.048 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:22.048 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:22.048 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:22.049 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:22.049 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:22.049 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:22.049 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:22.050 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:22.050 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:22.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:22.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:22.050 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:22.051 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:22.051 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:22.051 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:22.051 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:22.051 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:22.052 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:22.052 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:22.052 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:22.052 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:22.052 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:22.053 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:22.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:22.053 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:22.053 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:22.053 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:22.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:22.054 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:22Z" 2023-10-18T21:15:22.054 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:22.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:22Z" } 2023-10-18T21:15:22.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:22.056 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:22.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:22.057 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:27.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:27.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:27.059 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:27.116 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:27.116 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:27.148 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:27.146+0000 7f1d29b09780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:27.149 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:27.147+0000 7f1d29b09780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:27.163 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:27.163 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:27.163 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:27.163 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:27.164 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:27.164 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:27.164 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:27.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:27.164 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:27.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:27.165 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:27.165 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:27.165 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:27.165 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:27.166 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:27.166 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:27.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:27.166 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:27.169 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:27.169 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:27.169 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:27.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:27.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:27.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:27.170 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:27.170 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:27.170 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:27.170 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:27.171 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:27.171 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:27.171 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:27.171 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:27.171 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:27.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:27.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:27.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:27.172 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:27.172 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:27.173 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:27.173 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:27.173 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:27.173 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:27.173 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:27.173 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:27.174 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:27.174 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:27.174 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:27.174 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:27.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:27.175 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:27.175 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:27.175 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:27.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:27.176 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:27Z" 2023-10-18T21:15:27.176 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:27.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:27Z" } 2023-10-18T21:15:27.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:27.178 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:27.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:27.179 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:32.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:32.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:32.181 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:32.238 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:32.238 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:32.270 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:32.268+0000 7f532b6f2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:32.271 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:32.269+0000 7f532b6f2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:32.285 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:32Z" 2023-10-18T21:15:32.299 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:32.300 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:32Z" } 2023-10-18T21:15:32.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:32.301 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:32.301 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:32.302 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:37.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:37.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:37.304 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:37.362 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:37.363 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:37.394 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:37.393+0000 7fbfcdf23780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:37.395 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:37.394+0000 7fbfcdf23780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:37.408 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:37.408 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:37.409 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:37.409 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:37.409 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:37.409 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:37.410 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:37.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:37.410 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:37.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:37.411 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:37.411 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:37.411 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:37.411 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:37.411 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:37.412 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:37.412 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:37.412 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:37.412 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:37.413 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:37.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:37.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:37.413 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:37.413 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:37.414 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:37.414 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:37.414 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:37.414 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:37.415 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:37.415 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:37.415 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:37.415 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:37.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:37.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:37.416 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:37.416 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:37.416 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:37.417 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:37.417 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:37.417 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:37.417 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:37.418 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:37.418 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:37.419 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:37.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:37.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:37.422 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:37.422 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:37.423 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:37.423 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:37.423 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:37.424 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:37.424 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:37.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:37.425 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:37.425 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:37.426 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:37.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:37.427 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:37Z" 2023-10-18T21:15:37.427 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:37.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:37Z" } 2023-10-18T21:15:37.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:37.431 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:37.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:37.432 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:42.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:42.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:42.435 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:42.491 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:42.492 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:42.523 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:42.522+0000 7fa54c27c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:42.524 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:42.523+0000 7fa54c27c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:42.538 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:42.538 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:42.538 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:42.539 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:42.539 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:42.539 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:42.539 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:42.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:42.540 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:42.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:42.540 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:42.540 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:42.540 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:42.541 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:42.541 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:42.541 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:42.541 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:42.541 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:42.542 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:42.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:42.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:42.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:42.543 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:42.543 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:42.543 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:42.543 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:42.543 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:42.544 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:42.544 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:42.544 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:42.544 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:42.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:42.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:42.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:42.545 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:42.545 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:42.545 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:42.546 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:42.546 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:42.546 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:42.546 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:42.546 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:42.547 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:42.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:42.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:42.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:42.547 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:42.548 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:42.548 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:42.548 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:42.548 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:42.548 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:42.549 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:42.549 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:42.549 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:42.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:42.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:42.550 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:42.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:42.550 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:42.550 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:42.551 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:42.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:42.551 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:42Z" 2023-10-18T21:15:42.551 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:42.552 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:42Z" } 2023-10-18T21:15:42.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:42.553 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:42.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:42.554 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:47.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:47.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:47.556 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:47.614 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:47.615 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:47.646 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:47.645+0000 7f1e680f9780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:47.647 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:47.646+0000 7f1e680f9780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:47.660 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:47Z" 2023-10-18T21:15:47.673 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:47.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:47Z" } 2023-10-18T21:15:47.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:47.675 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:47.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:47.676 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:52.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:52.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:52.679 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:52.736 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:52.737 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:52.769 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:52.768+0000 7f9343deb780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:52.770 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:52.769+0000 7f9343deb780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:52.784 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:52.785 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:52.785 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:52.785 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:52.785 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:52.785 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:52.786 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:52.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:52.786 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:52.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:52.787 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:52.787 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:52.787 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:52.787 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:52.787 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:52.788 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:52.788 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:52.788 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:52.788 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:52.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:52.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:52.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:52.789 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:52.789 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:52.790 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:52.790 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:52.790 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:52.790 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:52.790 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:52.791 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:52.791 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:52.791 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:52.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:52.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:52.792 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:52.792 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:52.792 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:52.792 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:52.792 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:52.793 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:52.793 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:52.793 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:52.793 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:52.794 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:52.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:52.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:52.794 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:52.794 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:52.795 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:52.795 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:52.795 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:52.795 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:52.795 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:52.796 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:52.796 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:52.796 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:52.796 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:52.796 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:52.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:52.797 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:52.797 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:52.797 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:52.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:52.798 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:52Z" 2023-10-18T21:15:52.798 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:52.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:52Z" } 2023-10-18T21:15:52.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:52.800 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:52.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:52.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:15:57.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:57.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:15:57.803 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:15:57.862 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:15:57.862 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:15:57.894 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:57.894+0000 7f0d254d5780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:57.895 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:15:57.895+0000 7f0d254d5780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:15:57.909 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:15:57.909 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:15:57.909 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:15:57.909 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:15:57.910 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:15:57.910 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:15:57.910 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:57.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:57.911 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:15:57.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:57.911 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:15:57.911 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:57.911 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:57.912 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:15:57.912 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:57.912 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:15:57.912 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:57.912 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:15:57.913 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:57.913 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:57.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:57.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:57.913 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:15:57.914 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:57.914 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:57.914 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:15:57.914 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:57.915 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:57.915 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:57.915 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:15:57.915 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:57.915 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:57.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:57.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:57.916 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:15:57.916 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:57.916 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:57.917 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:15:57.917 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:57.917 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:57.917 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:57.917 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:15:57.918 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:57.918 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:57.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:57.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:15:57.918 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:15:57.919 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:15:57.919 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:15:57.919 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:15:57.919 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:15:57.919 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:15:57.920 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:15:57.920 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:15:57.920 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:15:57.920 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:57.920 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:15:57.921 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:15:57.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:57.921 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:15:57.921 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:15:57.921 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:15:57.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:15:57.922 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:15:57Z" 2023-10-18T21:15:57.922 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:15:57.923 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:15:57Z" } 2023-10-18T21:15:57.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:15:57.924 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:57.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:15:57.925 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:02.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:02.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:02.927 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:02.984 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:02.984 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:03.016 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:03.015+0000 7f7ebc9c6780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:03.017 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:03.016+0000 7f7ebc9c6780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:03.031 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:03.032 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:03.032 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:03.032 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:03.032 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:03.032 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:03.033 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:03.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:03.033 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:03.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:03.033 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:03.034 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:03.034 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:03.034 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:03.034 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:03.034 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:03.035 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:03.035 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:03.035 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:03.035 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:03.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:03.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:03.036 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:03.036 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:03.036 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:03.036 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:03.037 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:03.038 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:03.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:03.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:03.038 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:03.038 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:03.039 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:03.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:03.041 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:03.041 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:03.041 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:03.041 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:03.041 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:03.042 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:03.043 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:03.044 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:03.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:03.044 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:03Z" 2023-10-18T21:16:03.044 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:03.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:03Z" } 2023-10-18T21:16:03.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:03.046 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:03.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:03.047 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:08.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:08.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:08.049 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:08.106 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:08.106 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:08.139 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:08.138+0000 7f460d542780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:08.140 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:08.139+0000 7f460d542780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:08.154 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:08.155 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:08.155 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:08.155 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:08.155 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:08.155 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:08.156 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:08.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:08.156 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:08.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:08.156 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:08.157 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:08.157 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:08.157 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:08.157 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:08.157 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:08.158 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:08.158 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:08.158 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:08.158 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:08.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:08.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:08.159 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:08.159 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:08.159 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:08.160 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:08.160 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:08.160 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:08.160 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:08.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:08.162 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:08.162 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:08.162 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:08.162 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:08.162 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:08.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:08.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:08.168 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:08Z" 2023-10-18T21:16:08.168 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:08.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:08Z" } 2023-10-18T21:16:08.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:08.170 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:08.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:08.171 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:13.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:13.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:13.172 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:13.228 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:13.229 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:13.263 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:13.261+0000 7f9610427780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:13.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:13.262+0000 7f9610427780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:13.278 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:13.288 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:13.288 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:13.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:13.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:13.288 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:13.289 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:13.289 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:13.289 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:13.289 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:13.293 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:13Z" 2023-10-18T21:16:13.293 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:13.294 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:13Z" } 2023-10-18T21:16:13.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:13.295 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:13.296 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:13.296 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:18.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:18.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:18.299 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:18.357 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:18.357 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:18.389 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:18.387+0000 7f8cd4746780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:18.390 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:18.388+0000 7f8cd4746780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:18.404 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:18.404 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:18.404 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:18.405 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:18.405 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:18.405 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:18.405 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:18.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:18.406 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:18.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:18.406 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:18.406 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:18.407 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:18.407 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:18.407 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:18.407 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:18.407 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:18.408 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:18.408 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:18.408 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:18.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:18.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:18.409 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:18.409 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:18.409 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:18.409 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:18.410 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:18.410 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:18.410 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:18.410 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:18.411 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:18.411 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:18.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:18.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:18.411 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:18.412 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:18.412 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:18.412 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:18.412 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:18.412 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:18.413 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:18.413 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:18.413 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:18.413 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:18.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:18.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:18.414 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:18.414 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:18.414 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:18.414 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:18.415 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:18.415 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:18.415 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:18.415 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:18.415 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:18.416 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:18.416 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:18.416 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:18.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:18.417 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:18.417 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:18.417 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:18.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:18.417 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:18Z" 2023-10-18T21:16:18.417 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:18.419 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:18Z" } 2023-10-18T21:16:18.419 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:18.420 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:18.420 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:18.421 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:23.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:23.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:23.423 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:23.482 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:23.482 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:23.518 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:23.516+0000 7f491161f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:23.518 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:23.517+0000 7f491161f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:23.532 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:23.533 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:23.533 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:23.533 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:23.533 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:23.534 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:23.534 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:23.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:23.534 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:23.537 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:23.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:23.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:23.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:23.537 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:23.538 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:23.538 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:23.538 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:23.538 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:23.539 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:23.539 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:23.539 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:23.539 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:23.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:23.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:23.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:23.540 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:23.540 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:23.540 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:23.541 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:23.541 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:23.541 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:23.541 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:23.541 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:23.542 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:23.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:23.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:23.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:23.542 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:23.543 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:23.543 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:23.543 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:23.543 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:23.543 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:23.544 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:23.544 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:23.544 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:23.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:23.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:23.545 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:23.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:23.545 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:23.545 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:23.545 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:23.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:23.546 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:23Z" 2023-10-18T21:16:23.546 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:23.547 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:23Z" } 2023-10-18T21:16:23.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:23.548 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:23.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:23.549 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:28.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:28.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:28.551 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:28.610 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:28.610 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:28.642 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:28.642+0000 7f5ef0d64780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:28.643 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:28.643+0000 7f5ef0d64780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:28.657 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:28.657 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:28.657 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:28.658 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:28.658 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:28.658 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:28.658 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:28.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:28.659 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:28.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:28.659 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:28.659 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:28.660 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:28.660 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:28.660 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:28.660 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:28.661 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:28.661 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:28.661 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:28.661 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:28.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:28.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:28.662 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:28.662 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:28.662 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:28.662 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:28.663 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:28.663 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:28.663 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:28.663 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:28.664 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:28.664 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:28.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:28.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:28.664 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:28.665 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:28.665 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:28.665 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:28.665 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:28.665 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:28Z" 2023-10-18T21:16:28.671 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:28.672 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:28Z" } 2023-10-18T21:16:28.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:28.673 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:28.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:28.674 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:33.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:33.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:33.676 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:33.732 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:33.732 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:33.765 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:33.764+0000 7f1e678f7780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:33.765 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:33.765+0000 7f1e678f7780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:33.779 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:33.783 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:33.783 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:33.783 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:33.783 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:33.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:33.786 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:33.786 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:33.786 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:33.788 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:33.788 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:33.788 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:33.788 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:33.788 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:33.792 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:33.792 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:33.792 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:33.792 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:33.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:33.794 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:33Z" 2023-10-18T21:16:33.794 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:33.795 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:33Z" } 2023-10-18T21:16:33.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:33.797 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:33.797 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:33.798 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:38.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:38.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:38.800 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:38.858 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:38.858 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:38.891 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:38.890+0000 7f54b0eb1780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:38.892 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:38.891+0000 7f54b0eb1780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:38.905 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:38.906 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:38.906 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:38.906 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:38.906 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:38.907 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:38.907 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:38.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:38.907 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:38.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:38.908 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:38.908 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:38.908 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:38.908 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:38.908 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:38.909 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:38.909 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:38.909 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:38.909 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:38.909 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:38.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:38.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:38.910 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:38.910 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:38.910 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:38.911 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:38.911 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:38.911 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:38.911 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:38.911 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:38.912 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:38.912 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:38.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:38.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:38.912 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:38.913 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:38.913 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:38.913 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:38.913 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:38.913 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:38.914 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:38.914 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:38.914 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:38.914 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:38.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:38.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:38.915 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:38.915 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:38.915 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:38.915 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:38.916 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:38.916 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:38.916 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:38.916 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:38.916 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:38.917 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:38.917 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:38.917 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:38.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:38.917 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:38.918 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:38.918 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:38.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:38.918 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:38Z" 2023-10-18T21:16:38.919 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:38.920 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:38Z" } 2023-10-18T21:16:38.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:38.921 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:38.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:38.921 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:43.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:43.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:43.924 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:43.982 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:43.982 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:44.015 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:44.015+0000 7f9c6037d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:44.016 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:44.016+0000 7f9c6037d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:44.030 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:44.031 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:44.031 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:44.031 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:44.031 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:44.032 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:44.032 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:44.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:44.032 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:44.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:44.033 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:44.033 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:44.033 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:44.033 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:44.033 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:44.034 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:44.034 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:44.034 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:44.034 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:44.035 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:44.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:44.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:44.035 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:44.036 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:44.036 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:44.036 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:44.036 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:44.036 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:44.037 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:44.037 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:44.037 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:44.037 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:44.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:44.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:44.038 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:44.038 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:44.038 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:44.039 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:44.039 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:44.039 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:44.039 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:44.039 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:44.040 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:44.040 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:44.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:44.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:44.040 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:44.041 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:44.041 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:44.041 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:44.041 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:44.041 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:44.042 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:44.042 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:44.042 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:44.042 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:44.042 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:44.043 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:44.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:44.043 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:44.043 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:44.043 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:44.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:44.044 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:44Z" 2023-10-18T21:16:44.044 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:44.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:44Z" } 2023-10-18T21:16:44.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:44.046 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:44.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:44.047 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:49.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:49.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:49.049 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:49.105 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:49.105 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:49.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:49.135+0000 7f5405ee3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:49.137 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:49.136+0000 7f5405ee3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:49.151 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:49.151 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:49.152 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:49.152 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:49.152 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:49.153 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:49.153 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:49.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:49.154 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:49.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:49.155 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:49.155 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:49.156 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:49.156 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:49.156 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:49.157 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:49.157 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:49.157 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:49.158 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:49.158 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:49.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:49.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:49.159 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:49.160 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:49.160 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:49.161 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:49.161 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:49.161 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:49.162 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:49.162 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:49.162 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:49.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:49.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:49.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:49.164 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:49.164 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:49.165 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:49.165 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:49.166 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:49.166 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:49.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:49.167 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:49.167 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:49.168 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:49.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:49.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:49.169 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:49.169 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:49.170 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:49.170 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:49.170 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:49.171 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:49.171 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:49.171 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:49.172 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:49.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:49.173 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:49.173 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:49.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:49.174 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:49.174 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:49.175 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:49.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:49.175 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:49Z" 2023-10-18T21:16:49.176 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:49.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:49Z" } 2023-10-18T21:16:49.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:49.179 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:49.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:49.180 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:54.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:54.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:54.183 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:54.242 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:54.242 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:54.274 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:54.274+0000 7f3a24f27780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:54.275 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:54.275+0000 7f3a24f27780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:54.289 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:54.289 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:54.290 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:54.291 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:54.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:54.291 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:54.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:54.291 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663570.013765_1749.1", 2023-10-18T21:16:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:54.292 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:54.293 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:54.293 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:50.013765Z", 2023-10-18T21:16:54.293 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:54.293 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:54.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:54.294 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:54.295 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:54.296 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:54.297 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:54.297 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:54.297 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:54.297 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:54.297 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:54.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:54.299 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:54.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:54.301 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:54.301 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:54.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:54.301 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:54.301 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:54.302 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:54.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:54.302 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:54Z" 2023-10-18T21:16:54.302 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:54.303 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663570.013765_1749.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:12:50.013765Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:54Z" } 2023-10-18T21:16:54.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663570.013765_1749.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:12:50.013765Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:54.304 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:54.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663570.013765_1749.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:54.305 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663696.079492_1849.1 target=1_1697663570.013765_1749.1 2023-10-18T21:16:59.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.307 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:59.365 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:59.365 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:59.398 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:59.397+0000 7fc9f9ea3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:59.399 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:16:59.399+0000 7fc9f9ea3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:16:59.414 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:59.414 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:59.414 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:16:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:59.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.416 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:59.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.416 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:59.416 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:59.416 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:59.417 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663696.079492_1849.1", 2023-10-18T21:16:59.417 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:59.417 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:16:59.417 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:59.417 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.079492Z", 2023-10-18T21:16:59.418 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:59.418 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.418 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:59.419 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:16:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.421 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:59.421 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:59.421 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:59.421 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:59.421 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:16:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:59.422 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.423 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:59.423 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:59.424 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:16:59.424 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:59.424 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:59.424 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:16:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:16:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.426 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:59.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.426 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:59.426 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:16:59.426 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:16:59.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.427 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:59Z" 2023-10-18T21:16:59.427 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:59.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663696.079492_1849.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:14:56.079492Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:16:59Z" } 2023-10-18T21:16:59.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663696.079492_1849.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.079492Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:16:59.429 DEBUG:rgw_multi.tests:log_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:59.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:59.430 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:16:59.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.431 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:59.483 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:59.483 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:59.532 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:16:59.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.532 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:16:59.532 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:16:59.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.533 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:16:59.533 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:16:59.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.534 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:16:59.534 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:16:59.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.535 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:16:59.535 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:16:59.535 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.536 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:16:59.537 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:16:59.537 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:16:59.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.538 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:59.590 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:59.590 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:59.633 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:16:59.634 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:16:59.634 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:16:59.634 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:16:59.634 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:16:59.634 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:16:59.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.635 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:16:59.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.635 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:16:59.636 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:59.636 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:16:59.636 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:16:59.636 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:59.636 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:59.637 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:59.637 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:16:59.637 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.638 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:16:59.638 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:59.638 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:16:59.638 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:16:59.639 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:59.639 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:16:59.639 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:59.639 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:16:59.639 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.640 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:16:59.640 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:59.641 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:16:59.641 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:16:59.641 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:59.641 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:16:59.641 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:59.642 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:16:59.642 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.642 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:16:59.643 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:16:59.643 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:16:59.643 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:16:59.643 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:16:59.643 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:16:59.644 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:16:59.644 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:16:59.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.645 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:16:59.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.645 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:16:59.645 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:16:59.645 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:16:59.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.646 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:16:59Z" 2023-10-18T21:16:59.646 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:16:59.647 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:16:59Z" } 2023-10-18T21:16:59.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:16:59.648 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:16:59.648 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:16:59.649 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:16:59.649 DEBUG:boto:path=/bofrnf-68/ 2023-10-18T21:16:59.650 DEBUG:boto:auth_path=/bofrnf-68/ 2023-10-18T21:16:59.650 DEBUG:boto:Method: HEAD 2023-10-18T21:16:59.651 DEBUG:boto:Path: /bofrnf-68/ 2023-10-18T21:16:59.651 DEBUG:boto:Data: 2023-10-18T21:16:59.652 DEBUG:boto:Headers: {} 2023-10-18T21:16:59.652 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:16:59.652 DEBUG:boto:Port: 8001 2023-10-18T21:16:59.653 DEBUG:boto:Params: {} 2023-10-18T21:16:59.654 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:16:59.654 DEBUG:boto:Token: None 2023-10-18T21:16:59.655 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:16:59 GMT /bofrnf-68/ 2023-10-18T21:16:59.655 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:/+RyhjNIw2ANp+DgLy8SRAbnJsc= 2023-10-18T21:16:59.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/+RyhjNIw2ANp+DgLy8SRAbnJsc=', 'Content-Length': '0'} 2023-10-18T21:16:59.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089173b0003ea6629-0065304b4b-2492-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:16:59.661 DEBUG:boto:path=/bofrnf-68/obj3 2023-10-18T21:16:59.662 DEBUG:boto:auth_path=/bofrnf-68/obj3 2023-10-18T21:16:59.662 DEBUG:boto:Method: HEAD 2023-10-18T21:16:59.663 DEBUG:boto:Path: /bofrnf-68/obj3 2023-10-18T21:16:59.663 DEBUG:boto:Data: 2023-10-18T21:16:59.664 DEBUG:boto:Headers: {} 2023-10-18T21:16:59.664 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:16:59.664 DEBUG:boto:Port: 8001 2023-10-18T21:16:59.665 DEBUG:boto:Params: {} 2023-10-18T21:16:59.665 DEBUG:boto:Token: None 2023-10-18T21:16:59.666 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:16:59 GMT /bofrnf-68/obj3 2023-10-18T21:16:59.666 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xlV017u6WbpWu2T1jqY6w6yiWlQ= 2023-10-18T21:16:59.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xlV017u6WbpWu2T1jqY6w6yiWlQ=', 'Content-Length': '0'} 2023-10-18T21:16:59.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:14:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-68:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c77bbd487ba42d90-0065304b4b-2492-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:16:59.669 DEBUG:boto:path=/bofrnf-68/obj3 2023-10-18T21:16:59.670 DEBUG:boto:auth_path=/bofrnf-68/obj3 2023-10-18T21:16:59.670 DEBUG:boto:Method: GET 2023-10-18T21:16:59.671 DEBUG:boto:Path: /bofrnf-68/obj3 2023-10-18T21:16:59.671 DEBUG:boto:Data: 2023-10-18T21:16:59.671 DEBUG:boto:Headers: {} 2023-10-18T21:16:59.672 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:16:59.672 DEBUG:boto:Port: 8001 2023-10-18T21:16:59.673 DEBUG:boto:Params: {} 2023-10-18T21:16:59.673 DEBUG:boto:Token: None 2023-10-18T21:16:59.674 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:16:59 GMT /bofrnf-68/obj3 2023-10-18T21:16:59.674 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:os1a6QQS9VVXamos4a62qmNz54g= 2023-10-18T21:16:59.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:os1a6QQS9VVXamos4a62qmNz54g=', 'Content-Length': '0'} 2023-10-18T21:16:59.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:14:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-68:403a921b-fcfc-4ab2-b434-bf29d35a51b6.8730.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000689cd3a7fda5da1c-0065304b4b-2492-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:16:59.677 DEBUG:boto:path=/bofrnf-69/ 2023-10-18T21:16:59.678 DEBUG:boto:auth_path=/bofrnf-69/ 2023-10-18T21:16:59.678 DEBUG:boto:Method: HEAD 2023-10-18T21:16:59.679 DEBUG:boto:Path: /bofrnf-69/ 2023-10-18T21:16:59.679 DEBUG:boto:Data: 2023-10-18T21:16:59.679 DEBUG:boto:Headers: {} 2023-10-18T21:16:59.680 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:16:59.680 DEBUG:boto:Port: 8001 2023-10-18T21:16:59.680 DEBUG:boto:Params: {} 2023-10-18T21:16:59.681 DEBUG:boto:Token: None 2023-10-18T21:16:59.681 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:16:59 GMT /bofrnf-69/ 2023-10-18T21:16:59.682 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:TSa50YDArS79rifXSfy9oTKXsSs= 2023-10-18T21:16:59.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TSa50YDArS79rifXSfy9oTKXsSs=', 'Content-Length': '0'} 2023-10-18T21:16:59.685 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084e7bf0601bf9d9a-0065304b4b-2492-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:16:59.685 DEBUG:boto:path=/bofrnf-69/obj3 2023-10-18T21:16:59.685 DEBUG:boto:auth_path=/bofrnf-69/obj3 2023-10-18T21:16:59.686 DEBUG:boto:Method: HEAD 2023-10-18T21:16:59.686 DEBUG:boto:Path: /bofrnf-69/obj3 2023-10-18T21:16:59.687 DEBUG:boto:Data: 2023-10-18T21:16:59.687 DEBUG:boto:Headers: {} 2023-10-18T21:16:59.687 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:16:59.688 DEBUG:boto:Port: 8001 2023-10-18T21:16:59.688 DEBUG:boto:Params: {} 2023-10-18T21:16:59.689 DEBUG:boto:Token: None 2023-10-18T21:16:59.689 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:16:59 GMT /bofrnf-69/obj3 2023-10-18T21:16:59.689 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jeuiia72eIdmrISS4d63r/xd2GY= 2023-10-18T21:16:59.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:16:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jeuiia72eIdmrISS4d63r/xd2GY=', 'Content-Length': '0'} 2023-10-18T21:16:59.691 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001a8d0117d451f5d2-0065304b4b-2492-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:16:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:16:59.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:16:59.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:16:59.693 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket bofrnf-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:16:59.742 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:59.742 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:59.791 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:16:59.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:16:59.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:16:59.793 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:16:59.841 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:59.841 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:59.887 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:16:59.888 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2023-10-18T21:16:59.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.891 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:16:59.938 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:16:59.939 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:16:59.968 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:16:59.981 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:16:59.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.982 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663819.777022_1866.1", 2023-10-18T21:16:59.982 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:16:59.777022Z" 2023-10-18T21:16:59.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.983 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:16:59.983 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:16:59.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.984 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:16:59.984 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:16:59.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:16:59.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:16:59.984 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:16:59.985 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:16:59.985 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:16:59.985 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:16:59.986 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663819.777022_1866.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:16:59.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:16:59.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:16:59.988 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:00.037 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:00.038 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:00.066 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:00.065+0000 7fc98a07c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:00.066 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:00.066+0000 7fc98a07c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:00.079 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:00.080 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:00.080 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:00.080 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:00.080 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:00.080 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:00.081 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:00.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:00.081 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:00.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:00.082 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:00.082 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:00.082 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:00.082 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663696.079492_1849.1", 2023-10-18T21:17:00.083 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:00.083 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:17:00.083 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:00.083 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.079492Z", 2023-10-18T21:17:00.083 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:00.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:00.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:00.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:00.085 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:00.085 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:00.085 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:00.085 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:00.085 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:00.086 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:00.086 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:00.086 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:17:00.086 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:00.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:00.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:00.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:00.087 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:00.087 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:00.088 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:00.088 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:00.088 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:00.088 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:00.089 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:00.089 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:17:00.089 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:00.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:00.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:00.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:00.090 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:00.091 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:00.091 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:00.091 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:00.091 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:00.092 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:00.092 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:00.092 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:17:00.092 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:00.093 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:00.093 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:00.093 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:00.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:00.094 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:00.094 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:17:00.094 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:17:00.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:00.095 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:00Z" 2023-10-18T21:17:00.095 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:00.096 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663696.079492_1849.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:14:56.079492Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:17:00Z" } 2023-10-18T21:17:00.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663696.079492_1849.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.079492Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:17:00.097 DEBUG:rgw_multi.tests:log_status={0: '1_1697663819.777022_1866.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:00.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:00.098 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663819.777022_1866.1 target=1_1697663696.079492_1849.1 2023-10-18T21:17:05.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:05.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:05.101 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:05.161 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:05.161 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:05.194 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:05.192+0000 7fe7ebd26780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:05.195 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:05.193+0000 7fe7ebd26780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:05.209 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663696.079492_1849.1", 2023-10-18T21:17:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:17:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.079492Z", 2023-10-18T21:17:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:17:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:17:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:17:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:17:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:17:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:05.223 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:05Z" 2023-10-18T21:17:05.223 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:05.224 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663696.079492_1849.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:14:56.079492Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:17:05Z" } 2023-10-18T21:17:05.224 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663696.079492_1849.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.079492Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:17:05.225 DEBUG:rgw_multi.tests:log_status={0: '1_1697663819.777022_1866.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:05.225 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:05.226 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663819.777022_1866.1 target=1_1697663696.079492_1849.1 2023-10-18T21:17:10.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:10.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:10.229 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:10.286 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:10.286 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:10.320 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:10.318+0000 7faec3c5a780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:10.321 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:10.319+0000 7faec3c5a780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:10.335 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663696.079492_1849.1", 2023-10-18T21:17:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:17:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.079492Z", 2023-10-18T21:17:10.339 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:10.339 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:10.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:10.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:10.339 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:17:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:17:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:17:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:17:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:17:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:10Z" 2023-10-18T21:17:10.348 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:10.349 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663696.079492_1849.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:14:56.079492Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:17:10Z" } 2023-10-18T21:17:10.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663696.079492_1849.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.079492Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:17:10.350 DEBUG:rgw_multi.tests:log_status={0: '1_1697663819.777022_1866.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:10.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:10.351 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663819.777022_1866.1 target=1_1697663696.079492_1849.1 2023-10-18T21:17:15.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:15.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:15.353 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:15.409 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:15.410 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:15.441 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:15.440+0000 7fcc4d096780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:15.442 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:15.441+0000 7fcc4d096780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:15.464 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:15.464 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:15.465 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:15.465 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:15.465 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:15.465 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:15.465 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:15.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:15.466 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:15.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:15.466 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:15.467 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:15.467 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:15.467 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663696.079492_1849.1", 2023-10-18T21:17:15.467 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:15.467 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:17:15.468 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:15.468 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.079492Z", 2023-10-18T21:17:15.468 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:15.468 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:15.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:15.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:15.469 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:15.469 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:15.469 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:15.469 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:15.470 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:15.470 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:15.470 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:15.470 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:17:15.471 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:15.471 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:15.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:15.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:15.471 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:15.472 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:15.472 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:15.472 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:15.472 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:15.472 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:15.473 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:15.473 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:17:15.473 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:15.473 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:15.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:15.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:15.474 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:15.474 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:15.474 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:15.474 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:15.475 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:15.475 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:15.475 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:15.475 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:17:15.475 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:15.476 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:15.476 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:15.476 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:15.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:15.477 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:15.477 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:17:15.477 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:17:15.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:15.477 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:15Z" 2023-10-18T21:17:15.478 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:15.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663696.079492_1849.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:14:56.079492Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:17:15Z" } 2023-10-18T21:17:15.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663696.079492_1849.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.079492Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:17:15.480 DEBUG:rgw_multi.tests:log_status={0: '1_1697663819.777022_1866.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:15.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663696.079492_1849.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:15.480 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663819.777022_1866.1 target=1_1697663696.079492_1849.1 2023-10-18T21:17:20.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:20.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:20.483 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:20.540 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:20.541 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:20.576 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:20.575+0000 7f2ce83b3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:20.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:20.576+0000 7f2ce83b3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:20.592 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:20.592 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:20.593 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:20.593 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:20.593 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:20.594 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:20.594 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.595 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:20.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.595 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:20.596 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:20.596 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:20.597 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663819.777022_1866.1", 2023-10-18T21:17:20.597 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:20.597 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:17:20.598 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:20.598 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:16:59.777022Z", 2023-10-18T21:17:20.599 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.599 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.600 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:20.601 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:20.601 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:20.601 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:20.602 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:20.602 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:20.603 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:20.603 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:17:20.604 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.605 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:20.605 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:20.605 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:20.605 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:20.606 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:20.606 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:20.606 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:20.606 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:17:20.606 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.607 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.607 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:20.608 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:20.608 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:20.608 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:20.608 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:20.609 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:20.609 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:20.609 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:17:20.609 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.610 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.610 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:20.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.610 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:20.611 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:17:20.611 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:17:20.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.611 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:20Z" 2023-10-18T21:17:20.611 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:20.613 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663819.777022_1866.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:16:59.777022Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:17:20Z" } 2023-10-18T21:17:20.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663819.777022_1866.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:16:59.777022Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:17:20.614 DEBUG:rgw_multi.tests:log_status={0: '1_1697663819.777022_1866.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:20.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663819.777022_1866.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:20.615 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:17:20.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:20.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:20.616 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:17:20.663 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:20.663 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:20.708 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[]} 2023-10-18T21:17:20.708 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-70 2023-10-18T21:17:20.709 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:17:20.709 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:17:20.709 DEBUG:boto:Method: PUT 2023-10-18T21:17:20.710 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:17:20.710 DEBUG:boto:Data: 2023-10-18T21:17:20.711 DEBUG:boto:Headers: {} 2023-10-18T21:17:20.711 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:17:20.711 DEBUG:boto:Port: 8000 2023-10-18T21:17:20.712 DEBUG:boto:Params: {} 2023-10-18T21:17:20.712 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:17:20.713 DEBUG:boto:Token: None 2023-10-18T21:17:20.713 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:17:20 GMT /bofrnf-70/ 2023-10-18T21:17:20.714 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:sPtm8gfgGy08nt3lN4yOYdIdtUE= 2023-10-18T21:17:20.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:17:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sPtm8gfgGy08nt3lN4yOYdIdtUE=', 'Content-Length': '0'} 2023-10-18T21:17:20.730 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000082a91bc74e1b2e16-0065304b60-2486-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:17:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:17:20.731 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-71 2023-10-18T21:17:20.731 DEBUG:boto:path=/bofrnf-71/ 2023-10-18T21:17:20.732 DEBUG:boto:auth_path=/bofrnf-71/ 2023-10-18T21:17:20.732 DEBUG:boto:Method: PUT 2023-10-18T21:17:20.733 DEBUG:boto:Path: /bofrnf-71/ 2023-10-18T21:17:20.733 DEBUG:boto:Data: 2023-10-18T21:17:20.733 DEBUG:boto:Headers: {} 2023-10-18T21:17:20.734 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:17:20.734 DEBUG:boto:Port: 8000 2023-10-18T21:17:20.735 DEBUG:boto:Params: {} 2023-10-18T21:17:20.735 DEBUG:boto:Token: None 2023-10-18T21:17:20.736 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:17:20 GMT /bofrnf-71/ 2023-10-18T21:17:20.736 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:caAV/elaiGgPgKTmhuzgI2Rdils= 2023-10-18T21:17:20.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:17:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:caAV/elaiGgPgKTmhuzgI2Rdils=', 'Content-Length': '0'} 2023-10-18T21:17:20.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e35789df82def46f-0065304b60-2486-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:17:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:17:20.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:20.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:20.753 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:20.805 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:20.806 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:20.841 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:17:20.854 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:17:20.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.855 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:17:20.855 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:17:20.749643Z" 2023-10-18T21:17:20.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.856 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:20.856 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:17:20.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.857 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:20.857 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:17:20.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.857 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:20.858 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:17:20.858 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.858 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:17:20.859 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:20.860 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:17:20.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:20.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:20.861 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:20.911 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:20.911 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:20.942 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:20.941+0000 7f6208d0d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:20.942 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:20.942+0000 7f6208d0d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:20.955 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:20.955 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:20.956 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:20.956 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:20.956 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:20.956 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:20.956 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.957 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:20.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.957 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:20.957 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:20.958 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:20.958 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.723537_1867.1", 2023-10-18T21:17:20.958 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:20.958 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:20.723537Z", 2023-10-18T21:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:20.963 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:20.963 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:20.963 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:20.963 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:20.963 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:20.964 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:20.964 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:17:20.964 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.964 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:20.965 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:20.965 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:20.965 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:20.966 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:20.966 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:20.966 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:20.966 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:20.966 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:17:20.967 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:20.967 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.967 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:20.967 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:20.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.968 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:20.968 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:17:20.968 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:17:20.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:20.968 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:20Z" 2023-10-18T21:17:20.968 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:20.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663840.723537_1867.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:17:20.723537Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:17:20Z" } 2023-10-18T21:17:20.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663840.723537_1867.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:17:20.723537Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:17:20.971 DEBUG:rgw_multi.tests:log_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:20.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663840.723537_1867.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:20.971 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663840.749643_1881.1 target=1_1697663840.723537_1867.1 2023-10-18T21:17:25.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:25.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:25.974 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:26.031 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:26.031 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:26.064 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:26.063+0000 7f42c180e780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:26.065 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:26.064+0000 7f42c180e780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:26.080 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:26.080 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:26.080 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:26.081 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:26.081 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:26.081 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:26.081 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:26.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.082 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:26.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.082 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:26.082 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:26.083 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:26.083 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:17:26.083 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:26.083 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:17:26.084 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:26.084 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:20.749643Z", 2023-10-18T21:17:26.084 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:26.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.085 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:26.085 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:26.085 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:26.086 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:26.086 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:26.086 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:26.086 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:26.086 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:17:26.087 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:26.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.087 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:26.088 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:26.088 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:26.088 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:26.088 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:26.088 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:26.089 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:26.089 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:17:26.089 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:26.089 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.090 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:26.090 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:26.090 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:26.091 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:26.091 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:26.091 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:26.091 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:26.091 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:17:26.091 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:26.092 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.092 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.092 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.092 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:26.093 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:17:26.093 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:17:26.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.093 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:26Z" 2023-10-18T21:17:26.094 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:26.094 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663840.749643_1881.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:17:20.749643Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:17:26Z" } 2023-10-18T21:17:26.095 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663840.749643_1881.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:17:20.749643Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:17:26.095 DEBUG:rgw_multi.tests:log_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:26.096 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:26.096 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:17:26.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.098 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:17:26.148 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:26.148 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:26.198 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:17:26.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.199 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:17:26.253 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:26.253 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:26.297 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-18T21:17:26.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-70', '--dest-bucket=bofrnf-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-70', '--dest-bucket=bofrnf-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.299 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=bofrnf-70 --dest-bucket=bofrnf-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:17:26.353 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:26.354 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:26.400 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"bofrnf-70","zones":["a1"]},"dest":{"bucket":"bofrnf-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:17:26.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.402 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:17:26.451 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:26.451 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:26.496 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"bofrnf-70","zones":["a1"]},"dest":{"bucket":"bofrnf-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:17:26.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.498 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:17:26.545 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:26.546 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:26.598 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:26.599 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:26.599 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 18, 2023-10-18T21:17:26.599 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:17:26.599 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:17:26.600 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:17:26.600 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:26.600 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:17:26.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.600 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:17:26.601 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:17:26.601 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:17:26.601 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:17:26.601 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:17:26.601 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:17:26.602 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.602 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:17:26.602 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:17:26.602 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:17:26.602 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:17:26.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.603 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:17:26.603 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:17:26.603 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:17:26.603 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:17:26.604 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.604 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:17:26.604 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:17:26.604 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:17:26.605 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:17:26.605 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:17:26.605 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:17:26.605 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:17:26.605 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:17:26.606 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:17:26.606 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:17:26.606 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:17:26.606 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.607 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:17:26.607 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:17:26.607 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:17:26.608 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:17:26.608 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.608 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:17:26.608 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:17:26.608 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:17:26.609 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:17:26.609 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:17:26.609 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:17:26.609 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:17:26.610 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:17:26.610 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:17:26.610 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:17:26.610 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:17:26.610 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.611 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.611 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.611 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:17:26.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.611 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:17:26.612 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:17:26.612 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:17:26.612 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:17:26.612 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.613 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.613 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.613 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:17:26.613 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:17:26.613 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:17:26.614 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [ 2023-10-18T21:17:26.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.614 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-group", 2023-10-18T21:17:26.614 INFO:teuthology.orchestra.run.smithi082.stdout: "data_flow": { 2023-10-18T21:17:26.614 INFO:teuthology.orchestra.run.smithi082.stdout: "symmetrical": [ 2023-10-18T21:17:26.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.615 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-flow1", 2023-10-18T21:17:26.615 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:17:26.615 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:17:26.616 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:17:26.616 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.616 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.616 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.617 INFO:teuthology.orchestra.run.smithi082.stdout: "pipes": [ 2023-10-18T21:17:26.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.617 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe", 2023-10-18T21:17:26.617 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:17:26.617 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-70", 2023-10-18T21:17:26.618 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:17:26.618 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:17:26.618 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.619 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:17:26.619 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-71", 2023-10-18T21:17:26.619 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:17:26.619 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:17:26.619 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.620 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:17:26.620 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:17:26.620 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:17:26.620 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:17:26.621 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.621 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:17:26.621 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:17:26.621 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:17:26.622 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:17:26.622 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.622 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.622 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.622 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "enabled" 2023-10-18T21:17:26.623 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.623 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.623 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:17:26.624 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:17:26.624 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.624 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:17:26.625 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:17:26.625 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:17:26.625 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:17:26.625 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:17:26.625 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:17:26.626 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.626 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:17:26.626 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:17:26.626 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:17:26.626 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:17:26.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.627 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:17:26.627 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:17:26.627 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:17:26.627 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:17:26.628 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.628 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:17:26.628 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:17:26.628 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:17:26.628 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:17:26.629 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:17:26.629 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:17:26.629 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:17:26.629 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:17:26.629 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:17:26.630 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:17:26.630 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:17:26.630 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.631 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:17:26.631 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:17:26.631 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:17:26.631 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:17:26.631 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.632 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:17:26.632 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:17:26.632 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:17:26.632 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:17:26.632 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:17:26.633 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:17:26.633 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:17:26.633 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:17:26.633 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:17:26.633 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:17:26.634 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:17:26.634 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.634 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.634 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.634 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:17:26.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.635 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:17:26.635 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:17:26.635 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:17:26.636 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:17:26.636 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.636 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.636 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.636 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:17:26.637 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:17:26.637 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:17:26.637 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:17:26.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.637 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:17:26.638 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:17:26.638 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.638 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:17:26.638 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:17:26.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.639 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:17:26.639 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:17:26.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.640 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:17:26.640 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:17:26.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.640 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:17:26.641 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:17:26.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:26.642 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:17:26.642 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:17:26.643 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.643 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:26.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.643 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:17:26.644 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:17:26.644 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:17:26.644 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:17:26.644 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:17:26.644 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:17:26.645 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:17:26.645 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:17:26.645 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:17:26.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.645 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:17:26.646 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:17:26.646 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:17:26.646 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:17:26.646 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:17:26.647 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:17:26.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.647 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:17:26.647 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:17:26.647 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:17:26.648 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:17:26.648 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:17:26.648 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:17:26.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.648 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:17:26.649 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:17:26.649 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:17:26.649 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:17:26.649 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:17:26.650 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:17:26.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.650 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:17:26.650 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:17:26.650 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:17:26.651 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:17:26.651 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:17:26.651 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:17:26.651 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:26.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:26.652 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:17:26.652 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:17:26.652 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:26.652 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:26.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:17:26.654 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:17:26.699 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:26.700 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:26.743 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"bofrnf-70","zones":["a1"]},"dest":{"bucket":"bofrnf-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:17:26.744 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:17:26.744 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:17:26.745 DEBUG:boto:Method: HEAD 2023-10-18T21:17:26.745 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:17:26.746 DEBUG:boto:Data: 2023-10-18T21:17:26.746 DEBUG:boto:Headers: {} 2023-10-18T21:17:26.746 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:17:26.747 DEBUG:boto:Port: 8000 2023-10-18T21:17:26.747 DEBUG:boto:Params: {} 2023-10-18T21:17:26.748 DEBUG:boto:Token: None 2023-10-18T21:17:26.748 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:17:26 GMT /bofrnf-70/ 2023-10-18T21:17:26.749 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DqQG+DxYs3a1/jkaCEQW8t3OK/Q= 2023-10-18T21:17:26.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:17:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DqQG+DxYs3a1/jkaCEQW8t3OK/Q=', 'Content-Length': '0'} 2023-10-18T21:17:26.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000652c9dc313a35ba-0065304b66-262e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:17:26.887 DEBUG:boto:path=/bofrnf-70/obj1 2023-10-18T21:17:26.887 DEBUG:boto:auth_path=/bofrnf-70/obj1 2023-10-18T21:17:26.888 DEBUG:boto:Method: PUT 2023-10-18T21:17:26.888 DEBUG:boto:Path: /bofrnf-70/obj1 2023-10-18T21:17:26.889 DEBUG:boto:Data: 2023-10-18T21:17:26.889 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:17:26.890 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:17:26.890 DEBUG:boto:Port: 8000 2023-10-18T21:17:26.890 DEBUG:boto:Params: {} 2023-10-18T21:17:26.891 DEBUG:boto:Token: None 2023-10-18T21:17:26.891 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:17:26 GMT /bofrnf-70/obj1 2023-10-18T21:17:26.892 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:OssHAOHS9A+CWGxNjT/WiQEoMTg= 2023-10-18T21:17:26.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:17:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OssHAOHS9A+CWGxNjT/WiQEoMTg='} 2023-10-18T21:17:26.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab92f570cc164132-0065304b66-262e-a1'), ('Date', 'Wed, 18 Oct 2023 21:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:17:26.916 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:17:26.916 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:17:26.917 DEBUG:boto:Method: HEAD 2023-10-18T21:17:26.917 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:17:26.918 DEBUG:boto:Data: 2023-10-18T21:17:26.918 DEBUG:boto:Headers: {} 2023-10-18T21:17:26.918 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:17:26.919 DEBUG:boto:Port: 8000 2023-10-18T21:17:26.919 DEBUG:boto:Params: {} 2023-10-18T21:17:26.920 DEBUG:boto:Token: None 2023-10-18T21:17:26.920 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:17:26 GMT /bofrnf-70/ 2023-10-18T21:17:26.920 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DqQG+DxYs3a1/jkaCEQW8t3OK/Q= 2023-10-18T21:17:26.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:17:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DqQG+DxYs3a1/jkaCEQW8t3OK/Q=', 'Content-Length': '0'} 2023-10-18T21:17:26.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abff2d644c000884-0065304b66-262e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:17:26.925 DEBUG:boto:path=/bofrnf-70/obj2 2023-10-18T21:17:26.926 DEBUG:boto:auth_path=/bofrnf-70/obj2 2023-10-18T21:17:26.926 DEBUG:boto:Method: PUT 2023-10-18T21:17:26.927 DEBUG:boto:Path: /bofrnf-70/obj2 2023-10-18T21:17:26.927 DEBUG:boto:Data: 2023-10-18T21:17:26.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:17:26.928 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:17:26.928 DEBUG:boto:Port: 8000 2023-10-18T21:17:26.929 DEBUG:boto:Params: {} 2023-10-18T21:17:26.929 DEBUG:boto:Token: None 2023-10-18T21:17:26.930 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:17:26 GMT /bofrnf-70/obj2 2023-10-18T21:17:26.930 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:KRFJ2tHPjtBLCsqozmW6TODSYGY= 2023-10-18T21:17:26.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:17:26 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KRFJ2tHPjtBLCsqozmW6TODSYGY='} 2023-10-18T21:17:26.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1072244529bb813-0065304b66-262e-a1'), ('Date', 'Wed, 18 Oct 2023 21:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:17:26.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:26.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:26.976 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:27.048 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:27.048 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:27.131 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:17:27.165 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:17:27.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.166 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:17:27.166 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:17:20.749643Z" 2023-10-18T21:17:27.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:17:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.167 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:17:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:27.168 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:17:27.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.169 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:17:27.170 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:27.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:17:27.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:27.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:27.172 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:27.248 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:27.249 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:27.312 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:27.311+0000 7fec60e8d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:27.314 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:17:27.314+0000 7fec60e8d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:17:27.331 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:27.331 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:27.331 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:27.332 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:27.332 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:27.332 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:17:27.332 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:27.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.333 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:27.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.333 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:27.334 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:27.334 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:27.334 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:17:27.334 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:27.335 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:17:27.335 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:27.335 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:20.749643Z", 2023-10-18T21:17:27.335 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:27.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.336 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:27.337 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:27.337 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:27.337 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:17:27.337 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:27.337 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:27.338 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:27.338 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:17:27.338 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:27.338 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.339 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:27.339 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:27.339 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:27.339 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:17:27.340 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:27.340 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:27.340 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:27.340 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:17:27.340 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:27.340 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.341 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:27.341 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:27.341 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:17:27.342 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:17:27.342 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:27.342 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:27.342 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:27.343 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:17:27.343 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:17:27.343 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.343 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.343 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:27.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.344 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:27.344 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:17:27.344 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:17:27.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.345 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:27Z" 2023-10-18T21:17:27.345 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:27.346 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663840.749643_1881.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:17:20.749643Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:17:27Z" } 2023-10-18T21:17:27.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663840.749643_1881.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:17:20.749643Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:17:27.347 DEBUG:rgw_multi.tests:log_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:27.347 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:17:27.348 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:17:27.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:27.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:27.349 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:27.412 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:27.412 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:27.474 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:17:27.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.475 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:17:27.475 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:17:27.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.476 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:17:27.476 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:17:27.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.477 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:17:27.477 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:17:27.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.478 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:17:27.478 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:17:27.478 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.478 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:17:27.479 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:17:27.480 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:17:27.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:27.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:27.481 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:27.533 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:27.534 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:27.576 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:27.577 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:27.577 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:27.577 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:27.577 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:27.577 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:17:27.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.578 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:27.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.578 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:27.579 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:27.579 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:27.579 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:17:27.579 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:27.579 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:27.580 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:27.580 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:17:27.580 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.581 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:27.581 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:27.581 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:27.581 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:17:27.581 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:27.582 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:27.582 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:27.582 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:17:27.582 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.583 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:27.583 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:27.583 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:17:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:17:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:17:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:17:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:17:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:17:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:17:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:17:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:27Z" 2023-10-18T21:17:27.589 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:27.590 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:17:27Z" } 2023-10-18T21:17:27.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:17:27.591 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:17:27.591 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:17:27.591 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:17:32.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:32.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:32.594 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:32.654 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:32.654 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:32.702 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:32.703 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:32.703 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:32.703 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:32.703 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:32.704 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:17:32.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:32.704 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:32.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:32.705 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:32.705 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:32.705 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:32.705 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:17:32.705 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:32.706 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:32.706 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:32.706 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:17:32.706 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:32.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:32.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:32.707 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:32.707 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:32.707 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:32.708 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:17:32.708 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:32.708 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:32.708 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:32.708 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:17:32.709 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:32.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:32.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:32.709 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:32.709 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:32.710 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:32.710 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:17:32.710 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:32.710 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:17:32.711 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:32.711 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:17:32.711 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:32.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:32.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:32.712 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:32.712 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:32.712 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:32.712 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:17:32.712 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:32.713 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:17:32.713 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:32.713 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:17:32.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:32.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:32.714 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:32.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:32.714 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:32.714 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:17:32.714 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:17:32.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:32.715 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:32Z" 2023-10-18T21:17:32.715 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:32.716 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:17:32Z" } 2023-10-18T21:17:32.717 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:17:32.717 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:17:32.717 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:17:32.718 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:17:37.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:37.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:37.720 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:37.778 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:37.779 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:37.826 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:37.827 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:37.827 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:37.827 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:37.827 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:37.828 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:17:37.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:37.828 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:37.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:37.829 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:37.829 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:37.829 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:37.829 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:17:37.829 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:37.830 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:37.830 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:37.830 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:17:37.830 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:37.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:37.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:37.831 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:37.831 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:37.831 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:37.832 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:17:37.832 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:37.832 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:37.832 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:37.832 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:17:37.833 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:37.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:37.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:37.833 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:37.833 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:37.834 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:37.834 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:17:37.834 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:37.834 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:17:37.835 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:37.835 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:17:37.835 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:37.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:37.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:37.836 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:37.836 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:37.836 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:37.836 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:17:37.836 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:37.837 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:17:37.837 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:37.837 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:17:37.837 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:37.837 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:37.838 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:37.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:37.838 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:37.838 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:17:37.839 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:17:37.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:37.839 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:37Z" 2023-10-18T21:17:37.839 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:37.840 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:17:37Z" } 2023-10-18T21:17:37.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:17:37.841 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:17:37.842 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:17:37.842 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:17:42.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:42.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:42.845 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:42.903 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:42.904 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:42.951 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:17:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:17:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:17:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:17:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:17:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:17:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:17:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:17:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:17:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:17:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:17:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:17:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:17:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:42Z" 2023-10-18T21:17:42.964 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:42.965 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:17:42Z" } 2023-10-18T21:17:42.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:17:42.966 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:17:42.967 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:17:42.967 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:17:47.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:47.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:47.970 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:48.029 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:48.029 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:48.078 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:48.078 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:48.078 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:48.079 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:48.079 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:48.079 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:17:48.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:48.080 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:48.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:48.080 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:48.080 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:48.080 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:48.081 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:17:48.081 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:48.081 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:48.081 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:48.082 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:17:48.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:48.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:48.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:48.083 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:48.083 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:48.083 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:48.083 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:17:48.084 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:48.084 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:48.084 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:48.084 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:17:48.085 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:48.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:48.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:48.085 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:48.086 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:48.086 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:48.086 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:17:48.086 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:48.087 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:17:48.087 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:48.087 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:17:48.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:48.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:48.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:48.088 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:48.088 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:48.088 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:48.089 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:17:48.089 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:48.089 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:17:48.089 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:48.090 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:17:48.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:48.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:48.090 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:48.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:48.093 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:48.094 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:17:48.094 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:17:48.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:48.094 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:48Z" 2023-10-18T21:17:48.095 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:48.097 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:17:48Z" } 2023-10-18T21:17:48.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:17:48.098 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:17:48.098 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:17:48.099 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:17:53.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:53.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:53.101 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:53.161 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:53.161 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:53.209 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:53.209 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:53.209 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:53.210 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:53.210 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:53.210 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:17:53.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:53.210 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:53.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:53.211 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:53.211 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:53.211 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:53.212 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:17:53.212 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:53.212 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:53.212 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:53.212 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:17:53.213 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:53.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:53.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:53.213 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:53.213 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:53.214 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:53.214 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:17:53.214 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:53.214 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:53.215 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:53.215 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:17:53.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:53.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:53.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:53.216 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:53.216 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:53.216 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:53.216 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:17:53.216 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:53.217 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:17:53.217 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:53.217 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:17:53.217 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:53.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:53.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:53.218 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:53.218 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:53.218 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:53.218 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:17:53.219 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:53.219 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:17:53.219 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:53.219 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:17:53.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:53.220 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:53.220 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:53.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:53.220 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:53.220 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:17:53.221 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:17:53.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:53.221 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:53Z" 2023-10-18T21:17:53.221 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:53.222 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:17:53Z" } 2023-10-18T21:17:53.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:17:53.223 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:17:53.224 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:17:53.224 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:17:58.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:58.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:17:58.227 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:17:58.285 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:17:58.286 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:17:58.336 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:17:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:17:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:17:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:17:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:17:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:17:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:17:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:17:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:17:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:17:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:17:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:17:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:17:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:17:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:17:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:17:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:17:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:17:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:17:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:17:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:17:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:17:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:17:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:17:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:17:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:17:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:17:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:17:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:17:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:17:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:17:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:17:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:17:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:17:58Z" 2023-10-18T21:17:58.350 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:17:58.351 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:17:58Z" } 2023-10-18T21:17:58.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:17:58.352 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:17:58.352 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:17:58.352 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:03.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:03.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:03.356 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:03.414 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:03.415 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:03.462 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:03.462 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:03.463 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:03.463 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:03.463 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:03.463 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:03.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:03.464 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:03.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:03.464 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:03.464 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:03.465 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:03.465 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:03.465 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:03.465 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:03.466 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:03.466 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:03.466 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:03.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:03.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:03.467 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:03.467 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:03.467 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:03.467 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:18:03.468 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:03.468 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:03.468 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:03.468 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:18:03.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:03.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:03.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:03.469 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:03.469 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:03.470 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:03.470 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:03.470 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:03.470 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:03.471 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:03.471 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:03.471 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:03.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:03.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:03.472 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:03.472 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:03.472 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:03.472 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:03.472 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:03.473 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:03.473 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:03.473 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:03.473 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:03.473 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:03.474 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:03.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:03.474 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:03.474 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:03.475 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:03.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:03.478 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:03Z" 2023-10-18T21:18:03.478 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:03.479 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:03Z" } 2023-10-18T21:18:03.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:03.481 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:03.481 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:03.481 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:08.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:08.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:08.484 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:08.545 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:08.545 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:08.595 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:08.595 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:08.595 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:08.596 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:08.596 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:08.596 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:08.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:08.597 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:08.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:08.597 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:08.597 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:08.597 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:08.598 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:08.598 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:08.598 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:08.598 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:08.599 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:08.599 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:08.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:08.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:08.599 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:08.600 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:08.600 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:08.600 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:18:08.600 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:08.600 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:08.601 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:08.601 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:18:08.601 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:08.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:08.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:08.602 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:08.602 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:08.602 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:08.603 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:08.603 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:08.603 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:08.603 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:08.603 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:08.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:08.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:08.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:08.604 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:08.605 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:08.605 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:08.605 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:08.605 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:08.605 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:08.606 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:08.606 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:08.606 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:08.606 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:08.606 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:08.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:08.607 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:08.607 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:08.607 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:08.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:08.608 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:08Z" 2023-10-18T21:18:08.608 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:08.609 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:08Z" } 2023-10-18T21:18:08.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:08.610 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:08.610 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:08.611 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:13.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:13.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:13.613 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:13.673 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:13.673 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:13.721 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:13.721 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:13.721 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:18:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:18:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:13Z" 2023-10-18T21:18:13.735 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:13.736 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:13Z" } 2023-10-18T21:18:13.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:13.737 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:13.737 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:13.738 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:18.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:18.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:18.740 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:18.798 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:18.798 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:18.845 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:18.845 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:18.845 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:18.846 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:18.846 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:18.846 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:18.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:18.847 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:18.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:18.847 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:18.847 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:18.847 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:18.848 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:18.848 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:18.848 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:18.848 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:18.848 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:18.849 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:18.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:18.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:18.849 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:18.849 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:18.850 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:18.850 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:18:18.850 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:18.850 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:18.851 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:18.851 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:18:18.851 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:18.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:18.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:18.852 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:18.852 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:18.852 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:18.852 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:18.852 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:18.853 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:18.853 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:18.853 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:18.853 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:18.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:18.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:18.854 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:18.854 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:18.854 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:18.854 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:18.855 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:18.855 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:18.855 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:18.855 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:18.856 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:18.856 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:18.856 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:18.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:18.856 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:18.857 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:18.857 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:18.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:18.858 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:18Z" 2023-10-18T21:18:18.858 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:18.860 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:18Z" } 2023-10-18T21:18:18.861 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:18.861 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:18.862 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:18.863 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:23.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:23.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:23.865 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:23.926 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:23.926 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:23.975 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:23.976 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:23.976 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:23.976 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:23.976 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:23.977 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:23.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:23.977 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:23.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:23.978 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:23.978 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:23.978 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:23.978 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:23.978 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:23.979 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:23.979 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:23.979 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:23.979 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:23.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:23.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:23.981 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:23.981 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:23.981 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:23.982 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045519", 2023-10-18T21:18:23.982 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:23.982 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:23.982 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:23.983 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.126573Z" 2023-10-18T21:18:23.983 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:23.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:23.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:23.983 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:23.984 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:23.984 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:23.984 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:23.984 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:23.985 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:23.985 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:23.985 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:23.985 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:23.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:23.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:23.986 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:23.986 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:23.986 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:23.987 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:23.987 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:23.987 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:23.987 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:23.987 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:23.988 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:23.988 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:23.988 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:23.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:23.988 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:23.989 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:23.989 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:23.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:23.989 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:23Z" 2023-10-18T21:18:23.990 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:23.991 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045519", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.126573Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:23Z" } 2023-10-18T21:18:23.992 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045519', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.126573Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:23.992 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:23.993 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045519', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:23.993 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045659 target=00000000000000000000:00000000000000045519, shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:28.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:28.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:28.996 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:29.053 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:29.054 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:29.100 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:18:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:18:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:29Z" 2023-10-18T21:18:29.113 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:29.114 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:29Z" } 2023-10-18T21:18:29.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:29.115 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:29.115 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:29.116 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:34.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:34.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:34.119 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:34.178 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:34.178 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:34.228 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:34.228 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:34.229 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:34.229 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:34.229 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:34.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:34.229 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:34.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:34.230 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:34.230 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:34.230 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:34.232 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:34.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:34.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:34.232 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:34.232 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:34.233 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:34.233 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:18:34.233 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:34.233 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:34.233 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:34.234 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:18:34.234 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:34.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:34.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:34.235 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:34.235 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:34.235 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:34.235 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:34.235 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:34.236 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:34.236 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:34.236 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:34.236 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:34.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:34.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:34.237 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:34.237 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:34.237 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:34.238 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:34.238 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:34.238 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:34.238 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:34.239 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:34.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:34.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:34.239 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:34.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:34.240 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:34.240 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:34.240 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:34.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:34.241 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:34Z" 2023-10-18T21:18:34.241 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:34.242 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:34Z" } 2023-10-18T21:18:34.243 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:34.243 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:34.244 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:34.244 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:39.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:39.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:39.247 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:39.307 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:39.307 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:39.353 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:39.354 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:39.354 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:39.354 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:39.355 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:39.355 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:39.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:39.355 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:39.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:39.356 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:39.356 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:39.356 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:39.356 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:39.356 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:39.357 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:39.357 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:39.357 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:39.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:39.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:39.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:39.358 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:39.358 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:39.358 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:39.359 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:18:39.359 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:39.359 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:39.359 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:39.359 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:18:39.360 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:39.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:39.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:39.360 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:39.360 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:39.361 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:39.361 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:39.361 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:39.361 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:39.361 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:39.362 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:39.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:39.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:39.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:39.362 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:39.363 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:39.363 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:39.363 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:39.363 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:39.363 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:39.364 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:39.364 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:39.364 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:39.364 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:39.365 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:39.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:39.365 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:39.365 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:39.366 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:39.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:39.366 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:39Z" 2023-10-18T21:18:39.366 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:39.367 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:39Z" } 2023-10-18T21:18:39.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:39.368 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:39.369 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:39.369 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:44.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:44.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:44.371 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:44.430 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:44.431 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:44.478 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:44.479 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:44.479 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:44.479 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:44.479 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:44.480 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:44.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:44.480 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:44.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:44.481 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:44.481 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:44.481 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:44.481 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:44.481 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:44.482 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:44.482 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:44.482 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:44.482 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:44.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:44.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:44.483 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:44.483 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:44.483 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:44.484 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:18:44.484 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:44.484 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:44.484 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:44.484 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:18:44.485 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:44.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:44.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:44.485 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:44.485 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:44.486 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:44.486 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:44.486 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:44.486 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:44.487 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:44.487 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:44.487 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:44.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:44.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:44.488 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:44.488 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:44.488 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:44.488 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:44.488 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:44.489 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:44.489 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:44.489 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:44.489 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:44.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:44.490 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:44.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:44.490 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:44.490 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:44.491 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:44.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:44.491 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:44Z" 2023-10-18T21:18:44.491 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:44.492 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:44Z" } 2023-10-18T21:18:44.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:44.493 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:44.494 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:44.494 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:49.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:49.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:49.496 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:49.552 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:49.553 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:49.599 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:49.599 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:49.599 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:49.600 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:49.600 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:49.600 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:49.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:49.601 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:49.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:49.601 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:49.601 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:49.602 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:49.602 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:49.602 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:49.602 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:49.602 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:49.603 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:49.603 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:49.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:49.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:49.604 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:49.604 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:49.604 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:49.604 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:18:49.605 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:49.605 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:49.605 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:49.605 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:18:49.605 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:49.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:49.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:49.606 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:49.606 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:49.607 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:49.607 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:49.607 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:49.607 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:49.607 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:49.608 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:49.608 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:49.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:49.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:49.608 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:49.609 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:49.609 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:49.609 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:49.609 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:49.609 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:49.610 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:49.610 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:49.610 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:49.610 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:49.611 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:49.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:49.611 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:49.611 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:49.611 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:49.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:49.612 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:49Z" 2023-10-18T21:18:49.612 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:49.613 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:49Z" } 2023-10-18T21:18:49.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:49.614 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:49.614 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:49.615 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:54.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:54.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:54.618 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:54.676 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:54.677 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:54.725 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:54.725 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:54.725 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:54.726 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:54.726 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:54.726 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:54.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:54.726 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:54.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:54.727 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:54.727 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:54.727 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:54.728 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:54.728 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:54.728 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:54.728 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:54.728 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:54.729 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:54.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:54.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:54.729 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:54.729 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:54.730 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:54.730 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:18:54.730 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:54.730 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:54.731 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:54.731 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:18:54.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:54.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:54.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:54.732 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:54.732 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:54.732 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:54.732 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:54.732 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:54.733 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:54.733 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:54.733 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:54.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:54.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:54.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:54.734 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:54.734 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:54.734 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:54.734 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051533", 2023-10-18T21:18:54.735 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:54.735 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:54.735 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:54.735 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:14:56.232611Z" 2023-10-18T21:18:54.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:54.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:54.736 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:54.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:54.736 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:54.737 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:54.737 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:54.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:54.737 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:54Z" 2023-10-18T21:18:54.737 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:54.738 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051533", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:14:56.232611Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:54Z" } 2023-10-18T21:18:54.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051533', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:14:56.232611Z'}}] 2023-10-18T21:18:54.739 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:54.740 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051533'} 2023-10-18T21:18:54.740 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051533 2023-10-18T21:18:59.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:59.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:18:59.742 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:18:59.801 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:59.802 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:18:59.850 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:18:59.850 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:18:59.850 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:18:59.851 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:18:59.851 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:18:59.851 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:18:59.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:59.851 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:18:59.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:59.852 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:18:59.852 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:59.852 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:59.852 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:18:59.853 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:59.853 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:59.853 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:59.853 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:18:59.854 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:59.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:59.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:59.854 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:18:59.854 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:59.855 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:59.855 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:18:59.855 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:59.855 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:18:59.855 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:59.856 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:18:59.856 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:59.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:59.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:59.857 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:18:59.857 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:59.857 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:59.857 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:18:59.857 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:59.858 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:18:59.858 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:59.858 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:18:59.858 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:59.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:59.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:18:59.859 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:18:59.859 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:18:59.859 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:18:59.859 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:18:59.860 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:18:59.860 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:18:59.860 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:18:59.860 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:18:59.861 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:59.861 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:18:59.861 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:18:59.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:59.861 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:18:59.862 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:18:59.862 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:18:59.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:18:59.862 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:18:59Z" 2023-10-18T21:18:59.862 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:18:59.864 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:18:59Z" } 2023-10-18T21:18:59.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:18:59.865 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:59.865 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:18:59.865 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:18:59.866 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:18:59.866 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:18:59.867 DEBUG:boto:Method: HEAD 2023-10-18T21:18:59.867 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:18:59.867 DEBUG:boto:Data: 2023-10-18T21:18:59.868 DEBUG:boto:Headers: {} 2023-10-18T21:18:59.868 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:18:59.869 DEBUG:boto:Port: 8001 2023-10-18T21:18:59.869 DEBUG:boto:Params: {} 2023-10-18T21:18:59.870 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:18:59.870 DEBUG:boto:Token: None 2023-10-18T21:18:59.871 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:18:59 GMT /bofrnf-70/ 2023-10-18T21:18:59.871 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:3T0YVkQpFKtfUGw4kxd3drf86vc= 2023-10-18T21:18:59.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:18:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3T0YVkQpFKtfUGw4kxd3drf86vc=', 'Content-Length': '0'} 2023-10-18T21:18:59.877 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052fc1f7080f903bc-0065304bc3-262c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:18:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:18:59.878 DEBUG:boto:path=/bofrnf-70/obj1 2023-10-18T21:18:59.878 DEBUG:boto:auth_path=/bofrnf-70/obj1 2023-10-18T21:18:59.879 DEBUG:boto:Method: HEAD 2023-10-18T21:18:59.879 DEBUG:boto:Path: /bofrnf-70/obj1 2023-10-18T21:18:59.879 DEBUG:boto:Data: 2023-10-18T21:18:59.880 DEBUG:boto:Headers: {} 2023-10-18T21:18:59.880 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:18:59.881 DEBUG:boto:Port: 8001 2023-10-18T21:18:59.881 DEBUG:boto:Params: {} 2023-10-18T21:18:59.881 DEBUG:boto:Token: None 2023-10-18T21:18:59.882 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:18:59 GMT /bofrnf-70/obj1 2023-10-18T21:18:59.882 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1cPKfA+FtHKGR7Ewj2xhRvfFS48= 2023-10-18T21:18:59.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:18:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1cPKfA+FtHKGR7Ewj2xhRvfFS48=', 'Content-Length': '0'} 2023-10-18T21:18:59.884 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000025c52af9cffcc2a8-0065304bc3-262c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:18:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:18:59.885 DEBUG:boto:path=/bofrnf-70/obj2 2023-10-18T21:18:59.885 DEBUG:boto:auth_path=/bofrnf-70/obj2 2023-10-18T21:18:59.886 DEBUG:boto:Method: HEAD 2023-10-18T21:18:59.886 DEBUG:boto:Path: /bofrnf-70/obj2 2023-10-18T21:18:59.887 DEBUG:boto:Data: 2023-10-18T21:18:59.887 DEBUG:boto:Headers: {} 2023-10-18T21:18:59.887 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:18:59.888 DEBUG:boto:Port: 8001 2023-10-18T21:18:59.888 DEBUG:boto:Params: {} 2023-10-18T21:18:59.889 DEBUG:boto:Token: None 2023-10-18T21:18:59.889 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:18:59 GMT /bofrnf-70/obj2 2023-10-18T21:18:59.890 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:yojBC5qVM436NRDPxqZsX1gEF+c= 2023-10-18T21:18:59.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:18:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yojBC5qVM436NRDPxqZsX1gEF+c=', 'Content-Length': '0'} 2023-10-18T21:18:59.892 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004ded1bd8dbcb6e41-0065304bc3-262c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:18:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:18:59.892 DEBUG:boto:path=/bofrnf-71/ 2023-10-18T21:18:59.893 DEBUG:boto:auth_path=/bofrnf-71/ 2023-10-18T21:18:59.893 DEBUG:boto:Method: HEAD 2023-10-18T21:18:59.893 DEBUG:boto:Path: /bofrnf-71/ 2023-10-18T21:18:59.894 DEBUG:boto:Data: 2023-10-18T21:18:59.894 DEBUG:boto:Headers: {} 2023-10-18T21:18:59.895 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:18:59.895 DEBUG:boto:Port: 8001 2023-10-18T21:18:59.895 DEBUG:boto:Params: {} 2023-10-18T21:18:59.896 DEBUG:boto:Token: None 2023-10-18T21:18:59.896 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:18:59 GMT /bofrnf-71/ 2023-10-18T21:18:59.897 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W0TZc5oi7Sv8pb3D9oE4yECUzt8= 2023-10-18T21:18:59.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:18:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W0TZc5oi7Sv8pb3D9oE4yECUzt8=', 'Content-Length': '0'} 2023-10-18T21:18:59.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077cfaf51e8c166d7-0065304bc3-262c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:18:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:18:59.900 DEBUG:boto:path=/bofrnf-71/obj1 2023-10-18T21:18:59.901 DEBUG:boto:auth_path=/bofrnf-71/obj1 2023-10-18T21:18:59.901 DEBUG:boto:Method: HEAD 2023-10-18T21:18:59.902 DEBUG:boto:Path: /bofrnf-71/obj1 2023-10-18T21:18:59.902 DEBUG:boto:Data: 2023-10-18T21:18:59.903 DEBUG:boto:Headers: {} 2023-10-18T21:18:59.903 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:18:59.903 DEBUG:boto:Port: 8001 2023-10-18T21:18:59.904 DEBUG:boto:Params: {} 2023-10-18T21:18:59.905 DEBUG:boto:Token: None 2023-10-18T21:18:59.905 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:18:59 GMT /bofrnf-71/obj1 2023-10-18T21:18:59.905 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pNW1scsFTHVjwpM4R6J69r8Uj1o= 2023-10-18T21:18:59.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:18:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pNW1scsFTHVjwpM4R6J69r8Uj1o=', 'Content-Length': '0'} 2023-10-18T21:18:59.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:17:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-70:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bceea7243c86cd09-0065304bc3-262c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:18:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:18:59.909 DEBUG:boto:path=/bofrnf-71/obj1 2023-10-18T21:18:59.909 DEBUG:boto:auth_path=/bofrnf-71/obj1 2023-10-18T21:18:59.910 DEBUG:boto:Method: GET 2023-10-18T21:18:59.910 DEBUG:boto:Path: /bofrnf-71/obj1 2023-10-18T21:18:59.911 DEBUG:boto:Data: 2023-10-18T21:18:59.911 DEBUG:boto:Headers: {} 2023-10-18T21:18:59.911 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:18:59.912 DEBUG:boto:Port: 8001 2023-10-18T21:18:59.912 DEBUG:boto:Params: {} 2023-10-18T21:18:59.913 DEBUG:boto:Token: None 2023-10-18T21:18:59.913 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:18:59 GMT /bofrnf-71/obj1 2023-10-18T21:18:59.914 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Wyz1F3noSl59u6GgqQJAysWBFp8= 2023-10-18T21:18:59.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:18:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Wyz1F3noSl59u6GgqQJAysWBFp8=', 'Content-Length': '0'} 2023-10-18T21:18:59.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:17:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-70:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067fba2de53166112-0065304bc3-262c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:18:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:18:59.917 DEBUG:boto:path=/bofrnf-71/obj2 2023-10-18T21:18:59.917 DEBUG:boto:auth_path=/bofrnf-71/obj2 2023-10-18T21:18:59.918 DEBUG:boto:Method: HEAD 2023-10-18T21:18:59.918 DEBUG:boto:Path: /bofrnf-71/obj2 2023-10-18T21:18:59.919 DEBUG:boto:Data: 2023-10-18T21:18:59.919 DEBUG:boto:Headers: {} 2023-10-18T21:18:59.919 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:18:59.920 DEBUG:boto:Port: 8001 2023-10-18T21:18:59.920 DEBUG:boto:Params: {} 2023-10-18T21:18:59.921 DEBUG:boto:Token: None 2023-10-18T21:18:59.921 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:18:59 GMT /bofrnf-71/obj2 2023-10-18T21:18:59.921 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:926bcHTQ8thqeWZqZK9EX/CqASo= 2023-10-18T21:18:59.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:18:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:926bcHTQ8thqeWZqZK9EX/CqASo=', 'Content-Length': '0'} 2023-10-18T21:18:59.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:17:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-70:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb5ee2fa37979fe3-0065304bc3-262c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:18:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:18:59.924 DEBUG:boto:path=/bofrnf-71/obj2 2023-10-18T21:18:59.925 DEBUG:boto:auth_path=/bofrnf-71/obj2 2023-10-18T21:18:59.925 DEBUG:boto:Method: GET 2023-10-18T21:18:59.926 DEBUG:boto:Path: /bofrnf-71/obj2 2023-10-18T21:18:59.926 DEBUG:boto:Data: 2023-10-18T21:18:59.926 DEBUG:boto:Headers: {} 2023-10-18T21:18:59.927 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:18:59.927 DEBUG:boto:Port: 8001 2023-10-18T21:18:59.928 DEBUG:boto:Params: {} 2023-10-18T21:18:59.928 DEBUG:boto:Token: None 2023-10-18T21:18:59.929 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:18:59 GMT /bofrnf-71/obj2 2023-10-18T21:18:59.929 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:7GtwwI3Fss5ph22/Bh2A4jWfbO8= 2023-10-18T21:18:59.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:18:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7GtwwI3Fss5ph22/Bh2A4jWfbO8=', 'Content-Length': '0'} 2023-10-18T21:18:59.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:17:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-70:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072f829bbb9190284-0065304bc3-262c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:18:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:18:59.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:18:59.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:18:59.933 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:18:59.984 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:18:59.984 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:00.030 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-10-18T21:19:00.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.031 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:19:00.083 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:00.083 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:00.132 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:19:00.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.133 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:19:00.182 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:00.183 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:00.228 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:19:00.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.230 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:19:00.278 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:00.279 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:00.329 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 19, 2023-10-18T21:19:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:19:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:19:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:19:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:19:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:19:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:19:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:19:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:19:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:19:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:19:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:19:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:19:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:19:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:19:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:19:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:19:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:19:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:19:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:19:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:19:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:19:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:19:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:19:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:19:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:19:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:19:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:19:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:19:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:19:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:19:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:19:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:19:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:19:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:19:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:19:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:19:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:19:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:19:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:19:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:19:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:19:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:19:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:19:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:19:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:19:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:19:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:19:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:19:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:19:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:19:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:19:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:19:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [ 2023-10-18T21:19:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-group", 2023-10-18T21:19:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "data_flow": { 2023-10-18T21:19:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "symmetrical": [ 2023-10-18T21:19:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-flow1", 2023-10-18T21:19:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:19:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:19:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:19:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: "pipes": [ 2023-10-18T21:19:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe", 2023-10-18T21:19:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:19:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:19:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:19:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:19:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:19:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:19:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:19:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:19:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:19:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:19:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:19:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:19:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:19:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:19:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:19:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:19:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:19:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:19:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "allowed" 2023-10-18T21:19:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:19:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:19:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:19:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:19:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:19:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:19:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:19:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:19:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:19:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:19:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:19:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:19:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:19:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:19:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:19:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:19:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:19:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:19:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:19:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:19:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:19:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:19:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:19:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:19:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:19:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:19:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:19:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:19:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:19:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:19:00.363 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:19:00.363 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.363 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:19:00.363 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:19:00.363 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:19:00.364 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:19:00.364 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:19:00.364 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:19:00.364 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:19:00.364 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:19:00.365 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:19:00.365 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:19:00.365 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:19:00.365 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.365 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.366 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.366 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:19:00.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.366 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:19:00.367 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:19:00.367 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:19:00.367 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:19:00.367 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.367 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.368 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.368 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:19:00.368 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:19:00.368 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:19:00.368 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:19:00.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.369 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:19:00.369 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:19:00.369 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.369 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.370 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:19:00.370 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:19:00.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.370 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:19:00.370 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:19:00.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.371 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:19:00.371 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:19:00.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.373 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:19:00.373 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:19:00.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:00.374 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:19:00.374 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:19:00.374 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.375 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:00.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.375 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:19:00.375 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:19:00.376 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:19:00.376 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:19:00.376 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:19:00.376 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:19:00.377 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:19:00.377 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:19:00.377 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:19:00.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.377 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:19:00.378 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:19:00.378 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:19:00.378 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:19:00.378 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:19:00.379 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:19:00.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.379 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:19:00.379 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:19:00.379 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:19:00.380 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:19:00.380 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:19:00.380 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:19:00.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.381 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:19:00.381 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:19:00.381 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:19:00.381 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:19:00.381 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:19:00.382 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:19:00.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.382 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:19:00.382 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:19:00.383 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:19:00.383 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:19:00.383 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:19:00.383 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:19:00.383 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:00.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:00.384 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:19:00.384 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:19:00.384 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:00.384 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:00.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.386 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:19:00.432 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:00.432 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:00.476 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:19:00.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.478 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:19:00.539 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:00.539 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:00.636 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:19:00.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.637 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:19:00.717 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:00.718 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:00.821 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-18T21:19:00.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=bofrnf-70', '--source-bucket=*', '--dest-bucket=bofrnf-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=bofrnf-70', '--source-bucket=*', '--dest-bucket=bofrnf-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.823 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=bofrnf-70 '--source-bucket=*' --dest-bucket=bofrnf-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:19:00.897 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:00.898 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:00.974 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"bofrnf-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-18T21:19:00.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:00.976 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:19:01.041 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:01.042 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:01.114 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"bofrnf-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-18T21:19:01.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:01.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:19:01.115 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:19:01.162 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:01.163 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:01.207 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"bofrnf-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-18T21:19:01.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:01.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:01.209 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:01.257 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:01.257 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:01.287 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:19:01.300 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:19:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663941.096796_1971.1", 2023-10-18T21:19:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:19:01.096796Z" 2023-10-18T21:19:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:19:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:19:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:19:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:01.304 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:19:01.306 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:01.307 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:19:01.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:01.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:01.309 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:01.357 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:01.358 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:01.386 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:01.385+0000 7efed99cb780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:01.387 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:01.386+0000 7efed99cb780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:01.400 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:01.400 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:01.401 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:01.401 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:01.401 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:01.401 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:01.401 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:01.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:01.402 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:01.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:01.403 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:01.403 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:01.403 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:01.403 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:19:01.403 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:01.404 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:19:01.404 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:01.404 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:20.749643Z", 2023-10-18T21:19:01.404 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:01.405 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:01.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:01.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:01.405 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:01.405 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:01.406 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:01.406 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:01.406 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:01.407 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:01.407 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:01.407 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:19:01.407 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:01.407 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:01.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:01.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:01.408 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:01.408 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:01.409 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:01.409 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:01.409 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:01.409 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:01.409 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:01.410 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:19:01.410 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:01.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:01.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:01.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:01.411 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:01.411 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:01.411 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:01.411 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:01.412 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:01.412 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:01.412 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:01.412 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:19:01.413 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:01.413 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:01.413 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:01.413 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:01.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:01.414 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:01.414 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:19:01.414 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:19:01.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:01.415 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:01Z" 2023-10-18T21:19:01.416 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:01.417 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663840.749643_1881.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:17:20.749643Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:19:01Z" } 2023-10-18T21:19:01.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663840.749643_1881.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:17:20.749643Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:19:01.419 DEBUG:rgw_multi.tests:log_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:01.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:01.419 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663941.096796_1971.1 target=1_1697663840.749643_1881.1 2023-10-18T21:19:06.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:06.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:06.422 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:06.478 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:06.479 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:06.511 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:06.510+0000 7ff7cb976780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:06.512 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:06.511+0000 7ff7cb976780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:06.526 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:19:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:19:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:20.749643Z", 2023-10-18T21:19:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:19:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:19:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:19:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:19:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:19:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:06Z" 2023-10-18T21:19:06.541 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:06.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663840.749643_1881.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:17:20.749643Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:19:06Z" } 2023-10-18T21:19:06.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663840.749643_1881.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:17:20.749643Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:19:06.543 DEBUG:rgw_multi.tests:log_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:06.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:06.543 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663941.096796_1971.1 target=1_1697663840.749643_1881.1 2023-10-18T21:19:11.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:11.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:11.546 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:11.603 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:11.603 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:11.635 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:11.634+0000 7f27acc3d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:11.636 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:11.635+0000 7f27acc3d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:11.650 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:11.651 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:11.651 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:11.651 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:11.651 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:11.651 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:11.652 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:11.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:11.652 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:11.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:11.653 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:11.653 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:11.653 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:11.653 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:19:11.653 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:11.654 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:19:11.654 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:11.654 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:20.749643Z", 2023-10-18T21:19:11.654 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:11.654 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:11.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:11.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:11.655 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:11.655 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:11.656 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:11.656 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:11.656 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:11.656 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:11.656 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:11.657 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:19:11.657 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:11.657 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:11.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:11.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:11.658 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:11.658 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:11.658 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:11.658 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:11.659 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:11.659 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:11.659 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:11.659 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:19:11.659 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:11.660 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:11.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:11.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:11.660 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:11.660 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:11.661 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:11.661 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:11.661 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:11.661 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:11.661 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:11.662 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:19:11.662 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:11.662 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:11.662 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:11.663 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:11.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:11.663 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:11.663 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:19:11.663 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:19:11.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:11.664 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:11Z" 2023-10-18T21:19:11.664 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:11.665 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663840.749643_1881.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:17:20.749643Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:19:11Z" } 2023-10-18T21:19:11.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663840.749643_1881.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:17:20.749643Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:19:11.666 DEBUG:rgw_multi.tests:log_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:11.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:11.667 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663941.096796_1971.1 target=1_1697663840.749643_1881.1 2023-10-18T21:19:16.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:16.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:16.669 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:16.727 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:16.728 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:16.759 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:16.758+0000 7fd4b050f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:16.760 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:16.759+0000 7fd4b050f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:16.774 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:16.774 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:16.774 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:16.775 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:16.775 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:16.775 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:16.775 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:16.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:16.776 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:16.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:16.776 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:16.776 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:16.776 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:16.777 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:19:16.777 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:16.777 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:19:16.777 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:16.777 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:20.749643Z", 2023-10-18T21:19:16.778 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:16.778 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:16.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:16.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:16.778 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:16.779 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:16.779 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:16.779 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:16.779 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:16.779 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:16.780 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:16.780 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:19:16.780 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:16.780 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:16.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:16.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:16.781 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:16.781 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:16.781 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:16.782 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:16.782 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:16.782 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:16.782 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:16.782 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:19:16.783 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:16.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:16.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:16.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:16.784 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:16.784 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:16.784 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:16.784 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:16.784 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:16.785 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:16.785 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:16.785 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:19:16.785 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:16.785 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:16.786 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:16.786 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:16.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:16.786 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:16.786 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:19:16.787 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:19:16.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:16.787 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:16Z" 2023-10-18T21:19:16.787 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:16.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663840.749643_1881.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:17:20.749643Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:19:16Z" } 2023-10-18T21:19:16.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663840.749643_1881.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:17:20.749643Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:19:16.789 DEBUG:rgw_multi.tests:log_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:16.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:16.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663941.096796_1971.1 target=1_1697663840.749643_1881.1 2023-10-18T21:19:21.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:21.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:21.792 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:21.849 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:21.850 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:21.882 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:21.881+0000 7ff38aa9d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:21.883 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:21.882+0000 7ff38aa9d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:21.897 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:21.897 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:21.898 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:21.898 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:21.898 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:21.898 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:21.899 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:21.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:21.899 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:21.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:21.900 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:21.900 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:21.900 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:21.900 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:19:21.900 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:21.901 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:19:21.901 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:21.901 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:20.749643Z", 2023-10-18T21:19:21.901 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:21.902 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:21.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:21.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:21.902 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:21.903 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:21.903 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:21.903 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:21.903 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:21.903 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:21.904 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:21.904 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:19:21.904 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:21.904 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:21.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:21.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:21.905 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:21.905 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:21.905 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:21.905 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:21.906 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:21.906 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:21.906 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:21.906 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:19:21.906 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:21.907 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:21.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:21.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:21.907 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:21.907 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:21.908 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:21.908 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:21.908 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:21.908 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:21.908 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:21.909 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:19:21.909 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:21.909 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:21.909 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:21.909 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:21.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:21.910 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:21.910 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:19:21.910 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:19:21.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:21.911 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:21Z" 2023-10-18T21:19:21.911 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:21.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663840.749643_1881.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:17:20.749643Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:19:21Z" } 2023-10-18T21:19:21.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663840.749643_1881.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:17:20.749643Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:19:21.913 DEBUG:rgw_multi.tests:log_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:21.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:21.914 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663941.096796_1971.1 target=1_1697663840.749643_1881.1 2023-10-18T21:19:26.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:26.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:26.917 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:26.974 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:26.974 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:27.006 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:27.005+0000 7f1433fc1780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:27.007 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:27.006+0000 7f1433fc1780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:27.021 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:27.021 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:27.022 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:27.022 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:27.022 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:27.023 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:27.023 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:27.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:27.024 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:27.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:27.025 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:27.025 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:27.026 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:27.026 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663840.749643_1881.1", 2023-10-18T21:19:27.026 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:27.027 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:19:27.027 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:27.027 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:20.749643Z", 2023-10-18T21:19:27.027 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:27.028 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:27.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:27.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:27.029 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:27.029 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:27.029 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:27.030 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:27.030 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:27.030 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:27.030 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:27.031 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:19:27.031 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:27.031 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:27.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:27.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:27.032 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:27.032 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:27.032 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:27.032 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:27.033 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:27.033 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:27.033 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:27.033 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:19:27.034 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:27.034 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:27.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:27.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:27.035 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:27.035 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:27.035 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:27.036 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:27.036 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:27.036 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:27.036 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:27.037 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:19:27.037 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:27.037 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:27.037 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:27.038 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:27.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:27.038 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:27.038 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:19:27.038 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:19:27.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:27.039 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:27Z" 2023-10-18T21:19:27.039 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:27.040 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663840.749643_1881.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:17:20.749643Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:19:27Z" } 2023-10-18T21:19:27.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663840.749643_1881.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:17:20.749643Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:19:27.041 DEBUG:rgw_multi.tests:log_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:27.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663840.749643_1881.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:27.042 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697663941.096796_1971.1 target=1_1697663840.749643_1881.1 2023-10-18T21:19:32.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.045 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:32.103 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:32.103 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:32.135 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:32.134+0000 7f3fec1ea780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:32.136 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:32.135+0000 7f3fec1ea780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:32.150 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:32.150 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:32.151 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:32.151 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:32.151 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:32.151 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:32.152 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.152 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:32.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.152 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:32.153 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.153 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:32.153 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663941.096796_1971.1", 2023-10-18T21:19:32.153 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.154 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:19:32.154 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.154 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:01.096796Z", 2023-10-18T21:19:32.154 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.155 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.155 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:32.155 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.156 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:32.156 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:32.156 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.156 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:32.157 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.157 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:19:32.157 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.158 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:19:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.161 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:32.161 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.161 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:32.161 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:32.162 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.162 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:32.162 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.162 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:19:32.163 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.163 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:32.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.164 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:32.164 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:19:32.164 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:19:32.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.164 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:32Z" 2023-10-18T21:19:32.165 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:32.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663941.096796_1971.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:19:01.096796Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:19:32Z" } 2023-10-18T21:19:32.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663941.096796_1971.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:19:01.096796Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:19:32.167 DEBUG:rgw_multi.tests:log_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:32.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:32.168 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:19:32.168 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:19:32.168 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:19:32.169 DEBUG:boto:Method: HEAD 2023-10-18T21:19:32.169 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:19:32.170 DEBUG:boto:Data: 2023-10-18T21:19:32.170 DEBUG:boto:Headers: {} 2023-10-18T21:19:32.170 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:19:32.171 DEBUG:boto:Port: 8000 2023-10-18T21:19:32.171 DEBUG:boto:Params: {} 2023-10-18T21:19:32.172 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:19:32.172 DEBUG:boto:Token: None 2023-10-18T21:19:32.173 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:19:32 GMT /bofrnf-70/ 2023-10-18T21:19:32.173 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B7er5kirCJucrxZYaRGUm5dZVN8= 2023-10-18T21:19:32.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:19:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B7er5kirCJucrxZYaRGUm5dZVN8=', 'Content-Length': '0'} 2023-10-18T21:19:32.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf3e4e4aca1667bf-0065304be4-270f-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:19:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:19:32.180 DEBUG:boto:path=/bofrnf-70/obj5 2023-10-18T21:19:32.180 DEBUG:boto:auth_path=/bofrnf-70/obj5 2023-10-18T21:19:32.181 DEBUG:boto:Method: PUT 2023-10-18T21:19:32.181 DEBUG:boto:Path: /bofrnf-70/obj5 2023-10-18T21:19:32.182 DEBUG:boto:Data: 2023-10-18T21:19:32.182 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:19:32.183 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:19:32.183 DEBUG:boto:Port: 8000 2023-10-18T21:19:32.184 DEBUG:boto:Params: {} 2023-10-18T21:19:32.184 DEBUG:boto:Token: None 2023-10-18T21:19:32.184 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:19:32 GMT /bofrnf-70/obj5 2023-10-18T21:19:32.185 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4OaBFBcer6RBqcLosKJW4pOiUfk= 2023-10-18T21:19:32.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:19:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4OaBFBcer6RBqcLosKJW4pOiUfk='} 2023-10-18T21:19:32.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fba3c4690adc559a-0065304be4-270f-a1'), ('Date', 'Wed, 18 Oct 2023 21:19:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:19:32.194 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:19:32.195 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:19:32.195 DEBUG:boto:Method: HEAD 2023-10-18T21:19:32.196 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:19:32.196 DEBUG:boto:Data: 2023-10-18T21:19:32.197 DEBUG:boto:Headers: {} 2023-10-18T21:19:32.197 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:19:32.197 DEBUG:boto:Port: 8000 2023-10-18T21:19:32.198 DEBUG:boto:Params: {} 2023-10-18T21:19:32.202 DEBUG:boto:Token: None 2023-10-18T21:19:32.202 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:19:32 GMT /bofrnf-70/ 2023-10-18T21:19:32.203 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:B7er5kirCJucrxZYaRGUm5dZVN8= 2023-10-18T21:19:32.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:19:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B7er5kirCJucrxZYaRGUm5dZVN8=', 'Content-Length': '0'} 2023-10-18T21:19:32.206 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000df2145de8664d20-0065304be4-270f-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:19:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:19:32.206 DEBUG:boto:path=/bofrnf-70/obj6 2023-10-18T21:19:32.207 DEBUG:boto:auth_path=/bofrnf-70/obj6 2023-10-18T21:19:32.207 DEBUG:boto:Method: PUT 2023-10-18T21:19:32.208 DEBUG:boto:Path: /bofrnf-70/obj6 2023-10-18T21:19:32.208 DEBUG:boto:Data: 2023-10-18T21:19:32.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:19:32.209 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:19:32.209 DEBUG:boto:Port: 8000 2023-10-18T21:19:32.210 DEBUG:boto:Params: {} 2023-10-18T21:19:32.210 DEBUG:boto:Token: None 2023-10-18T21:19:32.211 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:19:32 GMT /bofrnf-70/obj6 2023-10-18T21:19:32.211 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bR5F7d09RE+25G1X0n3YTRf9kRI= 2023-10-18T21:19:32.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:19:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bR5F7d09RE+25G1X0n3YTRf9kRI='} 2023-10-18T21:19:32.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e88c371878bd3740-0065304be4-270f-a1'), ('Date', 'Wed, 18 Oct 2023 21:19:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:19:32.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.221 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:32.270 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:32.271 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:32.306 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:19:32.320 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:19:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663941.096796_1971.1", 2023-10-18T21:19:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:19:01.096796Z" 2023-10-18T21:19:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:19:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:19:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:19:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.323 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:19:32.324 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:32.325 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:19:32.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.326 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:32.375 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:32.376 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:32.406 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:32.404+0000 7f2a9624c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:32.407 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:19:32.405+0000 7f2a9624c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:19:32.419 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:32.419 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:32.419 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:32.419 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:32.420 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:32.420 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:19:32.420 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.420 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:32.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.421 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:32.421 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.421 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:32.421 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663941.096796_1971.1", 2023-10-18T21:19:32.422 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.422 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:19:32.422 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.422 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:01.096796Z", 2023-10-18T21:19:32.423 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.423 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.423 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:32.424 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.424 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:32.424 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:19:32.424 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.424 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:32.425 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.425 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:19:32.425 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.426 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:32.426 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.427 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:32.427 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:19:32.427 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.427 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:32.427 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.428 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:19:32.428 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.428 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.429 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:32.429 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.429 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:19:32.429 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:19:32.429 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.430 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:32.430 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.430 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:19:32.430 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:19:32.431 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.431 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.431 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:32.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.431 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:32.432 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:19:32.432 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:19:32.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.432 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:32Z" 2023-10-18T21:19:32.432 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:32.433 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663941.096796_1971.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:19:01.096796Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:19:32Z" } 2023-10-18T21:19:32.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663941.096796_1971.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:19:01.096796Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:19:32.434 DEBUG:rgw_multi.tests:log_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:32.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:19:32.435 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:19:32.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.436 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:32.484 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:32.485 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:32.533 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:19:32.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.533 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:19:32.534 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:19:32.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.534 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:19:32.535 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:19:32.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.535 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:19:32.535 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:19:32.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.536 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:19:32.536 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:19:32.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.536 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:19:32.537 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:19:32.538 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:19:32.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:32.539 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:32.591 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:32.591 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:32.637 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:32.637 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:32.637 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:32.637 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:32.638 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:32.638 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:19:32.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.638 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:32.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.639 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:32.639 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.639 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:32.639 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:19:32.640 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.640 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:32.640 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.640 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:19:32.640 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.641 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:32.641 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.641 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:32.642 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:19:32.642 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.642 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:32.642 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.642 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:19:32.643 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.643 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:32.644 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.644 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:32.644 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:19:32.644 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.644 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:19:32.645 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.645 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:19:32.645 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:32.646 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:32.646 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:32.646 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:32.646 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:19:32.647 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:32.647 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:19:32.647 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:32.647 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:19:32.647 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.648 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:32.648 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:32.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.648 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:32.648 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:19:32.649 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:19:32.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:32.649 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:32Z" 2023-10-18T21:19:32.649 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:32.650 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:19:32Z" } 2023-10-18T21:19:32.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:19:32.651 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:19:32.652 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:19:32.652 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:19:37.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:37.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:37.656 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:37.715 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:37.716 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:37.764 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:37.764 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:37.765 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:37.765 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:37.766 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:37.766 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:19:37.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:37.767 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:37.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:37.768 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:37.769 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:37.769 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:37.770 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:19:37.770 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:37.771 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:37.771 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:37.772 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:19:37.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:37.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:37.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:37.773 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:37.773 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:37.774 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:37.775 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:19:37.775 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:37.776 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:37.776 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:37.777 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:19:37.777 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:37.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:37.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:37.779 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:37.779 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:37.780 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:37.780 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:19:37.781 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:37.781 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:19:37.782 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:37.782 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:19:37.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:37.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:37.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:37.785 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:37.785 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:37.786 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:37.786 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:19:37.787 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:37.787 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:19:37.788 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:37.788 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:19:37.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:37.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:37.789 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:19:37.791 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:19:37.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:37.792 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:37Z" 2023-10-18T21:19:37.792 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:37.794 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:19:37Z" } 2023-10-18T21:19:37.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:19:37.795 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:19:37.796 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:19:37.796 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:19:42.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:42.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:42.799 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:42.858 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:42.859 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:42.907 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:19:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:19:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:19:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:19:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:19:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:19:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:19:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:19:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:19:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:19:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:19:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:19:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:19:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:42Z" 2023-10-18T21:19:42.920 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:42.921 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:19:42Z" } 2023-10-18T21:19:42.921 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:19:42.922 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:19:42.922 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:19:42.922 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:19:47.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:47.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:47.925 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:47.985 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:47.985 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:48.034 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:48.035 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:48.035 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:48.035 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:48.036 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:48.036 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:19:48.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:48.036 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:48.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:48.037 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:48.037 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:48.037 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:48.037 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:19:48.038 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:48.038 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:48.038 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:48.038 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:19:48.038 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:48.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:48.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:48.039 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:48.039 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:48.039 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:48.040 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:19:48.040 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:48.040 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:48.040 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:48.040 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:19:48.041 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:48.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:48.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:48.041 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:48.041 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:48.042 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:48.042 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:19:48.042 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:48.042 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:19:48.043 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:48.043 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:19:48.043 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:48.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:48.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:48.043 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:48.044 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:48.044 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:48.044 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:19:48.044 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:48.044 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:19:48.045 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:48.045 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:19:48.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:48.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:48.045 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:48.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:48.046 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:48.046 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:19:48.046 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:19:48.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:48.047 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:48Z" 2023-10-18T21:19:48.047 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:48.048 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:19:48Z" } 2023-10-18T21:19:48.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:19:48.049 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:19:48.049 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:19:48.049 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:19:53.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:53.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:53.052 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:53.111 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:53.111 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:53.161 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:53.161 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:53.162 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:53.162 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:53.162 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:53.162 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:19:53.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:53.163 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:53.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:53.163 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:53.163 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:53.164 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:53.164 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:19:53.164 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:53.164 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:53.165 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:53.165 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:19:53.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:53.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:53.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:53.166 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:53.166 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:53.166 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:53.166 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:19:53.167 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:53.167 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:53.167 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:53.167 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:19:53.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:53.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:53.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:53.168 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:53.168 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:53.168 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:53.169 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:19:53.169 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:53.169 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:19:53.169 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:53.169 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:19:53.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:53.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:53.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:53.170 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:53.170 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:53.171 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:53.171 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:19:53.171 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:53.171 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:19:53.172 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:53.172 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:19:53.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:53.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:53.172 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:53.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:53.173 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:53.173 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:19:53.173 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:19:53.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:53.174 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:53Z" 2023-10-18T21:19:53.174 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:53.175 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:19:53Z" } 2023-10-18T21:19:53.175 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:19:53.176 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:19:53.177 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:19:53.177 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:19:58.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:58.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:19:58.180 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:19:58.240 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:19:58.240 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:19:58.288 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:19:58.289 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:19:58.289 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:19:58.289 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:19:58.289 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:19:58.289 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:19:58.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:58.290 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:19:58.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:58.290 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:19:58.291 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:58.291 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:58.291 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:19:58.291 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:58.291 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:58.292 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:58.292 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:19:58.292 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:58.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:58.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:58.293 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:19:58.293 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:58.293 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:58.294 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:19:58.294 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:58.294 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:19:58.294 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:58.294 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:19:58.295 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:58.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:58.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:58.295 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:19:58.295 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:58.296 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:58.296 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:19:58.296 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:58.296 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:19:58.296 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:58.297 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:19:58.297 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:58.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:58.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:19:58.298 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:19:58.298 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:19:58.298 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:19:58.298 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:19:58.298 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:19:58.299 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:19:58.299 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:19:58.299 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:19:58.299 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:58.299 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:19:58.300 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:19:58.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:58.300 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:19:58.300 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:19:58.300 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:19:58.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:19:58.301 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:19:58Z" 2023-10-18T21:19:58.301 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:19:58.302 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:19:58Z" } 2023-10-18T21:19:58.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:19:58.303 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:19:58.303 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:19:58.304 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:03.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:03.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:03.306 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:03.367 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:03.367 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:03.415 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:03.415 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:03.415 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:03.415 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:03.416 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:03.416 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:03.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:03.416 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:03.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:03.417 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:03.417 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:03.417 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:03.417 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:03.418 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:03.418 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:03.418 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:03.418 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:03.419 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:03.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:03.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:03.419 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:03.419 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:03.420 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:03.420 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:20:03.420 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:03.420 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:03.420 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:03.421 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:20:03.421 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:03.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:03.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:03.422 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:03.422 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:03.422 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:03.422 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:03.423 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:03.423 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:03.423 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:03.423 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:03.423 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:03.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:03.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:03.424 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:03.424 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:03.424 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:03.425 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:03.425 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:03.425 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:03.425 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:03.426 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:03.426 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:03.426 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:03.426 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:03.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:03.427 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:03.427 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:03.427 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:03.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:03.428 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:03Z" 2023-10-18T21:20:03.428 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:03.429 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:03Z" } 2023-10-18T21:20:03.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:03.430 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:03.431 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:03.431 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:08.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:08.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:08.433 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:08.492 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:08.493 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:08.541 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:08.541 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:08.541 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:08.541 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:08.542 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:08.542 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:08.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:08.542 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:08.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:08.543 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:08.543 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:08.543 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:08.543 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:08.544 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:08.544 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:08.544 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:08.544 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:08.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:08.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:08.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:08.545 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:08.545 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:08.546 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:08.546 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:20:08.546 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:08.546 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:08.546 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:08.547 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:20:08.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:08.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:08.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:08.547 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:08.548 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:08.548 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:08.548 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:08.548 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:08.548 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:08.549 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:08.549 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:08.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:08.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:08.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:08.550 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:08.550 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:08.550 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:08.550 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:08.550 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:08.551 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:08.551 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:08.551 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:08.551 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:08.552 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:08.552 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:08.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:08.552 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:08.552 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:08.553 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:08.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:08.553 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:08Z" 2023-10-18T21:20:08.553 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:08.554 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:08Z" } 2023-10-18T21:20:08.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:08.555 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:08.556 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:08.556 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:13.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:13.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:13.558 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:13.617 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:13.618 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:13.664 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:13.665 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:13.665 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:13.665 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:13.665 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:13.666 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:13.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:13.666 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:13.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:13.666 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:13.667 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:13.667 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:13.667 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:13.667 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:13.668 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:13.668 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:13.668 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:13.668 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:13.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:13.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:13.669 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:13.669 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:13.669 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:13.669 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:20:13.670 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:13.670 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:13.670 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:13.670 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:20:13.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:13.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:13.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:13.671 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:13.671 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:13.672 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:13.672 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:13.672 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:13.672 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:13.672 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:13.673 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:13.673 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:13.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:13.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:13.673 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:13.674 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:13.674 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:13.674 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:13.674 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:13.674 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:13.675 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:13.675 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:13.675 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:13.675 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:13.675 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:13.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:13.676 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:13.676 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:13.676 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:13.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:13.677 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:13Z" 2023-10-18T21:20:13.677 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:13.678 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:13Z" } 2023-10-18T21:20:13.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:13.679 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:13.679 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:13.680 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:18.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:18.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:18.682 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:18.741 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:18.741 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:18.791 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:20:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:20:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:18Z" 2023-10-18T21:20:18.805 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:18.806 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:18Z" } 2023-10-18T21:20:18.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:18.807 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:18.807 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:18.808 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:23.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:23.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:23.810 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:23.869 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:23.870 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:23.918 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:20:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:20:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:23.932 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:23.932 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:23.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:23.932 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:23Z" 2023-10-18T21:20:23.932 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:23.933 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:23Z" } 2023-10-18T21:20:23.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:23.934 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:23.935 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:23.935 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:28.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:28.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:28.937 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:28.996 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:28.997 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:29.045 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045659", 2023-10-18T21:20:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.947053Z" 2023-10-18T21:20:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:29.058 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:29Z" 2023-10-18T21:20:29.058 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:29.059 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045659", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:17:26.947053Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:29Z" } 2023-10-18T21:20:29.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045659', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.947053Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:29.060 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:29.060 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045659', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:29.061 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045799 target=00000000000000000000:00000000000000045659, shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:34.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:34.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:34.063 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:34.123 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:34.124 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:34.179 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:34.180 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:34.180 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:34.180 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:34.181 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:34.181 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:34.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:34.181 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:34.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:34.182 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:34.182 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:34.182 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:34.182 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:34.182 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:34.183 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:34.183 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:34.183 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:34.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:34.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:34.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:34.184 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:34.184 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:34.184 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:34.185 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:20:34.185 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:34.185 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:34.185 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:34.185 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:20:34.186 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:34.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:34.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:34.186 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:34.186 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:34.187 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:34.187 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:34.187 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:34.187 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:34.187 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:34.188 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:34.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:34.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:34.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:34.188 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:34.189 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:34.189 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:34.189 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:34.189 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:34.189 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:34.190 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:34.190 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:34.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:34.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:34.191 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:34.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:34.191 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:34.191 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:34.192 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:34.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:34.192 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:34Z" 2023-10-18T21:20:34.192 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:34.193 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:34Z" } 2023-10-18T21:20:34.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:34.194 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:34.194 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:34.195 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:39.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:39.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:39.197 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:39.257 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:39.257 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:39.306 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:39.306 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:39.306 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:39.306 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:39.307 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:39.307 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:39.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:39.307 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:39.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:39.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:39.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:39.310 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:39.310 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:39.310 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:39.311 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:20:39.311 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:39.311 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:39.311 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:39.311 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:20:39.312 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:39.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:39.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:39.312 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:39.312 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:39.313 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:39.313 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:39.313 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:39.314 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:39.314 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:39.314 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:39.314 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:39.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:39.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:39.315 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:39.315 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:39.315 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:39.315 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:39.316 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:39.316 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:39.316 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:39.316 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:39.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:39.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:39.317 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:39.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:39.317 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:39.317 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:39.318 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:39.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:39.318 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:39Z" 2023-10-18T21:20:39.318 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:39.319 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:39Z" } 2023-10-18T21:20:39.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:39.320 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:39.321 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:39.321 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:44.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:44.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:44.324 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:44.383 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:44.383 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:44.432 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:44.432 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:44.433 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:44.433 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:44.433 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:44.433 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:44.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:44.434 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:44.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:44.434 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:44.434 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:44.435 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:44.435 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:44.435 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:44.435 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:44.436 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:44.436 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:44.436 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:44.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:44.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:44.437 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:44.437 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:44.437 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:44.437 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:20:44.437 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:44.438 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:44.438 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:44.438 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:20:44.439 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:44.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:44.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:44.439 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:44.439 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:44.440 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:44.440 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:44.440 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:44.440 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:44.440 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:44.441 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:44.441 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:44.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:44.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:44.441 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:44.442 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:44.442 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:44.442 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:44.442 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:44.443 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:44.443 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:44.443 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:44.443 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:44.443 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:44.444 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:44.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:44.444 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:44.444 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:44.444 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:44.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:44.445 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:44Z" 2023-10-18T21:20:44.445 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:44.446 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:44Z" } 2023-10-18T21:20:44.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:44.447 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:44.447 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:44.448 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:49.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:49.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:49.450 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:49.511 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:49.511 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:49.559 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:49.559 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:49.560 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:49.560 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:49.560 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:49.560 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:49.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:49.561 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:49.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:49.561 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:49.562 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:49.562 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:49.562 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:49.562 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:49.562 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:49.563 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:49.563 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:49.563 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:49.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:49.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:49.564 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:49.564 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:49.564 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:49.564 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:20:49.565 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:49.565 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:49.565 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:49.565 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:20:49.565 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:49.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:49.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:49.566 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:49.566 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:49.567 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:49.567 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:49.567 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:49.567 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:49.567 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:49.568 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:49.568 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:49.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:49.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:49.569 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:49.569 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:49.569 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:49.569 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:49.569 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:49.570 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:49.570 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:49.570 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:49.570 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:49.570 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:49.571 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:49.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:49.571 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:49.571 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:49.572 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:49.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:49.572 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:49Z" 2023-10-18T21:20:49.572 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:49.573 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:49Z" } 2023-10-18T21:20:49.574 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:49.574 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:49.575 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:49.575 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:54.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:54.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:54.578 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:54.637 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:54.638 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:54.687 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:54.687 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:54.687 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:54.687 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:54.688 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:54.688 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:54.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:54.688 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:54.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:54.689 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:54.689 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:54.689 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:54.689 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:54.690 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:54.690 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:54.690 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:54.690 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:54.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:54.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:54.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:54.691 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:54.691 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:54.691 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:54.692 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:20:54.692 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:54.692 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:54.692 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:54.692 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:20:54.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:54.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:54.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:54.693 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:54.694 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:54.694 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:54.694 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:54.694 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:54.694 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:54.695 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:54.695 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:54.695 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:54.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:54.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:54.696 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:54.696 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:54.696 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:54.696 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:54.696 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:54.697 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:54.697 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:54.697 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:54.697 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:54.698 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:54.698 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:54.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:54.698 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:54.698 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:54.699 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:54.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:54.699 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:54Z" 2023-10-18T21:20:54.699 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:54.701 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:54Z" } 2023-10-18T21:20:54.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:54.702 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:54.702 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:54.702 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:20:59.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:59.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:20:59.705 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:20:59.762 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:20:59.763 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:20:59.809 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:20:59.809 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:20:59.810 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:20:59.810 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:20:59.810 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:20:59.811 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:20:59.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:59.811 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:20:59.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:59.811 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:20:59.812 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:59.812 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:59.812 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:20:59.812 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:59.813 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:59.813 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:59.813 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:20:59.813 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:59.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:59.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:59.814 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:20:59.815 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:59.815 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:59.815 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:20:59.816 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:59.816 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:20:59.816 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:59.817 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:20:59.817 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:59.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:59.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:59.817 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:20:59.818 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:59.818 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:59.818 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:20:59.818 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:59.819 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:20:59.819 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:59.819 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:20:59.820 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:59.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:59.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:20:59.820 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:20:59.821 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:20:59.821 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:20:59.821 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-10-18T21:20:59.821 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:20:59.822 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:20:59.822 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:20:59.822 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:17:26.912680Z" 2023-10-18T21:20:59.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:59.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:20:59.823 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:20:59.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:59.823 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:20:59.824 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:20:59.824 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:20:59.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:20:59.824 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:20:59Z" 2023-10-18T21:20:59.824 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:20:59.825 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:17:26.912680Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:20:59Z" } 2023-10-18T21:20:59.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:17:26.912680Z'}}] 2023-10-18T21:20:59.826 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:20:59.827 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051673'} 2023-10-18T21:20:59.827 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000051813 target=00000000000000000000:00000000000000051673 2023-10-18T21:21:04.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:04.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:04.829 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:04.888 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:04.888 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:04.937 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:04.937 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:04.938 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:04.938 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:04.938 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:04.939 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:21:04.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:04.939 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:04.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:04.940 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:04.940 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:04.940 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:04.941 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044326", 2023-10-18T21:21:04.941 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:04.941 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:04.941 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:04.942 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:12:51.216994Z" 2023-10-18T21:21:04.942 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:04.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:04.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:04.943 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:04.943 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:04.944 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:04.944 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:21:04.945 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:04.945 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:04.945 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:04.946 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:21:04.946 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:04.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:04.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:04.947 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:04.948 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:04.948 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:04.949 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032342", 2023-10-18T21:21:04.949 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:04.949 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:21:04.950 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:04.950 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:04:37.458485Z" 2023-10-18T21:21:04.950 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:04.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:04.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:04.952 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:04.952 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:04.952 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:04.953 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:21:04.953 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:04.953 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:21:04.954 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:04.954 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:21:04.955 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:04.955 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:04.956 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:04.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:04.956 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:04.957 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:21:04.957 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:21:04.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:04.958 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:04Z" 2023-10-18T21:21:04.958 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:04.960 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044326", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:12:51.216994Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032342", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:04:37.458485Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:21:04Z" } 2023-10-18T21:21:04.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044326', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:12:51.216994Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032342', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:04:37.458485Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:21:04.961 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:21:04.962 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044326', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032342', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:21:04.962 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:21:04.963 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:21:04.963 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:21:04.963 DEBUG:boto:Method: HEAD 2023-10-18T21:21:04.964 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:21:04.964 DEBUG:boto:Data: 2023-10-18T21:21:04.965 DEBUG:boto:Headers: {} 2023-10-18T21:21:04.965 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:04.966 DEBUG:boto:Port: 8001 2023-10-18T21:21:04.966 DEBUG:boto:Params: {} 2023-10-18T21:21:04.967 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:21:04.967 DEBUG:boto:Token: None 2023-10-18T21:21:04.968 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:04 GMT /bofrnf-70/ 2023-10-18T21:21:04.969 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6yi6rTl0fCFqImDQjfEXwFB6gzA= 2023-10-18T21:21:04.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6yi6rTl0fCFqImDQjfEXwFB6gzA=', 'Content-Length': '0'} 2023-10-18T21:21:04.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e21901a2271fa30-0065304c40-270b-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:04.976 DEBUG:boto:path=/bofrnf-70/obj5 2023-10-18T21:21:04.976 DEBUG:boto:auth_path=/bofrnf-70/obj5 2023-10-18T21:21:04.977 DEBUG:boto:Method: HEAD 2023-10-18T21:21:04.977 DEBUG:boto:Path: /bofrnf-70/obj5 2023-10-18T21:21:04.977 DEBUG:boto:Data: 2023-10-18T21:21:04.978 DEBUG:boto:Headers: {} 2023-10-18T21:21:04.978 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:04.979 DEBUG:boto:Port: 8001 2023-10-18T21:21:04.979 DEBUG:boto:Params: {} 2023-10-18T21:21:04.979 DEBUG:boto:Token: None 2023-10-18T21:21:04.980 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:04 GMT /bofrnf-70/obj5 2023-10-18T21:21:04.980 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:qya+coueUHZ+bIQwET6ZDY0QK9k= 2023-10-18T21:21:04.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qya+coueUHZ+bIQwET6ZDY0QK9k=', 'Content-Length': '0'} 2023-10-18T21:21:04.983 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e43b3c5e49074969-0065304c40-270b-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:04.983 DEBUG:boto:path=/bofrnf-70/obj6 2023-10-18T21:21:04.984 DEBUG:boto:auth_path=/bofrnf-70/obj6 2023-10-18T21:21:04.984 DEBUG:boto:Method: HEAD 2023-10-18T21:21:04.985 DEBUG:boto:Path: /bofrnf-70/obj6 2023-10-18T21:21:04.985 DEBUG:boto:Data: 2023-10-18T21:21:04.985 DEBUG:boto:Headers: {} 2023-10-18T21:21:04.986 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:04.986 DEBUG:boto:Port: 8001 2023-10-18T21:21:04.987 DEBUG:boto:Params: {} 2023-10-18T21:21:04.987 DEBUG:boto:Token: None 2023-10-18T21:21:04.988 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:04 GMT /bofrnf-70/obj6 2023-10-18T21:21:04.988 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Gn/mbP01zZuwnr7tKlIRHSeAu3o= 2023-10-18T21:21:04.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gn/mbP01zZuwnr7tKlIRHSeAu3o=', 'Content-Length': '0'} 2023-10-18T21:21:04.990 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f7821cf8a0735958-0065304c40-270b-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:04.991 DEBUG:boto:path=/bofrnf-71/ 2023-10-18T21:21:04.991 DEBUG:boto:auth_path=/bofrnf-71/ 2023-10-18T21:21:04.992 DEBUG:boto:Method: HEAD 2023-10-18T21:21:04.992 DEBUG:boto:Path: /bofrnf-71/ 2023-10-18T21:21:04.993 DEBUG:boto:Data: 2023-10-18T21:21:04.994 DEBUG:boto:Headers: {} 2023-10-18T21:21:04.994 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:04.994 DEBUG:boto:Port: 8001 2023-10-18T21:21:04.995 DEBUG:boto:Params: {} 2023-10-18T21:21:04.995 DEBUG:boto:Token: None 2023-10-18T21:21:04.996 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:04 GMT /bofrnf-71/ 2023-10-18T21:21:04.996 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4POBwugCmrgGD1f1iDIUIY3cR4U= 2023-10-18T21:21:04.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4POBwugCmrgGD1f1iDIUIY3cR4U=', 'Content-Length': '0'} 2023-10-18T21:21:04.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2ba26ec4bbdbeea-0065304c40-270b-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:05.000 DEBUG:boto:path=/bofrnf-71/obj5 2023-10-18T21:21:05.000 DEBUG:boto:auth_path=/bofrnf-71/obj5 2023-10-18T21:21:05.001 DEBUG:boto:Method: HEAD 2023-10-18T21:21:05.001 DEBUG:boto:Path: /bofrnf-71/obj5 2023-10-18T21:21:05.001 DEBUG:boto:Data: 2023-10-18T21:21:05.002 DEBUG:boto:Headers: {} 2023-10-18T21:21:05.002 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:05.003 DEBUG:boto:Port: 8001 2023-10-18T21:21:05.003 DEBUG:boto:Params: {} 2023-10-18T21:21:05.003 DEBUG:boto:Token: None 2023-10-18T21:21:05.004 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:05 GMT /bofrnf-71/obj5 2023-10-18T21:21:05.005 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:F4UX2Gy/UM0ZAU69zWGyUNm36bU= 2023-10-18T21:21:05.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:F4UX2Gy/UM0ZAU69zWGyUNm36bU=', 'Content-Length': '0'} 2023-10-18T21:21:05.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:19:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-70:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0a0d3e21aed4bd0-0065304c41-270b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:05.008 DEBUG:boto:path=/bofrnf-71/obj5 2023-10-18T21:21:05.008 DEBUG:boto:auth_path=/bofrnf-71/obj5 2023-10-18T21:21:05.009 DEBUG:boto:Method: GET 2023-10-18T21:21:05.009 DEBUG:boto:Path: /bofrnf-71/obj5 2023-10-18T21:21:05.010 DEBUG:boto:Data: 2023-10-18T21:21:05.010 DEBUG:boto:Headers: {} 2023-10-18T21:21:05.011 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:05.011 DEBUG:boto:Port: 8001 2023-10-18T21:21:05.011 DEBUG:boto:Params: {} 2023-10-18T21:21:05.012 DEBUG:boto:Token: None 2023-10-18T21:21:05.012 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:21:05 GMT /bofrnf-71/obj5 2023-10-18T21:21:05.013 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PdG6U4kqpwqMs2hxIbulNyqo678= 2023-10-18T21:21:05.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PdG6U4kqpwqMs2hxIbulNyqo678=', 'Content-Length': '0'} 2023-10-18T21:21:05.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:19:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-70:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039e2e8c458b494f9-0065304c41-270b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:05.058 DEBUG:boto:path=/bofrnf-71/obj6 2023-10-18T21:21:05.059 DEBUG:boto:auth_path=/bofrnf-71/obj6 2023-10-18T21:21:05.059 DEBUG:boto:Method: HEAD 2023-10-18T21:21:05.060 DEBUG:boto:Path: /bofrnf-71/obj6 2023-10-18T21:21:05.060 DEBUG:boto:Data: 2023-10-18T21:21:05.060 DEBUG:boto:Headers: {} 2023-10-18T21:21:05.061 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:05.061 DEBUG:boto:Port: 8001 2023-10-18T21:21:05.062 DEBUG:boto:Params: {} 2023-10-18T21:21:05.062 DEBUG:boto:Token: None 2023-10-18T21:21:05.063 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:05 GMT /bofrnf-71/obj6 2023-10-18T21:21:05.063 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:AghVvqLENrkCgsaUpsA/ABOO+/s= 2023-10-18T21:21:05.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AghVvqLENrkCgsaUpsA/ABOO+/s=', 'Content-Length': '0'} 2023-10-18T21:21:05.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:19:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-70:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a3fec7a8c7ed08c-0065304c41-270b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:05.066 DEBUG:boto:path=/bofrnf-71/obj6 2023-10-18T21:21:05.067 DEBUG:boto:auth_path=/bofrnf-71/obj6 2023-10-18T21:21:05.067 DEBUG:boto:Method: GET 2023-10-18T21:21:05.067 DEBUG:boto:Path: /bofrnf-71/obj6 2023-10-18T21:21:05.068 DEBUG:boto:Data: 2023-10-18T21:21:05.068 DEBUG:boto:Headers: {} 2023-10-18T21:21:05.069 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:05.069 DEBUG:boto:Port: 8001 2023-10-18T21:21:05.069 DEBUG:boto:Params: {} 2023-10-18T21:21:05.070 DEBUG:boto:Token: None 2023-10-18T21:21:05.070 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:21:05 GMT /bofrnf-71/obj6 2023-10-18T21:21:05.071 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IDNjFPsy/8O7OiAzgRThqfmSexY= 2023-10-18T21:21:05.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:05 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IDNjFPsy/8O7OiAzgRThqfmSexY=', 'Content-Length': '0'} 2023-10-18T21:21:05.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:19:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-70:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085b2564467b41055-0065304c41-270b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:05.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:05.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:05.075 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:05.126 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:05.126 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:05.182 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:21:05.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:05.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:05.184 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:05.232 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:05.233 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:05.262 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:21:05.275 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:21:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664065.161640_1985.1", 2023-10-18T21:21:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:05.161640Z" 2023-10-18T21:21:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:21:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:21:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:05.279 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:05.279 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:21:05.279 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:05.279 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:21:05.280 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664065.161640_1985.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:05.281 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:21:05.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:05.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:05.282 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:05.332 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:05.332 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:05.361 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:05.360+0000 7f3d5d687780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:05.362 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:05.361+0000 7f3d5d687780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:05.375 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663941.096796_1971.1", 2023-10-18T21:21:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:01.096796Z", 2023-10-18T21:21:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:05.388 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:05.389 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:05.389 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:05.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:05.389 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:05Z" 2023-10-18T21:21:05.389 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:05.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663941.096796_1971.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:19:01.096796Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:05Z" } 2023-10-18T21:21:05.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663941.096796_1971.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:19:01.096796Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:05.391 DEBUG:rgw_multi.tests:log_status={0: '1_1697664065.161640_1985.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:05.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:05.392 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664065.161640_1985.1 target=1_1697663941.096796_1971.1 2023-10-18T21:21:10.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:10.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:10.394 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:10.451 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:10.451 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:10.483 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:10.482+0000 7fd48a01e780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:10.484 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:10.483+0000 7fd48a01e780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:10.498 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697663941.096796_1971.1", 2023-10-18T21:21:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:01.096796Z", 2023-10-18T21:21:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:10Z" 2023-10-18T21:21:10.512 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:10.513 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697663941.096796_1971.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:19:01.096796Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:10Z" } 2023-10-18T21:21:10.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663941.096796_1971.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:19:01.096796Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:10.514 DEBUG:rgw_multi.tests:log_status={0: '1_1697664065.161640_1985.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:10.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1697663941.096796_1971.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:10.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664065.161640_1985.1 target=1_1697663941.096796_1971.1 2023-10-18T21:21:15.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:15.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:15.517 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:15.575 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:15.576 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:15.609 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:15.608+0000 7f52c92af780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:15.610 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:15.609+0000 7f52c92af780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:15.623 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:15.624 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:15.624 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:15.624 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:15.624 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:15.625 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:15.625 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:15.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:15.625 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:15.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:15.626 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:15.626 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:15.626 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:15.626 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664065.161640_1985.1", 2023-10-18T21:21:15.627 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:15.627 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:15.627 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:15.627 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:05.161640Z", 2023-10-18T21:21:15.628 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:15.628 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:15.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:15.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:15.628 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:15.629 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:15.629 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:15.629 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:15.629 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:15.629 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:15.630 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:15.630 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:15.630 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:15.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:15.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:15.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:15.631 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:15.631 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:15.631 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:15.632 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:15.632 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:15.632 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:15.632 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:15.632 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:15.633 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:15.633 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:15.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:15.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:15.634 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:15.634 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:15.634 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:15.634 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:15.634 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:15.635 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:15.635 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:15.635 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:15.635 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:15.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:15.636 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:15.636 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:15.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:15.636 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:15.637 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:15.637 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:15.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:15.637 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:15Z" 2023-10-18T21:21:15.637 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:15.638 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664065.161640_1985.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:05.161640Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:15Z" } 2023-10-18T21:21:15.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664065.161640_1985.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:05.161640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:15.639 DEBUG:rgw_multi.tests:log_status={0: '1_1697664065.161640_1985.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:15.640 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664065.161640_1985.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:15.640 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:21:15.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:15.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:15.642 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:15.690 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:15.690 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:15.736 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[]} 2023-10-18T21:21:15.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:15.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:15.737 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:15.788 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:15.789 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:15.840 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:21:15.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:15.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:15.841 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:15.897 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:15.897 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:15.946 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-18T21:21:15.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=bofrnf-70', '--source-bucket=bofrnf-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:15.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=bofrnf-70', '--source-bucket=bofrnf-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:15.948 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=bofrnf-70 --source-bucket=bofrnf-71 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:16.003 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:16.003 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:16.055 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"bofrnf-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-18T21:21:16.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:16.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:16.056 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:16.109 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:16.110 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:16.162 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"bofrnf-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-18T21:21:16.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:16.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:16.164 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:16.211 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:16.212 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:16.257 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"bofrnf-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-18T21:21:16.258 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:21:16.258 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:21:16.259 DEBUG:boto:Method: HEAD 2023-10-18T21:21:16.259 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:21:16.260 DEBUG:boto:Data: 2023-10-18T21:21:16.260 DEBUG:boto:Headers: {} 2023-10-18T21:21:16.260 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:16.261 DEBUG:boto:Port: 8001 2023-10-18T21:21:16.261 DEBUG:boto:Params: {} 2023-10-18T21:21:16.262 DEBUG:boto:Token: None 2023-10-18T21:21:16.262 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:16 GMT /bofrnf-70/ 2023-10-18T21:21:16.263 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8LlrbtqyU5Ah5BmozfB0xp3ASsY= 2023-10-18T21:21:16.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8LlrbtqyU5Ah5BmozfB0xp3ASsY=', 'Content-Length': '0'} 2023-10-18T21:21:16.266 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c29f3dc0bea4b343-0065304c4c-270b-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:16.266 DEBUG:boto:path=/bofrnf-70/obj7 2023-10-18T21:21:16.267 DEBUG:boto:auth_path=/bofrnf-70/obj7 2023-10-18T21:21:16.267 DEBUG:boto:Method: PUT 2023-10-18T21:21:16.268 DEBUG:boto:Path: /bofrnf-70/obj7 2023-10-18T21:21:16.268 DEBUG:boto:Data: 2023-10-18T21:21:16.268 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:21:16.269 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:16.269 DEBUG:boto:Port: 8001 2023-10-18T21:21:16.270 DEBUG:boto:Params: {} 2023-10-18T21:21:16.270 DEBUG:boto:Token: None 2023-10-18T21:21:16.271 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:21:16 GMT /bofrnf-70/obj7 2023-10-18T21:21:16.271 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o37Lkp4DdeZVfFIFpluM8CfEKhQ= 2023-10-18T21:21:16.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o37Lkp4DdeZVfFIFpluM8CfEKhQ='} 2023-10-18T21:21:16.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000986f42cbff5a1ade-0065304c4c-270b-a2'), ('Date', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:16.278 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:21:16.279 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:21:16.279 DEBUG:boto:Method: HEAD 2023-10-18T21:21:16.280 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:21:16.280 DEBUG:boto:Data: 2023-10-18T21:21:16.280 DEBUG:boto:Headers: {} 2023-10-18T21:21:16.281 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:16.281 DEBUG:boto:Port: 8001 2023-10-18T21:21:16.282 DEBUG:boto:Params: {} 2023-10-18T21:21:16.282 DEBUG:boto:Token: None 2023-10-18T21:21:16.283 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:16 GMT /bofrnf-70/ 2023-10-18T21:21:16.283 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:8LlrbtqyU5Ah5BmozfB0xp3ASsY= 2023-10-18T21:21:16.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8LlrbtqyU5Ah5BmozfB0xp3ASsY=', 'Content-Length': '0'} 2023-10-18T21:21:16.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000245227cf0a1f1763-0065304c4c-270b-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:16.286 DEBUG:boto:path=/bofrnf-70/obj8 2023-10-18T21:21:16.287 DEBUG:boto:auth_path=/bofrnf-70/obj8 2023-10-18T21:21:16.287 DEBUG:boto:Method: PUT 2023-10-18T21:21:16.288 DEBUG:boto:Path: /bofrnf-70/obj8 2023-10-18T21:21:16.288 DEBUG:boto:Data: 2023-10-18T21:21:16.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:21:16.289 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:16.289 DEBUG:boto:Port: 8001 2023-10-18T21:21:16.290 DEBUG:boto:Params: {} 2023-10-18T21:21:16.290 DEBUG:boto:Token: None 2023-10-18T21:21:16.291 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:21:16 GMT /bofrnf-70/obj8 2023-10-18T21:21:16.291 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:q8y7hJ47LD+eghH6ZekQ9wO9l0g= 2023-10-18T21:21:16.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:q8y7hJ47LD+eghH6ZekQ9wO9l0g='} 2023-10-18T21:21:16.297 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a46f602d866bcb82-0065304c4c-270b-a2'), ('Date', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:16.297 DEBUG:boto:path=/bofrnf-71/ 2023-10-18T21:21:16.298 DEBUG:boto:auth_path=/bofrnf-71/ 2023-10-18T21:21:16.298 DEBUG:boto:Method: HEAD 2023-10-18T21:21:16.298 DEBUG:boto:Path: /bofrnf-71/ 2023-10-18T21:21:16.299 DEBUG:boto:Data: 2023-10-18T21:21:16.299 DEBUG:boto:Headers: {} 2023-10-18T21:21:16.300 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:16.300 DEBUG:boto:Port: 8001 2023-10-18T21:21:16.300 DEBUG:boto:Params: {} 2023-10-18T21:21:16.301 DEBUG:boto:Token: None 2023-10-18T21:21:16.301 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:16 GMT /bofrnf-71/ 2023-10-18T21:21:16.302 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MV2/YfBkKh5q2Bcjny37oK1zOb8= 2023-10-18T21:21:16.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MV2/YfBkKh5q2Bcjny37oK1zOb8=', 'Content-Length': '0'} 2023-10-18T21:21:16.304 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000429d4f74c122e019-0065304c4c-270b-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:16.305 DEBUG:boto:path=/bofrnf-71/obj9 2023-10-18T21:21:16.305 DEBUG:boto:auth_path=/bofrnf-71/obj9 2023-10-18T21:21:16.306 DEBUG:boto:Method: PUT 2023-10-18T21:21:16.306 DEBUG:boto:Path: /bofrnf-71/obj9 2023-10-18T21:21:16.307 DEBUG:boto:Data: 2023-10-18T21:21:16.307 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:21:16.307 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:16.308 DEBUG:boto:Port: 8001 2023-10-18T21:21:16.308 DEBUG:boto:Params: {} 2023-10-18T21:21:16.309 DEBUG:boto:Token: None 2023-10-18T21:21:16.309 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:21:16 GMT /bofrnf-71/obj9 2023-10-18T21:21:16.310 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Mj9DYjVlCM00Qp9xdt41oPpIJm0= 2023-10-18T21:21:16.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mj9DYjVlCM00Qp9xdt41oPpIJm0='} 2023-10-18T21:21:16.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff1fec4d952a1060-0065304c4c-270b-a2'), ('Date', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:16.356 DEBUG:boto:path=/bofrnf-71/ 2023-10-18T21:21:16.356 DEBUG:boto:auth_path=/bofrnf-71/ 2023-10-18T21:21:16.357 DEBUG:boto:Method: HEAD 2023-10-18T21:21:16.357 DEBUG:boto:Path: /bofrnf-71/ 2023-10-18T21:21:16.357 DEBUG:boto:Data: 2023-10-18T21:21:16.358 DEBUG:boto:Headers: {} 2023-10-18T21:21:16.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:16.359 DEBUG:boto:Port: 8001 2023-10-18T21:21:16.359 DEBUG:boto:Params: {} 2023-10-18T21:21:16.359 DEBUG:boto:Token: None 2023-10-18T21:21:16.360 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:16 GMT /bofrnf-71/ 2023-10-18T21:21:16.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MV2/YfBkKh5q2Bcjny37oK1zOb8= 2023-10-18T21:21:16.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MV2/YfBkKh5q2Bcjny37oK1zOb8=', 'Content-Length': '0'} 2023-10-18T21:21:16.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004aecbfaace84fa3b-0065304c4c-270b-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:16.364 DEBUG:boto:path=/bofrnf-71/obj10 2023-10-18T21:21:16.364 DEBUG:boto:auth_path=/bofrnf-71/obj10 2023-10-18T21:21:16.364 DEBUG:boto:Method: PUT 2023-10-18T21:21:16.365 DEBUG:boto:Path: /bofrnf-71/obj10 2023-10-18T21:21:16.365 DEBUG:boto:Data: 2023-10-18T21:21:16.366 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:21:16.366 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:21:16.367 DEBUG:boto:Port: 8001 2023-10-18T21:21:16.367 DEBUG:boto:Params: {} 2023-10-18T21:21:16.367 DEBUG:boto:Token: None 2023-10-18T21:21:16.368 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:21:16 GMT /bofrnf-71/obj10 2023-10-18T21:21:16.368 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ASqM59LD1XBdlz4KW3JkWeuqoRY= 2023-10-18T21:21:16.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ASqM59LD1XBdlz4KW3JkWeuqoRY='} 2023-10-18T21:21:16.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3c4030a443b0cb6-0065304c4c-270b-a2'), ('Date', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:16.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:16.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:16.376 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:16.424 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:16.424 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:16.454 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:21:16.467 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:21:16.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:16.467 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:21:16.468 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:16.148026Z" 2023-10-18T21:21:16.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:16.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:16.468 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:16.468 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:21:16.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:16.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:16.469 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:16.469 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:21:16.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:16.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:16.470 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:16.470 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:21:16.470 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:16.471 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:21:16.472 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:16.472 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:21:16.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:16.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:16.473 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:16.522 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:16.523 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:16.551 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:16.550+0000 7f31a987d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:16.552 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:16.551+0000 7f31a987d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:16.566 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:16.566 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:16.566 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:16.566 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:16.567 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:16.567 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:16.567 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:16.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:16.567 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:16.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:16.568 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:16.568 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:16.568 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:16.569 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664065.161640_1985.1", 2023-10-18T21:21:16.569 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:16.569 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:16.569 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:16.569 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:05.161640Z", 2023-10-18T21:21:16.570 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:16.570 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:16.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:16.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:16.570 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:16.571 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:16.571 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:16.571 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:16.571 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:16.571 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:16.572 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:16.572 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:16.572 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:16.572 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:16.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:16.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:16.573 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:16.573 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:16.573 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:16.574 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:16.574 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:16.574 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:16.574 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:16.575 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:16.575 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:16.575 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:16.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:16.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:16.576 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:16.576 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:16.576 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:16.576 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:16.576 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:16.577 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:16.577 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:16.577 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:16.577 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:16.577 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:16.578 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:16.578 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:16.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:16.578 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:16.579 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:16.579 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:16.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:16.579 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:16Z" 2023-10-18T21:21:16.579 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:16.580 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664065.161640_1985.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:05.161640Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:16Z" } 2023-10-18T21:21:16.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664065.161640_1985.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:05.161640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:16.581 DEBUG:rgw_multi.tests:log_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:16.582 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664065.161640_1985.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:16.582 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664076.148026_2005.1 target=1_1697664065.161640_1985.1 2023-10-18T21:21:21.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:21.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:21.585 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:21.642 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:21.643 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:21.676 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:21.675+0000 7fae50ded780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:21.677 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:21.676+0000 7fae50ded780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:21.691 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:21.691 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:21.692 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:21.692 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:21.692 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:21.692 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:21.693 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:21.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:21.693 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:21.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:21.694 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:21.694 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:21.694 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:21.695 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664065.161640_1985.1", 2023-10-18T21:21:21.695 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:21.695 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:21.696 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:21.696 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:05.161640Z", 2023-10-18T21:21:21.697 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:21.697 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:21.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:21.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:21.698 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:21.699 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:21.699 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:21.699 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:21.700 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:21.700 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:21.701 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:21.701 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:21.702 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:21.702 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:21.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:21.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:21.703 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:21.703 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:21.704 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:21.704 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:21.705 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:21.705 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:21.705 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:21.706 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:21.706 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:21.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:21.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:21.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:21.708 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:21.708 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:21.709 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:21.709 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:21.709 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:21.710 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:21.710 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:21.711 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:21.711 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:21.712 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:21.712 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:21.712 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:21.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:21.713 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:21.714 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:21.714 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:21.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:21.715 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:21Z" 2023-10-18T21:21:21.715 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:21.717 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664065.161640_1985.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:05.161640Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:21Z" } 2023-10-18T21:21:21.717 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664065.161640_1985.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:05.161640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:21.718 DEBUG:rgw_multi.tests:log_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:21.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664065.161640_1985.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:21.719 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664076.148026_2005.1 target=1_1697664065.161640_1985.1 2023-10-18T21:21:26.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:26.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:26.722 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:26.780 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:26.780 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:26.813 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:26.811+0000 7f5ac3420780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:26.814 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:26.812+0000 7f5ac3420780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:26.828 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:26.828 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:26.828 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:26.829 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:26.829 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:26.829 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:26.829 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:26.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:26.830 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:26.830 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:26.830 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:26.830 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:26.831 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:26.831 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664065.161640_1985.1", 2023-10-18T21:21:26.831 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:26.831 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:26.831 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:26.832 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:05.161640Z", 2023-10-18T21:21:26.832 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:26.832 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:26.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:26.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:26.833 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:26.833 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:26.833 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:26.833 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:26.834 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:26.834 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:26.834 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:26.834 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:26.834 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:26.835 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:26.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:26.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:26.835 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:26.836 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:26.836 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:26.836 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:26.836 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:26.836 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:26.837 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:26.837 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:26.837 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:26.837 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:26.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:26.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:26.838 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:26.838 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:26.838 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:26.839 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:26.839 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:26.839 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:26.839 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:26.839 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:26.840 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:26.840 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:26.840 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:26.840 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:26.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:26.841 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:26.841 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:26.841 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:26.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:26.841 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:26Z" 2023-10-18T21:21:26.842 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:26.843 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664065.161640_1985.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:05.161640Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:26Z" } 2023-10-18T21:21:26.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664065.161640_1985.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:05.161640Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:26.844 DEBUG:rgw_multi.tests:log_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:26.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664065.161640_1985.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:26.845 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664076.148026_2005.1 target=1_1697664065.161640_1985.1 2023-10-18T21:21:31.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:31.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:31.847 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:31.902 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:31.902 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:31.934 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:31.932+0000 7f1cc7ea3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:31.934 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:31.933+0000 7f1cc7ea3780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:31.948 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:31.948 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:31.948 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:31.949 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:31.949 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:31.949 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:31.949 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:31.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:31.950 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:31.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:31.950 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:31.950 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:31.951 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:31.951 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664075.821463_1986.1", 2023-10-18T21:21:31.951 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:31.951 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:31.951 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:31.952 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:15.821463Z", 2023-10-18T21:21:31.952 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:31.952 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:31.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:31.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:31.953 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:31.953 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:31.953 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:31.953 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:31.953 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:31.954 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:31.954 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:31.954 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:31.954 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:31.955 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:31.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:31.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:31.955 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:31.955 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:31.956 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:31.956 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:31.956 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:31.956 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:31.956 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:31.957 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:31.957 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:31.957 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:31.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:31.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:31.958 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:31.958 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:31.958 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:31.958 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:31.959 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:31.959 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:31.959 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:31.959 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:31.959 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:31.960 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:31.960 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:31.960 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:31.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:31.960 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:31.961 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:31.961 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:31.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:31.961 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:31Z" 2023-10-18T21:21:31.962 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:31.963 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664075.821463_1986.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:15.821463Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:31Z" } 2023-10-18T21:21:31.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664075.821463_1986.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:15.821463Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:31.964 DEBUG:rgw_multi.tests:log_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:31.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664075.821463_1986.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:31.964 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664076.148026_2005.1 target=1_1697664075.821463_1986.1 2023-10-18T21:21:36.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:36.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:36.967 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:37.025 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:37.025 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:37.058 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:37.056+0000 7f421080b780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:37.058 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:37.057+0000 7f421080b780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:37.072 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:37.072 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:37.073 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:37.073 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:37.073 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:37.073 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:37.074 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.074 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:37.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.075 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:37.075 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.075 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:37.075 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:21:37.075 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.076 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:37.076 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.076 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:21:37.076 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.076 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.077 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:37.077 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.077 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:37.078 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:37.078 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.078 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.078 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.079 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:37.079 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.079 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.080 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:37.080 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.080 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:37.080 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:37.080 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.081 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.081 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.081 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:37.081 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.081 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.082 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:37.082 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.082 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:37.083 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:37.083 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.083 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.083 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.083 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:37.083 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.084 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:37.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.084 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:37.085 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:37.085 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:37.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.085 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:37Z" 2023-10-18T21:21:37.086 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:37.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664076.148026_2005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:16.148026Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:37Z" } 2023-10-18T21:21:37.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664076.148026_2005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.148026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:37.088 DEBUG:rgw_multi.tests:log_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:37.088 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:37.088 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:21:37.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:37.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:37.090 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:37.139 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:37.139 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:37.190 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:21:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000016162", 2023-10-18T21:21:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:16.314833Z" 2023-10-18T21:21:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.192 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000013376", 2023-10-18T21:21:37.192 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:20:31.755928Z" 2023-10-18T21:21:37.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.193 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000006570", 2023-10-18T21:21:37.193 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:16.373441Z" 2023-10-18T21:21:37.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.193 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000017277", 2023-10-18T21:21:37.194 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:16.277146Z" 2023-10-18T21:21:37.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.194 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:21:37.195 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000016162', 1: '00000000000000000000:00000000000000013376', 2: '00000000000000000000:00000000000000006570', 3: '00000000000000000000:00000000000000017277'} 2023-10-18T21:21:37.195 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T21:21:37.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:37.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:37.197 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:37.247 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:37.247 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:37.291 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 17193016251575518912 2023-10-18T21:21:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000016162", 2023-10-18T21:21:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:16.314833Z" 2023-10-18T21:21:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000013376", 2023-10-18T21:21:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:20:31.755928Z" 2023-10-18T21:21:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000006570", 2023-10-18T21:21:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:16.373441Z" 2023-10-18T21:21:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000017277", 2023-10-18T21:21:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:16.277146Z" 2023-10-18T21:21:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 0, 2023-10-18T21:21:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 0 2023-10-18T21:21:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:37Z" 2023-10-18T21:21:37.304 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:37.305 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17193016251575518912 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016162", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:16.314833Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:20:31.755928Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006570", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:16.373441Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017277", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:16.277146Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-18T21:21:37Z" } 2023-10-18T21:21:37.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016162', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.314833Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:20:31.755928Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006570', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.373441Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017277', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.277146Z'}}] 2023-10-18T21:21:37.306 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000016162', 1: '00000000000000000000:00000000000000013376', 2: '00000000000000000000:00000000000000006570', 3: '00000000000000000000:00000000000000017277'} 2023-10-18T21:21:37.307 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000016162', 1: '00000000000000000000:00000000000000013376', 2: '00000000000000000000:00000000000000006570', 3: '00000000000000000000:00000000000000017277'} 2023-10-18T21:21:37.307 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-18T21:21:37.308 DEBUG:boto:path=/bofrnf-70/ 2023-10-18T21:21:37.308 DEBUG:boto:auth_path=/bofrnf-70/ 2023-10-18T21:21:37.308 DEBUG:boto:Method: HEAD 2023-10-18T21:21:37.309 DEBUG:boto:Path: /bofrnf-70/ 2023-10-18T21:21:37.309 DEBUG:boto:Data: 2023-10-18T21:21:37.310 DEBUG:boto:Headers: {} 2023-10-18T21:21:37.310 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:37.310 DEBUG:boto:Port: 8000 2023-10-18T21:21:37.311 DEBUG:boto:Params: {} 2023-10-18T21:21:37.311 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:21:37.312 DEBUG:boto:Token: None 2023-10-18T21:21:37.312 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:37 GMT /bofrnf-70/ 2023-10-18T21:21:37.313 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:bE+fPsScOTKO4lN5GmiVcFTGwxw= 2023-10-18T21:21:37.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bE+fPsScOTKO4lN5GmiVcFTGwxw=', 'Content-Length': '0'} 2023-10-18T21:21:37.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcf1e520e62eed89-0065304c61-270f-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:37.318 DEBUG:boto:path=/bofrnf-70/obj7 2023-10-18T21:21:37.319 DEBUG:boto:auth_path=/bofrnf-70/obj7 2023-10-18T21:21:37.319 DEBUG:boto:Method: HEAD 2023-10-18T21:21:37.320 DEBUG:boto:Path: /bofrnf-70/obj7 2023-10-18T21:21:37.320 DEBUG:boto:Data: 2023-10-18T21:21:37.320 DEBUG:boto:Headers: {} 2023-10-18T21:21:37.321 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:37.321 DEBUG:boto:Port: 8000 2023-10-18T21:21:37.322 DEBUG:boto:Params: {} 2023-10-18T21:21:37.322 DEBUG:boto:Token: None 2023-10-18T21:21:37.322 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:37 GMT /bofrnf-70/obj7 2023-10-18T21:21:37.323 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jX+/RMKkmGTtWEnt5+eyf0S1b/A= 2023-10-18T21:21:37.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jX+/RMKkmGTtWEnt5+eyf0S1b/A=', 'Content-Length': '0'} 2023-10-18T21:21:37.325 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d066e561652298f0-0065304c61-270f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:21:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:37.325 DEBUG:boto:path=/bofrnf-70/obj8 2023-10-18T21:21:37.326 DEBUG:boto:auth_path=/bofrnf-70/obj8 2023-10-18T21:21:37.326 DEBUG:boto:Method: HEAD 2023-10-18T21:21:37.327 DEBUG:boto:Path: /bofrnf-70/obj8 2023-10-18T21:21:37.327 DEBUG:boto:Data: 2023-10-18T21:21:37.327 DEBUG:boto:Headers: {} 2023-10-18T21:21:37.328 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:37.328 DEBUG:boto:Port: 8000 2023-10-18T21:21:37.329 DEBUG:boto:Params: {} 2023-10-18T21:21:37.329 DEBUG:boto:Token: None 2023-10-18T21:21:37.329 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:37 GMT /bofrnf-70/obj8 2023-10-18T21:21:37.330 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:uA30A7YN2f0PJCx1dkjMoULJyhw= 2023-10-18T21:21:37.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uA30A7YN2f0PJCx1dkjMoULJyhw=', 'Content-Length': '0'} 2023-10-18T21:21:37.332 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000af08215ec16fc18a-0065304c61-270f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:21:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:37.333 DEBUG:boto:path=/bofrnf-70/obj9 2023-10-18T21:21:37.333 DEBUG:boto:auth_path=/bofrnf-70/obj9 2023-10-18T21:21:37.333 DEBUG:boto:Method: HEAD 2023-10-18T21:21:37.334 DEBUG:boto:Path: /bofrnf-70/obj9 2023-10-18T21:21:37.334 DEBUG:boto:Data: 2023-10-18T21:21:37.335 DEBUG:boto:Headers: {} 2023-10-18T21:21:37.335 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:37.335 DEBUG:boto:Port: 8000 2023-10-18T21:21:37.336 DEBUG:boto:Params: {} 2023-10-18T21:21:37.336 DEBUG:boto:Token: None 2023-10-18T21:21:37.337 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:37 GMT /bofrnf-70/obj9 2023-10-18T21:21:37.337 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:5yLM7avBcEoH6bn2kkSjFyFJ2A8= 2023-10-18T21:21:37.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5yLM7avBcEoH6bn2kkSjFyFJ2A8=', 'Content-Length': '0'} 2023-10-18T21:21:37.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-71:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9faea568875c2a4-0065304c61-270f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:21:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:37.340 DEBUG:boto:path=/bofrnf-70/obj9 2023-10-18T21:21:37.341 DEBUG:boto:auth_path=/bofrnf-70/obj9 2023-10-18T21:21:37.341 DEBUG:boto:Method: GET 2023-10-18T21:21:37.342 DEBUG:boto:Path: /bofrnf-70/obj9 2023-10-18T21:21:37.342 DEBUG:boto:Data: 2023-10-18T21:21:37.342 DEBUG:boto:Headers: {} 2023-10-18T21:21:37.343 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:37.343 DEBUG:boto:Port: 8000 2023-10-18T21:21:37.344 DEBUG:boto:Params: {} 2023-10-18T21:21:37.344 DEBUG:boto:Token: None 2023-10-18T21:21:37.345 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:21:37 GMT /bofrnf-70/obj9 2023-10-18T21:21:37.345 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:BdCkiUUCktnd5ZwQ8x2a8V4X4L0= 2023-10-18T21:21:37.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BdCkiUUCktnd5ZwQ8x2a8V4X4L0=', 'Content-Length': '0'} 2023-10-18T21:21:37.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-71:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017c50ad1b37ef0bc-0065304c61-270f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:21:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:37.348 DEBUG:boto:path=/bofrnf-70/obj10 2023-10-18T21:21:37.348 DEBUG:boto:auth_path=/bofrnf-70/obj10 2023-10-18T21:21:37.349 DEBUG:boto:Method: HEAD 2023-10-18T21:21:37.349 DEBUG:boto:Path: /bofrnf-70/obj10 2023-10-18T21:21:37.349 DEBUG:boto:Data: 2023-10-18T21:21:37.350 DEBUG:boto:Headers: {} 2023-10-18T21:21:37.350 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:37.351 DEBUG:boto:Port: 8000 2023-10-18T21:21:37.351 DEBUG:boto:Params: {} 2023-10-18T21:21:37.352 DEBUG:boto:Token: None 2023-10-18T21:21:37.352 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:37 GMT /bofrnf-70/obj10 2023-10-18T21:21:37.352 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:FKKxDLBzK1VyDcbFKtNardyhHr0= 2023-10-18T21:21:37.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FKKxDLBzK1VyDcbFKtNardyhHr0=', 'Content-Length': '0'} 2023-10-18T21:21:37.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-71:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e8391b22596d60c-0065304c61-270f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:21:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:37.355 DEBUG:boto:path=/bofrnf-70/obj10 2023-10-18T21:21:37.356 DEBUG:boto:auth_path=/bofrnf-70/obj10 2023-10-18T21:21:37.356 DEBUG:boto:Method: GET 2023-10-18T21:21:37.357 DEBUG:boto:Path: /bofrnf-70/obj10 2023-10-18T21:21:37.357 DEBUG:boto:Data: 2023-10-18T21:21:37.357 DEBUG:boto:Headers: {} 2023-10-18T21:21:37.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:37.358 DEBUG:boto:Port: 8000 2023-10-18T21:21:37.359 DEBUG:boto:Params: {} 2023-10-18T21:21:37.359 DEBUG:boto:Token: None 2023-10-18T21:21:37.360 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:21:37 GMT /bofrnf-70/obj10 2023-10-18T21:21:37.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:EU2ppb2nWZoev1SdkCtyYyrsXi0= 2023-10-18T21:21:37.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:37 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EU2ppb2nWZoev1SdkCtyYyrsXi0=', 'Content-Length': '0'} 2023-10-18T21:21:37.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '419020ed-d3b2-4ce0-b36c-97abb284c459:bofrnf-71:403a921b-fcfc-4ab2-b434-bf29d35a51b6.9352.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b837d5b5d28bee1-0065304c61-270f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:21:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:37.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:37.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:37.364 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket bofrnf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:37.413 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:37.413 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:37.467 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:21:37.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:37.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:37.469 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:37.518 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:37.518 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:37.564 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:21:37.564 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2023-10-18T21:21:37.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:37.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:37.567 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:37.615 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:37.615 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:37.645 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:21:37.658 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:21:37.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.658 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664097.449895_2017.1", 2023-10-18T21:21:37.658 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:37.449895Z" 2023-10-18T21:21:37.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.659 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:37.659 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:21:37.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.660 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:37.660 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:21:37.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.661 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:37.661 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:21:37.661 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.661 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:21:37.662 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:37.663 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:21:37.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:37.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:37.664 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:37.713 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:37.714 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:37.742 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:37.741+0000 7feaeb73c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:37.743 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:37.742+0000 7feaeb73c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:37.756 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:37.756 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:37.757 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:37.757 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:37.757 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:37.757 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:37.758 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.758 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:37.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.759 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:37.759 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.759 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:37.759 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:21:37.759 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.760 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:37.760 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.760 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:21:37.760 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.760 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.761 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:37.761 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.761 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:37.761 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:37.762 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.762 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.762 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.762 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:37.763 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.763 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.763 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:37.764 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.764 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:37.764 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:37.764 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.764 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.765 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.765 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:37.765 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.765 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:37.766 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:37.766 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:37.766 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:37.766 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:37.767 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:37.767 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:37.767 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:37.767 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:37.767 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:37.767 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:37.768 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:37.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.768 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:37.769 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:37.769 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:37.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:37.769 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:37Z" 2023-10-18T21:21:37.769 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:37.770 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664076.148026_2005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:16.148026Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:37Z" } 2023-10-18T21:21:37.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664076.148026_2005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.148026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:37.771 DEBUG:rgw_multi.tests:log_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:37.772 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:37.772 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664097.449895_2017.1 target=1_1697664076.148026_2005.1 2023-10-18T21:21:42.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:42.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:42.774 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:42.831 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:42.832 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:42.865 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:42.864+0000 7fe761fbe780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:42.866 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:42.865+0000 7fe761fbe780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:42.880 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:42.880 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:42.880 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:42.881 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:42.881 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:42.881 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:42.882 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:42.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:42.882 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:42.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:42.883 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:42.883 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:42.883 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:42.884 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:21:42.884 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:42.884 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:42.884 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:42.885 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:21:42.885 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:42.885 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:42.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:42.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:42.886 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:42.886 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:42.886 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:42.886 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:42.886 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:42.887 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:42.887 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:42.887 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:42.887 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:42.887 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:42.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:42.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:42.888 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:42.888 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:42.888 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:42.889 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:42.889 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:42.889 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:42.889 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:42.889 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:42.890 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:42.890 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:42.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:42.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:42.891 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:42.891 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:42.891 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:42.891 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:42.891 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:42.892 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:42.892 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:42.892 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:42.892 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:42.892 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:42.893 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:42.893 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:42.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:42.893 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:42.893 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:42.893 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:42.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:42.894 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:42Z" 2023-10-18T21:21:42.894 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:42.896 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664076.148026_2005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:16.148026Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:42Z" } 2023-10-18T21:21:42.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664076.148026_2005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.148026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:42.897 DEBUG:rgw_multi.tests:log_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:42.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:42.899 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664097.449895_2017.1 target=1_1697664076.148026_2005.1 2023-10-18T21:21:47.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:47.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:47.901 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:47.958 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:47.958 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:47.990 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:47.990+0000 7f5748fdd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:47.991 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:47.991+0000 7f5748fdd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:48.005 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:21:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:21:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:47Z" 2023-10-18T21:21:48.019 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:48.020 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664076.148026_2005.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:16.148026Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:47Z" } 2023-10-18T21:21:48.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664076.148026_2005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.148026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:48.021 DEBUG:rgw_multi.tests:log_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:48.021 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:48.022 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664097.449895_2017.1 target=1_1697664076.148026_2005.1 2023-10-18T21:21:53.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:53.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:53.024 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:53.082 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:53.082 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:53.114 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:53.113+0000 7f7fc1bc8780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:53.115 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:53.115+0000 7f7fc1bc8780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:53.129 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:53.130 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:53.130 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:53.130 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:53.130 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:53.131 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:53.131 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.131 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:53.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.132 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:53.132 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:53.132 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:53.132 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664097.449895_2017.1", 2023-10-18T21:21:53.132 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:53.133 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:53.133 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:53.133 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:37.449895Z", 2023-10-18T21:21:53.133 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.133 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.134 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:53.134 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:53.135 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:53.135 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:53.135 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:53.135 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:53.135 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:53.136 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:53.136 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.137 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:53.137 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:53.137 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:53.137 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:53.138 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:53.138 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:53.138 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:53.138 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:53.138 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.139 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.139 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:53.140 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:53.140 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:53.140 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:53.140 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:53.140 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:53.141 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:53.141 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:53.141 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.141 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.141 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.142 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:53.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.142 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:53.142 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:53.142 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:53.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.143 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:53Z" 2023-10-18T21:21:53.143 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:53.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664097.449895_2017.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:37.449895Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:53Z" } 2023-10-18T21:21:53.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664097.449895_2017.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:37.449895Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:53.145 DEBUG:rgw_multi.tests:log_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:53.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:53.146 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:21:53.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.147 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:53.194 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:53.195 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:53.239 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[]} 2023-10-18T21:21:53.240 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-72 2023-10-18T21:21:53.241 DEBUG:boto:path=/bofrnf-72/ 2023-10-18T21:21:53.241 DEBUG:boto:auth_path=/bofrnf-72/ 2023-10-18T21:21:53.241 DEBUG:boto:Method: PUT 2023-10-18T21:21:53.242 DEBUG:boto:Path: /bofrnf-72/ 2023-10-18T21:21:53.242 DEBUG:boto:Data: 2023-10-18T21:21:53.243 DEBUG:boto:Headers: {} 2023-10-18T21:21:53.243 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:53.243 DEBUG:boto:Port: 8000 2023-10-18T21:21:53.245 DEBUG:boto:Params: {} 2023-10-18T21:21:53.246 DEBUG:boto:Token: None 2023-10-18T21:21:53.246 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:21:53 GMT /bofrnf-72/ 2023-10-18T21:21:53.247 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Xqh0ZCCaDABnjrO2IbpCnqF6hc0= 2023-10-18T21:21:53.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xqh0ZCCaDABnjrO2IbpCnqF6hc0=', 'Content-Length': '0'} 2023-10-18T21:21:53.262 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054aacebce12bd713-0065304c71-270f-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:53.263 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-73 2023-10-18T21:21:53.263 DEBUG:boto:path=/bofrnf-73/ 2023-10-18T21:21:53.263 DEBUG:boto:auth_path=/bofrnf-73/ 2023-10-18T21:21:53.264 DEBUG:boto:Method: PUT 2023-10-18T21:21:53.264 DEBUG:boto:Path: /bofrnf-73/ 2023-10-18T21:21:53.265 DEBUG:boto:Data: 2023-10-18T21:21:53.265 DEBUG:boto:Headers: {} 2023-10-18T21:21:53.265 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:53.266 DEBUG:boto:Port: 8000 2023-10-18T21:21:53.266 DEBUG:boto:Params: {} 2023-10-18T21:21:53.267 DEBUG:boto:Token: None 2023-10-18T21:21:53.267 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:21:53 GMT /bofrnf-73/ 2023-10-18T21:21:53.268 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:cWfc+cJzaPkc70mxpvZssEqDiFU= 2023-10-18T21:21:53.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWfc+cJzaPkc70mxpvZssEqDiFU=', 'Content-Length': '0'} 2023-10-18T21:21:53.283 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f211f7c82b00c23f-0065304c71-270f-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:53.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:53.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:53.284 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:53.332 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:53.333 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:53.371 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:21:53.387 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:21:53.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.388 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:21:53.388 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:53.281518Z" 2023-10-18T21:21:53.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.389 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:53.389 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:21:53.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.390 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:53.390 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:21:53.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.391 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:53.391 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:21:53.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.391 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:21:53.392 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:53.393 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:21:53.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:53.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:53.394 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:53.443 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:53.444 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:53.473 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:53.472+0000 7fe1a6d48780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:53.474 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:53.473+0000 7fe1a6d48780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:53.487 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:53.487 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:53.487 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:53.487 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:53.487 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:53.488 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:53.488 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.488 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:53.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.489 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:53.489 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:53.489 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:53.489 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:21:53.490 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:53.490 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:53.490 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:53.490 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:53.281518Z", 2023-10-18T21:21:53.490 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.491 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.491 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:53.491 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:53.492 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:53.492 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:53.492 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:53.492 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:53.492 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:53.493 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:53.493 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.493 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.494 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:53.494 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:53.494 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:53.494 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:53.495 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:53.496 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:53.497 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:53.497 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:53.497 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:53.497 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:53.497 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:53.498 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:53.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:53.500 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:53Z" 2023-10-18T21:21:53.500 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:53.501 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664113.281518_2033.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:53.281518Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:53Z" } 2023-10-18T21:21:53.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664113.281518_2033.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:53.281518Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:53.502 DEBUG:rgw_multi.tests:log_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:53.502 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:53.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:21:53.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.504 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:53.553 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:53.554 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:53.598 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:21:53.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.600 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:53.654 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:53.654 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:53.700 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-10-18T21:21:53.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-72', '--dest-bucket=bofrnf-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-72', '--dest-bucket=bofrnf-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.701 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-bofrnf-72 --source-zones=a1 --dest-zones=a2 --source-bucket=bofrnf-72 --dest-bucket=bofrnf-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:53.755 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:53.755 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:53.801 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:21:53.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-73', '--dest-bucket=bofrnf-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-73', '--dest-bucket=bofrnf-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.802 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-bofrnf-73 --source-zones=a1 --dest-zones=a2 --source-bucket=bofrnf-73 --dest-bucket=bofrnf-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:53.857 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:53.858 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:53.902 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:21:53.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.903 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:53.952 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:53.952 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:53.998 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:21:53.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:53.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:54.000 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:54.048 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:54.048 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:54.101 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:54.102 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:54.102 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 20, 2023-10-18T21:21:54.102 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:21:54.102 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:21:54.103 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:21:54.103 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:54.103 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:21:54.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.103 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:21:54.104 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:21:54.104 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:21:54.104 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:21:54.104 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:21:54.104 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:21:54.105 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.105 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:21:54.105 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:21:54.105 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:21:54.105 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:21:54.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.106 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:21:54.106 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:21:54.106 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:21:54.106 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:21:54.107 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.107 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:21:54.107 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:21:54.107 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:21:54.108 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:21:54.108 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:21:54.108 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:21:54.108 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:21:54.108 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:21:54.109 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:21:54.109 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:21:54.109 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:21:54.109 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.110 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:21:54.110 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:21:54.110 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:21:54.110 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:21:54.111 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.111 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:21:54.111 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:21:54.111 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:21:54.111 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:21:54.112 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:21:54.112 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:21:54.112 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:21:54.112 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:21:54.112 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:21:54.113 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:21:54.113 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:21:54.113 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.113 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.113 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.114 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:21:54.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.114 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:21:54.114 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:21:54.115 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:21:54.115 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:21:54.115 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.115 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.115 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.116 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:21:54.116 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:21:54.116 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:21:54.116 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [ 2023-10-18T21:21:54.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-group", 2023-10-18T21:21:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: "data_flow": { 2023-10-18T21:21:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: "directional": [ 2023-10-18T21:21:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: "source_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:21:54.118 INFO:teuthology.orchestra.run.smithi082.stdout: "dest_zone": "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:21:54.118 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.118 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.118 INFO:teuthology.orchestra.run.smithi082.stdout: "pipes": [ 2023-10-18T21:21:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-72", 2023-10-18T21:21:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:21:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-72", 2023-10-18T21:21:54.119 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:21:54.120 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:21:54.120 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.120 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:21:54.121 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-73", 2023-10-18T21:21:54.121 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:21:54.121 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:21:54.121 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.122 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:21:54.122 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:21:54.122 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:21:54.122 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:21:54.122 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.123 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:21:54.123 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:21:54.123 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:21:54.124 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:21:54.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.124 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-73", 2023-10-18T21:21:54.125 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:21:54.125 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-73", 2023-10-18T21:21:54.125 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:21:54.125 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:21:54.125 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.126 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:21:54.126 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-73", 2023-10-18T21:21:54.126 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:21:54.127 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:21:54.127 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.127 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:21:54.127 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:21:54.128 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:21:54.128 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:21:54.128 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.128 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:21:54.129 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:21:54.129 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:21:54.129 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:21:54.129 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.129 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.130 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.130 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "enabled" 2023-10-18T21:21:54.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.130 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.131 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:21:54.131 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:21:54.131 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.132 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:21:54.132 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:21:54.132 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:21:54.132 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:21:54.132 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:21:54.133 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:21:54.133 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.133 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:21:54.133 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:21:54.133 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:21:54.134 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:21:54.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.134 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:21:54.134 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:21:54.134 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:21:54.135 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:21:54.135 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.135 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:21:54.135 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:21:54.135 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:21:54.136 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:21:54.136 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:21:54.136 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:21:54.136 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:21:54.137 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:21:54.137 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:21:54.137 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:21:54.137 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:21:54.137 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.138 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:21:54.138 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:21:54.138 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:21:54.139 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:21:54.139 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.139 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:21:54.139 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:21:54.139 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:21:54.140 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:21:54.140 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:21:54.140 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:21:54.140 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:21:54.140 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:21:54.142 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:21:54.142 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:21:54.142 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:21:54.142 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.143 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.143 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:21:54.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.144 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:21:54.144 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:21:54.144 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:21:54.144 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:21:54.144 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.145 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.145 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.145 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:21:54.145 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:21:54.146 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:21:54.146 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:21:54.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.146 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:21:54.146 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:21:54.147 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.147 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.147 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:21:54.147 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:21:54.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.148 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:21:54.148 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:21:54.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.148 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:21:54.149 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:21:54.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.149 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:21:54.149 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:21:54.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.150 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:21:54.150 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:21:54.150 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.151 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.151 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:21:54.151 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:21:54.151 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:21:54.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:21:54.152 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:21:54.152 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:21:54.152 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:21:54.152 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:21:54.153 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:21:54.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.153 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:21:54.153 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:21:54.154 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:21:54.154 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:21:54.154 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:21:54.154 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:21:54.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.155 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:21:54.155 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:21:54.155 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:21:54.155 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:21:54.155 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:21:54.156 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:21:54.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.156 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:21:54.156 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:21:54.156 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:21:54.157 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:21:54.157 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:21:54.157 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:21:54.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.157 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:21:54.158 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:21:54.158 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:21:54.158 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:21:54.158 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:21:54.159 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:21:54.159 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.159 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:21:54.159 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:21:54.160 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:54.160 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:54.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:54.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:21:54.161 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:21:54.212 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:54.212 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:54.305 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:21:54.306 DEBUG:boto:path=/bofrnf-72/ 2023-10-18T21:21:54.306 DEBUG:boto:auth_path=/bofrnf-72/ 2023-10-18T21:21:54.307 DEBUG:boto:Method: HEAD 2023-10-18T21:21:54.307 DEBUG:boto:Path: /bofrnf-72/ 2023-10-18T21:21:54.307 DEBUG:boto:Data: 2023-10-18T21:21:54.308 DEBUG:boto:Headers: {} 2023-10-18T21:21:54.308 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:54.308 DEBUG:boto:Port: 8000 2023-10-18T21:21:54.309 DEBUG:boto:Params: {} 2023-10-18T21:21:54.309 DEBUG:boto:Token: None 2023-10-18T21:21:54.310 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:54 GMT /bofrnf-72/ 2023-10-18T21:21:54.310 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:si/R/4V8fIhC004U772ccu8zUIc= 2023-10-18T21:21:54.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:si/R/4V8fIhC004U772ccu8zUIc=', 'Content-Length': '0'} 2023-10-18T21:21:54.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dba44a018f5f9ab7-0065304c72-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:54.320 DEBUG:boto:path=/bofrnf-72/obj1 2023-10-18T21:21:54.320 DEBUG:boto:auth_path=/bofrnf-72/obj1 2023-10-18T21:21:54.321 DEBUG:boto:Method: PUT 2023-10-18T21:21:54.321 DEBUG:boto:Path: /bofrnf-72/obj1 2023-10-18T21:21:54.321 DEBUG:boto:Data: 2023-10-18T21:21:54.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:21:54.322 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:54.323 DEBUG:boto:Port: 8000 2023-10-18T21:21:54.323 DEBUG:boto:Params: {} 2023-10-18T21:21:54.323 DEBUG:boto:Token: None 2023-10-18T21:21:54.324 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:21:54 GMT /bofrnf-72/obj1 2023-10-18T21:21:54.324 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:pjXMvZC2rwaHbcGn5atN0UWu+4E= 2023-10-18T21:21:54.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pjXMvZC2rwaHbcGn5atN0UWu+4E='} 2023-10-18T21:21:54.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008837a000d6f6d8e8-0065304c72-2942-a1'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:54.371 DEBUG:boto:path=/bofrnf-72/ 2023-10-18T21:21:54.372 DEBUG:boto:auth_path=/bofrnf-72/ 2023-10-18T21:21:54.372 DEBUG:boto:Method: HEAD 2023-10-18T21:21:54.372 DEBUG:boto:Path: /bofrnf-72/ 2023-10-18T21:21:54.373 DEBUG:boto:Data: 2023-10-18T21:21:54.373 DEBUG:boto:Headers: {} 2023-10-18T21:21:54.374 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:54.374 DEBUG:boto:Port: 8000 2023-10-18T21:21:54.375 DEBUG:boto:Params: {} 2023-10-18T21:21:54.375 DEBUG:boto:Token: None 2023-10-18T21:21:54.376 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:54 GMT /bofrnf-72/ 2023-10-18T21:21:54.376 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:si/R/4V8fIhC004U772ccu8zUIc= 2023-10-18T21:21:54.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:si/R/4V8fIhC004U772ccu8zUIc=', 'Content-Length': '0'} 2023-10-18T21:21:54.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023bdb03a660346b8-0065304c72-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:54.381 DEBUG:boto:path=/bofrnf-72/obj2 2023-10-18T21:21:54.381 DEBUG:boto:auth_path=/bofrnf-72/obj2 2023-10-18T21:21:54.382 DEBUG:boto:Method: PUT 2023-10-18T21:21:54.382 DEBUG:boto:Path: /bofrnf-72/obj2 2023-10-18T21:21:54.383 DEBUG:boto:Data: 2023-10-18T21:21:54.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:21:54.384 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:54.384 DEBUG:boto:Port: 8000 2023-10-18T21:21:54.384 DEBUG:boto:Params: {} 2023-10-18T21:21:54.385 DEBUG:boto:Token: None 2023-10-18T21:21:54.385 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:21:54 GMT /bofrnf-72/obj2 2023-10-18T21:21:54.386 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gJgznvhWp4aGaeMR5ZCGWqmpwnM= 2023-10-18T21:21:54.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gJgznvhWp4aGaeMR5ZCGWqmpwnM='} 2023-10-18T21:21:54.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fe6fb41ab593284-0065304c72-2942-a1'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:54.396 DEBUG:boto:path=/bofrnf-73/ 2023-10-18T21:21:54.397 DEBUG:boto:auth_path=/bofrnf-73/ 2023-10-18T21:21:54.397 DEBUG:boto:Method: HEAD 2023-10-18T21:21:54.398 DEBUG:boto:Path: /bofrnf-73/ 2023-10-18T21:21:54.398 DEBUG:boto:Data: 2023-10-18T21:21:54.398 DEBUG:boto:Headers: {} 2023-10-18T21:21:54.399 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:54.399 DEBUG:boto:Port: 8000 2023-10-18T21:21:54.400 DEBUG:boto:Params: {} 2023-10-18T21:21:54.400 DEBUG:boto:Token: None 2023-10-18T21:21:54.401 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:54 GMT /bofrnf-73/ 2023-10-18T21:21:54.401 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wj5Ml4FWe7rTryQzHLhC1Hf7HJk= 2023-10-18T21:21:54.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wj5Ml4FWe7rTryQzHLhC1Hf7HJk=', 'Content-Length': '0'} 2023-10-18T21:21:54.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011856543536123a2-0065304c72-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:54.406 DEBUG:boto:path=/bofrnf-73/obj3 2023-10-18T21:21:54.406 DEBUG:boto:auth_path=/bofrnf-73/obj3 2023-10-18T21:21:54.407 DEBUG:boto:Method: PUT 2023-10-18T21:21:54.407 DEBUG:boto:Path: /bofrnf-73/obj3 2023-10-18T21:21:54.408 DEBUG:boto:Data: 2023-10-18T21:21:54.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:21:54.409 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:54.409 DEBUG:boto:Port: 8000 2023-10-18T21:21:54.409 DEBUG:boto:Params: {} 2023-10-18T21:21:54.410 DEBUG:boto:Token: None 2023-10-18T21:21:54.410 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:21:54 GMT /bofrnf-73/obj3 2023-10-18T21:21:54.411 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:6UT2N/clV/eruVzctJ1/Pta4H4I= 2023-10-18T21:21:54.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6UT2N/clV/eruVzctJ1/Pta4H4I='} 2023-10-18T21:21:54.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008481a9e3276c9813-0065304c72-2942-a1'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:54.455 DEBUG:boto:path=/bofrnf-73/ 2023-10-18T21:21:54.456 DEBUG:boto:auth_path=/bofrnf-73/ 2023-10-18T21:21:54.456 DEBUG:boto:Method: HEAD 2023-10-18T21:21:54.456 DEBUG:boto:Path: /bofrnf-73/ 2023-10-18T21:21:54.457 DEBUG:boto:Data: 2023-10-18T21:21:54.457 DEBUG:boto:Headers: {} 2023-10-18T21:21:54.458 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:54.458 DEBUG:boto:Port: 8000 2023-10-18T21:21:54.459 DEBUG:boto:Params: {} 2023-10-18T21:21:54.459 DEBUG:boto:Token: None 2023-10-18T21:21:54.460 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:21:54 GMT /bofrnf-73/ 2023-10-18T21:21:54.460 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wj5Ml4FWe7rTryQzHLhC1Hf7HJk= 2023-10-18T21:21:54.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wj5Ml4FWe7rTryQzHLhC1Hf7HJk=', 'Content-Length': '0'} 2023-10-18T21:21:54.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013170a774f048246-0065304c72-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:54.464 DEBUG:boto:path=/bofrnf-73/obj4 2023-10-18T21:21:54.465 DEBUG:boto:auth_path=/bofrnf-73/obj4 2023-10-18T21:21:54.465 DEBUG:boto:Method: PUT 2023-10-18T21:21:54.466 DEBUG:boto:Path: /bofrnf-73/obj4 2023-10-18T21:21:54.466 DEBUG:boto:Data: 2023-10-18T21:21:54.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:21:54.467 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:21:54.467 DEBUG:boto:Port: 8000 2023-10-18T21:21:54.468 DEBUG:boto:Params: {} 2023-10-18T21:21:54.468 DEBUG:boto:Token: None 2023-10-18T21:21:54.469 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:21:54 GMT /bofrnf-73/obj4 2023-10-18T21:21:54.469 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:4Glt+7KapLmZ+CehFJ6AmLbgVyI= 2023-10-18T21:21:54.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4Glt+7KapLmZ+CehFJ6AmLbgVyI='} 2023-10-18T21:21:54.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fa2712244d9bd92-0065304c72-2942-a1'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:21:54.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:54.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:54.483 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:54.559 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:54.560 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:54.623 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:21:54.655 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:21:54.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.655 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:21:54.656 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:53.281518Z" 2023-10-18T21:21:54.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.656 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:54.656 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:21:54.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.657 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:54.657 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:21:54.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.658 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:54.658 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:21:54.658 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.659 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:21:54.660 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:54.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:21:54.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:54.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:54.661 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:54.734 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:54.734 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:54.791 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:54.790+0000 7f3bc0aa2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:54.792 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:21:54.791+0000 7f3bc0aa2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:21:54.818 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:54.819 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:54.819 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:54.819 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:54.820 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:54.820 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:21:54.820 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:54.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.821 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:54.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.821 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:54.821 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:54.822 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:54.822 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:21:54.822 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:54.822 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:21:54.823 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:54.823 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:53.281518Z", 2023-10-18T21:21:54.823 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:54.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.824 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:54.824 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:54.824 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:54.825 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:21:54.825 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:54.825 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:54.825 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:54.826 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:21:54.826 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:54.826 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.827 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:54.827 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:54.827 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:54.828 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:21:54.828 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:54.828 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:54.828 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:54.829 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:21:54.829 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:54.830 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.831 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:54.831 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:54.832 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:21:54.832 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:21:54.833 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:54.833 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:54.833 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:54.834 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:21:54.834 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:21:54.835 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.835 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.836 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:54.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.836 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:54.837 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:21:54.837 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:21:54.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.838 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:54Z" 2023-10-18T21:21:54.838 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:54.839 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664113.281518_2033.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:53.281518Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:21:54Z" } 2023-10-18T21:21:54.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664113.281518_2033.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:53.281518Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:21:54.841 DEBUG:rgw_multi.tests:log_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:54.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:21:54.841 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:21:54.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:54.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:54.843 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:54.897 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:54.898 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:54.959 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:21:54.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.960 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:21:54.960 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:21:54.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.960 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:21:54.961 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:21:54.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.961 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:21:54.962 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:21:54.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:54.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:54.962 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:21:54.963 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:21:54.963 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:54.963 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:21:54.964 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:21:54.965 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:21:54.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:54.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:21:54.966 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:21:55.016 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:21:55.017 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:21:55.059 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:21:55.060 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:21:55.060 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:21:55.060 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:21:55.061 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:21:55.061 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:21:55.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:55.061 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:21:55.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:55.062 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:21:55.062 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:55.062 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:55.063 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:21:55.063 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:55.063 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:55.064 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:55.064 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:21:55.064 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:55.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:55.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:55.066 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:21:55.066 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:55.066 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:55.067 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:21:55.067 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:55.068 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:21:55.068 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:55.068 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:21:55.069 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:55.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:55.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:55.070 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:21:55.070 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:55.071 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:55.071 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:21:55.072 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:55.072 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:21:55.072 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:55.072 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:21:55.073 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:55.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:55.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:21:55.073 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:21:55.073 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:21:55.074 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:21:55.074 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:21:55.074 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:21:55.074 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:21:55.075 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:21:55.075 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:21:55.075 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:55.075 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:21:55.075 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:21:55.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:55.076 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:21:55.076 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:21:55.076 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:21:55.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:21:55.077 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:21:55Z" 2023-10-18T21:21:55.077 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:21:55.078 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:21:55Z" } 2023-10-18T21:21:55.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:21:55.079 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:21:55.079 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:21:55.080 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:00.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:00.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:00.082 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:00.142 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:00.142 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:00.191 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:00.192 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:00.192 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:00.192 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:00.192 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:00.192 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:00.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:00.193 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:00.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:00.193 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:00.194 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:00.194 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:00.194 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:00.194 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:00.194 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:00.195 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:00.195 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:00.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:00.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:00.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:00.196 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:00.196 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:00.196 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:00.196 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:00.196 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:00.197 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:00.197 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:00.197 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:00.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:00.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:00.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:00.198 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:00.198 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:00.198 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:00.198 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:00.199 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:00.199 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:00.199 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:00.199 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:00.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:00.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:00.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:00.200 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:00.200 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:00.201 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:00.201 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:00.201 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:00.201 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:00.201 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:00.202 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:00.202 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:00.202 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:00.202 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:00.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:00.203 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:00.203 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:00.203 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:00.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:00.203 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:00Z" 2023-10-18T21:22:00.204 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:00.205 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:00Z" } 2023-10-18T21:22:00.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:00.206 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:00.206 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:00.206 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:05.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:05.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:05.209 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:05.267 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:05.268 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:05.316 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:05.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:05.326 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:05.326 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:05.326 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:05.326 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:05.327 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:05.327 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:05.327 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:05.327 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:05.327 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:05.328 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:05.328 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:05.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:05.328 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:05Z" 2023-10-18T21:22:05.329 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:05.330 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:05Z" } 2023-10-18T21:22:05.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:05.331 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:05.332 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:05.332 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:10.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:10.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:10.335 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:10.394 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:10.394 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:10.443 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:10.443 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:10.443 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:10.444 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:10.444 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:10.444 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:10.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:10.444 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:10.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:10.445 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:10.445 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:10.445 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:10Z" 2023-10-18T21:22:10.456 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:10.457 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:10Z" } 2023-10-18T21:22:10.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:10.458 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:10.458 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:10.459 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:15.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:15.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:15.462 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:15.522 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:15.523 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:15.570 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:15.571 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:15.571 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:15.571 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:15.571 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:15.572 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:15.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:15.572 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:15.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:15.572 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:15.573 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:15.573 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:15.573 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:15.573 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:15.573 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:15.574 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:15.574 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:15.574 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:15.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:15.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:15.575 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:15.575 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:15.575 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:15.575 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:15.575 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:15.576 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:15.576 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:15.576 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:15.576 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:15.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:15.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:15.577 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:15.577 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:15.577 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:15.577 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:15.578 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:15.578 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:15.578 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:15.578 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:15.579 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:15.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:15.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:15.579 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:15.579 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:15.580 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:15.580 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:15.580 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:15.580 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:15.580 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:15.581 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:15.581 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:15.581 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:15.581 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:15.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:15.581 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:15.582 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:15.582 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:15.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:15.582 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:15Z" 2023-10-18T21:22:15.582 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:15.586 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:15Z" } 2023-10-18T21:22:15.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:15.587 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:15.587 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:15.588 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:20.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:20.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:20.590 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:20.648 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:20.649 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:20.697 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:20.697 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:20.697 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:20.698 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:20.698 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:20.698 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:20.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:20.698 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:20.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:20.699 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:20.699 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:20.699 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:20.700 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:20.700 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:20.700 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:20.700 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:20.700 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:20.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:20.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:20.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:20.701 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:20.701 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:20.702 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:20.702 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:20.702 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:20.702 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:20.702 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:20.703 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:20.703 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:20.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:20.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:20.703 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:20.704 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:20.704 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:20.704 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:20.704 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:20.704 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:20.705 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:20.705 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:20.705 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:20.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:20.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:20.706 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:20.706 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:20.706 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:20.706 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:20.706 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:20.707 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:20.707 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:20.707 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:20.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:20.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:20.708 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:20.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:20.708 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:20.708 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:20.708 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:20.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:20.709 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:20Z" 2023-10-18T21:22:20.709 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:20.710 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:20Z" } 2023-10-18T21:22:20.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:20.713 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:20.714 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:20.714 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:25.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:25.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:25.717 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:25.776 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:25.777 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:25.824 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:25.825 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:25.825 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:25.825 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:25.826 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:25.826 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:25.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:25.826 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:25.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:25.827 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:25.827 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:25.827 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:25.827 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:25.828 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:25.828 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:25.828 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:25.828 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:25.828 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:25.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:25.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:25.829 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:25.829 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:25.829 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:25.830 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:25.830 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:25.830 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:25.830 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:25.830 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:25.831 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:25.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:25.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:25.831 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:25.831 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:25.832 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:25.832 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:25.832 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:25.832 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:25.832 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:25.833 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:25.833 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:25.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:25.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:25.834 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:25.834 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:25.834 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:25.834 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:25.834 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:25.835 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:25.835 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:25.835 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:25.835 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:25.835 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:25.836 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:25.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:25.836 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:25.836 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:25.836 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:25.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:25.837 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:25Z" 2023-10-18T21:22:25.837 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:25.838 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:25Z" } 2023-10-18T21:22:25.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:25.839 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:25.840 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:25.840 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:30.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:30.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:30.843 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:30.901 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:30.902 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:30.950 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:30.951 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:30.951 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:30Z" 2023-10-18T21:22:30.964 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:30.965 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:30Z" } 2023-10-18T21:22:30.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:30.966 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:30.967 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:30.967 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:35.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:35.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:35.969 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:36.029 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:36.030 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:36.078 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:36.078 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:36.078 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:36.079 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:36.079 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:36.079 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:36.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:36.080 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:36.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:36.080 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:36.080 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:36.080 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:36.081 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:36.081 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:36.081 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:36.081 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:36.081 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:36.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:36.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:36.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:36.082 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:36.083 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:36.083 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:36.083 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:36.083 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:36.083 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:36.084 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:36.084 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:36.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:36.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:36.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:36.085 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:36.085 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:36.085 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:36.085 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:36.085 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:36.086 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:36.086 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:36.086 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:36.086 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:36.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:36.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:36.087 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:36.087 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:36.087 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:36.087 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:36.088 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:36.088 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:36.088 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:36.088 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:36Z" 2023-10-18T21:22:36.090 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:36.091 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:36Z" } 2023-10-18T21:22:36.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:36.092 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:36.093 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:36.093 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:41.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:41.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:41.096 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:41.155 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:41.155 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:41.206 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:41.206 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:41.206 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:41.207 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:41.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:41.208 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:41.208 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:41.208 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:41.208 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:41.210 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:41.211 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:41.212 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:41.213 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:41.214 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:41.215 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:41.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:41.217 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:41Z" 2023-10-18T21:22:41.218 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:41.219 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:41Z" } 2023-10-18T21:22:41.219 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:41.220 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:41.220 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:41.220 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:46.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:46.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:46.223 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:46.282 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:46.283 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:46.333 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:46.333 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:46.333 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:46.333 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:46.334 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:46.334 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:46.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:46.334 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:46.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:46.335 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:46.335 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:46.335 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:46.335 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:46.336 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:46.336 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:46.336 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:46.336 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:46.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:46.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:46.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:46.337 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:46.337 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:46.337 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:46.338 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:46.338 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:46.338 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:46.338 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:46.339 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:46.339 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:46.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:46.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:46.339 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:46.340 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:46.340 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:46.340 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:46.340 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:46.340 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:46.341 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:46.341 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:46.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:46.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:46.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:46.342 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:46.342 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:46.342 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:46.342 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:46.343 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:46.343 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:46.343 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:46.343 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:46.343 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:46.344 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:46.344 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:46.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:46.344 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:46.344 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:46.345 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:46.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:46.345 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:46Z" 2023-10-18T21:22:46.345 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:46.346 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:46Z" } 2023-10-18T21:22:46.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:46.347 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:46.348 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:46.348 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:51.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:51.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:51.350 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:51.409 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:51.410 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:51.458 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:51.459 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:51.459 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:51.459 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:51.460 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:51.460 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:51.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:51.460 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:51.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:51.461 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:51.461 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:51.461 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:51.461 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:51.461 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:51.462 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:51.462 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:51.462 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:51.462 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:51.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:51.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:51.463 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:51.463 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:51.463 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:51.464 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:51.464 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:51.464 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:51.464 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:51.464 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:51.465 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:51.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:51.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:51.465 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:51.465 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:51.466 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:51.466 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:51.466 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:51.466 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:51.467 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:51.467 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:51.467 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:51.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:51.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:51.468 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:51.468 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:51.468 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:51.468 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:51.468 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:51.469 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:51.469 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:51.469 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:51.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:51.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:51.470 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:51.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:51.470 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:51.470 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:51.470 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:51.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:51.471 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:51Z" 2023-10-18T21:22:51.471 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:51.472 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:51Z" } 2023-10-18T21:22:51.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:51.473 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:51.474 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:51.474 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:22:56.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:56.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:22:56.477 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:22:56.536 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:22:56.536 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:22:56.585 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:22:56.585 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:22:56.585 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:22:56.586 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:22:56.586 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:22:56.586 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:22:56.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:56.587 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:22:56.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:56.587 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:22:56.587 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:56.587 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:56.588 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:22:56.588 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:56.588 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:56.588 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:56.588 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:22:56.589 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:56.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:56.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:56.589 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:22:56.589 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:56.590 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:56.590 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:22:56.590 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:56.590 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:22:56.590 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:56.591 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:22:56.591 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:56.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:56.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:56.591 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:22:56.592 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:56.592 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:56.592 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:22:56.592 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:56.592 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:22:56.593 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:56.593 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:22:56.593 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:56.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:56.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:22:56.594 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:22:56.594 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:22:56.594 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:22:56.594 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:22:56.595 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:22:56.595 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:22:56.595 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:22:56.595 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:22:56.595 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:56.596 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:22:56.596 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:22:56.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:56.596 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:22:56.596 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:22:56.597 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:22:56.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:22:56.597 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:22:56Z" 2023-10-18T21:22:56.597 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:22:56.598 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:22:56Z" } 2023-10-18T21:22:56.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:22:56.599 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:22:56.599 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:22:56.600 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:23:01.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:01.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:01.603 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:01.662 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:01.662 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:01.712 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:01.712 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:01.712 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:01.712 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:01.713 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:01.713 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:01.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:01.713 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:01.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:01.714 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:01.714 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:01.714 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:01.714 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:01.714 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:01.715 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:01.715 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:01.715 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:01.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:01.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:01.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:01.716 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:01.716 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:01.716 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:01.716 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:01.717 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:01.717 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:01.717 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:01.717 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:01.718 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:01.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:01.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:01.718 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:01.718 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:01.719 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:01.719 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:23:01.719 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:01.719 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:01.719 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:01.720 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:23:01.720 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:01.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:01.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:01.720 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:01.721 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:01.721 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:01.721 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:23:01.721 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:01.721 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:01.722 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:01.722 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:23:01.722 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:01.722 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:01.722 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:01.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:01.723 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:01.723 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:01.723 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:01.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:01.724 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:01Z" 2023-10-18T21:23:01.724 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:01.725 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:01Z" } 2023-10-18T21:23:01.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:23:01.726 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:01.726 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:23:01.726 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:23:06.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:06.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:06.728 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:06.787 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:06.788 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:06.836 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:06.836 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:06.837 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:06.837 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:06.837 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:06.837 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:06.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:06.838 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:06.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:06.838 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:06.838 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:06.838 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:06.839 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:06.839 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:06.839 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:06.839 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:06.839 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:06.840 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:06.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:06.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:06.840 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:06.840 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:06.841 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:06.841 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:06.841 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:06.841 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:06.841 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:06.842 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:06.842 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:06.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:06.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:06.842 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:06.843 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:06.843 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:06.843 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:23:06.843 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:06.843 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:06.844 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:06.844 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:23:06.844 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:06.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:06.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:06.845 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:06.845 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:06.845 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:06.845 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:23:06.845 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:06.846 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:06.846 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:06.846 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:23:06.846 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:06.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:06.848 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:06Z" 2023-10-18T21:23:06.848 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:06.849 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:06Z" } 2023-10-18T21:23:06.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:23:06.850 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:06.850 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:23:06.851 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:23:11.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:11.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:11.853 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:11.912 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:11.913 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:11.961 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:11.961 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:11.962 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:11.962 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:11.962 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:11.962 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:11.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:11.963 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:11.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:11.963 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:11.963 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:11.964 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:11.964 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:11.964 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:11.964 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:11.964 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:11.965 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:11.965 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:11.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:11.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:11.965 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:11.966 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:11.966 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:11.966 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:11.966 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:11.967 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:11.967 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:11.967 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:11.967 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:11.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:11.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:11.968 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:11.968 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:11.968 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:11.969 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:23:11.969 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:11.969 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:11.969 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:11.969 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:23:11.970 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:11.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:11.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:11.971 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:11.971 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:11.971 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:11.971 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:23:11.971 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:11.972 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:11.972 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:11.972 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:23:11.972 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:11.973 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:11.973 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:11.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:11.973 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:11.973 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:11.973 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:11.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:11.974 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:11Z" 2023-10-18T21:23:11.974 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:11.975 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:11Z" } 2023-10-18T21:23:11.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:23:11.976 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:11.976 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:23:11.977 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:23:16.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:16.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:16.979 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:17.039 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:17.039 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:17.088 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:17.088 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:17.089 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:17.089 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:17.089 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:17.089 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:17.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:17.090 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:17.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:17.090 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:17.090 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:17.091 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:17.091 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:17.091 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:17.091 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:17.091 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:17.091 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:17.092 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:17.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:17.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:17.092 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:17.093 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:17.093 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:17.093 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:17.093 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:17.093 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:17.094 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:17.094 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:17.094 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:17.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:17.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:17.095 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:17.095 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:17.095 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:17.095 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:23:17.095 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:17.096 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:17.096 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:17.096 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:23:17.096 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:17.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:17.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:17.097 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:17.097 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:17.097 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:17.097 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:23:17.098 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:17.098 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:17.098 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:17.098 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:23:17.098 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:17.099 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:17.099 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:17.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:17.099 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:17.099 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:17.100 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:17.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:17.100 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:17Z" 2023-10-18T21:23:17.100 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:17.101 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:17Z" } 2023-10-18T21:23:17.102 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:23:17.102 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:17.103 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:23:17.103 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:23:22.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:22.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:22.106 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:22.165 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:22.165 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:22.215 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:22.215 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:22.215 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:22.216 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:22.216 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:22.216 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:22.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:22.216 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:22.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:22.217 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:22.217 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:22.217 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:22.218 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:22.218 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:22.218 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:22.218 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:22.218 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:22.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:22.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:22.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:22.219 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:22.220 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:22.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:22.222 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:23:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:22.223 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:23:22.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:22.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:22.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:22.224 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:22.224 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:22.225 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:22.225 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:23:22.225 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:22.225 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:22.225 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:22.226 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:23:22.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:22.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:22.226 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:22.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:22.227 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:22.227 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:22.227 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:22.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:22.228 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:22Z" 2023-10-18T21:23:22.228 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:22.229 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:20.728035Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051813", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:19:32.193271Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:22Z" } 2023-10-18T21:23:22.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:23:22.230 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:22.230 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:23:22.231 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:23:27.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:27.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:27.233 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:27.291 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:27.292 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:27.339 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:27.340 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:27.340 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:27.341 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:27.341 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:27.341 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:27.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:27.342 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:27.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:27.343 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:27.343 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:27.344 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:27.344 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:27.344 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:27.345 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:27.345 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:27.345 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:27.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:27.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:27.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:27.347 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:27.347 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:27.347 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:27.348 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:27.348 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:27.348 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:27.349 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:27.349 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:27.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:27.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:27.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:27.350 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:27.351 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:27.351 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:27.351 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:23:27.352 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:27.352 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:27.353 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:27.353 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:23:27.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:27.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:27.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:27.354 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:27.355 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:27.355 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:27.356 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:23:27.356 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:27.356 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:27.357 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:27.357 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:23:27.358 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:27.358 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:27.358 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:27.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:27.359 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:27.359 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:27.359 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:27.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:27.359 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:27Z" 2023-10-18T21:23:27.360 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:27.361 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:27Z" } 2023-10-18T21:23:27.361 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:23:27.362 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:27.362 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:27.363 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:23:32.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:32.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:32.365 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:32.424 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:32.424 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:32.472 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:32.473 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:32.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:32.474 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:32.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:32.474 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:32.478 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:32.478 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:32.478 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:32.478 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:32.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:32.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:32.479 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:32.479 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:32.479 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:32.479 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:23:32.480 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:32.480 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:32.480 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:32.480 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:23:32.480 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:32.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:32.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:32.481 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:32.481 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:32.481 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:32.482 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:23:32.482 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:32.482 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:32.482 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:32.482 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:23:32.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:32.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:32.483 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:32.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:32.483 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:32.484 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:32.484 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:32.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:32.484 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:32Z" 2023-10-18T21:23:32.484 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:32.485 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:32Z" } 2023-10-18T21:23:32.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:23:32.486 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:32.487 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:32.487 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:23:37.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:37.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:37.489 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:37.550 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:37.550 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:37.598 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:37.599 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:37.599 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:37.599 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:37.599 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:37.600 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:37.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:37.600 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:37.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:37.600 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:37.601 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:37.601 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:37.601 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:37.601 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:37.601 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:37.602 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:37.602 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:37.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:37.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:37.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:37.603 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:37.603 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:37.603 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:37.603 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:37.604 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:37.604 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:37.604 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:37.604 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:37.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:37.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:37.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:37.605 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:37.605 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:37.605 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:37.605 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:23:37.606 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:37.606 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:37.606 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:37.606 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:23:37.606 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:37.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:37.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:37.607 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:37.607 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:37.607 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:37.608 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:23:37.608 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:37.608 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:37.608 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:37.608 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:23:37.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:37.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:37.609 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:37.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:37.609 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:37.610 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:37.610 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:37.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:37.610 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:37Z" 2023-10-18T21:23:37.610 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:37.612 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:37Z" } 2023-10-18T21:23:37.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:23:37.613 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:37.613 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:37.613 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:23:42.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:42.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:42.616 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:42.675 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:42.675 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:42.722 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:42.723 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:42.723 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:42.723 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:42.723 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:42.724 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:42.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:42.724 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:42.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:42.724 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:42.725 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:42.725 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:42.725 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:42.725 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:42.726 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:42.726 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:42.726 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:42.726 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:42.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:42.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:42.727 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:42.727 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:42.727 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:42.727 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:42.727 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:42.728 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:42.728 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:42.728 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:42.728 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:42.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:42.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:42.729 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:42.729 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:42.729 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:42.730 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:23:42.730 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:42.730 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:42.730 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:42.730 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:23:42.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:42.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:42.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:42.731 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:42.731 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:42.732 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:42.732 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:23:42.732 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:42.732 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:42.732 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:42.733 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:23:42.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:42.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:42.733 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:42.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:42.734 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:42.734 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:42.734 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:42.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:42.734 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:42Z" 2023-10-18T21:23:42.735 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:42.736 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:42Z" } 2023-10-18T21:23:42.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:23:42.737 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:42.737 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:42.737 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:23:47.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:47.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:47.739 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:47.798 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:47.798 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:47.846 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:47.846 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:47.847 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:47.847 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:47.847 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:47.847 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:47.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:47.848 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:47.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:47.848 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:47.848 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:47.849 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:47.849 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:47.849 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:47.849 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:47.849 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:47.850 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:47.850 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:47.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:47.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:47.850 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:47.851 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:47.851 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:47.851 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:47.851 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:47.851 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:47.852 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:47.852 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:47.852 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:47.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:47.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:47.853 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:47.853 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:47.853 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:47.853 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:23:47.853 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:47.854 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:47.854 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:47.854 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:23:47.854 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:47.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:47.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:47.855 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:47.855 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:47.855 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:47.855 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:23:47.856 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:47.856 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:47.856 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:47.856 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:23:47.857 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:47.857 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:47.857 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:47.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:47.857 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:47.857 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:47.858 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:47.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:47.858 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:47Z" 2023-10-18T21:23:47.858 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:47.859 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:47Z" } 2023-10-18T21:23:47.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:23:47.860 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:47.861 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:47.861 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:23:52.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:52.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:52.863 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:52.923 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:52.923 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:52.972 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:52.972 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:52.973 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:52.973 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:52.973 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:52.973 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:52.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:52.974 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:52.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:52.974 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:52.974 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:52.975 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:52.975 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:52.975 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:52.975 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:52.975 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:52.976 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:52.976 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:52.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:52.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:52.976 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:52.977 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:52.977 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:52.977 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:23:52.977 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:52.977 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:52.978 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:52.978 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:23:52.978 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:52.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:52.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:52.979 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:52.979 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:52.979 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:52.979 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:23:52.979 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:52.980 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:52.980 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:52.980 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:23:52.980 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:52.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:52.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:52.981 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:52.981 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:52.981 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:52.981 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:23:52.982 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:52.982 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:52.982 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:52.982 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:23:52.982 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:52.983 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:52.983 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:52.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:52.983 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:52.983 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:52.984 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:52.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:52.984 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:52Z" 2023-10-18T21:23:52.984 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:52.985 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045799", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:19:32.218710Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:52Z" } 2023-10-18T21:23:52.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:23:52.986 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:52.986 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:52.987 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:23:57.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:57.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:57.989 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:58.048 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:58.048 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:58.097 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:58.098 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:58.098 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:58.098 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:58.098 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:58.099 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:23:58.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.099 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:58.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.099 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:58.100 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.100 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:58.100 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:23:58.100 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.100 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:58.100 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.101 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:23:58.101 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.102 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:58.102 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.102 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:58.102 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:23:58.102 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.103 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:58.103 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.103 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:23:58.103 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.104 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:58.104 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.104 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:58.104 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:23:58.105 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.105 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:23:58.105 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.105 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:23:58.105 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.106 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:58.106 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.106 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:23:58.107 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:23:58.107 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.107 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:23:58.107 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.107 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:23:58.108 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.108 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.108 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:58.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.108 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:58.109 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:23:58.109 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:23:58.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.109 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:58Z" 2023-10-18T21:23:58.109 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:58.110 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:23:58Z" } 2023-10-18T21:23:58.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:23:58.111 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:58.112 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:23:58.112 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:23:58.113 DEBUG:boto:path=/bofrnf-72/ 2023-10-18T21:23:58.113 DEBUG:boto:auth_path=/bofrnf-72/ 2023-10-18T21:23:58.113 DEBUG:boto:Method: HEAD 2023-10-18T21:23:58.114 DEBUG:boto:Path: /bofrnf-72/ 2023-10-18T21:23:58.114 DEBUG:boto:Data: 2023-10-18T21:23:58.115 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.115 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.115 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.116 DEBUG:boto:Params: {} 2023-10-18T21:23:58.116 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:23:58.117 DEBUG:boto:Token: None 2023-10-18T21:23:58.117 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-72/ 2023-10-18T21:23:58.118 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:43sOW77mFvJge1FBJGiOx595DKw= 2023-10-18T21:23:58.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:43sOW77mFvJge1FBJGiOx595DKw=', 'Content-Length': '0'} 2023-10-18T21:23:58.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c30ff2698bbc960-0065304cee-294e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.124 DEBUG:boto:path=/bofrnf-72/obj1 2023-10-18T21:23:58.124 DEBUG:boto:auth_path=/bofrnf-72/obj1 2023-10-18T21:23:58.125 DEBUG:boto:Method: HEAD 2023-10-18T21:23:58.125 DEBUG:boto:Path: /bofrnf-72/obj1 2023-10-18T21:23:58.126 DEBUG:boto:Data: 2023-10-18T21:23:58.126 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.126 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.127 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.127 DEBUG:boto:Params: {} 2023-10-18T21:23:58.129 DEBUG:boto:Token: None 2023-10-18T21:23:58.129 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-72/obj1 2023-10-18T21:23:58.130 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:oq8/tvTriwHvgT7a+VmfWiXWld0= 2023-10-18T21:23:58.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oq8/tvTriwHvgT7a+VmfWiXWld0=', 'Content-Length': '0'} 2023-10-18T21:23:58.132 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a1d5b8aea54b6121-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.132 DEBUG:boto:path=/bofrnf-72/obj2 2023-10-18T21:23:58.133 DEBUG:boto:auth_path=/bofrnf-72/obj2 2023-10-18T21:23:58.133 DEBUG:boto:Method: HEAD 2023-10-18T21:23:58.134 DEBUG:boto:Path: /bofrnf-72/obj2 2023-10-18T21:23:58.134 DEBUG:boto:Data: 2023-10-18T21:23:58.134 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.135 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.135 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.135 DEBUG:boto:Params: {} 2023-10-18T21:23:58.136 DEBUG:boto:Token: None 2023-10-18T21:23:58.136 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-72/obj2 2023-10-18T21:23:58.137 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:A83J0zmGzyFr8lZpb/ggPT3Qi/A= 2023-10-18T21:23:58.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A83J0zmGzyFr8lZpb/ggPT3Qi/A=', 'Content-Length': '0'} 2023-10-18T21:23:58.139 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f76e048952237c2e-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.139 DEBUG:boto:path=/bofrnf-72/obj3 2023-10-18T21:23:58.140 DEBUG:boto:auth_path=/bofrnf-72/obj3 2023-10-18T21:23:58.140 DEBUG:boto:Method: HEAD 2023-10-18T21:23:58.141 DEBUG:boto:Path: /bofrnf-72/obj3 2023-10-18T21:23:58.141 DEBUG:boto:Data: 2023-10-18T21:23:58.141 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.142 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.142 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.143 DEBUG:boto:Params: {} 2023-10-18T21:23:58.143 DEBUG:boto:Token: None 2023-10-18T21:23:58.144 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-72/obj3 2023-10-18T21:23:58.144 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:CmdoGHy1/7EUd9rXCg4Tng8frKs= 2023-10-18T21:23:58.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CmdoGHy1/7EUd9rXCg4Tng8frKs=', 'Content-Length': '0'} 2023-10-18T21:23:58.146 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f90279c5bf0b56ca-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.147 DEBUG:boto:path=/bofrnf-72/obj4 2023-10-18T21:23:58.147 DEBUG:boto:auth_path=/bofrnf-72/obj4 2023-10-18T21:23:58.148 DEBUG:boto:Method: HEAD 2023-10-18T21:23:58.148 DEBUG:boto:Path: /bofrnf-72/obj4 2023-10-18T21:23:58.148 DEBUG:boto:Data: 2023-10-18T21:23:58.149 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.149 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.149 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.150 DEBUG:boto:Params: {} 2023-10-18T21:23:58.150 DEBUG:boto:Token: None 2023-10-18T21:23:58.151 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-72/obj4 2023-10-18T21:23:58.151 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:jhhFnQiWed2XW4K32aAiuQwSemk= 2023-10-18T21:23:58.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jhhFnQiWed2XW4K32aAiuQwSemk=', 'Content-Length': '0'} 2023-10-18T21:23:58.153 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000433297b9d04513a4-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.154 DEBUG:boto:path=/bofrnf-73/ 2023-10-18T21:23:58.154 DEBUG:boto:auth_path=/bofrnf-73/ 2023-10-18T21:23:58.155 DEBUG:boto:Method: HEAD 2023-10-18T21:23:58.155 DEBUG:boto:Path: /bofrnf-73/ 2023-10-18T21:23:58.155 DEBUG:boto:Data: 2023-10-18T21:23:58.156 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.156 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.157 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.157 DEBUG:boto:Params: {} 2023-10-18T21:23:58.157 DEBUG:boto:Token: None 2023-10-18T21:23:58.158 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-73/ 2023-10-18T21:23:58.158 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:DDlach/lXAFB1SpYZbwx49fBsWA= 2023-10-18T21:23:58.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DDlach/lXAFB1SpYZbwx49fBsWA=', 'Content-Length': '0'} 2023-10-18T21:23:58.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9417b7d0f227af0-0065304cee-294e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.161 DEBUG:boto:path=/bofrnf-73/obj1 2023-10-18T21:23:58.162 DEBUG:boto:auth_path=/bofrnf-73/obj1 2023-10-18T21:23:58.162 DEBUG:boto:Method: HEAD 2023-10-18T21:23:58.163 DEBUG:boto:Path: /bofrnf-73/obj1 2023-10-18T21:23:58.163 DEBUG:boto:Data: 2023-10-18T21:23:58.164 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.164 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.164 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.165 DEBUG:boto:Params: {} 2023-10-18T21:23:58.165 DEBUG:boto:Token: None 2023-10-18T21:23:58.166 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-73/obj1 2023-10-18T21:23:58.166 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:ku4YHXBYUynJuaJ/krWnU8zN7tk= 2023-10-18T21:23:58.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ku4YHXBYUynJuaJ/krWnU8zN7tk=', 'Content-Length': '0'} 2023-10-18T21:23:58.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-72:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10007.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cab357aa5db6f731-0065304cee-294e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.169 DEBUG:boto:path=/bofrnf-73/obj1 2023-10-18T21:23:58.170 DEBUG:boto:auth_path=/bofrnf-73/obj1 2023-10-18T21:23:58.170 DEBUG:boto:Method: GET 2023-10-18T21:23:58.171 DEBUG:boto:Path: /bofrnf-73/obj1 2023-10-18T21:23:58.171 DEBUG:boto:Data: 2023-10-18T21:23:58.172 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.172 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.172 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.173 DEBUG:boto:Params: {} 2023-10-18T21:23:58.173 DEBUG:boto:Token: None 2023-10-18T21:23:58.174 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-73/obj1 2023-10-18T21:23:58.174 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:nrsWaT4GRvgt5Tvlf+U3F+DR2cU= 2023-10-18T21:23:58.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nrsWaT4GRvgt5Tvlf+U3F+DR2cU=', 'Content-Length': '0'} 2023-10-18T21:23:58.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-72:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10007.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003399749df10947e5-0065304cee-294e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.218 DEBUG:boto:path=/bofrnf-73/obj2 2023-10-18T21:23:58.219 DEBUG:boto:auth_path=/bofrnf-73/obj2 2023-10-18T21:23:58.219 DEBUG:boto:Method: HEAD 2023-10-18T21:23:58.220 DEBUG:boto:Path: /bofrnf-73/obj2 2023-10-18T21:23:58.220 DEBUG:boto:Data: 2023-10-18T21:23:58.220 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.221 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.221 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.222 DEBUG:boto:Params: {} 2023-10-18T21:23:58.222 DEBUG:boto:Token: None 2023-10-18T21:23:58.223 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-73/obj2 2023-10-18T21:23:58.223 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1CoaVFxANtno665i9A937VDJfVo= 2023-10-18T21:23:58.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1CoaVFxANtno665i9A937VDJfVo=', 'Content-Length': '0'} 2023-10-18T21:23:58.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-72:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10007.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e17146e02229f7f-0065304cee-294e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.226 DEBUG:boto:path=/bofrnf-73/obj2 2023-10-18T21:23:58.227 DEBUG:boto:auth_path=/bofrnf-73/obj2 2023-10-18T21:23:58.227 DEBUG:boto:Method: GET 2023-10-18T21:23:58.227 DEBUG:boto:Path: /bofrnf-73/obj2 2023-10-18T21:23:58.228 DEBUG:boto:Data: 2023-10-18T21:23:58.228 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.229 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.229 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.229 DEBUG:boto:Params: {} 2023-10-18T21:23:58.230 DEBUG:boto:Token: None 2023-10-18T21:23:58.230 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-73/obj2 2023-10-18T21:23:58.231 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MwNp4cYaPBbhHAgJZwIn1XIj8sA= 2023-10-18T21:23:58.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MwNp4cYaPBbhHAgJZwIn1XIj8sA=', 'Content-Length': '0'} 2023-10-18T21:23:58.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-72:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10007.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d28e6914bcf38789-0065304cee-294e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.235 DEBUG:boto:path=/bofrnf-73/obj3 2023-10-18T21:23:58.235 DEBUG:boto:auth_path=/bofrnf-73/obj3 2023-10-18T21:23:58.235 DEBUG:boto:Method: HEAD 2023-10-18T21:23:58.236 DEBUG:boto:Path: /bofrnf-73/obj3 2023-10-18T21:23:58.236 DEBUG:boto:Data: 2023-10-18T21:23:58.237 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.237 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:23:58.237 DEBUG:boto:Port: 8001 2023-10-18T21:23:58.238 DEBUG:boto:Params: {} 2023-10-18T21:23:58.238 DEBUG:boto:Token: None 2023-10-18T21:23:58.239 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-73/obj3 2023-10-18T21:23:58.239 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:o168U9TmEXs9dhXUwqxH5TBjGPQ= 2023-10-18T21:23:58.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o168U9TmEXs9dhXUwqxH5TBjGPQ=', 'Content-Length': '0'} 2023-10-18T21:23:58.241 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001b2da60537272b9e-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.244 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2023-10-18T21:23:58.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:58.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:58.246 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:58.296 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:58.297 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:58.328 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:23:58.341 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:23:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:23:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:53.281518Z" 2023-10-18T21:23:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:23:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:23:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:23:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:23:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:23:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:23:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.345 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:23:58.346 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:23:58.346 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:23:58.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:58.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:58.347 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:58.396 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:58.397 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:58.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:23:58.424+0000 7f3f14d7f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:23:58.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:23:58.425+0000 7f3f14d7f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:23:58.439 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:58.440 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:58.440 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:58.440 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:58.440 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:58.441 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:23:58.441 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.441 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:58.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.442 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:58.442 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.442 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:23:58.442 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:23:58.443 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.443 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:23:58.443 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.443 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:53.281518Z", 2023-10-18T21:23:58.443 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.444 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.444 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:58.445 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.445 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:23:58.445 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:23:58.445 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.445 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:58.445 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.446 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:23:58.446 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.446 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.447 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:58.447 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.447 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:23:58.447 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:23:58.448 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.448 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:58.448 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.448 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:23:58.448 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.449 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.449 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:58.449 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.450 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:23:58.450 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:23:58.450 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.450 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:58.450 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.451 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:23:58.451 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.451 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.451 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.451 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:58.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.452 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:58.452 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:23:58.452 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:23:58.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.453 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:58Z" 2023-10-18T21:23:58.453 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:58.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664113.281518_2033.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:21:53.281518Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:23:58Z" } 2023-10-18T21:23:58.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664113.281518_2033.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:53.281518Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:23:58.455 DEBUG:rgw_multi.tests:log_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:23:58.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:23:58.456 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:23:58.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:23:58.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:23:58.457 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:23:58.503 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:58.503 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:58.547 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:23:58.547 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-74 2023-10-18T21:23:58.548 DEBUG:boto:path=/bofrnf-74/ 2023-10-18T21:23:58.548 DEBUG:boto:auth_path=/bofrnf-74/ 2023-10-18T21:23:58.549 DEBUG:boto:Method: PUT 2023-10-18T21:23:58.549 DEBUG:boto:Path: /bofrnf-74/ 2023-10-18T21:23:58.549 DEBUG:boto:Data: 2023-10-18T21:23:58.550 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.550 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:23:58.551 DEBUG:boto:Port: 8000 2023-10-18T21:23:58.551 DEBUG:boto:Params: {} 2023-10-18T21:23:58.551 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:23:58.552 DEBUG:boto:Token: None 2023-10-18T21:23:58.552 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-74/ 2023-10-18T21:23:58.553 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:hYqJCvmmDgzqU2wofRJK4GKpS+I= 2023-10-18T21:23:58.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hYqJCvmmDgzqU2wofRJK4GKpS+I=', 'Content-Length': '0'} 2023-10-18T21:23:58.570 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006066e6e985e9e71f-0065304cee-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.571 INFO:rgw_multi.tests:create bucket zone=a1 name=bofrnf-75 2023-10-18T21:23:58.571 DEBUG:boto:path=/bofrnf-75/ 2023-10-18T21:23:58.571 DEBUG:boto:auth_path=/bofrnf-75/ 2023-10-18T21:23:58.572 DEBUG:boto:Method: PUT 2023-10-18T21:23:58.572 DEBUG:boto:Path: /bofrnf-75/ 2023-10-18T21:23:58.573 DEBUG:boto:Data: 2023-10-18T21:23:58.573 DEBUG:boto:Headers: {} 2023-10-18T21:23:58.573 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:23:58.574 DEBUG:boto:Port: 8000 2023-10-18T21:23:58.574 DEBUG:boto:Params: {} 2023-10-18T21:23:58.575 DEBUG:boto:Token: None 2023-10-18T21:23:58.575 DEBUG:boto:StringToSign: PUT Wed, 18 Oct 2023 21:23:58 GMT /bofrnf-75/ 2023-10-18T21:23:58.576 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:U2KJt//vBJNTedmtBmGuoetH4sI= 2023-10-18T21:23:58.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:U2KJt//vBJNTedmtBmGuoetH4sI=', 'Content-Length': '0'} 2023-10-18T21:23:58.591 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d4a1eb8c525a66b5-0065304cee-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:23:58.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:58.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:58.592 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:58.642 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:58.642 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:58.672 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:23:58.691 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:23:58.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.691 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:23:58.692 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:23:58.589509Z" 2023-10-18T21:23:58.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.692 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:23:58.692 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:23:58.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.693 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:23:58.693 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:23:58.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.694 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:23:58.694 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:23:58.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.695 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:23:58.695 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:23:58.695 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:23:58.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:58.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:23:58.697 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:23:58.748 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:23:58.748 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:23:58.776 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:23:58.774+0000 7fd193a63780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:23:58.778 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:23:58.776+0000 7fd193a63780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:23:58.790 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:23:58.791 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:23:58.791 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:23:58.791 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:23:58.791 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:23:58.792 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:23:58.792 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.792 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:23:58.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.793 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:23:58.793 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.793 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:23:58.793 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.562824_2075.1", 2023-10-18T21:23:58.793 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.794 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:23:58.794 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.794 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.562824Z", 2023-10-18T21:23:58.794 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.794 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.795 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:23:58.795 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.796 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:23:58.796 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:23:58.796 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.796 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:58.797 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.797 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:23:58.797 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.797 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.798 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:23:58.798 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.798 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:23:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:23:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.799 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:23:58.800 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.800 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:23:58.801 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:23:58.801 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:23:58.801 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:23:58.801 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:23:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:23:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:23:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:23:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:23:58.802 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:23:58.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:23:58.803 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:23:58.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:23:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:23:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:23:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:23:58.804 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:23:58Z" 2023-10-18T21:23:58.805 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:23:58.805 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.562824_2075.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.562824Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:23:58Z" } 2023-10-18T21:23:58.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.562824_2075.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.562824Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:23:58.806 DEBUG:rgw_multi.tests:log_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:23:58.807 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.562824_2075.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:23:58.807 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664238.589509_2088.1 target=1_1697664238.562824_2075.1 2023-10-18T21:24:03.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:03.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:03.810 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:03.867 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:03.867 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:03.899 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:24:03.897+0000 7f9f2da65780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:24:03.900 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:24:03.898+0000 7f9f2da65780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:24:03.914 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:03.914 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:03.914 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:03.915 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:03.915 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:03.915 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:24:03.915 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:03.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:03.916 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:03.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:03.916 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:03.917 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:03.917 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:24:03.917 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:24:03.917 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:03.918 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:24:03.918 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:03.918 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:24:03.918 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:03.919 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:03.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:03.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:03.919 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:03.920 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:03.920 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:24:03.920 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:24:03.920 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:03.921 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:03.921 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:03.921 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:24:03.921 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:03.922 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:03.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:03.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:03.922 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:03.923 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:03.923 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:24:03.923 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:24:03.923 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:03.924 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:03.924 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:03.924 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:24:03.924 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:03.925 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:03.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:03.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:03.925 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:03.926 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:03.926 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:24:03.926 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:24:03.926 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:03.927 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:03.927 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:03.927 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:24:03.927 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:03.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:03.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:03.928 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:03.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:03.929 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:03.929 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:24:03.929 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:24:03.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:03.930 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:03Z" 2023-10-18T21:24:03.930 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:03.931 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:24:03Z" } 2023-10-18T21:24:03.932 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:24:03.932 DEBUG:rgw_multi.tests:log_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:24:03.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:24:03.933 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:24:03.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:03.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:03.935 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:24:03.985 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:03.986 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:04.030 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:24:04.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.032 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:24:04.085 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:04.085 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:04.135 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:24:04.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-74', '--dest-bucket=bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-74', '--dest-bucket=bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.137 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-bofrnf-74 --source-zones=a1 --dest-zones=a2 --source-bucket=bofrnf-74 --dest-bucket=bofrnf-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:24:04.191 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:04.192 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:04.236 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:24:04.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-74', '--dest-bucket=bofrnf-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-74', '--dest-bucket=bofrnf-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.238 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-bofrnf-75 --source-zones=a1 --dest-zones=a2 --source-bucket=bofrnf-74 --dest-bucket=bofrnf-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:24:04.293 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:04.294 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:04.339 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:24:04.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-74', '--dest-bucket=bofrnf-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-74', '--dest-bucket=bofrnf-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.341 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=bofrnf-74 --dest-bucket=bofrnf-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:24:04.395 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:04.396 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:04.442 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:24:04.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.444 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:24:04.494 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:04.494 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:04.539 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:24:04.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.541 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:24:04.590 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:04.590 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:04.641 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:04.642 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:24:04.643 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 21, 2023-10-18T21:24:04.643 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:24:04.643 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:24:04.643 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:24:04.644 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:24:04.644 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:24:04.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.644 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:24:04.645 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:24:04.645 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:24:04.645 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:24:04.645 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:24:04.646 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:24:04.646 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.646 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:24:04.646 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:24:04.647 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:24:04.647 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.647 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:24:04.647 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:24:04.648 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:24:04.648 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:24:04.648 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.648 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:24:04.648 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:24:04.649 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:24:04.649 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:24:04.649 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:24:04.649 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:24:04.650 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:24:04.650 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:24:04.650 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:24:04.650 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:24:04.650 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:24:04.651 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.651 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:24:04.651 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:24:04.652 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:24:04.652 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:24:04.652 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.652 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:24:04.652 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:24:04.653 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:24:04.653 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:24:04.653 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:24:04.653 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:24:04.654 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:24:04.654 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:24:04.654 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:24:04.654 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:24:04.654 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:24:04.655 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.655 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.655 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.655 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:24:04.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.656 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:24:04.656 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:24:04.656 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:24:04.656 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:24:04.656 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.657 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.657 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.657 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:24:04.657 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:24:04.658 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:24:04.658 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [ 2023-10-18T21:24:04.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.658 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-group", 2023-10-18T21:24:04.658 INFO:teuthology.orchestra.run.smithi082.stdout: "data_flow": { 2023-10-18T21:24:04.659 INFO:teuthology.orchestra.run.smithi082.stdout: "symmetrical": [ 2023-10-18T21:24:04.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.659 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-flow1", 2023-10-18T21:24:04.659 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.660 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:24:04.660 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:24:04.660 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.660 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.661 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.661 INFO:teuthology.orchestra.run.smithi082.stdout: "directional": [ 2023-10-18T21:24:04.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.661 INFO:teuthology.orchestra.run.smithi082.stdout: "source_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:24:04.662 INFO:teuthology.orchestra.run.smithi082.stdout: "dest_zone": "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:24:04.662 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.662 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.662 INFO:teuthology.orchestra.run.smithi082.stdout: "pipes": [ 2023-10-18T21:24:04.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.663 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-72", 2023-10-18T21:24:04.663 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.664 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-72", 2023-10-18T21:24:04.664 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.664 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:24:04.664 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.665 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:24:04.665 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-73", 2023-10-18T21:24:04.665 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.665 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:24:04.666 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.666 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:24:04.666 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.667 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:24:04.667 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:24:04.667 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.668 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:24:04.668 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:24:04.668 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:24:04.668 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:24:04.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.669 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-73", 2023-10-18T21:24:04.670 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.670 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-73", 2023-10-18T21:24:04.670 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.671 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:24:04.671 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.671 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:24:04.672 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-73", 2023-10-18T21:24:04.672 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.672 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:24:04.672 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.673 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:24:04.673 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.673 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:24:04.673 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:24:04.674 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.674 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:24:04.674 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:24:04.675 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:24:04.675 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:24:04.675 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.676 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-74", 2023-10-18T21:24:04.676 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.676 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-74", 2023-10-18T21:24:04.677 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.677 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:24:04.677 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.677 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:24:04.678 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-74", 2023-10-18T21:24:04.678 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.678 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:24:04.678 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.679 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:24:04.679 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.679 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:24:04.680 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:24:04.680 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.680 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:24:04.681 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:24:04.681 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:24:04.681 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:24:04.681 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.682 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-75", 2023-10-18T21:24:04.682 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.682 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-74", 2023-10-18T21:24:04.683 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.683 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:24:04.683 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.683 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:24:04.684 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-75", 2023-10-18T21:24:04.684 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.684 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:24:04.684 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.685 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:24:04.685 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.685 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:24:04.685 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:24:04.685 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.686 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:24:04.686 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:24:04.686 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:24:04.686 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:24:04.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.687 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe", 2023-10-18T21:24:04.687 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-74", 2023-10-18T21:24:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:24:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:24:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-75", 2023-10-18T21:24:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.689 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:24:04.690 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.690 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:24:04.690 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:24:04.690 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:24:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:24:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:24:04.691 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:24:04.692 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:24:04.692 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:24:04.692 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.692 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.692 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.693 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "enabled" 2023-10-18T21:24:04.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.693 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.694 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:24:04.694 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:24:04.694 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.695 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:24:04.695 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:24:04.695 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:24:04.695 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:24:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:24:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:24:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:24:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:24:04.697 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:24:04.697 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:24:04.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.697 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:24:04.697 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:24:04.698 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:24:04.698 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:24:04.698 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.698 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:24:04.699 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:24:04.699 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:24:04.699 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:24:04.699 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:24:04.700 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:24:04.700 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:24:04.700 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:24:04.700 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:24:04.701 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:24:04.701 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:24:04.701 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.702 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:24:04.702 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:24:04.702 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:24:04.703 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:24:04.704 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.704 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:24:04.704 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:24:04.705 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:24:04.705 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:24:04.705 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:24:04.706 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:24:04.706 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:24:04.706 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:24:04.706 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:24:04.707 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:24:04.707 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:24:04.707 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.708 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.708 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:24:04.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.708 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:24:04.709 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:24:04.709 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:24:04.709 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:24:04.709 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.710 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.710 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:24:04.711 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:24:04.711 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:24:04.711 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:24:04.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.711 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:24:04.712 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:24:04.712 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.712 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.712 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:24:04.712 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:24:04.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.713 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:24:04.713 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:24:04.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.714 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:24:04.714 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:24:04.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.715 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:24:04.715 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:24:04.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:04.716 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:24:04.716 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:24:04.716 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.716 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:04.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.717 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:24:04.717 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:24:04.717 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:24:04.717 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:24:04.718 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:24:04.718 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:24:04.718 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:24:04.718 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:24:04.719 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:24:04.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.719 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:24:04.719 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:24:04.719 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:24:04.720 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:24:04.720 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:24:04.720 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:24:04.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.720 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:24:04.721 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:24:04.721 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:24:04.721 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:24:04.721 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:24:04.722 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:24:04.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.722 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:24:04.722 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:24:04.722 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:24:04.723 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:24:04.723 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:24:04.723 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:24:04.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.723 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:24:04.724 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:24:04.724 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:24:04.724 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:24:04.724 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:24:04.724 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:24:04.725 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:04.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:04.725 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:24:04.725 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:24:04.726 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:04.726 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:04.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:24:04.727 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:24:04.785 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:04.785 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:04.929 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:24:04.930 DEBUG:boto:path=/bofrnf-74/ 2023-10-18T21:24:04.930 DEBUG:boto:auth_path=/bofrnf-74/ 2023-10-18T21:24:04.931 DEBUG:boto:Method: HEAD 2023-10-18T21:24:04.932 DEBUG:boto:Path: /bofrnf-74/ 2023-10-18T21:24:04.932 DEBUG:boto:Data: 2023-10-18T21:24:04.932 DEBUG:boto:Headers: {} 2023-10-18T21:24:04.933 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:24:04.933 DEBUG:boto:Port: 8000 2023-10-18T21:24:04.934 DEBUG:boto:Params: {} 2023-10-18T21:24:04.935 DEBUG:boto:Token: None 2023-10-18T21:24:04.935 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:24:04 GMT /bofrnf-74/ 2023-10-18T21:24:04.936 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lr0876sHFwIeI9q4niDeOXsd+L0= 2023-10-18T21:24:04.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:24:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lr0876sHFwIeI9q4niDeOXsd+L0=', 'Content-Length': '0'} 2023-10-18T21:24:04.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b1f07d4d7faf219-0065304cf4-2a99-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:24:04.947 DEBUG:boto:path=/bofrnf-74/obj1 2023-10-18T21:24:04.948 DEBUG:boto:auth_path=/bofrnf-74/obj1 2023-10-18T21:24:04.949 DEBUG:boto:Method: PUT 2023-10-18T21:24:04.949 DEBUG:boto:Path: /bofrnf-74/obj1 2023-10-18T21:24:04.949 DEBUG:boto:Data: 2023-10-18T21:24:04.950 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:24:04.951 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:24:04.951 DEBUG:boto:Port: 8000 2023-10-18T21:24:04.952 DEBUG:boto:Params: {} 2023-10-18T21:24:04.952 DEBUG:boto:Token: None 2023-10-18T21:24:04.953 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:24:04 GMT /bofrnf-74/obj1 2023-10-18T21:24:04.953 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:MbSfvSREbf95ofhEZAAEun7Bw98= 2023-10-18T21:24:04.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:24:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MbSfvSREbf95ofhEZAAEun7Bw98='} 2023-10-18T21:24:04.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000850f1ec3331f0164-0065304cf4-2a99-a1'), ('Date', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:24:04.975 DEBUG:boto:path=/bofrnf-74/ 2023-10-18T21:24:04.976 DEBUG:boto:auth_path=/bofrnf-74/ 2023-10-18T21:24:04.976 DEBUG:boto:Method: HEAD 2023-10-18T21:24:04.976 DEBUG:boto:Path: /bofrnf-74/ 2023-10-18T21:24:04.977 DEBUG:boto:Data: 2023-10-18T21:24:04.977 DEBUG:boto:Headers: {} 2023-10-18T21:24:04.977 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:24:04.978 DEBUG:boto:Port: 8000 2023-10-18T21:24:04.978 DEBUG:boto:Params: {} 2023-10-18T21:24:04.979 DEBUG:boto:Token: None 2023-10-18T21:24:04.979 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:24:04 GMT /bofrnf-74/ 2023-10-18T21:24:04.980 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lr0876sHFwIeI9q4niDeOXsd+L0= 2023-10-18T21:24:04.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:24:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lr0876sHFwIeI9q4niDeOXsd+L0=', 'Content-Length': '0'} 2023-10-18T21:24:04.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005563ad72b0935a6d-0065304cf4-2a99-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:24:04.985 DEBUG:boto:path=/bofrnf-74/obj2 2023-10-18T21:24:04.985 DEBUG:boto:auth_path=/bofrnf-74/obj2 2023-10-18T21:24:04.985 DEBUG:boto:Method: PUT 2023-10-18T21:24:04.986 DEBUG:boto:Path: /bofrnf-74/obj2 2023-10-18T21:24:04.986 DEBUG:boto:Data: 2023-10-18T21:24:04.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:24:04.987 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:24:04.988 DEBUG:boto:Port: 8000 2023-10-18T21:24:04.988 DEBUG:boto:Params: {} 2023-10-18T21:24:04.989 DEBUG:boto:Token: None 2023-10-18T21:24:04.989 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:24:04 GMT /bofrnf-74/obj2 2023-10-18T21:24:04.990 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:xBh4SHZVSKizHo/TzPLuC0rCA8A= 2023-10-18T21:24:04.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:24:04 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xBh4SHZVSKizHo/TzPLuC0rCA8A='} 2023-10-18T21:24:05.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082894aadfce8f1bc-0065304cf4-2a99-a1'), ('Date', 'Wed, 18 Oct 2023 21:24:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:24:05.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:05.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:05.036 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:05.111 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:05.112 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:05.200 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:24:05.232 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:24:05.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.233 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:24:05.234 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:23:58.589509Z" 2023-10-18T21:24:05.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.234 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:24:05.234 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:24:05.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.235 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:24:05.236 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:24:05.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.236 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:24:05.237 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:24:05.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.237 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:24:05.238 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:24:05.239 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:24:05.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:05.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:05.240 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:05.308 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:05.308 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:05.349 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:24:05.347+0000 7f9c20397780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:24:05.351 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:24:05.349+0000 7f9c20397780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:24:05.370 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:24:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:24:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:24:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:24:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:24:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:24:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:24:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:24:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:24:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:24:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:24:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:24:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:24:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:24:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:24:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:24:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:24:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:05Z" 2023-10-18T21:24:05.384 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:05.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:24:05Z" } 2023-10-18T21:24:05.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:24:05.386 DEBUG:rgw_multi.tests:log_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:24:05.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:24:05.387 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:24:05.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:05.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:05.388 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:05.440 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:05.441 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:05.491 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:24:05.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.491 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:05.492 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:05.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.492 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-10-18T21:24:05.493 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:24:05.002600Z" 2023-10-18T21:24:05.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.493 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:05.494 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:05.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.495 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:24:05.495 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:24:05.495 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.495 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:24:05.497 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:05.497 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:24:05.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:05.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:05.499 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:05.551 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:05.552 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:05.595 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:05.596 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:05.596 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:05.596 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:05.596 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:05.597 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:05.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.597 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:05.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.597 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:05.598 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:05.598 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:05.598 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:05.598 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:05.599 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:05.599 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:05.599 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:05.599 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.600 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:05.600 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:05.600 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:05.601 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:05.601 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:05.601 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:05.601 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:05.601 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:05.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.602 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:05.603 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:05.604 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:05.604 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:05.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:05.605 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:05.606 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:05.606 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:05.606 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:05.606 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:05.607 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:05.608 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:05.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:05.608 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:05Z" 2023-10-18T21:24:05.608 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:05.609 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:05Z" } 2023-10-18T21:24:05.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:05.610 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:05.611 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:05.611 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:10.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:10.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:10.614 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:10.676 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:10.677 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:10.725 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:10Z" 2023-10-18T21:24:10.737 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:10.738 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:10Z" } 2023-10-18T21:24:10.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:10.739 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:10.739 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:10.740 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:15.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:15.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:15.745 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:15.803 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:15.803 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:15.855 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:15.856 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:15.856 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:15.856 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:15.856 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:15.857 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:15.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:15.857 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:15.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:15.858 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:15.858 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:15.858 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:15.858 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:15.859 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:15.859 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:15.859 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:15.859 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:15.860 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:15.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:15.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:15.860 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:15.861 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:15.861 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:15.861 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:15.861 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:15.861 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:15.862 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:15.862 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:15.862 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:15.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:15.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:15.863 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:15.863 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:15.863 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:15.863 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:15.864 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:15.864 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:15.864 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:15.864 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:15.865 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:15.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:15.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:15.865 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:15.866 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:15.866 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:15.866 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:15.866 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:15.866 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:15.867 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:15.867 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:15.867 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:15.867 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:15.868 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:15.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:15.868 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:15.868 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:15.869 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:15.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:15.869 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:15Z" 2023-10-18T21:24:15.869 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:15.870 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:15Z" } 2023-10-18T21:24:15.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:15.871 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:15.872 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:15.872 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:20.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:20.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:20.874 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:20.933 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:20.933 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:20.982 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:20.983 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:20.983 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:20.983 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:20.984 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:20.984 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:20.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:20.984 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:20.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:20.985 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:20.985 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:20.985 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:20.985 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:20.985 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:20.986 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:20.986 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:20.986 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:20.986 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:20.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:20.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:20.987 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:20.987 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:20.987 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:20.987 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:20.988 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:20.988 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:20.988 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:20.988 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:20.988 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:20.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:20.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:20.989 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:20.989 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:20.989 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:20.990 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:20.990 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:20.990 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:20.990 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:20.991 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:20.991 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:20.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:20.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:20.991 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:20.992 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:20.992 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:20.992 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:20.992 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:20.992 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:20.993 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:20.993 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:20.993 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:20.993 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:20.993 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:20.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:20.994 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:20.994 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:20.994 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:20.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:20.995 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:20Z" 2023-10-18T21:24:20.995 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:20.996 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:20Z" } 2023-10-18T21:24:20.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:20.997 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:20.997 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:20.998 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:25.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:26.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:26.000 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:26.059 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:26.060 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:26.112 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:26.112 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:26.112 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:26.113 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:26.113 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:26.113 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:26.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:26.113 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:26.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:26.114 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:26.114 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:26.114 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:26.115 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:26.115 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:26.115 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:26.115 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:26.115 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:26.116 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:26.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:26.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:26.116 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:26.117 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:26.117 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:26.117 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:26.117 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:26.117 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:26.118 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:26.118 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:26.118 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:26.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:26.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:26.119 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:26.119 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:26.119 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:26.119 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:26.119 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:26.120 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:26.120 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:26.120 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:26.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:26.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:26.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:26.121 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:26.121 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:26.121 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:26.121 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:26.122 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:26.122 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:26.122 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:26.122 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:26.122 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:26.123 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:26.123 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:26.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:26.123 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:26.123 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:26.124 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:26.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:26.124 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:26Z" 2023-10-18T21:24:26.124 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:26.125 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:26Z" } 2023-10-18T21:24:26.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:26.126 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:26.127 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:26.127 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:31.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:31.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:31.130 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:31.188 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:31.189 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:31.237 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:31.238 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:31.238 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:31.238 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:31.238 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:31.239 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:31.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:31.239 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:31.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:31.239 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:31.240 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:31.240 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:31.240 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:31.240 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:31.241 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:31.241 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:31.241 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:31.241 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:31.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:31.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:31.242 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:31.242 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:31.242 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:31.243 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:31.243 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:31.243 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:31.243 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:31.244 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:31.244 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:31.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:31.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:31.244 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:31.245 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:31.245 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:31.245 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:31.245 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:31.246 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:31.246 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:31.246 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:31.246 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:31.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:31.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:31.247 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:31.247 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:31.247 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:31.248 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:31.248 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:31.248 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:31.248 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:31.248 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:31.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:31.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:31.249 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:31.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:31.249 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:31.250 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:31.250 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:31.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:31.250 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:31Z" 2023-10-18T21:24:31.251 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:31.251 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:31Z" } 2023-10-18T21:24:31.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:31.252 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:31.253 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:31.253 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:36.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:36.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:36.255 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:36.313 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:36.314 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:36.362 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:36.363 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:36.363 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:36.363 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:36.363 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:36.364 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:36.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:36.364 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:36.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:36.364 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:36.365 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:36.365 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:36.365 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:36.365 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:36.365 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:36.366 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:36.366 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:36.366 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:36.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:36.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:36.367 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:36.367 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:36.367 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:36.367 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:36.368 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:36.368 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:36.368 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:36.368 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:36.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:36.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:36.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:36.369 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:36.369 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:36.369 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:36.370 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:36.370 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:36.370 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:36.370 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:36.370 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:36.371 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:36.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:36.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:36.371 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:36.371 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:36.372 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:36.372 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:36.372 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:36.372 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:36.372 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:36.373 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:36.373 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:36.373 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:36.373 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:36.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:36.374 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:36.374 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:36.374 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:36.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:36.375 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:36Z" 2023-10-18T21:24:36.375 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:36.376 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:36Z" } 2023-10-18T21:24:36.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:36.377 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:36.377 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:36.378 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:41.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:41.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:41.380 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:41.439 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:41.439 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:41.488 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:41Z" 2023-10-18T21:24:41.500 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:41.501 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:41Z" } 2023-10-18T21:24:41.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:41.502 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:41.502 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:41.503 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:46.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:46.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:46.505 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:46.564 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:46.565 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:46.614 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:46.614 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:46.615 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:46.615 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:46.615 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:46.615 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:46.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:46.616 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:46.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:46.616 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:46.616 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:46.617 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:46.617 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:46.617 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:46.617 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:46.617 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:46.618 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:46.618 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:46.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:46.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:46.619 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:46.619 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:46.619 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:46.619 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:46.619 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:46.620 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:46.620 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:46.620 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:46.620 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:46.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:46.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:46.621 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:46.621 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:46.621 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:46.621 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:46.622 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:46.622 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:46.622 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:46.622 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:46.622 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:46.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:46.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:46.623 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:46.623 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:46.624 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:46.624 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:46.624 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:46.624 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:46.624 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:46.625 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:46.625 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:46.625 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:46.625 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:46.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:46.626 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:46.626 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:46.626 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:46.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:46.626 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:46Z" 2023-10-18T21:24:46.627 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:46.627 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:46Z" } 2023-10-18T21:24:46.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:46.628 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:46.629 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:46.629 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:51.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:51.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:51.631 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:51.690 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:51.691 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:51.738 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:51.739 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:51.739 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:51.739 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:51.739 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:51.740 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:51.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:51.740 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:51.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:51.741 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:51.741 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:51.741 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:51.741 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:51.742 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:51.742 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:51.742 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:51.742 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:51.742 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:51.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:51.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:51.743 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:51.743 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:51.743 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:51.744 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:51.744 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:51.744 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:51.744 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:51.744 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:51.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:51.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:51.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:51.745 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:51.746 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:51.746 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:51.746 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:51.746 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:51.746 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:51.747 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:51.747 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:51.747 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:51.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:51.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:51.748 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:51.748 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:51.748 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:51.748 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:51.748 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:51.748 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:51.749 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:51.749 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:51.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:51.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:51.749 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:51.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:51.750 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:51.750 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:51.750 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:51.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:51.751 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:51Z" 2023-10-18T21:24:51.751 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:51.752 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:51Z" } 2023-10-18T21:24:51.752 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:51.753 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:51.753 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:51.753 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:24:56.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:56.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:24:56.756 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:24:56.815 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:24:56.815 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:24:56.865 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:24:56.865 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:24:56.865 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:24:56.865 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:24:56.866 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:24:56.866 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:24:56.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:56.866 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:24:56.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:56.867 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:24:56.867 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:56.867 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:56.867 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:24:56.868 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:56.868 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:56.868 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:56.868 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:24:56.869 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:56.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:56.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:56.869 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:24:56.869 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:24:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:24:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:24:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:24:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:24:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:24:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:24:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:24:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:24:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:24:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:24:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:24:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:24:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:24:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:24:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:24:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:24:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:24:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:24:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:24:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:24:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:24:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:24:56Z" 2023-10-18T21:24:56.878 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:24:56.879 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:24:56Z" } 2023-10-18T21:24:56.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:24:56.880 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:24:56.880 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:24:56.880 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:25:01.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:01.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:01.882 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:01.942 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:01.943 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:01.990 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:01.991 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:01.991 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:01.991 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:01.991 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:01.992 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:01.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:01.992 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:01.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:01.993 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:01.993 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:01.993 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:01.993 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:01.993 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:01.994 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:01.994 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:01.994 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:01.994 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:01.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:01.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:01.995 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:01.995 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:01.995 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:01.995 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:01.996 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:01.996 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:01.996 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:01.996 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:01.996 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:01.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:01.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:01.997 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:01.997 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:01.997 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:01.998 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:01.998 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:01.998 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:01.998 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:01.998 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:01.999 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:01.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:01.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:01.999 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:02.000 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:02.000 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:02.000 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:25:02.000 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:02.000 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:25:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:02.001 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:02.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:02.002 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:02.002 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:02.002 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:02.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:02.003 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:01Z" 2023-10-18T21:25:02.004 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:02.005 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:01Z" } 2023-10-18T21:25:02.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:25:02.006 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:02.006 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:25:02.007 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:25:07.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:07.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:07.010 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:07.068 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:07.069 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:07.116 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:07.116 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:07.117 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:07.117 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:07.117 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:07.117 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:07.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:07.118 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:07.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:07.118 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:07.118 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:07.119 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:07.119 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:07.119 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:07.119 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:07.120 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:07.120 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:07.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:07.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:07.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:07.120 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:07.121 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:07.121 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:07.121 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:07.121 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:07.121 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:07.122 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:07.122 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:07.122 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:07.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:07.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:07.123 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:07.123 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:07.123 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:07.123 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:07.123 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:07.124 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:07.124 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:07.124 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:07.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:07.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:07.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:07.125 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:07.125 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:07.125 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:07.125 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:25:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:25:07.126 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:07.127 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:07.128 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:07.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:07.128 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:07Z" 2023-10-18T21:25:07.128 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:07.129 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:07Z" } 2023-10-18T21:25:07.130 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:25:07.131 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:07.131 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:25:07.132 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:25:12.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:12.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:12.134 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:12.193 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:12.194 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:12.242 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:12.242 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:12.242 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:12.243 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:12.243 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:12.243 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:12.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:12.244 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:12.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:12.244 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:12.244 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:12.244 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:12.245 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:12.245 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:12.245 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:12.245 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:12.245 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:12.245 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:12.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:12.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:12.246 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:12.246 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:12.247 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:12.247 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:12.247 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:12.247 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:12.247 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:12.248 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:12.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:12.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:12.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:12.248 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:12.249 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:12.249 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:12.249 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:12.249 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:12.249 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:12.250 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:12.250 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:12.250 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:12.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:12.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:12.251 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:12.251 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:12.251 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:12.251 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:25:12.252 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:12.252 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:12.252 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:12.252 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:25:12.252 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:12.252 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:12.253 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:12.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:12.253 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:12.253 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:12.254 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:12.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:12.254 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:12Z" 2023-10-18T21:25:12.254 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:12.255 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:12Z" } 2023-10-18T21:25:12.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:25:12.256 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:12.256 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:25:12.257 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:25:17.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:17.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:17.259 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:17.318 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:17.318 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:17.367 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:17.367 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:17.367 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:17.368 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:17.368 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:17.368 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:17.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:17.368 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:17.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:17.369 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:17.369 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:17.369 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:17.369 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:17.370 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:17.370 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:17.370 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:17.370 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:17.371 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:17.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:17.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:17.371 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:17.371 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:17.372 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:17.372 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:17.372 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:17.372 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:17.372 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:17.373 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:17.373 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:17.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:17.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:17.374 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:17.374 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:17.374 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:17.374 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:17.374 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:17.375 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:17.375 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:17.375 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:17.375 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:17.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:17.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:17.376 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:17.376 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:17.376 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:17.376 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:25:17.376 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:17.377 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:17.377 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:17.377 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:25:17.377 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:17.377 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:17.378 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:17.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:17.378 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:17.378 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:17.378 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:17.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:17.379 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:17Z" 2023-10-18T21:25:17.379 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:17.380 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:17Z" } 2023-10-18T21:25:17.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:25:17.381 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:17.381 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:25:17.381 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:25:22.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:22.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:22.384 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:22.443 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:22.444 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:22.492 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:22.492 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:22.492 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:22.493 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:22.493 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:22.493 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:22.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:22.493 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:22.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:22.494 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:22.494 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:22.494 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:22.494 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:22.495 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:22.495 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:22.495 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:22.495 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:22.495 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:22.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:22.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:22.496 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:22.496 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:22.496 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:22.497 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:22.497 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:22.497 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:22.497 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:22.497 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:22.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:22.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:22.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:22.498 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:22.498 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:22.499 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:22.499 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:22.499 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:22.499 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:22.499 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:22.500 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:22.500 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:22.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:22.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:22.500 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:22.501 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:22.501 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:22.501 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:25:22.501 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:22.501 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:22.502 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:22.502 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:25:22.502 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:22.502 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:22.502 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:22.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:22.503 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:22.503 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:22.503 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:22.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:22.504 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:22Z" 2023-10-18T21:25:22.504 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:22.505 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:22Z" } 2023-10-18T21:25:22.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:25:22.506 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:22.506 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:25:22.507 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:25:27.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:27.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:27.509 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:27.568 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:27.568 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:27.616 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:25:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:25:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:27Z" 2023-10-18T21:25:27.632 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:27.634 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:27Z" } 2023-10-18T21:25:27.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:25:27.635 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:27.635 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:25:27.635 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:25:32.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:32.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:32.638 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:32.698 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:32.699 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:32.746 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:32.747 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:32.747 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:32.747 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:32.747 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:32.748 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:32.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:32.748 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:32.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:32.749 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:32.749 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:32.749 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:32.749 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:32.749 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:32.750 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:32.750 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:32.750 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:32.750 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:32.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:32.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:32.751 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:32.751 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:32.751 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:32.752 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:32.752 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:32.752 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:32.752 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:32.752 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:32.753 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:32.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:32.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:32.753 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:32.753 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:32.754 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:32.754 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:32.754 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:32.754 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:32.754 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:32.755 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:32.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:32.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:32.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:32.755 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:32.756 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:32.756 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:32.756 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:25:32.756 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:32.756 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:32.757 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:32.757 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:25:32.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:32.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:32.757 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:32.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:32.758 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:32.758 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:32.759 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:32.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:32.759 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:32Z" 2023-10-18T21:25:32.759 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:32.760 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052094", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:21:54.419859Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:32Z" } 2023-10-18T21:25:32.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:25:32.761 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:32.761 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:25:32.762 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939, shard=3 master=00000000000000000000:00000000000000052235 target=00000000000000000000:00000000000000052094 2023-10-18T21:25:35.933 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T21:25:35.932+0000 7fc3c73cb700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-18T21:25:37.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:37.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:37.764 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:37.822 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:37.823 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:37.874 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:37.874 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:37.874 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:37.875 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:37.875 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:37.875 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:37.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:37.875 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:37.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:37.876 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:37.876 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:37.876 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:37.877 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:37.877 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:37.877 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:37.877 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:37.877 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:37.878 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:37.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:37.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:37.878 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:37.878 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:37.879 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:37.879 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:37.879 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:37.879 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:37.879 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:37.880 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:37.880 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:37.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:37.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:37.880 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:37.881 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:37.881 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:37.881 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:37.881 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:37.882 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:37.882 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:37.882 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:37.882 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:37.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:37.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:37.883 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:37.883 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:37.883 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:37.883 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:25:37.884 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:37.884 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:37.884 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:37.884 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:25:37.884 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:37.885 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:37.885 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:37.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:37.885 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:37.885 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:37.886 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:37.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:37.886 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:37Z" 2023-10-18T21:25:37.886 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:37.887 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052235", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:24:04.972607Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:37Z" } 2023-10-18T21:25:37.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052235', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:24:04.972607Z'}}] 2023-10-18T21:25:37.888 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:37.888 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:37.889 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939 2023-10-18T21:25:42.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:42.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:42.891 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:42.950 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:42.950 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:42.998 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:25:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:25:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:42Z" 2023-10-18T21:25:43.011 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:43.012 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052235", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:24:04.972607Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:42Z" } 2023-10-18T21:25:43.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052235', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:24:04.972607Z'}}] 2023-10-18T21:25:43.013 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:43.013 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:43.014 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939 2023-10-18T21:25:48.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:48.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:48.016 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:48.076 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:48.076 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:48.127 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:48.127 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:48.128 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:48.128 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:48.128 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:48.128 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:48.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:48.129 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:48.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:48.129 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:48.129 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:48.130 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:48.130 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:48.130 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:48.130 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:48.130 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:48.131 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:48.131 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:48.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:48.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:48.131 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:48.132 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:48.132 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:48.132 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:48.132 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:48.132 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:48.133 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:48.133 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:48.133 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:48.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:48.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:48.133 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:48.134 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:48.134 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:48.134 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:48.134 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:48.135 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:48.135 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:48.135 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:48.135 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:48.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:48.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:48.136 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:48.136 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:48.136 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:48.136 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:25:48.136 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:48.137 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:48.137 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:48.137 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:25:48.137 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:48.137 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:48.137 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:48.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:48.138 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:48.138 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:48.138 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:48.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:48.139 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:48Z" 2023-10-18T21:25:48.139 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:48.140 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052235", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:24:04.972607Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:48Z" } 2023-10-18T21:25:48.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052235', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:24:04.972607Z'}}] 2023-10-18T21:25:48.141 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:48.141 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:48.141 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939 2023-10-18T21:25:53.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:53.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:53.144 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:53.203 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:53.203 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:53.251 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:53.251 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:53.251 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:53.251 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:53.252 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:53.252 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:53.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:53.252 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:53.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:53.253 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:53.253 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:53.253 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:53.253 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:53.253 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:53.254 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:53.254 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:53.254 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:53.254 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:53.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:53.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:53.255 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:53.255 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:53.255 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:53.255 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:53.256 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:53.256 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:53.256 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:53.257 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:53.257 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:53.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:53.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:53.258 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:53.258 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:53.258 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:53.258 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:53.258 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:53.259 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:53.259 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:53.259 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:53.259 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:53.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:53.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:53.260 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:53.260 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:53.260 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:53.261 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:25:53.261 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:53.261 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:53.261 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:53.261 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:25:53.262 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:53.262 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:53.262 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:53.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:53.262 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:53.263 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:53.263 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:53.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:53.263 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:53Z" 2023-10-18T21:25:53.264 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:53.264 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052235", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:24:04.972607Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:53Z" } 2023-10-18T21:25:53.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052235', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:24:04.972607Z'}}] 2023-10-18T21:25:53.265 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:53.266 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:53.266 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939 2023-10-18T21:25:58.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:58.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:25:58.269 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:25:58.328 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:25:58.329 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:25:58Z" 2023-10-18T21:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:25:58.390 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052235", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:24:04.972607Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:25:58Z" } 2023-10-18T21:25:58.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052235', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:24:04.972607Z'}}] 2023-10-18T21:25:58.391 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:58.391 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:25:58.392 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939 2023-10-18T21:26:03.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:03.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:03.394 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:03.453 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:03.453 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:03.506 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:03.506 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:03.506 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:03.506 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:03.507 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:03.507 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:26:03.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:03.507 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:03.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:03.508 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:03.508 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:03.508 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:26:03.508 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:26:03.508 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:03.509 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:03.509 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:03.509 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:26:03.509 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:03.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:03.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:03.510 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:03.510 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:03.510 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:26:03.511 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:26:03.511 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:03.511 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:03.511 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:03.511 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:26:03.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:03.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:03.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:03.512 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:03.512 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:03.513 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:26:03.513 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:26:03.513 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:03.513 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:26:03.513 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:03.514 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:26:03.514 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:03.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:03.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:03.514 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:03.515 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:03.515 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:26:03.515 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:26:03.515 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:03.515 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:26:03.515 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:03.516 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:26:03.516 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:03.516 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:03.516 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:03.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:03.517 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:03.517 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:26:03.517 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:26:03.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:03.517 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:03Z" 2023-10-18T21:26:03.518 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:03.518 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045939", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:54.394654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052235", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:24:04.972607Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:26:03Z" } 2023-10-18T21:26:03.519 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052235', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:24:04.972607Z'}}] 2023-10-18T21:26:03.519 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:26:03.520 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:26:03.520 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045939 2023-10-18T21:26:08.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:08.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:08.522 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:08.581 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:08.581 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:08.631 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:08.632 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:08.632 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:08.632 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:08.632 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:08.632 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:26:08.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:08.633 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:08.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:08.633 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:08.634 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:08.634 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:26:08.634 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:26:08.634 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:08.634 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:08.635 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:08.635 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:26:08.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:08.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:08.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:08.636 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:08.636 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:08.636 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:26:08.636 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-10-18T21:26:08.637 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:08.637 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:08.637 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:08.637 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:05.002600Z" 2023-10-18T21:26:08.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:08.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:08.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:08.638 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:08.638 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:08.639 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:26:08.639 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:26:08.639 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:08.639 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:26:08.640 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:08.640 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:26:08.640 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:08.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:08.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:08.641 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:08.641 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:08.641 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:26:08.641 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:26:08.641 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:08.641 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:26:08.642 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:08.642 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:26:08.642 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:08.642 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:08.643 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:08.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:08.643 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:08.643 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:26:08.643 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:26:08.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:08.644 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:08Z" 2023-10-18T21:26:08.644 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:08.645 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:24:05.002600Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052235", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:24:04.972607Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:26:08Z" } 2023-10-18T21:26:08.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:24:05.002600Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052235', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:24:04.972607Z'}}] 2023-10-18T21:26:08.646 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:26:08.646 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:26:08.646 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:26:08.647 DEBUG:boto:path=/bofrnf-74/ 2023-10-18T21:26:08.647 DEBUG:boto:auth_path=/bofrnf-74/ 2023-10-18T21:26:08.648 DEBUG:boto:Method: HEAD 2023-10-18T21:26:08.648 DEBUG:boto:Path: /bofrnf-74/ 2023-10-18T21:26:08.648 DEBUG:boto:Data: 2023-10-18T21:26:08.649 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.649 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.649 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.650 DEBUG:boto:Params: {} 2023-10-18T21:26:08.650 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:26:08.651 DEBUG:boto:Token: None 2023-10-18T21:26:08.651 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-74/ 2023-10-18T21:26:08.652 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:W0dFVqDqce0F4Dn76Rc7ZHe81gY= 2023-10-18T21:26:08.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W0dFVqDqce0F4Dn76Rc7ZHe81gY=', 'Content-Length': '0'} 2023-10-18T21:26:08.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007d051a85c00dc39-0065304d70-2a9d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.659 DEBUG:boto:path=/bofrnf-74/obj1 2023-10-18T21:26:08.659 DEBUG:boto:auth_path=/bofrnf-74/obj1 2023-10-18T21:26:08.659 DEBUG:boto:Method: HEAD 2023-10-18T21:26:08.660 DEBUG:boto:Path: /bofrnf-74/obj1 2023-10-18T21:26:08.660 DEBUG:boto:Data: 2023-10-18T21:26:08.661 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.661 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.661 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.662 DEBUG:boto:Params: {} 2023-10-18T21:26:08.662 DEBUG:boto:Token: None 2023-10-18T21:26:08.663 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-74/obj1 2023-10-18T21:26:08.663 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:eNtrUsreaIG+8ifoJHyBEsCQwn4= 2023-10-18T21:26:08.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:eNtrUsreaIG+8ifoJHyBEsCQwn4=', 'Content-Length': '0'} 2023-10-18T21:26:08.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c0ba96c3312f449-0065304d70-2a9d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.667 DEBUG:boto:path=/bofrnf-74/obj1 2023-10-18T21:26:08.667 DEBUG:boto:auth_path=/bofrnf-74/obj1 2023-10-18T21:26:08.667 DEBUG:boto:Method: GET 2023-10-18T21:26:08.668 DEBUG:boto:Path: /bofrnf-74/obj1 2023-10-18T21:26:08.668 DEBUG:boto:Data: 2023-10-18T21:26:08.669 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.669 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.670 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.670 DEBUG:boto:Params: {} 2023-10-18T21:26:08.670 DEBUG:boto:Token: None 2023-10-18T21:26:08.671 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-74/obj1 2023-10-18T21:26:08.671 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:axThbxaJERlS3BCW7L9WRuaaH0E= 2023-10-18T21:26:08.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:axThbxaJERlS3BCW7L9WRuaaH0E=', 'Content-Length': '0'} 2023-10-18T21:26:08.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcb5bae40b2d80a9-0065304d70-2a9d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.675 DEBUG:boto:path=/bofrnf-74/obj2 2023-10-18T21:26:08.675 DEBUG:boto:auth_path=/bofrnf-74/obj2 2023-10-18T21:26:08.675 DEBUG:boto:Method: HEAD 2023-10-18T21:26:08.676 DEBUG:boto:Path: /bofrnf-74/obj2 2023-10-18T21:26:08.676 DEBUG:boto:Data: 2023-10-18T21:26:08.676 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.677 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.677 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.678 DEBUG:boto:Params: {} 2023-10-18T21:26:08.678 DEBUG:boto:Token: None 2023-10-18T21:26:08.679 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-74/obj2 2023-10-18T21:26:08.679 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:R26WaGEtI/HfHvLj+/Se3ysG1Gg= 2023-10-18T21:26:08.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R26WaGEtI/HfHvLj+/Se3ysG1Gg=', 'Content-Length': '0'} 2023-10-18T21:26:08.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a78588ddc1bf880-0065304d70-2a9d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.681 DEBUG:boto:path=/bofrnf-74/obj2 2023-10-18T21:26:08.682 DEBUG:boto:auth_path=/bofrnf-74/obj2 2023-10-18T21:26:08.682 DEBUG:boto:Method: GET 2023-10-18T21:26:08.683 DEBUG:boto:Path: /bofrnf-74/obj2 2023-10-18T21:26:08.683 DEBUG:boto:Data: 2023-10-18T21:26:08.684 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.684 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.684 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.685 DEBUG:boto:Params: {} 2023-10-18T21:26:08.685 DEBUG:boto:Token: None 2023-10-18T21:26:08.685 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-74/obj2 2023-10-18T21:26:08.686 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:12PP1q8Ym+foXfpTC9L6dgehIi4= 2023-10-18T21:26:08.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:12PP1q8Ym+foXfpTC9L6dgehIi4=', 'Content-Length': '0'} 2023-10-18T21:26:08.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000612267bd2f095f82-0065304d70-2a9d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.689 DEBUG:boto:path=/bofrnf-75/ 2023-10-18T21:26:08.689 DEBUG:boto:auth_path=/bofrnf-75/ 2023-10-18T21:26:08.690 DEBUG:boto:Method: HEAD 2023-10-18T21:26:08.690 DEBUG:boto:Path: /bofrnf-75/ 2023-10-18T21:26:08.690 DEBUG:boto:Data: 2023-10-18T21:26:08.691 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.691 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.691 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.692 DEBUG:boto:Params: {} 2023-10-18T21:26:08.692 DEBUG:boto:Token: None 2023-10-18T21:26:08.693 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-75/ 2023-10-18T21:26:08.693 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:WcQ5AlhrlhFVjxsUQnzhspqwUe0= 2023-10-18T21:26:08.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:WcQ5AlhrlhFVjxsUQnzhspqwUe0=', 'Content-Length': '0'} 2023-10-18T21:26:08.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a019313aab7bdbd1-0065304d70-2a9d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.696 DEBUG:boto:path=/bofrnf-75/obj1 2023-10-18T21:26:08.697 DEBUG:boto:auth_path=/bofrnf-75/obj1 2023-10-18T21:26:08.697 DEBUG:boto:Method: HEAD 2023-10-18T21:26:08.698 DEBUG:boto:Path: /bofrnf-75/obj1 2023-10-18T21:26:08.698 DEBUG:boto:Data: 2023-10-18T21:26:08.698 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.699 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.699 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.700 DEBUG:boto:Params: {} 2023-10-18T21:26:08.700 DEBUG:boto:Token: None 2023-10-18T21:26:08.700 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-75/obj1 2023-10-18T21:26:08.701 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:1lmN5YThPO2dyebJQB6+u04GrOY= 2023-10-18T21:26:08.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1lmN5YThPO2dyebJQB6+u04GrOY=', 'Content-Length': '0'} 2023-10-18T21:26:08.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0f26bb87335f3a5-0065304d70-2a9d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.703 DEBUG:boto:path=/bofrnf-75/obj1 2023-10-18T21:26:08.704 DEBUG:boto:auth_path=/bofrnf-75/obj1 2023-10-18T21:26:08.704 DEBUG:boto:Method: GET 2023-10-18T21:26:08.704 DEBUG:boto:Path: /bofrnf-75/obj1 2023-10-18T21:26:08.705 DEBUG:boto:Data: 2023-10-18T21:26:08.705 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.706 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.706 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.706 DEBUG:boto:Params: {} 2023-10-18T21:26:08.707 DEBUG:boto:Token: None 2023-10-18T21:26:08.707 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-75/obj1 2023-10-18T21:26:08.708 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Z2VUB71+Me6dSqutESWQrWzuUTM= 2023-10-18T21:26:08.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Z2VUB71+Me6dSqutESWQrWzuUTM=', 'Content-Length': '0'} 2023-10-18T21:26:08.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af4ed5ec84449cdf-0065304d70-2a9d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.711 DEBUG:boto:path=/bofrnf-75/obj2 2023-10-18T21:26:08.711 DEBUG:boto:auth_path=/bofrnf-75/obj2 2023-10-18T21:26:08.711 DEBUG:boto:Method: HEAD 2023-10-18T21:26:08.712 DEBUG:boto:Path: /bofrnf-75/obj2 2023-10-18T21:26:08.712 DEBUG:boto:Data: 2023-10-18T21:26:08.713 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.713 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.713 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.714 DEBUG:boto:Params: {} 2023-10-18T21:26:08.714 DEBUG:boto:Token: None 2023-10-18T21:26:08.715 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-75/obj2 2023-10-18T21:26:08.715 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:m8Ai3vTsXEZ+SWFtVm7FM9KTTHI= 2023-10-18T21:26:08.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:m8Ai3vTsXEZ+SWFtVm7FM9KTTHI=', 'Content-Length': '0'} 2023-10-18T21:26:08.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f104715af16eb00-0065304d70-2a9d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.718 DEBUG:boto:path=/bofrnf-75/obj2 2023-10-18T21:26:08.718 DEBUG:boto:auth_path=/bofrnf-75/obj2 2023-10-18T21:26:08.719 DEBUG:boto:Method: GET 2023-10-18T21:26:08.719 DEBUG:boto:Path: /bofrnf-75/obj2 2023-10-18T21:26:08.719 DEBUG:boto:Data: 2023-10-18T21:26:08.720 DEBUG:boto:Headers: {} 2023-10-18T21:26:08.720 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:26:08.721 DEBUG:boto:Port: 8001 2023-10-18T21:26:08.721 DEBUG:boto:Params: {} 2023-10-18T21:26:08.721 DEBUG:boto:Token: None 2023-10-18T21:26:08.722 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:26:08 GMT /bofrnf-75/obj2 2023-10-18T21:26:08.722 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gcL4uYlmqaNL8pChAiCY3PRxs0o= 2023-10-18T21:26:08.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:26:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gcL4uYlmqaNL8pChAiCY3PRxs0o=', 'Content-Length': '0'} 2023-10-18T21:26:08.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:24:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d675cbf6456f8539-0065304d70-2a9d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:26:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:26:08.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:08.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:08.726 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:08.777 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:08.778 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:08.823 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:26:08.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:08.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:08.825 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:08.877 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:08.877 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:08.906 INFO:teuthology.orchestra.run.smithi082.stderr:WARNING: cannot find source zone id for name=* 2023-10-18T21:26:08.906 INFO:teuthology.orchestra.run.smithi082.stderr:WARNING: cannot find source zone id for name=* 2023-10-18T21:26:08.923 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:26:08.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:08.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:08.924 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:08.974 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:08.975 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:09.021 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:26:09.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.022 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:09.071 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:09.071 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:09.123 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:09.123 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:09.124 INFO:teuthology.orchestra.run.smithi082.stdout: "epoch": 22, 2023-10-18T21:26:09.124 INFO:teuthology.orchestra.run.smithi082.stdout: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:26:09.124 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": [], 2023-10-18T21:26:09.124 INFO:teuthology.orchestra.run.smithi082.stdout: "period_map": { 2023-10-18T21:26:09.125 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:09.125 INFO:teuthology.orchestra.run.smithi082.stdout: "zonegroups": [ 2023-10-18T21:26:09.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.125 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a", 2023-10-18T21:26:09.125 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a", 2023-10-18T21:26:09.126 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:26:09.126 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": true, 2023-10-18T21:26:09.126 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:26:09.126 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:26:09.126 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.127 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:26:09.127 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:26:09.127 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:26:09.127 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.128 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:26:09.128 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a1", 2023-10-18T21:26:09.128 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:26:09.128 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:26:09.128 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.129 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:26:09.129 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:26:09.129 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:26:09.129 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:26:09.129 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:26:09.130 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:26:09.130 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:26:09.130 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:26:09.130 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:26:09.130 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:26:09.131 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:26:09.131 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.131 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:26:09.132 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "a2", 2023-10-18T21:26:09.132 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:26:09.132 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:26:09.132 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.132 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:26:09.132 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:26:09.133 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:26:09.133 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:26:09.133 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:26:09.133 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:26:09.133 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:26:09.134 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:26:09.134 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:26:09.134 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:26:09.134 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:26:09.134 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.135 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.135 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.135 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:26:09.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.135 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:26:09.136 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:26:09.136 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:26:09.136 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:26:09.136 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.137 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.137 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:26:09.137 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:26:09.137 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:26:09.137 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [ 2023-10-18T21:26:09.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.138 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-group", 2023-10-18T21:26:09.138 INFO:teuthology.orchestra.run.smithi082.stdout: "data_flow": { 2023-10-18T21:26:09.138 INFO:teuthology.orchestra.run.smithi082.stdout: "symmetrical": [ 2023-10-18T21:26:09.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.139 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-flow1", 2023-10-18T21:26:09.139 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.139 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:26:09.139 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:26:09.139 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.140 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.140 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.140 INFO:teuthology.orchestra.run.smithi082.stdout: "directional": [ 2023-10-18T21:26:09.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.140 INFO:teuthology.orchestra.run.smithi082.stdout: "source_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:26:09.141 INFO:teuthology.orchestra.run.smithi082.stdout: "dest_zone": "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:26:09.141 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.141 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.141 INFO:teuthology.orchestra.run.smithi082.stdout: "pipes": [ 2023-10-18T21:26:09.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.142 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-72", 2023-10-18T21:26:09.142 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.142 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-72", 2023-10-18T21:26:09.142 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.142 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:26:09.143 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.143 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:26:09.143 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-73", 2023-10-18T21:26:09.143 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.144 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:26:09.144 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.144 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:26:09.144 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.145 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:26:09.145 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:26:09.145 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.145 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:26:09.145 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:26:09.146 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:26:09.146 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:26:09.146 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.147 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-73", 2023-10-18T21:26:09.147 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.147 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-73", 2023-10-18T21:26:09.147 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.147 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:26:09.148 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.148 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:26:09.148 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-73", 2023-10-18T21:26:09.149 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.149 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:26:09.149 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.149 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:26:09.150 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.150 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:26:09.150 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:26:09.150 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.151 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:26:09.151 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:26:09.151 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:26:09.151 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:26:09.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.152 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-74", 2023-10-18T21:26:09.152 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-74", 2023-10-18T21:26:09.153 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.153 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:26:09.153 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.153 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:26:09.154 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-74", 2023-10-18T21:26:09.154 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.154 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:26:09.154 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.155 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:26:09.155 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.155 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:26:09.155 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:26:09.155 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.156 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:26:09.156 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:26:09.156 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:26:09.157 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:26:09.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.157 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe-bofrnf-75", 2023-10-18T21:26:09.158 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.158 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-74", 2023-10-18T21:26:09.158 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.158 INFO:teuthology.orchestra.run.smithi082.stdout: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:26:09.158 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.159 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:26:09.159 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "bofrnf-75", 2023-10-18T21:26:09.159 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.159 INFO:teuthology.orchestra.run.smithi082.stdout: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:26:09.160 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.160 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:26:09.160 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.160 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:26:09.161 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:26:09.161 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.161 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:26:09.161 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:26:09.162 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:26:09.162 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:26:09.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.163 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "sync-pipe", 2023-10-18T21:26:09.163 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.163 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:26:09.163 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.163 INFO:teuthology.orchestra.run.smithi082.stdout: "*" 2023-10-18T21:26:09.164 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.164 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": { 2023-10-18T21:26:09.164 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket": "*", 2023-10-18T21:26:09.164 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.165 INFO:teuthology.orchestra.run.smithi082.stdout: "*" 2023-10-18T21:26:09.165 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.165 INFO:teuthology.orchestra.run.smithi082.stdout: "params": { 2023-10-18T21:26:09.165 INFO:teuthology.orchestra.run.smithi082.stdout: "source": { 2023-10-18T21:26:09.166 INFO:teuthology.orchestra.run.smithi082.stdout: "filter": { 2023-10-18T21:26:09.166 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [] 2023-10-18T21:26:09.166 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.166 INFO:teuthology.orchestra.run.smithi082.stdout: "dest": {}, 2023-10-18T21:26:09.167 INFO:teuthology.orchestra.run.smithi082.stdout: "priority": 0, 2023-10-18T21:26:09.167 INFO:teuthology.orchestra.run.smithi082.stdout: "mode": "system", 2023-10-18T21:26:09.167 INFO:teuthology.orchestra.run.smithi082.stdout: "user": "" 2023-10-18T21:26:09.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.167 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.168 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "allowed" 2023-10-18T21:26:09.168 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.168 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.169 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:26:09.169 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:26:09.169 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.169 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "b", 2023-10-18T21:26:09.170 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b", 2023-10-18T21:26:09.170 INFO:teuthology.orchestra.run.smithi082.stdout: "api_name": "", 2023-10-18T21:26:09.170 INFO:teuthology.orchestra.run.smithi082.stdout: "is_master": false, 2023-10-18T21:26:09.170 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:26:09.171 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:26:09.171 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.171 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames": [], 2023-10-18T21:26:09.171 INFO:teuthology.orchestra.run.smithi082.stdout: "hostnames_s3website": [], 2023-10-18T21:26:09.171 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:26:09.172 INFO:teuthology.orchestra.run.smithi082.stdout: "zones": [ 2023-10-18T21:26:09.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.172 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:26:09.172 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b1", 2023-10-18T21:26:09.172 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:26:09.172 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:26:09.173 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.173 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:26:09.173 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:26:09.173 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:26:09.174 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:26:09.174 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:26:09.174 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:26:09.174 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:26:09.174 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:26:09.175 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:26:09.175 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:26:09.175 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:26:09.175 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.176 INFO:teuthology.orchestra.run.smithi082.stdout: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:26:09.176 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "b2", 2023-10-18T21:26:09.176 INFO:teuthology.orchestra.run.smithi082.stdout: "endpoints": [ 2023-10-18T21:26:09.177 INFO:teuthology.orchestra.run.smithi082.stdout: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:26:09.177 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.177 INFO:teuthology.orchestra.run.smithi082.stdout: "log_meta": false, 2023-10-18T21:26:09.178 INFO:teuthology.orchestra.run.smithi082.stdout: "log_data": true, 2023-10-18T21:26:09.178 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_index_max_shards": 11, 2023-10-18T21:26:09.179 INFO:teuthology.orchestra.run.smithi082.stdout: "read_only": false, 2023-10-18T21:26:09.179 INFO:teuthology.orchestra.run.smithi082.stdout: "tier_type": "", 2023-10-18T21:26:09.179 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from_all": true, 2023-10-18T21:26:09.179 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_from": [], 2023-10-18T21:26:09.179 INFO:teuthology.orchestra.run.smithi082.stdout: "redirect_zone": "", 2023-10-18T21:26:09.180 INFO:teuthology.orchestra.run.smithi082.stdout: "supported_features": [ 2023-10-18T21:26:09.180 INFO:teuthology.orchestra.run.smithi082.stdout: "compress-encrypted", 2023-10-18T21:26:09.180 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:26:09.180 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.180 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.181 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.181 INFO:teuthology.orchestra.run.smithi082.stdout: "placement_targets": [ 2023-10-18T21:26:09.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.181 INFO:teuthology.orchestra.run.smithi082.stdout: "name": "default-placement", 2023-10-18T21:26:09.181 INFO:teuthology.orchestra.run.smithi082.stdout: "tags": [], 2023-10-18T21:26:09.181 INFO:teuthology.orchestra.run.smithi082.stdout: "storage_classes": [ 2023-10-18T21:26:09.182 INFO:teuthology.orchestra.run.smithi082.stdout: "STANDARD" 2023-10-18T21:26:09.182 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.182 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.182 INFO:teuthology.orchestra.run.smithi082.stdout: "default_placement": "default-placement", 2023-10-18T21:26:09.183 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:26:09.183 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_policy": { 2023-10-18T21:26:09.183 INFO:teuthology.orchestra.run.smithi082.stdout: "groups": [] 2023-10-18T21:26:09.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.183 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled_features": [ 2023-10-18T21:26:09.184 INFO:teuthology.orchestra.run.smithi082.stdout: "resharding" 2023-10-18T21:26:09.184 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.184 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.184 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2023-10-18T21:26:09.184 INFO:teuthology.orchestra.run.smithi082.stdout: "short_zone_ids": [ 2023-10-18T21:26:09.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.185 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:26:09.185 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 3791896933 2023-10-18T21:26:09.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.186 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:26:09.186 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2277887411 2023-10-18T21:26:09.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.186 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:26:09.187 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 953824181 2023-10-18T21:26:09.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:09.187 INFO:teuthology.orchestra.run.smithi082.stdout: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:26:09.187 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 2717429769 2023-10-18T21:26:09.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.188 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:09.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.188 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zonegroup": "a", 2023-10-18T21:26:09.188 INFO:teuthology.orchestra.run.smithi082.stdout: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:26:09.188 INFO:teuthology.orchestra.run.smithi082.stdout: "period_config": { 2023-10-18T21:26:09.189 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_quota": { 2023-10-18T21:26:09.189 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:26:09.189 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:26:09.189 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:26:09.189 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:26:09.190 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:26:09.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.190 INFO:teuthology.orchestra.run.smithi082.stdout: "user_quota": { 2023-10-18T21:26:09.190 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false, 2023-10-18T21:26:09.190 INFO:teuthology.orchestra.run.smithi082.stdout: "check_on_raw": false, 2023-10-18T21:26:09.191 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size": -1, 2023-10-18T21:26:09.191 INFO:teuthology.orchestra.run.smithi082.stdout: "max_size_kb": 0, 2023-10-18T21:26:09.191 INFO:teuthology.orchestra.run.smithi082.stdout: "max_objects": -1 2023-10-18T21:26:09.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.191 INFO:teuthology.orchestra.run.smithi082.stdout: "user_ratelimit": { 2023-10-18T21:26:09.192 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:26:09.192 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:26:09.192 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:26:09.192 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:26:09.192 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:26:09.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.193 INFO:teuthology.orchestra.run.smithi082.stdout: "bucket_ratelimit": { 2023-10-18T21:26:09.193 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:26:09.193 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:26:09.193 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:26:09.194 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:26:09.194 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:26:09.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.194 INFO:teuthology.orchestra.run.smithi082.stdout: "anonymous_ratelimit": { 2023-10-18T21:26:09.194 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_ops": 0, 2023-10-18T21:26:09.195 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_ops": 0, 2023-10-18T21:26:09.195 INFO:teuthology.orchestra.run.smithi082.stdout: "max_read_bytes": 0, 2023-10-18T21:26:09.195 INFO:teuthology.orchestra.run.smithi082.stdout: "max_write_bytes": 0, 2023-10-18T21:26:09.195 INFO:teuthology.orchestra.run.smithi082.stdout: "enabled": false 2023-10-18T21:26:09.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:09.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:09.196 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:26:09.196 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_name": "test-realm", 2023-10-18T21:26:09.196 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:09.196 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:09.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.198 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:09.246 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:09.246 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:09.308 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:26:09.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.309 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket bofrnf-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:09.373 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:09.374 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:09.474 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:26:09.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.475 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket bofrnf-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:09.558 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:09.558 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:09.649 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-10-18T21:26:09.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-bofrnf-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=bofrnf-74', '--source-bucket=*', '--dest-bucket=bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-bofrnf-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=bofrnf-74', '--source-bucket=*', '--dest-bucket=bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.650 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-bofrnf-74 --source-zones=a1 --dest-zones=a2 --bucket=bofrnf-74 '--source-bucket=*' --dest-bucket=bofrnf-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:09.723 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:09.723 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:09.802 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-bofrnf-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-18T21:26:09.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-bofrnf-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=bofrnf-74', '--source-bucket=*', '--dest-bucket=bofrnf-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-bofrnf-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=bofrnf-74', '--source-bucket=*', '--dest-bucket=bofrnf-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.803 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-bofrnf-75 --source-zones=a1 --dest-zones=a2 --bucket=bofrnf-74 '--source-bucket=*' --dest-bucket=bofrnf-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:09.869 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:09.869 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:09.935 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-bofrnf-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-18T21:26:09.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:09.937 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket bofrnf-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:09.988 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:09.989 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:10.037 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-bofrnf-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-18T21:26:10.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:10.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:26:10.039 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:26:10.084 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:10.084 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:10.131 INFO:teuthology.orchestra.run.smithi082.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-74","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-75","source":{"bucket":"bofrnf-74","zones":["a1"]},"dest":{"bucket":"bofrnf-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:26:10.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:10.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:10.132 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:10.180 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:10.180 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:10.209 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:26:10.224 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:26:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664370.023158_2186.1", 2023-10-18T21:26:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:26:10.023158Z" 2023-10-18T21:26:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:26:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:26:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:26:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:10.228 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:26:10.229 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:10.229 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:26:10.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:10.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:10.231 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:10.280 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:10.280 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:10.309 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:10.308+0000 7f610e780780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:10.310 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:10.309+0000 7f610e780780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:10.323 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:10Z" 2023-10-18T21:26:10.337 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:10.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:10Z" } 2023-10-18T21:26:10.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:10.339 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:10.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:10.339 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:26:15.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:15.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:15.342 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:15.399 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:15.399 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:15.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:15.431+0000 7f67325db780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:15.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:15.432+0000 7f67325db780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:15.446 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:15.447 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:15.447 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:15.447 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:15.447 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:15.448 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:15.448 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:15.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:15.448 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:15.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:15.449 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:15.449 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:15.449 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:15.449 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:15.450 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:15.450 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:15.450 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:15.450 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:15.450 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:15.451 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:15.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:15.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:15.451 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:15.451 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:15.452 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:15.452 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:15.452 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:15.452 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:15.453 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:15.453 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:15.453 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:15.453 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:15.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:15.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:15.454 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:15.454 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:15.454 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:15.455 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:15.455 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:15.455 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:15.455 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:15.455 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:15.456 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:15.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:15.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:15.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:15.457 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:15.457 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:15.457 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:15.457 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:15.457 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:15.458 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:15.458 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:15.458 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:15.458 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:15.459 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:15.459 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:15.459 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:15.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:15.460 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:15.460 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:15.460 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:15.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:15.460 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:15Z" 2023-10-18T21:26:15.461 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:15.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:15Z" } 2023-10-18T21:26:15.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:15.462 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:15.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:15.463 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:26:20.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:20.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:20.465 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:20.522 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:20.523 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:20.555 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:20.554+0000 7f7eaf3de780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:20.556 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:20.555+0000 7f7eaf3de780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:20.571 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:20.571 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:20.571 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:20.572 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:20.572 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:20.572 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:20.572 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:20.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:20.573 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:20.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:20.573 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:20.573 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:20.573 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:20.574 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:20.574 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:20.574 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:20.574 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:20.574 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:20.575 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:20.575 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:20.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:20.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:20.575 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:20.576 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:20.576 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:20.576 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:20.576 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:20.576 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:20.577 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:20.577 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:20.577 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:20.577 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:20.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:20.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:20.578 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:20.578 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:20.578 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:20.578 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:20.579 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:20.579 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:20.581 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:20.581 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:20.581 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:20.581 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:20.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:20.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:20.582 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:20.582 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:20.582 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:20.582 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:20.583 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:20.583 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:20.583 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:20.583 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:20.583 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:20.584 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:20.584 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:20.584 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:20.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:20.584 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:20.585 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:20.585 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:20.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:20.585 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:20Z" 2023-10-18T21:26:20.585 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:20.586 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:20Z" } 2023-10-18T21:26:20.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:20.587 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:20.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:20.588 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:26:25.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:25.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:25.590 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:25.647 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:25.647 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:25.680 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:25.680+0000 7f0cce81d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:25.681 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:25.681+0000 7f0cce81d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:25.695 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:25.702 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:25.702 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:25.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:25.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:25.702 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:25.703 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:25.703 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:25.703 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:25.703 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:25.703 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:25.704 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:25.704 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:25.704 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:25.704 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:25.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:25.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:25.705 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:25.705 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:25.705 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:25.705 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:25.706 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:25.706 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:25.706 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:25.706 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:25.706 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:25.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:25.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:25.707 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:25.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:25.707 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:25.708 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:25.708 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:25.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:25.708 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:25Z" 2023-10-18T21:26:25.708 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:25.709 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:25Z" } 2023-10-18T21:26:25.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:25.710 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:25.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:25.711 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:26:30.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:30.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:30.714 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:30.770 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:30.771 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:30.803 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:30.802+0000 7fe0778cc780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:30.804 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:30.803+0000 7fe0778cc780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:30.818 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:30.818 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:30.819 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:30.819 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:30.819 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:30.820 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:30.820 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:30.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:30.821 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:30.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:30.821 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:30.821 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:30.821 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:30.822 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:30.822 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:30.822 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:30.822 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:30.822 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:30.823 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:30.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:30.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:30.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:30.824 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:30.824 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:30.824 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:30.824 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:30.824 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:30.825 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:30.825 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:30.825 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:30.825 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:30.825 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:30.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:30.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:30.826 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:30.826 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:30.827 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:30.827 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:30.827 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:30.827 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:30.827 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:30.827 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:30.828 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:30.828 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:30.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:30.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:30.828 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:30.829 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:30.829 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:30.829 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:30.829 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:30.829 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:30.830 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:30.830 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:30.830 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:30.830 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:30.830 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:30.831 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:30.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:30.832 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:30.832 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:30.832 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:30.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:30.833 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:30Z" 2023-10-18T21:26:30.833 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:30.834 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:30Z" } 2023-10-18T21:26:30.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:30.836 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:30.836 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:30.837 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:26:35.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:35.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:35.839 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:35.896 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:35.896 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:35.928 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:35.926+0000 7fb501c65780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:35.929 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:35.927+0000 7fb501c65780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:35.943 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:35.943 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:35.943 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:35.943 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:35.944 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:35.944 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:35.944 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:35.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:35.944 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:35.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:35.945 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:35.945 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:35.945 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:35.945 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:35.946 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:35.946 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:35.946 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:35.946 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:35.946 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:35.947 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:35.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:35.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:35.947 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:35.947 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:35.948 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:35.948 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:35.948 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:35.948 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:35.948 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:35.949 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:35.949 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:35.949 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:35.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:35.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:35.950 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:35.950 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:35.950 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:35.950 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:35.950 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:35.951 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:35.951 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:35.951 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:35.951 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:35.951 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:35.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:35.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:35.952 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:35.952 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:35.952 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:35.953 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:35.953 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:35.953 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:35.953 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:35.953 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:35.954 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:35.954 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:35.954 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:35.954 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:35.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:35.955 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:35.955 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:35.955 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:35.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:35.955 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:35Z" 2023-10-18T21:26:35.956 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:35.956 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:35Z" } 2023-10-18T21:26:35.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:35.957 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:35.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:35.958 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:26:40.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:40.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:40.960 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:41.017 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:41.017 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:41.050 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:41.048+0000 7faf5cab1780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:41.050 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:41.049+0000 7faf5cab1780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:41.064 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:41.065 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:41.066 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:41.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:41.066 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:41.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:41.067 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:41.068 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:41.069 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:41.070 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:41.070 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:41.070 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:41.070 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:41.070 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:41.071 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:41.072 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:41.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:41.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:41.074 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:41.074 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:41.074 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:41.074 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:41.075 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:41.076 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:41.077 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:41.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:41.077 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:41Z" 2023-10-18T21:26:41.077 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:41.078 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:41Z" } 2023-10-18T21:26:41.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:41.079 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:41.079 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:41.080 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:26:46.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:46.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:46.083 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:46.139 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:46.139 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:46.173 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:46.171+0000 7f9809352780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:46.174 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:46.172+0000 7f9809352780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:46.187 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:46.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:46.190 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:46.190 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:46.190 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:46.190 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:46.191 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:46.192 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:46.193 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:46.194 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:46.194 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:46.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:46.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:46.195 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:46.196 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:46.196 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:46.196 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:46.196 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:46.197 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:46.198 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:46.198 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:46.198 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:46.198 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:46.198 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:46.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:46.200 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:46.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:46.200 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:46.200 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:46.200 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:46.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:46.201 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:46Z" 2023-10-18T21:26:46.201 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:46.202 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:46Z" } 2023-10-18T21:26:46.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:46.203 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:46.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:46.204 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:26:51.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:51.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:51.206 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:51.263 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:51.263 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:51.296 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:51.295+0000 7ffa15ef7780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:51.297 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:51.296+0000 7ffa15ef7780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:51.311 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:51.311 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:51.311 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:51.312 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:51.312 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:51.312 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:51.312 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:51.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:51.313 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:51.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:51.313 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:51.314 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:51.314 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:51.314 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:51.314 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:51.314 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:51.315 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:51.315 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:51.315 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:51.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:51.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:51.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:51.316 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:51.316 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:51.316 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:51.316 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:51.316 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:51.317 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:51.317 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:51.317 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:51.317 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:51.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:51.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:51.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:51.318 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:51.318 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:51.318 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:51.318 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:51.319 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:51.319 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:51.319 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:51.319 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:51.319 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:51.320 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:51.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:51.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:51.320 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:51.320 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:51.321 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:51.321 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:51.321 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:51.321 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:51.321 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:51.321 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:51.322 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:51.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:51.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:51.322 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:51.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:51.323 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:51.323 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:51.323 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:51.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:51.323 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:51Z" 2023-10-18T21:26:51.324 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:51.324 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:51Z" } 2023-10-18T21:26:51.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:51.325 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:51.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:51.326 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:26:56.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:56.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:26:56.328 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:26:56.385 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:26:56.386 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:26:56.420 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:56.419+0000 7f7dbabff780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:56.421 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:26:56.420+0000 7f7dbabff780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:26:56.435 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:26:56.435 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:26:56.435 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:26:56.436 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:26:56.436 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:26:56.436 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:26:56.436 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:56.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:56.437 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:26:56.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:56.437 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:26:56.437 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:56.437 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:56.438 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:26:56.438 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:56.438 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:26:56.438 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:56.438 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:26:56.439 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:56.439 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:56.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:56.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:56.440 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:26:56.440 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:56.440 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:56.440 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:26:56.440 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:56.441 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:56.441 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:56.441 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:26:56.441 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:56.441 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:56.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:56.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:56.442 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:26:56.442 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:56.442 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:56.443 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:26:56.443 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:56.443 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:56.443 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:56.443 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:26:56.444 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:56.444 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:56.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:56.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:26:56.444 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:26:56.445 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:26:56.445 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:26:56.445 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:26:56.445 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:26:56.446 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:26:56.446 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:26:56.446 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:26:56.446 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:26:56.446 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:56.447 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:26:56.447 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:26:56.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:56.447 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:26:56.447 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:26:56.448 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:26:56.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:26:56.448 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:26:56Z" 2023-10-18T21:26:56.448 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:26:56.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:26:56Z" } 2023-10-18T21:26:56.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:26:56.450 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:56.451 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:26:56.451 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:01.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:01.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:01.453 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:01.512 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:01.513 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:01.547 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:01.546+0000 7fe8451ea780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:01.548 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:01.547+0000 7fe8451ea780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:01.561 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:01.561 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:01.562 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:01.562 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:01.562 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:01.562 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:01.563 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:01.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:01.563 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:01.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:01.563 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:01.564 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:01.564 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:01.564 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:01.564 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:01.564 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:01.565 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:01.565 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:01.565 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:01.565 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:01.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:01.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:01.566 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:01.566 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:01.566 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:01.567 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:01.567 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:01.567 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:01.567 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:01.568 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:01.568 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:01.568 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:01.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:01.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:01.569 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:01.569 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:01.569 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:01.569 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:01.569 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:01.570 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:01.570 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:01.570 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:01.570 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:01.570 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:01.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:01.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:01.571 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:01.571 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:01.572 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:01.572 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:01.572 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:01.572 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:01.572 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:01.573 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:01.573 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:01.573 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:01.573 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:01.573 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:01.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:01.574 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:01.574 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:01.574 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:01.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:01.575 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:01Z" 2023-10-18T21:27:01.575 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:01.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:01Z" } 2023-10-18T21:27:01.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:01.577 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:01.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:01.578 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:06.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:06.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:06.580 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:06.637 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:06.637 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:06.669 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:06.668+0000 7f3b53d77780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:06.670 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:06.669+0000 7f3b53d77780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:06.684 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:06.684 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:06.684 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:06.684 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:06.685 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:06.685 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:06.685 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:06.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:06.685 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:06.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:06.686 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:06.686 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:06.686 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:06.686 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:06.687 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:06.687 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:06.687 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:06.687 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:06.687 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:06.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:06.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:06.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:06.688 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:06.688 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:06.689 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:06.689 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:06.689 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:06.689 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:06.690 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:06.690 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:06.690 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:06.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:06.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:06.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:06.691 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:06.691 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:06.691 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:06.691 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:06.692 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:06.692 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:06.692 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:06.692 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:06.692 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:06.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:06.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:06.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:06.693 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:06.694 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:06.694 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:06.694 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:06.694 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:06.694 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:06.695 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:06.695 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:06.695 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:06.695 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:06.695 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:06.696 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:06.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:06.696 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:06.696 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:06.696 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:06.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:06.697 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:06Z" 2023-10-18T21:27:06.697 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:06.698 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:06Z" } 2023-10-18T21:27:06.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:06.699 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:06.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:06.700 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:11.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:11.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:11.703 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:11.759 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:11.760 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:11.794 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:11.793+0000 7fe7e3973780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:11.795 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:11.794+0000 7fe7e3973780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:11.809 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:11.809 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:11.809 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:11.809 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:11.810 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:11.810 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:11.810 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:11.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:11.811 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:11.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:11.811 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:11.811 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:11.811 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:11.812 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:11.812 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:11.812 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:11.812 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:11.812 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:11.813 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:11.813 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:11.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:11.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:11.813 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:11.814 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:11.814 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:11.814 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:11.814 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:11.815 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:11.815 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:11.815 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:11.815 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:11.815 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:11.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:11.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:11.816 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:11.816 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:11.816 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:11.817 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:11.817 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:11.817 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:11.817 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:11.817 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:11.818 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:11.818 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:11.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:11.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:11.819 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:11.819 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:11.819 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:11.819 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:11.819 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:11.820 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:11.820 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:11.820 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:11.820 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:11.820 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:11.821 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:11.821 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:11.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:11.821 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:11.821 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:11.822 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:11.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:11.822 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:11Z" 2023-10-18T21:27:11.822 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:11.823 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:11Z" } 2023-10-18T21:27:11.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:11.824 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:11.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:11.825 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:16.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:16.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:16.828 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:16.884 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:16.884 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:16.916 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:16.916+0000 7efc5d2a4780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:16.917 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:16.917+0000 7efc5d2a4780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:16.931 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:16.932 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:16.932 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:16.932 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:16.932 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:16.932 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:16.933 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:16.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:16.933 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:16.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:16.934 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:16.934 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:16.934 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:16.935 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:16.935 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:16.935 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:16.935 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:16.935 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:16.936 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:16.936 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:16.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:16.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:16.937 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:16.937 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:16.937 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:16.937 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:16.938 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:16.938 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:16.938 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:16.938 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:16.939 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:16.939 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:16.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:16.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:16.940 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:16.940 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:16.940 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:16.940 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:16.940 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:16.941 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:16.941 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:16.941 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:16.941 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:16.942 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:16.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:16.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:16.942 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:16.943 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:16.943 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:16.943 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:16.943 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:16.944 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:16.944 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:16.944 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:16.944 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:16.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:16.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:16.945 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:16.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:16.945 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:16.946 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:16.946 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:16.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:16.946 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:16Z" 2023-10-18T21:27:16.947 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:16.948 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:16Z" } 2023-10-18T21:27:16.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:16.949 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:16.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:16.950 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:21.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:21.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:21.952 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:22.009 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:22.010 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:22.043 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:22.042+0000 7efc4f334780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:22.044 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:22.043+0000 7efc4f334780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:22.058 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:22.058 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:22.058 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:22.058 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:22.059 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:22.059 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:22.059 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:22.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:22.059 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:22.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:22.060 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:22.060 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:22.060 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:22.060 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:22.061 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:22.061 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:22.061 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:22.061 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:22.061 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:22.062 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:22.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:22.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:22.062 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:22.063 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:22.063 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:22.063 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:22.063 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:22.063 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:22.064 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:22.064 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:22.064 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:22.064 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:22.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:22.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:22.065 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:22.065 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:22.065 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:22.066 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:22.066 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:22.066 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:22.066 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:22.066 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:22.067 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:22.067 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:22.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:22.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:22.068 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:22.068 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:22.068 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:22.068 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:22.068 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:22.069 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:22.069 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:22.069 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:22.069 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:22.069 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:22.070 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:22.070 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:22.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:22.070 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:22.071 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:22.071 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:22.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:22.071 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:22Z" 2023-10-18T21:27:22.071 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:22.072 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:22Z" } 2023-10-18T21:27:22.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:22.073 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:22.074 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:22.074 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:27.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:27.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:27.076 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:27.132 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:27.132 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:27.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:27.162+0000 7f882b2bd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:27.165 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:27.163+0000 7f882b2bd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:27.179 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:27.179 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:27.179 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:27.180 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:27.180 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:27.180 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:27.180 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:27.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:27.181 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:27.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:27.181 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:27.181 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:27.181 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:27.182 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:27.182 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:27.182 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:27.182 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:27.182 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:27.183 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:27.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:27.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:27.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:27.183 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:27.184 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:27.184 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:27.184 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:27.184 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:27.184 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:27.185 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:27.185 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:27.185 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:27.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:27.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:27.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:27.186 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:27.186 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:27.186 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:27.186 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:27.187 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:27.187 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:27.187 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:27.187 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:27.187 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:27.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:27.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:27.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:27.188 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:27.188 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:27.189 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:27.189 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:27.189 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:27.189 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:27.189 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:27.190 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:27.190 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:27.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:27.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:27.190 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:27.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:27.191 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:27.191 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:27.191 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:27.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:27.191 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:27Z" 2023-10-18T21:27:27.192 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:27.192 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:27Z" } 2023-10-18T21:27:27.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:27.193 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:27.194 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:27.194 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:32.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:32.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:32.197 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:32.254 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:32.255 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:32.290 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:32.288+0000 7fdff0faf780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:32.291 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:32.289+0000 7fdff0faf780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:32.304 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:32.306 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:32.306 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:32.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:32.306 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:32Z" 2023-10-18T21:27:32.318 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:32.319 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:32Z" } 2023-10-18T21:27:32.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:32.320 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:32.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:32.321 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:37.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:37.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:37.323 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:37.380 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:37.380 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:37.412 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:37.410+0000 7f7c0b6a0780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:37.413 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:37.411+0000 7f7c0b6a0780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:37.427 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:37.427 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:37.427 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:37.427 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:37.428 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:37.428 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:37.428 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:37.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:37.429 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:37.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:37.429 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:37.429 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:37.430 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:37.430 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:37.430 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:37.430 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:37.431 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:37.431 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:37.431 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:37.432 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:37.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:37.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:37.432 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:37.433 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:37.433 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:37.433 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:37.433 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:37.434 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:37.434 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:37.434 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:37.434 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:37.435 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:37.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:37.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:37.435 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:37.435 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:37.436 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:37.436 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:37.436 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:37.436 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:37.436 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:37.437 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:37.437 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:37.437 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:37.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:37.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:37.438 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:37.438 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:37.438 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:37.438 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:37.439 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:37.439 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:37.439 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:37.439 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:37.439 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:37.440 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:37.440 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:37.440 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:37.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:37.440 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:37.441 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:37.441 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:37.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:37.441 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:37Z" 2023-10-18T21:27:37.442 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:37.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:37Z" } 2023-10-18T21:27:37.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:37.444 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:37.444 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:37.444 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:42.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:42.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:42.447 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:42.505 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:42.505 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:42.536+0000 7f2605508780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:42.538 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:42.537+0000 7f2605508780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:42.552 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:42.552 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:42.552 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:42.553 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:42.553 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:42.553 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:42.553 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:42.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:42.554 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:42.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:42.554 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:42.554 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:42.555 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:42.555 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:42.555 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:42.555 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:42.555 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:42.556 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:42.556 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:42.556 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:42.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:42.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:42.557 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:42.557 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:42.557 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:42.557 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:42.558 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:42.558 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:42.558 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:42.558 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:42.559 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:42.559 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:42.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:42.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:42.559 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:42.560 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:42.560 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:42.560 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:42.560 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:42.560 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:42.561 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:42.561 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:42.561 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:42.561 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:42.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:42.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:42.562 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:42.562 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:42.562 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:42.563 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:42.563 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:42.563 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:42.563 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:42.563 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:42.564 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:42.564 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:42.564 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:42.564 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:42.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:42.565 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:42.565 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:42.565 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:42.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:42.566 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:42Z" 2023-10-18T21:27:42.566 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:42.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:42Z" } 2023-10-18T21:27:42.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:42.568 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:42.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:42.569 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:47.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:47.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:47.572 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:47.630 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:47.631 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:47.663 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:47.662+0000 7f87ad501780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:47.664 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:47.663+0000 7f87ad501780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:47.677 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:47Z" 2023-10-18T21:27:47.691 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:47.692 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:47Z" } 2023-10-18T21:27:47.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:47.693 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:47.693 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:47.694 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:52.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:52.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:52.695 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:52.752 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:52.752 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:52.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:52.784+0000 7fe99f29c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:52.785 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:52.785+0000 7fe99f29c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:52.799 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:52.800 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:52.800 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:52.800 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:52.800 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:52.800 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:52.801 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:52.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:52.801 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:52.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:52.802 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:52.802 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:52.802 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:52.802 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:52.802 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:52.803 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:52.803 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:52.803 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:52.803 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:52.803 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:52.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:52.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:52.804 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:52.804 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:52.804 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:52.805 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:52.805 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:52.805 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:52.805 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:52.805 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:52.806 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:52.806 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:52.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:52.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:52.807 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:52.807 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:52.807 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:52.807 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:52.807 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:52.807 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:52.808 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:52.808 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:52.808 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:52.808 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:52.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:52.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:52.809 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:52.809 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:52.809 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:52.810 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:52.810 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:52.810 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:52.810 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:52.810 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:52.811 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:52.811 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:52.811 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:52.811 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:52.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:52.812 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:52.812 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:52.812 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:52.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:52.812 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:52Z" 2023-10-18T21:27:52.813 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:52.814 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:52Z" } 2023-10-18T21:27:52.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:52.815 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:52.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:52.815 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:27:57.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:57.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:27:57.819 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:27:57.876 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:27:57.877 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:27:57.908 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:57.907+0000 7f047796f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:57.909 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:27:57.908+0000 7f047796f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:27:57.923 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:27:57.923 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:27:57.924 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:27:57.924 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:27:57.924 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:27:57.924 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:27:57.925 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:57.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:57.925 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:27:57.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:57.926 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:27:57.926 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:57.926 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:57.926 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:27:57.926 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:57.927 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:27:57.927 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:57.927 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:27:57.929 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:57.929 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:57.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:57.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:57.930 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:27:57.930 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:57.930 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:57.930 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:27:57.931 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:57.931 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:57.931 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:57.932 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:27:57.932 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:57.932 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:57.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:57.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:57.933 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:27:57.933 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:57.933 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:57.933 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:27:57.934 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:57.934 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:57.934 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:57.934 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:27:57.935 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:57.935 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:57.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:57.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:27:57.936 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:27:57.936 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:27:57.936 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:27:57.936 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:27:57.937 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:27:57.937 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:27:57.937 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:27:57.937 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:27:57.937 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:27:57.938 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:57.938 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:27:57.938 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:27:57.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:57.939 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:27:57.939 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:27:57.939 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:27:57.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:27:57.940 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:27:57Z" 2023-10-18T21:27:57.940 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:27:57.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:27:57Z" } 2023-10-18T21:27:57.942 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:27:57.943 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:57.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:27:57.943 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:28:02.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:02.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:02.946 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:03.004 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:03.004 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:03.037 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:28:03.036+0000 7f6988b9d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:03.038 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:28:03.037+0000 7f6988b9d780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:03.051 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:28:03.052 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:28:03.052 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:28:03.052 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:28:03.052 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:28:03.053 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:03.053 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:03.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:03.053 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:28:03.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:03.054 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:28:03.054 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:03.054 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:03.057 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:28:03.058 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:03.058 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:28:03.058 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:03.058 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:28:03.059 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:03.059 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:03.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:03.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:03.059 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:28:03.060 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:03.060 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:03.060 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:03.060 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:03.060 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:03.061 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:03.061 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:03.061 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:03.061 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:03.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:03.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:03.062 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:28:03.062 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:03.062 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:03.062 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:28:03.063 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:03.063 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:03.063 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:03.063 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:28:03.063 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:03.064 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:03.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:03.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:03.064 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:28:03.064 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:03.065 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:03.065 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:03.065 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:03.065 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:03.066 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:03.066 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:03.066 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:03.066 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:03.067 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:03.067 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:28:03.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:03.067 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:28:03.067 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:28:03.068 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:28:03.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:03.068 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:28:03Z" 2023-10-18T21:28:03.068 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:28:03.069 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:28:03Z" } 2023-10-18T21:28:03.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:03.070 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:03.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:03.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:28:08.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:08.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:08.074 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:08.130 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:08.131 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:08.163 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:28:08.162+0000 7f1a8949a780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:08.164 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:28:08.163+0000 7f1a8949a780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:08.178 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:28:08.179 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:28:08.179 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:28:08.179 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:28:08.179 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:28:08.180 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:08.180 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:08.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:08.180 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:28:08.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:08.181 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:28:08.181 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:08.181 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:08.181 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664238.589509_2088.1", 2023-10-18T21:28:08.182 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:08.182 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:28:08.182 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:08.182 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:23:58.589509Z", 2023-10-18T21:28:08.183 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:08.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:08.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:08.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:08.183 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:28:08.184 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:08.184 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:08.184 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:08.184 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:08.184 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:08.185 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:08.185 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:08.185 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:08.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:08.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:08.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:08.186 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:28:08.186 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:08.186 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:08.187 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:28:08.187 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:08.187 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:08.187 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:08.187 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:28:08.188 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:08.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:08.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:08.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:08.188 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:28:08.189 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:08.189 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:08.189 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:08.189 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:08.190 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:08.190 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:08.190 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:08.190 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:08.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:08.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:08.191 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:28:08.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:08.191 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:28:08.191 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:28:08.192 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:28:08.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:08.192 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:28:08Z" 2023-10-18T21:28:08.192 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:28:08.193 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664238.589509_2088.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:23:58.589509Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:28:08Z" } 2023-10-18T21:28:08.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664238.589509_2088.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:23:58.589509Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:08.194 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:08.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664238.589509_2088.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:08.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697664370.023158_2186.1 target=1_1697664238.589509_2088.1 2023-10-18T21:28:13.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.197 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:13.253 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:13.254 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:13.286 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:28:13.286+0000 7f1dbbfd8780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:13.287 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:28:13.287+0000 7f1dbbfd8780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:13.301 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:28:13.301 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:28:13.301 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:28:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:28:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:28:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:28:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:28:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664370.023158_2186.1", 2023-10-18T21:28:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:28:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:26:10.023158Z", 2023-10-18T21:28:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:28:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:28:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:28:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:28:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:28:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:28:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:28:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:28:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:28:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:28:13Z" 2023-10-18T21:28:13.315 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:28:13.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664370.023158_2186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:26:10.023158Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:28:13Z" } 2023-10-18T21:28:13.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664370.023158_2186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:26:10.023158Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:13.318 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:13.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:13.319 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:28:13.320 DEBUG:boto:path=/bofrnf-74/ 2023-10-18T21:28:13.320 DEBUG:boto:auth_path=/bofrnf-74/ 2023-10-18T21:28:13.321 DEBUG:boto:Method: HEAD 2023-10-18T21:28:13.321 DEBUG:boto:Path: /bofrnf-74/ 2023-10-18T21:28:13.322 DEBUG:boto:Data: 2023-10-18T21:28:13.322 DEBUG:boto:Headers: {} 2023-10-18T21:28:13.323 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:13.323 DEBUG:boto:Port: 8000 2023-10-18T21:28:13.323 DEBUG:boto:Params: {} 2023-10-18T21:28:13.324 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:28:13.325 DEBUG:boto:Token: None 2023-10-18T21:28:13.325 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:28:13 GMT /bofrnf-74/ 2023-10-18T21:28:13.326 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gXBItNBAdeLzISBJVjcY/bHUMIs= 2023-10-18T21:28:13.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:13 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gXBItNBAdeLzISBJVjcY/bHUMIs=', 'Content-Length': '0'} 2023-10-18T21:28:13.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a791d2f38f1c4df5-0065304ded-2ba8-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:13.333 DEBUG:boto:path=/bofrnf-74/obj3 2023-10-18T21:28:13.334 DEBUG:boto:auth_path=/bofrnf-74/obj3 2023-10-18T21:28:13.334 DEBUG:boto:Method: PUT 2023-10-18T21:28:13.335 DEBUG:boto:Path: /bofrnf-74/obj3 2023-10-18T21:28:13.335 DEBUG:boto:Data: 2023-10-18T21:28:13.336 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:28:13.336 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:13.336 DEBUG:boto:Port: 8000 2023-10-18T21:28:13.337 DEBUG:boto:Params: {} 2023-10-18T21:28:13.337 DEBUG:boto:Token: None 2023-10-18T21:28:13.338 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:28:13 GMT /bofrnf-74/obj3 2023-10-18T21:28:13.338 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:mUac3ib8QSnlbaY+/zS7HC0OsxU= 2023-10-18T21:28:13.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:28:13 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mUac3ib8QSnlbaY+/zS7HC0OsxU='} 2023-10-18T21:28:13.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a126fbc2881c0036-0065304ded-2ba8-a1'), ('Date', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:13.348 DEBUG:boto:path=/bofrnf-74/ 2023-10-18T21:28:13.348 DEBUG:boto:auth_path=/bofrnf-74/ 2023-10-18T21:28:13.348 DEBUG:boto:Method: HEAD 2023-10-18T21:28:13.349 DEBUG:boto:Path: /bofrnf-74/ 2023-10-18T21:28:13.349 DEBUG:boto:Data: 2023-10-18T21:28:13.349 DEBUG:boto:Headers: {} 2023-10-18T21:28:13.350 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:13.350 DEBUG:boto:Port: 8000 2023-10-18T21:28:13.351 DEBUG:boto:Params: {} 2023-10-18T21:28:13.351 DEBUG:boto:Token: None 2023-10-18T21:28:13.352 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:28:13 GMT /bofrnf-74/ 2023-10-18T21:28:13.352 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:gXBItNBAdeLzISBJVjcY/bHUMIs= 2023-10-18T21:28:13.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:13 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gXBItNBAdeLzISBJVjcY/bHUMIs=', 'Content-Length': '0'} 2023-10-18T21:28:13.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d564d5e848c1555-0065304ded-2ba8-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:13.355 DEBUG:boto:path=/bofrnf-74/obj4 2023-10-18T21:28:13.356 DEBUG:boto:auth_path=/bofrnf-74/obj4 2023-10-18T21:28:13.356 DEBUG:boto:Method: PUT 2023-10-18T21:28:13.357 DEBUG:boto:Path: /bofrnf-74/obj4 2023-10-18T21:28:13.357 DEBUG:boto:Data: 2023-10-18T21:28:13.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:28:13.358 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:13.358 DEBUG:boto:Port: 8000 2023-10-18T21:28:13.359 DEBUG:boto:Params: {} 2023-10-18T21:28:13.359 DEBUG:boto:Token: None 2023-10-18T21:28:13.359 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 18 Oct 2023 21:28:13 GMT /bofrnf-74/obj4 2023-10-18T21:28:13.360 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:PufgGW/JE4t9xcR1Sz6ForlbKg0= 2023-10-18T21:28:13.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:28:13 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PufgGW/JE4t9xcR1Sz6ForlbKg0='} 2023-10-18T21:28:13.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000670d4b5617667ace-0065304ded-2ba8-a1'), ('Date', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:13.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.404 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:13.453 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:13.454 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:13.485 INFO:teuthology.orchestra.run.smithi082.stderr:No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:28:13.497 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:28:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664370.023158_2186.1", 2023-10-18T21:28:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:26:10.023158Z" 2023-10-18T21:28:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:28:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:28:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "0.000000" 2023-10-18T21:28:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:28:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.501 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:28:13.502 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:13.502 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-18T21:28:13.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.504 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:13.554 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:13.554 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:13.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:28:13.583+0000 7f8a4787f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:13.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-10-18T21:28:13.584+0000 7f8a4787f780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:13.597 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:28:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:28:13.598 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:28:13.598 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:28:13.598 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:28:13.598 INFO:teuthology.orchestra.run.smithi082.stdout: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:13.598 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.599 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:28:13.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.599 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:28:13.599 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.600 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:13.600 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697664370.023158_2186.1", 2023-10-18T21:28:13.600 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.600 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 1, 2023-10-18T21:28:13.600 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.601 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:26:10.023158Z", 2023-10-18T21:28:13.601 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.601 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.602 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:28:13.602 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.602 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:13.602 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:13.603 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.603 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:13.603 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.603 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:13.603 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.604 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:28:13.604 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.605 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:13.605 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "", 2023-10-18T21:28:13.605 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.605 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:13.605 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.606 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "0.000000", 2023-10-18T21:28:13.606 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.606 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.607 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:28:13.607 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.607 INFO:teuthology.orchestra.run.smithi082.stdout: "state": 1, 2023-10-18T21:28:13.607 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:13.607 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.608 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:13.608 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.608 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:13.608 INFO:teuthology.orchestra.run.smithi082.stdout: "realm_epoch": 2 2023-10-18T21:28:13.608 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.609 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:28:13.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.609 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:28:13.609 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 1, 2023-10-18T21:28:13.610 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 1 2023-10-18T21:28:13.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.610 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:28:13Z" 2023-10-18T21:28:13.610 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:28:13.611 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697664370.023158_2186.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-18T21:26:10.023158Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1697661951.933926_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.933926Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697661951.953788_570.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T20:45:51.953788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-18T21:28:13Z" } 2023-10-18T21:28:13.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664370.023158_2186.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:26:10.023158Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:13.612 DEBUG:rgw_multi.tests:log_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:13.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1697664370.023158_2186.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:13.613 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-18T21:28:13.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.614 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:13.662 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:13.663 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:13.711 INFO:teuthology.orchestra.run.smithi082.stdout:[ 2023-10-18T21:28:13.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.786 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:13.787 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:13.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.787 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-10-18T21:28:13.788 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:24:05.002600Z" 2023-10-18T21:28:13.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.788 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032764", 2023-10-18T21:28:13.788 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:28:13.365676Z" 2023-10-18T21:28:13.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.789 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052376", 2023-10-18T21:28:13.789 INFO:teuthology.orchestra.run.smithi082.stdout: "last_update": "2023-10-18T21:28:13.346520Z" 2023-10-18T21:28:13.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.790 INFO:teuthology.orchestra.run.smithi082.stdout:] 2023-10-18T21:28:13.791 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032764', 3: '00000000000000000000:00000000000000052376'} 2023-10-18T21:28:13.791 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:28:13.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:13.793 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:13.843 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:13.844 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:13.888 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:28:13.888 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:28:13.888 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:28:13.888 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:28:13.889 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:28:13.889 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:28:13.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.889 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:28:13.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.890 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:28:13.890 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.890 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:13.891 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:13.891 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.891 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:13.891 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.891 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:13.892 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.892 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:28:13.892 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.893 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:13.893 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-10-18T21:28:13.893 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.893 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:13.894 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.894 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:05.002600Z" 2023-10-18T21:28:13.894 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.895 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:28:13.895 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.895 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:13.897 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:13.897 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.898 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:28:13.898 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.898 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:13.898 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:13.899 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:28:13.899 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:13.899 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:13.899 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:28:13.900 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:13.900 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:28:13.900 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:13.900 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:28:13.901 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.901 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:13.901 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:28:13.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.901 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:28:13.901 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:28:13.902 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:28:13.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:13.902 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:28:13Z" 2023-10-18T21:28:13.902 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:28:13.903 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:24:05.002600Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052235", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:24:04.972607Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:28:13Z" } 2023-10-18T21:28:13.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:24:05.002600Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052235', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:24:04.972607Z'}}] 2023-10-18T21:28:13.904 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032764', 3: '00000000000000000000:00000000000000052376'} 2023-10-18T21:28:13.905 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:28:13.905 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032623, shard=3 master=00000000000000000000:00000000000000052376 target=00000000000000000000:00000000000000052235 2023-10-18T21:28:18.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:18.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:18.908 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:18.973 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:18.974 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:19.021 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:28:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:28:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:28:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:28:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:28:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:28:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:28:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:28:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:28:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-10-18T21:28:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:05.002600Z" 2023-10-18T21:28:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:28:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:28:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:28:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052235", 2023-10-18T21:28:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:28:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:04.972607Z" 2023-10-18T21:28:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:28:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:28:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:28:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:28:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:28:19Z" 2023-10-18T21:28:19.037 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:28:19.038 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:24:05.002600Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032623", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:21:54.480390Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052235", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:24:04.972607Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:28:19Z" } 2023-10-18T21:28:19.039 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:24:05.002600Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052235', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:24:04.972607Z'}}] 2023-10-18T21:28:19.039 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032764', 3: '00000000000000000000:00000000000000052376'} 2023-10-18T21:28:19.040 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052235'} 2023-10-18T21:28:19.040 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032623, shard=3 master=00000000000000000000:00000000000000052376 target=00000000000000000000:00000000000000052235 2023-10-18T21:28:24.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:24.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:24.043 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:24.102 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:24.103 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:24.152 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2023-10-18T21:28:24.152 INFO:teuthology.orchestra.run.smithi082.stdout: "sync_status": { 2023-10-18T21:28:24.153 INFO:teuthology.orchestra.run.smithi082.stdout: "info": { 2023-10-18T21:28:24.153 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "sync", 2023-10-18T21:28:24.153 INFO:teuthology.orchestra.run.smithi082.stdout: "num_shards": 4, 2023-10-18T21:28:24.153 INFO:teuthology.orchestra.run.smithi082.stdout: "instance_id": 5898477700290860314 2023-10-18T21:28:24.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:24.154 INFO:teuthology.orchestra.run.smithi082.stdout: "markers": [ 2023-10-18T21:28:24.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:24.154 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 0, 2023-10-18T21:28:24.154 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:24.155 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:24.155 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:24.155 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:24.155 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:24.155 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:24.156 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:24.156 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:24.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:24.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:24.156 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 1, 2023-10-18T21:28:24.157 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:24.157 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:24.157 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-10-18T21:28:24.157 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:24.158 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 0, 2023-10-18T21:28:24.158 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:24.158 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:24:05.002600Z" 2023-10-18T21:28:24.158 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:24.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:24.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:24.159 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 2, 2023-10-18T21:28:24.159 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:24.159 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:24.159 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000032764", 2023-10-18T21:28:24.160 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:24.160 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 9, 2023-10-18T21:28:24.160 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:24.160 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:28:13.365676Z" 2023-10-18T21:28:24.160 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:24.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:24.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2023-10-18T21:28:24.161 INFO:teuthology.orchestra.run.smithi082.stdout: "key": 3, 2023-10-18T21:28:24.161 INFO:teuthology.orchestra.run.smithi082.stdout: "val": { 2023-10-18T21:28:24.162 INFO:teuthology.orchestra.run.smithi082.stdout: "status": "incremental-sync", 2023-10-18T21:28:24.162 INFO:teuthology.orchestra.run.smithi082.stdout: "marker": "00000000000000000000:00000000000000052376", 2023-10-18T21:28:24.162 INFO:teuthology.orchestra.run.smithi082.stdout: "next_step_marker": "", 2023-10-18T21:28:24.162 INFO:teuthology.orchestra.run.smithi082.stdout: "total_entries": 55, 2023-10-18T21:28:24.162 INFO:teuthology.orchestra.run.smithi082.stdout: "pos": 0, 2023-10-18T21:28:24.163 INFO:teuthology.orchestra.run.smithi082.stdout: "timestamp": "2023-10-18T21:28:13.346520Z" 2023-10-18T21:28:24.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:24.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2023-10-18T21:28:24.163 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2023-10-18T21:28:24.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:24.164 INFO:teuthology.orchestra.run.smithi082.stdout: "full_sync": { 2023-10-18T21:28:24.164 INFO:teuthology.orchestra.run.smithi082.stdout: "total": 64, 2023-10-18T21:28:24.164 INFO:teuthology.orchestra.run.smithi082.stdout: "complete": 64 2023-10-18T21:28:24.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2023-10-18T21:28:24.164 INFO:teuthology.orchestra.run.smithi082.stdout: "current_time": "2023-10-18T21:28:24Z" 2023-10-18T21:28:24.165 INFO:teuthology.orchestra.run.smithi082.stdout:} 2023-10-18T21:28:24.166 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5898477700290860314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044466", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:21:20.727617Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-18T21:24:05.002600Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-18T21:28:13.365676Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052376", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-18T21:28:13.346520Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-18T21:28:24Z" } 2023-10-18T21:28:24.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:24:05.002600Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:28:13.365676Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052376', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:28:13.346520Z'}}] 2023-10-18T21:28:24.167 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032764', 3: '00000000000000000000:00000000000000052376'} 2023-10-18T21:28:24.167 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000046080', 2: '00000000000000000000:00000000000000032764', 3: '00000000000000000000:00000000000000052376'} 2023-10-18T21:28:24.168 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:28:24.168 DEBUG:boto:path=/bofrnf-74/ 2023-10-18T21:28:24.168 DEBUG:boto:auth_path=/bofrnf-74/ 2023-10-18T21:28:24.169 DEBUG:boto:Method: HEAD 2023-10-18T21:28:24.169 DEBUG:boto:Path: /bofrnf-74/ 2023-10-18T21:28:24.170 DEBUG:boto:Data: 2023-10-18T21:28:24.170 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.171 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.171 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.171 DEBUG:boto:Params: {} 2023-10-18T21:28:24.172 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:28:24.172 DEBUG:boto:Token: None 2023-10-18T21:28:24.173 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-74/ 2023-10-18T21:28:24.174 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:Uzdk3TgNmWmCIMWwHz2Ro6u5qo8= 2023-10-18T21:28:24.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uzdk3TgNmWmCIMWwHz2Ro6u5qo8=', 'Content-Length': '0'} 2023-10-18T21:28:24.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca184681d71c05b7-0065304df8-2bb4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.179 DEBUG:boto:path=/bofrnf-74/obj3 2023-10-18T21:28:24.180 DEBUG:boto:auth_path=/bofrnf-74/obj3 2023-10-18T21:28:24.180 DEBUG:boto:Method: HEAD 2023-10-18T21:28:24.181 DEBUG:boto:Path: /bofrnf-74/obj3 2023-10-18T21:28:24.181 DEBUG:boto:Data: 2023-10-18T21:28:24.181 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.182 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.182 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.183 DEBUG:boto:Params: {} 2023-10-18T21:28:24.183 DEBUG:boto:Token: None 2023-10-18T21:28:24.184 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-74/obj3 2023-10-18T21:28:24.184 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:lzjBFRJlb3txMx1vimTUoxaG0JE= 2023-10-18T21:28:24.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:lzjBFRJlb3txMx1vimTUoxaG0JE=', 'Content-Length': '0'} 2023-10-18T21:28:24.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be5be3e78c326983-0065304df8-2bb4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.188 DEBUG:boto:path=/bofrnf-74/obj3 2023-10-18T21:28:24.188 DEBUG:boto:auth_path=/bofrnf-74/obj3 2023-10-18T21:28:24.188 DEBUG:boto:Method: GET 2023-10-18T21:28:24.189 DEBUG:boto:Path: /bofrnf-74/obj3 2023-10-18T21:28:24.189 DEBUG:boto:Data: 2023-10-18T21:28:24.190 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.190 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.191 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.191 DEBUG:boto:Params: {} 2023-10-18T21:28:24.191 DEBUG:boto:Token: None 2023-10-18T21:28:24.192 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-74/obj3 2023-10-18T21:28:24.192 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dxmE6rpZMl/tjydKvX9cMs1ffZ8= 2023-10-18T21:28:24.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dxmE6rpZMl/tjydKvX9cMs1ffZ8=', 'Content-Length': '0'} 2023-10-18T21:28:24.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028e961b7c077d2cd-0065304df8-2bb4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.195 DEBUG:boto:path=/bofrnf-74/obj4 2023-10-18T21:28:24.196 DEBUG:boto:auth_path=/bofrnf-74/obj4 2023-10-18T21:28:24.196 DEBUG:boto:Method: HEAD 2023-10-18T21:28:24.197 DEBUG:boto:Path: /bofrnf-74/obj4 2023-10-18T21:28:24.197 DEBUG:boto:Data: 2023-10-18T21:28:24.198 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.198 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.199 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.199 DEBUG:boto:Params: {} 2023-10-18T21:28:24.200 DEBUG:boto:Token: None 2023-10-18T21:28:24.200 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-74/obj4 2023-10-18T21:28:24.200 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:IzDzm3kfZg307gg0QVv9f4fZi9k= 2023-10-18T21:28:24.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IzDzm3kfZg307gg0QVv9f4fZi9k=', 'Content-Length': '0'} 2023-10-18T21:28:24.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6ed69203d6e7ba9-0065304df8-2bb4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.203 DEBUG:boto:path=/bofrnf-74/obj4 2023-10-18T21:28:24.204 DEBUG:boto:auth_path=/bofrnf-74/obj4 2023-10-18T21:28:24.204 DEBUG:boto:Method: GET 2023-10-18T21:28:24.205 DEBUG:boto:Path: /bofrnf-74/obj4 2023-10-18T21:28:24.205 DEBUG:boto:Data: 2023-10-18T21:28:24.205 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.206 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.206 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.207 DEBUG:boto:Params: {} 2023-10-18T21:28:24.207 DEBUG:boto:Token: None 2023-10-18T21:28:24.208 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-74/obj4 2023-10-18T21:28:24.208 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:GcwE7hmn4OvTL4sARVcwA41R6q8= 2023-10-18T21:28:24.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GcwE7hmn4OvTL4sARVcwA41R6q8=', 'Content-Length': '0'} 2023-10-18T21:28:24.210 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009251dc58af6d5cb5-0065304df8-2bb4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.254 DEBUG:boto:path=/bofrnf-75/ 2023-10-18T21:28:24.255 DEBUG:boto:auth_path=/bofrnf-75/ 2023-10-18T21:28:24.255 DEBUG:boto:Method: HEAD 2023-10-18T21:28:24.255 DEBUG:boto:Path: /bofrnf-75/ 2023-10-18T21:28:24.256 DEBUG:boto:Data: 2023-10-18T21:28:24.256 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.257 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.257 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.257 DEBUG:boto:Params: {} 2023-10-18T21:28:24.258 DEBUG:boto:Token: None 2023-10-18T21:28:24.258 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-75/ 2023-10-18T21:28:24.259 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:K9P6J005KiKFIDi2OlV4pMXVEB8= 2023-10-18T21:28:24.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:K9P6J005KiKFIDi2OlV4pMXVEB8=', 'Content-Length': '0'} 2023-10-18T21:28:24.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bcd6171c23cf620-0065304df8-2bb4-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.262 DEBUG:boto:path=/bofrnf-75/obj3 2023-10-18T21:28:24.262 DEBUG:boto:auth_path=/bofrnf-75/obj3 2023-10-18T21:28:24.263 DEBUG:boto:Method: HEAD 2023-10-18T21:28:24.263 DEBUG:boto:Path: /bofrnf-75/obj3 2023-10-18T21:28:24.263 DEBUG:boto:Data: 2023-10-18T21:28:24.264 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.264 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.265 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.265 DEBUG:boto:Params: {} 2023-10-18T21:28:24.265 DEBUG:boto:Token: None 2023-10-18T21:28:24.266 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-75/obj3 2023-10-18T21:28:24.266 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:wicRBLz7buDvLKl/A2MlRfU98pE= 2023-10-18T21:28:24.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wicRBLz7buDvLKl/A2MlRfU98pE=', 'Content-Length': '0'} 2023-10-18T21:28:24.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003357ad1734ebc741-0065304df8-2bb4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.269 DEBUG:boto:path=/bofrnf-75/obj3 2023-10-18T21:28:24.269 DEBUG:boto:auth_path=/bofrnf-75/obj3 2023-10-18T21:28:24.270 DEBUG:boto:Method: GET 2023-10-18T21:28:24.270 DEBUG:boto:Path: /bofrnf-75/obj3 2023-10-18T21:28:24.271 DEBUG:boto:Data: 2023-10-18T21:28:24.271 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.271 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.272 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.272 DEBUG:boto:Params: {} 2023-10-18T21:28:24.273 DEBUG:boto:Token: None 2023-10-18T21:28:24.273 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-75/obj3 2023-10-18T21:28:24.274 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:tHaOXJQgZwWd5o4cJ8cihEJT61A= 2023-10-18T21:28:24.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tHaOXJQgZwWd5o4cJ8cihEJT61A=', 'Content-Length': '0'} 2023-10-18T21:28:24.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc5ede1d7f192207-0065304df8-2bb4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.277 DEBUG:boto:path=/bofrnf-75/obj4 2023-10-18T21:28:24.277 DEBUG:boto:auth_path=/bofrnf-75/obj4 2023-10-18T21:28:24.278 DEBUG:boto:Method: HEAD 2023-10-18T21:28:24.278 DEBUG:boto:Path: /bofrnf-75/obj4 2023-10-18T21:28:24.278 DEBUG:boto:Data: 2023-10-18T21:28:24.279 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.279 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.280 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.280 DEBUG:boto:Params: {} 2023-10-18T21:28:24.280 DEBUG:boto:Token: None 2023-10-18T21:28:24.281 DEBUG:boto:StringToSign: HEAD Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-75/obj4 2023-10-18T21:28:24.281 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:dz2DO24ATXe2xyife42HHajzN/s= 2023-10-18T21:28:24.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dz2DO24ATXe2xyife42HHajzN/s=', 'Content-Length': '0'} 2023-10-18T21:28:24.284 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000836bc82624959f54-0065304df8-2bb4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.284 DEBUG:boto:path=/bofrnf-75/obj4 2023-10-18T21:28:24.285 DEBUG:boto:auth_path=/bofrnf-75/obj4 2023-10-18T21:28:24.285 DEBUG:boto:Method: GET 2023-10-18T21:28:24.285 DEBUG:boto:Path: /bofrnf-75/obj4 2023-10-18T21:28:24.286 DEBUG:boto:Data: 2023-10-18T21:28:24.286 DEBUG:boto:Headers: {} 2023-10-18T21:28:24.287 DEBUG:boto:Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.287 DEBUG:boto:Port: 8001 2023-10-18T21:28:24.288 DEBUG:boto:Params: {} 2023-10-18T21:28:24.288 DEBUG:boto:Token: None 2023-10-18T21:28:24.288 DEBUG:boto:StringToSign: GET Wed, 18 Oct 2023 21:28:24 GMT /bofrnf-75/obj4 2023-10-18T21:28:24.289 DEBUG:boto:Signature: AWS 6TC2VRGERC8I036KRP19:0Z9wBBypL+qjZm0jYPn2MklEghs= 2023-10-18T21:28:24.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:28:24 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0Z9wBBypL+qjZm0jYPn2MklEghs=', 'Content-Length': '0'} 2023-10-18T21:28:24.291 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:28:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-74:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10564.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2ec41f47e0ee913-0065304df8-2bb4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:28:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:24.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'bofrnf-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:24.293 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket bofrnf-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:24.343 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:24.343 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:28:24.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:24.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:24.400 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:24.447 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setuser ceph since I am not root 2023-10-18T21:28:24.448 INFO:teuthology.orchestra.run.smithi082.stderr:ignoring --setgroup ceph since I am not root 2023-10-18T21:28:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:{"sync_policy":{"groups":[]}} 2023-10-18T21:28:24.494 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2023-10-18T21:28:24.495 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.496 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-18T21:28:24.496 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2023-10-18T21:28:24.496 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-18T21:28:24.498 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-18T21:28:24.498 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-18T21:28:24.498 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-18T21:28:24.499 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 1418, in test_bucket_reshard_index_log_trim 2023-10-18T21:28:24.499 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2023-10-18T21:28:24.499 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-18T21:28:24.500 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-18T21:28:24.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=bofrnf-55 2023-10-18T21:28:24.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:24.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:24.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.504 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:24.504 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.504 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.504 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:00 GMT 2023-10-18T21:28:24.505 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.505 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2uVPweGQnzfHFl3I8uWC/RUj2wE= 2023-10-18T21:28:24.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2uVPweGQnzfHFl3I8uWC/RUj2wE=', 'Content-Length': '0'} 2023-10-18T21:28:24.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000056104c21b7acc6b4-0065303e2c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:24.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.510 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.510 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.510 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.511 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:00 GMT 2023-10-18T21:28:24.511 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.512 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:x377Kl0Gb5i1GiRmpoCoxu2taWQ= 2023-10-18T21:28:24.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x377Kl0Gb5i1GiRmpoCoxu2taWQ=', 'Content-Length': '0'} 2023-10-18T21:28:24.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058af3ac74dbc75bb-0065303e2c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:24.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:24.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:24.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.516 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:24.516 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:24.517 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:24.517 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:00 GMT 2023-10-18T21:28:24.517 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:24.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.518 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Uln18/VYbHswwKUJGTxbB8W1Q30= 2023-10-18T21:28:24.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Uln18/VYbHswwKUJGTxbB8W1Q30='} 2023-10-18T21:28:24.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a016bd9be9b1def-0065303e2c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:24.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.522 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.522 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.523 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.523 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:00 GMT 2023-10-18T21:28:24.523 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.524 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:x377Kl0Gb5i1GiRmpoCoxu2taWQ= 2023-10-18T21:28:24.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:x377Kl0Gb5i1GiRmpoCoxu2taWQ=', 'Content-Length': '0'} 2023-10-18T21:28:24.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069ac03f711c23f08-0065303e2c-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:24.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:24.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:24.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.528 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:24.529 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:24.529 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:24.529 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:00 GMT 2023-10-18T21:28:24.529 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:24.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.530 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ZnamhqBVicTE0gDzmaajR2YDFfc= 2023-10-18T21:28:24.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:00 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZnamhqBVicTE0gDzmaajR2YDFfc='} 2023-10-18T21:28:24.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000975b2a83e71e3cf3-0065303e2c-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:24.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.535 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.535 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.535 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.536 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:01 GMT 2023-10-18T21:28:24.536 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.536 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rzZFnI8aZWQOCdxxA4sKxGYXa8o= 2023-10-18T21:28:24.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzZFnI8aZWQOCdxxA4sKxGYXa8o=', 'Content-Length': '0'} 2023-10-18T21:28:24.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086d70e9986a51ab9-0065303e2d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:24.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:24.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:24.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.541 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:24.541 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:24.541 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:24.542 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:01 GMT 2023-10-18T21:28:24.542 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:24.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.543 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:V+auCOajrI8u8tDxlNs3u3xGVhY= 2023-10-18T21:28:24.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:V+auCOajrI8u8tDxlNs3u3xGVhY='} 2023-10-18T21:28:24.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da5351333ea643d2-0065303e2d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:24.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.547 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.547 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.547 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.548 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:01 GMT 2023-10-18T21:28:24.548 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.549 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rzZFnI8aZWQOCdxxA4sKxGYXa8o= 2023-10-18T21:28:24.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rzZFnI8aZWQOCdxxA4sKxGYXa8o=', 'Content-Length': '0'} 2023-10-18T21:28:24.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f2be196f721e975-0065303e2d-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:24.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:24.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:24.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:24.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:24.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.553 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:24.553 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:24.554 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:24.554 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:01 GMT 2023-10-18T21:28:24.554 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:24.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.555 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:nUiO0AGtFyBsrDanrKv59PKuf3s= 2023-10-18T21:28:24.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:01 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nUiO0AGtFyBsrDanrKv59PKuf3s='} 2023-10-18T21:28:24.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d03f1398795df2f-0065303e2d-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:24.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:28:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [ 2023-10-18T21:28:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697660460.928951_896.1", 2023-10-18T21:28:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:21:00.928951Z" 2023-10-18T21:28:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697658841.435004_114.1", 2023-10-18T21:28:24.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T19:54:01.435004Z" 2023-10-18T21:28:24.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:24.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "0.000000" 2023-10-18T21:28:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:24.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:24.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "0.000000" 2023-10-18T21:28:24.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:24.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:24.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697660460.928951_896.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T21:28:24.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-18T21:28:24.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:24.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:01.254+0000 7f6b7ab59780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:01.255+0000 7f6b7ab59780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:24.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:24.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697660460.928951_896.1", 2023-10-18T21:28:24.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:24.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:24.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:24.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:21:00.928951Z", 2023-10-18T21:28:24.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:24.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:24.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:24.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:24.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:24.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:24.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:24.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697658841.435004_114.1", 2023-10-18T21:28:24.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:24.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T21:28:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:24.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:24.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:24.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:24.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:24.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:24.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:24.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:24.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:24.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:24.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:24.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:24.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:24.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:24.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:24.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T20:21:01Z" 2023-10-18T21:28:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:24.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:24.586 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:24.587 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:24.587 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:24.587 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:24.588 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:24.588 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:24.588 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:24.588 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:24.589 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:24.589 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:24.589 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:24.590 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:24.590 INFO:tasks.rgw_multisite_tests: "marker": "1_1697660460.928951_896.1", 2023-10-18T21:28:24.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:24.591 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:24.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:24.591 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:21:00.928951Z", 2023-10-18T21:28:24.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:24.592 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:24.592 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:24.592 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:24.593 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:24.593 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:24.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:24.594 INFO:tasks.rgw_multisite_tests: "marker": "1_1697658841.435004_114.1", 2023-10-18T21:28:24.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:24.594 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:24.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:24.595 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T19:54:01.435004Z", 2023-10-18T21:28:24.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:24.595 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:24.596 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:24.596 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:24.596 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:24.597 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:24.597 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:24.597 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:24.597 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:24.598 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:24.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:24.598 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:24.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:24.599 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:24.599 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:24.600 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:24.600 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:24.600 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:24.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:24.601 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:24.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:24.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:24.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:24.602 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:24.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:24.603 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:24.603 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:24.603 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:24.604 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:24.604 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:24.604 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:24.605 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:24.605 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:24.605 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T20:21:01Z" 2023-10-18T21:28:24.605 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:24.606 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697660460.928951_896.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:21:00.928951Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697658841.435004_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T19:54:01.435004Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-18T21:28:24.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697660460.928951_896.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T21:28:24.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697660460.928951_896.1', 1: '1_1697658841.435004_114.1', 2: '', 3: ''} 2023-10-18T21:28:24.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-18T21:28:24.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=bofrnf-55 2023-10-18T21:28:24.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:24.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:01.379+0000 7fdfbc03a780 1 waiting to reach incremental sync.. 2023-10-18T21:28:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:06.380+0000 7fdfbc03a780 1 waiting to reach incremental sync.. 2023-10-18T21:28:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:11.381+0000 7fdfbc03a780 1 waiting to reach incremental sync.. 2023-10-18T21:28:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:16.382+0000 7fdfbc03a780 1 waiting to reach incremental sync.. 2023-10-18T21:28:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:21.383+0000 7fdfbc03a780 1 bucket sync caught up with source: 2023-10-18T21:28:24.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000001.22.6, , , , 00000000001.53.6, , , 00000000001.62.6, , 00000000001.24.6] 2023-10-18T21:28:24.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000001.22.6, , , , 00000000001.53.6, , , 00000000001.62.6, , 00000000001.24.6] 2023-10-18T21:28:24.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:21.383+0000 7fdfbc03a780 0 bucket checkpoint complete 2023-10-18T21:28:24.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=bofrnf-55 2023-10-18T21:28:24.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:24.612 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:24.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:24.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:24.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:24.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:21.498+0000 7f4d58d54780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:24.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , ] 2023-10-18T21:28:24.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T21:28:24.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:26.500+0000 7f4d58d54780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:24.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , ] 2023-10-18T21:28:24.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T21:28:24.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:31.501+0000 7f4d58d54780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:24.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , ] 2023-10-18T21:28:24.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T21:28:24.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:36.502+0000 7f4d58d54780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:24.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , ] 2023-10-18T21:28:24.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T21:28:24.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:41.503+0000 7f4d58d54780 1 bucket sync caught up with source: 2023-10-18T21:28:24.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T21:28:24.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000001.47.6, , , , 00000000001.595.6, , , 00000000001.476.6, , 00000000001.33.6] 2023-10-18T21:28:24.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:41.503+0000 7f4d58d54780 0 bucket checkpoint complete 2023-10-18T21:28:24.618 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=bofrnf-55 zones={a2, a1} 2023-10-18T21:28:24.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:24.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.622 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.622 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.623 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.623 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.623 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.624 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:EzCimMfrmwk8njidnEYeicj0yLo= 2023-10-18T21:28:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EzCimMfrmwk8njidnEYeicj0yLo=', 'Content-Length': '0'} 2023-10-18T21:28:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c35b62503bcea76-0065303e55-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:24.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.628 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.628 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.629 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.629 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.629 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.630 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:EzCimMfrmwk8njidnEYeicj0yLo= 2023-10-18T21:28:24.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EzCimMfrmwk8njidnEYeicj0yLo=', 'Content-Length': '0'} 2023-10-18T21:28:24.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc8fe1cba2267e2b-0065303e55-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.631 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-18T21:28:24.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.635 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.636 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.636 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.636 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.636 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:24.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.637 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts= 2023-10-18T21:28:24.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts=', 'Content-Length': '0'} 2023-10-18T21:28:24.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a32928af39ce04b-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.638 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-18T21:28:24.639 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-18T21:28:24.639 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-18T21:28:24.639 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-18T21:28:24.640 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-18T21:28:24.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.644 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.644 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.645 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.645 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.645 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:24.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.646 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts= 2023-10-18T21:28:24.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts=', 'Content-Length': '0'} 2023-10-18T21:28:24.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b2158e088204886-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.647 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-18T21:28:24.647 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-18T21:28:24.648 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-18T21:28:24.648 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-18T21:28:24.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.652 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.652 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.653 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.653 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.653 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:24.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.654 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts= 2023-10-18T21:28:24.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts=', 'Content-Length': '0'} 2023-10-18T21:28:24.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fcfd26a2ac0cb92-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:24.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.659 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.660 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.660 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.660 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.660 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:24.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.661 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts= 2023-10-18T21:28:24.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qXe+rEb0Z5mEQVgVri+m7OXm5ts=', 'Content-Length': '0'} 2023-10-18T21:28:24.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d9a28acf1892d7-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.662 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-18T21:28:24.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:24.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.667 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.667 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.667 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.667 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.668 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.668 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY= 2023-10-18T21:28:24.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY=', 'Content-Length': '0'} 2023-10-18T21:28:24.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0280921e841cbb5-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:24.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.676 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.677 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.677 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.677 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.678 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.678 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY= 2023-10-18T21:28:24.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY=', 'Content-Length': '0'} 2023-10-18T21:28:24.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005019f9547219a8af-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:24.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.683 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.684 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.684 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.684 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.684 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.685 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:27sd5rlqNlya999wZlrtqmqG6XA= 2023-10-18T21:28:24.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:27sd5rlqNlya999wZlrtqmqG6XA=', 'Content-Length': '0'} 2023-10-18T21:28:24.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bd19b49ad0fdab4-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:24.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.690 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.690 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.691 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.691 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.691 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.692 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:27sd5rlqNlya999wZlrtqmqG6XA= 2023-10-18T21:28:24.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:27sd5rlqNlya999wZlrtqmqG6XA=', 'Content-Length': '0'} 2023-10-18T21:28:24.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1566f945e18b60d-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.693 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-18T21:28:24.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:24.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.697 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.697 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.698 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.698 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.698 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.699 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY= 2023-10-18T21:28:24.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY=', 'Content-Length': '0'} 2023-10-18T21:28:24.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008128dd1abef6c92f-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:24.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.704 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.704 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.704 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.705 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.705 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:24.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.706 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY= 2023-10-18T21:28:24.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:O/r8P1EXgpmj6269G8fJdDipLhY=', 'Content-Length': '0'} 2023-10-18T21:28:24.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000129555e14a1d8c65-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:24.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:24.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:24.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.711 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.711 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.711 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.711 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.712 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.712 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T21:28:24.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T21:28:24.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074dc656e0f9e5be8-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:24.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:24.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:24.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.718 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.718 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.718 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.718 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.719 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.719 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T21:28:24.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T21:28:24.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f50170520ee83b26-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:24.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.724 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.724 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.725 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.725 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.725 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:24.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.726 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:CMAlwgVDuXBGLv0lqaY9oiuEDDU= 2023-10-18T21:28:24.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CMAlwgVDuXBGLv0lqaY9oiuEDDU=', 'Content-Length': '0'} 2023-10-18T21:28:24.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18a19b53d16b663-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:24.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.730 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.730 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.731 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.731 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.731 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:24.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.732 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:CMAlwgVDuXBGLv0lqaY9oiuEDDU= 2023-10-18T21:28:24.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CMAlwgVDuXBGLv0lqaY9oiuEDDU=', 'Content-Length': '0'} 2023-10-18T21:28:24.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f633762180734e2-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.733 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-18T21:28:24.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:24.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.737 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.737 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.737 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.737 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.738 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:24.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.738 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:72ljtWOcoL0aZ9ikI5eW95ta+so= 2023-10-18T21:28:24.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:72ljtWOcoL0aZ9ikI5eW95ta+so=', 'Content-Length': '0'} 2023-10-18T21:28:24.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a544c997790d8301-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:24.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:24.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:24.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.743 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.743 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.743 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.743 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.744 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:24.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.744 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:72ljtWOcoL0aZ9ikI5eW95ta+so= 2023-10-18T21:28:24.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:72ljtWOcoL0aZ9ikI5eW95ta+so=', 'Content-Length': '0'} 2023-10-18T21:28:24.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000750b9676c35d792-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:24.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:24.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:24.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.749 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.749 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.750 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.750 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.750 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.751 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T21:28:24.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T21:28:24.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc178e45eca0c6f3-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:24.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:24.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:24.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.756 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.756 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.757 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.757 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.757 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.758 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T21:28:24.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T21:28:24.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000493a4de580b8468c-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.759 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-18T21:28:24.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:24.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.764 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.764 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.764 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.764 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.765 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.765 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4= 2023-10-18T21:28:24.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4=', 'Content-Length': '0'} 2023-10-18T21:28:24.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad4ab715d34bed4d-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:24.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.770 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.770 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.771 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.771 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.771 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.772 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4= 2023-10-18T21:28:24.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4=', 'Content-Length': '0'} 2023-10-18T21:28:24.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf79866089006e2-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:24.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.777 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.777 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.777 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.778 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.778 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.779 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:5VtbzXtJ+fpIpyfpwqzVr8RKDKU= 2023-10-18T21:28:24.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5VtbzXtJ+fpIpyfpwqzVr8RKDKU=', 'Content-Length': '0'} 2023-10-18T21:28:24.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd021948b5c2462-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:24.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.784 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.784 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.784 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.785 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.785 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.785 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:5VtbzXtJ+fpIpyfpwqzVr8RKDKU= 2023-10-18T21:28:24.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5VtbzXtJ+fpIpyfpwqzVr8RKDKU=', 'Content-Length': '0'} 2023-10-18T21:28:24.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e5f45c0083e1923-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.786 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-18T21:28:24.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:24.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.791 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.791 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.791 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.792 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.792 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.792 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4= 2023-10-18T21:28:24.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4=', 'Content-Length': '0'} 2023-10-18T21:28:24.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d9d435668f7593-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:24.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.797 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.798 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.798 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.798 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.799 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:24.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.799 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4= 2023-10-18T21:28:24.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xj/xY8ZWwaBNgUlmMzrF0lcFUS4=', 'Content-Length': '0'} 2023-10-18T21:28:24.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea0a0092c549751-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:24.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:24.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:24.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.804 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.804 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.805 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.805 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.805 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.806 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T21:28:24.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T21:28:24.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b18aa87746605b4-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:24.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:24.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:24.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.811 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.811 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.812 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.812 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.812 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.813 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T21:28:24.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T21:28:24.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1faf967e02f13d0-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:24.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.818 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.818 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.818 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.819 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.819 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:24.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.819 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KxPoZahOkpYAEAD4wmb+vEpGtX4= 2023-10-18T21:28:24.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KxPoZahOkpYAEAD4wmb+vEpGtX4=', 'Content-Length': '0'} 2023-10-18T21:28:24.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007978a015d6cb4d33-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:24.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.824 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.824 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.824 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.825 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.825 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:24.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.825 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KxPoZahOkpYAEAD4wmb+vEpGtX4= 2023-10-18T21:28:24.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KxPoZahOkpYAEAD4wmb+vEpGtX4=', 'Content-Length': '0'} 2023-10-18T21:28:24.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb0a38942c009f8-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.826 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-18T21:28:24.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:24.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.830 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.830 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.831 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.831 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.831 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:24.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.832 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1P3yaw8yvtthhxTp1kxo7HQ1nvs= 2023-10-18T21:28:24.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1P3yaw8yvtthhxTp1kxo7HQ1nvs=', 'Content-Length': '0'} 2023-10-18T21:28:24.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005379c903731241bb-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:24.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:24.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:24.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.836 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.836 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.837 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.837 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.837 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:24.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.838 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1P3yaw8yvtthhxTp1kxo7HQ1nvs= 2023-10-18T21:28:24.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1P3yaw8yvtthhxTp1kxo7HQ1nvs=', 'Content-Length': '0'} 2023-10-18T21:28:24.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4364a99dfe1248e-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:24.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:24.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:24.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.843 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.843 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.843 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.844 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.844 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.844 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T21:28:24.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T21:28:24.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000638f2a568a0e2ede-0065303e55-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:24.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:24.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:24.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.850 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.850 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.850 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.851 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.851 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.852 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124= 2023-10-18T21:28:24.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yqpc8YyaTPgIGDcF0u3bB6Ro124=', 'Content-Length': '0'} 2023-10-18T21:28:24.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071bcaecb6dfe663b-0065303e55-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.853 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-18T21:28:24.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:24.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.857 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.858 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.858 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.858 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.859 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.859 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:41ovE1ZvU5vrcUdqnwnIkbnXp5c= 2023-10-18T21:28:24.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:41ovE1ZvU5vrcUdqnwnIkbnXp5c=', 'Content-Length': '0'} 2023-10-18T21:28:24.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c7889b9496baf6-0065303e55-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:24.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.864 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.865 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.865 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.865 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:41 GMT 2023-10-18T21:28:24.865 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.866 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:41ovE1ZvU5vrcUdqnwnIkbnXp5c= 2023-10-18T21:28:24.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:41ovE1ZvU5vrcUdqnwnIkbnXp5c=', 'Content-Length': '0'} 2023-10-18T21:28:24.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea14ee9be02a8f56-0065303e55-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:24.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.871 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.871 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.872 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.872 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.872 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.873 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:5FMFMzt189MfHXhS1GI5hC0E6mA= 2023-10-18T21:28:24.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5FMFMzt189MfHXhS1GI5hC0E6mA=', 'Content-Length': '0'} 2023-10-18T21:28:24.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f01a3afed3e1afb-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:24.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.878 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.878 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.878 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.879 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.879 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.880 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:5FMFMzt189MfHXhS1GI5hC0E6mA= 2023-10-18T21:28:24.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5FMFMzt189MfHXhS1GI5hC0E6mA=', 'Content-Length': '0'} 2023-10-18T21:28:24.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005670f21fc88dbc83-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.880 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-18T21:28:24.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:24.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.885 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.885 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.885 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.886 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.886 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.887 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:57H4m8JGBgxh3fs0PZ9tFQHXbO0= 2023-10-18T21:28:24.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:57H4m8JGBgxh3fs0PZ9tFQHXbO0=', 'Content-Length': '0'} 2023-10-18T21:28:24.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009554d2b092579d3c-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:24.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.891 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.892 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.892 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.892 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.893 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:24.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.893 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:57H4m8JGBgxh3fs0PZ9tFQHXbO0= 2023-10-18T21:28:24.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:57H4m8JGBgxh3fs0PZ9tFQHXbO0=', 'Content-Length': '0'} 2023-10-18T21:28:24.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3da0683ce699d2-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:24.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:24.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:24.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.898 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.898 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.899 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.899 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.899 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.900 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T21:28:24.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T21:28:24.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d9262fe74fb463c-0065303e56-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:24.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:24.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:24.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.905 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.905 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.906 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.906 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.906 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.907 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T21:28:24.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T21:28:24.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d053b65a8e866bd6-0065303e56-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:24.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.911 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.912 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.912 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.912 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.913 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:24.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.913 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:dehh8R/IuGyB+l3H1IbvP2p3JXk= 2023-10-18T21:28:24.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dehh8R/IuGyB+l3H1IbvP2p3JXk=', 'Content-Length': '0'} 2023-10-18T21:28:24.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca36833d58edc82f-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:24.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.918 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.918 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.918 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.919 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.919 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:24.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.919 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:dehh8R/IuGyB+l3H1IbvP2p3JXk= 2023-10-18T21:28:24.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dehh8R/IuGyB+l3H1IbvP2p3JXk=', 'Content-Length': '0'} 2023-10-18T21:28:24.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c55160e32e1d8580-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.920 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-18T21:28:24.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:24.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.924 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.924 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.925 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.925 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.925 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:24.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.926 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:bxzv+VNCtBgbxxPKvuMjYItgnjg= 2023-10-18T21:28:24.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxzv+VNCtBgbxxPKvuMjYItgnjg=', 'Content-Length': '0'} 2023-10-18T21:28:24.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000918048bf345124ff-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:24.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:24.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:24.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.930 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.930 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.931 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.931 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.931 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:24.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.932 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:bxzv+VNCtBgbxxPKvuMjYItgnjg= 2023-10-18T21:28:24.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxzv+VNCtBgbxxPKvuMjYItgnjg=', 'Content-Length': '0'} 2023-10-18T21:28:24.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a9dea0fa09239f-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:24.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:24.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:24.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.937 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.937 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.937 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.938 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.938 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.939 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T21:28:24.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T21:28:24.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e5f03ac86590839-0065303e56-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:24.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:24.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:24.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.944 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.944 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.945 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.945 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.945 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.946 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T21:28:24.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T21:28:24.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be8e8c1886e81ae1-0065303e56-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.947 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-18T21:28:24.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:24.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:24.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:24.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.951 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.951 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.952 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.952 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.952 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.953 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM= 2023-10-18T21:28:24.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM=', 'Content-Length': '0'} 2023-10-18T21:28:24.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096e7039955a0fd33-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:24.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:24.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:24.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.958 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.958 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.958 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.959 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.959 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.960 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM= 2023-10-18T21:28:24.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM=', 'Content-Length': '0'} 2023-10-18T21:28:24.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c0ba3cdf06d9d2-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:24.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:24.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:24.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.964 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.965 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.965 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.965 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.966 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.966 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:OlowGsMXdnKwod4Th7Des/4DaRU= 2023-10-18T21:28:24.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlowGsMXdnKwod4Th7Des/4DaRU=', 'Content-Length': '0'} 2023-10-18T21:28:24.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001332fe4ba4de79aa-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:24.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:24.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:24.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:24.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.971 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:24.972 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.972 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.972 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.972 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.973 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:OlowGsMXdnKwod4Th7Des/4DaRU= 2023-10-18T21:28:24.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OlowGsMXdnKwod4Th7Des/4DaRU=', 'Content-Length': '0'} 2023-10-18T21:28:24.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31fbe1de578c189-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.974 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-18T21:28:24.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:24.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:24.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:24.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.978 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.979 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.979 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.979 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.979 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.980 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM= 2023-10-18T21:28:24.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM=', 'Content-Length': '0'} 2023-10-18T21:28:24.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a84f5d8143ccc41e-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:24.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:24.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:24.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.985 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.985 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.986 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.986 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.986 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:24.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.987 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM= 2023-10-18T21:28:24.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVT2nyKxGDjASGmY+vHW1KIVbtM=', 'Content-Length': '0'} 2023-10-18T21:28:24.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4f2acefdca7421c-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:24.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:24.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:24.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:24.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:24.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.992 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.992 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.992 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.993 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:24.993 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:24.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:24.993 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T21:28:24.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T21:28:24.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec3769eaa346aa3-0065303e56-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:24.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:24.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:24.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:24.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:24.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:24.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:24.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:24.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:24.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:24.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:24.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:24.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:24.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:24.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:24.999 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:24.999 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.999 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:24.999 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.000 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.000 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T21:28:25.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T21:28:25.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028326a75eb0dd4a3-0065303e56-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:25.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.005 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.005 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.005 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.006 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.006 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:25.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.007 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:/uIDX+rs7FfLM+UZguUTHSYXJDQ= 2023-10-18T21:28:25.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/uIDX+rs7FfLM+UZguUTHSYXJDQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc3d92d60bf355d8-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:25.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.011 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.011 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.012 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.012 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.012 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:25.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.013 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:/uIDX+rs7FfLM+UZguUTHSYXJDQ= 2023-10-18T21:28:25.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/uIDX+rs7FfLM+UZguUTHSYXJDQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f4aecac62e76e0-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.014 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-18T21:28:25.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:25.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.017 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.018 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.018 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.018 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.019 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:25.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.019 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:wbUwzp6z1qbVhDtEabXHt9v8Wpw= 2023-10-18T21:28:25.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wbUwzp6z1qbVhDtEabXHt9v8Wpw=', 'Content-Length': '0'} 2023-10-18T21:28:25.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a479ee35f1a9e9-0065303e56-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:25.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.023 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.024 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.024 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.024 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.025 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:25.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.025 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:wbUwzp6z1qbVhDtEabXHt9v8Wpw= 2023-10-18T21:28:25.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wbUwzp6z1qbVhDtEabXHt9v8Wpw=', 'Content-Length': '0'} 2023-10-18T21:28:25.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000947af33f560632b3-0065303e56-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:25.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.030 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.031 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.031 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.031 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.031 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.032 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T21:28:25.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T21:28:25.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d96be79510cf446-0065303e56-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:25.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.037 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.037 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.038 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.038 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.038 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.039 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8= 2023-10-18T21:28:25.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rWd25YyrtUXCxIMl9D/nhA+YkJ8=', 'Content-Length': '0'} 2023-10-18T21:28:25.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009db5466ad971893d-0065303e56-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.040 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=bofrnf-55 zones={a2, a1} 2023-10-18T21:28:25.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding": "None", 2023-10-18T21:28:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_index": { 2023-10-18T21:28:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 0, 2023-10-18T21:28:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "Normal", 2023-10-18T21:28:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "normal": { 2023-10-18T21:28:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 11, 2023-10-18T21:28:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "logs": [ 2023-10-18T21:28:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 0, 2023-10-18T21:28:25.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:25.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:25.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 0, 2023-10-18T21:28:25.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 11, 2023-10-18T21:28:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:25.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.051 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bofrnf-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:25.051 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bofrnf-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bofrnf-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [ 2023-10-18T21:28:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op_id": "1#00000000001.22.6", 2023-10-18T21:28:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.6485583521340136943", 2023-10-18T21:28:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op": "write", 2023-10-18T21:28:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "object": "a", 2023-10-18T21:28:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance": "", 2023-10-18T21:28:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": "complete", 2023-10-18T21:28:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "index_ver": 1, 2023-10-18T21:28:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:21:00.946035145Z", 2023-10-18T21:28:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "ver": { 2023-10-18T21:28:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pool": 9, 2023-10-18T21:28:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "epoch": 5 2023-10-18T21:28:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bilog_flags": 0, 2023-10-18T21:28:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "versioned": false, 2023-10-18T21:28:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "owner": "", 2023-10-18T21:28:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "owner_display_name": "", 2023-10-18T21:28:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones_trace": [ 2023-10-18T21:28:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24" 2023-10-18T21:28:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op_id": "5#00000000001.53.6", 2023-10-18T21:28:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.7855387404454453039", 2023-10-18T21:28:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op": "write", 2023-10-18T21:28:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "object": "d", 2023-10-18T21:28:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance": "", 2023-10-18T21:28:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": "complete", 2023-10-18T21:28:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "index_ver": 1, 2023-10-18T21:28:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:21:01.038076432Z", 2023-10-18T21:28:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "ver": { 2023-10-18T21:28:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pool": 9, 2023-10-18T21:28:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "epoch": 3 2023-10-18T21:28:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bilog_flags": 0, 2023-10-18T21:28:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "versioned": false, 2023-10-18T21:28:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "owner": "", 2023-10-18T21:28:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "owner_display_name": "", 2023-10-18T21:28:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones_trace": [ 2023-10-18T21:28:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24" 2023-10-18T21:28:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op_id": "8#00000000001.62.6", 2023-10-18T21:28:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.15732007203871540178", 2023-10-18T21:28:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op": "write", 2023-10-18T21:28:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "object": "c", 2023-10-18T21:28:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance": "", 2023-10-18T21:28:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": "complete", 2023-10-18T21:28:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "index_ver": 1, 2023-10-18T21:28:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:21:01.019275931Z", 2023-10-18T21:28:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "ver": { 2023-10-18T21:28:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pool": 9, 2023-10-18T21:28:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "epoch": 3 2023-10-18T21:28:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bilog_flags": 0, 2023-10-18T21:28:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "versioned": false, 2023-10-18T21:28:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "owner": "", 2023-10-18T21:28:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "owner_display_name": "", 2023-10-18T21:28:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones_trace": [ 2023-10-18T21:28:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24" 2023-10-18T21:28:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op_id": "10#00000000001.24.6", 2023-10-18T21:28:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op_tag": "403a921b-fcfc-4ab2-b434-bf29d35a51b6.5732.10906357668614388979", 2023-10-18T21:28:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "op": "write", 2023-10-18T21:28:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "object": "b", 2023-10-18T21:28:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance": "", 2023-10-18T21:28:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": "complete", 2023-10-18T21:28:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "index_ver": 1, 2023-10-18T21:28:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:21:01.001180380Z", 2023-10-18T21:28:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "ver": { 2023-10-18T21:28:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pool": 9, 2023-10-18T21:28:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "epoch": 1 2023-10-18T21:28:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bilog_flags": 0, 2023-10-18T21:28:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "versioned": false, 2023-10-18T21:28:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "owner": "", 2023-10-18T21:28:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "owner_display_name": "", 2023-10-18T21:28:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones_trace": [ 2023-10-18T21:28:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "entry": "403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24" 2023-10-18T21:28:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:25.081 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'bofrnf-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'bofrnf-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket bofrnf-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-55 2023-10-18T21:28:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 4 2023-10-18T21:28:25.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding": "None", 2023-10-18T21:28:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_index": { 2023-10-18T21:28:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 1, 2023-10-18T21:28:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "Normal", 2023-10-18T21:28:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "normal": { 2023-10-18T21:28:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "logs": [ 2023-10-18T21:28:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 0, 2023-10-18T21:28:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:25.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:25.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 0, 2023-10-18T21:28:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 11, 2023-10-18T21:28:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 1, 2023-10-18T21:28:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 1, 2023-10-18T21:28:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bofrnf-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:25.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bofrnf-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bofrnf-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [] 2023-10-18T21:28:25.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:25.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.104 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.104 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.104 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.105 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.105 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.106 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI= 2023-10-18T21:28:25.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI=', 'Content-Length': '0'} 2023-10-18T21:28:25.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000754800ce3f3ba7bc-0065303e56-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:25.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e 2023-10-18T21:28:25.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:25.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.110 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:25.110 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:25.110 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:25.111 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.111 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e 2023-10-18T21:28:25.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.112 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:GyTRN6mm4QW8oa+rsTZYaUXbjcM= 2023-10-18T21:28:25.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GyTRN6mm4QW8oa+rsTZYaUXbjcM='} 2023-10-18T21:28:25.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090d8adb23f86ed37-0065303e56-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:25.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.116 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.116 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.116 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.117 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.117 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.118 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI= 2023-10-18T21:28:25.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI=', 'Content-Length': '0'} 2023-10-18T21:28:25.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051eed752673acffc-0065303e56-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:25.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f 2023-10-18T21:28:25.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:25.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.122 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:25.122 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:25.122 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:25.123 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.123 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f 2023-10-18T21:28:25.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.124 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:IC+1wr+iUxmCHm2pB5srWC6JWlY= 2023-10-18T21:28:25.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IC+1wr+iUxmCHm2pB5srWC6JWlY='} 2023-10-18T21:28:25.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000946c0b97b12c85b9-0065303e56-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:25.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.128 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.128 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.128 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.129 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.129 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.130 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI= 2023-10-18T21:28:25.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2eI2uwjPBpwlSRC5mT1Jh4qUsqI=', 'Content-Length': '0'} 2023-10-18T21:28:25.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090557a50764eec5c-0065303e56-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:25.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g 2023-10-18T21:28:25.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:25.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.134 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:25.134 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:25.134 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:25.135 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:42 GMT 2023-10-18T21:28:25.135 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g 2023-10-18T21:28:25.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.136 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:trWagEhN+Z4ye27tKucu+Tsyc2w= 2023-10-18T21:28:25.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:trWagEhN+Z4ye27tKucu+Tsyc2w='} 2023-10-18T21:28:25.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5da45b902c93e52-0065303e56-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:25.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.140 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.140 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.140 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.141 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:43 GMT 2023-10-18T21:28:25.141 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.142 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6Y0T0shiz+ICcGEeDEYb8kX+gGU= 2023-10-18T21:28:25.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:21:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6Y0T0shiz+ICcGEeDEYb8kX+gGU=', 'Content-Length': '0'} 2023-10-18T21:28:25.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018f11b17168ef4e1-0065303e57-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:25.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h 2023-10-18T21:28:25.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:25.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.146 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:25.146 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:25.146 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:25.147 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:21:43 GMT 2023-10-18T21:28:25.147 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h 2023-10-18T21:28:25.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.148 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:coFmLIDH4CbCEGkgeynAPaRhoTY= 2023-10-18T21:28:25.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:21:43 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:coFmLIDH4CbCEGkgeynAPaRhoTY='} 2023-10-18T21:28:25.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e92a86e8296f51a-0065303e57-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=bofrnf-55 2023-10-18T21:28:25.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:25.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:43.107+0000 7f5400246780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T21:28:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:48.107+0000 7f5400246780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T21:28:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:53.107+0000 7f5400246780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T21:28:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:21:58.108+0000 7f5400246780 1 waiting to reach latest gen 1, on 0.. 2023-10-18T21:28:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:03.110+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, , ] 2023-10-18T21:28:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:08.111+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, , ] 2023-10-18T21:28:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:13.112+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, , ] 2023-10-18T21:28:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:18.113+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, , ] 2023-10-18T21:28:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:23.113+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:28.114+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:33.115+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:38.115+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:43.116+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:48.116+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:53.118+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:22:58.119+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:03.120+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:08.121+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:13.122+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:18.122+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:23.123+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:28.124+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:33.124+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:38.125+0000 7f5400246780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:43.126+0000 7f5400246780 1 bucket sync caught up with source: 2023-10-18T21:28:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [00000000003.46.6, 00000000002.37.6, 00000000002.95.6] 2023-10-18T21:28:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:43.126+0000 7f5400246780 0 bucket checkpoint complete 2023-10-18T21:28:25.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=bofrnf-55 2023-10-18T21:28:25.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:25.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:43.243+0000 7f353e08f780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000001.33.6] 2023-10-18T21:28:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T21:28:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:48.245+0000 7f353e08f780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000001.33.6] 2023-10-18T21:28:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T21:28:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:53.247+0000 7f353e08f780 1 waiting for incremental sync to catch up: 2023-10-18T21:28:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000001.33.6] 2023-10-18T21:28:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T21:28:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:58.248+0000 7f353e08f780 1 bucket sync caught up with source: 2023-10-18T21:28:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T21:28:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T21:28:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:23:58.248+0000 7f353e08f780 0 bucket checkpoint complete 2023-10-18T21:28:25.178 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=bofrnf-55 zones={a2, a1} 2023-10-18T21:28:25.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:25.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:28:25.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.182 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.183 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.183 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.183 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.184 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.184 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:f8J6oA462jMsqfFlKCxGlZ03jWg= 2023-10-18T21:28:25.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f8J6oA462jMsqfFlKCxGlZ03jWg=', 'Content-Length': '0'} 2023-10-18T21:28:25.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005eb0e81be5ead3d1-0065303ede-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:25.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:28:25.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.189 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.189 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.190 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.190 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.190 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.191 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:f8J6oA462jMsqfFlKCxGlZ03jWg= 2023-10-18T21:28:25.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:f8J6oA462jMsqfFlKCxGlZ03jWg=', 'Content-Length': '0'} 2023-10-18T21:28:25.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060ec187054c20c08-0065303ede-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.192 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-18T21:28:25.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.196 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.196 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.197 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.197 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.197 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:25.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.198 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU= 2023-10-18T21:28:25.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU=', 'Content-Length': '0'} 2023-10-18T21:28:25.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004833dab8e7eb3522-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.199 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-18T21:28:25.199 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-18T21:28:25.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-18T21:28:25.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-18T21:28:25.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-18T21:28:25.201 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-18T21:28:25.201 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-18T21:28:25.201 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-18T21:28:25.201 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-18T21:28:25.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.206 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.206 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.206 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.207 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.207 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:25.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.207 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU= 2023-10-18T21:28:25.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU=', 'Content-Length': '0'} 2023-10-18T21:28:25.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000877859b63eb4671b-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.209 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-18T21:28:25.209 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-18T21:28:25.209 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-18T21:28:25.210 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-18T21:28:25.210 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-18T21:28:25.210 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-18T21:28:25.211 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-18T21:28:25.211 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-18T21:28:25.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.215 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.215 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.216 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.216 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.216 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:25.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.217 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU= 2023-10-18T21:28:25.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU=', 'Content-Length': '0'} 2023-10-18T21:28:25.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fca159179740267-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:25.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.222 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.222 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.223 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.223 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.223 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:25.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.224 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU= 2023-10-18T21:28:25.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8F3h9vG8iwEmVpMTNvhu4qzVwBU=', 'Content-Length': '0'} 2023-10-18T21:28:25.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003330c97c4335ab4b-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.225 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-18T21:28:25.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:25.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.229 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.230 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.230 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.230 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.231 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.231 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM= 2023-10-18T21:28:25.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM=', 'Content-Length': '0'} 2023-10-18T21:28:25.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6bf22d9e23a641-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:25.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.236 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.236 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.237 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.237 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.237 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.238 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM= 2023-10-18T21:28:25.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM=', 'Content-Length': '0'} 2023-10-18T21:28:25.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b6751e78ac8ea3d-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:25.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.243 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.243 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.243 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.244 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.244 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.245 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:mds7q/yUg+cQBVJSEMLvMtV1TWI= 2023-10-18T21:28:25.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mds7q/yUg+cQBVJSEMLvMtV1TWI=', 'Content-Length': '0'} 2023-10-18T21:28:25.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005689680253f2b795-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:25.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.250 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.250 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.250 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.251 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.251 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.251 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:mds7q/yUg+cQBVJSEMLvMtV1TWI= 2023-10-18T21:28:25.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mds7q/yUg+cQBVJSEMLvMtV1TWI=', 'Content-Length': '0'} 2023-10-18T21:28:25.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002441e7adb49b6923-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.252 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-18T21:28:25.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:25.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.257 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.257 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.257 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.257 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.258 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.258 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM= 2023-10-18T21:28:25.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM=', 'Content-Length': '0'} 2023-10-18T21:28:25.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed3de3c63f75eeb7-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:25.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.263 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.263 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.264 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.264 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.264 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:25.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.265 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM= 2023-10-18T21:28:25.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mu9wSuYpbwwvP6E1nxHimar23WM=', 'Content-Length': '0'} 2023-10-18T21:28:25.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbdf432b0deb3953-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:25.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:25.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:25.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.270 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.270 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.271 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.271 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.271 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.272 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a2cd8aae3f4f2a-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:25.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:25.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:25.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.277 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.277 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.278 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.278 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.278 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.279 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef4fea989229a130-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:25.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.283 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.284 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.284 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.284 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.285 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:25.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.285 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:XYIJc8CVeP+7mipjKmcAuxo4vSk= 2023-10-18T21:28:25.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XYIJc8CVeP+7mipjKmcAuxo4vSk=', 'Content-Length': '0'} 2023-10-18T21:28:25.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebd81d05c3598f3-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:25.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.289 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.290 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.290 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.290 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.291 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:25.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.291 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:XYIJc8CVeP+7mipjKmcAuxo4vSk= 2023-10-18T21:28:25.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:XYIJc8CVeP+7mipjKmcAuxo4vSk=', 'Content-Length': '0'} 2023-10-18T21:28:25.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b364a417db127c23-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.292 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-18T21:28:25.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:25.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.296 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.296 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.296 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.297 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.297 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:25.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.298 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KeCUNxiv1G4GrF+n2z4FEAW035M= 2023-10-18T21:28:25.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KeCUNxiv1G4GrF+n2z4FEAW035M=', 'Content-Length': '0'} 2023-10-18T21:28:25.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a736e715ee21c711-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:25.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:25.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:25.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.302 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.302 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.303 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.303 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.303 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:25.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.304 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KeCUNxiv1G4GrF+n2z4FEAW035M= 2023-10-18T21:28:25.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KeCUNxiv1G4GrF+n2z4FEAW035M=', 'Content-Length': '0'} 2023-10-18T21:28:25.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea8a926f3500e3cc-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:25.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:25.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:25.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.309 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.309 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.309 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.310 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.310 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.311 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008be1e3c086d9b22b-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:25.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:25.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:25.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.316 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.316 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.316 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.317 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.317 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.317 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1f2c860acbb17b4-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.319 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-18T21:28:25.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:25.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.323 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.323 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.324 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.324 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.324 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.325 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g= 2023-10-18T21:28:25.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g=', 'Content-Length': '0'} 2023-10-18T21:28:25.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de10cfa24b1fa0be-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:25.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.330 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.330 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.330 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.330 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.331 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.331 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g= 2023-10-18T21:28:25.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g=', 'Content-Length': '0'} 2023-10-18T21:28:25.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b96991f0a0f57c-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:25.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.336 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.337 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.337 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.337 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.337 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.338 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z2A1w71eJB8L1ovmdhHOGpIY6t8= 2023-10-18T21:28:25.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z2A1w71eJB8L1ovmdhHOGpIY6t8=', 'Content-Length': '0'} 2023-10-18T21:28:25.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7eee11c68726c7-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:25.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.343 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.343 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.344 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.344 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.344 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.345 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z2A1w71eJB8L1ovmdhHOGpIY6t8= 2023-10-18T21:28:25.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z2A1w71eJB8L1ovmdhHOGpIY6t8=', 'Content-Length': '0'} 2023-10-18T21:28:25.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007559d99ee65a5b0-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.346 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-18T21:28:25.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:25.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.350 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.350 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.351 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.351 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.351 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.352 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g= 2023-10-18T21:28:25.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g=', 'Content-Length': '0'} 2023-10-18T21:28:25.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849ecec599b9292a-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:25.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.357 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.357 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.357 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.357 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.358 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:25.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.358 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g= 2023-10-18T21:28:25.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KBF6xjhVngkgxE5pp+3ER3VTA7g=', 'Content-Length': '0'} 2023-10-18T21:28:25.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cee3574b23660161-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:25.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:25.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:25.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.363 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.364 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.364 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.364 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.364 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.365 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a66d9255ad90c51d-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:25.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:25.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:25.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.370 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.370 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.371 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.371 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.371 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.372 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7331a7cbb56ba4d-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:25.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.377 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.377 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.377 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.378 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.378 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:25.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.378 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:qd598u9h42m0FbFvGW7dDqUn30g= 2023-10-18T21:28:25.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qd598u9h42m0FbFvGW7dDqUn30g=', 'Content-Length': '0'} 2023-10-18T21:28:25.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c72f1632ac02081-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:25.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.383 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.383 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.383 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.384 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.384 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:25.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.385 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:qd598u9h42m0FbFvGW7dDqUn30g= 2023-10-18T21:28:25.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qd598u9h42m0FbFvGW7dDqUn30g=', 'Content-Length': '0'} 2023-10-18T21:28:25.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011fbeb689c48d9cd-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.385 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-18T21:28:25.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:25.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.389 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.389 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.390 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.390 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.390 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:25.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.391 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8soovHvLKFAjoFsWC5uDD+YKud4= 2023-10-18T21:28:25.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8soovHvLKFAjoFsWC5uDD+YKud4=', 'Content-Length': '0'} 2023-10-18T21:28:25.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea8dcf9c5e42f5d-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:25.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:25.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:25.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.395 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.396 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.396 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.396 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.397 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:25.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.397 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8soovHvLKFAjoFsWC5uDD+YKud4= 2023-10-18T21:28:25.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8soovHvLKFAjoFsWC5uDD+YKud4=', 'Content-Length': '0'} 2023-10-18T21:28:25.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae018d3b97bb055f-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:25.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:25.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:25.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.402 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.402 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.403 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.403 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.403 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.404 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b07f7ea042c1dab-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:25.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:25.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:25.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.409 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.409 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.410 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.410 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.410 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.411 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000255deedb5e61c5ed-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.412 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-18T21:28:25.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:25.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.416 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.417 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.417 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.417 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.418 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.418 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ= 2023-10-18T21:28:25.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ce3bec564ac3a9-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:25.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.423 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.423 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.424 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.424 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.424 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.425 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ= 2023-10-18T21:28:25.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025305484fb3926e3-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:25.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.430 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.430 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.430 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.431 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.431 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.431 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:7Yrpln1bqgj64a8dGB85nT1seco= 2023-10-18T21:28:25.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7Yrpln1bqgj64a8dGB85nT1seco=', 'Content-Length': '0'} 2023-10-18T21:28:25.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc76c32a805cd20-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:25.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.436 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.437 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.437 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.437 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.438 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.438 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:7Yrpln1bqgj64a8dGB85nT1seco= 2023-10-18T21:28:25.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7Yrpln1bqgj64a8dGB85nT1seco=', 'Content-Length': '0'} 2023-10-18T21:28:25.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a10c7453ed39dfb-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.439 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-18T21:28:25.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:25.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.443 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.444 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.444 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.444 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.445 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.445 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ= 2023-10-18T21:28:25.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d307facc171e2f-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:25.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.450 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.451 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.451 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.451 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.451 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:25.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.452 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ= 2023-10-18T21:28:25.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JwMsnSABCutO3ZsH7/4EH9hOqfQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000951b9ed7bc46d6ed-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:25.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:25.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:25.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.457 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.457 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.457 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.458 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.458 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.459 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62dff7773096b39-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:25.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:25.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:25.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.464 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.464 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.465 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.465 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.465 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.466 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000145554170ab26a27-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:25.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.470 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.471 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.471 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.471 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.472 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:25.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.472 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ccGS0r8Ie0YPSFUBVynwu7MXtgo= 2023-10-18T21:28:25.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ccGS0r8Ie0YPSFUBVynwu7MXtgo=', 'Content-Length': '0'} 2023-10-18T21:28:25.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e13c25246267e192-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:25.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.477 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.477 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.477 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.477 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.478 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:25.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.478 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ccGS0r8Ie0YPSFUBVynwu7MXtgo= 2023-10-18T21:28:25.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ccGS0r8Ie0YPSFUBVynwu7MXtgo=', 'Content-Length': '0'} 2023-10-18T21:28:25.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000163d6630590ab3fb-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.479 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-18T21:28:25.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:25.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.483 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.483 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.484 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.484 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.484 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:25.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.485 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:nXYQBHClFfnxdpyqIt6bcQzRuCQ= 2023-10-18T21:28:25.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nXYQBHClFfnxdpyqIt6bcQzRuCQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef69a4fe4415599-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:25.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:25.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:25.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.489 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.489 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.490 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.490 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.490 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:25.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.491 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:nXYQBHClFfnxdpyqIt6bcQzRuCQ= 2023-10-18T21:28:25.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nXYQBHClFfnxdpyqIt6bcQzRuCQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e387901d23f23ae7-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:25.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:25.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:25.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.496 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.496 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.496 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.497 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.497 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.498 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb9707a32293ff6f-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:25.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:25.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:25.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.503 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.503 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.503 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.504 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.504 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.505 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6b0c19c8232d57-0065303ede-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.506 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-18T21:28:25.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:25.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.510 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.510 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.511 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.511 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.511 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.512 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY= 2023-10-18T21:28:25.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY=', 'Content-Length': '0'} 2023-10-18T21:28:25.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051cb853f4a055d5a-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:25.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.517 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.517 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.517 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.518 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.518 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.519 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY= 2023-10-18T21:28:25.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY=', 'Content-Length': '0'} 2023-10-18T21:28:25.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd8b9e175c451c9e-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:25.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.524 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.524 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.524 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.525 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.525 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.526 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:TUvRScOz5i5q8iRORQ2eJLNXnCc= 2023-10-18T21:28:25.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TUvRScOz5i5q8iRORQ2eJLNXnCc=', 'Content-Length': '0'} 2023-10-18T21:28:25.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697fcac8e3b72d8f-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:25.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.531 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.531 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.531 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.532 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.532 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.532 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:TUvRScOz5i5q8iRORQ2eJLNXnCc= 2023-10-18T21:28:25.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:TUvRScOz5i5q8iRORQ2eJLNXnCc=', 'Content-Length': '0'} 2023-10-18T21:28:25.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004909b62f150a0819-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.533 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-18T21:28:25.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:25.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.538 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.538 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.538 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.539 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.539 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.539 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY= 2023-10-18T21:28:25.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY=', 'Content-Length': '0'} 2023-10-18T21:28:25.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000306a6a5bf330878d-0065303ede-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:25.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.544 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.545 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.545 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.545 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.545 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:25.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.546 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY= 2023-10-18T21:28:25.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZKJn8gDDiTLKanH8NxJN3IsJhqY=', 'Content-Length': '0'} 2023-10-18T21:28:25.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002608140d20991d10-0065303ede-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:25.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.551 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.551 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.552 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.552 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:58 GMT 2023-10-18T21:28:25.552 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.553 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg= 2023-10-18T21:28:25.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oZtwEw9IjV/6oiffY4/cIYg7cmg=', 'Content-Length': '0'} 2023-10-18T21:28:25.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9722fbfa996c98-0065303ede-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:25.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.558 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.561 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.562 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.562 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.562 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.563 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cac177959c254ae-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:25.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.567 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.568 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.568 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.568 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.569 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:25.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.569 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:RizmJPOGJYJl5V8rD1Gnc1ktREs= 2023-10-18T21:28:25.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RizmJPOGJYJl5V8rD1Gnc1ktREs=', 'Content-Length': '0'} 2023-10-18T21:28:25.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047820ab1b7113bc0-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:25.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.574 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.574 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.574 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.574 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.575 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:25.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.575 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:RizmJPOGJYJl5V8rD1Gnc1ktREs= 2023-10-18T21:28:25.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RizmJPOGJYJl5V8rD1Gnc1ktREs=', 'Content-Length': '0'} 2023-10-18T21:28:25.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2194d6aadf14847-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.576 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-18T21:28:25.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:25.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.580 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.580 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.581 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.581 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.581 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:25.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.582 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4WdJ3/FR3ND+1/C4eXv3zmjlBLs= 2023-10-18T21:28:25.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4WdJ3/FR3ND+1/C4eXv3zmjlBLs=', 'Content-Length': '0'} 2023-10-18T21:28:25.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a23d9316681379f6-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:25.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:25.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:25.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.586 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.587 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.587 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.587 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.587 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:25.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.588 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4WdJ3/FR3ND+1/C4eXv3zmjlBLs= 2023-10-18T21:28:25.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4WdJ3/FR3ND+1/C4eXv3zmjlBLs=', 'Content-Length': '0'} 2023-10-18T21:28:25.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e6683433f074e1-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:25.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.593 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.593 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.594 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.594 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.594 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.595 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b7f36de4bda7824-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:25.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:25.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.600 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.600 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.600 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.601 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.601 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.602 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb8c907da4950bfd-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.603 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-18T21:28:25.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:25.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.607 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.607 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.608 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.608 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.608 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.609 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM= 2023-10-18T21:28:25.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM=', 'Content-Length': '0'} 2023-10-18T21:28:25.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6270d5aa322b8a-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:25.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.614 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.614 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.615 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.615 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.615 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.616 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM= 2023-10-18T21:28:25.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM=', 'Content-Length': '0'} 2023-10-18T21:28:25.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d3530cea960d1a-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:25.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.621 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.621 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.621 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.622 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.622 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.622 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:26t2Ce9Uj8E3AgFALSm2UbjBwV8= 2023-10-18T21:28:25.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:26t2Ce9Uj8E3AgFALSm2UbjBwV8=', 'Content-Length': '0'} 2023-10-18T21:28:25.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4df156bb0a6994-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:25.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.627 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.628 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.628 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.628 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.629 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.629 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:26t2Ce9Uj8E3AgFALSm2UbjBwV8= 2023-10-18T21:28:25.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:26t2Ce9Uj8E3AgFALSm2UbjBwV8=', 'Content-Length': '0'} 2023-10-18T21:28:25.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035746ce7076269e6-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.630 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-18T21:28:25.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:25.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.634 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.635 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.635 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.635 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.636 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.636 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM= 2023-10-18T21:28:25.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM=', 'Content-Length': '0'} 2023-10-18T21:28:25.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001709b1f21edc9f3b-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:25.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.641 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.641 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.642 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.642 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.642 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:25.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.643 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM= 2023-10-18T21:28:25.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CZjLMVn+4oyLzMm+mwX+4p0xaSM=', 'Content-Length': '0'} 2023-10-18T21:28:25.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd90dec48de190b-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=e 2023-10-18T21:28:25.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=e 2023-10-18T21:28:25.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=e 2023-10-18T21:28:25.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.648 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.648 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.648 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.648 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.649 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.649 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014899c4e5ac3b0e8-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=e 2023-10-18T21:28:25.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=e 2023-10-18T21:28:25.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=e 2023-10-18T21:28:25.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.655 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.655 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.655 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.656 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.656 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.656 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c4f197508e48918-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e 2023-10-18T21:28:25.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.661 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.661 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.662 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.662 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.662 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e 2023-10-18T21:28:25.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.663 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:l2gOwWUenThOHdxRbD8UorjpVDY= 2023-10-18T21:28:25.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:l2gOwWUenThOHdxRbD8UorjpVDY=', 'Content-Length': '0'} 2023-10-18T21:28:25.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004140a2def0083957-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e 2023-10-18T21:28:25.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.667 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.667 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.668 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.668 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.668 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e 2023-10-18T21:28:25.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.669 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:l2gOwWUenThOHdxRbD8UorjpVDY= 2023-10-18T21:28:25.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:l2gOwWUenThOHdxRbD8UorjpVDY=', 'Content-Length': '0'} 2023-10-18T21:28:25.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae54690d162ac4c3-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.670 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-18T21:28:25.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e 2023-10-18T21:28:25.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.673 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.674 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.674 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.674 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.675 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e 2023-10-18T21:28:25.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.675 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:noTU/pTZhNnxJfHH3puJQYiir7o= 2023-10-18T21:28:25.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:noTU/pTZhNnxJfHH3puJQYiir7o=', 'Content-Length': '0'} 2023-10-18T21:28:25.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c0906ce1917af8a-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:25.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:25.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e 2023-10-18T21:28:25.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.680 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.680 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.680 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.680 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.681 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e 2023-10-18T21:28:25.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.681 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:noTU/pTZhNnxJfHH3puJQYiir7o= 2023-10-18T21:28:25.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:noTU/pTZhNnxJfHH3puJQYiir7o=', 'Content-Length': '0'} 2023-10-18T21:28:25.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c176e77bc4d67321-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=e 2023-10-18T21:28:25.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=e 2023-10-18T21:28:25.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=e 2023-10-18T21:28:25.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.686 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.687 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.687 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.687 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.688 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.688 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d66545d9cfe896-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=e 2023-10-18T21:28:25.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=e 2023-10-18T21:28:25.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=e 2023-10-18T21:28:25.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.693 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.693 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.694 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.694 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.694 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.695 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2b0450cfa4cf4f2-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.696 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-18T21:28:25.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:25.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.700 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.701 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.701 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.701 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.702 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.702 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU= 2023-10-18T21:28:25.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU=', 'Content-Length': '0'} 2023-10-18T21:28:25.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9f400b1fce70b90-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:25.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.707 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.707 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.708 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.708 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.708 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.709 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU= 2023-10-18T21:28:25.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU=', 'Content-Length': '0'} 2023-10-18T21:28:25.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b02d459c17b7b4fe-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:25.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.714 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.714 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.714 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.715 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.715 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.716 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:jGynnztRkhaxIuzinxsiIyVrE1Y= 2023-10-18T21:28:25.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jGynnztRkhaxIuzinxsiIyVrE1Y=', 'Content-Length': '0'} 2023-10-18T21:28:25.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5a7ee4d917fe622-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:25.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.720 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.721 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.721 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.721 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.722 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.722 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:jGynnztRkhaxIuzinxsiIyVrE1Y= 2023-10-18T21:28:25.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jGynnztRkhaxIuzinxsiIyVrE1Y=', 'Content-Length': '0'} 2023-10-18T21:28:25.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa1246dba38ee23d-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.723 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-18T21:28:25.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:25.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.727 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.728 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.728 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.728 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.729 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.729 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU= 2023-10-18T21:28:25.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU=', 'Content-Length': '0'} 2023-10-18T21:28:25.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ec5345dcecea4fa-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:25.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.734 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.734 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.735 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.735 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.735 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:25.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.736 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU= 2023-10-18T21:28:25.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:rnwFWYyWI7/40QJgK1qGYYjyosU=', 'Content-Length': '0'} 2023-10-18T21:28:25.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b791356fde548bdb-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=f 2023-10-18T21:28:25.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=f 2023-10-18T21:28:25.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=f 2023-10-18T21:28:25.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.741 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.741 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.741 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.742 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.742 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.743 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e860933b9745c231-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=f 2023-10-18T21:28:25.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=f 2023-10-18T21:28:25.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=f 2023-10-18T21:28:25.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.748 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.748 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.749 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.749 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.749 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.750 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049b4af5f43f9a0a8-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f 2023-10-18T21:28:25.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.754 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.755 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.755 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.755 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.756 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f 2023-10-18T21:28:25.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.756 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:/dX0LLV4/ypNDP67ZcxOu9s6d04= 2023-10-18T21:28:25.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/dX0LLV4/ypNDP67ZcxOu9s6d04=', 'Content-Length': '0'} 2023-10-18T21:28:25.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d181d83001352d4-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f 2023-10-18T21:28:25.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.760 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.761 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.761 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.761 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.762 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f 2023-10-18T21:28:25.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.762 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:/dX0LLV4/ypNDP67ZcxOu9s6d04= 2023-10-18T21:28:25.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/dX0LLV4/ypNDP67ZcxOu9s6d04=', 'Content-Length': '0'} 2023-10-18T21:28:25.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752c0ea56a3cacc5-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.763 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-18T21:28:25.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f 2023-10-18T21:28:25.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.767 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.767 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.767 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.768 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.768 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f 2023-10-18T21:28:25.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.769 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:h7Z5rAbpNeiYAD1AM36dUtAm8XM= 2023-10-18T21:28:25.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h7Z5rAbpNeiYAD1AM36dUtAm8XM=', 'Content-Length': '0'} 2023-10-18T21:28:25.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e778487d794954-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:25.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:25.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f 2023-10-18T21:28:25.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.773 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.773 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.773 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.774 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.774 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f 2023-10-18T21:28:25.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.775 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:h7Z5rAbpNeiYAD1AM36dUtAm8XM= 2023-10-18T21:28:25.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h7Z5rAbpNeiYAD1AM36dUtAm8XM=', 'Content-Length': '0'} 2023-10-18T21:28:25.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c179b96d3715b773-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=f 2023-10-18T21:28:25.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=f 2023-10-18T21:28:25.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=f 2023-10-18T21:28:25.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.779 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.780 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.780 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.780 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.781 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.781 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089bda83c5debefd1-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=f 2023-10-18T21:28:25.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=f 2023-10-18T21:28:25.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=f 2023-10-18T21:28:25.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.786 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.787 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.787 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.787 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.788 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.788 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000165021f4f48a885b-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-18T21:28:25.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:25.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.794 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.794 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.794 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.795 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.795 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.796 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4= 2023-10-18T21:28:25.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4=', 'Content-Length': '0'} 2023-10-18T21:28:25.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053fd9b340dbbd5c7-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:25.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.801 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.801 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.801 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.801 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.802 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.802 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4= 2023-10-18T21:28:25.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4=', 'Content-Length': '0'} 2023-10-18T21:28:25.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080df66b79225e746-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:25.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.807 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.808 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.808 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.808 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.808 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.809 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:SHxbxTKPLzw3HPv60rBex12FfvQ= 2023-10-18T21:28:25.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SHxbxTKPLzw3HPv60rBex12FfvQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004112121f0e80634d-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:25.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.814 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.814 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.815 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.815 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.815 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.816 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:SHxbxTKPLzw3HPv60rBex12FfvQ= 2023-10-18T21:28:25.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SHxbxTKPLzw3HPv60rBex12FfvQ=', 'Content-Length': '0'} 2023-10-18T21:28:25.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dee9962d776226ca-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.817 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-18T21:28:25.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:25.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.821 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.821 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.822 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.822 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.822 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.823 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4= 2023-10-18T21:28:25.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4=', 'Content-Length': '0'} 2023-10-18T21:28:25.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483ca418e9818aeb-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:25.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.828 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.828 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.828 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.829 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.829 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:25.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.829 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4= 2023-10-18T21:28:25.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1Y6QeLxktH10W0uJYL3mHgPJ1j4=', 'Content-Length': '0'} 2023-10-18T21:28:25.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6632788d5e6fed0-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=g 2023-10-18T21:28:25.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=g 2023-10-18T21:28:25.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=g 2023-10-18T21:28:25.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.834 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.835 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.835 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.835 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.836 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.836 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044a3f570bdfed105-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=g 2023-10-18T21:28:25.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=g 2023-10-18T21:28:25.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=g 2023-10-18T21:28:25.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.841 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.841 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.842 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.842 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.842 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.843 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005049e7de0704f91f-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g 2023-10-18T21:28:25.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.848 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.848 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.848 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.848 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.849 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g 2023-10-18T21:28:25.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.849 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:GiMOUh0TGs9ftUDj57xa9Ln3gTE= 2023-10-18T21:28:25.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GiMOUh0TGs9ftUDj57xa9Ln3gTE=', 'Content-Length': '0'} 2023-10-18T21:28:25.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c412d7d45264e0-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g 2023-10-18T21:28:25.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.854 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.854 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.854 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.855 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.855 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g 2023-10-18T21:28:25.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.855 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:GiMOUh0TGs9ftUDj57xa9Ln3gTE= 2023-10-18T21:28:25.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GiMOUh0TGs9ftUDj57xa9Ln3gTE=', 'Content-Length': '0'} 2023-10-18T21:28:25.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b82e4090e883c1f-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-18T21:28:25.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g 2023-10-18T21:28:25.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.860 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.860 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.861 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.861 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.861 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g 2023-10-18T21:28:25.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.862 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:GLPR4Av3J//GKbAnIE3HPsyHiGE= 2023-10-18T21:28:25.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GLPR4Av3J//GKbAnIE3HPsyHiGE=', 'Content-Length': '0'} 2023-10-18T21:28:25.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006696f78f6853af74-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:25.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:25.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g 2023-10-18T21:28:25.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.866 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.866 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.867 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.867 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.867 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g 2023-10-18T21:28:25.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.868 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:GLPR4Av3J//GKbAnIE3HPsyHiGE= 2023-10-18T21:28:25.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GLPR4Av3J//GKbAnIE3HPsyHiGE=', 'Content-Length': '0'} 2023-10-18T21:28:25.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008979b3f284457427-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=g 2023-10-18T21:28:25.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=g 2023-10-18T21:28:25.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=g 2023-10-18T21:28:25.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.873 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.873 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.873 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.874 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.874 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.875 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a173eb2c7cdc7329-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=g 2023-10-18T21:28:25.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=g 2023-10-18T21:28:25.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=g 2023-10-18T21:28:25.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.880 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.880 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.880 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.881 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.881 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.881 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d741b8fe5aad225a-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.883 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-18T21:28:25.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:25.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.887 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.887 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.888 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.888 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.888 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.889 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss= 2023-10-18T21:28:25.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss=', 'Content-Length': '0'} 2023-10-18T21:28:25.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041848d2cc48fde29-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:25.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.894 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.894 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.894 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.895 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.895 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.895 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss= 2023-10-18T21:28:25.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss=', 'Content-Length': '0'} 2023-10-18T21:28:25.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010157719a226cc04-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:25.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.900 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.901 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.901 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.901 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.902 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.902 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:cDoJ+8PRRwIER7ntyIdd4PXKU/M= 2023-10-18T21:28:25.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cDoJ+8PRRwIER7ntyIdd4PXKU/M=', 'Content-Length': '0'} 2023-10-18T21:28:25.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002fe041bd987872a-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:25.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.907 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.907 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.908 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.908 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.908 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.909 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:cDoJ+8PRRwIER7ntyIdd4PXKU/M= 2023-10-18T21:28:25.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cDoJ+8PRRwIER7ntyIdd4PXKU/M=', 'Content-Length': '0'} 2023-10-18T21:28:25.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff85ebd16472031-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.910 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-18T21:28:25.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:25.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.914 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.914 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.915 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.915 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.915 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.916 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss= 2023-10-18T21:28:25.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss=', 'Content-Length': '0'} 2023-10-18T21:28:25.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001702364cc86ebac-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:25.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.921 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.921 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.921 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.922 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.922 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:25.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.922 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss= 2023-10-18T21:28:25.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3wIb6EC03OwSGdVp8azmLPMTrss=', 'Content-Length': '0'} 2023-10-18T21:28:25.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007261dd9e24b3a73d-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=h 2023-10-18T21:28:25.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=h 2023-10-18T21:28:25.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=h 2023-10-18T21:28:25.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.927 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.928 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.928 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.928 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.929 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.929 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5019b80a50c7b7f-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=h 2023-10-18T21:28:25.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=h 2023-10-18T21:28:25.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=h 2023-10-18T21:28:25.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.934 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.935 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.935 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.935 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.936 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.936 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cecbb84c8635d038-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h 2023-10-18T21:28:25.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.941 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.941 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.941 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.942 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.942 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h 2023-10-18T21:28:25.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.943 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:an234AdKI2DjD1h9YWgCQsNjsa4= 2023-10-18T21:28:25.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:an234AdKI2DjD1h9YWgCQsNjsa4=', 'Content-Length': '0'} 2023-10-18T21:28:25.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001019913fbcb7cd04-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h 2023-10-18T21:28:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.947 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:25.947 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.948 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.948 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.948 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h 2023-10-18T21:28:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.949 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:an234AdKI2DjD1h9YWgCQsNjsa4= 2023-10-18T21:28:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:an234AdKI2DjD1h9YWgCQsNjsa4=', 'Content-Length': '0'} 2023-10-18T21:28:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e7652665a85a09-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.950 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-18T21:28:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h 2023-10-18T21:28:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.953 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.954 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.954 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.954 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.955 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h 2023-10-18T21:28:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.955 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:51DIphLkMKb+nzztXQdLudW57Rw= 2023-10-18T21:28:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:51DIphLkMKb+nzztXQdLudW57Rw=', 'Content-Length': '0'} 2023-10-18T21:28:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000684cb9c6693ffa1b-0065303edf-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h 2023-10-18T21:28:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.959 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.960 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.960 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.960 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.961 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h 2023-10-18T21:28:25.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.961 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:51DIphLkMKb+nzztXQdLudW57Rw= 2023-10-18T21:28:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:51DIphLkMKb+nzztXQdLudW57Rw=', 'Content-Length': '0'} 2023-10-18T21:28:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde129f11b8f8444-0065303edf-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=h 2023-10-18T21:28:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=h 2023-10-18T21:28:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=h 2023-10-18T21:28:25.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.966 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.966 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.967 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.967 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.967 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.968 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2c24862b736ac3b-0065303edf-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=h 2023-10-18T21:28:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=h 2023-10-18T21:28:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:25.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=h 2023-10-18T21:28:25.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:25.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:25.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:25.973 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:25.973 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.974 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:25.974 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:23:59 GMT 2023-10-18T21:28:25.974 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:25.975 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc= 2023-10-18T21:28:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:23:59 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sCXQKnkAyIqucKFQrXQeS3FL2Xc=', 'Content-Length': '0'} 2023-10-18T21:28:25.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000255163e66fa67db4-0065303edf-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:23:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:25.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:25.976 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=bofrnf-55 zones={a2, a1} 2023-10-18T21:28:25.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'bofrnf-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.977 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'bofrnf-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket bofrnf-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-55 2023-10-18T21:28:25.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 8 2023-10-18T21:28:25.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding": "None", 2023-10-18T21:28:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_index": { 2023-10-18T21:28:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 2, 2023-10-18T21:28:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "Normal", 2023-10-18T21:28:25.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "normal": { 2023-10-18T21:28:25.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:25.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:25.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "logs": [ 2023-10-18T21:28:25.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 0, 2023-10-18T21:28:25.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:25.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:25.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 0, 2023-10-18T21:28:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 11, 2023-10-18T21:28:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 1, 2023-10-18T21:28:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 1, 2023-10-18T21:28:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 2, 2023-10-18T21:28:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 2, 2023-10-18T21:28:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:25.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:25.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=bofrnf-55 2023-10-18T21:28:25.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:25.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:25.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:25.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:25.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:25.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:24:00.199+0000 7f15ebab8780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T21:28:26.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:24:05.200+0000 7f15ebab8780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T21:28:26.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:24:10.201+0000 7f15ebab8780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T21:28:26.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:24:15.202+0000 7f15ebab8780 1 waiting to reach latest gen 2, on 1.. 2023-10-18T21:28:26.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:24:20.202+0000 7f15ebab8780 1 bucket sync caught up with empty source 2023-10-18T21:28:26.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:24:20.202+0000 7f15ebab8780 0 bucket checkpoint complete 2023-10-18T21:28:26.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=bofrnf-55 2023-10-18T21:28:26.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:26.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:24:20.321+0000 7f549f84a780 1 bucket sync caught up with source: 2023-10-18T21:28:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T21:28:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, 00000000002.53.6, , , 00000000001.554.6, 00000000001.595.6, , , 00000000002.484.6, , 00000000002.37.6] 2023-10-18T21:28:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:24:20.321+0000 7f549f84a780 0 bucket checkpoint complete 2023-10-18T21:28:26.004 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=bofrnf-55 zones={a2, a1} 2023-10-18T21:28:26.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:26.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.008 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.008 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.008 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.009 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.009 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.010 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Mhadj15V93uMESLg/uS93VOb3kE= 2023-10-18T21:28:26.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mhadj15V93uMESLg/uS93VOb3kE=', 'Content-Length': '0'} 2023-10-18T21:28:26.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ae5a31a48da1f96-0065303ef4-1668-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/ 2023-10-18T21:28:26.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.014 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.014 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.014 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.015 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.015 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.016 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Mhadj15V93uMESLg/uS93VOb3kE= 2023-10-18T21:28:26.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Mhadj15V93uMESLg/uS93VOb3kE=', 'Content-Length': '0'} 2023-10-18T21:28:26.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006925ecdf4d2e498e-0065303ef4-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.016 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-18T21:28:26.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.021 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.021 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.021 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.022 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.022 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:26.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.023 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs= 2023-10-18T21:28:26.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs=', 'Content-Length': '0'} 2023-10-18T21:28:26.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa0e1ab17fe7dc4-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.024 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-18T21:28:26.024 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-18T21:28:26.024 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-18T21:28:26.025 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-18T21:28:26.025 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-18T21:28:26.025 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-18T21:28:26.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-18T21:28:26.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-18T21:28:26.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-18T21:28:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.031 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.031 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.031 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.032 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.032 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:26.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.032 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs= 2023-10-18T21:28:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs=', 'Content-Length': '0'} 2023-10-18T21:28:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf8344ec2839924-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.034 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-18T21:28:26.034 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-18T21:28:26.034 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-18T21:28:26.035 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-18T21:28:26.035 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-18T21:28:26.035 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-18T21:28:26.035 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-18T21:28:26.036 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-18T21:28:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.040 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.040 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.041 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.041 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.041 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.042 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs= 2023-10-18T21:28:26.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs=', 'Content-Length': '0'} 2023-10-18T21:28:26.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc5ce2d6faf16c24-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?versions&max-keys=999 2023-10-18T21:28:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.047 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.047 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.048 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.048 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.048 INFO:tasks.rgw_multisite_tests:/bofrnf-55/?versions 2023-10-18T21:28:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.049 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs= 2023-10-18T21:28:26.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o33WDQOOj6Cp6BZ99dBlKHArTWs=', 'Content-Length': '0'} 2023-10-18T21:28:26.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c3b9687c74290ed-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55999falseanulltrue2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.050 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-18T21:28:26.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.054 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.055 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.055 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.055 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.056 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.056 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU= 2023-10-18T21:28:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU=', 'Content-Length': '0'} 2023-10-18T21:28:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f17c7fb2837efee-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.061 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.061 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.062 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.062 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.062 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.063 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU= 2023-10-18T21:28:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU=', 'Content-Length': '0'} 2023-10-18T21:28:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa8a964836f904ee-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.068 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.068 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.068 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.069 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.069 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.070 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:y84ufVmK3/zQ8W2VjBHAvQ4W4LE= 2023-10-18T21:28:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y84ufVmK3/zQ8W2VjBHAvQ4W4LE=', 'Content-Length': '0'} 2023-10-18T21:28:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f9131a0cc61e5e-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:26.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.074 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.075 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.075 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.075 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.076 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.076 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:y84ufVmK3/zQ8W2VjBHAvQ4W4LE= 2023-10-18T21:28:26.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y84ufVmK3/zQ8W2VjBHAvQ4W4LE=', 'Content-Length': '0'} 2023-10-18T21:28:26.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b39ae72631aed3-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.077 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-18T21:28:26.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:26.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.081 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.082 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.082 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.082 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.083 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.083 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU= 2023-10-18T21:28:26.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU=', 'Content-Length': '0'} 2023-10-18T21:28:26.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45030f7e84079f4-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a?versionId=null 2023-10-18T21:28:26.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.088 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.089 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.089 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.089 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.089 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a?versionId=null 2023-10-18T21:28:26.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.090 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU= 2023-10-18T21:28:26.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2IQsXlTymMMv6efEbOC0W5t//eU=', 'Content-Length': '0'} 2023-10-18T21:28:26.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c7ac62a4ee5c91e-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:26.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:26.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:26.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.095 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.095 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.096 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.096 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.096 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.097 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000593cb046d9c0aba7-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:26.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:26.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:26.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.102 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.102 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.103 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.103 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.103 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.104 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee355deaaa7febc2-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:26.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.108 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.109 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.109 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.109 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.109 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:26.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.110 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:zjXwn04OLDZRQkDHIXzbHIOb6Rk= 2023-10-18T21:28:26.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zjXwn04OLDZRQkDHIXzbHIOb6Rk=', 'Content-Length': '0'} 2023-10-18T21:28:26.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000269a8c653e84ef16-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:26.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.114 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.115 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.115 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.115 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.115 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:26.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.116 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:zjXwn04OLDZRQkDHIXzbHIOb6Rk= 2023-10-18T21:28:26.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:zjXwn04OLDZRQkDHIXzbHIOb6Rk=', 'Content-Length': '0'} 2023-10-18T21:28:26.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e4bce094c592e71-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.117 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-18T21:28:26.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:26.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.121 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.121 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.121 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.121 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.122 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.122 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:r4I+izeyPq9/cUpeLUXvAbmyVGM= 2023-10-18T21:28:26.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r4I+izeyPq9/cUpeLUXvAbmyVGM=', 'Content-Length': '0'} 2023-10-18T21:28:26.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6bd36e225cb943-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/a 2023-10-18T21:28:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/a 2023-10-18T21:28:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/a 2023-10-18T21:28:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.127 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.127 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.127 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.127 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.128 INFO:tasks.rgw_multisite_tests:/bofrnf-55/a 2023-10-18T21:28:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.128 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:r4I+izeyPq9/cUpeLUXvAbmyVGM= 2023-10-18T21:28:26.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:r4I+izeyPq9/cUpeLUXvAbmyVGM=', 'Content-Length': '0'} 2023-10-18T21:28:26.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b17c6f03fd281f-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.133 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.133 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.134 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.134 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.134 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.135 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4745b9f548832f5-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=a 2023-10-18T21:28:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=a 2023-10-18T21:28:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=a 2023-10-18T21:28:26.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.140 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.140 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.141 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.141 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.141 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.142 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008371df0ec411380f-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55a1000falsea2023-10-18T20:21:00.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.143 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-18T21:28:26.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:26.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.148 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.148 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.148 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.148 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.149 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.149 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8= 2023-10-18T21:28:26.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8=', 'Content-Length': '0'} 2023-10-18T21:28:26.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059fedec66fad5bc9-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:26.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.154 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.155 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.155 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.155 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.156 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.156 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8= 2023-10-18T21:28:26.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8=', 'Content-Length': '0'} 2023-10-18T21:28:26.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02fd0f2a5c1e3a1-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:26.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.161 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.161 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.162 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.162 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.162 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.163 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:nisZSm52Fu1gW5LCoYV6MlbeqCk= 2023-10-18T21:28:26.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nisZSm52Fu1gW5LCoYV6MlbeqCk=', 'Content-Length': '0'} 2023-10-18T21:28:26.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33402a8ea690b73-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:26.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.168 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.168 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.168 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.169 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.169 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.169 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:nisZSm52Fu1gW5LCoYV6MlbeqCk= 2023-10-18T21:28:26.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nisZSm52Fu1gW5LCoYV6MlbeqCk=', 'Content-Length': '0'} 2023-10-18T21:28:26.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d00d7d670b73b9-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.170 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-18T21:28:26.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:26.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.175 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.175 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.175 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.176 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.176 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.177 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8= 2023-10-18T21:28:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8=', 'Content-Length': '0'} 2023-10-18T21:28:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b991eb31cf8b5dfd-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b?versionId=null 2023-10-18T21:28:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.181 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.182 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.182 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.182 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.183 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b?versionId=null 2023-10-18T21:28:26.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.183 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8= 2023-10-18T21:28:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:B0PHK9lqXOF5H3pQLtnFy3Cfip8=', 'Content-Length': '0'} 2023-10-18T21:28:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389ff9173ab5c4ac-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:26.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.188 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.189 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.189 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.189 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.190 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.190 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d7a5f6e04352f45-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:26.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:26.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:26.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.195 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.196 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.196 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.196 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.197 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.197 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000139a641fabdd655c-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:26.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.202 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.202 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.202 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.203 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.203 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:26.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.204 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:hE0PBU4SG9FRrvD2QO7JDVdINCg= 2023-10-18T21:28:26.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hE0PBU4SG9FRrvD2QO7JDVdINCg=', 'Content-Length': '0'} 2023-10-18T21:28:26.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005daa74ff4d25f707-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:26.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.208 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.208 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.208 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.209 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.209 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:26.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.210 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:hE0PBU4SG9FRrvD2QO7JDVdINCg= 2023-10-18T21:28:26.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hE0PBU4SG9FRrvD2QO7JDVdINCg=', 'Content-Length': '0'} 2023-10-18T21:28:26.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5eae9a46ba0482-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.210 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-18T21:28:26.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:26.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.214 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.214 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.215 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.215 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.215 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:26.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.216 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3Hk6HYZ0AwZF8aswzSychJ5INgU= 2023-10-18T21:28:26.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Hk6HYZ0AwZF8aswzSychJ5INgU=', 'Content-Length': '0'} 2023-10-18T21:28:26.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ed0406bf92a4479-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/b 2023-10-18T21:28:26.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/b 2023-10-18T21:28:26.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/b 2023-10-18T21:28:26.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.220 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.220 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.221 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.221 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.221 INFO:tasks.rgw_multisite_tests:/bofrnf-55/b 2023-10-18T21:28:26.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.222 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3Hk6HYZ0AwZF8aswzSychJ5INgU= 2023-10-18T21:28:26.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Hk6HYZ0AwZF8aswzSychJ5INgU=', 'Content-Length': '0'} 2023-10-18T21:28:26.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49b5a5d14286bea-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:26.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:26.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:26.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.227 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.227 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.228 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.228 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.228 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.229 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006966e88320bbf339-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=b 2023-10-18T21:28:26.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=b 2023-10-18T21:28:26.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=b 2023-10-18T21:28:26.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.234 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.234 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.235 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.235 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.235 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.236 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013eba0e45571c5a2-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55b1000falseb2023-10-18T20:21:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.237 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-18T21:28:26.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:26.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.241 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.241 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.242 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.242 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.242 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.243 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA= 2023-10-18T21:28:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA=', 'Content-Length': '0'} 2023-10-18T21:28:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5078623edd49d68-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.248 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.248 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.248 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.249 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.249 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.250 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA= 2023-10-18T21:28:26.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA=', 'Content-Length': '0'} 2023-10-18T21:28:26.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd058875b88a68d-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.254 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.255 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.255 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.255 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.256 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.256 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:FaZzi79yTbHI2xI5MU8kRb2ta6o= 2023-10-18T21:28:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FaZzi79yTbHI2xI5MU8kRb2ta6o=', 'Content-Length': '0'} 2023-10-18T21:28:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000259e01ae3a89302d-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:26.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.261 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.261 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.262 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.262 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.262 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.263 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:FaZzi79yTbHI2xI5MU8kRb2ta6o= 2023-10-18T21:28:26.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FaZzi79yTbHI2xI5MU8kRb2ta6o=', 'Content-Length': '0'} 2023-10-18T21:28:26.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c63ae4e7740161-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.264 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-18T21:28:26.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:26.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.268 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.268 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.269 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.269 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.269 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.270 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA= 2023-10-18T21:28:26.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA=', 'Content-Length': '0'} 2023-10-18T21:28:26.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed32c02909044c4a-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c?versionId=null 2023-10-18T21:28:26.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.275 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.275 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.275 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.276 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.276 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c?versionId=null 2023-10-18T21:28:26.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.277 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA= 2023-10-18T21:28:26.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KTqFY3TxsR0BCeB54Vw9YVJMkbA=', 'Content-Length': '0'} 2023-10-18T21:28:26.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006642de39c70f677-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:26.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:26.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:26.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.281 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.282 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.282 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.282 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.283 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.283 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8b0cbc73b7d984-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:26.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:26.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:26.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.288 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.289 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.289 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.289 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.289 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.290 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f934a8567b60be42-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:26.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.295 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.295 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.295 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.296 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.296 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.296 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sEi4yJccpmT9L5LeGUX2oZmVY88= 2023-10-18T21:28:26.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sEi4yJccpmT9L5LeGUX2oZmVY88=', 'Content-Length': '0'} 2023-10-18T21:28:26.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd96062112bca7c-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:26.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.301 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.301 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.301 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.302 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.302 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:26.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.303 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sEi4yJccpmT9L5LeGUX2oZmVY88= 2023-10-18T21:28:26.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sEi4yJccpmT9L5LeGUX2oZmVY88=', 'Content-Length': '0'} 2023-10-18T21:28:26.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b47492ffd2a964-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.304 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-18T21:28:26.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:26.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.307 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.308 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.308 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.308 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.309 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:26.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.309 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:SiCEjAH22VfvZKFXKGShKNp1fiY= 2023-10-18T21:28:26.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SiCEjAH22VfvZKFXKGShKNp1fiY=', 'Content-Length': '0'} 2023-10-18T21:28:26.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff73cf761ad07ba-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/c 2023-10-18T21:28:26.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/c 2023-10-18T21:28:26.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/c 2023-10-18T21:28:26.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.313 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.314 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.314 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.314 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.315 INFO:tasks.rgw_multisite_tests:/bofrnf-55/c 2023-10-18T21:28:26.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.315 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:SiCEjAH22VfvZKFXKGShKNp1fiY= 2023-10-18T21:28:26.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SiCEjAH22VfvZKFXKGShKNp1fiY=', 'Content-Length': '0'} 2023-10-18T21:28:26.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b5ca264500940b-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:26.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:26.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:26.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.320 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.320 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.321 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.321 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.321 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.322 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad4ea85558e3282c-0065303ef4-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=c 2023-10-18T21:28:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=c 2023-10-18T21:28:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=c 2023-10-18T21:28:26.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.327 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.327 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.328 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.328 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.328 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.329 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk= 2023-10-18T21:28:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:J4GqZaIxg8OZJPKzWlz/gF3zwsk=', 'Content-Length': '0'} 2023-10-18T21:28:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025eb8624fbba5fe3-0065303ef4-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55c1000falsec2023-10-18T20:21:01.019Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.330 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-18T21:28:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:26.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.334 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.335 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.335 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.335 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.336 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.336 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI= 2023-10-18T21:28:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI=', 'Content-Length': '0'} 2023-10-18T21:28:26.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000492c08782f112c56-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:26.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.341 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.341 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.342 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.342 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.342 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.343 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI= 2023-10-18T21:28:26.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI=', 'Content-Length': '0'} 2023-10-18T21:28:26.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005b645e36c73e5c5-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:26.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.348 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.348 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.348 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.349 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.349 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.350 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4jv7kbyfhq1XrKReKOG0NC3Lmug= 2023-10-18T21:28:26.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4jv7kbyfhq1XrKReKOG0NC3Lmug=', 'Content-Length': '0'} 2023-10-18T21:28:26.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001066f6e2ee19d917-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:26.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.355 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.355 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.355 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.355 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.356 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.356 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4jv7kbyfhq1XrKReKOG0NC3Lmug= 2023-10-18T21:28:26.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4jv7kbyfhq1XrKReKOG0NC3Lmug=', 'Content-Length': '0'} 2023-10-18T21:28:26.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2af80ed35b835c0-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.357 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-18T21:28:26.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:26.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.361 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.362 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.362 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.362 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.363 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.363 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI= 2023-10-18T21:28:26.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI=', 'Content-Length': '0'} 2023-10-18T21:28:26.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a456e4a8782307f7-0065303ef4-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d?versionId=null 2023-10-18T21:28:26.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.368 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.368 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.369 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.369 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:20 GMT 2023-10-18T21:28:26.369 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d?versionId=null 2023-10-18T21:28:26.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.370 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI= 2023-10-18T21:28:26.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:20 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8EzNenOE1Y4WCM/a/X7pneXm4EI=', 'Content-Length': '0'} 2023-10-18T21:28:26.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053f942e6534aa6dc-0065303ef4-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:26.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:26.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:26.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.375 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.375 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.375 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.375 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.376 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.376 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a37b88677885194c-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:26.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:26.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:26.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.384 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.385 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.385 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.385 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.386 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.386 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097265c515256caa4-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:26.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.391 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.391 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.391 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.392 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.392 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:26.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.393 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:jg1JA9Ain07Hfi2VLbmtuS6Tahw= 2023-10-18T21:28:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jg1JA9Ain07Hfi2VLbmtuS6Tahw=', 'Content-Length': '0'} 2023-10-18T21:28:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000743f6ce08f0e19e0-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.397 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.397 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.398 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.398 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.398 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:26.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.399 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:jg1JA9Ain07Hfi2VLbmtuS6Tahw= 2023-10-18T21:28:26.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jg1JA9Ain07Hfi2VLbmtuS6Tahw=', 'Content-Length': '0'} 2023-10-18T21:28:26.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009314ebf438ccb496-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.400 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-18T21:28:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.403 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.404 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.404 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.404 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.405 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:26.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.405 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:IckQ0iH/CJg7dUW+95WfkdyqxLs= 2023-10-18T21:28:26.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IckQ0iH/CJg7dUW+95WfkdyqxLs=', 'Content-Length': '0'} 2023-10-18T21:28:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062957882fbe18c3b-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/d 2023-10-18T21:28:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/d 2023-10-18T21:28:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/d 2023-10-18T21:28:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.409 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.410 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.410 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.410 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.411 INFO:tasks.rgw_multisite_tests:/bofrnf-55/d 2023-10-18T21:28:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.411 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:IckQ0iH/CJg7dUW+95WfkdyqxLs= 2023-10-18T21:28:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:IckQ0iH/CJg7dUW+95WfkdyqxLs=', 'Content-Length': '0'} 2023-10-18T21:28:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fade565dca7ae4a2-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.416 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.416 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.417 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.417 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.417 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.418 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000432fb6cad655c18b-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=d 2023-10-18T21:28:26.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=d 2023-10-18T21:28:26.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=d 2023-10-18T21:28:26.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.423 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.423 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.424 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.424 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.424 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.425 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022e2354d56aa7655-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55d1000falsed2023-10-18T20:21:01.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.426 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-18T21:28:26.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:26.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.430 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.431 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.431 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.431 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.432 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.432 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q= 2023-10-18T21:28:26.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q=', 'Content-Length': '0'} 2023-10-18T21:28:26.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048ce49a41b5186ae-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:26.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.437 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.437 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.438 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.438 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.438 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.439 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q= 2023-10-18T21:28:26.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q=', 'Content-Length': '0'} 2023-10-18T21:28:26.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff7a00e647be0e51-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:26.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.444 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.444 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.444 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.445 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.445 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.446 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:nqGZcqvuxeKqvak6R2RWMDmMIgA= 2023-10-18T21:28:26.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nqGZcqvuxeKqvak6R2RWMDmMIgA=', 'Content-Length': '0'} 2023-10-18T21:28:26.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6904bd91cc710b-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:26.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.450 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.451 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.451 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.451 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.452 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.452 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:nqGZcqvuxeKqvak6R2RWMDmMIgA= 2023-10-18T21:28:26.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nqGZcqvuxeKqvak6R2RWMDmMIgA=', 'Content-Length': '0'} 2023-10-18T21:28:26.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024ef9ae549869e60-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.453 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-18T21:28:26.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:26.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.457 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.458 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.458 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.458 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.459 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.459 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q= 2023-10-18T21:28:26.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q=', 'Content-Length': '0'} 2023-10-18T21:28:26.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b31df91c1ec068-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e?versionId=null 2023-10-18T21:28:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.464 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.464 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.465 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.465 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.465 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e?versionId=null 2023-10-18T21:28:26.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.466 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q= 2023-10-18T21:28:26.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:uJbQw1F8JGAxkspvLZmFGPPtA1Q=', 'Content-Length': '0'} 2023-10-18T21:28:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80f2c66581dc10e-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=e 2023-10-18T21:28:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=e 2023-10-18T21:28:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=e 2023-10-18T21:28:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.471 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.471 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.471 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.472 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.472 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.473 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3e361b17abb6b3a-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=e 2023-10-18T21:28:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=e 2023-10-18T21:28:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=e 2023-10-18T21:28:26.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.478 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.478 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.478 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.479 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.479 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.480 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005adcc1afbaee70ac-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e 2023-10-18T21:28:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.484 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.484 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.485 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.485 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.485 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e 2023-10-18T21:28:26.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.486 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:qab1GfcLJN1fMPiU3L9E2QLQ/YI= 2023-10-18T21:28:26.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qab1GfcLJN1fMPiU3L9E2QLQ/YI=', 'Content-Length': '0'} 2023-10-18T21:28:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160e2bd12084f5af-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e 2023-10-18T21:28:26.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.490 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.490 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.491 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.491 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.491 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e 2023-10-18T21:28:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.492 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:qab1GfcLJN1fMPiU3L9E2QLQ/YI= 2023-10-18T21:28:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qab1GfcLJN1fMPiU3L9E2QLQ/YI=', 'Content-Length': '0'} 2023-10-18T21:28:26.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139c31ae1892bc0f-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.493 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-18T21:28:26.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e 2023-10-18T21:28:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.496 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.497 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.497 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.497 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.498 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e 2023-10-18T21:28:26.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.498 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4YMjfD4bfDWAvC86jdWyF87bks8= 2023-10-18T21:28:26.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4YMjfD4bfDWAvC86jdWyF87bks8=', 'Content-Length': '0'} 2023-10-18T21:28:26.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c292cd3edd79a7a2-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/e 2023-10-18T21:28:26.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/e 2023-10-18T21:28:26.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/e 2023-10-18T21:28:26.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.503 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.503 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.503 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.504 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.504 INFO:tasks.rgw_multisite_tests:/bofrnf-55/e 2023-10-18T21:28:26.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.504 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4YMjfD4bfDWAvC86jdWyF87bks8= 2023-10-18T21:28:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4YMjfD4bfDWAvC86jdWyF87bks8=', 'Content-Length': '0'} 2023-10-18T21:28:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212a15d21f556a04-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=e 2023-10-18T21:28:26.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=e 2023-10-18T21:28:26.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=e 2023-10-18T21:28:26.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.509 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.510 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.510 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.510 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.511 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.511 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097077375783ee6ec-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=e 2023-10-18T21:28:26.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=e 2023-10-18T21:28:26.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=e 2023-10-18T21:28:26.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.516 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.517 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.517 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.517 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.518 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.518 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d1f8872adf1c0c5-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55e1000falsee2023-10-18T20:21:42.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.519 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-18T21:28:26.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:26.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.524 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.524 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.524 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.525 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.525 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.526 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA= 2023-10-18T21:28:26.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA=', 'Content-Length': '0'} 2023-10-18T21:28:26.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2d89ec10bfae8f-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:26.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.530 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.531 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.531 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.531 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.532 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.532 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA= 2023-10-18T21:28:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA=', 'Content-Length': '0'} 2023-10-18T21:28:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053824e7f51d74b9c-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.537 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.537 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.538 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.538 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.538 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.539 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8mA9voj8wYW2FW98EUW+HmHG4rM= 2023-10-18T21:28:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8mA9voj8wYW2FW98EUW+HmHG4rM=', 'Content-Length': '0'} 2023-10-18T21:28:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000267d057999f16c65-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.544 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.544 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.545 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.545 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.545 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.546 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8mA9voj8wYW2FW98EUW+HmHG4rM= 2023-10-18T21:28:26.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8mA9voj8wYW2FW98EUW+HmHG4rM=', 'Content-Length': '0'} 2023-10-18T21:28:26.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae72e29d37395db-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.547 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-18T21:28:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.551 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.551 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.552 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.552 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.552 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.553 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA= 2023-10-18T21:28:26.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA=', 'Content-Length': '0'} 2023-10-18T21:28:26.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4167d42109bc0fd-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f?versionId=null 2023-10-18T21:28:26.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.558 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.558 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.558 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.559 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.559 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f?versionId=null 2023-10-18T21:28:26.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.560 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA= 2023-10-18T21:28:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VBZyCcRMo944X5VgTFYNUVBAgRA=', 'Content-Length': '0'} 2023-10-18T21:28:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cda27cb496cbcd17-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=f 2023-10-18T21:28:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=f 2023-10-18T21:28:26.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=f 2023-10-18T21:28:26.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.564 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.565 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.565 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.565 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.566 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.566 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033899ee9d21a65cd-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=f 2023-10-18T21:28:26.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=f 2023-10-18T21:28:26.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=f 2023-10-18T21:28:26.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.571 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.572 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.572 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.572 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.573 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.573 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000673f8b435ca33c83-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f 2023-10-18T21:28:26.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.578 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.578 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.578 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.579 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.579 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f 2023-10-18T21:28:26.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.579 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:s2R/StI8reiQYXj7UADNHY8WnLo= 2023-10-18T21:28:26.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s2R/StI8reiQYXj7UADNHY8WnLo=', 'Content-Length': '0'} 2023-10-18T21:28:26.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c92505149650543c-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f 2023-10-18T21:28:26.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.584 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.584 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.584 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.585 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.585 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f 2023-10-18T21:28:26.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.586 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:s2R/StI8reiQYXj7UADNHY8WnLo= 2023-10-18T21:28:26.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:s2R/StI8reiQYXj7UADNHY8WnLo=', 'Content-Length': '0'} 2023-10-18T21:28:26.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099f1684b0829b51-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.587 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-18T21:28:26.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f 2023-10-18T21:28:26.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.592 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.593 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.593 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.593 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.593 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f 2023-10-18T21:28:26.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.594 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:HubOixa8CKyavZ3TR+dZ1ZxLtZc= 2023-10-18T21:28:26.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HubOixa8CKyavZ3TR+dZ1ZxLtZc=', 'Content-Length': '0'} 2023-10-18T21:28:26.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007aeb56e1f580553-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/f 2023-10-18T21:28:26.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/f 2023-10-18T21:28:26.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/f 2023-10-18T21:28:26.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.598 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.599 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.599 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.599 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.599 INFO:tasks.rgw_multisite_tests:/bofrnf-55/f 2023-10-18T21:28:26.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.600 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:HubOixa8CKyavZ3TR+dZ1ZxLtZc= 2023-10-18T21:28:26.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HubOixa8CKyavZ3TR+dZ1ZxLtZc=', 'Content-Length': '0'} 2023-10-18T21:28:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4021df9bdf6b029-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=f 2023-10-18T21:28:26.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=f 2023-10-18T21:28:26.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=f 2023-10-18T21:28:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.605 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.605 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.605 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.606 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.606 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.607 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516a6e0bff154d93-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=f 2023-10-18T21:28:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=f 2023-10-18T21:28:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=f 2023-10-18T21:28:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.612 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.612 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.613 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.613 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.613 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.614 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016061c9bfe687155-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55f1000falsef2023-10-18T20:21:42.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.615 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-18T21:28:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.619 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.619 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.620 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.620 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.620 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.621 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ= 2023-10-18T21:28:26.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ=', 'Content-Length': '0'} 2023-10-18T21:28:26.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c93ef91b65cbe3b-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:26.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.626 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.626 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.627 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.627 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.627 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.628 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ= 2023-10-18T21:28:26.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ=', 'Content-Length': '0'} 2023-10-18T21:28:26.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af48cdecec7e74b-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.633 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.633 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.633 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.634 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.634 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.635 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1qnP6lr+/Cv2WWoUG4C9byn7oBg= 2023-10-18T21:28:26.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1qnP6lr+/Cv2WWoUG4C9byn7oBg=', 'Content-Length': '0'} 2023-10-18T21:28:26.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009849be6e41c5ff31-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:26.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.639 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.640 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.640 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.640 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.641 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.641 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1qnP6lr+/Cv2WWoUG4C9byn7oBg= 2023-10-18T21:28:26.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1qnP6lr+/Cv2WWoUG4C9byn7oBg=', 'Content-Length': '0'} 2023-10-18T21:28:26.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07e6cc49637b6b5-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.642 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-18T21:28:26.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:26.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.646 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.647 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.647 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.647 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.648 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.648 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ= 2023-10-18T21:28:26.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ=', 'Content-Length': '0'} 2023-10-18T21:28:26.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017ed8bdbf93c5bc9-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g?versionId=null 2023-10-18T21:28:26.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.653 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.653 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.654 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.654 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.654 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g?versionId=null 2023-10-18T21:28:26.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.655 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ= 2023-10-18T21:28:26.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:tkUCUwwoWj6HZrgfx8FHq27xIQQ=', 'Content-Length': '0'} 2023-10-18T21:28:26.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e224af1826dce0-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=g 2023-10-18T21:28:26.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=g 2023-10-18T21:28:26.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=g 2023-10-18T21:28:26.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.660 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.660 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.660 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.661 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.661 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.662 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025abbfbd46c3cd91-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=g 2023-10-18T21:28:26.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=g 2023-10-18T21:28:26.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=g 2023-10-18T21:28:26.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.667 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.667 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.668 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.668 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.668 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.669 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3ec1226cf8b009c-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g 2023-10-18T21:28:26.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.673 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.674 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.674 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.674 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.675 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g 2023-10-18T21:28:26.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.675 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4in5VvuIo9OLcUMacIlF2lc1P54= 2023-10-18T21:28:26.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4in5VvuIo9OLcUMacIlF2lc1P54=', 'Content-Length': '0'} 2023-10-18T21:28:26.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075a1730b9e8ceb5-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g 2023-10-18T21:28:26.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.679 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.680 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.680 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.680 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.681 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g 2023-10-18T21:28:26.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.681 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4in5VvuIo9OLcUMacIlF2lc1P54= 2023-10-18T21:28:26.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4in5VvuIo9OLcUMacIlF2lc1P54=', 'Content-Length': '0'} 2023-10-18T21:28:26.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f6c0a11de669b7-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.682 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-18T21:28:26.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g 2023-10-18T21:28:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.686 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.686 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.686 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.687 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.687 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g 2023-10-18T21:28:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.688 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:VV6trtJKbYsV81bVb8DsgqE+D9E= 2023-10-18T21:28:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VV6trtJKbYsV81bVb8DsgqE+D9E=', 'Content-Length': '0'} 2023-10-18T21:28:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e881b329942fc71f-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/g 2023-10-18T21:28:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/g 2023-10-18T21:28:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/g 2023-10-18T21:28:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.692 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.692 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.693 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.693 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.693 INFO:tasks.rgw_multisite_tests:/bofrnf-55/g 2023-10-18T21:28:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.694 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:VV6trtJKbYsV81bVb8DsgqE+D9E= 2023-10-18T21:28:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VV6trtJKbYsV81bVb8DsgqE+D9E=', 'Content-Length': '0'} 2023-10-18T21:28:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cde247104dec81c-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=g 2023-10-18T21:28:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=g 2023-10-18T21:28:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=g 2023-10-18T21:28:26.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.699 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.699 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.699 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.700 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.700 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.700 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7397fe1c3121466-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=g 2023-10-18T21:28:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=g 2023-10-18T21:28:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=g 2023-10-18T21:28:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.706 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.706 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.706 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.707 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.707 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.708 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087cc6aecdbd044b2-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55g1000falseg2023-10-18T20:21:42.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.709 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-18T21:28:26.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.713 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.713 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.714 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.714 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.714 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.715 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk= 2023-10-18T21:28:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk=', 'Content-Length': '0'} 2023-10-18T21:28:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b53736eba4cc7e-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.720 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.720 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.720 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.721 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.721 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.722 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk= 2023-10-18T21:28:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk=', 'Content-Length': '0'} 2023-10-18T21:28:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095c9322cc0d7ed31-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.726 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.727 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.727 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.727 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.728 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.728 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Oqm6z2Z7LsDrqKrcP2dB+CT3kcQ= 2023-10-18T21:28:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Oqm6z2Z7LsDrqKrcP2dB+CT3kcQ=', 'Content-Length': '0'} 2023-10-18T21:28:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000874fd60ef79fe3ac-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.733 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.733 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.734 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.734 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.734 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.735 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Oqm6z2Z7LsDrqKrcP2dB+CT3kcQ= 2023-10-18T21:28:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Oqm6z2Z7LsDrqKrcP2dB+CT3kcQ=', 'Content-Length': '0'} 2023-10-18T21:28:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44c043c4427993d-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.736 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-18T21:28:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.740 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.740 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.741 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.741 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.741 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.742 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk= 2023-10-18T21:28:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk=', 'Content-Length': '0'} 2023-10-18T21:28:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ca852170adb6d4-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h?versionId=null 2023-10-18T21:28:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.747 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.747 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.747 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.748 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.748 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h?versionId=null 2023-10-18T21:28:26.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.748 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk= 2023-10-18T21:28:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2E9a3CeunHYu30ZkYTuU0zwVCLk=', 'Content-Length': '0'} 2023-10-18T21:28:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eb79b8975ab0576-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=h 2023-10-18T21:28:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=h 2023-10-18T21:28:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=h 2023-10-18T21:28:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.753 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.754 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.754 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.754 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.755 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.755 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c5f5c86ffeb395e-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=h 2023-10-18T21:28:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=h 2023-10-18T21:28:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=h 2023-10-18T21:28:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.760 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.761 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.761 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.761 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.762 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.762 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007438b16e1b305340-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h 2023-10-18T21:28:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.767 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.767 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.768 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.768 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.768 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h 2023-10-18T21:28:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.769 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:W7vS7xrIqK76pI1e6vFD7DMH4Qs= 2023-10-18T21:28:26.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W7vS7xrIqK76pI1e6vFD7DMH4Qs=', 'Content-Length': '0'} 2023-10-18T21:28:26.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b6502b3bb955231-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h 2023-10-18T21:28:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.773 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.773 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.774 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.774 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.774 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h 2023-10-18T21:28:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.775 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:W7vS7xrIqK76pI1e6vFD7DMH4Qs= 2023-10-18T21:28:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:W7vS7xrIqK76pI1e6vFD7DMH4Qs=', 'Content-Length': '0'} 2023-10-18T21:28:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e7334c101821d0-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.776 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-18T21:28:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h 2023-10-18T21:28:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.780 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.780 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.780 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.780 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.781 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h 2023-10-18T21:28:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.781 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:loEfPDZEZ5s5Gqcv6XRNFq45/hs= 2023-10-18T21:28:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:loEfPDZEZ5s5Gqcv6XRNFq45/hs=', 'Content-Length': '0'} 2023-10-18T21:28:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-55:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007de1612eb7e2edf3-0065303ef5-1668-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/h 2023-10-18T21:28:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/h 2023-10-18T21:28:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/h 2023-10-18T21:28:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.786 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.786 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.786 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.787 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.787 INFO:tasks.rgw_multisite_tests:/bofrnf-55/h 2023-10-18T21:28:26.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.787 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:loEfPDZEZ5s5Gqcv6XRNFq45/hs= 2023-10-18T21:28:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:loEfPDZEZ5s5Gqcv6XRNFq45/hs=', 'Content-Length': '0'} 2023-10-18T21:28:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 20:21:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f98d16a1d56142f0-0065303ef5-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=h 2023-10-18T21:28:26.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=h 2023-10-18T21:28:26.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=h 2023-10-18T21:28:26.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:26.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:26.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.792 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.793 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.793 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.793 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.794 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.794 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b46e8db5ba7bdee0-0065303ef5-1668-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/ 2023-10-18T21:28:26.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/ 2023-10-18T21:28:26.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-55/?prefix=h 2023-10-18T21:28:26.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-55/?prefix=h 2023-10-18T21:28:26.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:26.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-55/?prefix=h 2023-10-18T21:28:26.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.799 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:26.800 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.800 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.800 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:24:21 GMT 2023-10-18T21:28:26.801 INFO:tasks.rgw_multisite_tests:/bofrnf-55/ 2023-10-18T21:28:26.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.801 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4= 2023-10-18T21:28:26.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:24:21 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ODzOUGk4gJlhQGK6awibHufOVA4=', 'Content-Length': '0'} 2023-10-18T21:28:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014f843139a16b54b-0065303ef5-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 20:24:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-55h1000falseh2023-10-18T20:21:43.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:26.802 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=bofrnf-55 zones={a2, a1} 2023-10-18T21:28:26.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:26.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:26.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:26.804 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:21.952+0000 7efce4c91700 -1 req 10418709809151165008 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-36:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.3,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-36:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.3,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.804 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:21.952+0000 7efdc5e53700 -1 req 17951071748088388098 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-24:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.24,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-24:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.24,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.805 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:21.952+0000 7efdbc640700 -1 req 7809433190435413888 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-5:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.5,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-5:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.5,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.805 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:21.955+0000 7efcdac7d700 -1 req 16622378275756042683 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-41:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.10,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-41:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.10,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.805 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:21.997+0000 7efdb3e2f700 -1 req 11600398951189980837 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-27:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.27,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-27:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.27,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.806 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.002+0000 7efdae624700 -1 req 15539175400740723401 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-19:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.19,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-19:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.19,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.806 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.008+0000 7efd96df5700 -1 req 9945040015104871972 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-3:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.3,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-3:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.3,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.806 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.039+0000 7efd8b5de700 -1 req 1080517829428759886 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-30:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.30,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-30:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.30,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.807 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.046+0000 7efda3e0f700 -1 req 9728147470867133463 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-40:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.9,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-40:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.9,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.807 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.046+0000 7efd775b6700 -1 req 1993393129796747981 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-25:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.25,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-25:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.25,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.807 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.051+0000 7efd90de9700 -1 req 14791781324847460765 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-22:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.22,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-22:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.22,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:26.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding": "None", 2023-10-18T21:28:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_index": { 2023-10-18T21:28:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 2, 2023-10-18T21:28:26.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "Normal", 2023-10-18T21:28:26.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "normal": { 2023-10-18T21:28:26.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:26.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:26.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "logs": [ 2023-10-18T21:28:26.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 1, 2023-10-18T21:28:26.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:26.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:26.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 1, 2023-10-18T21:28:26.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 2, 2023-10-18T21:28:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 2, 2023-10-18T21:28:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:26.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.822 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:26.822 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:26.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:26.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:26.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:26.824 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.296+0000 7efd7c5c0700 -1 req 15075530772322273016 0.000999981s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-39:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.8,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-39:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.8,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.824 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.298+0000 7efd5e584700 -1 req 13649186578147700193 0.000999982s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-37:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.6,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-37:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.6,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.824 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.338+0000 7efd6a59c700 -1 req 8082273709273409527 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-2:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.2,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-2:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.2,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.825 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.340+0000 7efd4bd5f700 -1 req 15487544499056391101 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-29:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.29,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-29:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.29,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.825 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.342+0000 7efd7fdc7700 -1 req 17200455680627767505 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-23:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.23,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-23:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.23,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.825 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.382+0000 7efd3953a700 -1 req 2889084971211465227 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-28:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.28,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-28:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.28,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.826 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:24:22.386+0000 7efd3dd43700 -1 req 1055971798316952861 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=bofrnf-20:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.20,z=403a921b-fcfc-4ab2-b434-bf29d35a51b6,az=0},d={b=bofrnf-20:403a921b-fcfc-4ab2-b434-bf29d35a51b6.4372.20,z=419020ed-d3b2-4ce0-b36c-97abb284c459,az=0}} returned ret=-2 2023-10-18T21:28:26.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:26.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'bofrnf-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:26.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket bofrnf-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:26.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:26.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:26.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding": "None", 2023-10-18T21:28:26.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_index": { 2023-10-18T21:28:26.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 2, 2023-10-18T21:28:26.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "Normal", 2023-10-18T21:28:26.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "normal": { 2023-10-18T21:28:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "logs": [ 2023-10-18T21:28:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 1, 2023-10-18T21:28:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 1, 2023-10-18T21:28:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 2, 2023-10-18T21:28:26.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "type": "InIndex", 2023-10-18T21:28:26.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "in_index": { 2023-10-18T21:28:26.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "gen": 2, 2023-10-18T21:28:26.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "layout": { 2023-10-18T21:28:26.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 3, 2023-10-18T21:28:26.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hash_type": "Mod" 2023-10-18T21:28:26.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.841 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-18T21:28:26.841 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.842 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-18T21:28:26.842 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2023-10-18T21:28:26.842 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-18T21:28:26.843 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-18T21:28:26.843 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-18T21:28:26.843 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-18T21:28:26.844 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 1776, in test_resharded_bucket_full_sync_after_data_sync_init 2023-10-18T21:28:26.844 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-18T21:28:26.844 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-18T21:28:26.845 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-18T21:28:26.845 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-18T21:28:26.845 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-18T21:28:26.846 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-18T21:28:26.846 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-18T21:28:26.846 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-18T21:28:26.847 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-18T21:28:26.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:26.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:26.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:26.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:26.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.850 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:26.851 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.851 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.851 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:28 GMT 2023-10-18T21:28:26.851 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:26.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.852 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:47AA5ASZ2oaCBXgRSh165N8FEFA= 2023-10-18T21:28:26.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:28 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:47AA5ASZ2oaCBXgRSh165N8FEFA=', 'Content-Length': '0'} 2023-10-18T21:28:26.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000079454eb1857653-0065304514-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=bofrnf-61 2023-10-18T21:28:26.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:26.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:28:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [ 2023-10-18T21:28:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697662228.817590_1239.1", 2023-10-18T21:28:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:50:28.817590Z" 2023-10-18T21:28:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:28:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:26.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "0.000000" 2023-10-18T21:28:26.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:26.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:28:26.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:26.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697662228.817590_1239.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:26.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-18T21:28:26.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:26.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:26.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:26.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:26.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:28.996+0000 7fbbd3950780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:28.997+0000 7fbbd3950780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:26.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:26.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:26.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:26.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:26.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:26.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:26.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697662228.811524_1230.1", 2023-10-18T21:28:26.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:26.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:26.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:26.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:50:28.811524Z", 2023-10-18T21:28:26.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:26.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:26.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:26.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:26.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:26.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:26.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:26.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:26.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:26.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:26.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:26.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:26.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:26.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:26.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:26.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:26.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:26.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:26.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:26.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:26.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:26.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:26.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:26.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:26.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:26.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:26.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:26.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:26.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T20:50:28Z" 2023-10-18T21:28:26.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:26.883 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:26.883 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:26.883 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:26.884 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:26.884 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:26.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.885 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.885 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:26.885 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:26.886 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:26.886 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:26.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:26.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1697662228.811524_1230.1", 2023-10-18T21:28:26.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:26.887 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:26.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:26.888 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:50:28.811524Z", 2023-10-18T21:28:26.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.888 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.889 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.889 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:26.889 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:26.890 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:26.890 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:26.890 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:26.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:26.891 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:26.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:26.891 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:26.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.892 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.892 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.893 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:26.893 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:26.893 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:26.893 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:26.894 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:26.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:26.894 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:26.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:26.895 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:26.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.896 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.896 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.896 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:26.897 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:26.897 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:26.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:26.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:26.898 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:26.898 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:26.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:26.899 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:26.899 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.899 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.900 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.900 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:26.900 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.900 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:26.901 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:26.901 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:26.901 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.902 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T20:50:28Z" 2023-10-18T21:28:26.902 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:26.902 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697662228.811524_1230.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:50:28.811524Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:26.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697662228.817590_1239.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:26.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697662228.811524_1230.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:26.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697662228.817590_1239.1 target=1_1697662228.811524_1230.1 2023-10-18T21:28:26.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:26.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:26.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:26.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:26.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:26.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:34.122+0000 7f16dd5d9780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:26.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:34.123+0000 7f16dd5d9780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:26.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:26.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:26.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:26.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:26.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:26.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:26.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:26.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:26.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:26.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697662228.817590_1239.1", 2023-10-18T21:28:26.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:26.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:26.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:26.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:50:28.817590Z", 2023-10-18T21:28:26.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:26.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:26.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:26.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:26.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:26.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:26.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:26.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:26.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:26.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:26.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:26.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:26.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:26.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:26.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:26.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:26.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:26.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:26.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:26.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:26.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:26.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:26.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:26.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:26.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:26.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:26.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:26.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:26.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:26.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T20:50:34Z" 2023-10-18T21:28:26.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:26.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:26.926 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:26.926 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:26.926 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:26.927 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:26.927 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:26.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.928 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.928 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:26.928 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:26.929 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:26.929 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:26.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:26.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1697662228.817590_1239.1", 2023-10-18T21:28:26.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:26.930 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:26.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:26.931 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:50:28.817590Z", 2023-10-18T21:28:26.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.931 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.931 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.932 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:26.932 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:26.932 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:26.933 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:26.933 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:26.933 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:26.934 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:26.934 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:26.934 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:26.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.935 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.935 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.935 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:26.936 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:26.936 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:26.936 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:26.937 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:26.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:26.937 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:26.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:26.938 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:26.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.938 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.939 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.939 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:26.939 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:26.940 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:26.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:26.940 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:26.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:26.941 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:26.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:26.941 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:26.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:26.942 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.942 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:26.943 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:26.943 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.943 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:26.944 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:26.944 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:26.944 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:26.944 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T20:50:34Z" 2023-10-18T21:28:26.945 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:26.945 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697662228.817590_1239.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:50:28.817590Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:26.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697662228.817590_1239.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:26.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697662228.817590_1239.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:26.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-18T21:28:26.947 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-10-18T21:28:26.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-10-18T21:28:26.947 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:50:34.157+0000 7f77112ab700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 172938) UID: 0 2023-10-18T21:28:26.947 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T20:50:34.157+0000 7f77155147c0 -1 shutting down 2023-10-18T21:28:26.948 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T21:28:26.948 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~6s 2023-10-18T21:28:26.948 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-10-18T21:28:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.952 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.952 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.953 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.953 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:26.953 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.954 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:26.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:26.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c90ab847996939df-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj11 2023-10-18T21:28:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj11 2023-10-18T21:28:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj11 2023-10-18T21:28:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.958 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:26.958 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:26.959 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:26.959 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:26.959 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj11 2023-10-18T21:28:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.960 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:+zCs3horFSsfJ9Q8cRTA1gigEUg= 2023-10-18T21:28:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+zCs3horFSsfJ9Q8cRTA1gigEUg='} 2023-10-18T21:28:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5004741568d9239-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.964 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.964 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.965 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.965 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:26.965 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.966 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:26.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:26.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093049e21dc437e2c-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj22 2023-10-18T21:28:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj22 2023-10-18T21:28:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj22 2023-10-18T21:28:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.970 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:26.970 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:26.970 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:26.971 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:26.971 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj22 2023-10-18T21:28:26.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.972 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3Z/0aXtkPLKIzLd4eWFlBO3EcSQ= 2023-10-18T21:28:26.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3Z/0aXtkPLKIzLd4eWFlBO3EcSQ='} 2023-10-18T21:28:26.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f338146c80ad387a-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:26.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:26.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:26.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.976 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.976 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.976 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.977 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:26.977 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:26.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.978 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fb63639a86c1f6a-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj33 2023-10-18T21:28:26.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj33 2023-10-18T21:28:26.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:26.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj33 2023-10-18T21:28:26.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:26.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.982 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:26.982 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:26.982 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:26.983 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:26.983 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj33 2023-10-18T21:28:26.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.984 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:mwqsHeGCbUI7F62nEipJJbzQa5I= 2023-10-18T21:28:26.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mwqsHeGCbUI7F62nEipJJbzQa5I='} 2023-10-18T21:28:26.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ccb49bf4980694a3-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:26.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:26.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:26.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.988 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:26.988 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.988 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:26.989 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:26.989 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:26.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.990 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:26.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:26.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015e1978443bef1e5-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj44 2023-10-18T21:28:26.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj44 2023-10-18T21:28:26.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:26.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj44 2023-10-18T21:28:26.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:26.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:26.994 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:26.994 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:26.994 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:26.995 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:26.995 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj44 2023-10-18T21:28:26.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:26.996 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:QqmSNimUVEQL6mbSG6kYV2S8ZvI= 2023-10-18T21:28:26.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QqmSNimUVEQL6mbSG6kYV2S8ZvI='} 2023-10-18T21:28:26.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087ad1c70cd9a20ba-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:26.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:26.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:26.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:26.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:26.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:26.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:26.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:26.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:26.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:26.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:26.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.000 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.000 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.000 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.001 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.001 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.001 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:27.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:27.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004732890c70be52bd-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj55 2023-10-18T21:28:27.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj55 2023-10-18T21:28:27.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj55 2023-10-18T21:28:27.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.006 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.006 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.006 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.006 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.007 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj55 2023-10-18T21:28:27.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.007 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:MaKOW0fBYJAoGaOUMTv57xg7TWw= 2023-10-18T21:28:27.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MaKOW0fBYJAoGaOUMTv57xg7TWw='} 2023-10-18T21:28:27.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddfd97f8f015cd2b-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.008 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.008 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-61 2023-10-18T21:28:27.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 5 2023-10-18T21:28:27.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.014 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.014 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.015 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.015 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.015 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.016 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f18ecc27e5cbbbd-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj17 2023-10-18T21:28:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj17 2023-10-18T21:28:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj17 2023-10-18T21:28:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.020 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.020 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.021 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.021 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.021 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj17 2023-10-18T21:28:27.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.022 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:5X5vRXOXbBl4waee9PSCNuzebh4= 2023-10-18T21:28:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5X5vRXOXbBl4waee9PSCNuzebh4='} 2023-10-18T21:28:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fd014ba2cc1cfb1-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.026 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.026 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.027 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.027 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.027 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.028 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:27.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:27.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4643c9fd002e102-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj34 2023-10-18T21:28:27.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj34 2023-10-18T21:28:27.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj34 2023-10-18T21:28:27.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.032 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.032 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.033 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.033 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.033 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj34 2023-10-18T21:28:27.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.034 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:5M45YM0XfKwfRwpOqsIRgLvndrM= 2023-10-18T21:28:27.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5M45YM0XfKwfRwpOqsIRgLvndrM='} 2023-10-18T21:28:27.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b958ecc194de3fbd-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.038 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.038 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.039 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.039 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.039 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.040 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:27.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:27.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd9e751965bcb58f-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj51 2023-10-18T21:28:27.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj51 2023-10-18T21:28:27.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj51 2023-10-18T21:28:27.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.044 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.044 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.045 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.045 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.045 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj51 2023-10-18T21:28:27.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.046 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ZwjUOfM9Rb1CI3BkB0cSRfwRzpI= 2023-10-18T21:28:27.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ZwjUOfM9Rb1CI3BkB0cSRfwRzpI='} 2023-10-18T21:28:27.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4bb2c7d85e2e1f1-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.050 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.050 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.051 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.051 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.051 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.052 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:27.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:27.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002441051dc06a3b82-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj68 2023-10-18T21:28:27.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj68 2023-10-18T21:28:27.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj68 2023-10-18T21:28:27.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.056 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.056 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.056 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.057 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.057 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj68 2023-10-18T21:28:27.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.058 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:P7r0zKFLMCP68ag0Y98P+3HdF10= 2023-10-18T21:28:27.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P7r0zKFLMCP68ag0Y98P+3HdF10='} 2023-10-18T21:28:27.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6a2dd6762636311-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.062 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.062 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.062 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.063 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.063 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.064 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:27.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:27.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a0dcb6c9716b5d4-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj85 2023-10-18T21:28:27.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj85 2023-10-18T21:28:27.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj85 2023-10-18T21:28:27.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.068 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.068 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.068 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.069 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.069 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj85 2023-10-18T21:28:27.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.070 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ERDYnguzDBJ5bvHz2psiWcpzwUc= 2023-10-18T21:28:27.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ERDYnguzDBJ5bvHz2psiWcpzwUc='} 2023-10-18T21:28:27.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000673e0bfd1813a046-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-61 2023-10-18T21:28:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 10 2023-10-18T21:28:27.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.076 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.077 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.077 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.077 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.078 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.078 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:27.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:27.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027ab0f3d003f4a9e-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj19 2023-10-18T21:28:27.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj19 2023-10-18T21:28:27.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj19 2023-10-18T21:28:27.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.082 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.083 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.083 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.083 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.084 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj19 2023-10-18T21:28:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.084 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:hO/RFzcTEKPT1MG/KH57NAWRE94= 2023-10-18T21:28:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hO/RFzcTEKPT1MG/KH57NAWRE94='} 2023-10-18T21:28:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a579d3a8ac891f21-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.088 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.089 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.089 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.089 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.090 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.090 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI= 2023-10-18T21:28:27.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PFiGl9Dj8iceCSl+9z7a2Jn0KsI=', 'Content-Length': '0'} 2023-10-18T21:28:27.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b0855f954684678-0065304520-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj38 2023-10-18T21:28:27.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj38 2023-10-18T21:28:27.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj38 2023-10-18T21:28:27.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.094 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.095 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.095 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.095 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:40 GMT 2023-10-18T21:28:27.096 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj38 2023-10-18T21:28:27.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.096 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Hc+7qJO3uA0saBzoOHaLzVm2e5A= 2023-10-18T21:28:27.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:40 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Hc+7qJO3uA0saBzoOHaLzVm2e5A='} 2023-10-18T21:28:27.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4fab1709946a2d8-0065304520-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.100 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.101 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.101 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.101 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.102 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.102 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093cf7d6ae72a6516-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj57 2023-10-18T21:28:27.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj57 2023-10-18T21:28:27.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj57 2023-10-18T21:28:27.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.106 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.107 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.107 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.107 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.108 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj57 2023-10-18T21:28:27.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.108 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:As+YGEoMJP7Vd1swA0373E3uNHw= 2023-10-18T21:28:27.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:As+YGEoMJP7Vd1swA0373E3uNHw='} 2023-10-18T21:28:27.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e03945061ef6763-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.112 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.113 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.113 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.113 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.114 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.114 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000241dd375b3d4e8ce-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj76 2023-10-18T21:28:27.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj76 2023-10-18T21:28:27.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj76 2023-10-18T21:28:27.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.119 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.119 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.119 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.119 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.120 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj76 2023-10-18T21:28:27.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.120 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:AcOHfRQZ0tP7XyxHBetqFwIjobs= 2023-10-18T21:28:27.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AcOHfRQZ0tP7XyxHBetqFwIjobs='} 2023-10-18T21:28:27.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000788b530c23c72b01-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.124 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.125 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.125 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.125 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.126 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.126 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edac04ba3a2851b0-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj95 2023-10-18T21:28:27.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj95 2023-10-18T21:28:27.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj95 2023-10-18T21:28:27.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.130 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.131 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.131 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.131 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.132 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj95 2023-10-18T21:28:27.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.132 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:LatHDjB7IOO1QMhS4GQR4BONr9Q= 2023-10-18T21:28:27.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:LatHDjB7IOO1QMhS4GQR4BONr9Q='} 2023-10-18T21:28:27.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d883cb0486ff6b60-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-61 2023-10-18T21:28:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 15 2023-10-18T21:28:27.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.139 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.139 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.139 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.140 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.140 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.140 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e6547115ca64973-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj23 2023-10-18T21:28:27.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj23 2023-10-18T21:28:27.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj23 2023-10-18T21:28:27.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.145 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.145 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.145 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.145 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.146 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj23 2023-10-18T21:28:27.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.146 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:JXfhaUcaKlwQHGZZxMzzRQkNxaw= 2023-10-18T21:28:27.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:JXfhaUcaKlwQHGZZxMzzRQkNxaw='} 2023-10-18T21:28:27.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006577e658af898f67-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.151 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.151 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.151 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.151 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.152 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.152 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc716804c106130d-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj46 2023-10-18T21:28:27.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj46 2023-10-18T21:28:27.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj46 2023-10-18T21:28:27.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.156 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.157 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.157 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.157 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.158 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj46 2023-10-18T21:28:27.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.158 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:EcXWhEheRv891gG5uk1m00F/kg8= 2023-10-18T21:28:27.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EcXWhEheRv891gG5uk1m00F/kg8='} 2023-10-18T21:28:27.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028bd3f971e3f53bb-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.162 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.163 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.163 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.163 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.164 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.164 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aaaca00babe5b4c-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj69 2023-10-18T21:28:27.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj69 2023-10-18T21:28:27.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj69 2023-10-18T21:28:27.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.168 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.169 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.169 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.169 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.170 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj69 2023-10-18T21:28:27.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.170 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yQxs/Cvmtyz3B+6pEtPQ62sk2K0= 2023-10-18T21:28:27.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yQxs/Cvmtyz3B+6pEtPQ62sk2K0='} 2023-10-18T21:28:27.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079a7200a2fb66790-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.174 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.175 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.175 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.175 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.175 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.176 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f62295d458aedb88-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj92 2023-10-18T21:28:27.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj92 2023-10-18T21:28:27.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj92 2023-10-18T21:28:27.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.180 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.181 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.181 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.181 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.181 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj92 2023-10-18T21:28:27.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.182 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:P+vWJizPVOLr1MjBBkG1sNVleEo= 2023-10-18T21:28:27.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:P+vWJizPVOLr1MjBBkG1sNVleEo='} 2023-10-18T21:28:27.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f535b1715b56b04-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.186 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.186 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.187 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.187 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.187 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.188 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e7ec169ad9e8e2a-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj115 2023-10-18T21:28:27.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj115 2023-10-18T21:28:27.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj115 2023-10-18T21:28:27.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.192 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.192 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.193 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.193 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.193 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj115 2023-10-18T21:28:27.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.194 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yPI1G5UlPXkmrYyOnY7pogDwP+Y= 2023-10-18T21:28:27.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yPI1G5UlPXkmrYyOnY7pogDwP+Y='} 2023-10-18T21:28:27.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009664eedf1f6b8f44-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.195 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.195 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-61 2023-10-18T21:28:27.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 20 2023-10-18T21:28:27.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.201 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.201 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.201 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.201 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.202 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.202 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e355fb93b69aee75-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj29 2023-10-18T21:28:27.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj29 2023-10-18T21:28:27.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj29 2023-10-18T21:28:27.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.207 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.207 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.207 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.208 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.208 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj29 2023-10-18T21:28:27.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.208 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KU/tlibYxiEjkDISWDyU7RIvKjo= 2023-10-18T21:28:27.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KU/tlibYxiEjkDISWDyU7RIvKjo='} 2023-10-18T21:28:27.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b69c3557b8763bf0-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.213 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.213 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.213 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.213 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.214 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.214 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf4abadc293bfcb5-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj58 2023-10-18T21:28:27.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj58 2023-10-18T21:28:27.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj58 2023-10-18T21:28:27.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.219 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.219 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.219 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.220 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.220 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj58 2023-10-18T21:28:27.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.220 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ECT3PO3ZOEmHa3ZLotUr/KFIH80= 2023-10-18T21:28:27.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ECT3PO3ZOEmHa3ZLotUr/KFIH80='} 2023-10-18T21:28:27.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046ef1a2b5b82e0c8-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.225 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.225 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.225 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.225 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.226 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.226 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff9e3d4ddd207ae1-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj87 2023-10-18T21:28:27.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj87 2023-10-18T21:28:27.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj87 2023-10-18T21:28:27.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.231 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.231 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.231 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.232 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.232 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj87 2023-10-18T21:28:27.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.232 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:BKoajGojlaOUkhSgSaZd68FkIq4= 2023-10-18T21:28:27.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BKoajGojlaOUkhSgSaZd68FkIq4='} 2023-10-18T21:28:27.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000481015a50fbb848-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.237 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.237 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.237 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.238 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.238 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.238 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061532d843257603a-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj116 2023-10-18T21:28:27.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj116 2023-10-18T21:28:27.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj116 2023-10-18T21:28:27.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.243 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.243 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.243 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.244 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.244 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj116 2023-10-18T21:28:27.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.245 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:hXcdwUyi6brj7Kt5bYVtSqZ+BM0= 2023-10-18T21:28:27.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hXcdwUyi6brj7Kt5bYVtSqZ+BM0='} 2023-10-18T21:28:27.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023a5734ef7d569b2-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.249 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.249 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.249 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.250 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.250 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.251 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d2ebb4179565dab-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj145 2023-10-18T21:28:27.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj145 2023-10-18T21:28:27.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj145 2023-10-18T21:28:27.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.255 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.255 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.256 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.256 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.256 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj145 2023-10-18T21:28:27.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.257 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:7PAcXBJ7iwzEGHK3brvBgF4ZMOU= 2023-10-18T21:28:27.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7PAcXBJ7iwzEGHK3brvBgF4ZMOU='} 2023-10-18T21:28:27.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b285c96010bb1a0e-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-61 2023-10-18T21:28:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 25 2023-10-18T21:28:27.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.263 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.264 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.264 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.264 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.264 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.265 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9939f072f0f9a36-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj31 2023-10-18T21:28:27.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj31 2023-10-18T21:28:27.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj31 2023-10-18T21:28:27.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.269 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.270 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.270 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.270 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.271 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj31 2023-10-18T21:28:27.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.271 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KWOYt6XP/F2DNu5XcWGZmOXwST4= 2023-10-18T21:28:27.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KWOYt6XP/F2DNu5XcWGZmOXwST4='} 2023-10-18T21:28:27.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ba98501fe123faa-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.275 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.276 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.276 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.276 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.276 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.277 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ= 2023-10-18T21:28:27.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:z3s1FzxKodh+kFkWQYhaS/SRvXQ=', 'Content-Length': '0'} 2023-10-18T21:28:27.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbbcfc24b9ab4394-0065304521-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj62 2023-10-18T21:28:27.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj62 2023-10-18T21:28:27.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj62 2023-10-18T21:28:27.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.281 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.281 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.282 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.282 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:41 GMT 2023-10-18T21:28:27.282 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj62 2023-10-18T21:28:27.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.283 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:p9yfzgZ14/6RN1dTEnE78uvpbJE= 2023-10-18T21:28:27.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:41 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:p9yfzgZ14/6RN1dTEnE78uvpbJE='} 2023-10-18T21:28:27.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9d5e22e0a94ad49-0065304521-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.287 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.287 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.288 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.288 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.288 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.289 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T21:28:27.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T21:28:27.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f08eb02e8d6a81d3-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj93 2023-10-18T21:28:27.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj93 2023-10-18T21:28:27.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj93 2023-10-18T21:28:27.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.293 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.293 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.294 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.294 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.294 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj93 2023-10-18T21:28:27.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.295 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:iINOw8rFtbRHBHbdM18CkO8cgyQ= 2023-10-18T21:28:27.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iINOw8rFtbRHBHbdM18CkO8cgyQ='} 2023-10-18T21:28:27.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcb24477849463e8-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.299 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.299 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.300 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.300 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.300 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.301 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T21:28:27.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T21:28:27.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018e47b021cfe3168-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj124 2023-10-18T21:28:27.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj124 2023-10-18T21:28:27.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj124 2023-10-18T21:28:27.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.305 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.305 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.306 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.306 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.306 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj124 2023-10-18T21:28:27.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.307 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i+ur+qcqTEwrj3EfKRY5xIQr1gE= 2023-10-18T21:28:27.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i+ur+qcqTEwrj3EfKRY5xIQr1gE='} 2023-10-18T21:28:27.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a53fbe1ce2b8406d-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.311 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.311 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.312 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.312 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.312 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.313 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T21:28:27.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T21:28:27.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000300c4006a84e89f5-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj155 2023-10-18T21:28:27.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj155 2023-10-18T21:28:27.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj155 2023-10-18T21:28:27.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.317 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.318 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.318 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.318 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.318 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj155 2023-10-18T21:28:27.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.319 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:y5aKCZnIdGhKfjUy/xkqOQqJBWI= 2023-10-18T21:28:27.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y5aKCZnIdGhKfjUy/xkqOQqJBWI='} 2023-10-18T21:28:27.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e106619f4fd2197-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.320 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.320 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-61 2023-10-18T21:28:27.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 30 2023-10-18T21:28:27.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.326 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.326 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.326 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.327 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.327 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.327 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T21:28:27.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T21:28:27.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6d025b11b77e4fe-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj37 2023-10-18T21:28:27.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj37 2023-10-18T21:28:27.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj37 2023-10-18T21:28:27.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.332 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.332 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.332 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.333 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.333 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj37 2023-10-18T21:28:27.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.334 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:dAuCCm8qDeTOx1mwhewHofiB0KA= 2023-10-18T21:28:27.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dAuCCm8qDeTOx1mwhewHofiB0KA='} 2023-10-18T21:28:27.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075b5bdd786afe7c6-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.338 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.338 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.338 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.339 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.339 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.340 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T21:28:27.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T21:28:27.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000677ec04ac70b676f-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj74 2023-10-18T21:28:27.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj74 2023-10-18T21:28:27.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj74 2023-10-18T21:28:27.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.344 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.344 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.344 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.345 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.345 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj74 2023-10-18T21:28:27.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.346 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:RbtSlfZpzFjwDpS1dznruxBa2yw= 2023-10-18T21:28:27.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RbtSlfZpzFjwDpS1dznruxBa2yw='} 2023-10-18T21:28:27.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000769eccab0ca46c58-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.350 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.350 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.350 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.351 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.351 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.352 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T21:28:27.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T21:28:27.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6d2707a8865fba0-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj111 2023-10-18T21:28:27.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj111 2023-10-18T21:28:27.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj111 2023-10-18T21:28:27.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.356 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.356 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.356 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.357 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.357 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj111 2023-10-18T21:28:27.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.357 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:iLrH5XE3LLuQVMmMKYJla6T2Bkg= 2023-10-18T21:28:27.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iLrH5XE3LLuQVMmMKYJla6T2Bkg='} 2023-10-18T21:28:27.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d3175f69fa98bef-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.362 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.362 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.362 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.363 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.363 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.364 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T21:28:27.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T21:28:27.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000651cb6076b73e2fa-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj148 2023-10-18T21:28:27.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj148 2023-10-18T21:28:27.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj148 2023-10-18T21:28:27.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.368 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.368 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.369 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.369 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.369 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj148 2023-10-18T21:28:27.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.370 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4GcIJKzhOlZAMq5YzkCDHhLPi1Y= 2023-10-18T21:28:27.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4GcIJKzhOlZAMq5YzkCDHhLPi1Y='} 2023-10-18T21:28:27.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015c5f6ac9873eb58-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/ 2023-10-18T21:28:27.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/ 2023-10-18T21:28:27.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/ 2023-10-18T21:28:27.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.374 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.374 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.375 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.375 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.375 INFO:tasks.rgw_multisite_tests:/bofrnf-61/ 2023-10-18T21:28:27.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.376 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8= 2023-10-18T21:28:27.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4gB822JTqA1GLIe0i/wNowJagY8=', 'Content-Length': '0'} 2023-10-18T21:28:27.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4c4717726dc6e30-0065304522-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-61/obj185 2023-10-18T21:28:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-61/obj185 2023-10-18T21:28:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-61/obj185 2023-10-18T21:28:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.380 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.380 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.381 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.381 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:50:42 GMT 2023-10-18T21:28:27.381 INFO:tasks.rgw_multisite_tests:/bofrnf-61/obj185 2023-10-18T21:28:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.382 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ByDtyjupSDpNvPEWRQiVgc4xjYo= 2023-10-18T21:28:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:50:42 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ByDtyjupSDpNvPEWRQiVgc4xjYo='} 2023-10-18T21:28:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2407c36c637750d-0065304522-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:50:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:42.575+0000 7f573e733780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-18T21:28:27.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:42.577+0000 7f573e733780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-18T21:28:27.385 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-10-18T21:28:27.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T21:28:27.385 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-10-18T21:28:27.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:27.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T21:28:27.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-18T21:28:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-10-18T21:28:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T21:28:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T21:28:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-18T21:28:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T21:28:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: anonymous 2023-10-18T21:28:27.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=bofrnf-61 2023-10-18T21:28:27.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:27.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:43.780+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.391 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-10-18T21:28:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:48.781+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:53.781+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:50:58.781+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:03.782+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:08.783+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:13.784+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:18.784+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:23.785+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:28.785+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:33.786+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:38.786+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:43.787+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:48.788+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:53.787+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:51:58.788+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:03.789+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:08.790+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:13.791+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:18.791+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:23.792+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:28.792+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:33.793+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:38.794+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:43.794+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:48.794+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:53.795+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:52:58.797+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:03.797+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:08.798+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:13.799+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:18.799+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:23.800+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:28.800+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:33.800+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:38.801+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:43.801+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:48.803+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:53.804+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:53:58.804+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:03.805+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:08.806+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:13.806+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:18.807+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:23.807+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:28.807+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:33.808+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:38.809+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:43.810+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:48.810+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:53.811+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:54:58.812+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:03.813+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:08.813+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:13.814+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:18.813+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:23.814+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:28.814+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:33.816+0000 7f7461192780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:38.817+0000 7f7461192780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-10-18T21:28:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:38.817+0000 7f7461192780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:28:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:55:38.817+0000 7f7461192780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:28:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-18T21:28:27.413 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-18T21:28:27.413 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.414 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-18T21:28:27.414 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-10-18T21:28:27.415 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-18T21:28:27.415 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-18T21:28:27.415 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-18T21:28:27.416 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-18T21:28:27.416 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 1842, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-10-18T21:28:27.416 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-18T21:28:27.417 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-18T21:28:27.417 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-18T21:28:27.417 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-18T21:28:27.418 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-18T21:28:27.418 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-18T21:28:27.418 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-18T21:28:27.418 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-18T21:28:27.419 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-18T21:28:27.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.422 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.423 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.423 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.423 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:29 GMT 2023-10-18T21:28:27.424 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.424 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:esiPySdIkgPqGBUYBPOOjzeMmCI= 2023-10-18T21:28:27.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:esiPySdIkgPqGBUYBPOOjzeMmCI=', 'Content-Length': '0'} 2023-10-18T21:28:27.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000cf0a49e39ccc2b1e-00653046f5-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=bofrnf-63 2023-10-18T21:28:27.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:27.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:28:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [ 2023-10-18T21:28:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697662709.769954_1341.1", 2023-10-18T21:28:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:58:29.769954Z" 2023-10-18T21:28:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:28:27.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "0.000000" 2023-10-18T21:28:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:27.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:28:27.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:27.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697662709.769954_1341.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:27.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-18T21:28:27.433 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:27.434 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:27.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:27.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:29.955+0000 7fc902dc2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:27.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:29.956+0000 7fc902dc2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:27.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:27.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:27.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:27.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:27.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:27.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:27.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:27.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:27.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:27.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697662709.769954_1341.1", 2023-10-18T21:28:27.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:27.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:27.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:27.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:58:29.769954Z", 2023-10-18T21:28:27.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:27.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:27.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:27.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:27.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:27.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:27.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:27.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:27.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:27.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:27.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:27.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:27.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:27.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:27.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:27.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:27.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:27.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:27.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:27.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T20:58:29Z" 2023-10-18T21:28:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:27.455 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:27.456 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:27.456 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:27.456 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:27.457 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:27.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.457 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.458 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:27.458 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:27.458 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:27.458 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:27.459 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:27.459 INFO:tasks.rgw_multisite_tests: "marker": "1_1697662709.769954_1341.1", 2023-10-18T21:28:27.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:27.460 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:27.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:27.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:58:29.769954Z", 2023-10-18T21:28:27.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.461 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.461 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.461 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:27.462 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:27.462 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:27.462 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:27.463 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:27.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:27.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:27.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:27.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:27.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.465 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.465 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.465 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:27.465 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:27.466 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:27.466 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:27.466 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:27.467 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:27.467 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:27.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:27.468 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:27.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.468 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.468 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.469 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:27.469 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:27.469 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:27.470 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:27.470 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:27.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:27.471 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:27.471 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:27.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:27.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.472 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.472 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.472 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:27.473 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.473 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:27.473 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:27.474 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:27.474 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.474 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T20:58:29Z" 2023-10-18T21:28:27.475 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:27.475 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697662709.769954_1341.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T20:58:29.769954Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:27.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697662709.769954_1341.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:27.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697662709.769954_1341.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:27.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-18T21:28:27.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.480 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.480 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.480 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.481 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:29 GMT 2023-10-18T21:28:27.481 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.482 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:+E8hJKqrJZOCO18BDa8B0RfdY5Q= 2023-10-18T21:28:27.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+E8hJKqrJZOCO18BDa8B0RfdY5Q=', 'Content-Length': '0'} 2023-10-18T21:28:27.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000626cc1889a7a7828-00653046f5-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj11 2023-10-18T21:28:27.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj11 2023-10-18T21:28:27.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj11 2023-10-18T21:28:27.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.486 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.486 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.486 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.487 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:29 GMT 2023-10-18T21:28:27.487 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj11 2023-10-18T21:28:27.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.488 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Xaru9oMWL8Zy5DnVtl0RzMm4kMM= 2023-10-18T21:28:27.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:29 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xaru9oMWL8Zy5DnVtl0RzMm4kMM='} 2023-10-18T21:28:27.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d7417adebad4228-00653046f5-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.492 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.492 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.492 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.493 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.493 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.494 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d87182a0a6cd6da-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj22 2023-10-18T21:28:27.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj22 2023-10-18T21:28:27.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj22 2023-10-18T21:28:27.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.498 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.498 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.498 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.499 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.499 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj22 2023-10-18T21:28:27.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.500 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3FXZjzhTaKOKqosy11k1TnYNLb4= 2023-10-18T21:28:27.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3FXZjzhTaKOKqosy11k1TnYNLb4='} 2023-10-18T21:28:27.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051336ccf498cfc5e-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.504 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.504 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.504 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.505 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.505 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.506 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009791abb35afb66b-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj33 2023-10-18T21:28:27.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj33 2023-10-18T21:28:27.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj33 2023-10-18T21:28:27.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.510 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.510 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.511 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.511 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.511 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj33 2023-10-18T21:28:27.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.512 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:FRMLYArKTWTcS3HRXOkQPqGQGG0= 2023-10-18T21:28:27.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FRMLYArKTWTcS3HRXOkQPqGQGG0='} 2023-10-18T21:28:27.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0583cec00b111e1-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.517 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.517 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.518 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.518 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.518 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.519 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029decb1f1c584234-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj44 2023-10-18T21:28:27.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj44 2023-10-18T21:28:27.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj44 2023-10-18T21:28:27.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.523 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.523 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.524 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.524 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.524 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj44 2023-10-18T21:28:27.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.525 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1g1pud1A0JKYLVBTdqxIk9TIE5Q= 2023-10-18T21:28:27.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1g1pud1A0JKYLVBTdqxIk9TIE5Q='} 2023-10-18T21:28:27.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029516b1c1b58f7c5-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.529 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.530 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.530 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.530 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.531 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.531 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095f0917cf1ff6a79-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj55 2023-10-18T21:28:27.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj55 2023-10-18T21:28:27.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj55 2023-10-18T21:28:27.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.536 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.536 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.536 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.536 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.537 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj55 2023-10-18T21:28:27.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.537 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Gqw50Q50/TdhqLeuyJRVCOTrd0M= 2023-10-18T21:28:27.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Gqw50Q50/TdhqLeuyJRVCOTrd0M='} 2023-10-18T21:28:27.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4eafa0d09e3b585-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-63 2023-10-18T21:28:27.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 5 2023-10-18T21:28:27.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.544 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.544 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.545 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.545 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.545 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.546 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004de7999d888f5688-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj17 2023-10-18T21:28:27.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj17 2023-10-18T21:28:27.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj17 2023-10-18T21:28:27.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.550 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.551 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.551 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.551 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.551 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj17 2023-10-18T21:28:27.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.552 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:/LrFkzu55NzrKRVhKtzKVZEkP7U= 2023-10-18T21:28:27.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/LrFkzu55NzrKRVhKtzKVZEkP7U='} 2023-10-18T21:28:27.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000185c546249df269c-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.556 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.557 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.557 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.557 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.558 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.558 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9fc55a1e2c67962-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj34 2023-10-18T21:28:27.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj34 2023-10-18T21:28:27.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj34 2023-10-18T21:28:27.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.562 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.563 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.563 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.563 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.564 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj34 2023-10-18T21:28:27.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.564 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:5e7OgFW/6X0qHaFaLpzc49irIq0= 2023-10-18T21:28:27.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5e7OgFW/6X0qHaFaLpzc49irIq0='} 2023-10-18T21:28:27.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fa1b60ac4e79875-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.568 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.569 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.569 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.569 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.570 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.570 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e782163d7cffa5e1-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj51 2023-10-18T21:28:27.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj51 2023-10-18T21:28:27.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj51 2023-10-18T21:28:27.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.575 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.575 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.575 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.576 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.576 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj51 2023-10-18T21:28:27.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.577 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ju6NJioPld1uceh4j2naIo7zYrw= 2023-10-18T21:28:27.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ju6NJioPld1uceh4j2naIo7zYrw='} 2023-10-18T21:28:27.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db20aade8ee4b112-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.581 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.581 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.581 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.582 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.582 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.582 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d561a278245fca0c-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj68 2023-10-18T21:28:27.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj68 2023-10-18T21:28:27.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj68 2023-10-18T21:28:27.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.587 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.587 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.587 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.588 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.588 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj68 2023-10-18T21:28:27.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.588 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:OWyT3IGRStQ7sf7T0dVggwvQRpw= 2023-10-18T21:28:27.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:OWyT3IGRStQ7sf7T0dVggwvQRpw='} 2023-10-18T21:28:27.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001753c9089e17cf88-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.593 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.593 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.593 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.593 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.594 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.594 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1fc373fe33130ca-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj85 2023-10-18T21:28:27.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj85 2023-10-18T21:28:27.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj85 2023-10-18T21:28:27.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.599 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.599 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.599 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.599 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.600 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj85 2023-10-18T21:28:27.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.600 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Viy5nZ/Ux9BaxM0wN1kIsF0EtlU= 2023-10-18T21:28:27.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Viy5nZ/Ux9BaxM0wN1kIsF0EtlU='} 2023-10-18T21:28:27.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001f45f489d65e81c-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-63 2023-10-18T21:28:27.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 10 2023-10-18T21:28:27.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.607 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.607 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.608 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.608 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.608 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.609 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000116dd4ccbda4bdc9-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj19 2023-10-18T21:28:27.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj19 2023-10-18T21:28:27.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj19 2023-10-18T21:28:27.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.613 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.613 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.614 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.614 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.614 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj19 2023-10-18T21:28:27.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.615 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:RBbDnHZMGztlmdj53XUImqMxxSA= 2023-10-18T21:28:27.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:RBbDnHZMGztlmdj53XUImqMxxSA='} 2023-10-18T21:28:27.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4a5513f305f6f48-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.619 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.619 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.620 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.620 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.620 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.621 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003af29c67302aec2e-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj38 2023-10-18T21:28:27.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj38 2023-10-18T21:28:27.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj38 2023-10-18T21:28:27.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.625 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.625 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.626 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.626 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.626 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj38 2023-10-18T21:28:27.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.627 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:PehESfneRAiSmw67BFXBTy+mnLM= 2023-10-18T21:28:27.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:PehESfneRAiSmw67BFXBTy+mnLM='} 2023-10-18T21:28:27.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000234d3bcb39ecd12f-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.631 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.631 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.631 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.632 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.632 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.632 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073860f8cfeb7647f-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj57 2023-10-18T21:28:27.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj57 2023-10-18T21:28:27.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj57 2023-10-18T21:28:27.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.637 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.637 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.637 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.637 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.638 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj57 2023-10-18T21:28:27.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.638 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:DVF64RZIRRdiDLgom7icLtbk6L4= 2023-10-18T21:28:27.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DVF64RZIRRdiDLgom7icLtbk6L4='} 2023-10-18T21:28:27.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007833e6b24e0fcaa3-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.642 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.642 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.643 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.643 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.643 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.644 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000924eaf0e8e55674e-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj76 2023-10-18T21:28:27.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj76 2023-10-18T21:28:27.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj76 2023-10-18T21:28:27.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.648 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.648 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.649 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.649 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.649 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj76 2023-10-18T21:28:27.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.650 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xZNcGuz5HJu72wQdWx3zU6npyvU= 2023-10-18T21:28:27.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xZNcGuz5HJu72wQdWx3zU6npyvU='} 2023-10-18T21:28:27.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000709f5a4399425dc6-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.654 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.654 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.655 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.655 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.655 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.656 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3a5c848236b71dc-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj95 2023-10-18T21:28:27.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj95 2023-10-18T21:28:27.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj95 2023-10-18T21:28:27.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.660 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.660 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.661 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.661 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.661 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj95 2023-10-18T21:28:27.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.662 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:g739m2NDOYln1G9PTRETiNKWK38= 2023-10-18T21:28:27.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:g739m2NDOYln1G9PTRETiNKWK38='} 2023-10-18T21:28:27.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000031123912827bfb5-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.663 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.663 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-63 2023-10-18T21:28:27.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 15 2023-10-18T21:28:27.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.669 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.669 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.669 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.669 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.670 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.670 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018320f291c3ab751-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj23 2023-10-18T21:28:27.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj23 2023-10-18T21:28:27.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj23 2023-10-18T21:28:27.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.675 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.675 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.675 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.675 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.676 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj23 2023-10-18T21:28:27.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.676 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:dK69cRXYuO0DR5azR8mHIX+hNdk= 2023-10-18T21:28:27.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:dK69cRXYuO0DR5azR8mHIX+hNdk='} 2023-10-18T21:28:27.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fd02e945a0428c4-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.680 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.681 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.681 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.681 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.682 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.682 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055943bc4977032ad-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj46 2023-10-18T21:28:27.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj46 2023-10-18T21:28:27.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj46 2023-10-18T21:28:27.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.686 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.687 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.687 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.687 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.688 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj46 2023-10-18T21:28:27.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.688 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:vBL45m0ayN+2ldl0KhlrjxIAUQk= 2023-10-18T21:28:27.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vBL45m0ayN+2ldl0KhlrjxIAUQk='} 2023-10-18T21:28:27.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022b0d1b9826f3300-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.692 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.693 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.693 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.693 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.694 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.694 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1a3b0d77cd31e60-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj69 2023-10-18T21:28:27.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj69 2023-10-18T21:28:27.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj69 2023-10-18T21:28:27.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.698 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.698 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.699 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.699 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.699 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj69 2023-10-18T21:28:27.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.700 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4Hjw6dnP7ew7GJX0K0e2FUCEv6o= 2023-10-18T21:28:27.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4Hjw6dnP7ew7GJX0K0e2FUCEv6o='} 2023-10-18T21:28:27.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6056d14e4660ef7-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.704 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.704 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.705 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.705 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.705 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.706 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c66a84565c859ede-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj92 2023-10-18T21:28:27.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj92 2023-10-18T21:28:27.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj92 2023-10-18T21:28:27.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.710 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.710 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.710 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.711 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.711 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj92 2023-10-18T21:28:27.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.712 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Fq/s6egPVugluMONwycQ1YSzVLM= 2023-10-18T21:28:27.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Fq/s6egPVugluMONwycQ1YSzVLM='} 2023-10-18T21:28:27.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1e6ba18ddef07d6-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.716 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.716 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.716 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.717 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.717 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.717 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8= 2023-10-18T21:28:27.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i4oFDOjh1IVc5Wra3H6yDsLqak8=', 'Content-Length': '0'} 2023-10-18T21:28:27.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e5aaa590b621fb1-00653046f6-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj115 2023-10-18T21:28:27.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj115 2023-10-18T21:28:27.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj115 2023-10-18T21:28:27.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.722 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.722 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.722 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.723 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:30 GMT 2023-10-18T21:28:27.723 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj115 2023-10-18T21:28:27.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.723 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:AlenhjI9HjheMPfoLWCMrzvNR5A= 2023-10-18T21:28:27.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:30 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:AlenhjI9HjheMPfoLWCMrzvNR5A='} 2023-10-18T21:28:27.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf040cbb49b271d2-00653046f6-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-63 2023-10-18T21:28:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 20 2023-10-18T21:28:27.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.730 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.730 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.731 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.731 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.731 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.732 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffddcee777021e01-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj29 2023-10-18T21:28:27.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj29 2023-10-18T21:28:27.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj29 2023-10-18T21:28:27.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.736 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.736 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.737 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.737 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.737 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj29 2023-10-18T21:28:27.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.738 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:5ESbkCNdQKcfMzDteJ+7X6TOobs= 2023-10-18T21:28:27.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:5ESbkCNdQKcfMzDteJ+7X6TOobs='} 2023-10-18T21:28:27.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000335695faa5c741dd-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.742 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.742 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.743 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.743 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.743 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.744 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bab2f24589b66548-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj58 2023-10-18T21:28:27.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj58 2023-10-18T21:28:27.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj58 2023-10-18T21:28:27.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.748 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.748 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.749 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.749 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.749 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj58 2023-10-18T21:28:27.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.750 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:K8gYmn+9/8nx7RYJlcbpoKnh9RY= 2023-10-18T21:28:27.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:K8gYmn+9/8nx7RYJlcbpoKnh9RY='} 2023-10-18T21:28:27.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6a1b19fe8a5f6f4-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.754 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.754 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.755 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.755 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.755 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.756 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da01c1c56aad83b2-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj87 2023-10-18T21:28:27.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj87 2023-10-18T21:28:27.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj87 2023-10-18T21:28:27.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.760 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.760 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.761 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.761 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.761 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj87 2023-10-18T21:28:27.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.762 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:8O3zSw8TgsuuqdRkaua31VJCSr4= 2023-10-18T21:28:27.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:8O3zSw8TgsuuqdRkaua31VJCSr4='} 2023-10-18T21:28:27.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d203ec4a370f934c-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.766 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.766 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.766 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.767 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.767 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.768 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c48a97173b63cbbc-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj116 2023-10-18T21:28:27.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj116 2023-10-18T21:28:27.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj116 2023-10-18T21:28:27.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.772 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.772 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.773 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.773 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.773 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj116 2023-10-18T21:28:27.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.774 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:UyP7Xv/QXuCJPyN7OllDoB6QzZc= 2023-10-18T21:28:27.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:UyP7Xv/QXuCJPyN7OllDoB6QzZc='} 2023-10-18T21:28:27.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff0ad44343c3bdc1-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.778 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.778 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.779 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.779 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.779 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.780 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdeb85826b692f38-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj145 2023-10-18T21:28:27.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj145 2023-10-18T21:28:27.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj145 2023-10-18T21:28:27.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.784 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.785 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.785 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.785 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.786 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj145 2023-10-18T21:28:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.786 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:VQ0CA4OgS+U8B76Xa25o7qKMrEE= 2023-10-18T21:28:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:VQ0CA4OgS+U8B76Xa25o7qKMrEE='} 2023-10-18T21:28:27.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbd5961a57906524-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-63 2023-10-18T21:28:27.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 25 2023-10-18T21:28:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.793 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.793 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.793 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.794 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.794 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.795 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a33b6c4497a214c-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj31 2023-10-18T21:28:27.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj31 2023-10-18T21:28:27.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj31 2023-10-18T21:28:27.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.799 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.799 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.799 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.800 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.800 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj31 2023-10-18T21:28:27.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.801 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:yhA18IgbRQKtt4OGQu3GGprw0zk= 2023-10-18T21:28:27.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:yhA18IgbRQKtt4OGQu3GGprw0zk='} 2023-10-18T21:28:27.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006909c8a8de842fd6-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.805 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.805 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.805 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.806 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.806 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.807 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000febdda13c97d1d37-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj62 2023-10-18T21:28:27.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj62 2023-10-18T21:28:27.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj62 2023-10-18T21:28:27.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.811 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.811 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.811 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.811 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.812 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj62 2023-10-18T21:28:27.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.812 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:kqYLlzD5bid1hbllk/ILJUqOqMY= 2023-10-18T21:28:27.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kqYLlzD5bid1hbllk/ILJUqOqMY='} 2023-10-18T21:28:27.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ed4c172487ff10f-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.816 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.817 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.817 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.817 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.818 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.818 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003009fab0fe2bbebc-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj93 2023-10-18T21:28:27.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj93 2023-10-18T21:28:27.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj93 2023-10-18T21:28:27.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.823 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.823 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.823 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.824 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.824 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj93 2023-10-18T21:28:27.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.825 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:549fEJ8JMkb+OmW/mWJCquzOKW0= 2023-10-18T21:28:27.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:549fEJ8JMkb+OmW/mWJCquzOKW0='} 2023-10-18T21:28:27.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3ede4cb7edbaf32-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.829 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.829 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.829 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.830 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.830 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.831 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000193a4f1ba1568d9f-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj124 2023-10-18T21:28:27.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj124 2023-10-18T21:28:27.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj124 2023-10-18T21:28:27.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.835 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.835 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.836 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.836 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.836 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj124 2023-10-18T21:28:27.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.837 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:69J+VS4By/LO7oCgneBR1SHi2hA= 2023-10-18T21:28:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:69J+VS4By/LO7oCgneBR1SHi2hA='} 2023-10-18T21:28:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087ccd83f93e157b1-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.841 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.841 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.842 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.842 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.842 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.843 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052b2c9b864e91441-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj155 2023-10-18T21:28:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj155 2023-10-18T21:28:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj155 2023-10-18T21:28:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.847 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.848 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.848 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.848 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.849 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj155 2023-10-18T21:28:27.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.849 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:21cV7j+LgtEy904i+NiTgPfmzZQ= 2023-10-18T21:28:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:21cV7j+LgtEy904i+NiTgPfmzZQ='} 2023-10-18T21:28:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b83e059bfecf3d44-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.851 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:27.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:27.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:27.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-63 2023-10-18T21:28:27.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 30 2023-10-18T21:28:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.856 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.857 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.857 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.857 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.858 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.858 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4fc535d8f9c4c12-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj37 2023-10-18T21:28:27.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj37 2023-10-18T21:28:27.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj37 2023-10-18T21:28:27.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.863 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.863 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.863 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.863 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.864 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj37 2023-10-18T21:28:27.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.864 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:49Pd/txzmMHEV6cZRNk0xyg0Eqo= 2023-10-18T21:28:27.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:49Pd/txzmMHEV6cZRNk0xyg0Eqo='} 2023-10-18T21:28:27.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2e7dd6e3d5906ad-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.868 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.869 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.869 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.869 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.870 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.870 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000065fac21ccbd2e5e-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj74 2023-10-18T21:28:27.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj74 2023-10-18T21:28:27.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj74 2023-10-18T21:28:27.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.874 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.875 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.875 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.875 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.875 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj74 2023-10-18T21:28:27.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.876 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Vw+HWikspvpRCIeDc5uBHh4Ju1s= 2023-10-18T21:28:27.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Vw+HWikspvpRCIeDc5uBHh4Ju1s='} 2023-10-18T21:28:27.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e64a1c647e0be4c-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.880 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.880 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.881 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.881 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.881 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.882 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001101a6b242c7da31-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj111 2023-10-18T21:28:27.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj111 2023-10-18T21:28:27.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj111 2023-10-18T21:28:27.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.886 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.886 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.887 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.887 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.887 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj111 2023-10-18T21:28:27.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.888 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xfose7/t8dcmGGPtR3RSbC+8dyY= 2023-10-18T21:28:27.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xfose7/t8dcmGGPtR3RSbC+8dyY='} 2023-10-18T21:28:27.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2b21b626b826885-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.892 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.892 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.893 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.893 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.893 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.894 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk= 2023-10-18T21:28:27.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ofk1eFdaNRAeU+G6UXJu6NIKFrk=', 'Content-Length': '0'} 2023-10-18T21:28:27.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8cb5f567c959d3e-00653046f7-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj148 2023-10-18T21:28:27.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj148 2023-10-18T21:28:27.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj148 2023-10-18T21:28:27.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.898 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.898 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.899 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.899 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:31 GMT 2023-10-18T21:28:27.899 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj148 2023-10-18T21:28:27.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.900 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:b1SvQ0yI8nVfjmRe82+qYX7eqzQ= 2023-10-18T21:28:27.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:31 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:b1SvQ0yI8nVfjmRe82+qYX7eqzQ='} 2023-10-18T21:28:27.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f410356dd80a8847-00653046f7-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/ 2023-10-18T21:28:27.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/ 2023-10-18T21:28:27.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:27.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/ 2023-10-18T21:28:27.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.904 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:27.905 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.905 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.905 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:32 GMT 2023-10-18T21:28:27.906 INFO:tasks.rgw_multisite_tests:/bofrnf-63/ 2023-10-18T21:28:27.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.906 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:aBBUFVdTZH43i7We+FzM8Ltyi9k= 2023-10-18T21:28:27.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 20:58:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:aBBUFVdTZH43i7We+FzM8Ltyi9k=', 'Content-Length': '0'} 2023-10-18T21:28:27.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041144c3132dc6283-00653046f8-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 20:58:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-63/obj185 2023-10-18T21:28:27.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-63/obj185 2023-10-18T21:28:27.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-63/obj185 2023-10-18T21:28:27.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:27.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.910 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.911 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:27.911 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:27.911 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 20:58:32 GMT 2023-10-18T21:28:27.911 INFO:tasks.rgw_multisite_tests:/bofrnf-63/obj185 2023-10-18T21:28:27.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.912 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ph+pyzTtiVbh2DwlaBIhbK2JFGI= 2023-10-18T21:28:27.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 20:58:32 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ph+pyzTtiVbh2DwlaBIhbK2JFGI='} 2023-10-18T21:28:27.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a2f44a441cb6827-00653046f8-1664-a1'), ('Date', 'Wed, 18 Oct 2023 20:58:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=bofrnf-63 2023-10-18T21:28:27.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:27.914 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:32.121+0000 7f1d7a60d780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:37.122+0000 7f1d7a60d780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:42.123+0000 7f1d7a60d780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:47.122+0000 7f1d7a60d780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:52.126+0000 7f1d7a60d780 1 bucket sync caught up with source: 2023-10-18T21:28:27.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, , , , , , 00000000002.116.6, , , , , , 00000000002.115.6, , , , , 00000000002.129.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.115.6] 2023-10-18T21:28:27.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, , , , , , 00000000002.116.6, , , , , , 00000000002.115.6, , , , , 00000000002.129.6, , , , , , , , , , , , , , , 00000000001.116.6, , , , 00000000001.115.6] 2023-10-18T21:28:27.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:52.126+0000 7f1d7a60d780 0 bucket checkpoint complete 2023-10-18T21:28:27.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=bofrnf-63 2023-10-18T21:28:27.918 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:27.918 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:27.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:27.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:52.244+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:58:57.245+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:02.247+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:07.247+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:12.247+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:17.248+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:22.249+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:27.250+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:32.250+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:37.251+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:42.251+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:47.251+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:52.252+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T20:59:57.254+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:02.254+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:07.255+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:12.255+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:17.256+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:22.256+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:27.257+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:32.257+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:37.258+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:42.259+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:47.260+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:52.260+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:00:57.261+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:02.262+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:07.262+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:12.263+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:17.264+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:22.263+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:27.264+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:32.265+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:37.266+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:42.266+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:47.267+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:52.268+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:01:57.269+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:02.269+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:07.270+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:12.271+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:17.271+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:22.271+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:27.272+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:32.273+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:37.274+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:42.274+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:47.275+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:52.275+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:02:57.276+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:02.277+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:07.277+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:12.278+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:17.279+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:22.280+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:27.281+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:32.281+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:37.282+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:42.282+0000 7fc6c5775780 1 waiting to reach incremental sync.. 2023-10-18T21:28:27.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:47.283+0000 7fc6c5775780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-10-18T21:28:27.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:47.283+0000 7fc6c5775780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:28:27.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:47.283+0000 7fc6c5775780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:28:27.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-18T21:28:27.938 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-18T21:28:27.939 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.939 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-18T21:28:27.939 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-10-18T21:28:27.940 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-18T21:28:27.940 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-18T21:28:27.941 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-18T21:28:27.941 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-18T21:28:27.941 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 1904, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-10-18T21:28:27.942 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-18T21:28:27.942 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-18T21:28:27.942 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-18T21:28:27.943 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-18T21:28:27.943 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-18T21:28:27.943 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-18T21:28:27.944 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-18T21:28:27.944 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-18T21:28:27.944 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-18T21:28:27.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:27.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:27.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:27.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:27.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:27.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:27.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:27.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:27.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:27.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-18T21:28:27.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:27.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:27.948 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:27.948 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.949 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.949 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:27.949 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:27.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:27.950 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:gsRi8Y+QHfxrf7UcdER9jNlSw3I= 2023-10-18T21:28:27.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gsRi8Y+QHfxrf7UcdER9jNlSw3I=', 'Content-Length': '0'} 2023-10-18T21:28:27.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000037e0ae8e2045dba8-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:27.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=bofrnf-64 2023-10-18T21:28:27.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:27.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:27.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:28:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [ 2023-10-18T21:28:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697663027.328641_1439.1", 2023-10-18T21:28:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T21:03:47.328641Z" 2023-10-18T21:28:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:28:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "0.000000" 2023-10-18T21:28:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:27.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:28:27.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:27.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697663027.328641_1439.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:27.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-18T21:28:27.959 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:27.959 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:27.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:27.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:27.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:27.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:47.519+0000 7fc0aed04780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:27.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:47.520+0000 7fc0aed04780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:27.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:27.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697663027.328641_1439.1", 2023-10-18T21:28:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:03:47.328641Z", 2023-10-18T21:28:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:27.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:27.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:27.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:27.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:27.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:27.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:27.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:27.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:27.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:03:47Z" 2023-10-18T21:28:27.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:27.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:27.980 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:27.980 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:27.980 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:27.981 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:27.981 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:27.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.981 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.982 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:27.982 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:27.982 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:27.983 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:27.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:27.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1697663027.328641_1439.1", 2023-10-18T21:28:27.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:27.984 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:27.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:27.984 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:03:47.328641Z", 2023-10-18T21:28:27.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.985 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.985 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.985 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:27.986 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:27.986 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:27.986 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:27.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:27.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:27.987 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:27.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:27.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:27.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.989 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.989 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.989 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:27.989 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:27.990 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:27.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:27.990 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:27.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:27.991 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:27.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:27.992 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:27.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.992 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.992 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.993 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:27.993 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:27.993 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:27.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:27.994 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:27.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:27.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:27.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:27.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:27.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:27.996 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.996 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:27.996 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:27.997 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.997 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:27.997 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:27.998 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:27.998 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:27.998 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:03:47Z" 2023-10-18T21:28:27.999 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:27.999 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:27.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697663027.328641_1439.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:03:47.328641Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:28.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697663027.328641_1439.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:28.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697663027.328641_1439.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:28.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-18T21:28:28.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.004 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.004 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.004 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.005 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.005 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.006 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:28:28.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:28:28.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052425925bacd91f5-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11 2023-10-18T21:28:28.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.010 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.010 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.011 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.011 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.011 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11 2023-10-18T21:28:28.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.012 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:kJMMs1aTQBEnQLDuENU2svQJT5o= 2023-10-18T21:28:28.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kJMMs1aTQBEnQLDuENU2svQJT5o='} 2023-10-18T21:28:28.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d5ea8c2adba5a35-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.016 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.016 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.017 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.017 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.017 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.018 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:28:28.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:28:28.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000918fd72caeb25936-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22 2023-10-18T21:28:28.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.025 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.025 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.026 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.026 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.026 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22 2023-10-18T21:28:28.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.027 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:MzfOEmfuUqZZzd3GSY/GhHAumQk= 2023-10-18T21:28:28.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MzfOEmfuUqZZzd3GSY/GhHAumQk='} 2023-10-18T21:28:28.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1d4c187c68c5381-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.031 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.031 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.032 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.032 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.032 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.033 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:28:28.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:28:28.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000005b29a1c0a7ef32-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33 2023-10-18T21:28:28.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.037 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.037 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.038 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.038 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.038 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33 2023-10-18T21:28:28.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.039 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ojAOBVnkTxoLgi0d+Oel1PODrR4= 2023-10-18T21:28:28.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ojAOBVnkTxoLgi0d+Oel1PODrR4='} 2023-10-18T21:28:28.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcf6867ec619b352-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.043 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.043 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.044 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.044 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.044 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.045 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:28:28.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:28:28.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060bcabd7027a5c9e-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44 2023-10-18T21:28:28.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.049 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.049 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.050 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.050 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.050 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44 2023-10-18T21:28:28.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.051 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:7tuPfoz37Jich9jiSdpIQS/f4/M= 2023-10-18T21:28:28.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7tuPfoz37Jich9jiSdpIQS/f4/M='} 2023-10-18T21:28:28.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0005c618f635a67-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.055 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.055 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.056 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.056 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.056 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.057 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo= 2023-10-18T21:28:28.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:R3CrxksFop9eVtuqa+rBoXMjXWo=', 'Content-Length': '0'} 2023-10-18T21:28:28.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7f0f333f81cb2e5-0065304833-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55 2023-10-18T21:28:28.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.061 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.061 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.061 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.062 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:03:47 GMT 2023-10-18T21:28:28.062 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55 2023-10-18T21:28:28.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.063 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:mVWsxe1jfrRVn25Qb2iO8pJtgek= 2023-10-18T21:28:28.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:03:47 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:mVWsxe1jfrRVn25Qb2iO8pJtgek='} 2023-10-18T21:28:28.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091f23c3124d4da68-0065304833-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=bofrnf-64 2023-10-18T21:28:28.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:28.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:28.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:28.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:47.778+0000 7ff04123e780 1 waiting to reach incremental sync.. 2023-10-18T21:28:28.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:52.779+0000 7ff04123e780 1 waiting to reach incremental sync.. 2023-10-18T21:28:28.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:03:57.779+0000 7ff04123e780 1 waiting to reach incremental sync.. 2023-10-18T21:28:28.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:02.781+0000 7ff04123e780 1 bucket sync caught up with source: 2023-10-18T21:28:28.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, , , , , , , , 00000000001.119.6, 00000000002.97.6, 00000000002.121.6] 2023-10-18T21:28:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, , , , , , , , 00000000001.119.6, 00000000002.97.6, 00000000002.121.6] 2023-10-18T21:28:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:02.781+0000 7ff04123e780 0 bucket checkpoint complete 2023-10-18T21:28:28.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=bofrnf-64 2023-10-18T21:28:28.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:28.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:02.897+0000 7f466f907780 1 waiting to reach incremental sync.. 2023-10-18T21:28:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:07.899+0000 7f466f907780 1 bucket sync caught up with source: 2023-10-18T21:28:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: local status: [, , , , , , , , 00000000001.332.6, 00000000002.855.6, 00000000002.725.6] 2023-10-18T21:28:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: remote markers: [, , , , , , , , 00000000001.332.6, 00000000002.855.6, 00000000002.725.6] 2023-10-18T21:28:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:07.899+0000 7f466f907780 0 bucket checkpoint complete 2023-10-18T21:28:28.070 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=bofrnf-64 zones={a2, a1} 2023-10-18T21:28:28.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:28:28.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.074 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.075 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.075 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.075 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:07 GMT 2023-10-18T21:28:28.076 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.076 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:hG7lL7f4EFehigKdiQG8WSihlzE= 2023-10-18T21:28:28.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hG7lL7f4EFehigKdiQG8WSihlzE=', 'Content-Length': '0'} 2023-10-18T21:28:28.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007112ade23e62e733-0065304847-1f24-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.081 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.081 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.081 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.081 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:07 GMT 2023-10-18T21:28:28.082 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.082 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:hG7lL7f4EFehigKdiQG8WSihlzE= 2023-10-18T21:28:28.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hG7lL7f4EFehigKdiQG8WSihlzE=', 'Content-Length': '0'} 2023-10-18T21:28:28.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007303c5a32d3b4855-0065304847-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.083 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-18T21:28:28.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.087 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.087 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.088 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.088 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:07 GMT 2023-10-18T21:28:28.088 INFO:tasks.rgw_multisite_tests:/bofrnf-64/?versions 2023-10-18T21:28:28.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.089 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY= 2023-10-18T21:28:28.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY=', 'Content-Length': '0'} 2023-10-18T21:28:28.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e1d385468474f87-0065304847-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64999falseobj11nulltrue2023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.090 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-18T21:28:28.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-18T21:28:28.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-18T21:28:28.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-18T21:28:28.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-18T21:28:28.092 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-18T21:28:28.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.096 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.096 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.097 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.097 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:07 GMT 2023-10-18T21:28:28.097 INFO:tasks.rgw_multisite_tests:/bofrnf-64/?versions 2023-10-18T21:28:28.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.098 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY= 2023-10-18T21:28:28.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY=', 'Content-Length': '0'} 2023-10-18T21:28:28.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002424d6c1cc8bc0cd-0065304847-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64999falseobj11nulltrue2023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.099 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-18T21:28:28.099 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-18T21:28:28.100 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-18T21:28:28.100 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-18T21:28:28.100 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-18T21:28:28.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.104 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.105 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.105 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.105 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:07 GMT 2023-10-18T21:28:28.106 INFO:tasks.rgw_multisite_tests:/bofrnf-64/?versions 2023-10-18T21:28:28.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.106 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY= 2023-10-18T21:28:28.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY=', 'Content-Length': '0'} 2023-10-18T21:28:28.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000201f6f2f5efbd4d4-0065304847-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64999falseobj11nulltrue2023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?versions&max-keys=999 2023-10-18T21:28:28.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.111 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.111 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.111 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.112 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:07 GMT 2023-10-18T21:28:28.112 INFO:tasks.rgw_multisite_tests:/bofrnf-64/?versions 2023-10-18T21:28:28.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.113 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY= 2023-10-18T21:28:28.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:bxvOKNEZTLovBBNG5JWyWJawbsY=', 'Content-Length': '0'} 2023-10-18T21:28:28.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c168948e6c42f1eb-0065304847-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64999falseobj11nulltrue2023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.114 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-18T21:28:28.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.118 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.118 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.118 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.119 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:07 GMT 2023-10-18T21:28:28.119 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.119 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:/AuSZUhqAF7a4UhpftlnXM/ocvU= 2023-10-18T21:28:28.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/AuSZUhqAF7a4UhpftlnXM/ocvU=', 'Content-Length': '0'} 2023-10-18T21:28:28.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9cd14cf86f0edd7-0065304847-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.124 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.125 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.125 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.125 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:07 GMT 2023-10-18T21:28:28.125 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.126 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:/AuSZUhqAF7a4UhpftlnXM/ocvU= 2023-10-18T21:28:28.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:07 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:/AuSZUhqAF7a4UhpftlnXM/ocvU=', 'Content-Length': '0'} 2023-10-18T21:28:28.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f2c4f944f89b8b4-0065304847-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.131 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.131 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.131 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.131 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.132 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.132 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i7/YOoHOM7OQ1ZaF3Rc7NukMy2I= 2023-10-18T21:28:28.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i7/YOoHOM7OQ1ZaF3Rc7NukMy2I=', 'Content-Length': '0'} 2023-10-18T21:28:28.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b49759cda6b6e454-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.137 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.137 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.137 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.138 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.138 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.138 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:i7/YOoHOM7OQ1ZaF3Rc7NukMy2I= 2023-10-18T21:28:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:i7/YOoHOM7OQ1ZaF3Rc7NukMy2I=', 'Content-Length': '0'} 2023-10-18T21:28:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004299d20e76f56e9a-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.139 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-18T21:28:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.143 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.144 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.144 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.144 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.145 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.145 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:cJV3yEnOIq45DVMCWS2ZSOpScz0= 2023-10-18T21:28:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cJV3yEnOIq45DVMCWS2ZSOpScz0=', 'Content-Length': '0'} 2023-10-18T21:28:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f5e99f9fa24d8af-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.150 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.150 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.150 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.151 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.151 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11?versionId=null 2023-10-18T21:28:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.151 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:cJV3yEnOIq45DVMCWS2ZSOpScz0= 2023-10-18T21:28:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cJV3yEnOIq45DVMCWS2ZSOpScz0=', 'Content-Length': '0'} 2023-10-18T21:28:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6ac73bb0d15f436-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.156 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.156 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.157 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.157 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.157 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.158 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fbafe1d1c1751b2-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj111000falseobj112023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.163 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.163 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.163 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.164 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.164 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.165 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b1e277ac4edc535-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj111000falseobj112023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11 2023-10-18T21:28:28.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.169 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.170 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.170 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.170 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.171 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11 2023-10-18T21:28:28.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.171 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:QQiSYTTjqoxdtHrmRyjvdyqn7II= 2023-10-18T21:28:28.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QQiSYTTjqoxdtHrmRyjvdyqn7II=', 'Content-Length': '0'} 2023-10-18T21:28:28.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1d59c1256d68b4-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11 2023-10-18T21:28:28.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.176 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.176 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.176 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.177 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.177 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11 2023-10-18T21:28:28.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.177 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:QQiSYTTjqoxdtHrmRyjvdyqn7II= 2023-10-18T21:28:28.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QQiSYTTjqoxdtHrmRyjvdyqn7II=', 'Content-Length': '0'} 2023-10-18T21:28:28.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd5b0d8c54439f88-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.178 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-18T21:28:28.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11 2023-10-18T21:28:28.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.182 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.182 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.183 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.183 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.183 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11 2023-10-18T21:28:28.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.184 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:HL2xnvxnwazUweeaApxAhDfHl6I= 2023-10-18T21:28:28.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HL2xnvxnwazUweeaApxAhDfHl6I=', 'Content-Length': '0'} 2023-10-18T21:28:28.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017b7e6dcf64c2cdc-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj11 2023-10-18T21:28:28.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj11 2023-10-18T21:28:28.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj11 2023-10-18T21:28:28.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.188 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.188 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.189 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.189 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.189 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj11 2023-10-18T21:28:28.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.190 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:HL2xnvxnwazUweeaApxAhDfHl6I= 2023-10-18T21:28:28.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:HL2xnvxnwazUweeaApxAhDfHl6I=', 'Content-Length': '0'} 2023-10-18T21:28:28.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025208d87e3d5e90b-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.195 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.195 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.196 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.196 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.196 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.197 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3d91fb7f81aeee-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj111000falseobj112023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj11 2023-10-18T21:28:28.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.201 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.202 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.202 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.202 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.203 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.203 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b95fa17385459138-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj111000falseobj112023-10-18T21:03:47.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.204 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-18T21:28:28.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.208 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.209 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.209 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.209 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.210 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.210 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U= 2023-10-18T21:28:28.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U=', 'Content-Length': '0'} 2023-10-18T21:28:28.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007637789e8ec3bad1-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.215 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.215 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.216 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.216 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.216 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.217 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U= 2023-10-18T21:28:28.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U=', 'Content-Length': '0'} 2023-10-18T21:28:28.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003323f3592deb9a7a-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.221 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.221 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.222 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.222 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.222 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.223 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:EDmF9U34rcI0fuRJJYp1YqI7Wuw= 2023-10-18T21:28:28.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EDmF9U34rcI0fuRJJYp1YqI7Wuw=', 'Content-Length': '0'} 2023-10-18T21:28:28.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000600b5cfd13e002e4-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.229 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.229 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.230 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.230 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.230 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.231 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:EDmF9U34rcI0fuRJJYp1YqI7Wuw= 2023-10-18T21:28:28.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EDmF9U34rcI0fuRJJYp1YqI7Wuw=', 'Content-Length': '0'} 2023-10-18T21:28:28.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f0746f4f16bac5c-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.232 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-18T21:28:28.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.236 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.236 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.236 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.237 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.237 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.238 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U= 2023-10-18T21:28:28.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U=', 'Content-Length': '0'} 2023-10-18T21:28:28.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd1ee416b9a926f3-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.242 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.242 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.243 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.243 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.243 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22?versionId=null 2023-10-18T21:28:28.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.244 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U= 2023-10-18T21:28:28.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Yc1zZdSEXAuqrdY5QVyIEb2zt1U=', 'Content-Length': '0'} 2023-10-18T21:28:28.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0e4ea0cc5f31165-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.248 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.249 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.249 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.249 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.250 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.250 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cc0326b295bf4b7-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj221000falseobj222023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.255 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.255 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.256 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.256 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.256 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.257 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009366328567674dd3-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj221000falseobj222023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22 2023-10-18T21:28:28.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.261 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.262 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.262 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.262 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.263 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22 2023-10-18T21:28:28.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.263 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ytP5Bp79TzjZ5xFFqvp8rIRqe3A= 2023-10-18T21:28:28.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ytP5Bp79TzjZ5xFFqvp8rIRqe3A=', 'Content-Length': '0'} 2023-10-18T21:28:28.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7eda0c0a3f6b92d-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22 2023-10-18T21:28:28.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.267 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.268 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.268 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.268 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.269 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22 2023-10-18T21:28:28.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.269 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ytP5Bp79TzjZ5xFFqvp8rIRqe3A= 2023-10-18T21:28:28.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ytP5Bp79TzjZ5xFFqvp8rIRqe3A=', 'Content-Length': '0'} 2023-10-18T21:28:28.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a52a2e3c57ca05-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.270 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-18T21:28:28.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22 2023-10-18T21:28:28.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.274 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.274 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.274 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.275 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.275 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22 2023-10-18T21:28:28.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.276 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:SbmjAegI+JRqs8QjN8BqODkQuJc= 2023-10-18T21:28:28.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SbmjAegI+JRqs8QjN8BqODkQuJc=', 'Content-Length': '0'} 2023-10-18T21:28:28.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ca79fcfcf51ac7-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj22 2023-10-18T21:28:28.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj22 2023-10-18T21:28:28.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj22 2023-10-18T21:28:28.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.280 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.280 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.280 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.280 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.281 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj22 2023-10-18T21:28:28.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.281 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:SbmjAegI+JRqs8QjN8BqODkQuJc= 2023-10-18T21:28:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:SbmjAegI+JRqs8QjN8BqODkQuJc=', 'Content-Length': '0'} 2023-10-18T21:28:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3de6eabf0f15b1e-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.286 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.286 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.286 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.287 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.287 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.288 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd65c08550f0c68d-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj221000falseobj222023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj22 2023-10-18T21:28:28.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.292 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.293 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.293 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.293 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.293 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.294 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094932f06439e4f52-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj221000falseobj222023-10-18T21:03:47.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.295 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-18T21:28:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.299 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.299 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.300 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.300 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.300 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.301 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM= 2023-10-18T21:28:28.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM=', 'Content-Length': '0'} 2023-10-18T21:28:28.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a385edb1cd5b3eba-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.306 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.306 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.306 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.307 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.307 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.307 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM= 2023-10-18T21:28:28.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM=', 'Content-Length': '0'} 2023-10-18T21:28:28.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d67d46279b1fea6a-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.312 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.312 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.313 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.313 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.313 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.314 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oBSj3j9hej3U7r8Ycbn+5uyUCAA= 2023-10-18T21:28:28.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oBSj3j9hej3U7r8Ycbn+5uyUCAA=', 'Content-Length': '0'} 2023-10-18T21:28:28.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4f4fa7e0b0198d0-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.319 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.319 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.319 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.320 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.320 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.321 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oBSj3j9hej3U7r8Ycbn+5uyUCAA= 2023-10-18T21:28:28.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oBSj3j9hej3U7r8Ycbn+5uyUCAA=', 'Content-Length': '0'} 2023-10-18T21:28:28.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d7fd2e64ff5476-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.321 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-18T21:28:28.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.325 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.326 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.326 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.326 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.327 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.327 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM= 2023-10-18T21:28:28.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM=', 'Content-Length': '0'} 2023-10-18T21:28:28.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006757be7645d5225a-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.332 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.332 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.332 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.333 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.333 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33?versionId=null 2023-10-18T21:28:28.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.333 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM= 2023-10-18T21:28:28.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ABbeapoWuqFPgfHRBRCTKlFTUZM=', 'Content-Length': '0'} 2023-10-18T21:28:28.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d6876897510989-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.338 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.338 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.339 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.339 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.339 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.340 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14b0a78060f2152-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj331000falseobj332023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.345 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.345 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.345 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.346 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.346 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.347 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e507b31224dfb643-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj331000falseobj332023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33 2023-10-18T21:28:28.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.351 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.351 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.351 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.352 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.352 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33 2023-10-18T21:28:28.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.353 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:X0idiJytnT3d0liI/gZqF44JGYM= 2023-10-18T21:28:28.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X0idiJytnT3d0liI/gZqF44JGYM=', 'Content-Length': '0'} 2023-10-18T21:28:28.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a290aa9b05386a1-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33 2023-10-18T21:28:28.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.357 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.357 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.357 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.358 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.358 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33 2023-10-18T21:28:28.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.359 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:X0idiJytnT3d0liI/gZqF44JGYM= 2023-10-18T21:28:28.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X0idiJytnT3d0liI/gZqF44JGYM=', 'Content-Length': '0'} 2023-10-18T21:28:28.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04829eb2252c50f-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.360 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-18T21:28:28.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33 2023-10-18T21:28:28.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.363 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.363 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.364 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.364 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.364 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33 2023-10-18T21:28:28.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.365 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:n4ne0CYB9cwBB0Ihg+5MjECrtyI= 2023-10-18T21:28:28.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n4ne0CYB9cwBB0Ihg+5MjECrtyI=', 'Content-Length': '0'} 2023-10-18T21:28:28.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e896ff27003a76fc-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj33 2023-10-18T21:28:28.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj33 2023-10-18T21:28:28.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj33 2023-10-18T21:28:28.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.369 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.369 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.370 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.370 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.370 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj33 2023-10-18T21:28:28.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.371 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:n4ne0CYB9cwBB0Ihg+5MjECrtyI= 2023-10-18T21:28:28.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:n4ne0CYB9cwBB0Ihg+5MjECrtyI=', 'Content-Length': '0'} 2023-10-18T21:28:28.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af1ceae86ab746a-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.375 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.376 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.376 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.376 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.377 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.377 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c3cbc4bbde9b095-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj331000falseobj332023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj33 2023-10-18T21:28:28.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.382 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.383 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.383 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.383 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.384 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.384 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044724e334710008b-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj331000falseobj332023-10-18T21:03:47.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.385 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-18T21:28:28.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.389 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.389 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.390 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.390 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.390 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.391 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc= 2023-10-18T21:28:28.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc=', 'Content-Length': '0'} 2023-10-18T21:28:28.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd82f5facbbe61ba-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.396 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.396 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.396 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.397 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.397 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.397 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc= 2023-10-18T21:28:28.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc=', 'Content-Length': '0'} 2023-10-18T21:28:28.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b96e10ecedefc6-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.402 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.402 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.403 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.403 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.403 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.404 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:FVO9Q8v1tbeYdo86GBSXbwAQrF8= 2023-10-18T21:28:28.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVO9Q8v1tbeYdo86GBSXbwAQrF8=', 'Content-Length': '0'} 2023-10-18T21:28:28.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000223c444ebc60d24-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.409 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.409 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.409 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.410 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.410 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.411 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:FVO9Q8v1tbeYdo86GBSXbwAQrF8= 2023-10-18T21:28:28.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:FVO9Q8v1tbeYdo86GBSXbwAQrF8=', 'Content-Length': '0'} 2023-10-18T21:28:28.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cf572b0f8a626b4-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.412 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-18T21:28:28.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.416 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.416 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.416 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.417 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.417 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.417 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc= 2023-10-18T21:28:28.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc=', 'Content-Length': '0'} 2023-10-18T21:28:28.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2860e3ed40d7195-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.422 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.423 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.423 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.423 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.423 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44?versionId=null 2023-10-18T21:28:28.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.424 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc= 2023-10-18T21:28:28.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:kDat/mrmb+hGap12AUW5j+KDQSc=', 'Content-Length': '0'} 2023-10-18T21:28:28.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423bdf3bc88b443b-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.432 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.432 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.432 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.433 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.433 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.433 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e2d53ff4b16568c-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj441000falseobj442023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.439 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.439 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.439 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.440 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.440 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.440 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082e721551ce07362-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj441000falseobj442023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44 2023-10-18T21:28:28.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.445 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.445 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.446 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.446 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.446 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44 2023-10-18T21:28:28.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.447 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:hlRgC63GcHmaaGRJRKTVfn0gaK0= 2023-10-18T21:28:28.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hlRgC63GcHmaaGRJRKTVfn0gaK0=', 'Content-Length': '0'} 2023-10-18T21:28:28.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c173bb352443933d-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44 2023-10-18T21:28:28.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.451 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.451 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.452 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.452 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.452 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44 2023-10-18T21:28:28.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.453 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:hlRgC63GcHmaaGRJRKTVfn0gaK0= 2023-10-18T21:28:28.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:hlRgC63GcHmaaGRJRKTVfn0gaK0=', 'Content-Length': '0'} 2023-10-18T21:28:28.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d6d1c095708447-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.454 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-18T21:28:28.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44 2023-10-18T21:28:28.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.458 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.458 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.458 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.459 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.459 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44 2023-10-18T21:28:28.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.460 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xvDDh+1pBRgQdQGssUcVGO2WQwQ= 2023-10-18T21:28:28.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xvDDh+1pBRgQdQGssUcVGO2WQwQ=', 'Content-Length': '0'} 2023-10-18T21:28:28.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdcca31f34a3678b-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj44 2023-10-18T21:28:28.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj44 2023-10-18T21:28:28.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj44 2023-10-18T21:28:28.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.464 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.464 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.464 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.465 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.465 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj44 2023-10-18T21:28:28.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.465 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xvDDh+1pBRgQdQGssUcVGO2WQwQ= 2023-10-18T21:28:28.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xvDDh+1pBRgQdQGssUcVGO2WQwQ=', 'Content-Length': '0'} 2023-10-18T21:28:28.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d6a40e3ffec1fc9-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.470 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.471 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.471 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.471 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.472 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.472 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009afab75c98500265-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj441000falseobj442023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj44 2023-10-18T21:28:28.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.477 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.478 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.478 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.478 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.479 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.479 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026362bcec413ffae-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj441000falseobj442023-10-18T21:03:47.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.480 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-18T21:28:28.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.485 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.485 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.485 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.486 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.486 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.487 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I= 2023-10-18T21:28:28.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I=', 'Content-Length': '0'} 2023-10-18T21:28:28.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040838bdaf698856d-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.491 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.492 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.492 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.492 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.493 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.493 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I= 2023-10-18T21:28:28.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I=', 'Content-Length': '0'} 2023-10-18T21:28:28.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0a5ada9b7ed25e-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.498 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.498 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.499 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.499 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.499 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.500 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1E+jsSsKOzITrXWjvt/DjXRBMLA= 2023-10-18T21:28:28.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1E+jsSsKOzITrXWjvt/DjXRBMLA=', 'Content-Length': '0'} 2023-10-18T21:28:28.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b21cd94c3f919f6-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.505 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.505 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.505 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.506 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.506 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.507 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1E+jsSsKOzITrXWjvt/DjXRBMLA= 2023-10-18T21:28:28.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1E+jsSsKOzITrXWjvt/DjXRBMLA=', 'Content-Length': '0'} 2023-10-18T21:28:28.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000979812c52392e4a1-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.507 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-18T21:28:28.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.512 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.512 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.512 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.513 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.513 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.514 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I= 2023-10-18T21:28:28.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I=', 'Content-Length': '0'} 2023-10-18T21:28:28.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0ef29856799dd8-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.518 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.519 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.519 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.519 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.520 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55?versionId=null 2023-10-18T21:28:28.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.520 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I= 2023-10-18T21:28:28.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KOn2AiXRww1+UsgccGQVjSFCm/I=', 'Content-Length': '0'} 2023-10-18T21:28:28.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c617669dbc499db4-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.525 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.525 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.526 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.526 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.526 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.527 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8b745fa6dd862d-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj551000falseobj552023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.532 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.532 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.533 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.533 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.533 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.534 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7976a524d3060fa-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj551000falseobj552023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55 2023-10-18T21:28:28.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.538 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.539 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.539 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.539 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.539 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55 2023-10-18T21:28:28.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.540 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:BTV67fGnRvtsk/wfqM4PNjih7lw= 2023-10-18T21:28:28.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BTV67fGnRvtsk/wfqM4PNjih7lw=', 'Content-Length': '0'} 2023-10-18T21:28:28.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e83a48c8f5f85d-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55 2023-10-18T21:28:28.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.544 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.545 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.545 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.545 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.545 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55 2023-10-18T21:28:28.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.546 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:BTV67fGnRvtsk/wfqM4PNjih7lw= 2023-10-18T21:28:28.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:BTV67fGnRvtsk/wfqM4PNjih7lw=', 'Content-Length': '0'} 2023-10-18T21:28:28.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c84247aacdd854af-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.547 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-18T21:28:28.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55 2023-10-18T21:28:28.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.550 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.551 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.551 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.551 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.552 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55 2023-10-18T21:28:28.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.552 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:GbHY1U4oQD8UHBm9ViYtJlGzeyg= 2023-10-18T21:28:28.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GbHY1U4oQD8UHBm9ViYtJlGzeyg=', 'Content-Length': '0'} 2023-10-18T21:28:28.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-64:403a921b-fcfc-4ab2-b434-bf29d35a51b6.5734.33'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca128905657ccbda-0065304848-1f24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj55 2023-10-18T21:28:28.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj55 2023-10-18T21:28:28.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj55 2023-10-18T21:28:28.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.556 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.557 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.557 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.557 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.558 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj55 2023-10-18T21:28:28.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.558 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:GbHY1U4oQD8UHBm9ViYtJlGzeyg= 2023-10-18T21:28:28.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:GbHY1U4oQD8UHBm9ViYtJlGzeyg=', 'Content-Length': '0'} 2023-10-18T21:28:28.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:03:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c11dc2278cf7ac7e-0065304848-1664-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:28.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.563 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.563 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.564 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.564 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.564 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.565 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ebfd5bdcdd4c695-0065304848-1f24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj551000falseobj552023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/?prefix=obj55 2023-10-18T21:28:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.570 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:28.570 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.570 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.571 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:08 GMT 2023-10-18T21:28:28.571 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.572 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o= 2023-10-18T21:28:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:08 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6gdK7K+B+fXtz0bg/q5wuA4tO4o=', 'Content-Length': '0'} 2023-10-18T21:28:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c218ae6ebc70c56-0065304848-1664-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:04:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'bofrnf-64obj551000falseobj552023-10-18T21:03:47.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-18T21:28:28.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=bofrnf-64 zones={a2, a1} 2023-10-18T21:28:28.573 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-10-18T21:28:28.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-10-18T21:28:28.574 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T21:04:08.922+0000 7ff122b99700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 176793) UID: 0 2023-10-18T21:28:28.574 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi082.stdout: INFO: 2023-10-18T21:04:08.922+0000 7ff126e027c0 -1 shutting down 2023-10-18T21:28:28.574 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T21:28:28.575 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-10-18T21:28:28.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:28.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:28.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-64 2023-10-18T21:28:28.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 5 2023-10-18T21:28:28.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.581 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.581 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.581 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.581 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.582 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.582 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e34ebf078087c9f6-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj17 2023-10-18T21:28:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj17 2023-10-18T21:28:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj17 2023-10-18T21:28:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.587 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.587 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.587 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.588 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.588 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj17 2023-10-18T21:28:28.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.588 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:NrntbniOlHTRPoDcOJ1lmck1jcY= 2023-10-18T21:28:28.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:NrntbniOlHTRPoDcOJ1lmck1jcY='} 2023-10-18T21:28:28.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000878e82c71c1ff91c-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.593 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.593 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.593 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.594 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.594 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.594 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a463868ff2a34b6f-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj34 2023-10-18T21:28:28.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj34 2023-10-18T21:28:28.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj34 2023-10-18T21:28:28.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.599 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.599 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.599 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.600 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.600 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj34 2023-10-18T21:28:28.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.601 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:vcgPA7yxDmJfdTcr4IXL5mNUFH4= 2023-10-18T21:28:28.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vcgPA7yxDmJfdTcr4IXL5mNUFH4='} 2023-10-18T21:28:28.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003900c8c3e0f89c5-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.605 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.605 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.605 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.605 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.606 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.606 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046a27bab10418f7a-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj51 2023-10-18T21:28:28.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj51 2023-10-18T21:28:28.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj51 2023-10-18T21:28:28.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.611 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.611 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.611 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.612 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.612 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj51 2023-10-18T21:28:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.612 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3qVJiRpwwE/va4AFiu3lpipEVo8= 2023-10-18T21:28:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3qVJiRpwwE/va4AFiu3lpipEVo8='} 2023-10-18T21:28:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcd9347c3ca7b063-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.617 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.617 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.617 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.618 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.618 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.618 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000616c3a480b1d5cfc-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj68 2023-10-18T21:28:28.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj68 2023-10-18T21:28:28.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj68 2023-10-18T21:28:28.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.623 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.623 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.623 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.624 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.624 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj68 2023-10-18T21:28:28.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.625 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:7UcniNPxTCAT+Ah9Alow1WuuqJs= 2023-10-18T21:28:28.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7UcniNPxTCAT+Ah9Alow1WuuqJs='} 2023-10-18T21:28:28.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b99573fd1f910b54-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.629 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.629 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.629 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.630 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.630 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.631 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b31327c990e8619e-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj85 2023-10-18T21:28:28.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj85 2023-10-18T21:28:28.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj85 2023-10-18T21:28:28.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.635 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.635 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.635 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.636 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.636 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj85 2023-10-18T21:28:28.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.637 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:QvXzdjkfA9yew1UfL71X/+uUR3Q= 2023-10-18T21:28:28.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:QvXzdjkfA9yew1UfL71X/+uUR3Q='} 2023-10-18T21:28:28.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d1f47a3acb569fe-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-64 2023-10-18T21:28:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 10 2023-10-18T21:28:28.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.643 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.643 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.644 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.644 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.644 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.645 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5f5f0cc5352b663-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj19 2023-10-18T21:28:28.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj19 2023-10-18T21:28:28.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj19 2023-10-18T21:28:28.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.649 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.649 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.650 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.650 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.650 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj19 2023-10-18T21:28:28.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.651 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:qfAaWmCKK3Xtp3R9UBXL12y+Phk= 2023-10-18T21:28:28.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:qfAaWmCKK3Xtp3R9UBXL12y+Phk='} 2023-10-18T21:28:28.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac5f5a801ee266ff-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.655 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.655 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.655 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.656 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.656 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.657 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7af20a156552ed9-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj38 2023-10-18T21:28:28.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj38 2023-10-18T21:28:28.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj38 2023-10-18T21:28:28.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.661 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.661 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.661 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.661 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.662 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj38 2023-10-18T21:28:28.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.662 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:0I6V8oTnRABjnoipRUsNxe/NiOY= 2023-10-18T21:28:28.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0I6V8oTnRABjnoipRUsNxe/NiOY='} 2023-10-18T21:28:28.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053de607309a416dd-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.666 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.667 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.667 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.667 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.668 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.668 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3d5884aa07b40de-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj57 2023-10-18T21:28:28.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj57 2023-10-18T21:28:28.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj57 2023-10-18T21:28:28.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.672 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.673 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.673 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.673 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.674 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj57 2023-10-18T21:28:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.674 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:y2kC1j3Pg8elMPGROtZvUx2LfHk= 2023-10-18T21:28:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:y2kC1j3Pg8elMPGROtZvUx2LfHk='} 2023-10-18T21:28:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f254fe238ab7793-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.678 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.679 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.679 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.679 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.680 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.680 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002df6f52828d70801-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj76 2023-10-18T21:28:28.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj76 2023-10-18T21:28:28.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj76 2023-10-18T21:28:28.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.684 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.685 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.685 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.685 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.686 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj76 2023-10-18T21:28:28.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.686 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:w+N2Y4sDo99mqeEz0mQzGZTkp9U= 2023-10-18T21:28:28.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:w+N2Y4sDo99mqeEz0mQzGZTkp9U='} 2023-10-18T21:28:28.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d050df8199c34619-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.691 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.691 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.691 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.691 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.692 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.692 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7012c5660098b59-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj95 2023-10-18T21:28:28.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj95 2023-10-18T21:28:28.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj95 2023-10-18T21:28:28.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.697 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.697 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.697 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.697 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.698 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj95 2023-10-18T21:28:28.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.698 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:2x8q8PGQT2A9vebMvcoWdgr+kqs= 2023-10-18T21:28:28.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:2x8q8PGQT2A9vebMvcoWdgr+kqs='} 2023-10-18T21:28:28.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006532da93468e1789-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:28.703 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-10-18T21:28:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-64 2023-10-18T21:28:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 15 2023-10-18T21:28:28.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.708 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.709 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.709 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.709 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.710 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.710 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cf4183dc6f92875-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj23 2023-10-18T21:28:28.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj23 2023-10-18T21:28:28.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj23 2023-10-18T21:28:28.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.714 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.715 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.715 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.715 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.716 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj23 2023-10-18T21:28:28.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.716 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:miTEU3Qs0a6J0Fvnu2YZXWwvS3U= 2023-10-18T21:28:28.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:miTEU3Qs0a6J0Fvnu2YZXWwvS3U='} 2023-10-18T21:28:28.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072fde9172bade3d7-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.721 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.721 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.721 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.721 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.722 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.722 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ae40b2f57a4d70d-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj46 2023-10-18T21:28:28.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj46 2023-10-18T21:28:28.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj46 2023-10-18T21:28:28.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.727 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.727 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.727 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.727 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.728 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj46 2023-10-18T21:28:28.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.728 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:EH6nySVuYDwb1kFRQH8KYyzmWAc= 2023-10-18T21:28:28.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:EH6nySVuYDwb1kFRQH8KYyzmWAc='} 2023-10-18T21:28:28.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a51e0a2f99e7f6c9-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.733 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.733 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.733 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.734 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.734 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.735 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb135227f6a304a4-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj69 2023-10-18T21:28:28.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj69 2023-10-18T21:28:28.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj69 2023-10-18T21:28:28.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.739 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.739 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.739 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.740 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.740 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj69 2023-10-18T21:28:28.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.740 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:cTMrgEmuoMwr5UPeDW3yr+ejkrc= 2023-10-18T21:28:28.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cTMrgEmuoMwr5UPeDW3yr+ejkrc='} 2023-10-18T21:28:28.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cfaa12c00ed7046-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.744 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.745 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.745 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.745 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.745 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.746 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039d069f3dc773b21-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj92 2023-10-18T21:28:28.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj92 2023-10-18T21:28:28.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj92 2023-10-18T21:28:28.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.750 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.750 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.751 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.751 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.751 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj92 2023-10-18T21:28:28.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.752 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:7ZY+Op2OHhvH8eKLcNU0kDtn0ko= 2023-10-18T21:28:28.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:7ZY+Op2OHhvH8eKLcNU0kDtn0ko='} 2023-10-18T21:28:28.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7f7795450a219d1-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.756 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.756 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.756 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.757 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.757 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.758 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4= 2023-10-18T21:28:28.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:d+O8FaOi2md05ZoJjY8jpk6bjY4=', 'Content-Length': '0'} 2023-10-18T21:28:28.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ae88fc788cd65e3-006530484f-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj115 2023-10-18T21:28:28.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj115 2023-10-18T21:28:28.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj115 2023-10-18T21:28:28.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.762 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.762 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.763 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.763 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:15 GMT 2023-10-18T21:28:28.763 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj115 2023-10-18T21:28:28.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.764 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:sARDa4dDHpXEEhHXw23FZEaMcJk= 2023-10-18T21:28:28.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:15 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:sARDa4dDHpXEEhHXw23FZEaMcJk='} 2023-10-18T21:28:28.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbda3440f5d43242-006530484f-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-64 2023-10-18T21:28:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 20 2023-10-18T21:28:28.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.770 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.770 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.771 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.771 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.771 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.772 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a44dbf8bc780768b-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj29 2023-10-18T21:28:28.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj29 2023-10-18T21:28:28.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj29 2023-10-18T21:28:28.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.776 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.776 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.777 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.777 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.777 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj29 2023-10-18T21:28:28.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.778 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:0Us8prGuf5odkhB8y0x0V87Gn+8= 2023-10-18T21:28:28.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:0Us8prGuf5odkhB8y0x0V87Gn+8='} 2023-10-18T21:28:28.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ba1d1d74b284cc9-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.782 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.782 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.783 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.783 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.783 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.784 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000585a07531bc6f836-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj58 2023-10-18T21:28:28.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj58 2023-10-18T21:28:28.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj58 2023-10-18T21:28:28.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.788 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.788 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.788 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.789 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.789 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj58 2023-10-18T21:28:28.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.790 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:h97oeUJNH7pdFjzRTyBs65lBn9Q= 2023-10-18T21:28:28.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:h97oeUJNH7pdFjzRTyBs65lBn9Q='} 2023-10-18T21:28:28.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c6837b18aa15330-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.794 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.794 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.794 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.795 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.795 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.795 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2f6bd7dde4cb31f-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj87 2023-10-18T21:28:28.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj87 2023-10-18T21:28:28.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj87 2023-10-18T21:28:28.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.800 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.800 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.800 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.801 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.801 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj87 2023-10-18T21:28:28.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.801 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Ts+wGPw7TGvrHRKdnuK7JoQ/UVw= 2023-10-18T21:28:28.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Ts+wGPw7TGvrHRKdnuK7JoQ/UVw='} 2023-10-18T21:28:28.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4926ed1810bf30e-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.806 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.806 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.806 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.807 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.807 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.807 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae54454c83d4ed9e-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj116 2023-10-18T21:28:28.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj116 2023-10-18T21:28:28.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj116 2023-10-18T21:28:28.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.812 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.812 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.812 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.813 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.813 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj116 2023-10-18T21:28:28.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.813 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:X7YHmXXN7wIgkn2eib7KcZBb7E4= 2023-10-18T21:28:28.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:X7YHmXXN7wIgkn2eib7KcZBb7E4='} 2023-10-18T21:28:28.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052695e33bdab3229-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.818 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.818 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.818 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.819 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.819 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.820 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b14524279f99536-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj145 2023-10-18T21:28:28.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj145 2023-10-18T21:28:28.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj145 2023-10-18T21:28:28.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.824 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.824 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.824 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.825 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.825 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj145 2023-10-18T21:28:28.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.826 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:3z1nEnAqaBZ8CCCt6JNnGUZ43a0= 2023-10-18T21:28:28.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:3z1nEnAqaBZ8CCCt6JNnGUZ43a0='} 2023-10-18T21:28:28.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4edf970137237be-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:28.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:28.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-64 2023-10-18T21:28:28.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 25 2023-10-18T21:28:28.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.832 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.832 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.833 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.833 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.833 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.834 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ff3c1121842b9af-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj31 2023-10-18T21:28:28.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj31 2023-10-18T21:28:28.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj31 2023-10-18T21:28:28.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.838 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.839 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.839 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.839 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.840 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj31 2023-10-18T21:28:28.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.840 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KdTQMJjvA8AouOzapYe0slspVXE= 2023-10-18T21:28:28.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KdTQMJjvA8AouOzapYe0slspVXE='} 2023-10-18T21:28:28.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060b1f058816e6e82-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.844 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.844 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.845 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.845 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.845 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.846 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb72a33d78af0973-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj62 2023-10-18T21:28:28.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj62 2023-10-18T21:28:28.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj62 2023-10-18T21:28:28.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.850 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.850 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.851 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.851 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.851 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj62 2023-10-18T21:28:28.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.852 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:KvTTm5bzOtibUCIPg67x0XCCgOg= 2023-10-18T21:28:28.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:KvTTm5bzOtibUCIPg67x0XCCgOg='} 2023-10-18T21:28:28.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4d43e9e61690295-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.856 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.857 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.857 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.857 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.857 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.858 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050a8e00abb777bdd-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj93 2023-10-18T21:28:28.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj93 2023-10-18T21:28:28.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj93 2023-10-18T21:28:28.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.862 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.863 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.863 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.863 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.863 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj93 2023-10-18T21:28:28.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.864 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:MxyOkgIVaie8N43mgQTZoRtK2zo= 2023-10-18T21:28:28.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MxyOkgIVaie8N43mgQTZoRtK2zo='} 2023-10-18T21:28:28.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003c9dc32640781fc-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.868 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.868 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.869 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.869 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.869 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.870 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008418cd34e02f81b6-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj124 2023-10-18T21:28:28.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj124 2023-10-18T21:28:28.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj124 2023-10-18T21:28:28.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.874 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.874 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.874 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.875 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.875 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj124 2023-10-18T21:28:28.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.876 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:gxvNwd5khEDxaCdoE1jfJAlZGlk= 2023-10-18T21:28:28.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gxvNwd5khEDxaCdoE1jfJAlZGlk='} 2023-10-18T21:28:28.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b79e309d069d2dea-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.880 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.880 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.880 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.881 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.881 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.882 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009319aff4aa795c9a-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj155 2023-10-18T21:28:28.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj155 2023-10-18T21:28:28.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj155 2023-10-18T21:28:28.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.886 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.886 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.887 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.887 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.887 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj155 2023-10-18T21:28:28.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.888 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:iE9yV78dHB/DmxME41XExQnypKY= 2023-10-18T21:28:28.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:iE9yV78dHB/DmxME41XExQnypKY='} 2023-10-18T21:28:28.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083843a762e259a8d-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'bofrnf-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket bofrnf-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:28.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: tenant: 2023-10-18T21:28:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: bucket name: bofrnf-64 2023-10-18T21:28:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: total entries: 30 2023-10-18T21:28:28.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.894 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.895 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.895 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.895 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.895 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.896 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa1b437e9f6df9bd-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj37 2023-10-18T21:28:28.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj37 2023-10-18T21:28:28.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj37 2023-10-18T21:28:28.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.900 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.901 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.901 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.901 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.901 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj37 2023-10-18T21:28:28.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.902 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:+Ca/zmIoD+dI2+oAdi98dG9Lj5w= 2023-10-18T21:28:28.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:+Ca/zmIoD+dI2+oAdi98dG9Lj5w='} 2023-10-18T21:28:28.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e9c3ff44de601d1-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.906 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.907 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.907 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.907 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.908 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.908 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037341a399e7bbd0a-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj74 2023-10-18T21:28:28.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj74 2023-10-18T21:28:28.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj74 2023-10-18T21:28:28.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.912 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.913 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.913 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.913 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.914 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj74 2023-10-18T21:28:28.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.914 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:vLxMfXLH0Mmf8TNAN3geaG0E35c= 2023-10-18T21:28:28.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:vLxMfXLH0Mmf8TNAN3geaG0E35c='} 2023-10-18T21:28:28.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000aa850b1b5f27463-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.918 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.919 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.919 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.919 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.920 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.920 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs= 2023-10-18T21:28:28.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:xuQvwXVJwtsVJjczxM72df1LEgs=', 'Content-Length': '0'} 2023-10-18T21:28:28.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a29644f9e4bf98d-0065304850-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj111 2023-10-18T21:28:28.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj111 2023-10-18T21:28:28.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj111 2023-10-18T21:28:28.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.925 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.925 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.925 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.926 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:16 GMT 2023-10-18T21:28:28.926 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj111 2023-10-18T21:28:28.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.927 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:gF2QzDgB596Pwj69zv1+dq74/CU= 2023-10-18T21:28:28.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:16 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gF2QzDgB596Pwj69zv1+dq74/CU='} 2023-10-18T21:28:28.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5e195fc62f8062e-0065304850-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.931 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.931 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.932 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.932 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:17 GMT 2023-10-18T21:28:28.932 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.933 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Q0hKGy4sF1ZCPfe1ADjrYonNX28= 2023-10-18T21:28:28.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q0hKGy4sF1ZCPfe1ADjrYonNX28=', 'Content-Length': '0'} 2023-10-18T21:28:28.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a404ecaf6e8cad03-0065304851-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj148 2023-10-18T21:28:28.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj148 2023-10-18T21:28:28.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj148 2023-10-18T21:28:28.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.937 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.937 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.938 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.938 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:17 GMT 2023-10-18T21:28:28.938 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj148 2023-10-18T21:28:28.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.939 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:T+fyg5esgpGNE1c/Li0P5q7Cfao= 2023-10-18T21:28:28.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:T+fyg5esgpGNE1c/Li0P5q7Cfao='} 2023-10-18T21:28:28.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005357316cb5806c97-0065304851-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/ 2023-10-18T21:28:28.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/ 2023-10-18T21:28:28.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:28.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/ 2023-10-18T21:28:28.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:28.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.943 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:28.944 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.944 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.944 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:17 GMT 2023-10-18T21:28:28.944 INFO:tasks.rgw_multisite_tests:/bofrnf-64/ 2023-10-18T21:28:28.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.945 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Q0hKGy4sF1ZCPfe1ADjrYonNX28= 2023-10-18T21:28:28.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:04:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Q0hKGy4sF1ZCPfe1ADjrYonNX28=', 'Content-Length': '0'} 2023-10-18T21:28:28.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e2e25f096c31456-0065304851-1664-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:04:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-64/obj185 2023-10-18T21:28:28.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-64/obj185 2023-10-18T21:28:28.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:28.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-64/obj185 2023-10-18T21:28:28.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:28.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-18T21:28:28.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:28.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:28.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:28.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:28.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:28.949 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:28.949 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-18T21:28:28.950 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:28.950 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:04:17 GMT 2023-10-18T21:28:28.950 INFO:tasks.rgw_multisite_tests:/bofrnf-64/obj185 2023-10-18T21:28:28.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:28.951 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:k9CLn46GT6lafE0QhpfXPu3rhMA= 2023-10-18T21:28:28.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:04:17 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:k9CLn46GT6lafE0QhpfXPu3rhMA='} 2023-10-18T21:28:28.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b925caa5dbc73e7-0065304851-1664-a1'), ('Date', 'Wed, 18 Oct 2023 21:04:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:28.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:17.156+0000 7efd078c3780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-18T21:28:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:17.158+0000 7efd078c3780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-18T21:28:28.954 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-10-18T21:28:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-18T21:28:28.955 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-10-18T21:28:28.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-10-18T21:28:28.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T21:28:28.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-18T21:28:28.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi082.front.sepia.ceph.com port 8001: Connection refused 2023-10-18T21:28:28.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > curl http://smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:28.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-18T21:28:28.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-18T21:28:28.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-18T21:28:28.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: anonymous 2023-10-18T21:28:28.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=bofrnf-64 2023-10-18T21:28:28.958 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:28.959 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'bofrnf-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-18T21:28:28.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket bofrnf-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-18T21:28:28.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:18.369+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:23.370+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:28.371+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:33.371+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:38.372+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:43.373+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:48.373+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:53.373+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:04:58.374+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:03.375+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:08.376+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:13.377+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:18.377+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:23.378+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:28.379+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:33.380+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:38.380+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:43.380+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:48.381+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:53.381+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:05:58.383+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:03.383+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:08.384+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:13.385+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:18.385+0000 7fdae3324780 1 waiting to reach latest gen 6, on 0.. 2023-10-18T21:28:28.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:23.386+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:28.387+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:33.387+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:38.387+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:43.388+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:48.389+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:53.390+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:06:58.391+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:03.391+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:08.392+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:13.393+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:18.393+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:23.394+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:28.394+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:33.394+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:38.395+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:43.397+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:48.397+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:53.398+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:07:58.398+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:03.399+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:08.400+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:13.401+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:18.401+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:23.401+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:28.402+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:33.402+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:38.403+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:43.404+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:48.405+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:53.405+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:08:58.406+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:09:03.407+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:09:08.407+0000 7fdae3324780 1 waiting to reach latest gen 6, on 2.. 2023-10-18T21:28:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:09:13.407+0000 7fdae3324780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-18T21:28:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:09:13.407+0000 7fdae3324780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:28:28.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:09:13.407+0000 7fdae3324780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-18T21:28:28.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-18T21:28:28.978 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-18T21:28:28.978 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:28.979 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-18T21:28:28.979 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2023-10-18T21:28:28.979 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-18T21:28:28.980 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-18T21:28:28.981 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-18T21:28:28.981 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-18T21:28:28.981 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 2697, in test_sync_multiple_buckets_to_single 2023-10-18T21:28:28.982 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesB, content) 2023-10-18T21:28:28.982 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-10-18T21:28:28.982 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-10-18T21:28:28.982 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-10-18T21:28:28.983 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-10-18T21:28:28.983 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-10-18T21:28:28.983 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-18T21:28:28.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:28.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:28.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:28.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:28:28.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [ 2023-10-18T21:28:28.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:28.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697664097.449895_2017.1", 2023-10-18T21:28:28.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T21:21:37.449895Z" 2023-10-18T21:28:28.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:28.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:28.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:28.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:28:28.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:28.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:28.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:28.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "0.000000" 2023-10-18T21:28:28.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:28.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:28.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:28.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:28:28.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:28.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:28.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:28.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-18T21:28:28.991 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:28.992 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:28.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:28.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:28.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:28.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:37.741+0000 7feaeb73c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:28.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:37.742+0000 7feaeb73c780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:28.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:28.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:28.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:28.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:28.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:28.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:28.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:28.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:28.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:28.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:28.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:28.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:28.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:28.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:28:28.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:28.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:28.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:28.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:28:28.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:28.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:28.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:29.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:29.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:29.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:29.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:21:37Z" 2023-10-18T21:28:29.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:29.013 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.014 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.014 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.014 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.014 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.015 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.015 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.015 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.016 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.016 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.016 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.016 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.017 INFO:tasks.rgw_multisite_tests: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:28:29.017 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.017 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:29.018 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.018 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:28:29.018 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.019 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.019 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.019 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.020 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.020 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.020 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.021 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.022 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.022 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.023 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.023 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.023 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.024 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.024 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:29.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.025 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:29.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.026 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.026 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.027 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.027 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.027 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.028 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.028 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.029 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.030 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.030 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.030 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.031 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.031 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.031 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:29.031 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:29.032 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.032 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:21:37Z" 2023-10-18T21:28:29.032 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.033 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664076.148026_2005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.148026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:29.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697664097.449895_2017.1 target=1_1697664076.148026_2005.1 2023-10-18T21:28:29.034 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:42.864+0000 7fe761fbe780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:42.865+0000 7fe761fbe780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:28:29.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:29.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:28:29.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:29.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:29.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:29.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:29.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:21:42Z" 2023-10-18T21:28:29.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:29.057 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.057 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.057 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.058 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.058 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.058 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.059 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.059 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.059 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.060 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.060 INFO:tasks.rgw_multisite_tests: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:28:29.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.061 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:29.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:28:29.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.062 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.062 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.063 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.063 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.066 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.066 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.066 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.067 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.067 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.067 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.068 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:29.068 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.068 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.068 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.069 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:29.069 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.069 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.070 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.070 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.070 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.071 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.071 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.071 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.072 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.072 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.073 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.073 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.074 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.074 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.074 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.075 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:29.075 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:29.075 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.076 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:21:42Z" 2023-10-18T21:28:29.076 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.076 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664076.148026_2005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.148026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:29.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697664097.449895_2017.1 target=1_1697664076.148026_2005.1 2023-10-18T21:28:29.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:47.990+0000 7f5748fdd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:47.991+0000 7f5748fdd780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:28:29.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:29.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:28:29.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:29.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:29.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:29.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:21:47Z" 2023-10-18T21:28:29.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:29.100 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.100 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.101 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.101 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.101 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.102 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.102 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.102 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.103 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.103 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.103 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.104 INFO:tasks.rgw_multisite_tests: "marker": "1_1697664076.148026_2005.1", 2023-10-18T21:28:29.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.104 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:29.105 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.105 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:16.148026Z", 2023-10-18T21:28:29.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.105 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.106 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.106 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.106 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.107 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.107 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.108 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.109 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.109 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.109 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.110 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.110 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.110 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.111 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.111 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:29.111 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.112 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.112 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.112 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:29.113 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.113 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.113 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.113 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.114 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.114 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.115 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.115 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.117 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.117 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.117 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.117 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.118 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.118 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:29.118 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:29.119 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.119 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:21:47Z" 2023-10-18T21:28:29.119 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.120 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664076.148026_2005.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:16.148026Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:29.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697664076.148026_2005.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=0 master=1_1697664097.449895_2017.1 target=1_1697664076.148026_2005.1 2023-10-18T21:28:29.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.122 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:53.113+0000 7f7fc1bc8780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:53.115+0000 7f7fc1bc8780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697664097.449895_2017.1", 2023-10-18T21:28:29.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:29.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:37.449895Z", 2023-10-18T21:28:29.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:29.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:29.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:29.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:29.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:21:53Z" 2023-10-18T21:28:29.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:29.143 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.144 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.144 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.144 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.145 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.145 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.145 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.145 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.146 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.146 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.146 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.147 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.147 INFO:tasks.rgw_multisite_tests: "marker": "1_1697664097.449895_2017.1", 2023-10-18T21:28:29.147 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.148 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:29.148 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.148 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:37.449895Z", 2023-10-18T21:28:29.148 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.149 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.149 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.149 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.150 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.150 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.150 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.152 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.152 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.153 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.153 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.153 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.154 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.154 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.154 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:29.154 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.155 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.155 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.155 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:29.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.156 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.156 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.157 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.157 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.157 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.158 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.158 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.160 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.160 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.160 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.160 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.161 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.161 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:29.161 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:29.162 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.162 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:21:53Z" 2023-10-18T21:28:29.162 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.163 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664097.449895_2017.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:37.449895Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:29.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697664097.449895_2017.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-18T21:28:29.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:29.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: {"groups":[]} 2023-10-18T21:28:29.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=bofrnf-72 2023-10-18T21:28:29.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/ 2023-10-18T21:28:29.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/ 2023-10-18T21:28:29.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:29.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/ 2023-10-18T21:28:29.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:29.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.169 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:29.170 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.170 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.170 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:53 GMT 2023-10-18T21:28:29.171 INFO:tasks.rgw_multisite_tests:/bofrnf-72/ 2023-10-18T21:28:29.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.171 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:Xqh0ZCCaDABnjrO2IbpCnqF6hc0= 2023-10-18T21:28:29.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:Xqh0ZCCaDABnjrO2IbpCnqF6hc0=', 'Content-Length': '0'} 2023-10-18T21:28:29.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000054aacebce12bd713-0065304c71-270f-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=bofrnf-73 2023-10-18T21:28:29.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/ 2023-10-18T21:28:29.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/ 2023-10-18T21:28:29.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:29.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/ 2023-10-18T21:28:29.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:29.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.176 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:29.176 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.176 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.177 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:53 GMT 2023-10-18T21:28:29.177 INFO:tasks.rgw_multisite_tests:/bofrnf-73/ 2023-10-18T21:28:29.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.178 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:cWfc+cJzaPkc70mxpvZssEqDiFU= 2023-10-18T21:28:29.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:53 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:cWfc+cJzaPkc70mxpvZssEqDiFU=', 'Content-Length': '0'} 2023-10-18T21:28:29.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f211f7c82b00c23f-0065304c71-270f-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.179 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:28:29.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [ 2023-10-18T21:28:29.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:28:29.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T21:21:53.281518Z" 2023-10-18T21:28:29.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:28:29.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:29.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "0.000000" 2023-10-18T21:28:29.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:28:29.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-18T21:28:29.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:53.472+0000 7fe1a6d48780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:53.473+0000 7fe1a6d48780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:28:29.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:29.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:53.281518Z", 2023-10-18T21:28:29.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:29.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:29.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:29.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:29.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:21:53Z" 2023-10-18T21:28:29.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:29.208 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.209 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.209 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.209 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.210 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.210 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.211 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.211 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.211 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.211 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.212 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.212 INFO:tasks.rgw_multisite_tests: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:28:29.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.213 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:29.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:53.281518Z", 2023-10-18T21:28:29.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.214 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.214 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.214 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.215 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.215 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.217 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.217 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.218 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.218 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.218 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.219 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.219 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.219 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:29.220 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.220 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.220 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.220 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:29.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.221 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.221 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.222 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.222 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.222 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.223 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.224 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.225 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.225 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.225 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.225 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.226 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.226 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:29.226 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:29.227 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.227 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:21:53Z" 2023-10-18T21:28:29.227 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.228 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664113.281518_2033.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:53.281518Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:29.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-18T21:28:29.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:29.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-18T21:28:29.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:29.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-10-18T21:28:29.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-72', '--dest-bucket=bofrnf-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-72', '--dest-bucket=bofrnf-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-bofrnf-72 --source-zones=a1 --dest-zones=a2 --source-bucket=bofrnf-72 --dest-bucket=bofrnf-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:29.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:28:29.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-73', '--dest-bucket=bofrnf-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-bofrnf-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=bofrnf-73', '--dest-bucket=bofrnf-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-bofrnf-73 --source-zones=a1 --dest-zones=a2 --source-bucket=bofrnf-73 --dest-bucket=bofrnf-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:29.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-18T21:28:29.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:29.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:28:29.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:29.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "epoch": 20, 2023-10-18T21:28:29.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "predecessor_uuid": "aaf2721a-72fd-4d7e-add3-5a97d3603481", 2023-10-18T21:28:29.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": [], 2023-10-18T21:28:29.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period_map": { 2023-10-18T21:28:29.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zonegroups": [ 2023-10-18T21:28:29.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "a", 2023-10-18T21:28:29.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "name": "a", 2023-10-18T21:28:29.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "api_name": "", 2023-10-18T21:28:29.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "is_master": true, 2023-10-18T21:28:29.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "endpoints": [ 2023-10-18T21:28:29.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:28:29.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hostnames": [], 2023-10-18T21:28:29.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hostnames_s3website": [], 2023-10-18T21:28:29.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:28:29.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones": [ 2023-10-18T21:28:29.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:28:29.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "name": "a1", 2023-10-18T21:28:29.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "endpoints": [ 2023-10-18T21:28:29.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "http://smithi082.front.sepia.ceph.com:8000" 2023-10-18T21:28:29.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "log_meta": false, 2023-10-18T21:28:29.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "log_data": true, 2023-10-18T21:28:29.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-18T21:28:29.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "read_only": false, 2023-10-18T21:28:29.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "tier_type": "", 2023-10-18T21:28:29.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_from_all": true, 2023-10-18T21:28:29.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_from": [], 2023-10-18T21:28:29.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "redirect_zone": "", 2023-10-18T21:28:29.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "supported_features": [ 2023-10-18T21:28:29.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "compress-encrypted", 2023-10-18T21:28:29.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding" 2023-10-18T21:28:29.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:28:29.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "name": "a2", 2023-10-18T21:28:29.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "endpoints": [ 2023-10-18T21:28:29.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "http://smithi082.front.sepia.ceph.com:8001" 2023-10-18T21:28:29.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "log_meta": false, 2023-10-18T21:28:29.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "log_data": true, 2023-10-18T21:28:29.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-18T21:28:29.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "read_only": false, 2023-10-18T21:28:29.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "tier_type": "", 2023-10-18T21:28:29.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_from_all": true, 2023-10-18T21:28:29.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_from": [], 2023-10-18T21:28:29.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "redirect_zone": "", 2023-10-18T21:28:29.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "supported_features": [ 2023-10-18T21:28:29.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "compress-encrypted", 2023-10-18T21:28:29.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding" 2023-10-18T21:28:29.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "placement_targets": [ 2023-10-18T21:28:29.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "name": "default-placement", 2023-10-18T21:28:29.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "tags": [], 2023-10-18T21:28:29.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "storage_classes": [ 2023-10-18T21:28:29.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "STANDARD" 2023-10-18T21:28:29.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "default_placement": "default-placement", 2023-10-18T21:28:29.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:28:29.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_policy": { 2023-10-18T21:28:29.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "groups": [ 2023-10-18T21:28:29.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "sync-group", 2023-10-18T21:28:29.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "data_flow": { 2023-10-18T21:28:29.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "directional": [ 2023-10-18T21:28:29.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "source_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:28:29.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "dest_zone": "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:28:29.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pipes": [ 2023-10-18T21:28:29.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "sync-pipe-bofrnf-72", 2023-10-18T21:28:29.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "source": { 2023-10-18T21:28:29.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket": "bofrnf-72", 2023-10-18T21:28:29.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones": [ 2023-10-18T21:28:29.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:28:29.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "dest": { 2023-10-18T21:28:29.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket": "bofrnf-73", 2023-10-18T21:28:29.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones": [ 2023-10-18T21:28:29.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:28:29.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "params": { 2023-10-18T21:28:29.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "source": { 2023-10-18T21:28:29.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "filter": { 2023-10-18T21:28:29.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "tags": [] 2023-10-18T21:28:29.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "dest": {}, 2023-10-18T21:28:29.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "priority": 0, 2023-10-18T21:28:29.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "mode": "system", 2023-10-18T21:28:29.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "user": "" 2023-10-18T21:28:29.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "sync-pipe-bofrnf-73", 2023-10-18T21:28:29.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "source": { 2023-10-18T21:28:29.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket": "bofrnf-73", 2023-10-18T21:28:29.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones": [ 2023-10-18T21:28:29.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "403a921b-fcfc-4ab2-b434-bf29d35a51b6" 2023-10-18T21:28:29.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "dest": { 2023-10-18T21:28:29.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket": "bofrnf-73", 2023-10-18T21:28:29.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones": [ 2023-10-18T21:28:29.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "419020ed-d3b2-4ce0-b36c-97abb284c459" 2023-10-18T21:28:29.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "params": { 2023-10-18T21:28:29.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "source": { 2023-10-18T21:28:29.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "filter": { 2023-10-18T21:28:29.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "tags": [] 2023-10-18T21:28:29.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "dest": {}, 2023-10-18T21:28:29.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "priority": 0, 2023-10-18T21:28:29.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "mode": "system", 2023-10-18T21:28:29.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "user": "" 2023-10-18T21:28:29.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "enabled" 2023-10-18T21:28:29.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "enabled_features": [ 2023-10-18T21:28:29.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding" 2023-10-18T21:28:29.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "b", 2023-10-18T21:28:29.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "name": "b", 2023-10-18T21:28:29.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "api_name": "", 2023-10-18T21:28:29.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "is_master": false, 2023-10-18T21:28:29.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "endpoints": [ 2023-10-18T21:28:29.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:28:29.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hostnames": [], 2023-10-18T21:28:29.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "hostnames_s3website": [], 2023-10-18T21:28:29.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "master_zone": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:28:29.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "zones": [ 2023-10-18T21:28:29.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:28:29.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "name": "b1", 2023-10-18T21:28:29.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "endpoints": [ 2023-10-18T21:28:29.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "http://smithi156.front.sepia.ceph.com:8000" 2023-10-18T21:28:29.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "log_meta": false, 2023-10-18T21:28:29.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "log_data": true, 2023-10-18T21:28:29.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-18T21:28:29.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "read_only": false, 2023-10-18T21:28:29.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "tier_type": "", 2023-10-18T21:28:29.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_from_all": true, 2023-10-18T21:28:29.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_from": [], 2023-10-18T21:28:29.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "redirect_zone": "", 2023-10-18T21:28:29.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "supported_features": [ 2023-10-18T21:28:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "compress-encrypted", 2023-10-18T21:28:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding" 2023-10-18T21:28:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "id": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:28:29.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "name": "b2", 2023-10-18T21:28:29.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "endpoints": [ 2023-10-18T21:28:29.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "http://smithi156.front.sepia.ceph.com:8001" 2023-10-18T21:28:29.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "log_meta": false, 2023-10-18T21:28:29.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "log_data": true, 2023-10-18T21:28:29.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-18T21:28:29.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "read_only": false, 2023-10-18T21:28:29.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "tier_type": "", 2023-10-18T21:28:29.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_from_all": true, 2023-10-18T21:28:29.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_from": [], 2023-10-18T21:28:29.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "redirect_zone": "", 2023-10-18T21:28:29.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "supported_features": [ 2023-10-18T21:28:29.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "compress-encrypted", 2023-10-18T21:28:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding" 2023-10-18T21:28:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "placement_targets": [ 2023-10-18T21:28:29.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "name": "default-placement", 2023-10-18T21:28:29.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "tags": [], 2023-10-18T21:28:29.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "storage_classes": [ 2023-10-18T21:28:29.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "STANDARD" 2023-10-18T21:28:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "default_placement": "default-placement", 2023-10-18T21:28:29.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:28:29.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_policy": { 2023-10-18T21:28:29.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "groups": [] 2023-10-18T21:28:29.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "enabled_features": [ 2023-10-18T21:28:29.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "resharding" 2023-10-18T21:28:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ], 2023-10-18T21:28:29.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "short_zone_ids": [ 2023-10-18T21:28:29.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:28:29.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": 3791896933 2023-10-18T21:28:29.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": "419020ed-d3b2-4ce0-b36c-97abb284c459", 2023-10-18T21:28:29.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": 2277887411 2023-10-18T21:28:29.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": "576f1485-1a9f-4b57-9613-5ba13a3f0a36", 2023-10-18T21:28:29.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": 953824181 2023-10-18T21:28:29.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": "a08fbfd2-309c-4698-8327-be8d0aa187fe", 2023-10-18T21:28:29.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": 2717429769 2023-10-18T21:28:29.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "master_zonegroup": "a", 2023-10-18T21:28:29.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "master_zone": "403a921b-fcfc-4ab2-b434-bf29d35a51b6", 2023-10-18T21:28:29.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period_config": { 2023-10-18T21:28:29.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket_quota": { 2023-10-18T21:28:29.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "enabled": false, 2023-10-18T21:28:29.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "check_on_raw": false, 2023-10-18T21:28:29.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_size": -1, 2023-10-18T21:28:29.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_size_kb": 0, 2023-10-18T21:28:29.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_objects": -1 2023-10-18T21:28:29.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "user_quota": { 2023-10-18T21:28:29.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "enabled": false, 2023-10-18T21:28:29.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "check_on_raw": false, 2023-10-18T21:28:29.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_size": -1, 2023-10-18T21:28:29.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_size_kb": 0, 2023-10-18T21:28:29.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_objects": -1 2023-10-18T21:28:29.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "user_ratelimit": { 2023-10-18T21:28:29.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_read_ops": 0, 2023-10-18T21:28:29.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_write_ops": 0, 2023-10-18T21:28:29.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_read_bytes": 0, 2023-10-18T21:28:29.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_write_bytes": 0, 2023-10-18T21:28:29.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "enabled": false 2023-10-18T21:28:29.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "bucket_ratelimit": { 2023-10-18T21:28:29.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_read_ops": 0, 2023-10-18T21:28:29.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_write_ops": 0, 2023-10-18T21:28:29.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_read_bytes": 0, 2023-10-18T21:28:29.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_write_bytes": 0, 2023-10-18T21:28:29.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "enabled": false 2023-10-18T21:28:29.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "anonymous_ratelimit": { 2023-10-18T21:28:29.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_read_ops": 0, 2023-10-18T21:28:29.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_write_ops": 0, 2023-10-18T21:28:29.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_read_bytes": 0, 2023-10-18T21:28:29.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "max_write_bytes": 0, 2023-10-18T21:28:29.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "enabled": false 2023-10-18T21:28:29.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_id": "7565323d-5a72-4e51-af8d-a123f7dfc36a", 2023-10-18T21:28:29.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_name": "test-realm", 2023-10-18T21:28:29.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-18T21:28:29.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-18T21:28:29.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-bofrnf-72","source":{"bucket":"bofrnf-72","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-bofrnf-73","source":{"bucket":"bofrnf-73","zones":["a1"]},"dest":{"bucket":"bofrnf-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-18T21:28:29.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/ 2023-10-18T21:28:29.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/ 2023-10-18T21:28:29.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:29.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/ 2023-10-18T21:28:29.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:29.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.333 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:29.333 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.334 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.334 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:54 GMT 2023-10-18T21:28:29.334 INFO:tasks.rgw_multisite_tests:/bofrnf-72/ 2023-10-18T21:28:29.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.335 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:si/R/4V8fIhC004U772ccu8zUIc= 2023-10-18T21:28:29.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:si/R/4V8fIhC004U772ccu8zUIc=', 'Content-Length': '0'} 2023-10-18T21:28:29.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dba44a018f5f9ab7-0065304c72-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/obj1 2023-10-18T21:28:29.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/obj1 2023-10-18T21:28:29.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:29.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/obj1 2023-10-18T21:28:29.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:28:29.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.339 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:29.339 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-18T21:28:29.339 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:29.340 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:54 GMT 2023-10-18T21:28:29.340 INFO:tasks.rgw_multisite_tests:/bofrnf-72/obj1 2023-10-18T21:28:29.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.341 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:pjXMvZC2rwaHbcGn5atN0UWu+4E= 2023-10-18T21:28:29.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:pjXMvZC2rwaHbcGn5atN0UWu+4E='} 2023-10-18T21:28:29.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008837a000d6f6d8e8-0065304c72-2942-a1'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/ 2023-10-18T21:28:29.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/ 2023-10-18T21:28:29.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:29.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/ 2023-10-18T21:28:29.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:29.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.345 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:29.345 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.345 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.345 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:54 GMT 2023-10-18T21:28:29.346 INFO:tasks.rgw_multisite_tests:/bofrnf-72/ 2023-10-18T21:28:29.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.346 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:si/R/4V8fIhC004U772ccu8zUIc= 2023-10-18T21:28:29.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:si/R/4V8fIhC004U772ccu8zUIc=', 'Content-Length': '0'} 2023-10-18T21:28:29.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023bdb03a660346b8-0065304c72-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/obj2 2023-10-18T21:28:29.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/obj2 2023-10-18T21:28:29.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:29.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/obj2 2023-10-18T21:28:29.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:28:29.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.350 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:29.351 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-18T21:28:29.351 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:29.351 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:54 GMT 2023-10-18T21:28:29.351 INFO:tasks.rgw_multisite_tests:/bofrnf-72/obj2 2023-10-18T21:28:29.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.352 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:gJgznvhWp4aGaeMR5ZCGWqmpwnM= 2023-10-18T21:28:29.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:gJgznvhWp4aGaeMR5ZCGWqmpwnM='} 2023-10-18T21:28:29.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fe6fb41ab593284-0065304c72-2942-a1'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/ 2023-10-18T21:28:29.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/ 2023-10-18T21:28:29.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:29.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/ 2023-10-18T21:28:29.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:29.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.356 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:29.356 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.357 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.357 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:54 GMT 2023-10-18T21:28:29.357 INFO:tasks.rgw_multisite_tests:/bofrnf-73/ 2023-10-18T21:28:29.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.358 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:wj5Ml4FWe7rTryQzHLhC1Hf7HJk= 2023-10-18T21:28:29.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wj5Ml4FWe7rTryQzHLhC1Hf7HJk=', 'Content-Length': '0'} 2023-10-18T21:28:29.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011856543536123a2-0065304c72-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/obj3 2023-10-18T21:28:29.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/obj3 2023-10-18T21:28:29.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:29.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/obj3 2023-10-18T21:28:29.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:28:29.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.362 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:29.362 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-18T21:28:29.362 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:29.363 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:54 GMT 2023-10-18T21:28:29.363 INFO:tasks.rgw_multisite_tests:/bofrnf-73/obj3 2023-10-18T21:28:29.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.364 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:6UT2N/clV/eruVzctJ1/Pta4H4I= 2023-10-18T21:28:29.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:6UT2N/clV/eruVzctJ1/Pta4H4I='} 2023-10-18T21:28:29.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008481a9e3276c9813-0065304c72-2942-a1'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/ 2023-10-18T21:28:29.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/ 2023-10-18T21:28:29.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:29.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/ 2023-10-18T21:28:29.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:29.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.368 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:29.368 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.368 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.368 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:54 GMT 2023-10-18T21:28:29.369 INFO:tasks.rgw_multisite_tests:/bofrnf-73/ 2023-10-18T21:28:29.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.369 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:wj5Ml4FWe7rTryQzHLhC1Hf7HJk= 2023-10-18T21:28:29.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:wj5Ml4FWe7rTryQzHLhC1Hf7HJk=', 'Content-Length': '0'} 2023-10-18T21:28:29.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013170a774f048246-0065304c72-2942-a1'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/obj4 2023-10-18T21:28:29.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/obj4 2023-10-18T21:28:29.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-18T21:28:29.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/obj4 2023-10-18T21:28:29.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:29.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-18T21:28:29.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8000 2023-10-18T21:28:29.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-18T21:28:29.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:29.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:29.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:29.373 INFO:tasks.rgw_multisite_tests:PUT 2023-10-18T21:28:29.374 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-18T21:28:29.374 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-18T21:28:29.374 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:21:54 GMT 2023-10-18T21:28:29.375 INFO:tasks.rgw_multisite_tests:/bofrnf-73/obj4 2023-10-18T21:28:29.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:29.375 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:4Glt+7KapLmZ+CehFJ6AmLbgVyI= 2023-10-18T21:28:29.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 18 Oct 2023 21:21:54 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:4Glt+7KapLmZ+CehFJ6AmLbgVyI='} 2023-10-18T21:28:29.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fa2712244d9bd92-0065304c72-2942-a1'), ('Date', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:29.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: No --period given, using current period=67bf1060-aa43-46d9-8036-efd54ef55d7d 2023-10-18T21:28:29.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [ 2023-10-18T21:28:29.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:28:29.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T21:21:53.281518Z" 2023-10-18T21:28:29.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.933926Z" 2023-10-18T21:28:29.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:29.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "0.000000" 2023-10-18T21:28:29.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T20:45:51.953788Z" 2023-10-18T21:28:29.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-18T21:28:29.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:54.790+0000 7f3bc0aa2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: 2023-10-18T21:21:54.791+0000 7f3bc0aa2780 20 RGW-SYNC:meta: read sync status 2023-10-18T21:28:29.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:28:29.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 1, 2023-10-18T21:28:29.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:53.281518Z", 2023-10-18T21:28:29.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "", 2023-10-18T21:28:29.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "0.000000", 2023-10-18T21:28:29.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "state": 1, 2023-10-18T21:28:29.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "realm_epoch": 2 2023-10-18T21:28:29.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 1, 2023-10-18T21:28:29.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 1 2023-10-18T21:28:29.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:21:54Z" 2023-10-18T21:28:29.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-18T21:28:29.405 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.406 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.406 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.406 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.406 INFO:tasks.rgw_multisite_tests: "period": "67bf1060-aa43-46d9-8036-efd54ef55d7d", 2023-10-18T21:28:29.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.407 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.407 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.408 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.408 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.408 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.409 INFO:tasks.rgw_multisite_tests: "marker": "1_1697664113.281518_2033.1", 2023-10-18T21:28:29.409 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.409 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-18T21:28:29.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.410 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:53.281518Z", 2023-10-18T21:28:29.410 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.411 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.411 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.411 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.412 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.412 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.412 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.412 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.933926_367.1", 2023-10-18T21:28:29.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.413 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.413 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.414 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.933926Z", 2023-10-18T21:28:29.414 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.414 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.415 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.415 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.415 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.415 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.416 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.416 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-18T21:28:29.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.417 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-18T21:28:29.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.418 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.418 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.418 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.419 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.419 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.419 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-18T21:28:29.420 INFO:tasks.rgw_multisite_tests: "marker": "1_1697661951.953788_570.1", 2023-10-18T21:28:29.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.420 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.421 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T20:45:51.953788Z", 2023-10-18T21:28:29.421 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-18T21:28:29.422 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.422 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.422 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.422 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.423 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.423 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-18T21:28:29.423 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-18T21:28:29.424 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.424 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:21:54Z" 2023-10-18T21:28:29.424 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.425 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697664113.281518_2033.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-18T21:21:53.281518Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1697661951.933926_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.933926Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697661951.953788_570.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T20:45:51.953788Z', 'realm_epoch': 2}}] 2023-10-18T21:28:29.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697664113.281518_2033.1', 1: '1_1697661951.933926_367.1', 2: '', 3: '1_1697661951.953788_570.1'} 2023-10-18T21:28:29.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-18T21:28:29.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: [ 2023-10-18T21:28:29.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:28:29.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:28:29.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:29.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:29.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:29.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "last_update": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:29.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:28:29.434 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.434 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:21:55Z" 2023-10-18T21:28:29.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.454 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.454 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.454 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.455 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.455 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.455 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.455 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.456 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.456 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.456 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.457 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.457 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.457 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.458 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.458 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.459 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.459 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.459 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.460 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.460 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.460 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.461 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.462 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.462 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.462 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.463 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.463 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.463 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.464 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.464 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.465 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.465 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.466 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.466 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.466 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.466 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.467 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.467 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.467 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.468 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.469 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.469 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.469 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.469 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.470 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.470 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.470 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.471 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.471 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:21:55Z" 2023-10-18T21:28:29.471 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.472 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:00Z" 2023-10-18T21:28:29.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.493 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.494 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.494 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.494 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.495 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.495 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.495 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.496 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.496 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.496 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.496 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.497 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.497 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.498 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.499 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.499 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.499 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.499 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.500 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.500 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.501 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.502 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.502 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.502 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.503 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.503 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.503 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.504 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.504 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.505 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.505 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.505 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.506 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.506 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.506 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.507 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.507 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.508 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.508 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.509 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.509 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.509 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.510 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.510 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.510 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.511 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:00Z" 2023-10-18T21:28:29.511 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.511 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:05Z" 2023-10-18T21:28:29.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.533 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.533 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.534 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.534 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.534 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.535 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.535 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.535 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.535 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.536 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.536 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.536 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.537 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.538 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.538 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.538 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.539 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.539 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.539 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.540 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.540 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.541 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.542 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.542 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.542 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.542 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.543 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.543 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.544 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.545 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.545 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.545 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.545 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.546 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.546 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.546 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.547 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.547 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.548 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.548 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.548 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.549 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.549 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.549 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.549 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.550 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.550 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.550 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:05Z" 2023-10-18T21:28:29.551 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.551 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:10Z" 2023-10-18T21:28:29.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.572 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.573 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.573 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.573 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.574 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.574 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.574 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.575 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.575 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.575 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.575 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.576 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.576 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.577 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.578 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.578 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.578 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.578 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.579 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.579 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.581 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.581 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.581 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.581 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.582 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.582 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.582 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.583 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.584 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.584 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.584 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.585 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.585 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.585 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.586 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.586 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.586 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.587 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.587 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.587 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.588 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.588 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.588 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.589 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.589 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.589 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.590 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:10Z" 2023-10-18T21:28:29.590 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.591 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:15Z" 2023-10-18T21:28:29.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.613 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.614 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.614 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.614 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.615 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.615 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.615 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.615 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.616 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.616 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.616 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.617 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.617 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.618 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.618 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.619 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.619 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.619 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.620 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.620 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.620 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.621 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.622 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.622 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.622 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.623 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.623 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.623 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.624 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.624 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.625 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.625 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.625 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.626 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.626 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.626 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.626 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.627 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.627 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.628 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.628 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.629 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.629 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.629 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.629 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.630 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.630 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.630 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.631 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:15Z" 2023-10-18T21:28:29.631 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.631 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:20Z" 2023-10-18T21:28:29.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.653 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.654 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.654 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.654 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.654 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.655 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.655 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.655 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.656 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.656 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.656 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.657 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.657 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.657 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.658 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.662 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.662 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.662 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.663 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.663 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.663 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.664 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.664 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.665 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.665 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.666 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.666 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.666 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.667 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.667 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.667 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.668 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.669 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.669 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.670 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.670 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.670 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.671 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.671 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.671 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.672 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.673 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.673 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.673 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.674 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.674 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.674 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.675 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.675 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:20Z" 2023-10-18T21:28:29.675 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.675 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:25Z" 2023-10-18T21:28:29.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.697 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.697 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.698 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.698 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.698 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.699 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.699 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.699 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.700 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.700 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.700 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.700 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.701 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.702 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.702 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.703 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.703 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.703 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.704 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.704 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.705 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.706 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.706 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.706 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.707 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.707 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.707 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.708 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.709 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.709 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.710 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.710 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.710 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.711 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.711 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.712 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.713 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.713 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.713 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.713 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.714 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.714 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.714 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.715 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.715 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:25Z" 2023-10-18T21:28:29.715 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.716 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:30Z" 2023-10-18T21:28:29.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.737 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.738 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.738 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.738 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.739 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.739 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.739 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.740 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.740 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.740 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.740 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.741 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.742 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.743 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.743 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.743 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.743 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.744 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.744 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.745 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.746 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.746 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.746 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.747 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.747 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.747 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.747 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.748 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.749 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.750 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.750 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.750 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.751 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.751 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.751 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.752 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.753 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.753 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.753 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.754 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.754 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.754 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.755 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.755 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.755 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:30Z" 2023-10-18T21:28:29.756 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.756 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:36Z" 2023-10-18T21:28:29.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.778 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.778 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.778 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.779 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.779 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.779 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.780 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.780 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.780 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.781 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.781 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.781 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.783 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.783 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.783 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.784 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.784 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.784 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.785 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.785 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.786 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.787 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.787 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.787 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.787 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.788 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.788 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.788 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.789 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.789 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.790 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.790 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.791 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.791 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.791 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.792 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.792 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.793 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.793 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.794 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.794 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.794 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.795 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.795 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.795 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.796 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.796 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:36Z" 2023-10-18T21:28:29.796 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.796 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:41Z" 2023-10-18T21:28:29.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.819 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.819 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.819 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.820 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.820 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.820 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.821 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.821 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.821 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.822 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.822 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.822 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.824 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.824 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.824 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.825 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.825 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.826 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.826 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.827 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.827 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.827 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.828 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.828 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.828 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.829 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.829 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.829 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.830 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.831 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.831 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.832 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.832 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.832 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.833 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.833 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.833 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.834 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.835 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.835 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.835 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.836 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.836 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.836 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.836 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.837 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:41Z" 2023-10-18T21:28:29.837 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.837 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:46Z" 2023-10-18T21:28:29.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.860 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.860 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.861 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.861 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.861 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.861 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.862 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.862 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.865 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.866 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.866 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.866 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.867 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.868 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.868 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.868 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.869 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.869 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.869 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.870 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.870 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.870 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.871 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.872 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.872 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.872 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.873 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.873 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.873 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.874 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.874 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.875 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.875 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.875 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.876 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.876 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.876 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.877 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.877 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.878 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.878 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.878 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.879 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.879 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.879 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.879 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.880 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.880 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.880 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.881 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:46Z" 2023-10-18T21:28:29.881 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.881 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:51Z" 2023-10-18T21:28:29.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.903 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.904 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.904 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.904 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.904 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.905 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.905 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.906 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.906 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.906 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.907 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.907 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.907 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.908 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.909 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.909 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.909 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.910 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.910 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.910 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.911 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.912 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.912 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.912 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.913 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.913 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.913 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.914 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.914 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.915 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.915 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.916 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.916 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.916 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.917 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.917 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.917 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.918 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.919 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.919 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.919 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.920 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.920 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.920 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.920 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.921 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:51Z" 2023-10-18T21:28:29.921 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.921 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:22:56Z" 2023-10-18T21:28:29.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.944 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.944 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.944 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.944 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.945 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.945 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.945 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.946 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.946 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.946 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.947 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.947 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.948 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.948 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.949 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.949 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.949 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.950 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.950 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.950 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.952 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.952 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.953 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.953 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.953 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.954 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.954 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.954 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.955 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.956 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.956 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.956 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.957 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.957 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.957 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.958 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.959 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.959 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.959 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:29.960 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.960 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:29.960 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:29.961 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:29.961 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.961 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:22:56Z" 2023-10-18T21:28:29.962 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:29.962 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:29.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:29.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:29.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:29.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:29.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:29.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:29.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:29.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:29.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:29.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:29.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:29.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:29.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:29.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:29.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:29.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:29.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:29.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:29.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:29.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:29.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:29.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:29.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:29.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:29.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:29.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:29.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:29.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:29.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:29.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:29.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:29.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:01Z" 2023-10-18T21:28:29.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:29.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:29.984 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:29.984 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:29.984 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:29.985 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:29.985 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:29.985 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.985 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:29.986 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.986 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:29.986 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.987 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.987 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:29.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:29.989 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.989 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.989 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.990 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:29.990 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.990 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.990 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:29.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.991 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:29.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.992 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:29.992 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.992 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.993 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.993 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:29.993 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.993 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.994 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:29.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.995 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:29.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:29.996 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.996 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:29.996 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:29.996 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:29.997 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:29.997 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:29.997 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:29.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:29.998 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:29.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:29.999 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:29.999 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.999 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:29.999 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.000 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.000 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.001 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.001 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.001 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.002 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:01Z" 2023-10-18T21:28:30.002 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.003 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:30.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:30.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:30.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:30.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:30.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:30.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:30.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:06Z" 2023-10-18T21:28:30.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.025 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.025 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.025 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.025 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.026 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.026 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.026 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.027 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.027 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.028 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.028 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.028 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.029 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.029 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.030 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.030 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.030 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.031 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.031 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.031 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.031 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.032 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.032 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.033 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.033 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.034 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.034 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.034 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.035 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.035 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.035 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:30.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.036 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:30.037 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.037 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.037 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.038 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.038 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.039 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:30.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.039 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:30.040 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.041 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.041 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.041 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.041 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.042 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.042 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.042 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.043 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:06Z" 2023-10-18T21:28:30.043 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.043 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:30.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:30.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:30.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:30.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:30.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:30.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:30.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:11Z" 2023-10-18T21:28:30.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.065 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.065 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.066 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.066 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.066 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.067 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.067 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.067 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.068 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.068 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.068 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.069 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.069 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.070 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.070 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.070 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.070 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.071 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.071 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.071 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.072 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.072 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.073 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.073 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.073 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.074 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.074 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.074 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.075 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.075 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.075 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:30.076 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.076 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.076 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.077 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:30.077 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.077 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.078 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.078 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.078 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.078 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.079 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:30.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.079 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.080 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:30.080 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.081 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.081 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.081 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.081 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.082 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.082 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.082 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.083 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:11Z" 2023-10-18T21:28:30.083 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.083 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:30.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:30.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:30.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.085 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:30.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:30.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:30.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:30.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:17Z" 2023-10-18T21:28:30.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.105 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.105 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.105 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.106 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.106 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.106 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.107 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.107 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.107 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.108 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.108 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.108 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.110 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.110 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.110 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.111 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.111 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.111 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.111 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.112 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.112 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.112 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.113 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.113 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.113 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.114 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.114 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.114 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.115 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.115 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.115 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:30.116 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.116 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:30.117 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.117 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.117 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.118 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.118 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.118 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.119 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:30.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.119 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.120 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:30.120 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.121 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.121 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.121 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.122 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.122 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.122 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.122 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.123 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:17Z" 2023-10-18T21:28:30.123 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.124 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:30.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:30.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:30.125 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.125 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:30.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:30.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:30.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:30.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:22Z" 2023-10-18T21:28:30.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.146 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.146 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.147 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.147 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.147 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.148 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.148 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.148 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.149 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.149 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.149 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.149 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.151 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.151 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.151 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.152 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.152 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.152 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.153 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.153 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.153 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.153 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.154 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.154 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.154 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.155 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.155 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.155 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.156 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.156 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.156 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032483", 2023-10-18T21:28:30.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.157 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.158 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.728035Z" 2023-10-18T21:28:30.158 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.158 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.159 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.159 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.159 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.159 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.160 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051813", 2023-10-18T21:28:30.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.160 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.161 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.193271Z" 2023-10-18T21:28:30.161 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.162 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.162 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.162 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.163 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.163 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.163 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.163 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.164 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:22Z" 2023-10-18T21:28:30.164 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.165 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.728035Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051813', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.193271Z'}}] 2023-10-18T21:28:30.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032483', 3: '00000000000000000000:00000000000000051813'} 2023-10-18T21:28:30.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799, shard=2 master=00000000000000000000:00000000000000032623 target=00000000000000000000:00000000000000032483, shard=3 master=00000000000000000000:00000000000000052094 target=00000000000000000000:00000000000000051813 2023-10-18T21:28:30.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:27Z" 2023-10-18T21:28:30.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.188 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.188 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.188 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.189 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.189 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.189 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.190 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.190 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.190 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.191 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.191 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.191 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.192 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.192 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.192 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.193 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.193 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.193 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.194 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.194 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.194 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.195 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.196 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.197 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.197 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.197 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.197 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.198 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.198 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.199 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.199 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.199 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.200 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.200 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.200 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.201 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.201 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.201 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.201 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.202 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.202 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.203 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.203 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.203 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.204 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.204 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.204 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.205 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.205 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.205 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.205 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.206 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:27Z" 2023-10-18T21:28:30.206 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.206 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:28:30.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:28:30.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:32Z" 2023-10-18T21:28:30.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.229 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.229 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.230 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.230 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.230 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.231 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.231 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.231 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.231 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.232 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.232 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.232 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.233 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.234 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.234 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.235 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.235 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.235 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.236 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.236 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.237 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.237 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.237 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.238 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.238 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.238 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.239 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.239 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.239 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.240 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.240 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.240 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.241 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.241 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.241 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.242 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.242 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.242 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.243 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.243 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.243 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.243 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.244 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.244 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.244 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.245 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.245 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.245 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.246 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.246 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.246 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.247 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.247 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.247 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:32Z" 2023-10-18T21:28:30.248 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.248 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:28:30.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:28:30.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:37Z" 2023-10-18T21:28:30.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.270 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.271 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.271 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.271 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.272 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.272 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.272 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.273 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.273 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.273 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.274 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.274 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.274 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.275 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.275 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.276 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.276 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.276 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.277 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.277 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.277 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.278 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.279 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.279 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.279 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.280 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.280 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.280 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.281 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.281 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.282 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.283 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.283 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.283 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.284 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.284 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.284 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.285 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.285 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.285 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.285 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.286 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.286 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.286 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.287 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.287 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.287 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.288 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.288 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.288 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:37Z" 2023-10-18T21:28:30.289 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.289 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:28:30.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:28:30.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:42Z" 2023-10-18T21:28:30.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.310 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.311 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.311 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.311 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.311 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.312 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.312 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.312 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.313 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.313 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.313 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.313 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.315 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.315 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.316 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.316 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.316 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.316 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.317 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.317 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.318 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.318 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.319 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.319 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.319 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.320 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.320 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.320 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.321 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.321 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.322 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.322 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.322 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.323 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.323 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.323 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.324 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.324 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.325 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.326 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.326 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.326 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.327 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.327 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.327 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.327 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.328 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:42Z" 2023-10-18T21:28:30.328 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.328 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:28:30.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:28:30.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:47Z" 2023-10-18T21:28:30.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.351 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.351 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.352 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.352 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.352 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.353 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.353 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.353 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.353 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.354 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.354 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.354 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.355 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.356 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.356 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.357 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.357 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.357 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.357 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.358 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.358 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.359 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.360 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.360 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.360 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.360 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.361 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.361 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.361 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.362 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.362 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.362 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.363 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.363 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.363 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.364 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.364 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.364 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.365 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.365 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.365 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.366 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.366 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.367 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.367 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.367 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.368 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.368 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.368 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.369 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.369 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:47Z" 2023-10-18T21:28:30.369 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.370 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:28:30.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:28:30.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:52Z" 2023-10-18T21:28:30.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.391 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.391 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.392 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.392 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.392 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.392 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.393 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.393 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.393 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.394 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.394 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.394 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.396 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.396 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.396 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.397 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.397 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.397 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.397 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.398 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045799", 2023-10-18T21:28:30.398 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.398 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.399 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:19:32.218710Z" 2023-10-18T21:28:30.399 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.400 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.400 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.400 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.401 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.401 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.401 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.402 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.402 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.403 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.403 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.403 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.404 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.404 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.404 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.405 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.405 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.406 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.407 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.407 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.407 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.408 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.408 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.408 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.409 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.409 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:52Z" 2023-10-18T21:28:30.409 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.410 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045799', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:19:32.218710Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:28:30.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045799', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000045939 target=00000000000000000000:00000000000000045799 2023-10-18T21:28:30.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-18T21:28:30.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-18T21:28:30.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-18T21:28:30.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-18T21:28:30.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "sync_status": { 2023-10-18T21:28:30.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "info": { 2023-10-18T21:28:30.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "sync", 2023-10-18T21:28:30.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "num_shards": 4, 2023-10-18T21:28:30.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "instance_id": 5898477700290860314 2023-10-18T21:28:30.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "markers": [ 2023-10-18T21:28:30.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 0, 2023-10-18T21:28:30.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 1, 2023-10-18T21:28:30.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:28:30.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 0, 2023-10-18T21:28:30.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:28:30.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 2, 2023-10-18T21:28:30.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 9, 2023-10-18T21:28:30.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: { 2023-10-18T21:28:30.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "key": 3, 2023-10-18T21:28:30.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "val": { 2023-10-18T21:28:30.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "status": "incremental-sync", 2023-10-18T21:28:30.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "next_step_marker": "", 2023-10-18T21:28:30.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total_entries": 55, 2023-10-18T21:28:30.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "pos": 0, 2023-10-18T21:28:30.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: ] 2023-10-18T21:28:30.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "full_sync": { 2023-10-18T21:28:30.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "total": 64, 2023-10-18T21:28:30.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "complete": 64 2023-10-18T21:28:30.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: }, 2023-10-18T21:28:30.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: "current_time": "2023-10-18T21:23:58Z" 2023-10-18T21:28:30.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi082.stdout: INFO: } 2023-10-18T21:28:30.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-18T21:28:30.432 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-18T21:28:30.432 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-18T21:28:30.432 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-18T21:28:30.433 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-18T21:28:30.433 INFO:tasks.rgw_multisite_tests: "instance_id": 5898477700290860314 2023-10-18T21:28:30.433 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.434 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-18T21:28:30.434 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.434 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-18T21:28:30.435 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.435 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.435 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044466", 2023-10-18T21:28:30.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.436 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.436 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:20.727617Z" 2023-10-18T21:28:30.437 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.437 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.437 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.438 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-18T21:28:30.438 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.438 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.439 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045939", 2023-10-18T21:28:30.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.439 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-18T21:28:30.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.440 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.394654Z" 2023-10-18T21:28:30.440 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.440 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.441 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.441 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-18T21:28:30.441 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.442 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.442 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032623", 2023-10-18T21:28:30.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.443 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-18T21:28:30.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.480390Z" 2023-10-18T21:28:30.444 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.444 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.444 INFO:tasks.rgw_multisite_tests: { 2023-10-18T21:28:30.444 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-18T21:28:30.445 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-18T21:28:30.445 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-18T21:28:30.445 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052094", 2023-10-18T21:28:30.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-18T21:28:30.446 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-18T21:28:30.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-18T21:28:30.447 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-18T21:21:54.419859Z" 2023-10-18T21:28:30.447 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.447 INFO:tasks.rgw_multisite_tests: } 2023-10-18T21:28:30.448 INFO:tasks.rgw_multisite_tests: ] 2023-10-18T21:28:30.448 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.448 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-18T21:28:30.448 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-18T21:28:30.449 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-18T21:28:30.449 INFO:tasks.rgw_multisite_tests: }, 2023-10-18T21:28:30.449 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-18T21:23:58Z" 2023-10-18T21:28:30.450 INFO:tasks.rgw_multisite_tests:} 2023-10-18T21:28:30.450 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044466', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:20.727617Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045939', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.394654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032623', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.480390Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052094', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-18T21:21:54.419859Z'}}] 2023-10-18T21:28:30.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044466', 1: '00000000000000000000:00000000000000045939', 2: '00000000000000000000:00000000000000032623', 3: '00000000000000000000:00000000000000052094'} 2023-10-18T21:28:30.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-18T21:28:30.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/ 2023-10-18T21:28:30.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/ 2023-10-18T21:28:30.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:30.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/ 2023-10-18T21:28:30.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-18T21:28:30.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.455 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:30.455 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.455 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.456 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.456 INFO:tasks.rgw_multisite_tests:/bofrnf-72/ 2023-10-18T21:28:30.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.457 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:43sOW77mFvJge1FBJGiOx595DKw= 2023-10-18T21:28:30.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:43sOW77mFvJge1FBJGiOx595DKw=', 'Content-Length': '0'} 2023-10-18T21:28:30.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c30ff2698bbc960-0065304cee-294e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/obj1 2023-10-18T21:28:30.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/obj1 2023-10-18T21:28:30.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:30.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/obj1 2023-10-18T21:28:30.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.461 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:30.461 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.462 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.462 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.462 INFO:tasks.rgw_multisite_tests:/bofrnf-72/obj1 2023-10-18T21:28:30.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.463 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:oq8/tvTriwHvgT7a+VmfWiXWld0= 2023-10-18T21:28:30.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:oq8/tvTriwHvgT7a+VmfWiXWld0=', 'Content-Length': '0'} 2023-10-18T21:28:30.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a1d5b8aea54b6121-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/obj2 2023-10-18T21:28:30.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/obj2 2023-10-18T21:28:30.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:30.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/obj2 2023-10-18T21:28:30.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.467 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:30.467 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.468 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.468 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.468 INFO:tasks.rgw_multisite_tests:/bofrnf-72/obj2 2023-10-18T21:28:30.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.469 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:A83J0zmGzyFr8lZpb/ggPT3Qi/A= 2023-10-18T21:28:30.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:A83J0zmGzyFr8lZpb/ggPT3Qi/A=', 'Content-Length': '0'} 2023-10-18T21:28:30.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f76e048952237c2e-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/obj3 2023-10-18T21:28:30.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/obj3 2023-10-18T21:28:30.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:30.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/obj3 2023-10-18T21:28:30.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.473 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:30.473 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.474 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.474 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.474 INFO:tasks.rgw_multisite_tests:/bofrnf-72/obj3 2023-10-18T21:28:30.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.475 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:CmdoGHy1/7EUd9rXCg4Tng8frKs= 2023-10-18T21:28:30.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:CmdoGHy1/7EUd9rXCg4Tng8frKs=', 'Content-Length': '0'} 2023-10-18T21:28:30.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f90279c5bf0b56ca-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-72/obj4 2023-10-18T21:28:30.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-72/obj4 2023-10-18T21:28:30.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:30.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-72/obj4 2023-10-18T21:28:30.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.481 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:30.481 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.482 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.482 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.482 INFO:tasks.rgw_multisite_tests:/bofrnf-72/obj4 2023-10-18T21:28:30.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.483 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:jhhFnQiWed2XW4K32aAiuQwSemk= 2023-10-18T21:28:30.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:jhhFnQiWed2XW4K32aAiuQwSemk=', 'Content-Length': '0'} 2023-10-18T21:28:30.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000433297b9d04513a4-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/ 2023-10-18T21:28:30.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/ 2023-10-18T21:28:30.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:30.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/ 2023-10-18T21:28:30.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.487 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:30.487 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.487 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.487 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.488 INFO:tasks.rgw_multisite_tests:/bofrnf-73/ 2023-10-18T21:28:30.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.488 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:DDlach/lXAFB1SpYZbwx49fBsWA= 2023-10-18T21:28:30.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:DDlach/lXAFB1SpYZbwx49fBsWA=', 'Content-Length': '0'} 2023-10-18T21:28:30.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9417b7d0f227af0-0065304cee-294e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/obj1 2023-10-18T21:28:30.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/obj1 2023-10-18T21:28:30.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:30.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/obj1 2023-10-18T21:28:30.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.492 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:30.493 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.493 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.493 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.494 INFO:tasks.rgw_multisite_tests:/bofrnf-73/obj1 2023-10-18T21:28:30.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.494 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:ku4YHXBYUynJuaJ/krWnU8zN7tk= 2023-10-18T21:28:30.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:ku4YHXBYUynJuaJ/krWnU8zN7tk=', 'Content-Length': '0'} 2023-10-18T21:28:30.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-72:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10007.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cab357aa5db6f731-0065304cee-294e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/obj1 2023-10-18T21:28:30.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/obj1 2023-10-18T21:28:30.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:30.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/obj1 2023-10-18T21:28:30.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.499 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:30.499 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.499 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.500 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.500 INFO:tasks.rgw_multisite_tests:/bofrnf-73/obj1 2023-10-18T21:28:30.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.501 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:nrsWaT4GRvgt5Tvlf+U3F+DR2cU= 2023-10-18T21:28:30.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:nrsWaT4GRvgt5Tvlf+U3F+DR2cU=', 'Content-Length': '0'} 2023-10-18T21:28:30.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-72:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10007.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003399749df10947e5-0065304cee-294e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/obj2 2023-10-18T21:28:30.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/obj2 2023-10-18T21:28:30.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:30.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/obj2 2023-10-18T21:28:30.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.505 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:30.505 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.506 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.506 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.506 INFO:tasks.rgw_multisite_tests:/bofrnf-73/obj2 2023-10-18T21:28:30.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.507 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:1CoaVFxANtno665i9A937VDJfVo= 2023-10-18T21:28:30.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:1CoaVFxANtno665i9A937VDJfVo=', 'Content-Length': '0'} 2023-10-18T21:28:30.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-72:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10007.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e17146e02229f7f-0065304cee-294e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/obj2 2023-10-18T21:28:30.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/obj2 2023-10-18T21:28:30.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-18T21:28:30.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/obj2 2023-10-18T21:28:30.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.512 INFO:tasks.rgw_multisite_tests:GET 2023-10-18T21:28:30.512 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.512 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.513 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.513 INFO:tasks.rgw_multisite_tests:/bofrnf-73/obj2 2023-10-18T21:28:30.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.514 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:MwNp4cYaPBbhHAgJZwIn1XIj8sA= 2023-10-18T21:28:30.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:MwNp4cYaPBbhHAgJZwIn1XIj8sA=', 'Content-Length': '0'} 2023-10-18T21:28:30.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 18 Oct 2023 21:21:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '403a921b-fcfc-4ab2-b434-bf29d35a51b6:bofrnf-72:403a921b-fcfc-4ab2-b434-bf29d35a51b6.10007.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d28e6914bcf38789-0065304cee-294e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/bofrnf-73/obj3 2023-10-18T21:28:30.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/bofrnf-73/obj3 2023-10-18T21:28:30.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-18T21:28:30.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /bofrnf-73/obj3 2023-10-18T21:28:30.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-18T21:28:30.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-18T21:28:30.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi082.front.sepia.ceph.com:8001 2023-10-18T21:28:30.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-18T21:28:30.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-18T21:28:30.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-18T21:28:30.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-18T21:28:30.518 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-18T21:28:30.519 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.519 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.519 INFO:tasks.rgw_multisite_tests:Wed, 18 Oct 2023 21:23:58 GMT 2023-10-18T21:28:30.519 INFO:tasks.rgw_multisite_tests:/bofrnf-73/obj3 2023-10-18T21:28:30.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-18T21:28:30.520 INFO:tasks.rgw_multisite_tests:AWS 6TC2VRGERC8I036KRP19:o168U9TmEXs9dhXUwqxH5TBjGPQ= 2023-10-18T21:28:30.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 18 Oct 2023 21:23:58 GMT', 'Authorization': 'AWS 6TC2VRGERC8I036KRP19:o168U9TmEXs9dhXUwqxH5TBjGPQ=', 'Content-Length': '0'} 2023-10-18T21:28:30.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001b2da60537272b9e-0065304cee-294e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 18 Oct 2023 21:23:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-18T21:28:30.521 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-18T21:28:30.522 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.522 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-18T21:28:30.522 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 5942.939s 2023-10-18T21:28:30.523 INFO:tasks.rgw_multisite_tests: 2023-10-18T21:28:30.523 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=5) 2023-10-18T21:28:30.524 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-10-18T21:28:30.590 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=83dca322ab0f40e88905fbe1a67312e0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_07c1e9af8fb194e5ad8a71499ead549c30345076/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-10-18T21:28:30.592 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-10-18T21:28:30.603 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-10-18T21:28:30.619 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-18T21:28:30.628 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-18T21:28:30.629 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:30.633 INFO:tasks.rgw.c1.client.0.smithi082.stdout:2023-10-18T21:28:30.629+0000 7f83ac18f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 122366) UID: 0 2023-10-18T21:28:30.633 INFO:tasks.rgw.c1.client.0.smithi082.stdout:2023-10-18T21:28:30.629+0000 7f83b03f87c0 -1 shutting down 2023-10-18T21:28:36.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-18T21:28:36.731 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-18T21:28:36.732 DEBUG:teuthology.orchestra.run.smithi082:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-10-18T21:28:36.754 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/vault-root-token 2023-10-18T21:28:36.827 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-18T21:28:36.828 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:36.869 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T21:28:36.868+0000 7fc3e4c06700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 180930) UID: 0 2023-10-18T21:28:36.869 INFO:tasks.rgw.c1.client.1.smithi082.stdout:2023-10-18T21:28:36.868+0000 7fc3e8e6f7c0 -1 shutting down 2023-10-18T21:28:42.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-18T21:28:42.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-18T21:28:42.930 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-18T21:28:42.930 DEBUG:teuthology.orchestra.run.smithi082:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-10-18T21:28:42.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/vault-root-token 2023-10-18T21:28:43.024 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-18T21:28:43.025 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:43.026 INFO:tasks.rgw.c2.client.0.smithi156.stdout:2023-10-18T21:28:43.025+0000 7f29ef34a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 121490) UID: 0 2023-10-18T21:28:43.026 INFO:tasks.rgw.c2.client.0.smithi156.stdout:2023-10-18T21:28:43.025+0000 7f29f35b37c0 -1 shutting down 2023-10-18T21:28:47.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-18T21:28:48.352 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-18T21:28:49.127 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-18T21:28:49.128 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-10-18T21:28:49.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/vault-root-token 2023-10-18T21:28:49.228 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-18T21:28:49.229 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:49.270 INFO:tasks.rgw.c2.client.1.smithi156.stdout:2023-10-18T21:28:49.269+0000 7f07233f9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 122786) UID: 0 2023-10-18T21:28:49.271 INFO:tasks.rgw.c2.client.1.smithi156.stdout:2023-10-18T21:28:49.269+0000 7f07276627c0 -1 shutting down 2023-10-18T21:28:53.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-18T21:28:53.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-18T21:28:55.331 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-18T21:28:55.332 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-10-18T21:28:55.354 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/vault-root-token 2023-10-18T21:28:55.426 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-18T21:28:55.437 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-10-18T21:28:55.811 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-18T21:28:55.812 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-18T21:28:55.812 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-10-18T21:28:55.813 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:55.869 INFO:tasks.ceph.c2.osd.0:Stopped 2023-10-18T21:28:55.869 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-10-18T21:28:55.870 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:55.916 INFO:tasks.ceph.c2.osd.1:Stopped 2023-10-18T21:28:55.917 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-10-18T21:28:55.917 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:55.966 INFO:tasks.ceph.c2.osd.2:Stopped 2023-10-18T21:28:55.966 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-18T21:28:55.967 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-10-18T21:28:55.967 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:55.991 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-10-18T21:28:55.992 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-18T21:28:55.992 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-10-18T21:28:55.993 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:56.050 INFO:tasks.ceph.c2.mon.a:Stopped 2023-10-18T21:28:56.051 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-18T21:28:56.051 DEBUG:teuthology.orchestra.run.smithi156:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-10-18T21:28:56.120 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T21:28:56.120 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-10-18T21:28:56.571 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T21:28:56.572 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-10-18T21:28:56.669 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T21:28:56.670 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-10-18T21:28:56.748 INFO:tasks.ceph:Archiving mon data... 2023-10-18T21:28:56.750 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650/data/c2.mon.a.tgz 2023-10-18T21:28:56.750 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2023-10-18T21:28:56.768 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.gKiUbJBnRl 2023-10-18T21:28:56.769 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.gKiUbJBnRl 2023-10-18T21:28:56.928 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.gKiUbJBnRl 2023-10-18T21:28:57.016 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.gKiUbJBnRl is 354KB 2023-10-18T21:28:57.087 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.gKiUbJBnRl 2023-10-18T21:28:57.104 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-18T21:28:57.106 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-18T21:28:57.108 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-18T21:28:57.171 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-18T21:28:57.183 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-10-18T21:28:57.534 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-18T21:28:57.535 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-18T21:28:57.536 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-10-18T21:28:57.536 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:57.582 INFO:tasks.ceph.c1.osd.0:Stopped 2023-10-18T21:28:57.583 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-10-18T21:28:57.583 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:57.630 INFO:tasks.ceph.c1.osd.1:Stopped 2023-10-18T21:28:57.631 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-10-18T21:28:57.631 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:57.677 INFO:tasks.ceph.c1.osd.2:Stopped 2023-10-18T21:28:57.677 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-18T21:28:57.678 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-10-18T21:28:57.678 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:57.702 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-10-18T21:28:57.703 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-18T21:28:57.703 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-10-18T21:28:57.704 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T21:28:57.762 INFO:tasks.ceph.c1.mon.a:Stopped 2023-10-18T21:28:57.763 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-18T21:28:57.763 DEBUG:teuthology.orchestra.run.smithi082:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-10-18T21:28:57.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T21:28:57.832 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-10-18T21:28:58.132 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T21:28:58.132 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-10-18T21:28:58.231 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T21:28:58.232 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-10-18T21:28:58.311 INFO:tasks.ceph:Archiving mon data... 2023-10-18T21:28:58.312 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650/data/c1.mon.a.tgz 2023-10-18T21:28:58.312 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2023-10-18T21:28:58.330 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.6XbVqBhfqY 2023-10-18T21:28:58.331 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.6XbVqBhfqY 2023-10-18T21:28:58.511 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.6XbVqBhfqY 2023-10-18T21:28:58.599 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.6XbVqBhfqY is 422KB 2023-10-18T21:28:58.694 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.6XbVqBhfqY 2023-10-18T21:28:58.712 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-18T21:28:58.713 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-18T21:28:58.754 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-18T21:28:58.787 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-18T21:28:58.787 DEBUG:teuthology.orchestra.run.smithi082:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-18T21:28:58.821 DEBUG:teuthology.orchestra.run.smithi156:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-18T21:28:58.846 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-18T21:28:58.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-18T21:28:58.860 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-18T21:28:58.866 INFO:tasks.ceph:Archiving crash dumps... 2023-10-18T21:28:58.869 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650/remote/smithi082/crash 2023-10-18T21:28:58.870 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-18T21:28:58.919 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650/remote/smithi156/crash 2023-10-18T21:28:58.920 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-18T21:28:58.950 INFO:tasks.ceph:Compressing logs... 2023-10-18T21:28:58.951 DEBUG:teuthology.orchestra.run.smithi082:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T21:28:58.964 DEBUG:teuthology.orchestra.run.smithi156:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T21:32:13.544 INFO:tasks.ceph:Archiving logs... 2023-10-18T21:32:13.545 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650/remote/smithi082/log 2023-10-18T21:32:13.547 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-18T21:32:28.324 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650/remote/smithi156/log 2023-10-18T21:32:28.325 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-18T21:32:50.699 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-18T21:32:50.709 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-18T21:32:50.710 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-18T21:32:50.713 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-18T21:32:50.805 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-18T21:32:50.816 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-18T21:32:50.816 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T21:32:50.819 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T21:32:50.836 INFO:teuthology.orchestra.run.smithi082.stderr:bash: ntpq: command not found 2023-10-18T21:32:50.839 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2023-10-18T21:32:50.841 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T21:32:50.841 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2023-10-18T21:32:50.842 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv01.front.sepia.ceph.com 3 7 377 113 +3119us[+3230us] +/- 57ms 2023-10-18T21:32:50.842 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -3786us[-3786us] +/- 64ms 2023-10-18T21:32:50.842 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 114 -16us[ +95us] +/- 81ms 2023-10-18T21:32:50.843 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T21:32:50.844 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T21:32:50.844 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-10-18T21:32:50.844 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv01.front.sepia.ceph.com 3 8 377 246 +3207us[+3304us] +/- 55ms 2023-10-18T21:32:50.845 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -3733us[-3733us] +/- 64ms 2023-10-18T21:32:50.845 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 115 +72us[ +72us] +/- 81ms 2023-10-18T21:32:50.845 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T21:32:50.846 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T21:32:50.856 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T21:32:50.857 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T21:32:50.867 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T21:32:50.982 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T21:32:51.079 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T21:32:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T21:32:51.114 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T21:32:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T21:32:51.148 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T21:32:51.164 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T21:32:51.174 INFO:teuthology.task.internal:Duration was 7163.481165 seconds 2023-10-18T21:32:51.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T21:32:51.185 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T21:32:51.186 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T21:32:51.188 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T21:32:51.234 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T21:32:51.236 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T21:32:51.410 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T21:32:51.411 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2023-10-18T21:32:51.411 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T21:32:51.434 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2023-10-18T21:32:51.435 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T21:32:51.457 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T21:32:51.457 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T21:32:51.477 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T21:32:51.535 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T21:32:51.535 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T21:32:51.555 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T21:32:51.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T21:32:51.653 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T21:32:51.654 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T21:32:51.680 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T21:32:51.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T21:32:51.719 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T21:32:51.723 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T21:32:51.747 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2023-10-18T21:32:51.775 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2023-10-18T21:32:51.798 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T21:32:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T21:32:51.827 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T21:32:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T21:32:51.857 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T21:32:51.868 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T21:32:51.869 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650/remote/smithi082 2023-10-18T21:32:51.869 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T21:32:51.908 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650/remote/smithi156 2023-10-18T21:32:51.909 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T21:32:51.946 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T21:32:51.947 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T21:32:51.949 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T21:32:51.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T21:32:52.012 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T21:32:52.012 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T21:32:52.023 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T21:32:52.023 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T21:32:52.026 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T21:32:52.041 INFO:teuthology.orchestra.run.smithi082.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 21:32 /home/ubuntu/cephtest 2023-10-18T21:32:52.057 INFO:teuthology.orchestra.run.smithi156.stdout: 525846 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 21:32 /home/ubuntu/cephtest 2023-10-18T21:32:52.058 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T21:32:52.070 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T21:32:52.093 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T21:32:52.123 DEBUG:teuthology.nuke:shortname: smithi082 2023-10-18T21:32:52.123 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T21:32:52.149 DEBUG:teuthology.nuke:shortname: smithi156 2023-10-18T21:32:52.149 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T21:32:52.153 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 19:24:55.861670', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNvyj4AREUB5UtfXpWW0uvjhICdkbhKvR7vk+X63lvYbeR9OxwLhNt/cWMOwhoABu8Uzx6tjPXC/XE/KcEEbuYc='} 2023-10-18T21:32:52.173 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_19:20:22-rgw-wip-yuri-testing-2023-10-18-0812-reef-distro-default-smithi/7431650', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 19:24:55.863459', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEoPyxVfTkY/yghFEHuNDraaBPv1OdiYKJ4EOPuHZdPIGA5KTcBjR8Q6W0QaX3LPal9vMY0E9N+bsBvCnjgETOg='} 2023-10-18T21:32:52.207 INFO:teuthology.orchestra.console.smithi082:Power off 2023-10-18T21:32:52.208 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T21:32:52.232 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2023-10-18T21:32:52.233 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T21:32:52.252 INFO:teuthology.orchestra.console.smithi156:Power off 2023-10-18T21:32:52.253 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T21:32:52.265 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2023-10-18T21:32:52.278 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2023-10-18T21:32:52.278 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T21:32:52.304 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-10-18T21:32:56.267 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T21:32:56.306 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T21:32:56.380 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2023-10-18T21:32:56.419 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-10-18T21:33:00.382 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T21:33:00.421 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T21:33:00.494 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2023-10-18T21:33:00.533 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2023-10-18T21:33:00.595 INFO:teuthology.orchestra.console.smithi082:Power off completed 2023-10-18T21:33:00.634 INFO:teuthology.orchestra.console.smithi156:Power off completed 2023-10-18T21:33:00.739 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} duration: 7163.48116517067 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=83dca322ab0f40e88905fbe1a67312e0 status: fail success: false 2023-10-18T21:33:00.740 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T21:33:00.825 INFO:teuthology.run:FAIL